Merge pull request 'feat(parity): rsync parity tracks 1-6 (protocol 2.28.0)' (#303) from feat/parity-2.28 into dev
CI / lint (push) Successful in 2m0s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 21s
CI / sanitizers (address) (push) Successful in 51s
CI / build-and-test (push) Successful in 1m7s
CI / sanitizers (undefined) (push) Successful in 43s
CI / fuzz-build (push) Successful in 47s
CI / coverage (push) Successful in 43s
CI / valgrind (push) Successful in 2m13s
CI / lint (push) Successful in 2m0s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 21s
CI / sanitizers (address) (push) Successful in 51s
CI / build-and-test (push) Successful in 1m7s
CI / sanitizers (undefined) (push) Successful in 43s
CI / fuzz-build (push) Successful in 47s
CI / coverage (push) Successful in 43s
CI / valgrind (push) Successful in 2m13s
This commit was merged in pull request #303.
This commit is contained in:
@@ -4,6 +4,37 @@ All notable changes to FastSync are documented here. Versions match
|
|||||||
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
|
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
|
||||||
run the same version because the handshake is strict.
|
run the same version because the handshake is strict.
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **`--delete` now defaults to delete-during (rsync `--del`) timing.** With no
|
||||||
|
explicit timing flag, a plain `--delete` removes each directory's extras as
|
||||||
|
that directory is processed instead of committing one whole-tree deletion only
|
||||||
|
after the entire transfer succeeds. This matches rsync, frees destination
|
||||||
|
space progressively, and avoids the whole-old+new-tree peak that could
|
||||||
|
`ENOSPC` a tight destination. The client maps the default onto the existing
|
||||||
|
`delete_during` wire boolean, so `PROTOCOL_VERSION` stays `2.28.0`.
|
||||||
|
- Added the FastSync-only long option **`--delete-commit`** (implies
|
||||||
|
`--delete`): it selects the old late whole-tree commit and is timing-identical
|
||||||
|
to `--delete-after` (the same `delete_after` wire boolean). Explicit timing
|
||||||
|
flags always win over the default, at most one timing flag may be given, and a
|
||||||
|
timing flag combined with `--no-delete` is still rejected.
|
||||||
|
- The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag:
|
||||||
|
the one-shot per-run config block (protected prefixes, size-pruned mirrors,
|
||||||
|
`--delete-missing-args` exact paths) is now always transmitted first on a
|
||||||
|
config-only carrier (`apply=false`), fixing a latent bug where a
|
||||||
|
`--delete-missing-args` run whose `--files-from` list synchronized no directory
|
||||||
|
never sent its exact deletions.
|
||||||
|
|
||||||
|
### Migration
|
||||||
|
|
||||||
|
- Scripts that relied on plain `--delete` deleting nothing until the transfer
|
||||||
|
fully succeeded must pass **`--delete-commit`** (or `--delete-after`) to keep
|
||||||
|
that behavior. Plain `--delete` now removes reached directories' extras during
|
||||||
|
the transfer, exactly like rsync's default; on a completed run the final tree
|
||||||
|
is unchanged.
|
||||||
|
|
||||||
## [2.26.0] - 2026-09-17
|
## [2.26.0] - 2026-09-17
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
cmake_minimum_required(VERSION 3.22)
|
cmake_minimum_required(VERSION 3.22)
|
||||||
|
|
||||||
project(FastFileTransfer VERSION 2.27.0)
|
project(FastFileTransfer VERSION 2.28.0)
|
||||||
|
|
||||||
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
|
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
|
||||||
set(CMAKE_C_STANDARD 11)
|
set(CMAKE_C_STANDARD 11)
|
||||||
|
|||||||
+115
-4
@@ -52,9 +52,10 @@
|
|||||||
`tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
|
`tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
|
||||||
over daemon/TCP (no argv channel in FastSync's binary config handshake;
|
over daemon/TCP (no argv channel in FastSync's binary config handshake;
|
||||||
rsync-daemon differential pins the rsync behavior) and receiver-side
|
rsync-daemon differential pins the rsync behavior) and receiver-side
|
||||||
`protect`/`risk` re-derivation for destination-only entries (would need a
|
`protect`/`risk` re-derivation for destination-only entries (would need a
|
||||||
receiver filter engine; differential pins the divergence). The options pass
|
receiver filter engine; differential pins the divergence — **reversed by
|
||||||
stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
|
track 4a below**, which adds that engine). The options pass
|
||||||
|
stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
|
||||||
holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
|
holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
|
||||||
`--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
|
`--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
|
||||||
|
|
||||||
@@ -74,13 +75,123 @@
|
|||||||
heuristic with a 10× size window, not rsync's matcher), but its residual is the
|
heuristic with a 10× size window, not rsync's matcher), but its residual is the
|
||||||
candidate-selection heuristic itself: the final tree is byte-exact by design, so
|
candidate-selection heuristic itself: the final tree is byte-exact by design, so
|
||||||
it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
|
it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
|
||||||
differential. The parity-review pass then moved `--delete-delay` to ⚠️ (the
|
differential. (Track 5b later found the name heuristic is rsync's own and moved
|
||||||
|
the row ❌ → ⚠️, leaving only the narrower delta size window; see entry 15.) The parity-review pass then moved `--delete-delay` to ⚠️ (the
|
||||||
plan-time `--max-delete` charge and non-recursive deferred removal differ from
|
plan-time `--max-delete` charge and non-recursive deferred removal differ from
|
||||||
rsync when a snapshotted entry fails removal). Differential-gate allowlist
|
rsync when a snapshotted entry fails removal). Differential-gate allowlist
|
||||||
entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
|
entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
|
||||||
integrated stats+options+fs branch stands at **111 ✅ / 13 ⚠️ / 33 ❌ = 157**;
|
integrated stats+options+fs branch stands at **111 ✅ / 13 ⚠️ / 33 ❌ = 157**;
|
||||||
full suite + ASan + clang-format + cppcheck clean.
|
full suite + ASan + clang-format + cppcheck clean.
|
||||||
|
|
||||||
|
11. **No-wire parity track 1** on `feat/parity-2.28` (no protocol change):
|
||||||
|
`-n --delete` now sends the same filter-excluded + size-pruned protected
|
||||||
|
prefixes and synchronized-directory scope as a real run (dry-run would-delete
|
||||||
|
matches rsync for source-derived protections; the destination-only exclude
|
||||||
|
residual was later closed by track 4a, readdir ordering remains);
|
||||||
|
`--delete-delay` now charges
|
||||||
|
`--max-delete` on actual removals and re-scans a queued directory at commit
|
||||||
|
to remove content created after the plan, with an independent deferred-list
|
||||||
|
cap (only partial-delete ordering remains); and `--info=name2` emits `NAME is
|
||||||
|
uptodate` plus the leading `./` root name line for `--info=name` (only the
|
||||||
|
root-line trigger condition and receiver-side `skip` wording remain). Matrix
|
||||||
|
now **111 ✅ / 14 ⚠️ / 32 ❌ = 157**; differential + unit tests added in
|
||||||
|
`test_features.py`, `test_option_parity.py`, `test_delete_plan.c`,
|
||||||
|
`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`.
|
||||||
|
12. **No-wire parity track 2b** on `feat/parity-2.28` (no protocol change):
|
||||||
|
`--progress`/`-P`/`--info=progress` (when not `--quiet`) now run an opt-in
|
||||||
|
paths-only metadata pre-count (no file reads/hashing) that supplies rsync's
|
||||||
|
full file-list total for the `to-chk` denominator and the directory names,
|
||||||
|
and emits per-directory/symlink/special name lines, in both the sequential
|
||||||
|
and `--threads` paths. `--delete-during`/`--delete-delay` reuse their
|
||||||
|
keep-set pre-scan instead of a second walk; non-progress runs are
|
||||||
|
unaffected. Differential tests (`progress`/`progress_threads` over a new
|
||||||
|
`multidir` corpus) match rsync's name set and `to-chk` denominator on a
|
||||||
|
fresh transfer, and the single-file byte-identical test still passes;
|
||||||
|
emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream)
|
||||||
|
plus re-run over-naming (unconditional `./`, ancestor dirs named with a
|
||||||
|
transferred child, and no quick-check for symlinks/empty dirs) remain the
|
||||||
|
caveats, so the row stays ⚠️ and the matrix is unchanged at
|
||||||
|
**111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
|
||||||
|
|
||||||
|
13. **Wire parity track 4a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||||
|
`2.28.0`): the receiver now has a delete-time filter engine. The sender
|
||||||
|
compiles its root-level selection rules exactly as the scanner does
|
||||||
|
(`filter_base_build`) and streams them as one bounded, self-describing
|
||||||
|
config-frame block (action, sides, anchored, dir-only, negate, owner,
|
||||||
|
pattern; bounded rule count and pattern bytes, unknown action/sides is a
|
||||||
|
protocol error). The receiver reconstructs `protect_rules` and applies them
|
||||||
|
first-match-wins to each extraneous destination path in every delete timing
|
||||||
|
(the whole-tree commit walker, the `--delete-during`/`--delete-delay`
|
||||||
|
per-directory plans, and the `-n` would-delete enumeration), so a
|
||||||
|
`P *.log` rule protects a destination-only `extra.log` like rsync (with
|
||||||
|
`risk` cancelling); the sender-derived protected-prefix behavior is
|
||||||
|
preserved when no rules are sent and `--delete-excluded` semantics are
|
||||||
|
unchanged. Per-directory merge (`:`/`.`) receiver re-derivation remains the
|
||||||
|
residual. `TestFilterProtect` (real + dry-run) plus differential cases
|
||||||
|
`filter_protect`, `filter_protect_during`, `filter_protect_delay` added and
|
||||||
|
the `--filter=RULE` row moves ❌ → ✅: matrix now
|
||||||
|
**115 ✅ / 11 ⚠️ / 31 ❌ = 157**; unit tests, the three named integration
|
||||||
|
files, clang-format and cppcheck clean.
|
||||||
|
|
||||||
|
14. **Wire parity track 5a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||||
|
`2.28.0` by project decision): the three basis-dir options now default to
|
||||||
|
rsync's metadata quick-check (equal size + equal mtime, or size alone under
|
||||||
|
`--size-only`; `-I` disables matching) instead of FastSync's historical
|
||||||
|
xxHash64 content equality, so a same-size/different-content basis is trusted
|
||||||
|
exactly as rsync trusts it. A new FastSync-only, long-only `--verify-basis`
|
||||||
|
flag restores the strict whole-file content equality; its bool is appended to
|
||||||
|
the basis block of the config frame (golden wire frame 882 → 886 bytes).
|
||||||
|
`--verify-basis` streams the confined basis descriptor to hash it, and a
|
||||||
|
basis hit is no longer capped at the 256 MiB whole-file payload bound:
|
||||||
|
`--copy-dest` streams the basis through a bounded buffer and `--link-dest`'s
|
||||||
|
copy fallback streams from the basis, so an over-limit hit materializes (a
|
||||||
|
basis MISS still falls back to the normal transfer and keeps its own bound).
|
||||||
|
A `--copy-dest` hit re-applies the SOURCE attributes (the sender transmits
|
||||||
|
the source metadata with the basis check frame), matching rsync's
|
||||||
|
"copy then fix attributes"; a `--link-dest` success keeps the shared inode's
|
||||||
|
attributes (writing through it would mutate the basis). Differential cases
|
||||||
|
`copy_dest` and `verify_basis` added; `test_basis_dir_size_only_content_residual`
|
||||||
|
converted to a passing parity assertion; `TestBasisDestDirs` updated for the
|
||||||
|
new default + `--verify-basis`; unit tests cover the quick-check/verify
|
||||||
|
decision and the same-size/different-content handshake. The
|
||||||
|
`--compare-dest`/`--copy-dest`/`--link-dest` rows move ❌ → ⚠️ (relative-DIR
|
||||||
|
resolution base and over-limit MISS refusal): matrix now
|
||||||
|
**116 ✅ / 13 ⚠️ / 28 ❌ = 157**.
|
||||||
|
|
||||||
|
15. **No-wire parity track 5b** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||||
|
`2.28.0` by project decision): `-y`/`--fuzzy` reclassified ❌ → ⚠️. A probe
|
||||||
|
against real rsync 3.4.1 (pinned `-B8192`, repeated-content 64 KiB corpus)
|
||||||
|
showed the name heuristic is already rsync's (`util1.c fuzzy_distance` /
|
||||||
|
`find_filename_suffix` + the exact size+mtime pass) and the output is always
|
||||||
|
byte-exact; the only residual is candidate ELIGIBILITY, because FastSync's
|
||||||
|
`delta_should_attempt` gate caps the size ratio at 10× and requires both
|
||||||
|
files ≥ 16 KiB while rsync will reuse a basis from 0.25× to 10000× and below
|
||||||
|
16 KiB. The choice is observable only as `--stats` bandwidth counters. Added
|
||||||
|
differential case `fuzzy_basis` (same-suffix sibling, one name edit,
|
||||||
|
identical content, block size pinned) asserting tree **and** normalized
|
||||||
|
`--stats` parity where the choices coincide, plus `TestFuzzy` pinning the
|
||||||
|
window boundary on both sides (>10× and <16 KiB siblings declined by
|
||||||
|
FastSync while rsync uses them, both trees byte-identical). Matrix now
|
||||||
|
**116 ✅ / 14 ⚠️ / 27 ❌ = 157**.
|
||||||
|
|
||||||
|
16. **Lockstep delete-default track 6** on `feat/parity-2.28` (`PROTOCOL_VERSION`
|
||||||
|
stays `2.28.0`): plain `--delete` now defaults to rsync's delete-during
|
||||||
|
(`--del`) timing, normalized on the client onto the existing `delete_during`
|
||||||
|
wire bool. The old late whole-tree commit is opt-in via `--delete-after` or
|
||||||
|
the FastSync-only long `--delete-commit` (identical `delete_after` timing).
|
||||||
|
`-d/--dirs` still falls back to the end commit, `--delay-updates` still
|
||||||
|
deletes before publication, and `--files-from`/`-R` scope is unchanged. The
|
||||||
|
`STATUS_DELETE_PLAN` frame gained a one-int `apply` flag so the per-run
|
||||||
|
config block (including `--delete-missing-args` exact paths) is always
|
||||||
|
transmitted, on a config-only carrier when the scope allows no directory
|
||||||
|
plan — fixing a latent bug with a file-only `--files-from` list. Differential
|
||||||
|
cases `delete`/`delete_commit`/`filter_protect_after` plus the extended
|
||||||
|
`test_delete_timing_parity.py` (plain `--delete` mid-abort removes reached
|
||||||
|
extras, `--delete-commit` defers) pass; full `-m "not setpriv"` suite,
|
||||||
|
clang-format and cppcheck clean. Matrix unchanged at
|
||||||
|
**116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay
|
||||||
|
⚠️ for the abort boundary; `--delete-after` stays ✅).
|
||||||
|
|
||||||
## Next steps
|
## Next steps
|
||||||
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
|
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
|
||||||
2. **Deferred security items** (documented, not implemented):
|
2. **Deferred security items** (documented, not implemented):
|
||||||
|
|||||||
@@ -113,13 +113,18 @@ matrix is classified as parity, caveat, or divergent in
|
|||||||
(`-B1000`, `-essh`, `-MOPT`, `--opt=value`) are accepted, matching rsync.
|
(`-B1000`, `-essh`, `-MOPT`, `--opt=value`) are accepted, matching rsync.
|
||||||
- `-r`, `-b`, `-L`, and `-B` are parsed with the rsync short names.
|
- `-r`, `-b`, `-L`, and `-B` are parsed with the rsync short names.
|
||||||
- `--stats` prints the counters FastSync can observe plus the receiver-only
|
- `--stats` prints the counters FastSync can observe plus the receiver-only
|
||||||
counters (`Matched data`, deleted files) reported over the wire; rsync's
|
counters reported over the wire (`Matched data`, deleted files, and the
|
||||||
per-type `Number of files` breakdown is not reproduced. `--progress` prints
|
created/literal counters); `Number of files` and `Number of created files`
|
||||||
rsync-style per-file blocks (without rsync's leading `./` line).
|
carry rsync's per-type breakdown. `--progress` prints rsync-style per-file
|
||||||
|
blocks including the leading `./` line, and (when progress is requested) a
|
||||||
|
paths-only pre-count supplies rsync's `to-chk` denominator.
|
||||||
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
|
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
|
||||||
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums, negotiated with
|
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums. `auto` honors
|
||||||
`auto`; `zlibx` behaves as `zlib`, and the transfer checksum is not separately
|
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` and otherwise follows rsync's
|
||||||
selectable.
|
compiled-in order. An omitted `--compress-level` uses the codec's rsync
|
||||||
|
default (zstd 3, zlib/zlibx 6, lz4 ignored); `zlib`/`zlibx` share the
|
||||||
|
literal-only zlib path (rsync's zlibx semantics), and the transfer checksum is
|
||||||
|
not separately selectable.
|
||||||
|
|
||||||
The detailed flag matrix is maintained in
|
The detailed flag matrix is maintained in
|
||||||
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It reports each row as **parity**,
|
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It reports each row as **parity**,
|
||||||
@@ -202,11 +207,13 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
|||||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`) |
|
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`) |
|
||||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination |
|
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination |
|
||||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win) |
|
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win) |
|
||||||
| `--delete` | Delete files on receiver not present in source (default timing: delete-after, i.e. only after the whole transfer succeeded). Scoped to the synchronized directories, so `--files-from` subsets are safe |
|
| `--verify-basis` | FastSync-only: require a basis hit (`--compare-dest`/`--copy-dest`/`--link-dest`) to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync) |
|
||||||
|
| `--delete` | Delete files on receiver not present in source (default timing: delete-during, matching rsync, so destination space is freed progressively). Scoped to the synchronized directories, so `--files-from` subsets are safe |
|
||||||
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
|
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
|
||||||
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
|
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
|
||||||
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`) |
|
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`) |
|
||||||
| `--delete-after` | Explicit delete-after timing (implies `--delete`) |
|
| `--delete-after` | Explicit delete-after timing (implies `--delete`) |
|
||||||
|
| `--delete-commit` | FastSync-only: keep the pre-2.28 atomic timing — delete only after the whole transfer succeeded (identical timing to `--delete-after`) |
|
||||||
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
|
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
|
||||||
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
|
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
|
||||||
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication) |
|
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication) |
|
||||||
@@ -505,8 +512,8 @@ features without changing the meaning of ordinary compatibility options.
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. |
|
| `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. |
|
||||||
| `-z [level]`, `--compress [level]` | Enable streaming compression (default `zstd`), levels 1-22. |
|
| `-z [level]`, `--compress [level]` | Enable streaming compression (default `zstd`), levels 1-22. |
|
||||||
| `--compress-level <n>` | Set the compression level. |
|
| `--compress-level <n>` | Set the compression level (1-22). Omitted, each codec uses its rsync default: zstd 3, zlib/zlibx 6, lz4 ignored. |
|
||||||
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlibx` behaves as `zlib`. |
|
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlib`/`zlibx` share the same literal-only zlib path. |
|
||||||
| `--zl <n>` | Alias for `--compress-level`. |
|
| `--zl <n>` | Alias for `--compress-level`. |
|
||||||
| `--skip-compress <list>` | Skip compression for `/`- or `,`-separated suffixes; defaults to rsync 3.4.1's built-in list. Incompatible with `--chunk-serialization`. |
|
| `--skip-compress <list>` | Skip compression for `/`- or `,`-separated suffixes; defaults to rsync 3.4.1's built-in list. Incompatible with `--chunk-serialization`. |
|
||||||
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
|
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
|
||||||
@@ -564,6 +571,7 @@ remote SSH argv is already built injection-safe.
|
|||||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`). |
|
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`). |
|
||||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination. |
|
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination. |
|
||||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win). |
|
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win). |
|
||||||
|
| `--verify-basis` | FastSync-only: require a basis hit to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync). |
|
||||||
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
|
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
|
||||||
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
|
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
|
||||||
| `--append-verify` | Like `--append`, but verifies the retained prefix checksum first (falls back to a full transfer on mismatch). |
|
| `--append-verify` | Like `--append`, but verifies the retained prefix checksum first (falls back to a full transfer on mismatch). |
|
||||||
@@ -571,6 +579,7 @@ remote SSH argv is already built injection-safe.
|
|||||||
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
|
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
|
||||||
| `--delete-during`, `--del` | Delete extras once the keep-set manifest is known, before data is applied (implies `--delete`; early mode, same engine behaviour as `--delete-before`). |
|
| `--delete-during`, `--del` | Delete extras once the keep-set manifest is known, before data is applied (implies `--delete`; early mode, same engine behaviour as `--delete-before`). |
|
||||||
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`; commit mode, same behaviour as `--delete-after`). |
|
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`; commit mode, same behaviour as `--delete-after`). |
|
||||||
|
| `--delete-commit` | FastSync-only: atomic delete-after timing (only after the whole transfer succeeded). |
|
||||||
| `--delete-after` | Explicit delete-after timing: delete only after the transfer succeeded (implies `--delete`). |
|
| `--delete-after` | Explicit delete-after timing: delete only after the transfer succeeded (implies `--delete`). |
|
||||||
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected). |
|
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected). |
|
||||||
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync. |
|
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync. |
|
||||||
|
|||||||
+81
-45
File diff suppressed because one or more lines are too long
@@ -75,6 +75,14 @@ bool change_list_enabled(const Config* config) {
|
|||||||
(config->info_level & LOG_INFO_NAME) != 0);
|
(config->info_level & LOG_INFO_NAME) != 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Emitted once, lazily, ahead of the first --info=name entry: rsync prints the
|
||||||
|
* transfer-root `./` name line when the root directory is (re)created. */
|
||||||
|
static bool name_root_printed = false;
|
||||||
|
|
||||||
|
void change_reset_name_root(void) {
|
||||||
|
name_root_printed = false;
|
||||||
|
}
|
||||||
|
|
||||||
/* ---- Itemize code ---- */
|
/* ---- Itemize code ---- */
|
||||||
|
|
||||||
/* Format the permission bits as an `ls -l` string, e.g. `-rw-r--r--`. */
|
/* Format the permission bits as an `ls -l` string, e.g. `-rw-r--r--`. */
|
||||||
@@ -212,6 +220,23 @@ static char* change_render_name(const ChangeEvent* event) {
|
|||||||
return line.data;
|
return line.data;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* rsync's `--info=name2` line for an unchanged entry: `NAME is uptodate`. */
|
||||||
|
static char* change_render_name_uptodate(const ChangeEvent* event) {
|
||||||
|
char* name = change_render_name(event);
|
||||||
|
if (name == NULL)
|
||||||
|
return NULL;
|
||||||
|
size_t length = strlen(name);
|
||||||
|
char* line = malloc(length + sizeof(" is uptodate"));
|
||||||
|
if (line == NULL) {
|
||||||
|
free(name);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
memcpy(line, name, length);
|
||||||
|
memcpy(line + length, " is uptodate", sizeof(" is uptodate"));
|
||||||
|
free(name);
|
||||||
|
return line;
|
||||||
|
}
|
||||||
|
|
||||||
/* ---- --out-format / --log-file-format ---- */
|
/* ---- --out-format / --log-file-format ---- */
|
||||||
|
|
||||||
/* rsync 3.4.1's `%C` uses the negotiated TRANSFER checksum (the first name of a
|
/* rsync 3.4.1's `%C` uses the negotiated TRANSFER checksum (the first name of a
|
||||||
@@ -461,10 +486,24 @@ static void print_escaped_line(FILE* stream, const char* line, bool eight_bit_ou
|
|||||||
void change_emit(const Config* config, const ChangeEvent* event) {
|
void change_emit(const Config* config, const ChangeEvent* event) {
|
||||||
if (event == NULL || !change_list_enabled(config))
|
if (event == NULL || !change_list_enabled(config))
|
||||||
return;
|
return;
|
||||||
if (event->decision == CHANGE_UP_TO_DATE)
|
|
||||||
return;
|
|
||||||
bool to_stdout = config->itemize_changes || config->out_format != NULL;
|
bool to_stdout = config->itemize_changes || config->out_format != NULL;
|
||||||
bool to_log = config->log_file != NULL && config->log_file_format != NULL;
|
bool to_log = config->log_file != NULL && config->log_file_format != NULL;
|
||||||
|
bool progress_active = config->show_progress || (config->info_level & LOG_INFO_PROGRESS);
|
||||||
|
if (event->decision == CHANGE_UP_TO_DATE) {
|
||||||
|
/* --info=name2 prints `NAME is uptodate` for entries the receiver already
|
||||||
|
had. An itemize/out-format run reports them through its own format (or
|
||||||
|
not at all), the progress stream has no frame for them, and neither the
|
||||||
|
itemize nor the log-file stream previously reported an up-to-date entry,
|
||||||
|
so nothing else here changes. */
|
||||||
|
if (!to_stdout && (config->info_level & LOG_INFO_NAME_UPTODATE) != 0 && !progress_active) {
|
||||||
|
char* line = change_render_name_uptodate(event);
|
||||||
|
if (line != NULL) {
|
||||||
|
print_escaped_line(stdout, line, config->eight_bit_output);
|
||||||
|
free(line);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
if (to_stdout) {
|
if (to_stdout) {
|
||||||
char* line = config->out_format != NULL
|
char* line = config->out_format != NULL
|
||||||
? change_render_format(config->out_format, config, event)
|
? change_render_format(config->out_format, config, event)
|
||||||
@@ -473,11 +512,15 @@ void change_emit(const Config* config, const ChangeEvent* event) {
|
|||||||
print_escaped_line(stdout, line, config->eight_bit_output);
|
print_escaped_line(stdout, line, config->eight_bit_output);
|
||||||
free(line);
|
free(line);
|
||||||
}
|
}
|
||||||
} else if ((config->info_level & LOG_INFO_NAME) != 0 &&
|
} else if ((config->info_level & LOG_INFO_NAME) != 0 && !progress_active) {
|
||||||
!(config->show_progress || (config->info_level & LOG_INFO_PROGRESS))) {
|
|
||||||
/* --info=name without -i/--out-format: print the updated entry's name. The
|
/* --info=name without -i/--out-format: print the updated entry's name. The
|
||||||
--progress path owns the name line when progress output is active (it
|
--progress path owns the name line when progress output is active (it
|
||||||
emits the same names before the progress frames), so do not duplicate. */
|
emits the same names before the progress frames), so do not duplicate.
|
||||||
|
The transfer-root `./` line precedes the first such name. */
|
||||||
|
if (!name_root_printed) {
|
||||||
|
name_root_printed = true;
|
||||||
|
fputs("./\n", stdout);
|
||||||
|
}
|
||||||
char* line = change_render_name(event);
|
char* line = change_render_name(event);
|
||||||
if (line != NULL) {
|
if (line != NULL) {
|
||||||
print_escaped_line(stdout, line, config->eight_bit_output);
|
print_escaped_line(stdout, line, config->eight_bit_output);
|
||||||
@@ -643,6 +686,29 @@ void change_emit_file_sent(const Config* config, const File* file) {
|
|||||||
change_emit_file_sent_bytes(config, file, payload, 0);
|
change_emit_file_sent_bytes(config, file, payload, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void change_emit_file_uptodate(const Config* config, const File* file) {
|
||||||
|
if (file == NULL || !change_list_enabled(config))
|
||||||
|
return;
|
||||||
|
ChangeEvent event;
|
||||||
|
memset(&event, 0, sizeof(event));
|
||||||
|
event.decision = CHANGE_UP_TO_DATE;
|
||||||
|
event.is_directory = false;
|
||||||
|
event.is_symlink = file->is_symlink;
|
||||||
|
event.is_special = file->is_special;
|
||||||
|
event.is_hardlink = file->link_group != 0 && !file->link_first;
|
||||||
|
event.symlink_target = file->symlink_target;
|
||||||
|
event.hardlink_target = file->hardlink_target;
|
||||||
|
event.size = file->data != NULL ? file->data->size : 0;
|
||||||
|
event.dest = file->dest_state;
|
||||||
|
char* name = NULL;
|
||||||
|
char* path = NULL;
|
||||||
|
fill_event_from_file(config, file, &event, &name, &path);
|
||||||
|
if (name != NULL && path != NULL)
|
||||||
|
change_emit(config, &event);
|
||||||
|
free(name);
|
||||||
|
free(path);
|
||||||
|
}
|
||||||
|
|
||||||
void change_emit_dir_sent(const Config* config, const File* file) {
|
void change_emit_dir_sent(const Config* config, const File* file) {
|
||||||
if (file == NULL || !change_list_enabled(config))
|
if (file == NULL || !change_list_enabled(config))
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -102,4 +102,13 @@ void change_emit_file_sent(const Config* config, const File* file);
|
|||||||
/* Build and emit a CHANGE_SENT event for an explicit directory entry (-d). */
|
/* Build and emit a CHANGE_SENT event for an explicit directory entry (-d). */
|
||||||
void change_emit_dir_sent(const Config* config, const File* file);
|
void change_emit_dir_sent(const Config* config, const File* file);
|
||||||
|
|
||||||
|
/* Build and emit a CHANGE_UP_TO_DATE event for a file the receiver already had.
|
||||||
|
* With --info=name2 it renders rsync's "NAME is uptodate" line (no output
|
||||||
|
* otherwise). */
|
||||||
|
void change_emit_file_uptodate(const Config* config, const File* file);
|
||||||
|
|
||||||
|
/* Reset the lazy transfer-root `./` line emitted ahead of the first
|
||||||
|
* --info=name entry. Call once at the start of a transfer. */
|
||||||
|
void change_reset_name_root(void);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|||||||
+101
-12
@@ -161,10 +161,16 @@ static int set_compression_choice(Config* config, const char* value) {
|
|||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
int algo;
|
int algo;
|
||||||
if (strcasecmp(value, "auto") == 0)
|
if (strcasecmp(value, "auto") == 0) {
|
||||||
algo = (int)compression_negotiate_default();
|
algo = compression_choice_resolve();
|
||||||
else
|
if (algo < 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
|
||||||
|
config->cli_exit_code = 4;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
algo = compression_algo_from_name(value);
|
algo = compression_algo_from_name(value);
|
||||||
|
}
|
||||||
if (algo < 0) {
|
if (algo < 0) {
|
||||||
log_message(LOG_LEVEL_ERROR,
|
log_message(LOG_LEVEL_ERROR,
|
||||||
"--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, "
|
"--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, "
|
||||||
@@ -228,16 +234,25 @@ static int set_checksum_choice(Config* config, const char* value) {
|
|||||||
config->cli_exit_code = 4;
|
config->cli_exit_code = 4;
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
ChecksumAlgo negotiated = checksum_negotiate_default();
|
int negotiated = -1;
|
||||||
|
if (rc1 == 1 || rc2 == 1) {
|
||||||
|
negotiated = checksum_choice_resolve();
|
||||||
|
if (negotiated < 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
|
||||||
|
config->cli_exit_code = 4;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
if (rc1 == 1)
|
if (rc1 == 1)
|
||||||
transfer = (int)negotiated;
|
transfer = negotiated;
|
||||||
if (!name2)
|
if (!name2)
|
||||||
pre = transfer;
|
pre = transfer;
|
||||||
else if (rc2 == 1)
|
else if (rc2 == 1)
|
||||||
pre = (int)negotiated;
|
pre = negotiated;
|
||||||
|
|
||||||
config->checksum_algo = pre;
|
config->checksum_algo = pre;
|
||||||
config->checksum_transfer_algo = transfer;
|
config->checksum_transfer_algo = transfer;
|
||||||
|
config->checksum_choice_set = true;
|
||||||
/* rsync: "none" for the transfer checksum forces --whole-file. */
|
/* rsync: "none" for the transfer checksum forces --whole-file. */
|
||||||
if (transfer == (int)CHECKSUM_ALGO_NONE)
|
if (transfer == (int)CHECKSUM_ALGO_NONE)
|
||||||
config->whole_file = true;
|
config->whole_file = true;
|
||||||
@@ -614,9 +629,19 @@ static int parse_info_flags(const char* value, Config* config) {
|
|||||||
}
|
}
|
||||||
if (strcmp(name, "copy") == 0)
|
if (strcmp(name, "copy") == 0)
|
||||||
flag = LOG_INFO_COPY;
|
flag = LOG_INFO_COPY;
|
||||||
else if (strcmp(name, "name") == 0)
|
else if (strcmp(name, "name") == 0) {
|
||||||
flag = LOG_INFO_NAME;
|
/* name level 2 adds rsync's "is uptodate" lines. */
|
||||||
else if (strcmp(name, "misc") == 0)
|
if (level == 0)
|
||||||
|
parsed &= ~(uint32_t)(LOG_INFO_NAME | LOG_INFO_NAME_UPTODATE);
|
||||||
|
else {
|
||||||
|
parsed |= LOG_INFO_NAME;
|
||||||
|
if (level >= 2)
|
||||||
|
parsed |= LOG_INFO_NAME_UPTODATE;
|
||||||
|
else
|
||||||
|
parsed &= ~(uint32_t)LOG_INFO_NAME_UPTODATE;
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
} else if (strcmp(name, "misc") == 0)
|
||||||
flag = LOG_INFO_MISC;
|
flag = LOG_INFO_MISC;
|
||||||
else if (strcmp(name, "skip") == 0)
|
else if (strcmp(name, "skip") == 0)
|
||||||
flag = LOG_INFO_SKIP;
|
flag = LOG_INFO_SKIP;
|
||||||
@@ -972,6 +997,12 @@ static const OptionEntry OPTION_TABLE[] = {
|
|||||||
{"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)},
|
{"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)},
|
||||||
{"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)},
|
{"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)},
|
||||||
{"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)},
|
{"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)},
|
||||||
|
/* FastSync-only long spelling of the late whole-tree commit, which selects
|
||||||
|
the same timing as rsync's --delete-after in FastSync (the whole-tree
|
||||||
|
keep-set manifest is committed only after the entire transfer succeeded).
|
||||||
|
Plain --delete now defaults to delete-during, so this restores the old
|
||||||
|
FastSync behavior; it maps onto the same delete_after wire field. */
|
||||||
|
{"--delete-commit", NULL, OPT_FLAG, offsetof(Config, delete_after)},
|
||||||
{"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)},
|
{"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)},
|
||||||
{"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)},
|
{"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)},
|
||||||
{"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)},
|
{"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)},
|
||||||
@@ -1029,6 +1060,11 @@ static const OptionEntry OPTION_TABLE[] = {
|
|||||||
* --remote-option is parsed. --trust-sender is a local receiver policy and
|
* --remote-option is parsed. --trust-sender is a local receiver policy and
|
||||||
* never travels to the remote peer. */
|
* never travels to the remote peer. */
|
||||||
{"--trust-sender", NULL, OPT_FLAG, offsetof(Config, trust_sender)},
|
{"--trust-sender", NULL, OPT_FLAG, offsetof(Config, trust_sender)},
|
||||||
|
/* FastSync-only (not an rsync option): require a basis-hit's content to
|
||||||
|
* match the source by whole-file digest instead of trusting rsync's
|
||||||
|
* size+mtime quick-check. Long-only; crosses the wire so the receiver
|
||||||
|
* performs the extra read/hash. */
|
||||||
|
{"--verify-basis", NULL, OPT_FLAG, offsetof(Config, verify_basis)},
|
||||||
};
|
};
|
||||||
|
|
||||||
/* Only boolean options with no required argument are safe to negate. */
|
/* Only boolean options with no required argument are safe to negate. */
|
||||||
@@ -1071,6 +1107,7 @@ static const NegatableOption NEGATABLE_OPTIONS[] = {
|
|||||||
{"xattrs", "X", offsetof(Config, preserve_xattrs)},
|
{"xattrs", "X", offsetof(Config, preserve_xattrs)},
|
||||||
{"acls", "A", offsetof(Config, preserve_acls)},
|
{"acls", "A", offsetof(Config, preserve_acls)},
|
||||||
{"fake-super", NULL, offsetof(Config, fake_super)},
|
{"fake-super", NULL, offsetof(Config, fake_super)},
|
||||||
|
{"verify-basis", NULL, offsetof(Config, verify_basis)},
|
||||||
};
|
};
|
||||||
|
|
||||||
static bool opt_is(const char* arg, const char* name, const char* alias) {
|
static bool opt_is(const char* arg, const char* name, const char* alias) {
|
||||||
@@ -1454,6 +1491,8 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
|
|||||||
ctx->exit_code = -1;
|
ctx->exit_code = -1;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
if (entry->offset == offsetof(Config, compression_level))
|
||||||
|
config->compression_level_set = true;
|
||||||
if (entry->offset == offsetof(Config, chmod_spec)) {
|
if (entry->offset == offsetof(Config, chmod_spec)) {
|
||||||
mode_t ignored;
|
mode_t ignored;
|
||||||
if (!chmod_apply(0, config->chmod_spec, &ignored)) {
|
if (!chmod_apply(0, config->chmod_spec, &ignored)) {
|
||||||
@@ -1480,7 +1519,9 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
|
|||||||
if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX)
|
if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX)
|
||||||
config->per_dir_filter_count++;
|
config->per_dir_filter_count++;
|
||||||
/* A delete-timing flag selects when --delete removes extras, so it
|
/* A delete-timing flag selects when --delete removes extras, so it
|
||||||
implies --delete exactly like the rsync options do. */
|
implies --delete exactly like the rsync options do. --delete-commit (the
|
||||||
|
FastSync-only late-commit spelling) is mapped onto delete_after and so is
|
||||||
|
covered here too. */
|
||||||
if (entry->offset == offsetof(Config, delete_before) ||
|
if (entry->offset == offsetof(Config, delete_before) ||
|
||||||
entry->offset == offsetof(Config, delete_during) ||
|
entry->offset == offsetof(Config, delete_during) ||
|
||||||
entry->offset == offsetof(Config, delete_delay) ||
|
entry->offset == offsetof(Config, delete_delay) ||
|
||||||
@@ -1738,6 +1779,7 @@ static bool cli_handle_transfer_flags(CliParseCtx* ctx) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
config->compression_level = (int)level;
|
config->compression_level = (int)level;
|
||||||
|
config->compression_level_set = true;
|
||||||
log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level);
|
log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level);
|
||||||
ctx->i++;
|
ctx->i++;
|
||||||
}
|
}
|
||||||
@@ -2533,6 +2575,18 @@ static bool cli_handle_outbuf_option(CliParseCtx* ctx) {
|
|||||||
* -1 on error. */
|
* -1 on error. */
|
||||||
static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) {
|
static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) {
|
||||||
set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
|
set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
|
||||||
|
/* rsync's plain --delete defaults to delete-during (--del): each directory's
|
||||||
|
extras are removed as that directory is processed, so space is freed
|
||||||
|
progressively and a tight destination never has to hold the whole old+new
|
||||||
|
tree at once. The late whole-tree commit FastSync historically used is
|
||||||
|
still selected explicitly by --delete-after or by the FastSync-only long
|
||||||
|
spelling --delete-commit (an exact alias for --delete-after). Resolve the
|
||||||
|
default on the client, before validation and before the config crosses the
|
||||||
|
wire, so exactly one timing flag is ever set; an explicit timing (including
|
||||||
|
--delete-commit) always wins. */
|
||||||
|
if (config->use_delete && !config->delete_before && !config->delete_during &&
|
||||||
|
!config->delete_delay && !config->delete_after)
|
||||||
|
config->delete_during = true;
|
||||||
if (config->compress_choice) {
|
if (config->compress_choice) {
|
||||||
int algo = compression_algo_from_name(config->compress_choice);
|
int algo = compression_algo_from_name(config->compress_choice);
|
||||||
if (algo >= 0) {
|
if (algo >= 0) {
|
||||||
@@ -2540,8 +2594,43 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
|
|||||||
config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE);
|
config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (config->use_compression && config->compression_algo == (int)COMPRESSION_ALGO_NONE)
|
/* A bare -z (no --compress-choice) resolves like rsync's "auto": the
|
||||||
config->compression_algo = (int)compression_negotiate_default();
|
* RSYNC_COMPRESS_LIST preference list first, then the compiled-in order. A
|
||||||
|
* list that names no supported codec is rsync's failed negotiation (exit 4). */
|
||||||
|
if (config->use_compression && !config->compress_choice) {
|
||||||
|
int resolved = compression_choice_resolve();
|
||||||
|
if (resolved < 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
|
||||||
|
config->cli_exit_code = 4;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
config->compression_algo = resolved;
|
||||||
|
if (resolved == (int)COMPRESSION_ALGO_NONE)
|
||||||
|
config->use_compression = false;
|
||||||
|
}
|
||||||
|
/* Apply rsync's per-codec compression level: an explicit --compress-level is
|
||||||
|
* clamped to the codec's range, otherwise the codec's own default is used. */
|
||||||
|
if (config->use_compression) {
|
||||||
|
CompressionAlgo algo = (CompressionAlgo)config->compression_algo;
|
||||||
|
config->compression_level = config->compression_level_set
|
||||||
|
? compression_clamp_level(algo, config->compression_level)
|
||||||
|
: compression_default_level(algo);
|
||||||
|
log_debug_message(LOG_DEBUG_UTIL, "Client compression: %s (level %d)",
|
||||||
|
compression_algo_name(algo), config->compression_level);
|
||||||
|
}
|
||||||
|
/* The negotiated checksum is always resolved (rsync negotiates one for the
|
||||||
|
* delta strong sum even without --checksum): RSYNC_CHECKSUM_LIST first, then
|
||||||
|
* the compiled-in order. An explicit --checksum-choice already set it. */
|
||||||
|
if (!config->checksum_choice_set) {
|
||||||
|
int resolved = checksum_choice_resolve();
|
||||||
|
if (resolved < 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
|
||||||
|
config->cli_exit_code = 4;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
config->checksum_algo = resolved;
|
||||||
|
config->checksum_transfer_algo = resolved;
|
||||||
|
}
|
||||||
/* rsync parity: "none" as the pre-transfer checksum cannot be combined with
|
/* rsync parity: "none" as the pre-transfer checksum cannot be combined with
|
||||||
* --checksum (exit 4). The check runs here because --checksum may appear on
|
* --checksum (exit 4). The check runs here because --checksum may appear on
|
||||||
* either side of --checksum-choice. */
|
* either side of --checksum-choice. */
|
||||||
|
|||||||
+408
-126
@@ -93,12 +93,11 @@ static const char* stats_bytes(const Config* config, unsigned long long bytes, c
|
|||||||
return buffer;
|
return buffer;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Build rsync's `Number of files` parenthetical: each non-zero category, in
|
/* Build rsync's per-type parenthetical: each non-zero category, in
|
||||||
reg/dir/link/special order. Empty when the flist counted nothing. */
|
reg/dir/link/special order. Empty when every count is zero. */
|
||||||
static void stats_type_breakdown(const TransferStats* stats, char* out, size_t out_size) {
|
static void type_breakdown(unsigned long long reg, unsigned long long dir, unsigned long long link,
|
||||||
unsigned long long total =
|
unsigned long long special, char* out, size_t out_size) {
|
||||||
stats->flist_reg + stats->flist_dir + stats->flist_link + stats->flist_special;
|
if (reg + dir + link + special == 0) {
|
||||||
if (total == 0) {
|
|
||||||
out[0] = '\0';
|
out[0] = '\0';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
@@ -107,10 +106,7 @@ static void stats_type_breakdown(const TransferStats* stats, char* out, size_t o
|
|||||||
const struct {
|
const struct {
|
||||||
const char* name;
|
const char* name;
|
||||||
unsigned long long count;
|
unsigned long long count;
|
||||||
} parts[4] = {{"reg", stats->flist_reg},
|
} parts[4] = {{"reg", reg}, {"dir", dir}, {"link", link}, {"special", special}};
|
||||||
{"dir", stats->flist_dir},
|
|
||||||
{"link", stats->flist_link},
|
|
||||||
{"special", stats->flist_special}};
|
|
||||||
bool first = true;
|
bool first = true;
|
||||||
for (size_t i = 0; i < 4; i++) {
|
for (size_t i = 0; i < 4; i++) {
|
||||||
if (parts[i].count == 0)
|
if (parts[i].count == 0)
|
||||||
@@ -127,6 +123,12 @@ static void stats_type_breakdown(const TransferStats* stats, char* out, size_t o
|
|||||||
out[used] = '\0';
|
out[used] = '\0';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Build rsync's `Number of files` parenthetical from the scan's flist counts. */
|
||||||
|
static void stats_type_breakdown(const TransferStats* stats, char* out, size_t out_size) {
|
||||||
|
type_breakdown(stats->flist_reg, stats->flist_dir, stats->flist_link, stats->flist_special, out,
|
||||||
|
out_size);
|
||||||
|
}
|
||||||
|
|
||||||
/* Print the rsync `--stats` block on stdout. The source-side flist and
|
/* Print the rsync `--stats` block on stdout. The source-side flist and
|
||||||
transferred counters come from `stats` (filled while scanning/sending), the
|
transferred counters come from `stats` (filled while scanning/sending), the
|
||||||
receiver-only counters from the STATUS_STATS frame, and the wire byte totals
|
receiver-only counters from the STATUS_STATS frame, and the wire byte totals
|
||||||
@@ -151,6 +153,7 @@ static void report_transfer_stats(const Config* config, const TransferStats* sta
|
|||||||
char total_buffer[32];
|
char total_buffer[32];
|
||||||
char transferred_buffer[32];
|
char transferred_buffer[32];
|
||||||
char literal_buffer[32];
|
char literal_buffer[32];
|
||||||
|
char matched_buffer[32];
|
||||||
char sent_buffer[32];
|
char sent_buffer[32];
|
||||||
char recv_buffer[32];
|
char recv_buffer[32];
|
||||||
char rate_buffer[32] = {0};
|
char rate_buffer[32] = {0};
|
||||||
@@ -159,8 +162,15 @@ static void report_transfer_stats(const Config* config, const TransferStats* sta
|
|||||||
stats_bytes(config, stats->total_file_size, total_buffer, sizeof(total_buffer));
|
stats_bytes(config, stats->total_file_size, total_buffer, sizeof(total_buffer));
|
||||||
const char* transferred = stats_bytes(config, stats->transferred_file_size, transferred_buffer,
|
const char* transferred = stats_bytes(config, stats->transferred_file_size, transferred_buffer,
|
||||||
sizeof(transferred_buffer));
|
sizeof(transferred_buffer));
|
||||||
const char* literal =
|
/* Protocol 2.28.0: the receiver reports the bytes it literally stored, which
|
||||||
stats_bytes(config, stats->literal_data, literal_buffer, sizeof(literal_buffer));
|
is exact for a delta transfer (the sender's own literal_data counts each
|
||||||
|
stored file's whole source size and is only an upper bound). Fall back to
|
||||||
|
the sender total when the receiver reported no delta/literal accounting
|
||||||
|
(e.g. a local no-server path). */
|
||||||
|
unsigned long long literal_bytes = (recv->literal_bytes != 0 || recv->matched_data != 0)
|
||||||
|
? recv->literal_bytes
|
||||||
|
: stats->literal_data;
|
||||||
|
const char* literal = stats_bytes(config, literal_bytes, literal_buffer, sizeof(literal_buffer));
|
||||||
const char* sent_s = stats_bytes(config, sent, sent_buffer, sizeof(sent_buffer));
|
const char* sent_s = stats_bytes(config, sent, sent_buffer, sizeof(sent_buffer));
|
||||||
const char* recv_s = stats_bytes(config, received, recv_buffer, sizeof(recv_buffer));
|
const char* recv_s = stats_bytes(config, received, recv_buffer, sizeof(recv_buffer));
|
||||||
const char* rate_str = rate_buffer;
|
const char* rate_str = rate_buffer;
|
||||||
@@ -177,21 +187,30 @@ static void report_transfer_stats(const Config* config, const TransferStats* sta
|
|||||||
stats_type_breakdown(stats, breakdown, sizeof(breakdown));
|
stats_type_breakdown(stats, breakdown, sizeof(breakdown));
|
||||||
unsigned long long flist_total =
|
unsigned long long flist_total =
|
||||||
stats->flist_reg + stats->flist_dir + stats->flist_link + stats->flist_special;
|
stats->flist_reg + stats->flist_dir + stats->flist_link + stats->flist_special;
|
||||||
|
char created_breakdown[128];
|
||||||
|
type_breakdown(recv->created_reg, recv->created_dir, recv->created_link, recv->created_special,
|
||||||
|
created_breakdown, sizeof(created_breakdown));
|
||||||
|
unsigned long long created_total =
|
||||||
|
recv->created_reg + recv->created_dir + recv->created_link + recv->created_special;
|
||||||
printf("\n");
|
printf("\n");
|
||||||
if (breakdown[0] != '\0')
|
if (breakdown[0] != '\0')
|
||||||
printf("Number of files: %llu %s\n", flist_total, breakdown);
|
printf("Number of files: %llu %s\n", flist_total, breakdown);
|
||||||
else
|
else
|
||||||
printf("Number of files: %llu\n", flist_total);
|
printf("Number of files: %llu\n", flist_total);
|
||||||
/* FastSync cannot tell which entries the receiver newly created, so it
|
/* Protocol 2.28.0: the receiver reports which destination entries it newly
|
||||||
reports the transferred regular files (which are created on a fresh
|
created, split by type, so this line matches rsync exactly. */
|
||||||
destination). See RSYNC_COMPAT.md for the documented residual. */
|
if (created_breakdown[0] != '\0')
|
||||||
printf("Number of created files: %llu\n", stats->transferred_regular);
|
printf("Number of created files: %llu %s\n", created_total, created_breakdown);
|
||||||
|
else
|
||||||
|
printf("Number of created files: %llu\n", created_total);
|
||||||
printf("Number of deleted files: %llu\n", recv->deleted_files);
|
printf("Number of deleted files: %llu\n", recv->deleted_files);
|
||||||
printf("Number of regular files transferred: %llu\n", stats->transferred_regular);
|
printf("Number of regular files transferred: %llu\n", stats->transferred_regular);
|
||||||
printf("Total file size: %s bytes\n", total);
|
printf("Total file size: %s bytes\n", total);
|
||||||
printf("Total transferred file size: %s bytes\n", transferred);
|
printf("Total transferred file size: %s bytes\n", transferred);
|
||||||
printf("Literal data: %s bytes\n", literal);
|
printf("Literal data: %s bytes\n", literal);
|
||||||
printf("Matched data: %llu bytes\n", recv->matched_data);
|
const char* matched =
|
||||||
|
stats_bytes(config, recv->matched_data, matched_buffer, sizeof(matched_buffer));
|
||||||
|
printf("Matched data: %s bytes\n", matched);
|
||||||
printf("File list size: 0\n");
|
printf("File list size: 0\n");
|
||||||
printf("File list generation time: 0.000 seconds\n");
|
printf("File list generation time: 0.000 seconds\n");
|
||||||
printf("File list transfer time: 0.000 seconds\n");
|
printf("File list transfer time: 0.000 seconds\n");
|
||||||
@@ -256,10 +275,61 @@ static void transfer_stats_note_transferred(TransferStats* stats, const File* fi
|
|||||||
|
|
||||||
static const char* delete_display_path(const Config* config, const char* path);
|
static const char* delete_display_path(const Config* config, const char* path);
|
||||||
|
|
||||||
|
/* Paths-only pre-count of the source file list, built once at transfer start
|
||||||
|
* when progress output is requested. rsync's `to-chk` denominator is the whole
|
||||||
|
* file list -- every regular file, directory, symlink and special plus the
|
||||||
|
* transfer root -- while the streaming scan never emits directories. A
|
||||||
|
* metadata-only walk (no file reads, no hashing) supplies that total and the
|
||||||
|
* directory names, so the opt-in pass leaves non-progress runs untouched. */
|
||||||
|
typedef struct {
|
||||||
|
unsigned long long total;
|
||||||
|
ArrayList* dir_paths; /* owned char* in transfer-relative display form */
|
||||||
|
} ProgressPrecount;
|
||||||
|
|
||||||
static bool g_progress_active;
|
static bool g_progress_active;
|
||||||
static unsigned long long g_progress_xferred;
|
static unsigned long long g_progress_xferred;
|
||||||
static unsigned long long g_progress_seen;
|
static unsigned long long g_progress_index;
|
||||||
|
static unsigned long long g_progress_total;
|
||||||
static struct timespec g_progress_file_start;
|
static struct timespec g_progress_file_start;
|
||||||
|
static ProgressPrecount g_progress_precount;
|
||||||
|
static PathIndex g_progress_dir_index;
|
||||||
|
static bool g_progress_dir_index_valid;
|
||||||
|
static StrHashSet g_progress_emitted;
|
||||||
|
static bool g_progress_emitted_valid;
|
||||||
|
static ArrayList* g_progress_emitted_keys;
|
||||||
|
|
||||||
|
static bool progress_requested(const Config* config) {
|
||||||
|
return config != NULL && !config->quiet &&
|
||||||
|
(config->show_progress || (config->info_level & LOG_INFO_PROGRESS) != 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void progress_precount_dispose(ProgressPrecount* p) {
|
||||||
|
if (p->dir_paths != NULL) {
|
||||||
|
array_list_delete(p->dir_paths);
|
||||||
|
p->dir_paths = NULL;
|
||||||
|
}
|
||||||
|
p->total = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void client_progress_cleanup(void) {
|
||||||
|
if (g_progress_dir_index_valid) {
|
||||||
|
path_index_free(&g_progress_dir_index);
|
||||||
|
g_progress_dir_index_valid = false;
|
||||||
|
}
|
||||||
|
if (g_progress_emitted_valid) {
|
||||||
|
str_hash_set_free(&g_progress_emitted);
|
||||||
|
g_progress_emitted_valid = false;
|
||||||
|
}
|
||||||
|
if (g_progress_emitted_keys != NULL) {
|
||||||
|
array_list_delete(g_progress_emitted_keys);
|
||||||
|
g_progress_emitted_keys = NULL;
|
||||||
|
}
|
||||||
|
progress_precount_dispose(&g_progress_precount);
|
||||||
|
g_progress_active = false;
|
||||||
|
g_progress_total = 0;
|
||||||
|
g_progress_index = 0;
|
||||||
|
g_progress_xferred = 0;
|
||||||
|
}
|
||||||
|
|
||||||
static void progress_first_frame(unsigned long long size, char* out, size_t out_size) {
|
static void progress_first_frame(unsigned long long size, char* out, size_t out_size) {
|
||||||
char ofs_buf[32];
|
char ofs_buf[32];
|
||||||
@@ -296,12 +366,12 @@ static void progress_final_frame(unsigned long long size, char* out, size_t out_
|
|||||||
unsigned long long remain = (unsigned long long)(diff_ms / 1000);
|
unsigned long long remain = (unsigned long long)(diff_ms / 1000);
|
||||||
snprintf(rembuf, sizeof(rembuf), "%4u:%02u:%02u", (unsigned)(remain / 3600),
|
snprintf(rembuf, sizeof(rembuf), "%4u:%02u:%02u", (unsigned)(remain / 3600),
|
||||||
(unsigned)((remain / 60) % 60), (unsigned)(remain % 60));
|
(unsigned)((remain / 60) % 60), (unsigned)(remain % 60));
|
||||||
/* rsync's `to-chk` denominator is the whole file list, which includes the
|
/* rsync's `to-chk` denominator is the whole file list (the pre-count); the
|
||||||
transfer-root directory FastSync never emits as a transfer entry. Count
|
numerator falls as each entry is processed, root first. Without a
|
||||||
that root entry so a single-file transfer matches rsync exactly. */
|
pre-count (the paths-only walk failed) fall back to the transferred-file
|
||||||
unsigned long long total = g_progress_seen + 1;
|
count so the single-file layout stays intact. */
|
||||||
unsigned long long to_chk =
|
unsigned long long total = g_progress_total > 0 ? g_progress_total : g_progress_xferred + 1;
|
||||||
g_progress_seen > g_progress_xferred ? g_progress_seen - g_progress_xferred : 0;
|
unsigned long long to_chk = total > g_progress_index ? total - g_progress_index - 1 : 0;
|
||||||
snprintf(out, out_size, "\r%15s %3d%% %7.2f%s %s (xfr#%llu, to-chk=%llu/%llu)\n", ofs_buf, 100,
|
snprintf(out, out_size, "\r%15s %3d%% %7.2f%s %s (xfr#%llu, to-chk=%llu/%llu)\n", ofs_buf, 100,
|
||||||
rate, units, rembuf, g_progress_xferred, to_chk, total);
|
rate, units, rembuf, g_progress_xferred, to_chk, total);
|
||||||
}
|
}
|
||||||
@@ -349,11 +419,73 @@ static void print_delete_reports(const Config* config, const ArrayList* paths) {
|
|||||||
fflush(stdout);
|
fflush(stdout);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static void client_progress_emit_ancestors(const Config* config, const char* rel) {
|
||||||
|
if (!g_progress_dir_index_valid || !g_progress_emitted_valid || g_progress_emitted_keys == NULL ||
|
||||||
|
rel == NULL)
|
||||||
|
return;
|
||||||
|
size_t rel_len = strlen(rel);
|
||||||
|
for (size_t i = 0; i < rel_len; i++) {
|
||||||
|
if (rel[i] != '/')
|
||||||
|
continue;
|
||||||
|
char* prefix = malloc(i + 1);
|
||||||
|
if (prefix == NULL)
|
||||||
|
return;
|
||||||
|
memcpy(prefix, rel, i);
|
||||||
|
prefix[i] = '\0';
|
||||||
|
if (path_index_contains(&g_progress_dir_index, prefix) &&
|
||||||
|
!str_hash_set_lookup(&g_progress_emitted, prefix)) {
|
||||||
|
char* key = str_dup(prefix);
|
||||||
|
if (key != NULL && array_list_add(g_progress_emitted_keys, key)) {
|
||||||
|
str_hash_set_insert_ref(&g_progress_emitted, key);
|
||||||
|
char* escaped = output_escape(prefix, config->eight_bit_output);
|
||||||
|
printf("%s/\n", escaped ? escaped : prefix);
|
||||||
|
free(escaped);
|
||||||
|
g_progress_index++;
|
||||||
|
} else {
|
||||||
|
free(key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
free(prefix);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* rsync's --info=name/progress line for one entry: transfer-relative name (a
|
||||||
|
* trailing slash for directories) plus the ` -> target` symlink suffix. */
|
||||||
|
static char* progress_entry_line(const File* file, const char* rel) {
|
||||||
|
const char* arrow = NULL;
|
||||||
|
const char* target = NULL;
|
||||||
|
if (file->is_symlink && file->symlink_target != NULL) {
|
||||||
|
arrow = " -> ";
|
||||||
|
target = file->symlink_target;
|
||||||
|
} else if (file->link_group != 0 && !file->link_first && file->hardlink_target != NULL) {
|
||||||
|
arrow = " => ";
|
||||||
|
target = file->hardlink_target;
|
||||||
|
}
|
||||||
|
size_t rel_len = strlen(rel);
|
||||||
|
bool dir_slash = file->is_dir && (rel_len == 0 || rel[rel_len - 1] != '/');
|
||||||
|
size_t extra = (dir_slash ? 1u : 0u) + (target != NULL ? 4u + strlen(target) : 0u);
|
||||||
|
char* line = malloc(rel_len + extra + 1);
|
||||||
|
if (line == NULL)
|
||||||
|
return NULL;
|
||||||
|
memcpy(line, rel, rel_len);
|
||||||
|
size_t off = rel_len;
|
||||||
|
if (dir_slash)
|
||||||
|
line[off++] = '/';
|
||||||
|
if (target != NULL) {
|
||||||
|
memcpy(line + off, arrow, 4);
|
||||||
|
off += 4;
|
||||||
|
memcpy(line + off, target, strlen(target));
|
||||||
|
off += strlen(target);
|
||||||
|
}
|
||||||
|
line[off] = '\0';
|
||||||
|
return line;
|
||||||
|
}
|
||||||
|
|
||||||
static void client_progress_begin(const Config* config) {
|
static void client_progress_begin(const Config* config) {
|
||||||
g_progress_active =
|
change_reset_name_root();
|
||||||
(config->show_progress || info_flag_enabled(config, LOG_INFO_PROGRESS)) && !config->quiet;
|
g_progress_active = progress_requested(config);
|
||||||
g_progress_xferred = 0;
|
g_progress_xferred = 0;
|
||||||
g_progress_seen = 0;
|
g_progress_index = 1; /* the transfer root is file-list entry #0 */
|
||||||
if (!g_progress_active) {
|
if (!g_progress_active) {
|
||||||
/* `--info=flist` prints rsync's file-list header even without progress. */
|
/* `--info=flist` prints rsync's file-list header even without progress. */
|
||||||
if (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST)) {
|
if (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST)) {
|
||||||
@@ -375,12 +507,14 @@ static void client_progress_begin(const Config* config) {
|
|||||||
static void client_progress_file(const Config* config, const File* file) {
|
static void client_progress_file(const Config* config, const File* file) {
|
||||||
if (!g_progress_active || file == NULL || !file->data)
|
if (!g_progress_active || file == NULL || !file->data)
|
||||||
return;
|
return;
|
||||||
g_progress_seen++;
|
|
||||||
g_progress_xferred++;
|
g_progress_xferred++;
|
||||||
unsigned long long size = file->data->size;
|
unsigned long long size = file->data->size;
|
||||||
if (!config->itemize_changes && config->out_format == NULL) {
|
if (!config->itemize_changes && config->out_format == NULL) {
|
||||||
const char* name = delete_display_path(config, file_wire_path(file));
|
const char* rel = delete_display_path(config, file_wire_path(file));
|
||||||
printf("%s\n", name ? name : "");
|
client_progress_emit_ancestors(config, rel);
|
||||||
|
char* escaped = output_escape(rel, config->eight_bit_output);
|
||||||
|
printf("%s\n", escaped ? escaped : (rel ? rel : ""));
|
||||||
|
free(escaped);
|
||||||
}
|
}
|
||||||
clock_gettime(CLOCK_MONOTONIC, &g_progress_file_start);
|
clock_gettime(CLOCK_MONOTONIC, &g_progress_file_start);
|
||||||
char frame[160];
|
char frame[160];
|
||||||
@@ -388,9 +522,41 @@ static void client_progress_file(const Config* config, const File* file) {
|
|||||||
fputs(frame, stdout);
|
fputs(frame, stdout);
|
||||||
progress_final_frame(size, frame, sizeof(frame));
|
progress_final_frame(size, frame, sizeof(frame));
|
||||||
fputs(frame, stdout);
|
fputs(frame, stdout);
|
||||||
|
g_progress_index++;
|
||||||
fflush(stdout);
|
fflush(stdout);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Emit the name line for a transferred non-regular entry (directory, symlink,
|
||||||
|
* special or hard-link sibling): rsync prints these in the file list but has no
|
||||||
|
* progress frame for them. */
|
||||||
|
static void client_progress_name(const Config* config, const File* file) {
|
||||||
|
if (!g_progress_active || file == NULL)
|
||||||
|
return;
|
||||||
|
const char* rel = delete_display_path(config, file_wire_path(file));
|
||||||
|
if (!config->itemize_changes && config->out_format == NULL) {
|
||||||
|
client_progress_emit_ancestors(config, rel);
|
||||||
|
char* line = progress_entry_line(file, rel ? rel : "");
|
||||||
|
if (line != NULL) {
|
||||||
|
char* escaped = output_escape(line, config->eight_bit_output);
|
||||||
|
printf("%s\n", escaped ? escaped : line);
|
||||||
|
free(escaped);
|
||||||
|
free(line);
|
||||||
|
fflush(stdout);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
g_progress_index++;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* An entry the receiver already had prints no name under --progress but still
|
||||||
|
* occupies a file-list slot in the `to-chk` numerator. */
|
||||||
|
static void client_progress_uptodate(const Config* config, const File* file) {
|
||||||
|
(void)config;
|
||||||
|
(void)file;
|
||||||
|
if (!g_progress_active)
|
||||||
|
return;
|
||||||
|
g_progress_index++;
|
||||||
|
}
|
||||||
|
|
||||||
/* Compiled scanner inputs that are shared read-only across scanner instances
|
/* Compiled scanner inputs that are shared read-only across scanner instances
|
||||||
* and, in -m mode, across worker threads. `base_filters` owns the compiled
|
* and, in -m mode, across worker threads. `base_filters` owns the compiled
|
||||||
* command-line + -C rules; the FileListSet allow-set lives in the Config.
|
* command-line + -C rules; the FileListSet allow-set lives in the Config.
|
||||||
@@ -535,6 +701,126 @@ static void prepared_scanner_destroy(PreparedScanner* prepared) {
|
|||||||
prepared->relative_prefix = NULL;
|
prepared->relative_prefix = NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static bool progress_precount_add_dir(ProgressPrecount* p, const char* path) {
|
||||||
|
if (path == NULL || path[0] == '\0')
|
||||||
|
return true;
|
||||||
|
char* dup = str_dup(path);
|
||||||
|
if (dup == NULL)
|
||||||
|
return false;
|
||||||
|
if (array_list_add(p->dir_paths, dup))
|
||||||
|
return true;
|
||||||
|
free(dup);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Metadata-only walk collecting the full file-list total and every directory
|
||||||
|
* name. It uses its own scanner (fresh filter compilation and hard-link table)
|
||||||
|
* so the data pass's link-group state is never perturbed. */
|
||||||
|
static bool progress_precount_scan(const Config* config, ProgressPrecount* out) {
|
||||||
|
out->dir_paths = array_list_create(free);
|
||||||
|
if (out->dir_paths == NULL)
|
||||||
|
return false;
|
||||||
|
out->total = 0;
|
||||||
|
PreparedScanner prepared;
|
||||||
|
memset(&prepared, 0, sizeof(prepared));
|
||||||
|
if (!prepare_scanner(config, 0, &prepared)) {
|
||||||
|
progress_precount_dispose(out);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
ScannerOptions local = prepared.options;
|
||||||
|
local.list_dirs = true;
|
||||||
|
local.note_nonreg = false;
|
||||||
|
local.use_metadata = false;
|
||||||
|
local.preserve_xattrs = false;
|
||||||
|
local.preserve_acls = false;
|
||||||
|
local.checksum = false;
|
||||||
|
local.capture_dir_times = false;
|
||||||
|
local.excluded_paths = NULL;
|
||||||
|
local.size_skipped_paths = NULL;
|
||||||
|
local.synced_dirs = NULL;
|
||||||
|
local.plan_dirs = NULL;
|
||||||
|
local.dir_entries = NULL;
|
||||||
|
local.dir_entries_mutex = NULL;
|
||||||
|
local.hardlinks = NULL;
|
||||||
|
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
|
||||||
|
bool ok = scanner != NULL;
|
||||||
|
if (scanner != NULL) {
|
||||||
|
Chunk* chunk;
|
||||||
|
while (ok && (chunk = directory_scanner_next(scanner)) != NULL) {
|
||||||
|
out->total += (unsigned long long)chunk->element_count;
|
||||||
|
for (int i = 0; i < chunk->element_count && ok; i++) {
|
||||||
|
const File* f = chunk->items[i];
|
||||||
|
if (f != NULL && f->is_dir)
|
||||||
|
ok = progress_precount_add_dir(out, delete_display_path(config, file_wire_path(f)));
|
||||||
|
}
|
||||||
|
chunk_destroy(chunk);
|
||||||
|
}
|
||||||
|
if (ok && directory_scanner_failed(scanner))
|
||||||
|
ok = false;
|
||||||
|
directory_scanner_destroy(scanner);
|
||||||
|
}
|
||||||
|
prepared_scanner_destroy(&prepared);
|
||||||
|
if (!ok) {
|
||||||
|
progress_precount_dispose(out);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
out->total += 1; /* the transfer root "." */
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Reuse the --delete-during/--delete-delay keep-set pre-scan: its traversed
|
||||||
|
* directory list already holds every directory and `non_dir_count` the entries
|
||||||
|
* counted during that same pass, so progress costs no second walk. */
|
||||||
|
static bool progress_precount_from_plan_dirs(const Config* config, const ArrayList* plan_dirs,
|
||||||
|
unsigned long long non_dir_count,
|
||||||
|
ProgressPrecount* out) {
|
||||||
|
out->dir_paths = array_list_create(free);
|
||||||
|
if (out->dir_paths == NULL)
|
||||||
|
return false;
|
||||||
|
out->total = non_dir_count + 1;
|
||||||
|
for (int i = 0; i < plan_dirs->size; i++) {
|
||||||
|
const char* path = (const char*)plan_dirs->items[i];
|
||||||
|
const char* rel = config->send_directory != NULL
|
||||||
|
? utils_strip_transfer_root(path, config->send_directory)
|
||||||
|
: path;
|
||||||
|
if (!progress_precount_add_dir(out, rel)) {
|
||||||
|
progress_precount_dispose(out);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out->total += (unsigned long long)out->dir_paths->size;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Build the optional progress pre-count. A failed pre-count is non-fatal: the
|
||||||
|
* transfer proceeds and the progress denominator falls back to the transferred
|
||||||
|
* file count. */
|
||||||
|
static void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
|
||||||
|
unsigned long long plan_non_dir_count) {
|
||||||
|
client_progress_cleanup();
|
||||||
|
g_progress_active = progress_requested(config);
|
||||||
|
if (!g_progress_active)
|
||||||
|
return;
|
||||||
|
bool ok = plan_dirs != NULL ? progress_precount_from_plan_dirs(
|
||||||
|
config, plan_dirs, plan_non_dir_count, &g_progress_precount)
|
||||||
|
: progress_precount_scan(config, &g_progress_precount);
|
||||||
|
if (!ok) {
|
||||||
|
g_progress_total = 0;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
g_progress_total = g_progress_precount.total;
|
||||||
|
if (g_progress_precount.dir_paths != NULL && g_progress_precount.dir_paths->size > 0 &&
|
||||||
|
path_index_build(&g_progress_dir_index,
|
||||||
|
(const char* const*)g_progress_precount.dir_paths->items,
|
||||||
|
(size_t)g_progress_precount.dir_paths->size))
|
||||||
|
g_progress_dir_index_valid = true;
|
||||||
|
if (str_hash_set_init(&g_progress_emitted, (size_t)(g_progress_precount.dir_paths != NULL
|
||||||
|
? g_progress_precount.dir_paths->size + 1
|
||||||
|
: 1)))
|
||||||
|
g_progress_emitted_valid = true;
|
||||||
|
g_progress_emitted_keys = array_list_create(free);
|
||||||
|
}
|
||||||
|
|
||||||
/* -R/--relative implied directories: rsync transmits the metadata of the
|
/* -R/--relative implied directories: rsync transmits the metadata of the
|
||||||
* parent directories implied by the source path (every prefix component above
|
* parent directories implied by the source path (every prefix component above
|
||||||
* the source root) so the receiver applies their attributes to the created
|
* the source root) so the receiver applies their attributes to the created
|
||||||
@@ -761,53 +1047,6 @@ static bool files_from_list_check(const Config* config, ArrayList* missing_dest,
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Basis directories are honored by the receiver's per-file incremental check,
|
|
||||||
which (like every whole-file payload path in FastSync) is bounded by
|
|
||||||
MAX_RECEIVE_WHOLE_FILE_SIZE. rsync would apply basis dirs to files of any
|
|
||||||
size; FastSync cannot, so when basis dirs are requested this preflight scan
|
|
||||||
refuses the run up front with a clear diagnostic instead of letting the
|
|
||||||
receiver abort the whole transfer mid-stream with no client explanation.
|
|
||||||
Returns true when the tree can be transferred. */
|
|
||||||
static bool basis_oversize_preflight(const Config* config) {
|
|
||||||
PreparedScanner prepared;
|
|
||||||
if (!prepare_scanner(config, 0, &prepared))
|
|
||||||
return false;
|
|
||||||
DirectoryScanner* scanner =
|
|
||||||
directory_scanner_create_with_options(config->send_directory, &prepared.options);
|
|
||||||
if (!scanner) {
|
|
||||||
prepared_scanner_destroy(&prepared);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
bool ok = true;
|
|
||||||
Chunk* chunk;
|
|
||||||
while ((chunk = directory_scanner_next(scanner)) != NULL) {
|
|
||||||
for (int i = 0; i < chunk->element_count; i++) {
|
|
||||||
File* f = chunk->items[i];
|
|
||||||
if (f == NULL || f->is_dir || f->data == NULL || f->data->size <= MAX_RECEIVE_WHOLE_FILE_SIZE)
|
|
||||||
continue;
|
|
||||||
char* escaped = output_escape(file_wire_path(f), config->eight_bit_output);
|
|
||||||
log_message(LOG_LEVEL_ERROR,
|
|
||||||
"%s is %llu bytes, larger than the %llu-byte whole-file transfer limit; "
|
|
||||||
"--compare-dest/--copy-dest/--link-dest cannot sync files above this limit",
|
|
||||||
escaped ? escaped : "<allocation failed>", (unsigned long long)f->data->size,
|
|
||||||
(unsigned long long)MAX_RECEIVE_WHOLE_FILE_SIZE);
|
|
||||||
free(escaped);
|
|
||||||
ok = false;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
chunk_destroy(chunk);
|
|
||||||
if (!ok)
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
if (directory_scanner_failed(scanner) || directory_scanner_had_io_error(scanner))
|
|
||||||
ok = false;
|
|
||||||
/* The scanner borrows prepared.options' base_filters/hardlinks pointers, so
|
|
||||||
prepared must outlive the scanner. */
|
|
||||||
directory_scanner_destroy(scanner);
|
|
||||||
prepared_scanner_destroy(&prepared);
|
|
||||||
return ok;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Read the daemon's MOTD frame and, unless --no-motd, display it on stdout.
|
/* Read the daemon's MOTD frame and, unless --no-motd, display it on stdout.
|
||||||
*
|
*
|
||||||
* The daemon sends the MOTD as the first thing after the config-frame STATUS_OK
|
* The daemon sends the MOTD as the first thing after the config-frame STATUS_OK
|
||||||
@@ -1549,9 +1788,12 @@ static bool send_delete_manifest_early(Client* client, ArrayList* manifest,
|
|||||||
directory and *io_error_out reports it (the caller still performs the
|
directory and *io_error_out reports it (the caller still performs the
|
||||||
deletion but reports the run as errored). */
|
deletion but reports the run as errored). */
|
||||||
static bool scan_paths_only(const Config* config, const ScannerOptions* options,
|
static bool scan_paths_only(const Config* config, const ScannerOptions* options,
|
||||||
ArrayList* manifest, DeletePlanSender* plans, bool* io_error_out) {
|
ArrayList* manifest, DeletePlanSender* plans, bool* io_error_out,
|
||||||
|
unsigned long long* non_dir_count_out) {
|
||||||
if (io_error_out)
|
if (io_error_out)
|
||||||
*io_error_out = false;
|
*io_error_out = false;
|
||||||
|
if (non_dir_count_out)
|
||||||
|
*non_dir_count_out = 0;
|
||||||
ScannerOptions local = *options;
|
ScannerOptions local = *options;
|
||||||
/* The pre-scan is a paths-only pass with no client output; it must not emit
|
/* The pre-scan is a paths-only pass with no client output; it must not emit
|
||||||
--info=nonreg lines (the data pass does that once). */
|
--info=nonreg lines (the data pass does that once). */
|
||||||
@@ -1562,6 +1804,13 @@ static bool scan_paths_only(const Config* config, const ScannerOptions* options,
|
|||||||
bool ok = true;
|
bool ok = true;
|
||||||
Chunk* chunk;
|
Chunk* chunk;
|
||||||
while ((chunk = directory_scanner_next(scanner)) != NULL) {
|
while ((chunk = directory_scanner_next(scanner)) != NULL) {
|
||||||
|
if (non_dir_count_out) {
|
||||||
|
for (int i = 0; i < chunk->element_count; i++) {
|
||||||
|
const File* f = chunk->items[i];
|
||||||
|
if (f && !f->is_dir)
|
||||||
|
(*non_dir_count_out)++;
|
||||||
|
}
|
||||||
|
}
|
||||||
if (manifest && !add_chunk_to_manifest(manifest, chunk)) {
|
if (manifest && !add_chunk_to_manifest(manifest, chunk)) {
|
||||||
ok = false;
|
ok = false;
|
||||||
chunk_destroy(chunk);
|
chunk_destroy(chunk);
|
||||||
@@ -1643,11 +1892,13 @@ static int incremental_check(Client* client, File* file, const Config* config,
|
|||||||
return -1;
|
return -1;
|
||||||
if (!send_n_data(client->file_descriptor, &mtime_nsec, sizeof(mtime_nsec)))
|
if (!send_n_data(client->file_descriptor, &mtime_nsec, sizeof(mtime_nsec)))
|
||||||
return -1;
|
return -1;
|
||||||
/* With alternate basis directories the receiver must be able to verify the
|
/* The whole-file digest (negotiated --checksum-choice algorithm and
|
||||||
* content of every candidate basis file, so the sender supplies its whole-file
|
* --checksum-seed) is only needed when it drives a decision: --checksum's
|
||||||
* digest (computed with the negotiated --checksum-choice algorithm and
|
* per-file quick check, or a --verify-basis content equality. Under the
|
||||||
* --checksum-seed) for every file even when --checksum was not requested. */
|
* default metadata quick-check the receiver never reads it, so the sender
|
||||||
if (config->checksum || config_has_basis(config)) {
|
* skips the full-file read/hash exactly as rsync does for a plain
|
||||||
|
* --link-dest run. */
|
||||||
|
if (config->checksum || config->verify_basis) {
|
||||||
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
size_t digest_len = 0;
|
size_t digest_len = 0;
|
||||||
if (!file_checksum(file, (ChecksumAlgo)config->checksum_algo, config->checksum_seed, digest,
|
if (!file_checksum(file, (ChecksumAlgo)config->checksum_algo, config->checksum_seed, digest,
|
||||||
@@ -1658,6 +1909,15 @@ static int incremental_check(Client* client, File* file, const Config* config,
|
|||||||
!send_n_data(client->file_descriptor, digest, wire_len))
|
!send_n_data(client->file_descriptor, digest, wire_len))
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
|
/* Basis directories: the receiver materializes a hit from the basis without a
|
||||||
|
* data frame, so it would otherwise only have the basis inode's metadata.
|
||||||
|
* Transmit the SOURCE metadata with the check (rsync's copy-then-fix) so a
|
||||||
|
* --copy-dest hit / --link-dest copy fallback applies the source's
|
||||||
|
* attributes. Symmetric with incremental_check_receive_request. */
|
||||||
|
if (config_has_basis(config) && config->use_metadata) {
|
||||||
|
if (!metadata_send(client->file_descriptor, file->metadata))
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
Status s;
|
Status s;
|
||||||
if (!receive_status(client->file_descriptor, &s))
|
if (!receive_status(client->file_descriptor, &s))
|
||||||
return -1;
|
return -1;
|
||||||
@@ -1897,12 +2157,6 @@ static int send_dry_run_remote(Config* config) {
|
|||||||
}
|
}
|
||||||
if (missing_args)
|
if (missing_args)
|
||||||
array_list_delete(missing_args);
|
array_list_delete(missing_args);
|
||||||
/* Alternate basis dirs force the whole-file per-file check on the real
|
|
||||||
receiver; refuse an oversize source up front exactly as send_files does so
|
|
||||||
dry-run reports the same clear diagnostic instead of aborting mid-stream. */
|
|
||||||
if (config_has_basis(config) && !basis_oversize_preflight(config))
|
|
||||||
return 1;
|
|
||||||
|
|
||||||
/* A live session may follow, so arm graceful abort handling. */
|
/* A live session may follow, so arm graceful abort handling. */
|
||||||
client_set_abort_armed(true);
|
client_set_abort_armed(true);
|
||||||
Client* client = connect_transfer_client(config);
|
Client* client = connect_transfer_client(config);
|
||||||
@@ -1928,11 +2182,41 @@ static int send_dry_run_remote(Config* config) {
|
|||||||
DirectoryScanner* scanner = NULL;
|
DirectoryScanner* scanner = NULL;
|
||||||
ArrayList* dry_manifest = NULL;
|
ArrayList* dry_manifest = NULL;
|
||||||
ArrayList* dry_dirs = NULL;
|
ArrayList* dry_dirs = NULL;
|
||||||
|
ArrayList* dry_excluded = NULL;
|
||||||
|
ArrayList* dry_size_skipped = NULL;
|
||||||
if (!config_send(client->file_descriptor, config))
|
if (!config_send(client->file_descriptor, config))
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
receive_daemon_motd(client, config);
|
receive_daemon_motd(client, config);
|
||||||
if (!prepare_scanner(config, 0, &prepared))
|
if (!prepare_scanner(config, 0, &prepared))
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
|
/* -n --delete: build the same keep-set manifest, protected prefixes, and
|
||||||
|
synchronized-directory scope a real run would send, so the receiver's
|
||||||
|
read-only extras walk enumerates exactly the deletions a real run makes. */
|
||||||
|
if (config->use_delete) {
|
||||||
|
dry_manifest = array_list_create(free);
|
||||||
|
dry_dirs = array_list_create(free);
|
||||||
|
dry_size_skipped = array_list_create(free);
|
||||||
|
if (!dry_manifest || !dry_dirs || !dry_size_skipped)
|
||||||
|
goto dry_fail;
|
||||||
|
if (!config->delete_excluded) {
|
||||||
|
dry_excluded = array_list_create(free);
|
||||||
|
if (!dry_excluded)
|
||||||
|
goto dry_fail;
|
||||||
|
prepared.options.excluded_paths = dry_excluded;
|
||||||
|
}
|
||||||
|
prepared.options.size_skipped_paths = dry_size_skipped;
|
||||||
|
/* A --files-from subset confines the extras walk to the directories the
|
||||||
|
scan synchronized; a full recursive transfer marks the root itself. */
|
||||||
|
if (config->files_from_set == NULL) {
|
||||||
|
char* root_marker = delete_scope_root_marker(config);
|
||||||
|
if (!root_marker || !array_list_add(dry_dirs, root_marker)) {
|
||||||
|
free(root_marker);
|
||||||
|
goto dry_fail;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
prepared.options.synced_dirs = dry_dirs;
|
||||||
|
}
|
||||||
|
}
|
||||||
scanner = directory_scanner_create_with_options(config->send_directory, &prepared.options);
|
scanner = directory_scanner_create_with_options(config->send_directory, &prepared.options);
|
||||||
if (!scanner)
|
if (!scanner)
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
@@ -1940,26 +2224,6 @@ static int send_dry_run_remote(Config* config) {
|
|||||||
int file_count = 0;
|
int file_count = 0;
|
||||||
unsigned long long total_bytes = 0;
|
unsigned long long total_bytes = 0;
|
||||||
char size_buffer[32];
|
char size_buffer[32];
|
||||||
/* -n --delete: build the same keep-set manifest a real run would send so the
|
|
||||||
receiver can enumerate (read-only) the destination extras. Filter-excluded
|
|
||||||
and size-pruned protections are not propagated here, so a filtered dry-run
|
|
||||||
may over-report; the no-filter case is exact. */
|
|
||||||
dry_manifest = config->use_delete ? array_list_create(free) : NULL;
|
|
||||||
if (config->use_delete && !dry_manifest)
|
|
||||||
goto dry_fail;
|
|
||||||
/* Scope the receiver-side extras walk to the receive root (the "." sentinel),
|
|
||||||
exactly as the recursive transfer path does. */
|
|
||||||
if (config->use_delete) {
|
|
||||||
dry_dirs = array_list_create(free);
|
|
||||||
char* root_marker = dry_dirs ? str_dup(".") : NULL;
|
|
||||||
if (!dry_dirs || !root_marker || !array_list_add(dry_dirs, root_marker)) {
|
|
||||||
free(root_marker);
|
|
||||||
if (dry_dirs)
|
|
||||||
array_list_delete(dry_dirs);
|
|
||||||
dry_dirs = NULL;
|
|
||||||
goto dry_fail;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (!config->quiet)
|
if (!config->quiet)
|
||||||
printf("Dry run: files to be transferred\n");
|
printf("Dry run: files to be transferred\n");
|
||||||
Chunk* chunk;
|
Chunk* chunk;
|
||||||
@@ -2034,8 +2298,8 @@ static int send_dry_run_remote(Config* config) {
|
|||||||
the terminal FINISHED. */
|
the terminal FINISHED. */
|
||||||
bool early_delete = config->use_delete && config_delete_timing_early(config);
|
bool early_delete = config->use_delete && config_delete_timing_early(config);
|
||||||
if (dry_manifest) {
|
if (dry_manifest) {
|
||||||
if (send_delete_manifest(client->file_descriptor, dry_manifest, NULL, NULL, NULL, dry_dirs) !=
|
if (send_delete_manifest(client->file_descriptor, dry_manifest, dry_excluded, dry_size_skipped,
|
||||||
0)
|
NULL, dry_dirs) != 0)
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
if (early_delete) {
|
if (early_delete) {
|
||||||
Status ack;
|
Status ack;
|
||||||
@@ -2091,6 +2355,10 @@ dry_fail:
|
|||||||
array_list_delete(dry_manifest);
|
array_list_delete(dry_manifest);
|
||||||
if (dry_dirs)
|
if (dry_dirs)
|
||||||
array_list_delete(dry_dirs);
|
array_list_delete(dry_dirs);
|
||||||
|
if (dry_excluded)
|
||||||
|
array_list_delete(dry_excluded);
|
||||||
|
if (dry_size_skipped)
|
||||||
|
array_list_delete(dry_size_skipped);
|
||||||
if (scanner)
|
if (scanner)
|
||||||
directory_scanner_destroy(scanner);
|
directory_scanner_destroy(scanner);
|
||||||
prepared_scanner_destroy(&prepared);
|
prepared_scanner_destroy(&prepared);
|
||||||
@@ -2383,6 +2651,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
if (!send_directory_entry(client, f, config))
|
if (!send_directory_entry(client, f, config))
|
||||||
return -1;
|
return -1;
|
||||||
change_emit_dir_sent(config, f);
|
change_emit_dir_sent(config, f);
|
||||||
|
client_progress_name(config, f);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
/* --hard-links/-H sibling: a later member of a hard-link group that has no
|
/* --hard-links/-H sibling: a later member of a hard-link group that has no
|
||||||
@@ -2396,6 +2665,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
!send_wire_str(client->file_descriptor, f->hardlink_target))
|
!send_wire_str(client->file_descriptor, f->hardlink_target))
|
||||||
return -1;
|
return -1;
|
||||||
change_emit_file_sent(config, f);
|
change_emit_file_sent(config, f);
|
||||||
|
client_progress_name(config, f);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
/* Symlink entry (-l / -k keep-as-symlink): only the target rides the wire. */
|
/* Symlink entry (-l / -k keep-as-symlink): only the target rides the wire. */
|
||||||
@@ -2403,6 +2673,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
if (!send_symlink_entry(client, f, config))
|
if (!send_symlink_entry(client, f, config))
|
||||||
return -1;
|
return -1;
|
||||||
change_emit_file_sent(config, f);
|
change_emit_file_sent(config, f);
|
||||||
|
client_progress_name(config, f);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
/* --devices/--specials: a device/special node is recreated on the receiver,
|
/* --devices/--specials: a device/special node is recreated on the receiver,
|
||||||
@@ -2411,6 +2682,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
if (!file_send_special(f, client->file_descriptor, config->use_metadata))
|
if (!file_send_special(f, client->file_descriptor, config->use_metadata))
|
||||||
return -1;
|
return -1;
|
||||||
change_emit_file_sent(config, f);
|
change_emit_file_sent(config, f);
|
||||||
|
client_progress_name(config, f);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
bool stream = f->data->data == NULL && f->data->size > 0;
|
bool stream = f->data->data == NULL && f->data->size > 0;
|
||||||
@@ -2423,6 +2695,8 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
int rc = send_single_file(client, f, config, config->use_incremental, use_sendfile);
|
int rc = send_single_file(client, f, config, config->use_incremental, use_sendfile);
|
||||||
if (rc == 1) {
|
if (rc == 1) {
|
||||||
source_file_destroy(source);
|
source_file_destroy(source);
|
||||||
|
change_emit_file_uptodate(config, f);
|
||||||
|
client_progress_uptodate(config, f);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (rc < 0) {
|
if (rc < 0) {
|
||||||
@@ -2924,11 +3198,6 @@ int send_files(Config* config) {
|
|||||||
array_list_delete(missing_args);
|
array_list_delete(missing_args);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
if (config_has_basis(config) && !basis_oversize_preflight(config)) {
|
|
||||||
if (missing_args)
|
|
||||||
array_list_delete(missing_args);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* From here on a server session may be live, so Ctrl-C/SIGTERM should set the
|
/* From here on a server session may be live, so Ctrl-C/SIGTERM should set the
|
||||||
abort flag (and be forwarded as STATUS_ABORT) instead of terminating. */
|
abort flag (and be forwarded as STATUS_ABORT) instead of terminating. */
|
||||||
@@ -2969,6 +3238,7 @@ int send_files(Config* config) {
|
|||||||
bool delete_per_dir = config->use_delete && config_delete_timing_per_dir(config) && !config->dirs;
|
bool delete_per_dir = config->use_delete && config_delete_timing_per_dir(config) && !config->dirs;
|
||||||
bool send_failed = false;
|
bool send_failed = false;
|
||||||
bool had_scan_io = false;
|
bool had_scan_io = false;
|
||||||
|
unsigned long long per_dir_non_dir_count = 0;
|
||||||
PreparedScanner prepared;
|
PreparedScanner prepared;
|
||||||
memset(&prepared, 0, sizeof(prepared));
|
memset(&prepared, 0, sizeof(prepared));
|
||||||
if (!config_send(client->file_descriptor, config))
|
if (!config_send(client->file_descriptor, config))
|
||||||
@@ -3017,7 +3287,8 @@ int send_files(Config* config) {
|
|||||||
prepared.options.synced_dirs = synced_dirs;
|
prepared.options.synced_dirs = synced_dirs;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/* The late-timing modes (plain --delete / --delete-after) build the manifest
|
/* The late-timing modes (--delete-after/--delete-commit and a plain --delete
|
||||||
|
that fell back from per-dir mode because of -d/--dirs) build the manifest
|
||||||
while streaming and send it after the last data frame. --delete-before
|
while streaming and send it after the last data frame. --delete-before
|
||||||
sends a whole-tree keep-set up front; --delete-during/--delete-delay build a
|
sends a whole-tree keep-set up front; --delete-during/--delete-delay build a
|
||||||
per-directory plan set up front (paths only) and stream the plans alongside
|
per-directory plan set up front (paths only) and stream the plans alongside
|
||||||
@@ -3030,7 +3301,7 @@ int send_files(Config* config) {
|
|||||||
if (!early_manifest)
|
if (!early_manifest)
|
||||||
goto send_fail;
|
goto send_fail;
|
||||||
bool prescan_ok =
|
bool prescan_ok =
|
||||||
scan_paths_only(config, &prepared.options, early_manifest, NULL, &had_scan_io);
|
scan_paths_only(config, &prepared.options, early_manifest, NULL, &had_scan_io, NULL);
|
||||||
bool early_ok = false;
|
bool early_ok = false;
|
||||||
bool skip_delete = false;
|
bool skip_delete = false;
|
||||||
if (prescan_ok) {
|
if (prescan_ok) {
|
||||||
@@ -3072,7 +3343,8 @@ int send_files(Config* config) {
|
|||||||
if (!plan_sender || !plan_dirs)
|
if (!plan_sender || !plan_dirs)
|
||||||
goto send_fail;
|
goto send_fail;
|
||||||
prepared.options.plan_dirs = plan_dirs;
|
prepared.options.plan_dirs = plan_dirs;
|
||||||
bool prescan_ok = scan_paths_only(config, &prepared.options, NULL, plan_sender, &had_scan_io);
|
bool prescan_ok = scan_paths_only(config, &prepared.options, NULL, plan_sender, &had_scan_io,
|
||||||
|
&per_dir_non_dir_count);
|
||||||
bool plans_ok = false;
|
bool plans_ok = false;
|
||||||
bool skip_delete = false;
|
bool skip_delete = false;
|
||||||
if (prescan_ok) {
|
if (prescan_ok) {
|
||||||
@@ -3110,6 +3382,11 @@ int send_files(Config* config) {
|
|||||||
if (!manifest)
|
if (!manifest)
|
||||||
goto send_fail;
|
goto send_fail;
|
||||||
}
|
}
|
||||||
|
/* --progress/--info=progress: pre-count the file list for rsync's to-chk
|
||||||
|
denominator. When a --delete-during/--delete-delay pre-scan already ran,
|
||||||
|
reuse its traversed directory list instead of walking the tree again. */
|
||||||
|
if (progress_requested(config))
|
||||||
|
client_progress_prepare(config, plan_dirs, per_dir_non_dir_count);
|
||||||
/* Phase 6: compute the client-only stop deadline once at transfer start. The
|
/* Phase 6: compute the client-only stop deadline once at transfer start. The
|
||||||
early-delete pre-scan above deliberately ignores it so the keep-set (and
|
early-delete pre-scan above deliberately ignores it so the keep-set (and
|
||||||
its committed deletion) is always complete and correct. */
|
its committed deletion) is always complete and correct. */
|
||||||
@@ -3331,6 +3608,7 @@ send_fail:
|
|||||||
if (scanner)
|
if (scanner)
|
||||||
directory_scanner_destroy(scanner);
|
directory_scanner_destroy(scanner);
|
||||||
prepared_scanner_destroy(&prepared);
|
prepared_scanner_destroy(&prepared);
|
||||||
|
client_progress_cleanup();
|
||||||
disconnect_transfer_client(client);
|
disconnect_transfer_client(client);
|
||||||
protocol_session_unbind();
|
protocol_session_unbind();
|
||||||
client_set_abort_armed(false);
|
client_set_abort_armed(false);
|
||||||
@@ -3358,11 +3636,6 @@ int send_files_multithreaded(Config** config_ptr) {
|
|||||||
array_list_delete(missing_args);
|
array_list_delete(missing_args);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
if (config_has_basis(config) && !basis_oversize_preflight(config)) {
|
|
||||||
if (missing_args)
|
|
||||||
array_list_delete(missing_args);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Armed only once a session may go live (see send_files). */
|
/* Armed only once a session may go live (see send_files). */
|
||||||
client_set_abort_armed(true);
|
client_set_abort_armed(true);
|
||||||
@@ -3414,6 +3687,7 @@ int send_files_multithreaded(Config** config_ptr) {
|
|||||||
stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins, config->stop_at_set,
|
stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins, config->stop_at_set,
|
||||||
config->stop_at, now_mono);
|
config->stop_at, now_mono);
|
||||||
bool collect_excluded = config->use_delete && !config->delete_excluded;
|
bool collect_excluded = config->use_delete && !config->delete_excluded;
|
||||||
|
unsigned long long pre_scan_non_dir = 0;
|
||||||
if (config->use_delete) {
|
if (config->use_delete) {
|
||||||
if (collect_excluded) {
|
if (collect_excluded) {
|
||||||
context->excluded_paths = array_list_create(free);
|
context->excluded_paths = array_list_create(free);
|
||||||
@@ -3472,8 +3746,9 @@ int send_files_multithreaded(Config** config_ptr) {
|
|||||||
prepared_ok = prepared_ok && context->manifest != NULL;
|
prepared_ok = prepared_ok && context->manifest != NULL;
|
||||||
}
|
}
|
||||||
bool prebuilt =
|
bool prebuilt =
|
||||||
prepared_ok && scan_paths_only(config, &prepared.options, context->manifest,
|
prepared_ok &&
|
||||||
context->delete_plans, &context->scan_had_io_error);
|
scan_paths_only(config, &prepared.options, context->manifest, context->delete_plans,
|
||||||
|
&context->scan_had_io_error, &pre_scan_non_dir);
|
||||||
prepared_scanner_destroy(&prepared);
|
prepared_scanner_destroy(&prepared);
|
||||||
if (per_dir && prebuilt) {
|
if (per_dir && prebuilt) {
|
||||||
const char* walk_root = delete_plan_walk_root(config, context->synced_dirs);
|
const char* walk_root = delete_plan_walk_root(config, context->synced_dirs);
|
||||||
@@ -3536,6 +3811,11 @@ int send_files_multithreaded(Config** config_ptr) {
|
|||||||
pipeline_context_sender_destroy(context);
|
pipeline_context_sender_destroy(context);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
/* --progress/--info=progress: pre-count the file list for rsync's to-chk
|
||||||
|
denominator, reusing a --delete-during/--delete-delay pre-scan when one
|
||||||
|
already ran. */
|
||||||
|
if (progress_requested(config))
|
||||||
|
client_progress_prepare(config, context->plan_dirs, pre_scan_non_dir);
|
||||||
|
|
||||||
thrd_t scanner, loader, sender;
|
thrd_t scanner, loader, sender;
|
||||||
bool scanner_created = false;
|
bool scanner_created = false;
|
||||||
@@ -3561,6 +3841,7 @@ int send_files_multithreaded(Config** config_ptr) {
|
|||||||
if (scanner_created)
|
if (scanner_created)
|
||||||
thrd_join(scanner, NULL);
|
thrd_join(scanner, NULL);
|
||||||
pipeline_context_sender_destroy(context);
|
pipeline_context_sender_destroy(context);
|
||||||
|
client_progress_cleanup();
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -3580,6 +3861,7 @@ int send_files_multithreaded(Config** config_ptr) {
|
|||||||
transfer (exit 23). A --max-delete-capped commit is a successful transfer
|
transfer (exit 23). A --max-delete-capped commit is a successful transfer
|
||||||
that rsync reports with exit code 25. */
|
that rsync reports with exit code 25. */
|
||||||
pipeline_context_sender_destroy(context);
|
pipeline_context_sender_destroy(context);
|
||||||
|
client_progress_cleanup();
|
||||||
client_set_abort_armed(false);
|
client_set_abort_armed(false);
|
||||||
if (!sender_ok)
|
if (!sender_ok)
|
||||||
return 1;
|
return 1;
|
||||||
|
|||||||
+9
-4
@@ -62,8 +62,7 @@ void print_usage(void) {
|
|||||||
printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n");
|
printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n");
|
||||||
printf(" files (different container format); do not mix the two tools.\n");
|
printf(" files (different container format); do not mix the two tools.\n");
|
||||||
printf(" --delete Delete files on receiver not in source\n");
|
printf(" --delete Delete files on receiver not in source\n");
|
||||||
printf(" (default timing: delete only after the whole\n");
|
printf(" (default timing: delete-during, like rsync --del)\n");
|
||||||
printf(" transfer has succeeded)\n");
|
|
||||||
printf(" --delete-before Delete extras before the transfer starts\n");
|
printf(" --delete-before Delete extras before the transfer starts\n");
|
||||||
printf(" (implies --delete)\n");
|
printf(" (implies --delete)\n");
|
||||||
printf(" --delete-during Delete a directory's extras as that directory is\n");
|
printf(" --delete-during Delete a directory's extras as that directory is\n");
|
||||||
@@ -72,7 +71,9 @@ void print_usage(void) {
|
|||||||
printf(" --delete-delay Record the extras during the scan but remove them\n");
|
printf(" --delete-delay Record the extras during the scan but remove them\n");
|
||||||
printf(" only after a successful transfer (implies --delete)\n");
|
printf(" only after a successful transfer (implies --delete)\n");
|
||||||
printf(" --delete-after Delete only after the whole transfer succeeded\n");
|
printf(" --delete-after Delete only after the whole transfer succeeded\n");
|
||||||
printf(" (the default --delete timing; implies --delete)\n");
|
printf(" (implies --delete)\n");
|
||||||
|
printf(" --delete-commit FastSync-only: restore the late whole-tree commit\n");
|
||||||
|
printf(" (identical to --delete-after; implies --delete)\n");
|
||||||
printf(" --delete-excluded Also delete destination files that were excluded on\n");
|
printf(" --delete-excluded Also delete destination files that were excluded on\n");
|
||||||
printf(" the source (default protects them, matching rsync)\n");
|
printf(" the source (default protects them, matching rsync)\n");
|
||||||
printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n");
|
printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n");
|
||||||
@@ -93,7 +94,8 @@ void print_usage(void) {
|
|||||||
printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n");
|
printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n");
|
||||||
printf(" otherwise recreates them, like rsync)\n");
|
printf(" otherwise recreates them, like rsync)\n");
|
||||||
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
|
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
|
||||||
printf(" --no-delete (in either order) is rejected as a config error.\n");
|
printf(" --no-delete (in either order) is rejected as a config error, as is more\n");
|
||||||
|
printf(" than one timing flag.\n");
|
||||||
printf(" --ignore-existing Skip files that already exist on receiver\n");
|
printf(" --ignore-existing Skip files that already exist on receiver\n");
|
||||||
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
|
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
|
||||||
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
|
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
|
||||||
@@ -138,6 +140,9 @@ void print_usage(void) {
|
|||||||
printf(" into the destination instead of transferring its data\n");
|
printf(" into the destination instead of transferring its data\n");
|
||||||
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
|
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
|
||||||
printf(" into the destination (repeatable; earlier DIRs win)\n");
|
printf(" into the destination (repeatable; earlier DIRs win)\n");
|
||||||
|
printf(" --verify-basis FastSync-only: require a basis hit's content to match the\n");
|
||||||
|
printf(" source by whole-file digest instead of trusting rsync's\n");
|
||||||
|
printf(" size+mtime (or --size-only) quick-check\n");
|
||||||
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
|
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
|
||||||
printf(" --checksum compares. Accepted: xxh128 (default), xxh3, xxh64\n");
|
printf(" --checksum compares. Accepted: xxh128 (default), xxh3, xxh64\n");
|
||||||
printf(" (aka xxhash), md5, md4, sha1, or none. A two-name\n");
|
printf(" (aka xxhash), md5, md4, sha1, or none. A two-name\n");
|
||||||
|
|||||||
+14
-5
@@ -305,14 +305,16 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
|
|||||||
|
|
||||||
/* Runs the whole receive loop. The delete manifest may legitimately arrive
|
/* Runs the whole receive loop. The delete manifest may legitimately arrive
|
||||||
either FIRST (--delete-before / --delete-during: the sender transmits the
|
either FIRST (--delete-before / --delete-during: the sender transmits the
|
||||||
validated keep-set before any file data) or LAST (plain --delete /
|
validated keep-set before any file data) or LAST (--delete-after /
|
||||||
--delete-after / --delete-delay: the manifest closes the data stream). In
|
--delete-commit / --delete-delay: the manifest closes the data stream). In
|
||||||
the early modes the receiver deletes as soon as the manifest has been read
|
the early modes the receiver deletes as soon as the manifest has been read
|
||||||
and acknowledges with STATUS_OK so the sender only starts streaming once the
|
and acknowledges with STATUS_OK so the sender only starts streaming once the
|
||||||
deletion has committed (or failed); in the late modes the manifest is held
|
deletion has committed (or failed); in the late modes the manifest is held
|
||||||
and the deletion is committed only after the terminal STATUS_FINISHED proves
|
and the deletion is committed only after the terminal STATUS_FINISHED proves
|
||||||
the whole transfer succeeded. See receiver_process_pending() for how the -m
|
the whole transfer succeeded. A plain --delete defaults to the per-directory
|
||||||
receiver defers that commit until its disk writer has drained. */
|
delete-during plan mode (no manifest at all). See
|
||||||
|
receiver_process_pending() for how the -m receiver defers that commit until
|
||||||
|
its disk writer has drained. */
|
||||||
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||||
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
|
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
|
||||||
Status status;
|
Status status;
|
||||||
@@ -613,6 +615,8 @@ typedef struct {
|
|||||||
static bool receiver_save_file(File* file, void* context_pointer) {
|
static bool receiver_save_file(File* file, void* context_pointer) {
|
||||||
ReceiverSaveContext* context = context_pointer;
|
ReceiverSaveContext* context = context_pointer;
|
||||||
FileSaveResult result = FILE_SAVE_ERROR;
|
FileSaveResult result = FILE_SAVE_ERROR;
|
||||||
|
bool created = false;
|
||||||
|
unsigned created_dirs = 0;
|
||||||
if (context->config->dry_run) {
|
if (context->config->dry_run) {
|
||||||
/* Defense in depth: a dry-run receiver mutates nothing even if a data
|
/* Defense in depth: a dry-run receiver mutates nothing even if a data
|
||||||
frame reaches the sink (the sender is not supposed to send one). */
|
frame reaches the sink (the sender is not supposed to send one). */
|
||||||
@@ -622,12 +626,17 @@ static bool receiver_save_file(File* file, void* context_pointer) {
|
|||||||
--remove-source-files sender keeps its source. */
|
--remove-source-files sender keeps its source. */
|
||||||
result = FILE_SAVE_SKIPPED;
|
result = FILE_SAVE_SKIPPED;
|
||||||
} else {
|
} else {
|
||||||
result = file_save_to_disk_full(context->config->receive_root_directory, file, context->config);
|
result = file_save_to_disk_full_ex(context->config->receive_root_directory, file,
|
||||||
|
context->config, &created, &created_dirs);
|
||||||
}
|
}
|
||||||
/* Wire-stats tally: bytes reconstructed from the basis file (delta matches)
|
/* Wire-stats tally: bytes reconstructed from the basis file (delta matches)
|
||||||
count as matched data in the end-of-transfer report. */
|
count as matched data in the end-of-transfer report. */
|
||||||
if (result != FILE_SAVE_ERROR && file->matched_bytes > 0)
|
if (result != FILE_SAVE_ERROR && file->matched_bytes > 0)
|
||||||
context->stats.matched_data += file->matched_bytes;
|
context->stats.matched_data += file->matched_bytes;
|
||||||
|
/* Protocol 2.28.0: receiver-observed literal bytes and the created-entry
|
||||||
|
breakdown (regular/dir/link/special) for the `--stats` report. */
|
||||||
|
if (result == FILE_SAVE_WRITTEN)
|
||||||
|
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
|
||||||
/* A directory's metadata is deferred, never applied inline: collect it now
|
/* A directory's metadata is deferred, never applied inline: collect it now
|
||||||
and apply it at the end. -O/--omit-dir-times and --preserve_perms/-times
|
and apply it at the end. -O/--omit-dir-times and --preserve_perms/-times
|
||||||
are honored by dir_metadata_list_apply's caller (see
|
are honored by dir_metadata_list_apply's caller (see
|
||||||
|
|||||||
@@ -247,12 +247,23 @@ int write_thread(void* pipeline_context) {
|
|||||||
}
|
}
|
||||||
size_t file_bytes = file->data ? file->data->size : 0;
|
size_t file_bytes = file->data ? file->data->size : 0;
|
||||||
FileSaveResult result = FILE_SAVE_SKIPPED;
|
FileSaveResult result = FILE_SAVE_SKIPPED;
|
||||||
|
bool created = false;
|
||||||
|
unsigned created_dirs = 0;
|
||||||
/* Server-contacting --dry-run: never write. The receiver thread does not
|
/* Server-contacting --dry-run: never write. The receiver thread does not
|
||||||
enqueue anything on the dry-run path, but this keeps the writer thread
|
enqueue anything on the dry-run path, but this keeps the writer thread
|
||||||
provably mutation-free if a data frame ever reached it. */
|
provably mutation-free if a data frame ever reached it. */
|
||||||
bool dry_run = context->config->dry_run;
|
bool dry_run = context->config->dry_run;
|
||||||
if (save_to_disk && !dry_run) {
|
if (save_to_disk && !dry_run) {
|
||||||
result = file_save_to_disk_full(root_directory, file, context->config);
|
result =
|
||||||
|
file_save_to_disk_full_ex(root_directory, file, context->config, &created, &created_dirs);
|
||||||
|
if (result == FILE_SAVE_WRITTEN) {
|
||||||
|
/* Protocol 2.28.0: fold the receiver-observed literal bytes and the
|
||||||
|
created-entry type into the shared stats block under its mutex (the
|
||||||
|
receive thread also writes stats.matched_data). */
|
||||||
|
mtx_lock(&context->mutex);
|
||||||
|
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
|
||||||
|
mtx_unlock(&context->mutex);
|
||||||
|
}
|
||||||
if (result == FILE_SAVE_ERROR) {
|
if (result == FILE_SAVE_ERROR) {
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
pipeline_context_receiver_note_bytes_released(context, file_bytes);
|
pipeline_context_receiver_note_bytes_released(context, file_bytes);
|
||||||
|
|||||||
+28
-12
@@ -1,4 +1,5 @@
|
|||||||
#include "checksum.h"
|
#include "checksum.h"
|
||||||
|
#include "utils.h"
|
||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
#include <openssl/evp.h>
|
#include <openssl/evp.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
@@ -223,23 +224,31 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
if (fd < 0)
|
if (fd < 0)
|
||||||
return false;
|
return false;
|
||||||
|
|
||||||
|
bool ok = checksum_digest_fd(algo, seed, fd, out, out_capacity, out_len);
|
||||||
|
close(fd);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
|
||||||
|
size_t* out_len) {
|
||||||
|
if (fd < 0 || !out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
|
||||||
|
return false;
|
||||||
|
|
||||||
if (algo == CHECKSUM_ALGO_NONE) {
|
if (algo == CHECKSUM_ALGO_NONE) {
|
||||||
/* No checksum requested: nothing to read; an empty digest succeeds. */
|
/* No checksum requested: nothing to read; an empty digest succeeds. */
|
||||||
close(fd);
|
|
||||||
*out_len = 0;
|
*out_len = 0;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
uint8_t buffer[64 * 1024];
|
uint8_t buffer[64 * 1024];
|
||||||
bool ok = false;
|
bool ok = false;
|
||||||
|
lseek(fd, 0, SEEK_SET);
|
||||||
|
|
||||||
if (algo == CHECKSUM_ALGO_MD5 || algo == CHECKSUM_ALGO_SHA1) {
|
if (algo == CHECKSUM_ALGO_MD5 || algo == CHECKSUM_ALGO_SHA1) {
|
||||||
const EVP_MD* md = algo == CHECKSUM_ALGO_MD5 ? EVP_md5() : EVP_sha1();
|
const EVP_MD* md = algo == CHECKSUM_ALGO_MD5 ? EVP_md5() : EVP_sha1();
|
||||||
EVP_MD_CTX* ctx = EVP_MD_CTX_new();
|
EVP_MD_CTX* ctx = EVP_MD_CTX_new();
|
||||||
if (!ctx) {
|
if (!ctx)
|
||||||
close(fd);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
|
||||||
unsigned int digest_len = 0;
|
unsigned int digest_len = 0;
|
||||||
if (EVP_DigestInit_ex(ctx, md, NULL) == 1) {
|
if (EVP_DigestInit_ex(ctx, md, NULL) == 1) {
|
||||||
ok = true;
|
ok = true;
|
||||||
@@ -258,7 +267,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
ok = false;
|
ok = false;
|
||||||
}
|
}
|
||||||
EVP_MD_CTX_free(ctx);
|
EVP_MD_CTX_free(ctx);
|
||||||
close(fd);
|
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -275,7 +283,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
md4_final(&ctx, out);
|
md4_final(&ctx, out);
|
||||||
*out_len = 16;
|
*out_len = 16;
|
||||||
}
|
}
|
||||||
close(fd);
|
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -285,16 +292,13 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
XXH64_reset(&xxh64, seed);
|
XXH64_reset(&xxh64, seed);
|
||||||
} else if (algo == CHECKSUM_ALGO_XXH3 || algo == CHECKSUM_ALGO_XXH128) {
|
} else if (algo == CHECKSUM_ALGO_XXH3 || algo == CHECKSUM_ALGO_XXH128) {
|
||||||
xxh3 = XXH3_createState();
|
xxh3 = XXH3_createState();
|
||||||
if (!xxh3) {
|
if (!xxh3)
|
||||||
close(fd);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
|
||||||
if (algo == CHECKSUM_ALGO_XXH3)
|
if (algo == CHECKSUM_ALGO_XXH3)
|
||||||
XXH3_64bits_reset_withSeed(xxh3, seed);
|
XXH3_64bits_reset_withSeed(xxh3, seed);
|
||||||
else
|
else
|
||||||
XXH3_128bits_reset_withSeed(xxh3, seed);
|
XXH3_128bits_reset_withSeed(xxh3, seed);
|
||||||
} else {
|
} else {
|
||||||
close(fd);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -328,7 +332,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
}
|
}
|
||||||
if (xxh3)
|
if (xxh3)
|
||||||
XXH3_freeState(xxh3);
|
XXH3_freeState(xxh3);
|
||||||
close(fd);
|
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -396,7 +399,7 @@ uint8_t checksum_digest_len(ChecksumAlgo algo) {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
ChecksumAlgo checksum_negotiate_default(void) {
|
static ChecksumAlgo compiled_checksum_preference_first(void) {
|
||||||
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
||||||
* resolves to xxh128. */
|
* resolves to xxh128. */
|
||||||
static const ChecksumAlgo preference[] = {
|
static const ChecksumAlgo preference[] = {
|
||||||
@@ -409,3 +412,16 @@ ChecksumAlgo checksum_negotiate_default(void) {
|
|||||||
}
|
}
|
||||||
return CHECKSUM_ALGO_XXH64;
|
return CHECKSUM_ALGO_XXH64;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int checksum_choice_resolve(void) {
|
||||||
|
bool specified = false;
|
||||||
|
int env = env_choice_first("RSYNC_CHECKSUM_LIST", checksum_algo_from_name, &specified);
|
||||||
|
if (specified)
|
||||||
|
return env; /* -1 = the list named no supported checksum */
|
||||||
|
return (int)compiled_checksum_preference_first();
|
||||||
|
}
|
||||||
|
|
||||||
|
ChecksumAlgo checksum_negotiate_default(void) {
|
||||||
|
int resolved = checksum_choice_resolve();
|
||||||
|
return resolved >= 0 ? (ChecksumAlgo)resolved : compiled_checksum_preference_first();
|
||||||
|
}
|
||||||
|
|||||||
@@ -51,6 +51,13 @@ bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t
|
|||||||
bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, uint8_t* out,
|
bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, uint8_t* out,
|
||||||
size_t out_capacity, size_t* out_len);
|
size_t out_capacity, size_t* out_len);
|
||||||
|
|
||||||
|
/* Descriptor form of the streaming digest: rewinds `fd` to the start and hashes
|
||||||
|
* to EOF without closing it. Used by the --verify-basis path to hash an
|
||||||
|
* already-open, root-confined basis descriptor. Same contract as
|
||||||
|
* checksum_digest_file. */
|
||||||
|
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
|
||||||
|
size_t* out_len);
|
||||||
|
|
||||||
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
|
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
|
||||||
* Accepts "xxh64"/"xxhash", "xxh3", "xxh128", "md5", "md4", "sha1", "none".
|
* Accepts "xxh64"/"xxhash", "xxh3", "xxh128", "md5", "md4", "sha1", "none".
|
||||||
* "auto" is not an algorithm here; the caller resolves it to the negotiated
|
* "auto" is not an algorithm here; the caller resolves it to the negotiated
|
||||||
@@ -72,4 +79,11 @@ uint8_t checksum_digest_len(ChecksumAlgo algo);
|
|||||||
* xxh128 xxh3 xxh64 md5 md4 sha1 none). Used to resolve "auto". */
|
* xxh128 xxh3 xxh64 md5 md4 sha1 none). Used to resolve "auto". */
|
||||||
ChecksumAlgo checksum_negotiate_default(void);
|
ChecksumAlgo checksum_negotiate_default(void);
|
||||||
|
|
||||||
|
/* Resolve "auto" the way rsync does: the first supported name in
|
||||||
|
* RSYNC_CHECKSUM_LIST (whitespace-separated, client half ends at '&'), then the
|
||||||
|
* compiled-in preference order when the variable is unset/blank. Returns -1
|
||||||
|
* when the variable is set but names no supported checksum (rsync's failed
|
||||||
|
* negotiation), otherwise a valid ChecksumAlgo id. */
|
||||||
|
int checksum_choice_resolve(void);
|
||||||
|
|
||||||
#endif /* CHECKSUM_H */
|
#endif /* CHECKSUM_H */
|
||||||
|
|||||||
@@ -2,6 +2,7 @@
|
|||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
|
#include "utils.h"
|
||||||
#include <limits.h>
|
#include <limits.h>
|
||||||
#include <lz4.h>
|
#include <lz4.h>
|
||||||
#include <stdatomic.h>
|
#include <stdatomic.h>
|
||||||
@@ -119,7 +120,7 @@ bool compression_algo_enabled(CompressionAlgo algo) {
|
|||||||
return algo != COMPRESSION_ALGO_NONE;
|
return algo != COMPRESSION_ALGO_NONE;
|
||||||
}
|
}
|
||||||
|
|
||||||
CompressionAlgo compression_negotiate_default(void) {
|
static CompressionAlgo compiled_preference_first(void) {
|
||||||
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
||||||
* resolves to zstd. */
|
* resolves to zstd. */
|
||||||
static const CompressionAlgo preference[] = {
|
static const CompressionAlgo preference[] = {
|
||||||
@@ -133,6 +134,57 @@ CompressionAlgo compression_negotiate_default(void) {
|
|||||||
return COMPRESSION_ALGO_ZSTD;
|
return COMPRESSION_ALGO_ZSTD;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int compression_choice_resolve(void) {
|
||||||
|
bool specified = false;
|
||||||
|
int env = env_choice_first("RSYNC_COMPRESS_LIST", compression_algo_from_name, &specified);
|
||||||
|
if (specified)
|
||||||
|
return env; /* -1 = the list named no supported codec */
|
||||||
|
return (int)compiled_preference_first();
|
||||||
|
}
|
||||||
|
|
||||||
|
CompressionAlgo compression_negotiate_default(void) {
|
||||||
|
int resolved = compression_choice_resolve();
|
||||||
|
return resolved >= 0 ? (CompressionAlgo)resolved : compiled_preference_first();
|
||||||
|
}
|
||||||
|
|
||||||
|
int compression_default_level(CompressionAlgo algo) {
|
||||||
|
switch (algo) {
|
||||||
|
case COMPRESSION_ALGO_ZSTD:
|
||||||
|
return ZSTD_CLEVEL_DEFAULT;
|
||||||
|
case COMPRESSION_ALGO_ZLIB:
|
||||||
|
case COMPRESSION_ALGO_ZLIBX:
|
||||||
|
return 6; /* rsync resolves zlib's Z_DEFAULT_COMPRESSION (-1) to 6 */
|
||||||
|
case COMPRESSION_ALGO_LZ4:
|
||||||
|
return 1; /* rsync lz4 level is 0/ignored; positive keeps the gate on */
|
||||||
|
case COMPRESSION_ALGO_NONE:
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
int compression_clamp_level(CompressionAlgo algo, int level) {
|
||||||
|
switch (algo) {
|
||||||
|
case COMPRESSION_ALGO_ZSTD:
|
||||||
|
if (level < 1)
|
||||||
|
return 1;
|
||||||
|
if (level > 22)
|
||||||
|
return 22;
|
||||||
|
return level;
|
||||||
|
case COMPRESSION_ALGO_ZLIB:
|
||||||
|
case COMPRESSION_ALGO_ZLIBX:
|
||||||
|
if (level < 1)
|
||||||
|
return 1;
|
||||||
|
if (level > 9)
|
||||||
|
return 9;
|
||||||
|
return level;
|
||||||
|
case COMPRESSION_ALGO_LZ4:
|
||||||
|
return 1; /* ignored by lz4_compress; keeps the "compress" gate on */
|
||||||
|
case COMPRESSION_ALGO_NONE:
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
return level;
|
||||||
|
}
|
||||||
|
|
||||||
void compression_set_algo(CompressionAlgo algo) {
|
void compression_set_algo(CompressionAlgo algo) {
|
||||||
if (compression_algo_valid((int)algo))
|
if (compression_algo_valid((int)algo))
|
||||||
atomic_store(&g_compression_algo, (int)algo);
|
atomic_store(&g_compression_algo, (int)algo);
|
||||||
|
|||||||
@@ -35,6 +35,26 @@ bool compression_algo_valid(int algo);
|
|||||||
* "auto". */
|
* "auto". */
|
||||||
CompressionAlgo compression_negotiate_default(void);
|
CompressionAlgo compression_negotiate_default(void);
|
||||||
|
|
||||||
|
/* Resolve "auto" the way rsync does: the first supported name in
|
||||||
|
* RSYNC_COMPRESS_LIST (whitespace-separated, client half ends at '&'), then the
|
||||||
|
* compiled-in preference order when the variable is unset/blank. Returns -1
|
||||||
|
* when the variable is set but names no supported codec (rsync's failed
|
||||||
|
* negotiation), otherwise a valid CompressionAlgo id. */
|
||||||
|
int compression_choice_resolve(void);
|
||||||
|
|
||||||
|
/* rsync 3.4.1's per-codec default level, applied when the user did not pass
|
||||||
|
* --compress-level/--zl. zstd uses ZSTD_CLEVEL_DEFAULT (3) and zlib/zlibx the
|
||||||
|
* resolved Z_DEFAULT_COMPRESSION (6). lz4 has no tunable level in rsync
|
||||||
|
* (always the default acceleration); FastSync returns a positive placeholder so
|
||||||
|
* its "level > 0" compression gate stays engaged, and lz4_compress ignores the
|
||||||
|
* value, so the output is identical to rsync's. none is 0. */
|
||||||
|
int compression_default_level(CompressionAlgo algo);
|
||||||
|
|
||||||
|
/* Clamp an explicit --compress-level to the codec's accepted range the way
|
||||||
|
* rsync's init_compression_level() does: zstd 1..22, zlib/zlibx 1..9, lz4
|
||||||
|
* ignored (fixed positive placeholder), none 0. */
|
||||||
|
int compression_clamp_level(CompressionAlgo algo, int level);
|
||||||
|
|
||||||
/* True when the algorithm actually compresses (i.e. is not NONE). */
|
/* True when the algorithm actually compresses (i.e. is not NONE). */
|
||||||
bool compression_algo_enabled(CompressionAlgo algo);
|
bool compression_algo_enabled(CompressionAlgo algo);
|
||||||
|
|
||||||
|
|||||||
+131
-2
@@ -70,6 +70,8 @@ static void config_set_defaults(Config* config) {
|
|||||||
config->ignore_missing_args = false;
|
config->ignore_missing_args = false;
|
||||||
config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
|
config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
|
||||||
config->cli_exit_code = 0;
|
config->cli_exit_code = 0;
|
||||||
|
config->compression_level_set = false;
|
||||||
|
config->checksum_choice_set = false;
|
||||||
config->filters = NULL;
|
config->filters = NULL;
|
||||||
config->files_from = NULL;
|
config->files_from = NULL;
|
||||||
config->files_from_set = NULL;
|
config->files_from_set = NULL;
|
||||||
@@ -789,6 +791,8 @@ void config_delete(Config* config) {
|
|||||||
if (config->filters) {
|
if (config->filters) {
|
||||||
array_list_delete(config->filters);
|
array_list_delete(config->filters);
|
||||||
}
|
}
|
||||||
|
filter_rule_list_free(config->protect_rules);
|
||||||
|
config->protect_rules = NULL;
|
||||||
/* A --delay-updates staging tree is transient receiver state: remove any
|
/* A --delay-updates staging tree is transient receiver state: remove any
|
||||||
leftovers on every exit path (success already emptied it). */
|
leftovers on every exit path (success already emptied it). */
|
||||||
if (config->delay_context)
|
if (config->delay_context)
|
||||||
@@ -1024,6 +1028,124 @@ static bool receive_basis_entries(int fd, Config* c, ConfigStringBudget* budget)
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Receiver-side delete-protection rules (protocol 2.28.0). The sender compiles
|
||||||
|
* its command-line selection rules exactly as the scanner does and streams the
|
||||||
|
* result as one bounded, self-describing block (count + per-rule records); the
|
||||||
|
* receiver reconstructs a FilterRuleList for the --delete extras walk. owner
|
||||||
|
* and pattern are charged through the shared ConfigStringBudget and the block
|
||||||
|
* additionally enforces MAX_FILTER_RULES / MAX_FILTER_BYTES. */
|
||||||
|
static bool send_protect_entries(int fd, const Config* c) {
|
||||||
|
int count = c->filters ? c->filters->size : 0;
|
||||||
|
const char** texts = NULL;
|
||||||
|
if (count > 0) {
|
||||||
|
texts = malloc((size_t)count * sizeof(char*));
|
||||||
|
if (!texts)
|
||||||
|
return false;
|
||||||
|
for (int i = 0; i < count; i++)
|
||||||
|
texts[i] = (const char*)c->filters->items[i];
|
||||||
|
}
|
||||||
|
char err[160];
|
||||||
|
FilterRuleList* rules =
|
||||||
|
filter_base_build(texts, count, c->cvs_exclude, c->delete_excluded, err, sizeof(err));
|
||||||
|
free(texts);
|
||||||
|
if (!rules) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "invalid filter rule: %s", err);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = send_int(fd, rules->count);
|
||||||
|
for (int i = 0; ok && i < rules->count; i++) {
|
||||||
|
const FilterRule* r = rules->items[i];
|
||||||
|
/* Mirror the receiver's limit so the peer never receives a rule it will
|
||||||
|
reject as a protocol error. */
|
||||||
|
if (r->pattern && strlen(r->pattern) > MAX_PROTECT_PATTERN_LEN) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "filter pattern exceeds %d bytes", MAX_PROTECT_PATTERN_LEN);
|
||||||
|
filter_rule_list_free(rules);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
ok = send_int(fd, (int)r->action) && send_int(fd, (int)r->sides) &&
|
||||||
|
send_int(fd, r->anchored ? 1 : 0) && send_int(fd, r->dir_only ? 1 : 0) &&
|
||||||
|
send_int(fd, r->negate ? 1 : 0) && send_str(fd, r->owner ? r->owner : "") &&
|
||||||
|
send_str(fd, r->pattern ? r->pattern : "");
|
||||||
|
}
|
||||||
|
filter_rule_list_free(rules);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool receive_protect_entries(int fd, Config* c, ConfigStringBudget* budget) {
|
||||||
|
int count;
|
||||||
|
if (!receive_int(fd, &count))
|
||||||
|
return false;
|
||||||
|
if (count < 0 || count > MAX_FILTER_RULES)
|
||||||
|
return false;
|
||||||
|
if (count == 0)
|
||||||
|
return true;
|
||||||
|
FilterRuleList* list = filter_rule_list_create();
|
||||||
|
if (!list)
|
||||||
|
return false;
|
||||||
|
size_t pattern_bytes = 0;
|
||||||
|
for (int i = 0; i < count; i++) {
|
||||||
|
int action;
|
||||||
|
int sides;
|
||||||
|
bool anchored;
|
||||||
|
bool dir_only;
|
||||||
|
bool negate;
|
||||||
|
if (!receive_int(fd, &action) ||
|
||||||
|
(action != FILTER_ACTION_EXCLUDE && action != FILTER_ACTION_INCLUDE) ||
|
||||||
|
!receive_int(fd, &sides) || sides < (int)FILTER_SIDE_SENDER ||
|
||||||
|
sides > (int)(FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER) ||
|
||||||
|
!receive_wire_bool(fd, &anchored) || !receive_wire_bool(fd, &dir_only) ||
|
||||||
|
!receive_wire_bool(fd, &negate))
|
||||||
|
goto fail;
|
||||||
|
char* owner = config_receive_str(fd, budget);
|
||||||
|
if (!owner)
|
||||||
|
goto fail;
|
||||||
|
char* pattern = config_receive_str(fd, budget);
|
||||||
|
if (!pattern || pattern[0] == '\0') {
|
||||||
|
free(owner);
|
||||||
|
free(pattern);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
/* A pattern too long to be evaluated by glob_match against a PATH_MAX path
|
||||||
|
would silently fail to match and leave a protect rule inert (fail-open:
|
||||||
|
the entry is then deleted). Reject it up front as a protocol error
|
||||||
|
rather than accept a rule that can never shield anything. */
|
||||||
|
if (strlen(pattern) > MAX_PROTECT_PATTERN_LEN) {
|
||||||
|
free(owner);
|
||||||
|
free(pattern);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
size_t bytes = strlen(owner) + strlen(pattern);
|
||||||
|
if (bytes > MAX_FILTER_BYTES - pattern_bytes) {
|
||||||
|
free(owner);
|
||||||
|
free(pattern);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
pattern_bytes += bytes;
|
||||||
|
FilterRule* rule = calloc(1, sizeof(FilterRule));
|
||||||
|
if (!rule) {
|
||||||
|
free(owner);
|
||||||
|
free(pattern);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
rule->action = (FilterAction)action;
|
||||||
|
rule->sides = (unsigned)sides;
|
||||||
|
rule->anchored = anchored;
|
||||||
|
rule->dir_only = dir_only;
|
||||||
|
rule->negate = negate;
|
||||||
|
rule->owner = owner;
|
||||||
|
rule->pattern = pattern;
|
||||||
|
if (!filter_rule_list_add(list, rule)) {
|
||||||
|
filter_rule_free(rule);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
c->protect_rules = list;
|
||||||
|
return true;
|
||||||
|
fail:
|
||||||
|
filter_rule_list_free(list);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
static bool send_identity_entries(int fd, const IdentityMap* map, int count) {
|
static bool send_identity_entries(int fd, const IdentityMap* map, int count) {
|
||||||
for (int i = 0; i < count; i++) {
|
for (int i = 0; i < count; i++) {
|
||||||
if (!send_int(fd, map[i].from) || !send_int(fd, map[i].from_hi) || !send_int(fd, map[i].to) ||
|
if (!send_int(fd, map[i].from) || !send_int(fd, map[i].from_hi) || !send_int(fd, map[i].to) ||
|
||||||
@@ -1151,6 +1273,9 @@ fail:
|
|||||||
#define CONFIG_RECV_BLOCK_IDMAP(name) \
|
#define CONFIG_RECV_BLOCK_IDMAP(name) \
|
||||||
receive_identity_entries(fd, budget, c->name##_count, &c->name)
|
receive_identity_entries(fd, budget, c->name##_count, &c->name)
|
||||||
|
|
||||||
|
#define CONFIG_SEND_BLOCK_PROTECT_RULES(name) send_protect_entries(fd, c)
|
||||||
|
#define CONFIG_RECV_BLOCK_PROTECT_RULES(name) receive_protect_entries(fd, c, budget)
|
||||||
|
|
||||||
/* One table entry, applied in sequence. XSEND/XRECV are statement macros so
|
/* One table entry, applied in sequence. XSEND/XRECV are statement macros so
|
||||||
* consecutive entries read as a plain sequence of assignments. */
|
* consecutive entries read as a plain sequence of assignments. */
|
||||||
#define XSEND(name, ctype, def, kind) ok = ok && (CONFIG_SEND_##kind(name));
|
#define XSEND(name, ctype, def, kind) ok = ok && (CONFIG_SEND_##kind(name));
|
||||||
@@ -1188,6 +1313,7 @@ CONFIG_DEFINE_SEND(send_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
|
|||||||
CONFIG_DEFINE_SEND(send_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
CONFIG_DEFINE_SEND(send_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
||||||
CONFIG_DEFINE_SEND(send_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
CONFIG_DEFINE_SEND(send_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
||||||
CONFIG_DEFINE_SEND(send_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
CONFIG_DEFINE_SEND(send_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
||||||
|
CONFIG_DEFINE_SEND(send_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
|
||||||
|
|
||||||
CONFIG_DEFINE_RECV(receive_core_fields, CONFIG_WIRE_CORE_FIELDS)
|
CONFIG_DEFINE_RECV(receive_core_fields, CONFIG_WIRE_CORE_FIELDS)
|
||||||
CONFIG_DEFINE_RECV(receive_delta_fields, CONFIG_WIRE_DELTA_FIELDS)
|
CONFIG_DEFINE_RECV(receive_delta_fields, CONFIG_WIRE_DELTA_FIELDS)
|
||||||
@@ -1208,6 +1334,7 @@ CONFIG_DEFINE_RECV(receive_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
|
|||||||
CONFIG_DEFINE_RECV(receive_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
CONFIG_DEFINE_RECV(receive_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
||||||
CONFIG_DEFINE_RECV(receive_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
CONFIG_DEFINE_RECV(receive_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
||||||
CONFIG_DEFINE_RECV(receive_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
CONFIG_DEFINE_RECV(receive_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
||||||
|
CONFIG_DEFINE_RECV(receive_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
|
||||||
|
|
||||||
#undef XSEND
|
#undef XSEND
|
||||||
#undef XRECV
|
#undef XRECV
|
||||||
@@ -1326,7 +1453,8 @@ bool config_send_wire_block(int file_descriptor, const Config* config) {
|
|||||||
send_privilege_options(file_descriptor, config) &&
|
send_privilege_options(file_descriptor, config) &&
|
||||||
send_copy_as_options(file_descriptor, config) &&
|
send_copy_as_options(file_descriptor, config) &&
|
||||||
send_output_options(file_descriptor, config) &&
|
send_output_options(file_descriptor, config) &&
|
||||||
send_codec_options(file_descriptor, config);
|
send_codec_options(file_descriptor, config) &&
|
||||||
|
send_protect_options(file_descriptor, config);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool config_send(int file_descriptor, const Config* config) {
|
bool config_send(int file_descriptor, const Config* config) {
|
||||||
@@ -1398,7 +1526,8 @@ Config* config_receive_with_validate(int file_descriptor, ConfigValidateFunc val
|
|||||||
!receive_privilege_options(file_descriptor, config, &budget) ||
|
!receive_privilege_options(file_descriptor, config, &budget) ||
|
||||||
!receive_copy_as_options(file_descriptor, config, &budget) ||
|
!receive_copy_as_options(file_descriptor, config, &budget) ||
|
||||||
!receive_output_options(file_descriptor, config, &budget) ||
|
!receive_output_options(file_descriptor, config, &budget) ||
|
||||||
!receive_codec_options(file_descriptor, config, &budget))
|
!receive_codec_options(file_descriptor, config, &budget) ||
|
||||||
|
!receive_protect_options(file_descriptor, config, &budget))
|
||||||
goto error;
|
goto error;
|
||||||
/* Validate/normalize the negotiated codec. compress_choice is the human
|
/* Validate/normalize the negotiated codec. compress_choice is the human
|
||||||
* spelling (NULL or "" when -z was not given); compression_algo is the
|
* spelling (NULL or "" when -z was not given); compression_algo is the
|
||||||
|
|||||||
+81
-10
@@ -4,6 +4,7 @@
|
|||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
#include "checksum.h"
|
#include "checksum.h"
|
||||||
#include "compression.h"
|
#include "compression.h"
|
||||||
|
#include "filter.h"
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
#include <stdint.h>
|
#include <stdint.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
@@ -82,7 +83,7 @@ typedef struct {
|
|||||||
typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF = 2 } SuperMode;
|
typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF = 2 } SuperMode;
|
||||||
|
|
||||||
/* ===========================================================================
|
/* ===========================================================================
|
||||||
* Config wire-field table (single source of truth for protocol 2.27.0).
|
* Config wire-field table (single source of truth for protocol 2.28.0).
|
||||||
*
|
*
|
||||||
* Every field below crosses the wire. The table is the ONLY place a
|
* Every field below crosses the wire. The table is the ONLY place a
|
||||||
* serialized field is named: config.h expands CONFIG_WIRE_FIELDS() to declare
|
* serialized field is named: config.h expands CONFIG_WIRE_FIELDS() to declare
|
||||||
@@ -197,9 +198,18 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
|||||||
X(skip_compress_count, int, 0, INT_SKIPCOUNT) \
|
X(skip_compress_count, int, 0, INT_SKIPCOUNT) \
|
||||||
X(skip_compress_suffixes, char**, NULL, BLOCK_SKIP_SUFFIXES)
|
X(skip_compress_suffixes, char**, NULL, BLOCK_SKIP_SUFFIXES)
|
||||||
|
|
||||||
|
/* FastSync-only --verify-basis (protocol 2.28.0, no version bump by project
|
||||||
|
* decision): restores the stricter content equality on a basis hit. By
|
||||||
|
* default a basis hit is accepted on rsync's metadata quick-check alone (equal
|
||||||
|
* size plus equal mtime, or size alone under --size-only); with this flag the
|
||||||
|
* receiver ALSO requires the basis bytes' whole-file digest (the negotiated
|
||||||
|
* --checksum-choice algorithm) to equal the sender's, exactly FastSync's
|
||||||
|
* historical behavior. It is a receiver policy and crosses the wire so the
|
||||||
|
* receiver knows whether to read and hash the basis content. */
|
||||||
#define CONFIG_WIRE_BASIS_FIELDS(X) \
|
#define CONFIG_WIRE_BASIS_FIELDS(X) \
|
||||||
X(basis_count, int, 0, INT_BASISCOUNT) \
|
X(basis_count, int, 0, INT_BASISCOUNT) \
|
||||||
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS)
|
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS) \
|
||||||
|
X(verify_basis, bool, false, BOOL)
|
||||||
|
|
||||||
#define CONFIG_WIRE_FUZZY_FIELDS(X) X(fuzzy, bool, false, BOOL)
|
#define CONFIG_WIRE_FUZZY_FIELDS(X) X(fuzzy, bool, false, BOOL)
|
||||||
|
|
||||||
@@ -293,6 +303,20 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
|||||||
#define CONFIG_WIRE_CODEC_FIELDS(X) \
|
#define CONFIG_WIRE_CODEC_FIELDS(X) \
|
||||||
X(compression_algo, int, COMPRESSION_ALGO_ZSTD, INT_COMPRESSION_ALGO)
|
X(compression_algo, int, COMPRESSION_ALGO_ZSTD, INT_COMPRESSION_ALGO)
|
||||||
|
|
||||||
|
/* Receiver-side delete-protection filter rules (protocol 2.28.0). The sender
|
||||||
|
* compiles its root-level selection rules exactly as the scanner does
|
||||||
|
* (filter_base_build over --filter/-f/--exclude/--include/-C) and streams them
|
||||||
|
* as one self-describing, bounded block (count followed by per-rule records).
|
||||||
|
* The receiver reconstructs `protect_rules` and evaluates them against
|
||||||
|
* DESTINATION-ONLY entries during the --delete extras walk, so a
|
||||||
|
* `protect`/`P` rule protects an extra that never appeared on the sender
|
||||||
|
* (rsync re-derives deletion protection from the filter list; FastSync
|
||||||
|
* historically derived it only from the source scan). `protect_rules` is NULL
|
||||||
|
* on the sender and is owned/freed by the receiver Config. Bounded by
|
||||||
|
* MAX_FILTER_RULES and MAX_FILTER_BYTES; an unknown action/sides is a protocol
|
||||||
|
* error. */
|
||||||
|
#define CONFIG_WIRE_PROTECT_FIELDS(X) X(protect_rules, FilterRuleList*, NULL, BLOCK_PROTECT_RULES)
|
||||||
|
|
||||||
/* All serialized fields, in exact wire order. Concatenating the per-segment
|
/* All serialized fields, in exact wire order. Concatenating the per-segment
|
||||||
* lists here is what keeps the declaration order = the wire order. */
|
* lists here is what keeps the declaration order = the wire order. */
|
||||||
#define CONFIG_WIRE_FIELDS(X) \
|
#define CONFIG_WIRE_FIELDS(X) \
|
||||||
@@ -315,7 +339,8 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
|||||||
CONFIG_WIRE_PRIVILEGE_FIELDS(X) \
|
CONFIG_WIRE_PRIVILEGE_FIELDS(X) \
|
||||||
CONFIG_WIRE_COPY_AS_FIELDS(X) \
|
CONFIG_WIRE_COPY_AS_FIELDS(X) \
|
||||||
CONFIG_WIRE_OUTPUT_FIELDS(X) \
|
CONFIG_WIRE_OUTPUT_FIELDS(X) \
|
||||||
CONFIG_WIRE_CODEC_FIELDS(X)
|
CONFIG_WIRE_CODEC_FIELDS(X) \
|
||||||
|
CONFIG_WIRE_PROTECT_FIELDS(X)
|
||||||
|
|
||||||
typedef struct Config {
|
typedef struct Config {
|
||||||
/* -j/--threads=N: number of parallel scanner worker threads for the -m
|
/* -j/--threads=N: number of parallel scanner worker threads for the -m
|
||||||
@@ -419,6 +444,12 @@ typedef struct Config {
|
|||||||
* for an unsupported checksum/compress algorithm) so main() can mirror it. */
|
* for an unsupported checksum/compress algorithm) so main() can mirror it. */
|
||||||
int checksum_transfer_algo;
|
int checksum_transfer_algo;
|
||||||
int cli_exit_code;
|
int cli_exit_code;
|
||||||
|
/* Client-only "the user explicitly chose" bits. They let the per-codec
|
||||||
|
* default level / checksum list be applied only when the corresponding
|
||||||
|
* rsync option was omitted (an explicit --compress-level / --checksum-choice
|
||||||
|
* always wins). Never serialized. */
|
||||||
|
bool compression_level_set;
|
||||||
|
bool checksum_choice_set;
|
||||||
|
|
||||||
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
|
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
|
||||||
// never serialized to the wire (the receiver must not learn them).
|
// never serialized to the wire (the receiver must not learn them).
|
||||||
@@ -611,10 +642,14 @@ typedef struct Config {
|
|||||||
source directory is streamed in directory order, and the receiver removes
|
source directory is streamed in directory order, and the receiver removes
|
||||||
each directory's extras when its plan arrives (during) or snapshots them
|
each directory's extras when its plan arrives (during) or snapshots them
|
||||||
and removes them only after a successful transfer (delay). delete_after
|
and removes them only after a successful transfer (delay). delete_after
|
||||||
(and plain --delete) keep the whole-tree commit mode: extras are removed
|
keeps the whole-tree commit mode: extras are removed from a fresh
|
||||||
from a fresh end-of-transfer destination scan only after the whole transfer
|
end-of-transfer destination scan only after the whole transfer succeeded.
|
||||||
succeeded. See config_delete_timing_early()/config_delete_timing_per_dir()
|
A plain --delete with no explicit timing flag defaults to delete_during on
|
||||||
below. */
|
the client (cli_finalize_config), matching rsync's --del default; the old
|
||||||
|
late-commit behavior is selected explicitly by --delete-after or the
|
||||||
|
FastSync-only long spelling --delete-commit (an exact alias for
|
||||||
|
--delete-after, mapped onto the same wire field). See
|
||||||
|
config_delete_timing_early()/config_delete_timing_per_dir() below. */
|
||||||
/* partial_dir */
|
/* partial_dir */
|
||||||
// PR #174: Partial transfer resumption
|
// PR #174: Partial transfer resumption
|
||||||
/* suffix */
|
/* suffix */
|
||||||
@@ -1006,11 +1041,44 @@ typedef struct Config {
|
|||||||
* the sender can print rsync's `deleting PATH` lines for a real deletion. No
|
* the sender can print rsync's `deleting PATH` lines for a real deletion. No
|
||||||
* change to the fixed STATUS_STATS record itself; only a new trailing config
|
* change to the fixed STATUS_STATS record itself; only a new trailing config
|
||||||
* bool, which still requires the version bump for the strict lockstep. */
|
* bool, which still requires the version bump for the strict lockstep. */
|
||||||
#define PROTOCOL_VERSION "2.27.0"
|
/* (8) --stats receiver-observed counters (protocol 2.28.0): the fixed
|
||||||
|
* STATUS_STATS record grows from three counters to eight. The receiver now
|
||||||
|
* reports the bytes it literally stored (`literal_data`) and the count of
|
||||||
|
* destination entries it newly CREATED, split by type
|
||||||
|
* (reg/dir/link/special), so the sender can print rsync's exact
|
||||||
|
* `Number of created files: N (reg: X, dir: Y, link: Z, special: W)` line and
|
||||||
|
* an exact `Literal data` total even for delta transfers. The config-frame
|
||||||
|
* LAYOUT is unchanged (no new config field), but the STATUS_STATS body grows,
|
||||||
|
* so a 2.27 peer that does not consume the five new fixed-width counters would
|
||||||
|
* desynchronize on the trailing would-delete path list; the strict
|
||||||
|
* same-version handshake (config_receive rejects a mismatched version before
|
||||||
|
* parsing anything else) keeps mixed deployments from ever reaching that
|
||||||
|
* state. */
|
||||||
|
/* (9) Receiver-side delete protection (still protocol 2.28.0): the config frame
|
||||||
|
* gains one trailing self-describing block carrying the sender's compiled base
|
||||||
|
* filter rules so the receiver can protect DESTINATION-ONLY entries from
|
||||||
|
* --delete with `protect`/`risk` rules (rsync parity). The block appends after
|
||||||
|
* compression_algo; see CONFIG_WIRE_PROTECT_FIELDS. */
|
||||||
|
#define PROTOCOL_VERSION "2.28.0"
|
||||||
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
|
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
|
||||||
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
|
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
|
||||||
#define MAX_BASIS_DIRS 64
|
#define MAX_BASIS_DIRS 64
|
||||||
|
|
||||||
|
/* Bounds on the received receiver-side delete-protection rule block. The rule
|
||||||
|
* count and the aggregate pattern+owner bytes are each capped so a hostile
|
||||||
|
* peer cannot pin unbounded pre-auth memory; both are validated strictly on
|
||||||
|
* receive (alongside the per-string ConfigStringBudget). */
|
||||||
|
/* A peer may supply protect rules; cap the list so a crafted config cannot make
|
||||||
|
* the receiver's delete walk evaluate an unbounded number of glob patterns per
|
||||||
|
* destination entry (glob_match is O(pattern x path)). 1024 is far above any
|
||||||
|
* legitimate selection. */
|
||||||
|
#define MAX_FILTER_RULES 1024
|
||||||
|
#define MAX_FILTER_BYTES (256 * 1024)
|
||||||
|
/* glob_match's DP is capped at 64 Mi work units; a pattern longer than this
|
||||||
|
* could exceed the cap against a PATH_MAX path and silently stop matching,
|
||||||
|
* leaving a protect rule inert. Reject such a rule at receive time. */
|
||||||
|
#define MAX_PROTECT_PATTERN_LEN 8192
|
||||||
|
|
||||||
/* Upper bound on the number of --skip-compress suffixes accepted from the wire.
|
/* Upper bound on the number of --skip-compress suffixes accepted from the wire.
|
||||||
* Each suffix is an independent wire string (up to MAX_STRING_SIZE = 64 KiB), so
|
* Each suffix is an independent wire string (up to MAX_STRING_SIZE = 64 KiB), so
|
||||||
* without this a hostile pre-auth client could otherwise retain
|
* without this a hostile pre-auth client could otherwise retain
|
||||||
@@ -1112,8 +1180,11 @@ bool config_delete_timing_early(const Config* config);
|
|||||||
* commits them only after a fully-successful transfer (delay). */
|
* commits them only after a fully-successful transfer (delay). */
|
||||||
bool config_delete_timing_per_dir(const Config* config);
|
bool config_delete_timing_per_dir(const Config* config);
|
||||||
/* Delete-timing sanity: with deletion enabled at most one timing flag may be
|
/* Delete-timing sanity: with deletion enabled at most one timing flag may be
|
||||||
* set (none = the default delete-after commit timing); without deletion no
|
* set; without deletion no timing flag may be set (each timing flag implies
|
||||||
* timing flag may be set (each timing flag implies --delete). */
|
* --delete). A plain --delete is normalized to delete_during by
|
||||||
|
* cli_finalize_config on the client, so a transmitted use_delete config always
|
||||||
|
* carries exactly one timing; the zero-timing case remains valid only for a
|
||||||
|
* config that has not been through the CLI. */
|
||||||
bool config_has_valid_delete_timing(const Config* config);
|
bool config_has_valid_delete_timing(const Config* config);
|
||||||
|
|
||||||
/* Single source of truth for the cross-field ("combination") invariants a
|
/* Single source of truth for the cross-field ("combination") invariants a
|
||||||
|
|||||||
+150
-30
@@ -331,6 +331,8 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
|||||||
return -1;
|
return -1;
|
||||||
sender->config_sent = true;
|
sender->config_sent = true;
|
||||||
}
|
}
|
||||||
|
if (!send_int(fd, 1)) /* apply = true */
|
||||||
|
return -1;
|
||||||
if (!send_wire_str(fd, node->dir))
|
if (!send_wire_str(fd, node->dir))
|
||||||
return -1;
|
return -1;
|
||||||
if (send_str_section(fd, node->dirs) != 0 || send_str_section(fd, node->files) != 0)
|
if (send_str_section(fd, node->dirs) != 0 || send_str_section(fd, node->files) != 0)
|
||||||
@@ -339,6 +341,29 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Transmit the one-shot per-run config block (protected prefixes, size-pruned
|
||||||
|
* mirrors, --delete-missing-args exact paths) on its own carrier frame, with
|
||||||
|
* apply=false so the receiver consumes the config but walks nothing. This is
|
||||||
|
* how the config still reaches the receiver when the scope allows no directory
|
||||||
|
* plan at all (a --files-from list of bare files synchronizes no directory):
|
||||||
|
* without it, the missing-args exact deletions would be lost. Idempotent. */
|
||||||
|
static int send_config_only(int fd, DeletePlanSender* sender) {
|
||||||
|
if (!sender || sender->config_sent)
|
||||||
|
return 0;
|
||||||
|
if (!send_status(fd, STATUS_DELETE_PLAN) || !send_int(fd, 1))
|
||||||
|
return -1;
|
||||||
|
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
|
||||||
|
send_str_section(fd, sender->size_skipped) != 0 ||
|
||||||
|
send_str_section(fd, sender->missing_args) != 0)
|
||||||
|
return -1;
|
||||||
|
sender->config_sent = true;
|
||||||
|
if (!send_int(fd, 0)) /* apply = false */
|
||||||
|
return -1;
|
||||||
|
if (!send_wire_str(fd, ".") || !send_int(fd, 0) || !send_int(fd, 0))
|
||||||
|
return -1;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
static int send_prefix_plan(int fd, DeletePlanSender* sender, const char* dir) {
|
static int send_prefix_plan(int fd, DeletePlanSender* sender, const char* dir) {
|
||||||
PlanNode* node = plan_find(sender, dir);
|
PlanNode* node = plan_find(sender, dir);
|
||||||
if (!node || node->sent)
|
if (!node || node->sent)
|
||||||
@@ -354,6 +379,10 @@ int delete_plan_send_root(int fd, DeletePlanSender* sender) {
|
|||||||
const char* root = sender->walk_root ? sender->walk_root : ".";
|
const char* root = sender->walk_root ? sender->walk_root : ".";
|
||||||
if (!plan_ensure(sender, root))
|
if (!plan_ensure(sender, root))
|
||||||
return -1;
|
return -1;
|
||||||
|
/* Put the config block on the wire first, on its own carrier frame, so the
|
||||||
|
receiver always sees it even when the scope permits no directory plan. */
|
||||||
|
if (send_config_only(fd, sender) != 0)
|
||||||
|
return -1;
|
||||||
return send_prefix_plan(fd, sender, root);
|
return send_prefix_plan(fd, sender, root);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -412,11 +441,17 @@ struct DeletePlanSession {
|
|||||||
bool dry_run;
|
bool dry_run;
|
||||||
size_t max_delete;
|
size_t max_delete;
|
||||||
size_t deleted;
|
size_t deleted;
|
||||||
/* Removals charged against --max-delete. In --delete-delay mode a path is
|
/* Removals charged against --max-delete. The budget is charged on ACTUAL
|
||||||
planned (and the budget consumed) while scanning, but `deleted` advances
|
removals (an unlink/rmdir that succeeded), matching rsync: a snapshotted
|
||||||
only when the commit actually unlinks it, so an entry that survives the
|
entry that fails removal consumes nothing, so a later extra is still
|
||||||
commit (a directory refilled mid-transfer -> ENOTEMPTY) is not reported. */
|
deleted. `planned` and `deleted` advance together for the inline paths and
|
||||||
|
`apply_missing`; `deleted` is the reported count. */
|
||||||
size_t planned;
|
size_t planned;
|
||||||
|
/* Hard bound on the deferred snapshot list. Because the budget is no longer
|
||||||
|
charged at snapshot time, this independent cap keeps a huge destination
|
||||||
|
from growing the list without limit (it matches the receiver's overall
|
||||||
|
deletion bound). */
|
||||||
|
size_t defer_cap;
|
||||||
size_t skipped;
|
size_t skipped;
|
||||||
bool limit_hit;
|
bool limit_hit;
|
||||||
bool limit_logged;
|
bool limit_logged;
|
||||||
@@ -448,6 +483,7 @@ DeletePlanSession* delete_plan_session_create(const Config* config) {
|
|||||||
config->max_delete >= 0 && (size_t)config->max_delete < DELETE_PLAN_SERVER_LIMIT;
|
config->max_delete >= 0 && (size_t)config->max_delete < DELETE_PLAN_SERVER_LIMIT;
|
||||||
session->max_delete =
|
session->max_delete =
|
||||||
user_limited ? (size_t)config->max_delete : (size_t)DELETE_PLAN_SERVER_LIMIT;
|
user_limited ? (size_t)config->max_delete : (size_t)DELETE_PLAN_SERVER_LIMIT;
|
||||||
|
session->defer_cap = DELETE_PLAN_SERVER_LIMIT;
|
||||||
session->protected_prefixes = array_list_create(free);
|
session->protected_prefixes = array_list_create(free);
|
||||||
session->size_skipped = array_list_create(free);
|
session->size_skipped = array_list_create(free);
|
||||||
session->missing = array_list_create(free);
|
session->missing = array_list_create(free);
|
||||||
@@ -539,12 +575,18 @@ static int open_plan_dir(const Config* config, const char* dir) {
|
|||||||
typedef struct PlanSkips {
|
typedef struct PlanSkips {
|
||||||
DeleteSkipEntry* entries;
|
DeleteSkipEntry* entries;
|
||||||
int count;
|
int count;
|
||||||
|
/* Receiver-side delete-protection rules received on the config frame (NULL
|
||||||
|
when the sender sent none). Evaluated per extra so a protect/risk rule is
|
||||||
|
honored under --delete-during/--delete-delay exactly like the whole-tree
|
||||||
|
commit walker. */
|
||||||
|
const FilterRuleList* protect_rules;
|
||||||
} PlanSkips;
|
} PlanSkips;
|
||||||
|
|
||||||
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
|
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
|
||||||
PlanSkips* out) {
|
PlanSkips* out) {
|
||||||
out->entries = NULL;
|
out->entries = NULL;
|
||||||
out->count = 0;
|
out->count = 0;
|
||||||
|
out->protect_rules = config->protect_rules;
|
||||||
int count = (config->delay_updates ? 1 : 0) + config->basis_count +
|
int count = (config->delay_updates ? 1 : 0) + config->basis_count +
|
||||||
session->protected_prefixes->size + session->size_skipped->size;
|
session->protected_prefixes->size + session->size_skipped->size;
|
||||||
if (count == 0)
|
if (count == 0)
|
||||||
@@ -592,10 +634,15 @@ static void log_deleted(const char* rel) {
|
|||||||
free(escaped);
|
free(escaped);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Append a snapshot path for --delete-delay. The budget is charged here, but
|
/* Append a snapshot path for --delete-delay. The budget is NOT charged here:
|
||||||
* `deleted` is not: the path counts only once apply_deferred_path truly
|
* the remover charges --max-delete only when a path is actually unlinked (see
|
||||||
* unlinks it. */
|
* apply_deferred_path), so a snapshotted entry that survives ENOTEMPTY cannot
|
||||||
|
* deny budget to a later extra. The independent `defer_cap` bounds the list. */
|
||||||
static bool defer_add(DeletePlanSession* session, const char* rel) {
|
static bool defer_add(DeletePlanSession* session, const char* rel) {
|
||||||
|
if ((size_t)session->deferred->size >= session->defer_cap) {
|
||||||
|
note_skipped(session);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
char* copy = str_dup(rel);
|
char* copy = str_dup(rel);
|
||||||
if (!copy)
|
if (!copy)
|
||||||
return false;
|
return false;
|
||||||
@@ -603,7 +650,6 @@ static bool defer_add(DeletePlanSession* session, const char* rel) {
|
|||||||
free(copy);
|
free(copy);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
session->planned++;
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -636,16 +682,16 @@ static bool process_extra_dir(int dirfd, const char* name, const char* child_rel
|
|||||||
return false;
|
return false;
|
||||||
if (survives)
|
if (survives)
|
||||||
return true;
|
return true;
|
||||||
if (!budget_available(session)) {
|
|
||||||
note_skipped(session);
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
if (session->defer && !force_now) {
|
if (session->defer && !force_now) {
|
||||||
if (!defer_add(session, child_rel))
|
if (!defer_add(session, child_rel))
|
||||||
return false;
|
return false;
|
||||||
*removed = true;
|
*removed = true;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
if (!budget_available(session)) {
|
||||||
|
note_skipped(session);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
if (unlinkat(dirfd, name, AT_REMOVEDIR) == 0) {
|
if (unlinkat(dirfd, name, AT_REMOVEDIR) == 0) {
|
||||||
session->deleted++;
|
session->deleted++;
|
||||||
session->planned++;
|
session->planned++;
|
||||||
@@ -665,13 +711,13 @@ static bool process_extra_dir(int dirfd, const char* name, const char* child_rel
|
|||||||
|
|
||||||
static bool process_extra_file(int dirfd, const char* name, const char* child_rel, bool force_now,
|
static bool process_extra_file(int dirfd, const char* name, const char* child_rel, bool force_now,
|
||||||
DeletePlanSession* session) {
|
DeletePlanSession* session) {
|
||||||
|
if (session->defer && !force_now) {
|
||||||
|
return defer_add(session, child_rel);
|
||||||
|
}
|
||||||
if (!budget_available(session)) {
|
if (!budget_available(session)) {
|
||||||
note_skipped(session);
|
note_skipped(session);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
if (session->defer && !force_now) {
|
|
||||||
return defer_add(session, child_rel);
|
|
||||||
}
|
|
||||||
if (unlinkat(dirfd, name, 0) == 0) {
|
if (unlinkat(dirfd, name, 0) == 0) {
|
||||||
session->deleted++;
|
session->deleted++;
|
||||||
session->planned++;
|
session->planned++;
|
||||||
@@ -722,6 +768,10 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
|
|||||||
bool is_dir = S_ISDIR(st.st_mode);
|
bool is_dir = S_ISDIR(st.st_mode);
|
||||||
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entry->d_name);
|
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entry->d_name);
|
||||||
bool in_keep_files = !is_dir && list_contains_str(keep_files, entry->d_name);
|
bool in_keep_files = !is_dir && list_contains_str(keep_files, entry->d_name);
|
||||||
|
bool rule_protected =
|
||||||
|
skips->protect_rules &&
|
||||||
|
filter_rules_apply_side(skips->protect_rules, child_rel, entry->d_name, is_dir,
|
||||||
|
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT;
|
||||||
if (in_keep_dirs) {
|
if (in_keep_dirs) {
|
||||||
local_survives = true;
|
local_survives = true;
|
||||||
} else if (keep_dirs && !is_dir && list_contains_str(keep_dirs, entry->d_name)) {
|
} else if (keep_dirs && !is_dir && list_contains_str(keep_dirs, entry->d_name)) {
|
||||||
@@ -739,11 +789,18 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
|
|||||||
else if (!removed)
|
else if (!removed)
|
||||||
local_survives = true;
|
local_survives = true;
|
||||||
} else if (is_dir) {
|
} else if (is_dir) {
|
||||||
bool removed = false;
|
if (rule_protected) {
|
||||||
if (!process_extra_dir(dirfd, entry->d_name, child_rel, force_now, skips, session, &removed))
|
|
||||||
operation_ok = false;
|
|
||||||
else if (!removed)
|
|
||||||
local_survives = true;
|
local_survives = true;
|
||||||
|
} else {
|
||||||
|
bool removed = false;
|
||||||
|
if (!process_extra_dir(dirfd, entry->d_name, child_rel, force_now, skips, session,
|
||||||
|
&removed))
|
||||||
|
operation_ok = false;
|
||||||
|
else if (!removed)
|
||||||
|
local_survives = true;
|
||||||
|
}
|
||||||
|
} else if (rule_protected) {
|
||||||
|
local_survives = true;
|
||||||
} else {
|
} else {
|
||||||
if (!process_extra_file(dirfd, entry->d_name, child_rel, force_now, session))
|
if (!process_extra_file(dirfd, entry->d_name, child_rel, force_now, session))
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
@@ -822,6 +879,14 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
|||||||
}
|
}
|
||||||
session->config_seen = true;
|
session->config_seen = true;
|
||||||
}
|
}
|
||||||
|
/* apply=false is the config-only carrier frame: the receiver consumes the
|
||||||
|
config (and the missing-args exact deletions) but must not walk any
|
||||||
|
directory. Every real plan carries apply=true. */
|
||||||
|
int apply;
|
||||||
|
if (!receive_int(fd, &apply) || (apply != 0 && apply != 1)) {
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
char* dir = receive_wire_str(fd);
|
char* dir = receive_wire_str(fd);
|
||||||
ArrayList* dirs = array_list_create(free);
|
ArrayList* dirs = array_list_create(free);
|
||||||
ArrayList* files = array_list_create(free);
|
ArrayList* files = array_list_create(free);
|
||||||
@@ -839,7 +904,7 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
|||||||
if (!session->dry_run && enabled) {
|
if (!session->dry_run && enabled) {
|
||||||
if (!session->defer && !apply_missing(session, config))
|
if (!session->defer && !apply_missing(session, config))
|
||||||
ok = false;
|
ok = false;
|
||||||
if (ok && !apply_plan_dir(session, config, dir, dirs, files))
|
if (ok && apply && !apply_plan_dir(session, config, dir, dirs, files))
|
||||||
ok = false;
|
ok = false;
|
||||||
}
|
}
|
||||||
free(dir);
|
free(dir);
|
||||||
@@ -858,7 +923,9 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
|||||||
}
|
}
|
||||||
|
|
||||||
/* Apply one snapshotted --delete-delay path (post-order: children precede their
|
/* Apply one snapshotted --delete-delay path (post-order: children precede their
|
||||||
* parent directory). */
|
* parent directory). A directory that is still present is re-scanned so content
|
||||||
|
* created after the plan is removed too; every actual removal charges
|
||||||
|
* --max-delete. */
|
||||||
static bool apply_deferred_path(DeletePlanSession* session, const Config* config, const char* rel) {
|
static bool apply_deferred_path(DeletePlanSession* session, const Config* config, const char* rel) {
|
||||||
char* full = path_cat(config->receive_root_directory, rel);
|
char* full = path_cat(config->receive_root_directory, rel);
|
||||||
if (!full)
|
if (!full)
|
||||||
@@ -872,25 +939,78 @@ static bool apply_deferred_path(DeletePlanSession* session, const Config* config
|
|||||||
}
|
}
|
||||||
struct stat st;
|
struct stat st;
|
||||||
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
||||||
bool absent = errno == ENOENT;
|
bool absent = errno == ENOENT || errno == ENOTDIR;
|
||||||
close(parent_fd);
|
close(parent_fd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
return absent;
|
return absent;
|
||||||
}
|
}
|
||||||
int rc;
|
if (S_ISDIR(st.st_mode)) {
|
||||||
if (S_ISDIR(st.st_mode))
|
if (!budget_available(session)) {
|
||||||
rc = unlinkat(parent_fd, leaf, AT_REMOVEDIR);
|
note_skipped(session);
|
||||||
else
|
close(parent_fd);
|
||||||
rc = unlinkat(parent_fd, leaf, 0);
|
free(leaf);
|
||||||
bool ok = rc == 0 || errno == ENOENT || errno == ENOTEMPTY || errno == EEXIST;
|
return true;
|
||||||
if (rc == 0) {
|
}
|
||||||
|
int dirfd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
|
if (dirfd < 0) {
|
||||||
|
bool absent = errno == ENOENT || errno == ENOTDIR;
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
return absent;
|
||||||
|
}
|
||||||
|
PlanSkips skips;
|
||||||
|
if (!build_plan_skips(config, session, &skips)) {
|
||||||
|
close(dirfd);
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool survives = false;
|
||||||
|
bool ok = process_children(dirfd, rel, NULL, NULL, false, true, &skips, session, &survives);
|
||||||
|
free(skips.entries);
|
||||||
|
close(dirfd);
|
||||||
|
if (!ok) {
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!survives) {
|
||||||
|
if (!budget_available(session)) {
|
||||||
|
note_skipped(session);
|
||||||
|
} else if (unlinkat(parent_fd, leaf, AT_REMOVEDIR) == 0) {
|
||||||
|
session->deleted++;
|
||||||
|
session->planned++;
|
||||||
|
log_deleted(rel);
|
||||||
|
notify_deleted(session, rel);
|
||||||
|
} else if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST) {
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (!budget_available(session)) {
|
||||||
|
note_skipped(session);
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (unlinkat(parent_fd, leaf, 0) == 0) {
|
||||||
session->deleted++;
|
session->deleted++;
|
||||||
|
session->planned++;
|
||||||
log_deleted(rel);
|
log_deleted(rel);
|
||||||
notify_deleted(session, rel);
|
notify_deleted(session, rel);
|
||||||
|
} else if (errno != ENOENT) {
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
}
|
}
|
||||||
close(parent_fd);
|
close(parent_fd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
return ok;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
void delete_plan_session_set_delete_observer(DeletePlanSession* session,
|
void delete_plan_session_set_delete_observer(DeletePlanSession* session,
|
||||||
|
|||||||
@@ -48,11 +48,14 @@ void delete_plan_sender_finalize(DeletePlanSender* sender, const ArrayList* sync
|
|||||||
Directory keep entries do not count, so an I/O error that hid every file
|
Directory keep entries do not count, so an I/O error that hid every file
|
||||||
still refuses to delete. */
|
still refuses to delete. */
|
||||||
bool delete_plan_sender_empty(const DeletePlanSender* sender);
|
bool delete_plan_sender_empty(const DeletePlanSender* sender);
|
||||||
/* Attach the global config sections advertised on the first plan frame. */
|
/* Attach the global config sections advertised on the first plan frame. The
|
||||||
|
* block is always transmitted by delete_plan_send_root(), on a config-only
|
||||||
|
* carrier frame when the scope allows no directory plan. */
|
||||||
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
||||||
const ArrayList* size_skipped, const ArrayList* missing_args);
|
const ArrayList* size_skipped, const ArrayList* missing_args);
|
||||||
/* Send the root plan (even before any data, so root extras are handled like
|
/* Send the root plan (even before any data, so root extras are handled like
|
||||||
* rsync's first generator directory). Returns -1 on I/O error. */
|
* rsync's first generator directory), after transmitting the per-run config
|
||||||
|
* block on its own carrier frame. Returns -1 on I/O error. */
|
||||||
int delete_plan_send_root(int fd, DeletePlanSender* sender);
|
int delete_plan_send_root(int fd, DeletePlanSender* sender);
|
||||||
/* Send the plans for every ancestor of `path` (root-first) and, when is_dir,
|
/* Send the plans for every ancestor of `path` (root-first) and, when is_dir,
|
||||||
* for `path` itself; already-sent plans are skipped. */
|
* for `path` itself; already-sent plans are skipped. */
|
||||||
|
|||||||
+365
-22
@@ -15,6 +15,7 @@
|
|||||||
#include <unistd.h>
|
#include <unistd.h>
|
||||||
|
|
||||||
#include "data.h"
|
#include "data.h"
|
||||||
|
#include "checksum.h"
|
||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "file_store.h"
|
#include "file_store.h"
|
||||||
@@ -24,6 +25,13 @@
|
|||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "xattr.h"
|
#include "xattr.h"
|
||||||
|
#include <fcntl.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
/* Files larger than this are not loaded whole for transfer (the sender streams
|
||||||
|
* them); a whole-file digest is computed from the path instead. Kept in sync
|
||||||
|
* with the sender's streaming threshold. */
|
||||||
|
#define STREAM_THRESHOLD (64ULL * 1024 * 1024)
|
||||||
|
|
||||||
static bool write_all(int fd, const void* data, unsigned long long size) {
|
static bool write_all(int fd, const void* data, unsigned long long size) {
|
||||||
const unsigned char* p = data;
|
const unsigned char* p = data;
|
||||||
@@ -39,6 +47,31 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Streaming copy of an open source descriptor into the just-created destination
|
||||||
|
`fd` (already at offset 0). Used by the --copy-dest basis install so a basis
|
||||||
|
larger than any in-memory whole-file bound still materializes without
|
||||||
|
buffering the entire file. `expected_size` is the caller-verified basis
|
||||||
|
size; the copy must produce exactly that many bytes (a short source is a hard
|
||||||
|
error, never a silently truncated destination). The final ftruncate drops
|
||||||
|
any residual tail a raced-in longer source might have left. */
|
||||||
|
static bool copy_fd_all(int dst_fd, int src_fd, unsigned long long expected_size) {
|
||||||
|
unsigned char buf[1 << 20];
|
||||||
|
unsigned long long done = 0;
|
||||||
|
while (done < expected_size) {
|
||||||
|
unsigned long long remaining = expected_size - done;
|
||||||
|
size_t want = remaining < sizeof(buf) ? (size_t)remaining : sizeof(buf);
|
||||||
|
ssize_t n = read(src_fd, buf, want);
|
||||||
|
if (n < 0 && errno == EINTR)
|
||||||
|
continue;
|
||||||
|
if (n <= 0)
|
||||||
|
return false;
|
||||||
|
if (!write_all(dst_fd, buf, (unsigned long long)n))
|
||||||
|
return false;
|
||||||
|
done += (unsigned long long)n;
|
||||||
|
}
|
||||||
|
return ftruncate(dst_fd, (off_t)expected_size) == 0;
|
||||||
|
}
|
||||||
|
|
||||||
/* Preallocate `size` bytes on `fd` before any data is written (--preallocate).
|
/* Preallocate `size` bytes on `fd` before any data is written (--preallocate).
|
||||||
* fallocate(2) reserves real disk blocks, so an out-of-space condition
|
* fallocate(2) reserves real disk blocks, so an out-of-space condition
|
||||||
* (ENOSPC/EDQUOT) surfaces up front instead of partway through a transfer;
|
* (ENOSPC/EDQUOT) surfaces up front instead of partway through a transfer;
|
||||||
@@ -140,6 +173,12 @@ bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, s
|
|||||||
if (file->data->size == 0) {
|
if (file->data->size == 0) {
|
||||||
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
|
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
|
||||||
}
|
}
|
||||||
|
/* A streamed source (data not loaded) may exceed any in-memory whole-file
|
||||||
|
bound; hash it from the file path in bounded buffers instead of forcing a
|
||||||
|
full load. This is the same digest the receiver recomputes on the basis. */
|
||||||
|
if (!file->data->data && file->path && file->data->size > STREAM_THRESHOLD &&
|
||||||
|
checksum_digest_file(algo, seed, file->path, out, out_capacity, out_len))
|
||||||
|
return true;
|
||||||
if (!file->data->data && !file_load_data(file))
|
if (!file->data->data && !file_load_data(file))
|
||||||
return false;
|
return false;
|
||||||
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
|
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
|
||||||
@@ -176,6 +215,7 @@ File* file_create(const char* path) {
|
|||||||
file->is_dir = false;
|
file->is_dir = false;
|
||||||
file->dir_time_only = false;
|
file->dir_time_only = false;
|
||||||
file->basis_link = NULL;
|
file->basis_link = NULL;
|
||||||
|
file->basis_copy = NULL;
|
||||||
file->link_group = 0;
|
file->link_group = 0;
|
||||||
file->link_first = false;
|
file->link_first = false;
|
||||||
file->hardlink_target = NULL;
|
file->hardlink_target = NULL;
|
||||||
@@ -187,6 +227,7 @@ File* file_create(const char* path) {
|
|||||||
file->xattrs = NULL;
|
file->xattrs = NULL;
|
||||||
file->dest_state = (OutputDestState){0};
|
file->dest_state = (OutputDestState){0};
|
||||||
file->matched_bytes = 0;
|
file->matched_bytes = 0;
|
||||||
|
file->literal_bytes = 0;
|
||||||
return file;
|
return file;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -204,6 +245,8 @@ void file_destroy(void* item) {
|
|||||||
file->send_path = NULL;
|
file->send_path = NULL;
|
||||||
free(file->basis_link);
|
free(file->basis_link);
|
||||||
file->basis_link = NULL;
|
file->basis_link = NULL;
|
||||||
|
free(file->basis_copy);
|
||||||
|
file->basis_copy = NULL;
|
||||||
free(file->hardlink_target);
|
free(file->hardlink_target);
|
||||||
file->hardlink_target = NULL;
|
file->hardlink_target = NULL;
|
||||||
free(file->symlink_target);
|
free(file->symlink_target);
|
||||||
@@ -614,6 +657,103 @@ static int open_dir_beneath_root(const char* resolved, const char* root) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs) {
|
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs) {
|
||||||
|
return file_open_secure_parent_counted(path, leaf_out, create_dirs, NULL, NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The logical transfer root expressed in the same coordinate as the secure
|
||||||
|
* parent walk's `rel_buf` (relative to the authorized root, with a leading
|
||||||
|
* '/'), used as the floor at or below which a created directory is a real
|
||||||
|
* file-list entry. The on-disk transfer root is the receive root joined to the
|
||||||
|
* wire path; the mirror scaffolding above it (the absolute source path below
|
||||||
|
* the destination root) is not an rsync entry. Returns an allocated string or
|
||||||
|
* NULL (count every created component). */
|
||||||
|
static char* transfer_root_floor(const Config* config) {
|
||||||
|
if (!config || !config->send_directory || config->send_directory[0] == '\0')
|
||||||
|
return NULL;
|
||||||
|
const char* spec = config->send_directory;
|
||||||
|
const char* after = spec;
|
||||||
|
if (spec[0] == '.' && spec[1] == '/') {
|
||||||
|
after = spec + 2;
|
||||||
|
} else {
|
||||||
|
const char* cut = strstr(spec, "/./");
|
||||||
|
if (cut)
|
||||||
|
after = cut + 3;
|
||||||
|
}
|
||||||
|
while (*after == '/')
|
||||||
|
after++;
|
||||||
|
char* wire_root = str_dup(after);
|
||||||
|
if (!wire_root)
|
||||||
|
return NULL;
|
||||||
|
size_t wlen = strlen(wire_root);
|
||||||
|
while (wlen > 0 && wire_root[wlen - 1] == '/')
|
||||||
|
wire_root[--wlen] = '\0';
|
||||||
|
if (wlen == 0) {
|
||||||
|
free(wire_root);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
char* disk_root = config->receive_root_directory
|
||||||
|
? path_cat(config->receive_root_directory, wire_root)
|
||||||
|
: str_dup(wire_root);
|
||||||
|
free(wire_root);
|
||||||
|
if (!disk_root)
|
||||||
|
return NULL;
|
||||||
|
const char* root_path = utils_get_authorized_root_path();
|
||||||
|
const char* floor = disk_root;
|
||||||
|
if (root_path && root_path[0] == '/') {
|
||||||
|
size_t rl = strlen(root_path);
|
||||||
|
while (rl > 0 && root_path[rl - 1] == '/')
|
||||||
|
rl--;
|
||||||
|
if (strncmp(disk_root, root_path, rl) == 0 && (disk_root[rl] == '/' || disk_root[rl] == '\0'))
|
||||||
|
floor = disk_root + rl;
|
||||||
|
}
|
||||||
|
while (*floor == '/')
|
||||||
|
floor++;
|
||||||
|
char* out = str_dup(floor);
|
||||||
|
free(disk_root);
|
||||||
|
if (!out)
|
||||||
|
return NULL;
|
||||||
|
if (out[0] == '\0') {
|
||||||
|
free(out);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* A created parent component counts toward `Number of created files` only when
|
||||||
|
* its receive-root-relative path is at or below the logical transfer root
|
||||||
|
* (`count_floor`). The transfer root itself corresponds to rsync's `.` entry
|
||||||
|
* (created on a fresh destination, pre-existing otherwise); the mirror
|
||||||
|
* scaffolding above it is FastSync's absolute-path layout, not an rsync entry. */
|
||||||
|
static bool created_dir_counts(const char* count_floor, const char* rel_buf,
|
||||||
|
const char* component) {
|
||||||
|
if (!count_floor)
|
||||||
|
return true;
|
||||||
|
char candidate[PATH_MAX];
|
||||||
|
int n = snprintf(candidate, sizeof(candidate), "%s/%s", rel_buf, component);
|
||||||
|
if (n < 0 || (size_t)n >= sizeof(candidate))
|
||||||
|
return false;
|
||||||
|
const char* cand = candidate;
|
||||||
|
while (*cand == '/')
|
||||||
|
cand++;
|
||||||
|
size_t fl = strlen(count_floor);
|
||||||
|
if (strncmp(cand, count_floor, fl) != 0)
|
||||||
|
return false;
|
||||||
|
return cand[fl] == '\0' || cand[fl] == '/';
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Public wrapper for the receiver's created-directory accounting: the logical
|
||||||
|
* transfer root expressed receive-root-relative, or NULL when the wire paths
|
||||||
|
* carry no mirror scaffolding above it (--relative and --files-from, whose
|
||||||
|
* paths are already relative to the transfer root). The caller frees a
|
||||||
|
* non-NULL result. */
|
||||||
|
char* file_transfer_root_floor(const Config* config) {
|
||||||
|
if (!config || config->relative || config->files_from_set != NULL)
|
||||||
|
return NULL;
|
||||||
|
return transfer_root_floor(config);
|
||||||
|
}
|
||||||
|
|
||||||
|
int file_open_secure_parent_counted(const char* path, char** leaf_out, bool create_dirs,
|
||||||
|
unsigned* dirs_created, const char* count_floor) {
|
||||||
char* copy = str_dup(path);
|
char* copy = str_dup(path);
|
||||||
if (!copy)
|
if (!copy)
|
||||||
return -1;
|
return -1;
|
||||||
@@ -674,6 +814,14 @@ int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs)
|
|||||||
if (next < 0 && create_dirs && errno == ENOENT) {
|
if (next < 0 && create_dirs && errno == ENOENT) {
|
||||||
bool created = mkdirat(fd, component, (mode_t)(0777 & ~(mode_t)file_process_umask())) == 0;
|
bool created = mkdirat(fd, component, (mode_t)(0777 & ~(mode_t)file_process_umask())) == 0;
|
||||||
if (created || errno == EEXIST) {
|
if (created || errno == EEXIST) {
|
||||||
|
/* Protocol 2.28.0: only directories the logical file list would
|
||||||
|
create count toward `Number of created files`; the mirror
|
||||||
|
scaffolding above the transfer root (e.g. the absolute source path
|
||||||
|
under the destination root) is not an rsync entry. `count_floor`
|
||||||
|
is a receive-root-relative prefix that must be reached before a
|
||||||
|
created component is counted. */
|
||||||
|
if (created && dirs_created && created_dir_counts(count_floor, rel_buf, component))
|
||||||
|
(*dirs_created)++;
|
||||||
/* P7 Wave E: --copy-as owns EVERY entry, including the intermediate
|
/* P7 Wave E: --copy-as owns EVERY entry, including the intermediate
|
||||||
directories this walk creates implicitly. Its target ids are a
|
directories this walk creates implicitly. Its target ids are a
|
||||||
global policy, so they are available here without per-entry source
|
global policy, so they are available here without per-entry source
|
||||||
@@ -1026,15 +1174,14 @@ static void restore_extra_fd(int fd, const FileMetadata* metadata, const FileXat
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
static bool file_to_disk_secure_impl(const char* path, const void* data,
|
static bool
|
||||||
unsigned long long data_size, bool inplace, bool sparse,
|
file_to_disk_secure_impl(const char* path, const void* data, unsigned long long data_size,
|
||||||
bool preallocate, const FileMetadata* metadata,
|
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
|
||||||
FileAttrPolicy policy, bool update, bool no_replace,
|
FileAttrPolicy policy, bool update, bool no_replace, bool use_fsync,
|
||||||
bool use_fsync, const char* temp_dir,
|
const char* temp_dir, const FileXattrList* xattrs, bool fake_super,
|
||||||
const FileXattrList* xattrs, bool fake_super,
|
bool keep_partial, unsigned* dirs_created, const char* count_floor) {
|
||||||
bool keep_partial) {
|
|
||||||
char* leaf = NULL;
|
char* leaf = NULL;
|
||||||
int dirfd = file_open_secure_parent(path, &leaf, true);
|
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||||
if (dirfd < 0)
|
if (dirfd < 0)
|
||||||
return false;
|
return false;
|
||||||
int fd = -1;
|
int fd = -1;
|
||||||
@@ -1325,7 +1472,7 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
|
|||||||
"non-atomic copy into the destination directory");
|
"non-atomic copy into the destination directory");
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, update, no_replace, use_fsync, NULL, xattrs, fake_super,
|
policy, update, no_replace, use_fsync, NULL, xattrs, fake_super,
|
||||||
keep_partial);
|
keep_partial, dirs_created, count_floor);
|
||||||
}
|
}
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
@@ -1334,7 +1481,8 @@ bool file_to_disk_secure(const char* path, const void* data, unsigned long long
|
|||||||
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
|
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
|
||||||
FileAttrPolicy policy, const char* temp_dir) {
|
FileAttrPolicy policy, const char* temp_dir) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, false, false, false, temp_dir, NULL, false, false);
|
policy, false, false, false, temp_dir, NULL, false, false, NULL,
|
||||||
|
NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
|
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
|
||||||
@@ -1342,7 +1490,8 @@ bool file_to_disk_secure_update(const char* path, const void* data, unsigned lon
|
|||||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||||
const char* temp_dir) {
|
const char* temp_dir) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, true, false, false, temp_dir, NULL, false, false);
|
policy, true, false, false, temp_dir, NULL, false, false, NULL,
|
||||||
|
NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
|
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
|
||||||
@@ -1350,7 +1499,8 @@ bool file_to_disk_secure_with_fsync(const char* path, const void* data,
|
|||||||
bool preallocate, const FileMetadata* metadata,
|
bool preallocate, const FileMetadata* metadata,
|
||||||
FileAttrPolicy policy, bool use_fsync, const char* temp_dir) {
|
FileAttrPolicy policy, bool use_fsync, const char* temp_dir) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, false, false, use_fsync, temp_dir, NULL, false, false);
|
policy, false, false, use_fsync, temp_dir, NULL, false, false,
|
||||||
|
NULL, NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_to_disk_secure_no_replace(const char* path, const void* data,
|
bool file_to_disk_secure_no_replace(const char* path, const void* data,
|
||||||
@@ -1358,7 +1508,8 @@ bool file_to_disk_secure_no_replace(const char* path, const void* data,
|
|||||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||||
const char* temp_dir) {
|
const char* temp_dir) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
|
||||||
policy, false, true, false, temp_dir, NULL, false, false);
|
policy, false, true, false, temp_dir, NULL, false, false, NULL,
|
||||||
|
NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
|
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
|
||||||
@@ -1371,9 +1522,21 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
|
|||||||
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||||
bool no_replace, bool use_fsync, const FileXattrList* xattrs,
|
bool no_replace, bool use_fsync, const FileXattrList* xattrs,
|
||||||
bool fake_super, bool keep_partial, const char* temp_dir) {
|
bool fake_super, bool keep_partial, const char* temp_dir) {
|
||||||
|
return file_to_disk_secure_attrs_counted(path, data, data_size, inplace, sparse, preallocate,
|
||||||
|
metadata, policy, update, no_replace, use_fsync, xattrs,
|
||||||
|
fake_super, keep_partial, temp_dir, NULL, NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
|
||||||
|
unsigned long long data_size, bool inplace, bool sparse,
|
||||||
|
bool preallocate, const FileMetadata* metadata,
|
||||||
|
FileAttrPolicy policy, bool update, bool no_replace,
|
||||||
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
|
bool keep_partial, const char* temp_dir,
|
||||||
|
unsigned* dirs_created, const char* count_floor) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, update, no_replace, use_fsync, temp_dir, xattrs,
|
policy, update, no_replace, use_fsync, temp_dir, xattrs,
|
||||||
fake_super, keep_partial);
|
fake_super, keep_partial, dirs_created, count_floor);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Atomic --link-dest install. The destination is replaced (via a temporary
|
/* Atomic --link-dest install. The destination is replaced (via a temporary
|
||||||
@@ -1391,16 +1554,176 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
|
|||||||
* basis). Likewise `xattrs`/`fake_super` are applied only on the copy
|
* basis). Likewise `xattrs`/`fake_super` are applied only on the copy
|
||||||
* fallback, so a fallback copy preserves the per-file attributes instead of
|
* fallback, so a fallback copy preserves the per-file attributes instead of
|
||||||
* silently dropping them. */
|
* silently dropping them. */
|
||||||
|
/* Streaming --copy-dest basis install: atomically materialize `path` from the
|
||||||
|
* bytes of `basis_path` without holding the file in memory, so a basis larger
|
||||||
|
* than any whole-file bound still works. Mirrors the ordinary secure store
|
||||||
|
* path (confined parent walk, temp + rename, --update/--ignore-existing/
|
||||||
|
* --preallocate/--temp-dir) but sources the data from the basis descriptor
|
||||||
|
* rather than a caller buffer, and applies the SOURCE metadata (rsync copies
|
||||||
|
* then fixes attributes). A hard-link install that falls back to a byte copy
|
||||||
|
* also routes through here when the caller supplies the basis path. */
|
||||||
|
static bool file_copy_basis_stream_impl(const char* path, const char* basis_path,
|
||||||
|
unsigned long long expected_size, bool preallocate,
|
||||||
|
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||||
|
bool update, bool no_replace, bool use_fsync,
|
||||||
|
const FileXattrList* xattrs, bool fake_super,
|
||||||
|
const char* temp_dir, unsigned* dirs_created,
|
||||||
|
const char* count_floor) {
|
||||||
|
if (!path || !basis_path)
|
||||||
|
return false;
|
||||||
|
char* leaf = NULL;
|
||||||
|
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||||
|
if (dirfd < 0)
|
||||||
|
return false;
|
||||||
|
|
||||||
|
char* basis_leaf = NULL;
|
||||||
|
int basis_dirfd = file_open_secure_parent(basis_path, &basis_leaf, false);
|
||||||
|
int src_fd = -1;
|
||||||
|
if (basis_dirfd >= 0 && basis_leaf != NULL) {
|
||||||
|
/* O_NONBLOCK rejects a raced-in FIFO without blocking; the S_ISREG gate
|
||||||
|
below is the real type check. */
|
||||||
|
src_fd = openat(basis_dirfd, basis_leaf, O_RDONLY | O_CLOEXEC | O_NOFOLLOW | O_NONBLOCK);
|
||||||
|
struct stat src_st;
|
||||||
|
if (src_fd >= 0 && (fstat(src_fd, &src_st) != 0 || !S_ISREG(src_st.st_mode))) {
|
||||||
|
close(src_fd);
|
||||||
|
src_fd = -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (basis_dirfd >= 0)
|
||||||
|
close(basis_dirfd);
|
||||||
|
free(basis_leaf);
|
||||||
|
if (src_fd < 0) {
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct stat destination_stat;
|
||||||
|
bool destination_is_regular = fstatat(dirfd, leaf, &destination_stat, AT_SYMLINK_NOFOLLOW) == 0 &&
|
||||||
|
S_ISREG(destination_stat.st_mode);
|
||||||
|
if (update && metadata && destination_is_regular && stat_is_newer(&destination_stat, metadata)) {
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (no_replace && file_path_exists_secure(path)) {
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
int scratch_dirfd = -1;
|
||||||
|
if (temp_dir) {
|
||||||
|
scratch_dirfd = file_open_temp_dir(temp_dir);
|
||||||
|
if (scratch_dirfd < 0) {
|
||||||
|
int saved_errno = errno;
|
||||||
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"--temp-dir '%s' could not be opened (rsync requires it to already exist): %s",
|
||||||
|
temp_dir, strerror(saved_errno));
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int target_dirfd = scratch_dirfd >= 0 ? scratch_dirfd : dirfd;
|
||||||
|
int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%llu", leaf, (long)getpid(), ~0ULL);
|
||||||
|
char* tmp = NULL;
|
||||||
|
bool ok = false;
|
||||||
|
if (tmp_size >= 0)
|
||||||
|
tmp = malloc((size_t)tmp_size + 1);
|
||||||
|
if (tmp) {
|
||||||
|
for (unsigned int i = 0; i < 100 && !ok; ++i) {
|
||||||
|
if (scratch_dirfd >= 0)
|
||||||
|
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%llu", leaf, (long)getpid(),
|
||||||
|
next_temp_sequence());
|
||||||
|
else
|
||||||
|
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
|
||||||
|
int fd =
|
||||||
|
openat(target_dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
|
||||||
|
if (fd < 0) {
|
||||||
|
if (errno != EEXIST)
|
||||||
|
break;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
bool wrote = true;
|
||||||
|
if (preallocate && expected_size > 0 && preallocate_fd(fd, expected_size) != 0)
|
||||||
|
wrote = false;
|
||||||
|
if (wrote)
|
||||||
|
wrote = copy_fd_all(fd, src_fd, expected_size);
|
||||||
|
if (wrote && metadata) {
|
||||||
|
if (!policy.perms &&
|
||||||
|
fchmod(fd, file_mode_base(metadata, destination_is_regular,
|
||||||
|
destination_is_regular ? destination_stat.st_mode & 0777
|
||||||
|
: 0)) != 0)
|
||||||
|
wrote = false;
|
||||||
|
if (wrote)
|
||||||
|
wrote = file_restore_metadata_fd(fd, metadata, policy);
|
||||||
|
} else if (wrote && fchmod(fd, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH) != 0) {
|
||||||
|
wrote = false;
|
||||||
|
}
|
||||||
|
if (wrote)
|
||||||
|
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
|
||||||
|
if (wrote && use_fsync)
|
||||||
|
wrote = fsync(fd) == 0;
|
||||||
|
if (close(fd) != 0)
|
||||||
|
wrote = false;
|
||||||
|
if (wrote && renameat(target_dirfd, tmp, dirfd, leaf) != 0)
|
||||||
|
wrote = false;
|
||||||
|
if (!wrote)
|
||||||
|
unlinkat(target_dirfd, tmp, 0);
|
||||||
|
ok = wrote;
|
||||||
|
}
|
||||||
|
free(tmp);
|
||||||
|
}
|
||||||
|
if (!ok && scratch_dirfd >= 0) {
|
||||||
|
/* Retry once with no scratch dir (rsync's EXDEV fallback). */
|
||||||
|
close(scratch_dirfd);
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata,
|
||||||
|
policy, update, no_replace, use_fsync, xattrs, fake_super,
|
||||||
|
NULL, dirs_created, count_floor);
|
||||||
|
}
|
||||||
|
if (scratch_dirfd >= 0)
|
||||||
|
close(scratch_dirfd);
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* --copy-dest basis install (streaming). Applies the source metadata and the
|
||||||
|
per-file xattrs / --fake-super record. */
|
||||||
|
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
|
||||||
|
unsigned long long expected_size, bool preallocate,
|
||||||
|
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||||
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
|
const char* temp_dir) {
|
||||||
|
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata, policy,
|
||||||
|
update, false, use_fsync, xattrs, fake_super, temp_dir, NULL,
|
||||||
|
NULL);
|
||||||
|
}
|
||||||
|
|
||||||
static bool file_to_disk_secure_link_impl(const char* path, const char* basis_path,
|
static bool file_to_disk_secure_link_impl(const char* path, const char* basis_path,
|
||||||
const void* data, unsigned long long data_size,
|
const void* data, unsigned long long data_size,
|
||||||
bool preallocate, const FileMetadata* metadata,
|
bool preallocate, const FileMetadata* metadata,
|
||||||
FileAttrPolicy policy, bool use_fsync,
|
FileAttrPolicy policy, bool use_fsync,
|
||||||
const FileXattrList* xattrs, bool fake_super,
|
const FileXattrList* xattrs, bool fake_super,
|
||||||
const char* temp_dir) {
|
const char* temp_dir, unsigned* dirs_created,
|
||||||
|
const char* count_floor) {
|
||||||
if (!path || !basis_path)
|
if (!path || !basis_path)
|
||||||
return false;
|
return false;
|
||||||
|
/* The caller-supplied buffer is no longer used: the copy fallback streams
|
||||||
|
from the basis path (which may hold an over-limit file). Kept in the
|
||||||
|
signature for the existing API. */
|
||||||
|
(void)data;
|
||||||
char* leaf = NULL;
|
char* leaf = NULL;
|
||||||
int dirfd = file_open_secure_parent(path, &leaf, true);
|
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||||
if (dirfd < 0)
|
if (dirfd < 0)
|
||||||
return false;
|
return false;
|
||||||
|
|
||||||
@@ -1482,10 +1805,17 @@ static bool file_to_disk_secure_link_impl(const char* path, const char* basis_pa
|
|||||||
close(dirfd);
|
close(dirfd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
/* The basis file could not be linked in (missing, cross-device, refused
|
/* The basis file could not be linked in (missing, cross-device, refused
|
||||||
by the filesystem). Write a byte-identical local copy instead. */
|
by the filesystem). Stream a byte-identical local copy from the basis
|
||||||
return file_to_disk_secure_attrs(path, data, data_size, false, false, preallocate, metadata,
|
itself (never the possibly-absent caller buffer) so an over-limit basis
|
||||||
policy, false, false, use_fsync, xattrs, fake_super, false,
|
still materializes. When the basis path is not a readable regular file
|
||||||
temp_dir);
|
(e.g. a directory raced in), fall back to the caller-supplied bytes. */
|
||||||
|
if (file_copy_basis_stream_impl(path, basis_path, data_size, preallocate, metadata, policy,
|
||||||
|
false, false, use_fsync, xattrs, fake_super, temp_dir,
|
||||||
|
dirs_created, count_floor))
|
||||||
|
return true;
|
||||||
|
return file_to_disk_secure_attrs_counted(
|
||||||
|
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
|
||||||
|
xattrs, fake_super, false, temp_dir, dirs_created, count_floor);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (scratch_dirfd >= 0)
|
if (scratch_dirfd >= 0)
|
||||||
@@ -1500,7 +1830,7 @@ bool file_to_disk_secure_link(const char* path, const char* basis_path, const vo
|
|||||||
const FileMetadata* metadata, FileAttrPolicy policy, bool use_fsync,
|
const FileMetadata* metadata, FileAttrPolicy policy, bool use_fsync,
|
||||||
const char* temp_dir) {
|
const char* temp_dir) {
|
||||||
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
|
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
|
||||||
policy, use_fsync, NULL, false, temp_dir);
|
policy, use_fsync, NULL, false, temp_dir, NULL, NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, const void* data,
|
bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, const void* data,
|
||||||
@@ -1508,8 +1838,21 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
|
|||||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
const char* temp_dir) {
|
const char* temp_dir) {
|
||||||
|
return file_to_disk_secure_link_attrs_counted(path, basis_path, data, data_size, preallocate,
|
||||||
|
metadata, policy, use_fsync, xattrs, fake_super,
|
||||||
|
temp_dir, NULL, NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
|
||||||
|
const void* data, unsigned long long data_size,
|
||||||
|
bool preallocate, const FileMetadata* metadata,
|
||||||
|
FileAttrPolicy policy, bool use_fsync,
|
||||||
|
const FileXattrList* xattrs, bool fake_super,
|
||||||
|
const char* temp_dir, unsigned* dirs_created,
|
||||||
|
const char* count_floor) {
|
||||||
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
|
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
|
||||||
policy, use_fsync, xattrs, fake_super, temp_dir);
|
policy, use_fsync, xattrs, fake_super, temp_dir,
|
||||||
|
dirs_created, count_floor);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
|
bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
|
||||||
|
|||||||
@@ -87,6 +87,11 @@ bool file_path_exists_secure(const char* path);
|
|||||||
bool file_stat_secure(const char* path, struct stat* st);
|
bool file_stat_secure(const char* path, struct stat* st);
|
||||||
bool file_destination_is_newer_secure(const char* path, const FileMetadata* metadata);
|
bool file_destination_is_newer_secure(const char* path, const FileMetadata* metadata);
|
||||||
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
|
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
|
||||||
|
/* Protocol 2.28.0 variant: also increments *dirs_created for every missing
|
||||||
|
* parent directory this walk creates that lies strictly below `count_floor`
|
||||||
|
* (a receive-root-relative path, or NULL to count all of them). */
|
||||||
|
int file_open_secure_parent_counted(const char* path, char** leaf_out, bool create_dirs,
|
||||||
|
unsigned* dirs_created, const char* count_floor);
|
||||||
bool file_ensure_directory_secure(const char* path);
|
bool file_ensure_directory_secure(const char* path);
|
||||||
bool file_directory_exists_secure(const char* path);
|
bool file_directory_exists_secure(const char* path);
|
||||||
bool file_rename_secure(const char* old_path, const char* new_path);
|
bool file_rename_secure(const char* old_path, const char* new_path);
|
||||||
@@ -158,5 +163,37 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
|
|||||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
const char* temp_dir);
|
const char* temp_dir);
|
||||||
|
/* Streaming --copy-dest install: atomically materialize `path` by copying the
|
||||||
|
* bytes of `basis_path` through a bounded buffer (no whole-file buffering, so
|
||||||
|
* an arbitrarily large basis works), applying the SOURCE metadata and the
|
||||||
|
* per-file xattrs / --fake-super record. `update` honors a newer destination;
|
||||||
|
* a --temp-dir scratch location falls back to a direct write on EXDEV. */
|
||||||
|
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
|
||||||
|
unsigned long long expected_size, bool preallocate,
|
||||||
|
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||||
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
|
const char* temp_dir);
|
||||||
|
/* Protocol 2.28.0 receiver-stat variants: like the two above but additionally
|
||||||
|
* report through `dirs_created` (when non-NULL) how many parent directories the
|
||||||
|
* confined secure walk had to create that lie strictly below `count_floor` (a
|
||||||
|
* receive-root-relative prefix, or NULL for all). Used to reproduce rsync's
|
||||||
|
* `Number of created files` directory count on a fresh destination. */
|
||||||
|
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
|
||||||
|
unsigned long long data_size, bool inplace, bool sparse,
|
||||||
|
bool preallocate, const FileMetadata* metadata,
|
||||||
|
FileAttrPolicy policy, bool update, bool no_replace,
|
||||||
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
|
bool keep_partial, const char* temp_dir,
|
||||||
|
unsigned* dirs_created, const char* count_floor);
|
||||||
|
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
|
||||||
|
const void* data, unsigned long long data_size,
|
||||||
|
bool preallocate, const FileMetadata* metadata,
|
||||||
|
FileAttrPolicy policy, bool use_fsync,
|
||||||
|
const FileXattrList* xattrs, bool fake_super,
|
||||||
|
const char* temp_dir, unsigned* dirs_created,
|
||||||
|
const char* count_floor);
|
||||||
|
/* The logical transfer root expressed receive-root-relative, or NULL when the
|
||||||
|
* wire paths carry no mirror scaffolding above it. Caller frees non-NULL. */
|
||||||
|
char* file_transfer_root_floor(const Config* config);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|||||||
+233
-113
@@ -37,7 +37,12 @@
|
|||||||
#define MANIFEST_ENTRY_OVERHEAD (sizeof(char*) + 16)
|
#define MANIFEST_ENTRY_OVERHEAD (sizeof(char*) + 16)
|
||||||
|
|
||||||
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config) {
|
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config) {
|
||||||
return file_save_to_disk_full(root_directory, file, config) != FILE_SAVE_ERROR;
|
return file_save_to_disk_full_ex(root_directory, file, config, NULL, NULL) != FILE_SAVE_ERROR;
|
||||||
|
}
|
||||||
|
|
||||||
|
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
|
||||||
|
const Config* config) {
|
||||||
|
return file_save_to_disk_full_ex(root_directory, file, config, NULL, NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* --delay-updates receiver path: write the file into a private staging tree
|
/* --delay-updates receiver path: write the file into a private staging tree
|
||||||
@@ -94,6 +99,12 @@ static FileSaveResult file_stage_delayed_update(const char* root_directory,
|
|||||||
if (file->basis_link) {
|
if (file->basis_link) {
|
||||||
ok = file_to_disk_secure_link(staged_path, file->basis_link, file->data->data, file->data->size,
|
ok = file_to_disk_secure_link(staged_path, file->basis_link, file->data->data, file->data->size,
|
||||||
config->preallocate, metadata, policy, config->use_fsync, NULL);
|
config->preallocate, metadata, policy, config->use_fsync, NULL);
|
||||||
|
} else if (file->basis_copy) {
|
||||||
|
/* --copy-dest basis hit: stream the basis into the staging tree (bounded
|
||||||
|
buffers, so an over-limit basis still stages). */
|
||||||
|
ok = file_copy_basis_stream_attrs(staged_path, file->basis_copy, file->data->size,
|
||||||
|
config->preallocate, metadata, policy, config->update,
|
||||||
|
config->use_fsync, file->xattrs, config->fake_super, NULL);
|
||||||
} else {
|
} else {
|
||||||
ok =
|
ok =
|
||||||
file_to_disk_secure_attrs(staged_path, file->data->data, file->data->size, false, sparse,
|
file_to_disk_secure_attrs(staged_path, file->data->data, file->data->size, false, sparse,
|
||||||
@@ -208,13 +219,14 @@ static FileSaveResult hardlink_sibling_absent_first(const char* destination_path
|
|||||||
--existing/--ignore-existing/--update policies are decided against the final
|
--existing/--ignore-existing/--update policies are decided against the final
|
||||||
destination like every normal write. */
|
destination like every normal write. */
|
||||||
static FileSaveResult file_save_hardlink_sibling(const char* root_directory, const File* file,
|
static FileSaveResult file_save_hardlink_sibling(const char* root_directory, const File* file,
|
||||||
const Config* config) {
|
const Config* config, bool* created) {
|
||||||
Config* cfg = (Config*)config;
|
Config* cfg = (Config*)config;
|
||||||
if (!root_directory || !file || !file->path || !file->hardlink_target)
|
if (!root_directory || !file || !file->path || !file->hardlink_target)
|
||||||
return FILE_SAVE_ERROR;
|
return FILE_SAVE_ERROR;
|
||||||
char* destination_path = path_cat(root_directory, file->path);
|
char* destination_path = path_cat(root_directory, file->path);
|
||||||
if (!destination_path)
|
if (!destination_path)
|
||||||
return FILE_SAVE_ERROR;
|
return FILE_SAVE_ERROR;
|
||||||
|
bool existed = file_path_exists_secure(destination_path);
|
||||||
|
|
||||||
if (cfg->existing && !file_path_exists_secure(destination_path)) {
|
if (cfg->existing && !file_path_exists_secure(destination_path)) {
|
||||||
free(destination_path);
|
free(destination_path);
|
||||||
@@ -278,6 +290,8 @@ static FileSaveResult file_save_hardlink_sibling(const char* root_directory, con
|
|||||||
free(staged_first);
|
free(staged_first);
|
||||||
free(staged_sibling);
|
free(staged_sibling);
|
||||||
free(destination_path);
|
free(destination_path);
|
||||||
|
if (ok && created && !existed)
|
||||||
|
*created = true;
|
||||||
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -324,6 +338,8 @@ static FileSaveResult file_save_hardlink_sibling(const char* root_directory, con
|
|||||||
free(content);
|
free(content);
|
||||||
free(first_disk);
|
free(first_disk);
|
||||||
free(destination_path);
|
free(destination_path);
|
||||||
|
if (ok && created && !existed)
|
||||||
|
*created = true;
|
||||||
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -360,7 +376,7 @@ bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode) {
|
|||||||
* non-device entry must carry an empty rdev.
|
* non-device entry must carry an empty rdev.
|
||||||
*/
|
*/
|
||||||
static FileSaveResult file_save_special_to_disk(const char* root_directory, const File* file,
|
static FileSaveResult file_save_special_to_disk(const char* root_directory, const File* file,
|
||||||
const Config* config) {
|
const Config* config, bool* created) {
|
||||||
/* The empty-path and structural checks stay unconditional; the redundant
|
/* The empty-path and structural checks stay unconditional; the redundant
|
||||||
".." list-path re-check is skipped under --trust-sender exactly like the
|
".." list-path re-check is skipped under --trust-sender exactly like the
|
||||||
receive layer (confinement is deferred to the secure parent walk below,
|
receive layer (confinement is deferred to the secure parent walk below,
|
||||||
@@ -412,6 +428,7 @@ static FileSaveResult file_save_special_to_disk(const char* root_directory, cons
|
|||||||
char* destination = path_cat(root_directory, file->path);
|
char* destination = path_cat(root_directory, file->path);
|
||||||
if (!destination)
|
if (!destination)
|
||||||
return FILE_SAVE_ERROR;
|
return FILE_SAVE_ERROR;
|
||||||
|
bool existed = file_path_exists_secure(destination);
|
||||||
char* leaf = NULL;
|
char* leaf = NULL;
|
||||||
int parent_fd = file_open_secure_parent(destination, &leaf, true);
|
int parent_fd = file_open_secure_parent(destination, &leaf, true);
|
||||||
if (parent_fd < 0) {
|
if (parent_fd < 0) {
|
||||||
@@ -532,6 +549,8 @@ static FileSaveResult file_save_special_to_disk(const char* root_directory, cons
|
|||||||
free(destination);
|
free(destination);
|
||||||
/* A failed required --copy-as ownership marks the node as failed; every other
|
/* A failed required --copy-as ownership marks the node as failed; every other
|
||||||
* identity policy stays best-effort. */
|
* identity policy stays best-effort. */
|
||||||
|
if (owner_ok && created && !existed)
|
||||||
|
*created = true;
|
||||||
return owner_ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
return owner_ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -610,8 +629,13 @@ static FileSaveResult file_save_write_device(const char* root_directory, const F
|
|||||||
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_SKIPPED;
|
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_SKIPPED;
|
||||||
}
|
}
|
||||||
|
|
||||||
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
|
FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File* file,
|
||||||
const Config* config) {
|
const Config* config, bool* created,
|
||||||
|
unsigned* created_dirs) {
|
||||||
|
if (created)
|
||||||
|
*created = false;
|
||||||
|
if (created_dirs)
|
||||||
|
*created_dirs = 0;
|
||||||
/* Central no-mutation guard: a server-contacting --dry-run (or a local batch
|
/* Central no-mutation guard: a server-contacting --dry-run (or a local batch
|
||||||
apply that somehow carries dry_run) must never touch the destination, no
|
apply that somehow carries dry_run) must never touch the destination, no
|
||||||
matter which caller reached this primitive. The per-caller guards remain,
|
matter which caller reached this primitive. The per-caller guards remain,
|
||||||
@@ -634,7 +658,8 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
char* destination_path = NULL;
|
char* destination_path = NULL;
|
||||||
char *backup_path = NULL, *parent_copy = NULL;
|
char *backup_path = NULL, *parent_copy = NULL;
|
||||||
|
|
||||||
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data) ||
|
if (!file || !file->path || !file->data ||
|
||||||
|
(file->data->size != 0 && !file->data->data && !file->basis_link && !file->basis_copy) ||
|
||||||
(!file_get_trust_sender() && has_path_traversal(file->path)) ||
|
(!file_get_trust_sender() && has_path_traversal(file->path)) ||
|
||||||
(backup_enabled &&
|
(backup_enabled &&
|
||||||
(!backup_suffix || backup_suffix[0] == '\0' || strchr(backup_suffix, '/') != NULL ||
|
(!backup_suffix || backup_suffix[0] == '\0' || strchr(backup_suffix, '/') != NULL ||
|
||||||
@@ -658,7 +683,7 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
/* Device/special node (--devices/--specials): recreate the node instead of
|
/* Device/special node (--devices/--specials): recreate the node instead of
|
||||||
writing content (privilege-gated, confined, rdev-validated). */
|
writing content (privilege-gated, confined, rdev-validated). */
|
||||||
if (file->is_special)
|
if (file->is_special)
|
||||||
return file_save_special_to_disk(root_directory, file, config);
|
return file_save_special_to_disk(root_directory, file, config, created);
|
||||||
/* --write-devices: write straight into an existing device node. Writing
|
/* --write-devices: write straight into an existing device node. Writing
|
||||||
into a device is a super-user activity, so --no-super must suppress it just
|
into a device is a super-user activity, so --no-super must suppress it just
|
||||||
like device-node creation; the default AUTO/--super attempt it (the wide
|
like device-node creation; the default AUTO/--super attempt it (the wide
|
||||||
@@ -689,6 +714,7 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
char* dir_path = path_cat(root_directory, file->path);
|
char* dir_path = path_cat(root_directory, file->path);
|
||||||
if (!dir_path)
|
if (!dir_path)
|
||||||
return FILE_SAVE_ERROR;
|
return FILE_SAVE_ERROR;
|
||||||
|
bool dir_existed = file_path_exists_secure(dir_path);
|
||||||
bool ok = file_ensure_directory_secure(dir_path);
|
bool ok = file_ensure_directory_secure(dir_path);
|
||||||
/* P7 Wave E: apply the negotiated ownership to the directory ITSELF (not
|
/* P7 Wave E: apply the negotiated ownership to the directory ITSELF (not
|
||||||
just the files inside it). --copy-as and every explicit identity policy
|
just the files inside it). --copy-as and every explicit identity policy
|
||||||
@@ -712,6 +738,8 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
free(leaf);
|
free(leaf);
|
||||||
}
|
}
|
||||||
free(dir_path);
|
free(dir_path);
|
||||||
|
if (ok && created && !dir_existed)
|
||||||
|
*created = true;
|
||||||
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -728,6 +756,7 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
char* link_path = path_cat(root_directory, file->path);
|
char* link_path = path_cat(root_directory, file->path);
|
||||||
if (!link_path)
|
if (!link_path)
|
||||||
return FILE_SAVE_ERROR;
|
return FILE_SAVE_ERROR;
|
||||||
|
bool link_existed = file_path_exists_secure(link_path);
|
||||||
/* The link value is stored verbatim (rsync -l parity: absolute and
|
/* The link value is stored verbatim (rsync -l parity: absolute and
|
||||||
".."-bearing targets are preserved; the scanner's --safe-links /
|
".."-bearing targets are preserved; the scanner's --safe-links /
|
||||||
--copy-unsafe-links decide which links are sent at all). --munge-links
|
--copy-unsafe-links decide which links are sent at all). --munge-links
|
||||||
@@ -768,6 +797,8 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
ok = file_restore_symlink_metadata(link_path, file->metadata, link_policy,
|
ok = file_restore_symlink_metadata(link_path, file->metadata, link_policy,
|
||||||
config->omit_link_times);
|
config->omit_link_times);
|
||||||
}
|
}
|
||||||
|
if (ok && created && !link_existed)
|
||||||
|
*created = true;
|
||||||
free(link_path);
|
free(link_path);
|
||||||
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
|
||||||
}
|
}
|
||||||
@@ -777,7 +808,7 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
byte-identical copy of) the group's first member. Handled entirely here,
|
byte-identical copy of) the group's first member. Handled entirely here,
|
||||||
before the normal data-write paths (which would create an empty file). */
|
before the normal data-write paths (which would create an empty file). */
|
||||||
if (file->link_group != 0 && !file->link_first && file->hardlink_target != NULL) {
|
if (file->link_group != 0 && !file->link_first && file->hardlink_target != NULL) {
|
||||||
return file_save_hardlink_sibling(root_directory, file, config);
|
return file_save_hardlink_sibling(root_directory, file, config, created);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* These options arrive from the client. --backup-dir, --partial-dir and
|
/* These options arrive from the client. --backup-dir, --partial-dir and
|
||||||
@@ -808,12 +839,18 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
free(disk_path);
|
free(disk_path);
|
||||||
return FILE_SAVE_ERROR;
|
return FILE_SAVE_ERROR;
|
||||||
}
|
}
|
||||||
|
/* Snapshot the final destination's existence BEFORE any backup/force/partial
|
||||||
|
step can move or remove it, so the receiver can report rsync's
|
||||||
|
`Number of created files` (protocol 2.28.0). */
|
||||||
|
bool dest_existed = file_path_exists_secure(destination_path);
|
||||||
|
|
||||||
/* --delay-updates diverts the whole write into the staging tree; the rest of
|
/* --delay-updates diverts the whole write into the staging tree; the rest of
|
||||||
this function is the immediate-install path. */
|
this function is the immediate-install path. */
|
||||||
if (config && config->delay_updates) {
|
if (config && config->delay_updates) {
|
||||||
FileSaveResult result =
|
FileSaveResult result =
|
||||||
file_stage_delayed_update(root_directory, destination_path, file, (Config*)config);
|
file_stage_delayed_update(root_directory, destination_path, file, (Config*)config);
|
||||||
|
if (result == FILE_SAVE_WRITTEN && created && !dest_existed)
|
||||||
|
*created = true;
|
||||||
free(confined_backup);
|
free(confined_backup);
|
||||||
free(confined_partial);
|
free(confined_partial);
|
||||||
free(destination_path);
|
free(destination_path);
|
||||||
@@ -939,19 +976,30 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
existing/ignore-existing/update/backup preamble above has already made the
|
existing/ignore-existing/update/backup preamble above has already made the
|
||||||
policy decision. */
|
policy decision. */
|
||||||
bool ok;
|
bool ok;
|
||||||
|
char* count_floor = file_transfer_root_floor(config);
|
||||||
if (config && file->basis_link) {
|
if (config && file->basis_link) {
|
||||||
ok = file_to_disk_secure_link_attrs(
|
ok = file_to_disk_secure_link_attrs_counted(
|
||||||
disk_path, file->basis_link, file->data->data, file->data->size, config->preallocate,
|
disk_path, file->basis_link, file->data->data, file->data->size, config->preallocate,
|
||||||
metadata, policy, config->use_fsync, file->xattrs, config->fake_super, confined_temp);
|
metadata, policy, config->use_fsync, file->xattrs, config->fake_super, confined_temp,
|
||||||
|
created_dirs, count_floor);
|
||||||
|
} else if (config && file->basis_copy) {
|
||||||
|
/* --copy-dest: stream the basis bytes through a bounded buffer so a basis
|
||||||
|
larger than any whole-file bound still materializes. The source
|
||||||
|
metadata was transmitted with the check frame. */
|
||||||
|
ok = file_copy_basis_stream_attrs(
|
||||||
|
disk_path, file->basis_copy, file->data->size, config->preallocate, metadata, policy,
|
||||||
|
config->update, config->use_fsync, file->xattrs, config->fake_super, confined_temp);
|
||||||
} else {
|
} else {
|
||||||
/* The plain no-replace / update / with-fsync engines, plus per-file xattr
|
/* The plain no-replace / update / with-fsync engines, plus per-file xattr
|
||||||
(-X/-A) and --fake-super application on the written fd. */
|
(-X/-A) and --fake-super application on the written fd. */
|
||||||
ok = file_to_disk_secure_attrs(
|
ok = file_to_disk_secure_attrs_counted(
|
||||||
disk_path, file->data->data, file->data->size, inplace, sparse,
|
disk_path, file->data->data, file->data->size, inplace, sparse,
|
||||||
config && config->preallocate, metadata, policy, config && config->update,
|
config && config->preallocate, metadata, policy, config && config->update,
|
||||||
config && config->ignore_existing, config && config->use_fsync, file->xattrs,
|
config && config->ignore_existing, config && config->use_fsync, file->xattrs,
|
||||||
config ? config->fake_super : false, config ? config->partial : false, confined_temp);
|
config ? config->fake_super : false, config ? config->partial : false, confined_temp,
|
||||||
|
created_dirs, count_floor);
|
||||||
}
|
}
|
||||||
|
free(count_floor);
|
||||||
free(confined_temp);
|
free(confined_temp);
|
||||||
confined_temp = NULL;
|
confined_temp = NULL;
|
||||||
if (!ok)
|
if (!ok)
|
||||||
@@ -972,6 +1020,8 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
|
|||||||
free(confined_partial);
|
free(confined_partial);
|
||||||
free(destination_path);
|
free(destination_path);
|
||||||
free(disk_path);
|
free(disk_path);
|
||||||
|
if (created && !dest_existed)
|
||||||
|
*created = true;
|
||||||
return FILE_SAVE_WRITTEN;
|
return FILE_SAVE_WRITTEN;
|
||||||
|
|
||||||
fail:
|
fail:
|
||||||
@@ -984,6 +1034,37 @@ fail:
|
|||||||
return FILE_SAVE_ERROR;
|
return FILE_SAVE_ERROR;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void receiver_stats_note_saved(ReceiverStats* stats, const File* file, bool created,
|
||||||
|
unsigned created_dirs) {
|
||||||
|
if (!stats || !file)
|
||||||
|
return;
|
||||||
|
/* A basis-dir hit (--link-dest/--copy-dest) materializes bytes the sender
|
||||||
|
* never transferred. rsync reports no literal data and no created entry for
|
||||||
|
* such a file, and does not count the parent directories it creates only to
|
||||||
|
* hold it, so exclude the whole entry from the receiver tallies. */
|
||||||
|
bool basis_sourced = file->basis_link != NULL || file->basis_copy != NULL;
|
||||||
|
if (basis_sourced)
|
||||||
|
return;
|
||||||
|
bool is_sibling = file->link_group != 0 && !file->link_first;
|
||||||
|
if (!file->is_dir && !file->is_symlink && !file->is_special && !is_sibling) {
|
||||||
|
unsigned long long literal = file->literal_bytes;
|
||||||
|
if (literal == 0 && file->matched_bytes == 0)
|
||||||
|
literal = file->data ? file->data->size : 0;
|
||||||
|
stats->literal_bytes += literal;
|
||||||
|
}
|
||||||
|
stats->created_dir += created_dirs;
|
||||||
|
if (!created)
|
||||||
|
return;
|
||||||
|
if (file->is_dir)
|
||||||
|
stats->created_dir++;
|
||||||
|
else if (file->is_symlink)
|
||||||
|
stats->created_link++;
|
||||||
|
else if (file->is_special)
|
||||||
|
stats->created_special++;
|
||||||
|
else
|
||||||
|
stats->created_reg++;
|
||||||
|
}
|
||||||
|
|
||||||
/* Receive a file's xattr block (when the config enables xattr transport) and
|
/* Receive a file's xattr block (when the config enables xattr transport) and
|
||||||
* attach it to `file`. Returns false on a malformed/oversized frame. */
|
* attach it to `file`. Returns false on a malformed/oversized frame. */
|
||||||
static bool receive_file_xattrs(File* file, int fd, const Config* config) {
|
static bool receive_file_xattrs(File* file, int fd, const Config* config) {
|
||||||
@@ -1094,11 +1175,15 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
|
|||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
/* Wire-stats tally: bytes taken straight from the basis file (matched
|
/* Wire-stats tally: bytes taken straight from the basis file (matched
|
||||||
delta blocks). Computed before the delta is destroyed. */
|
delta blocks) and bytes shipped literally (protocol 2.28.0). Computed
|
||||||
|
before the delta is destroyed. */
|
||||||
unsigned long long matched = 0;
|
unsigned long long matched = 0;
|
||||||
|
unsigned long long literal = 0;
|
||||||
for (uint32_t k = 0; k < delta->instruction_count; k++) {
|
for (uint32_t k = 0; k < delta->instruction_count; k++) {
|
||||||
if (delta->instructions[k].type == DELTA_INSTR_BLOCK_MATCH)
|
if (delta->instructions[k].type == DELTA_INSTR_BLOCK_MATCH)
|
||||||
matched += delta->instructions[k].match.length;
|
matched += delta->instructions[k].match.length;
|
||||||
|
else if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
|
||||||
|
literal += delta->instructions[k].literal.length;
|
||||||
}
|
}
|
||||||
void* new_data = delta_apply(old_data, old_size, delta, config->delta_block_size);
|
void* new_data = delta_apply(old_data, old_size, delta, config->delta_block_size);
|
||||||
delta_destroy(delta);
|
delta_destroy(delta);
|
||||||
@@ -1119,6 +1204,7 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
|
|||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
file->matched_bytes = matched;
|
file->matched_bytes = matched;
|
||||||
|
file->literal_bytes = literal;
|
||||||
|
|
||||||
if (config->use_metadata) {
|
if (config->use_metadata) {
|
||||||
int meta_ok = 1;
|
int meta_ok = 1;
|
||||||
@@ -1233,16 +1319,17 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
|
|||||||
|
|
||||||
/* ---- Alternate basis directories (--compare-dest / --copy-dest / --link-dest) ----
|
/* ---- Alternate basis directories (--compare-dest / --copy-dest / --link-dest) ----
|
||||||
* The receiver consults the ordered basis-dir list only when the destination
|
* The receiver consults the ordered basis-dir list only when the destination
|
||||||
* entry is NOT already up to date. An "exact match" requires an equal size,
|
* entry is NOT already up to date. By default an "exact match" is rsync's
|
||||||
* an equal mtime (unless --size-only), and an equal content xxHash64, so a
|
* metadata quick-check: an equal size and an equal mtime (unless --size-only).
|
||||||
* hard link / local copy is only ever made from byte-identical content. */
|
* The FastSync-only --verify-basis additionally requires an equal whole-file
|
||||||
|
* content digest, so a hard link / local copy is only then made from
|
||||||
|
* byte-verified content. */
|
||||||
|
|
||||||
typedef struct BasisMatch {
|
typedef struct BasisMatch {
|
||||||
bool hit;
|
bool hit;
|
||||||
BasisDestType type;
|
BasisDestType type;
|
||||||
char* basis_path; /* owned absolute path of the matched basis file */
|
char* basis_path; /* owned absolute path of the matched basis file */
|
||||||
struct stat st; /* fstat() of the matched basis file */
|
struct stat st; /* fstat() of the matched basis file */
|
||||||
Data* content; /* owned basis bytes (or empty Data), NULL when not loaded */
|
|
||||||
} BasisMatch;
|
} BasisMatch;
|
||||||
|
|
||||||
static void basis_match_free(BasisMatch* match) {
|
static void basis_match_free(BasisMatch* match) {
|
||||||
@@ -1250,8 +1337,6 @@ static void basis_match_free(BasisMatch* match) {
|
|||||||
return;
|
return;
|
||||||
free(match->basis_path);
|
free(match->basis_path);
|
||||||
match->basis_path = NULL;
|
match->basis_path = NULL;
|
||||||
data_destroy(match->content);
|
|
||||||
match->content = NULL;
|
|
||||||
match->hit = false;
|
match->hit = false;
|
||||||
match->type = BASIS_DEST_NONE;
|
match->type = BASIS_DEST_NONE;
|
||||||
}
|
}
|
||||||
@@ -1283,32 +1368,10 @@ static bool basis_open_regular(const char* path, unsigned long long expected_siz
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Read the whole remaining content of an open descriptor. A zero-length file
|
|
||||||
yields an empty Data (data pointer NULL). */
|
|
||||||
static Data* basis_read_content(int fd, unsigned long long size) {
|
|
||||||
if (size == 0)
|
|
||||||
return data_create_reserve(0);
|
|
||||||
if (size > MAX_RECEIVE_WHOLE_FILE_SIZE || size > SIZE_MAX)
|
|
||||||
return NULL;
|
|
||||||
void* buf = protocol_alloc((size_t)size);
|
|
||||||
if (!buf)
|
|
||||||
return NULL;
|
|
||||||
size_t got = 0;
|
|
||||||
while (got < (size_t)size) {
|
|
||||||
ssize_t n = read(fd, (char*)buf + got, (size_t)size - got);
|
|
||||||
if (n <= 0) {
|
|
||||||
free(buf);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
got += (size_t)n;
|
|
||||||
}
|
|
||||||
return data_create(buf, (size_t)size);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* --ignore-times forces every file to be updated, so no basis hit is ever
|
/* --ignore-times forces every file to be updated, so no basis hit is ever
|
||||||
declared (matching rsync, where -I prevents link-dest from linking). */
|
declared (matching rsync, where -I prevents link-dest from linking). */
|
||||||
static bool basis_quick_matches(const Config* config, const struct stat* st, time_t check_mtime,
|
bool file_basis_quick_match(const Config* config, const struct stat* st, time_t check_mtime,
|
||||||
long check_mtime_nsec) {
|
long check_mtime_nsec) {
|
||||||
if (config->size_only)
|
if (config->size_only)
|
||||||
return true;
|
return true;
|
||||||
long mtime_nsec = 0;
|
long mtime_nsec = 0;
|
||||||
@@ -1319,28 +1382,39 @@ static bool basis_quick_matches(const Config* config, const struct stat* st, tim
|
|||||||
config->modify_window);
|
config->modify_window);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Search the basis-dir list in command-line order and return the first exact
|
/* True when a basis hit must be confirmed by a whole-file content digest
|
||||||
match. When load_content is true the matched bytes are kept in out->content
|
(--verify-basis). False is the rsync-parity default: the metadata
|
||||||
so the caller can materialize the file without re-reading it.
|
quick-check alone decides a hit. */
|
||||||
|
bool file_basis_content_required(const Config* config) {
|
||||||
|
return config != NULL && config->verify_basis;
|
||||||
|
}
|
||||||
|
|
||||||
An exact match ALSO requires the basis bytes' digest to equal the source's,
|
/* Search the basis-dir list in command-line order and return the first match.
|
||||||
so `hash_content` gates the content read/hash itself. A server-contacting
|
By default (no --verify-basis) rsync's metadata quick-check is sufficient:
|
||||||
--dry-run passes hash_content=false: no basis file may be read or hashed
|
basis_open_regular has already required an equal size, and
|
||||||
(that would be a 1-bit content oracle against a client-supplied digest), so a
|
file_basis_quick_match applies rsync's mtime (or --size-only) rule.
|
||||||
metadata-only pass can never confirm a hit and declines it. The real path
|
--verify-basis additionally requires the basis bytes' whole-file digest to
|
||||||
always passes hash_content=true, keeping its behavior byte-for-byte. */
|
equal the sender's, restoring FastSync's historical content equality; that
|
||||||
|
digest is computed by streaming the open basis descriptor, so an arbitrarily
|
||||||
|
large basis is verified without buffering it. A copy/link install re-reads
|
||||||
|
the basis from its path in bounded buffers, so no content buffer is kept.
|
||||||
|
|
||||||
|
`hash_content` gates content READS under --verify-basis: a server-contacting
|
||||||
|
--dry-run passes false because hashing a basis against a client-supplied
|
||||||
|
digest would be a 1-bit content oracle. Without --verify-basis a dry-run can
|
||||||
|
still confirm the metadata-only hit without reading any basis bytes, matching
|
||||||
|
rsync's read-only quick-check. */
|
||||||
static bool basis_match_find(const Config* config, const char* check_path,
|
static bool basis_match_find(const Config* config, const char* check_path,
|
||||||
unsigned long long check_size, time_t check_mtime,
|
unsigned long long check_size, time_t check_mtime,
|
||||||
long check_mtime_nsec, const uint8_t* check_digest,
|
long check_mtime_nsec, const uint8_t* check_digest,
|
||||||
size_t check_digest_len, bool load_content, bool hash_content,
|
size_t check_digest_len, bool hash_content, BasisMatch* out) {
|
||||||
BasisMatch* out) {
|
|
||||||
memset(out, 0, sizeof(*out));
|
memset(out, 0, sizeof(*out));
|
||||||
if (!config || !config_has_basis(config) || config->ignore_times)
|
if (!config || !config_has_basis(config) || config->ignore_times)
|
||||||
return false;
|
return false;
|
||||||
/* Dry-run: never read/hash basis content. A hit cannot be decided from
|
/* --verify-basis needs the basis content; a content-blind (dry-run) pass can
|
||||||
metadata alone, so report no match (the caller treats it as would-transfer)
|
never confirm it and must not read the file, so decline without touching
|
||||||
without touching the file's contents. */
|
the basis bytes. */
|
||||||
if (!hash_content)
|
if (file_basis_content_required(config) && !hash_content)
|
||||||
return false;
|
return false;
|
||||||
for (int i = 0; i < config->basis_count; i++) {
|
for (int i = 0; i < config->basis_count; i++) {
|
||||||
const BasisDest* entry = &config->basis_dirs[i];
|
const BasisDest* entry = &config->basis_dirs[i];
|
||||||
@@ -1359,29 +1433,26 @@ static bool basis_match_find(const Config* config, const char* check_path,
|
|||||||
int fd;
|
int fd;
|
||||||
struct stat st;
|
struct stat st;
|
||||||
if (basis_open_regular(candidate, check_size, &fd, &st)) {
|
if (basis_open_regular(candidate, check_size, &fd, &st)) {
|
||||||
if (basis_quick_matches(config, &st, check_mtime, check_mtime_nsec)) {
|
if (file_basis_quick_match(config, &st, check_mtime, check_mtime_nsec)) {
|
||||||
Data* content = basis_read_content(fd, check_size);
|
bool hit = true;
|
||||||
if (content) {
|
if (file_basis_content_required(config)) {
|
||||||
uint8_t basis_digest[CHECKSUM_MAX_DIGEST_LEN];
|
uint8_t basis_digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
size_t basis_len = 0;
|
size_t basis_len = 0;
|
||||||
bool hashed = checksum_digest((ChecksumAlgo)config->checksum_algo, config->checksum_seed,
|
bool hashed =
|
||||||
content->data, content->size, basis_digest,
|
checksum_digest_fd((ChecksumAlgo)config->checksum_algo, config->checksum_seed, fd,
|
||||||
sizeof(basis_digest), &basis_len);
|
basis_digest, sizeof(basis_digest), &basis_len);
|
||||||
if (hashed && basis_len == check_digest_len && check_digest_len > 0 &&
|
hit = hashed && basis_len == check_digest_len && check_digest_len > 0 &&
|
||||||
memcmp(basis_digest, check_digest, check_digest_len) == 0) {
|
memcmp(basis_digest, check_digest, check_digest_len) == 0;
|
||||||
out->hit = true;
|
}
|
||||||
out->type = entry->type;
|
if (hit) {
|
||||||
out->basis_path = candidate;
|
out->hit = true;
|
||||||
candidate = NULL; /* ownership transferred to out */
|
out->type = entry->type;
|
||||||
out->st = st;
|
out->basis_path = candidate;
|
||||||
out->content = load_content ? content : NULL;
|
candidate = NULL; /* ownership transferred to out */
|
||||||
if (!load_content)
|
out->st = st;
|
||||||
data_destroy(content);
|
close(fd);
|
||||||
close(fd);
|
return true;
|
||||||
return true;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
data_destroy(content);
|
|
||||||
}
|
}
|
||||||
close(fd);
|
close(fd);
|
||||||
}
|
}
|
||||||
@@ -1806,6 +1877,12 @@ typedef struct {
|
|||||||
long long check_mtime_nsec;
|
long long check_mtime_nsec;
|
||||||
uint8_t check_digest[CHECKSUM_MAX_DIGEST_LEN];
|
uint8_t check_digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
size_t check_digest_len;
|
size_t check_digest_len;
|
||||||
|
/* Source metadata carried alongside the check frame whenever a basis dir is
|
||||||
|
configured (rsync keeps the whole file list; FastSync's sender-driven
|
||||||
|
incremental path otherwise never transmits metadata for a SKIPPED file).
|
||||||
|
A basis materialization applies these SOURCE attributes instead of the
|
||||||
|
basis inode's, matching rsync's "copy then fix attributes". */
|
||||||
|
FileMetadata* source_metadata;
|
||||||
bool dest_exists; /* any destination entry exists (lstat succeeded) */
|
bool dest_exists; /* any destination entry exists (lstat succeeded) */
|
||||||
bool has_old_file;
|
bool has_old_file;
|
||||||
int old_fd;
|
int old_fd;
|
||||||
@@ -1838,6 +1915,8 @@ static void incremental_check_state_cleanup(IncrementalCheckState* state) {
|
|||||||
if (state->old_fd >= 0)
|
if (state->old_fd >= 0)
|
||||||
close(state->old_fd);
|
close(state->old_fd);
|
||||||
state->old_fd = -1;
|
state->old_fd = -1;
|
||||||
|
file_metadata_destroy(state->source_metadata);
|
||||||
|
state->source_metadata = NULL;
|
||||||
free(state->full_path);
|
free(state->full_path);
|
||||||
state->full_path = NULL;
|
state->full_path = NULL;
|
||||||
free(state->check_path);
|
free(state->check_path);
|
||||||
@@ -1862,7 +1941,7 @@ static IncrementalCheckOutcome incremental_check_receive_request(IncrementalChec
|
|||||||
send_error_detail(fd, "invalid check mtime nanoseconds");
|
send_error_detail(fd, "invalid check mtime nanoseconds");
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
if ((config->checksum || config_has_basis(config))) {
|
if ((config->checksum || config->verify_basis)) {
|
||||||
uint8_t wire_len;
|
uint8_t wire_len;
|
||||||
if (!receive_n_data(fd, &wire_len, sizeof(wire_len)) || wire_len == 0 ||
|
if (!receive_n_data(fd, &wire_len, sizeof(wire_len)) || wire_len == 0 ||
|
||||||
wire_len > CHECKSUM_MAX_DIGEST_LEN ||
|
wire_len > CHECKSUM_MAX_DIGEST_LEN ||
|
||||||
@@ -1874,8 +1953,24 @@ static IncrementalCheckOutcome incremental_check_receive_request(IncrementalChec
|
|||||||
if (!receive_n_data(fd, state->check_digest, state->check_digest_len))
|
if (!receive_n_data(fd, state->check_digest, state->check_digest_len))
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
|
/* The sender transmits the source metadata with every basis-configured check
|
||||||
|
so a basis hit can be materialized with the SOURCE's attributes (rsync
|
||||||
|
copies/copies-then-fixes; the receiver would otherwise only have the basis
|
||||||
|
inode's stat). The block is symmetric and consumed unconditionally here,
|
||||||
|
whether or not this file ends up as a basis hit. */
|
||||||
|
if (config_has_basis(config) && config->use_metadata) {
|
||||||
|
int meta_ok = 1;
|
||||||
|
state->source_metadata = metadata_receive(fd, &meta_ok);
|
||||||
|
if (!meta_ok)
|
||||||
|
return INCREMENTAL_ERROR;
|
||||||
|
}
|
||||||
|
|
||||||
if (state->check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
|
/* A basis-configured run may materialize a file larger than the whole-file
|
||||||
|
payload bound: a basis hit is streamed from the basis path (bounded
|
||||||
|
buffers), so the check size is not itself an allocation. Every other
|
||||||
|
path (delta/append/full) still applies MAX_RECEIVE_WHOLE_FILE_SIZE, and a
|
||||||
|
miss simply falls through to the normal transfer with its own bound. */
|
||||||
|
if (!config_has_basis(config) && state->check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
|
||||||
send_error_detail(fd, "check size exceeds receiver limit");
|
send_error_detail(fd, "check size exceeds receiver limit");
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
@@ -1965,6 +2060,20 @@ incremental_check_ignore_existing(const IncrementalCheckState* state) {
|
|||||||
return INCREMENTAL_SKIP;
|
return INCREMENTAL_SKIP;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Metadata for a materialized basis hit: prefer the SOURCE metadata the sender
|
||||||
|
transmitted with the check frame (rsync copies then fixes the destination to
|
||||||
|
the source's attributes); fall back to the basis inode's own stat when
|
||||||
|
metadata was not negotiated. Consumes state->source_metadata on success. */
|
||||||
|
static FileMetadata* basis_take_metadata(IncrementalCheckState* state,
|
||||||
|
const struct stat* basis_st) {
|
||||||
|
if (state->source_metadata) {
|
||||||
|
FileMetadata* meta = state->source_metadata;
|
||||||
|
state->source_metadata = NULL;
|
||||||
|
return meta;
|
||||||
|
}
|
||||||
|
return file_metadata_create(NULL, basis_st, false, false);
|
||||||
|
}
|
||||||
|
|
||||||
/* --link-dest relink of an already up-to-date destination. rsync hard-links a
|
/* --link-dest relink of an already up-to-date destination. rsync hard-links a
|
||||||
destination entry to a matching basis even when the entry is already correct,
|
destination entry to a matching basis even when the entry is already correct,
|
||||||
so a run over an existing tree still maximizes sharing with the basis. Only a
|
so a run over an existing tree still maximizes sharing with the basis. Only a
|
||||||
@@ -1982,7 +2091,7 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
|
|||||||
BasisMatch basis;
|
BasisMatch basis;
|
||||||
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
||||||
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
||||||
true, true, &basis);
|
true, &basis);
|
||||||
/* Only a link-dest hit relinks; a copy-dest/compare-dest hit (or a miss) lets
|
/* Only a link-dest hit relinks; a copy-dest/compare-dest hit (or a miss) lets
|
||||||
the up-to-date check below keep the existing destination. */
|
the up-to-date check below keep the existing destination. */
|
||||||
if (!basis.hit || basis.type != BASIS_DEST_LINK) {
|
if (!basis.hit || basis.type != BASIS_DEST_LINK) {
|
||||||
@@ -1995,11 +2104,16 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
|
|||||||
return INCREMENTAL_CONTINUE;
|
return INCREMENTAL_CONTINUE;
|
||||||
}
|
}
|
||||||
File* materialized = file_create(state->check_path);
|
File* materialized = file_create(state->check_path);
|
||||||
if (materialized && basis.content) {
|
if (materialized) {
|
||||||
data_destroy(materialized->data);
|
data_destroy(materialized->data);
|
||||||
materialized->data = basis.content;
|
materialized->data = data_create_reserve((size_t)state->check_size);
|
||||||
basis.content = NULL;
|
if (!materialized->data) {
|
||||||
materialized->metadata = file_metadata_create(NULL, &basis.st, false, false);
|
file_destroy(materialized);
|
||||||
|
materialized = NULL;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (materialized) {
|
||||||
|
materialized->metadata = basis_take_metadata(state, &basis.st);
|
||||||
materialized->skip = true;
|
materialized->skip = true;
|
||||||
materialized->basis_link = basis.basis_path;
|
materialized->basis_link = basis.basis_path;
|
||||||
basis.basis_path = NULL;
|
basis.basis_path = NULL;
|
||||||
@@ -2007,9 +2121,6 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
|
|||||||
file_destroy(materialized);
|
file_destroy(materialized);
|
||||||
materialized = NULL;
|
materialized = NULL;
|
||||||
}
|
}
|
||||||
} else {
|
|
||||||
file_destroy(materialized);
|
|
||||||
materialized = NULL;
|
|
||||||
}
|
}
|
||||||
if (materialized) {
|
if (materialized) {
|
||||||
if (!send_status(state->fd, STATUS_OK)) {
|
if (!send_status(state->fd, STATUS_OK)) {
|
||||||
@@ -2110,12 +2221,13 @@ static IncrementalCheckOutcome incremental_check_quick_skip(IncrementalCheckStat
|
|||||||
materialize nothing (no basis link/copy, no append/delta/full transfer) and
|
materialize nothing (no basis link/copy, no append/delta/full transfer) and
|
||||||
the sender must send no data, so answer STATUS_DRY_RUN_TRANSFER and stop.
|
the sender must send no data, so answer STATUS_DRY_RUN_TRANSFER and stop.
|
||||||
|
|
||||||
The basis lookup is deliberately content-blind: a real run would only accept
|
The basis lookup is content-blind: under the default metadata quick-check a
|
||||||
a --compare-dest exact hit after hashing the basis file and comparing it with
|
hit needs no basis bytes and is honored here just as in a real run; under
|
||||||
the client-supplied digest, which in a dry-run is a 1-bit content oracle.
|
--verify-basis a real run hashes the basis against the client-supplied digest,
|
||||||
Under dry_run no basis bytes may be read, so an otherwise-matching entry is
|
which in a dry-run is a 1-bit content oracle, so no basis bytes may be read
|
||||||
treated as would-transfer instead of a skip. Everything read here (the
|
and an otherwise-matching entry is reported as would-transfer. Everything
|
||||||
destination file's metadata, basis candidates' metadata) is read-only. */
|
read here (the destination file's metadata, basis candidates' metadata) is
|
||||||
|
read-only. */
|
||||||
static IncrementalCheckOutcome incremental_check_dry_run_shortcut(IncrementalCheckState* state,
|
static IncrementalCheckOutcome incremental_check_dry_run_shortcut(IncrementalCheckState* state,
|
||||||
bool* skipped,
|
bool* skipped,
|
||||||
bool* would_transfer) {
|
bool* would_transfer) {
|
||||||
@@ -2126,12 +2238,14 @@ static IncrementalCheckOutcome incremental_check_dry_run_shortcut(IncrementalChe
|
|||||||
bool skip_via_compare = false;
|
bool skip_via_compare = false;
|
||||||
if (config_has_basis(config) && !config->ignore_times) {
|
if (config_has_basis(config) && !config->ignore_times) {
|
||||||
BasisMatch basis;
|
BasisMatch basis;
|
||||||
/* hash_content=false: a dry-run must not read or hash the basis file. No
|
/* hash_content=false: a dry-run must not read or hash the basis file, so
|
||||||
content comparison is possible, so no compare-dest hit can be confirmed
|
under --verify-basis no compare-dest hit can be confirmed and an
|
||||||
and an otherwise-matching file is reported as would-transfer. */
|
otherwise-matching file is reported as would-transfer. Without
|
||||||
|
--verify-basis the metadata quick-check confirms it without touching any
|
||||||
|
basis bytes. */
|
||||||
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
||||||
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
||||||
false, false, &basis);
|
false, &basis);
|
||||||
if (basis.hit && basis.type == BASIS_DEST_COMPARE && !state->has_old_file)
|
if (basis.hit && basis.type == BASIS_DEST_COMPARE && !state->has_old_file)
|
||||||
skip_via_compare = true;
|
skip_via_compare = true;
|
||||||
basis_match_free(&basis);
|
basis_match_free(&basis);
|
||||||
@@ -2159,7 +2273,7 @@ static IncrementalCheckOutcome incremental_check_try_basis(IncrementalCheckState
|
|||||||
BasisMatch basis;
|
BasisMatch basis;
|
||||||
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
||||||
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
||||||
true, true, &basis);
|
true, &basis);
|
||||||
if (basis.hit) {
|
if (basis.hit) {
|
||||||
if (basis.type == BASIS_DEST_COMPARE) {
|
if (basis.type == BASIS_DEST_COMPARE) {
|
||||||
basis_match_free(&basis);
|
basis_match_free(&basis);
|
||||||
@@ -2169,24 +2283,30 @@ static IncrementalCheckOutcome incremental_check_try_basis(IncrementalCheckState
|
|||||||
return INCREMENTAL_SKIP;
|
return INCREMENTAL_SKIP;
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
|
/* Copy/link installs source their bytes from the basis PATH at install
|
||||||
|
time (bounded buffers), so no whole-file content buffer is needed here
|
||||||
|
even for an over-limit basis. */
|
||||||
File* materialized = file_create(state->check_path);
|
File* materialized = file_create(state->check_path);
|
||||||
if (materialized && basis.content) {
|
if (materialized) {
|
||||||
data_destroy(materialized->data);
|
data_destroy(materialized->data);
|
||||||
materialized->data = basis.content;
|
materialized->data = data_create_reserve((size_t)state->check_size);
|
||||||
basis.content = NULL;
|
if (!materialized->data) {
|
||||||
materialized->metadata = file_metadata_create(NULL, &basis.st, false, false);
|
file_destroy(materialized);
|
||||||
materialized->skip = true; /* receiver must not ack this as a data file */
|
materialized = NULL;
|
||||||
if (basis.type == BASIS_DEST_LINK) {
|
|
||||||
materialized->basis_link = basis.basis_path;
|
|
||||||
basis.basis_path = NULL;
|
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
if (materialized) {
|
||||||
|
materialized->metadata = basis_take_metadata(state, &basis.st);
|
||||||
|
materialized->skip = true; /* receiver must not ack this as a data file */
|
||||||
|
if (basis.type == BASIS_DEST_LINK)
|
||||||
|
materialized->basis_link = basis.basis_path;
|
||||||
|
else
|
||||||
|
materialized->basis_copy = basis.basis_path;
|
||||||
|
basis.basis_path = NULL;
|
||||||
if (!materialized->metadata) {
|
if (!materialized->metadata) {
|
||||||
file_destroy(materialized);
|
file_destroy(materialized);
|
||||||
materialized = NULL;
|
materialized = NULL;
|
||||||
}
|
}
|
||||||
} else {
|
|
||||||
file_destroy(materialized);
|
|
||||||
materialized = NULL;
|
|
||||||
}
|
}
|
||||||
if (materialized) {
|
if (materialized) {
|
||||||
if (!send_status(fd, STATUS_OK)) {
|
if (!send_status(fd, STATUS_OK)) {
|
||||||
@@ -3263,7 +3383,7 @@ static bool delete_extras_budgeted_observed(const Config* config, DeleteManifest
|
|||||||
size_t skipped = 0;
|
size_t skipped = 0;
|
||||||
DeleteWalkResult result = delete_extras_limited_observed(
|
DeleteWalkResult result = delete_extras_limited_observed(
|
||||||
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips, used,
|
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips, used,
|
||||||
&deleted, &skipped, observer, observer_context);
|
config->protect_rules, &deleted, &skipped, observer, observer_context);
|
||||||
if (owned_prefixes) {
|
if (owned_prefixes) {
|
||||||
for (int i = 0; i < config->basis_count; i++)
|
for (int i = 0; i < config->basis_count; i++)
|
||||||
free(owned_prefixes[i]);
|
free(owned_prefixes[i]);
|
||||||
@@ -3444,7 +3564,7 @@ static bool delete_missing_args_budgeted_observed(const Config* config, DeleteMa
|
|||||||
PrefixedDeleteObserver nested = {observer, observer_context, rel};
|
PrefixedDeleteObserver nested = {observer, observer_context, rel};
|
||||||
DeleteWalkResult walk =
|
DeleteWalkResult walk =
|
||||||
no_keeps ? delete_extras_limited_observed(full, no_keeps, NULL, remaining, NULL, 0,
|
no_keeps ? delete_extras_limited_observed(full, no_keeps, NULL, remaining, NULL, 0,
|
||||||
&contents_deleted, &contents_skipped,
|
NULL, &contents_deleted, &contents_skipped,
|
||||||
observer ? prefixed_delete_observer : NULL,
|
observer ? prefixed_delete_observer : NULL,
|
||||||
observer ? &nested : NULL)
|
observer ? &nested : NULL)
|
||||||
: DELETE_WALK_ERROR;
|
: DELETE_WALK_ERROR;
|
||||||
@@ -3555,7 +3675,7 @@ bool manifest_would_delete_list(const Config* config, DeleteManifest* manifest,
|
|||||||
used = idx;
|
used = idx;
|
||||||
}
|
}
|
||||||
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
|
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
|
||||||
skips, used, out, count_out);
|
skips, used, config->protect_rules, out, count_out);
|
||||||
if (owned_prefixes) {
|
if (owned_prefixes) {
|
||||||
for (int i = 0; i < config->basis_count; i++)
|
for (int i = 0; i < config->basis_count; i++)
|
||||||
free(owned_prefixes[i]);
|
free(owned_prefixes[i]);
|
||||||
|
|||||||
@@ -24,6 +24,16 @@ File* file_receive_hardlink(int file_descriptor);
|
|||||||
File* file_receive_symlink(int file_descriptor, const Config* config);
|
File* file_receive_symlink(int file_descriptor, const Config* config);
|
||||||
File* file_receive_special(int file_descriptor);
|
File* file_receive_special(int file_descriptor);
|
||||||
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
|
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
|
||||||
|
/* Testable basis quick-check / verification policy. file_basis_quick_match is
|
||||||
|
* rsync's metadata quick-check for a basis candidate (equal size is required
|
||||||
|
* separately by the caller; this adds the --size-only / mtime / --modify-window
|
||||||
|
* leg). file_basis_content_required reports whether a hit must ALSO be
|
||||||
|
* confirmed by a whole-file content digest (--verify-basis; false is the
|
||||||
|
* default rsync-parity behavior). */
|
||||||
|
bool file_basis_quick_match(const Config* config, const struct stat* st, time_t check_mtime,
|
||||||
|
long check_mtime_nsec);
|
||||||
|
bool file_basis_content_required(const Config* config);
|
||||||
|
|
||||||
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
|
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
|
||||||
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
|
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
|
||||||
* true only on the server-contacting --dry-run path when the file is not up to
|
* true only on the server-contacting --dry-run path when the file is not up to
|
||||||
@@ -179,6 +189,22 @@ typedef enum { FILE_SAVE_ERROR = 0, FILE_SAVE_WRITTEN = 1, FILE_SAVE_SKIPPED = 2
|
|||||||
|
|
||||||
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
|
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
|
||||||
const Config* config);
|
const Config* config);
|
||||||
|
/* Protocol 2.28.0 variant: also reports through `created` (when non-NULL)
|
||||||
|
* whether the destination entry did not exist before this save, and through
|
||||||
|
* `created_dirs` how many parent directories the confined walk created, so the
|
||||||
|
* receiver can build rsync's `Number of created files` breakdown. The plain
|
||||||
|
* file_save_to_disk_full() is this with both out-params NULL. */
|
||||||
|
FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File* file,
|
||||||
|
const Config* config, bool* created,
|
||||||
|
unsigned* created_dirs);
|
||||||
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
|
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
|
||||||
|
|
||||||
|
/* Protocol 2.28.0 receiver counter accumulator: fold one successfully saved
|
||||||
|
* entry into `stats`, adding its receiver-observed literal bytes and, when
|
||||||
|
* `created`, the matching created-by-type counter (regular file / symlink /
|
||||||
|
* special) plus `created_dirs` implicitly-created parent directories.
|
||||||
|
* Non-first hardlink siblings contribute no literal bytes. */
|
||||||
|
void receiver_stats_note_saved(ReceiverStats* stats, const File* file, bool created,
|
||||||
|
unsigned created_dirs);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|||||||
@@ -57,6 +57,11 @@ typedef struct {
|
|||||||
* equals the incoming file, and `data` is kept as the cross-filesystem
|
* equals the incoming file, and `data` is kept as the cross-filesystem
|
||||||
* fallback (a local copy) if the hard link cannot be created. */
|
* fallback (a local copy) if the hard link cannot be created. */
|
||||||
char* basis_link;
|
char* basis_link;
|
||||||
|
/* Receiver-only, --copy-dest: when set (and basis_link is NULL), stream the
|
||||||
|
* basis file's bytes into the destination instead of `data`/`data->size`.
|
||||||
|
* This lets a basis larger than any whole-file bound materialize without
|
||||||
|
* buffering it; the source metadata on `metadata` is applied afterwards. */
|
||||||
|
char* basis_copy;
|
||||||
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
|
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
|
||||||
* id shared by every member of one source inode (0 = not part of a group).
|
* id shared by every member of one source inode (0 = not part of a group).
|
||||||
* The FIRST member (link_first == true) carries its data on the wire and is
|
* The FIRST member (link_first == true) carries its data on the wire and is
|
||||||
@@ -97,6 +102,11 @@ typedef struct {
|
|||||||
* basis file (matched delta blocks) for this entry. 0 when the file was sent
|
* basis file (matched delta blocks) for this entry. 0 when the file was sent
|
||||||
* whole. Accumulated into ReceiverStats.matched_data by the receiver sink. */
|
* whole. Accumulated into ReceiverStats.matched_data by the receiver sink. */
|
||||||
unsigned long long matched_bytes;
|
unsigned long long matched_bytes;
|
||||||
|
/* Receiver-only (protocol 2.28.0) wire-stats tally: the literal delta fragment
|
||||||
|
* bytes this entry carried (DELTA_INSTR_LITERAL). 0 when the file was sent
|
||||||
|
* whole; the sink then falls back to the whole payload size. Accumulated
|
||||||
|
* into ReceiverStats.literal_bytes. */
|
||||||
|
unsigned long long literal_bytes;
|
||||||
} File;
|
} File;
|
||||||
|
|
||||||
/* The path that should be sent on the wire and used for the receiver-side
|
/* The path that should be sent on the wire and used for the receiver-side
|
||||||
|
|||||||
+1
-1
@@ -53,7 +53,7 @@ typedef struct {
|
|||||||
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
|
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
|
||||||
} FilterRule;
|
} FilterRule;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct FilterRuleList {
|
||||||
FilterRule** items; /* owned array of rule pointers */
|
FilterRule** items; /* owned array of rule pointers */
|
||||||
int count;
|
int count;
|
||||||
int capacity;
|
int capacity;
|
||||||
|
|||||||
+15
-13
@@ -105,25 +105,27 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
|
|||||||
bool format_stats_send(int fd, const ReceiverStats* stats) {
|
bool format_stats_send(int fd, const ReceiverStats* stats) {
|
||||||
if (!stats)
|
if (!stats)
|
||||||
return false;
|
return false;
|
||||||
unsigned long long matched = stats->matched_data;
|
unsigned long long fields[8] = {
|
||||||
unsigned long long deleted = stats->deleted_files;
|
stats->matched_data, stats->deleted_files, stats->would_delete_count, stats->literal_bytes,
|
||||||
unsigned long long would = stats->would_delete_count;
|
stats->created_reg, stats->created_dir, stats->created_link, stats->created_special,
|
||||||
return send_n_data(fd, &matched, sizeof(matched)) && send_n_data(fd, &deleted, sizeof(deleted)) &&
|
};
|
||||||
send_n_data(fd, &would, sizeof(would));
|
return send_n_data(fd, fields, sizeof(fields));
|
||||||
}
|
}
|
||||||
|
|
||||||
bool format_stats_receive(int fd, ReceiverStats* stats) {
|
bool format_stats_receive(int fd, ReceiverStats* stats) {
|
||||||
if (!stats)
|
if (!stats)
|
||||||
return false;
|
return false;
|
||||||
unsigned long long matched = 0;
|
unsigned long long fields[8] = {0};
|
||||||
unsigned long long deleted = 0;
|
if (!receive_n_data(fd, fields, sizeof(fields)))
|
||||||
unsigned long long would = 0;
|
|
||||||
if (!receive_n_data(fd, &matched, sizeof(matched)) ||
|
|
||||||
!receive_n_data(fd, &deleted, sizeof(deleted)) || !receive_n_data(fd, &would, sizeof(would)))
|
|
||||||
return false;
|
return false;
|
||||||
memset(stats, 0, sizeof(*stats));
|
memset(stats, 0, sizeof(*stats));
|
||||||
stats->matched_data = matched;
|
stats->matched_data = fields[0];
|
||||||
stats->deleted_files = deleted;
|
stats->deleted_files = fields[1];
|
||||||
stats->would_delete_count = would;
|
stats->would_delete_count = fields[2];
|
||||||
|
stats->literal_bytes = fields[3];
|
||||||
|
stats->created_reg = fields[4];
|
||||||
|
stats->created_dir = fields[5];
|
||||||
|
stats->created_link = fields[6];
|
||||||
|
stats->created_special = fields[7];
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|||||||
+16
-4
@@ -57,14 +57,26 @@ bool format_dest_state_send(int fd, const OutputDestState* state);
|
|||||||
bool format_dest_state_receive(int fd, OutputDestState* state);
|
bool format_dest_state_receive(int fd, OutputDestState* state);
|
||||||
|
|
||||||
/* End-of-transfer receiver counters reported through STATUS_STATS (protocol
|
/* End-of-transfer receiver counters reported through STATUS_STATS (protocol
|
||||||
* 2.25.0) when the wire config carries report_stats. `would_delete_count` is
|
* 2.25.0, extended in 2.28.0) when the wire config carries report_stats.
|
||||||
* the number of destination-relative paths the receiver would have deleted in a
|
* `would_delete_count` is the number of destination-relative paths the receiver
|
||||||
* -n/--dry-run --delete run; that many wire strings immediately follow the
|
* would have deleted in a -n/--dry-run --delete run; that many wire strings
|
||||||
* fixed record (sent/read by the caller). */
|
* immediately follow the fixed record (sent/read by the caller).
|
||||||
|
*
|
||||||
|
* Protocol 2.28.0 adds the receiver-observed counters the sender cannot see:
|
||||||
|
* `literal_bytes` is the file data the receiver actually stored literally
|
||||||
|
* (whole files plus the literal fragments of a delta) and the four `created_*`
|
||||||
|
* counters split the destination entries the receiver newly created by type,
|
||||||
|
* reproducing rsync's `Number of created files` breakdown and an exact
|
||||||
|
* `Literal data` for a delta run. */
|
||||||
typedef struct {
|
typedef struct {
|
||||||
unsigned long long matched_data;
|
unsigned long long matched_data;
|
||||||
unsigned long long deleted_files;
|
unsigned long long deleted_files;
|
||||||
unsigned long long would_delete_count;
|
unsigned long long would_delete_count;
|
||||||
|
unsigned long long literal_bytes;
|
||||||
|
unsigned long long created_reg;
|
||||||
|
unsigned long long created_dir;
|
||||||
|
unsigned long long created_link;
|
||||||
|
unsigned long long created_special;
|
||||||
} ReceiverStats;
|
} ReceiverStats;
|
||||||
|
|
||||||
/* Fixed-width STATUS_STATS counter record. The status frame and the optional
|
/* Fixed-width STATUS_STATS counter record. The status frame and the optional
|
||||||
|
|||||||
@@ -33,6 +33,11 @@ typedef enum {
|
|||||||
LOG_INFO_FLIST = 1u << 7,
|
LOG_INFO_FLIST = 1u << 7,
|
||||||
LOG_INFO_NONREG = 1u << 8,
|
LOG_INFO_NONREG = 1u << 8,
|
||||||
LOG_INFO_PROGRESS = 1u << 9,
|
LOG_INFO_PROGRESS = 1u << 9,
|
||||||
|
/* Marker for `--info=name2` and higher: also print rsync's
|
||||||
|
"NAME is uptodate" line for entries the receiver already has. It rides in
|
||||||
|
the info_level bitset (there is no separate Config field) and is never set
|
||||||
|
by --info=all (which selects level 1). */
|
||||||
|
LOG_INFO_NAME_UPTODATE = 1u << 10,
|
||||||
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS | LOG_INFO_DEL |
|
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS | LOG_INFO_DEL |
|
||||||
LOG_INFO_REMOVE | LOG_INFO_NAME | LOG_INFO_FLIST | LOG_INFO_NONREG |
|
LOG_INFO_REMOVE | LOG_INFO_NAME | LOG_INFO_FLIST | LOG_INFO_NONREG |
|
||||||
LOG_INFO_PROGRESS,
|
LOG_INFO_PROGRESS,
|
||||||
|
|||||||
@@ -191,7 +191,9 @@ enum NET_STATUS {
|
|||||||
* (--delete-delay). Payload: an int32 has_config flag (1 on the first plan
|
* (--delete-delay). Payload: an int32 has_config flag (1 on the first plan
|
||||||
* of the run, 0 afterwards); when set, the three global config sections
|
* of the run, 0 afterwards); when set, the three global config sections
|
||||||
* (protected-prefix count+paths, size-skipped count+paths, missing-args
|
* (protected-prefix count+paths, size-skipped count+paths, missing-args
|
||||||
* count+paths); then the destination-relative directory path wire string
|
* count+paths); then an int32 apply flag (1 for a real plan, 0 for a
|
||||||
|
* config-only carrier frame that must not walk a directory); then the
|
||||||
|
* destination-relative directory path wire string
|
||||||
* ("." for the receive root); then the child-directory count + names and the
|
* ("." for the receive root); then the child-directory count + names and the
|
||||||
* child-file count + names that must be kept. Appended after
|
* child-file count + names that must be kept. Appended after
|
||||||
* STATUS_DEST_INFO so no existing status is renumbered. */
|
* STATUS_DEST_INFO so no existing status is renumbered. */
|
||||||
|
|||||||
+82
-14
@@ -2,6 +2,7 @@
|
|||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include <arpa/inet.h>
|
#include <arpa/inet.h>
|
||||||
|
#include <ctype.h>
|
||||||
#include <dirent.h>
|
#include <dirent.h>
|
||||||
#include <errno.h>
|
#include <errno.h>
|
||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
@@ -142,6 +143,47 @@ char* str_dup(const char* string) {
|
|||||||
return new_string;
|
return new_string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified) {
|
||||||
|
if (specified)
|
||||||
|
*specified = false;
|
||||||
|
if (!env_name || !resolve)
|
||||||
|
return -1;
|
||||||
|
const char* env = getenv(env_name);
|
||||||
|
if (!env)
|
||||||
|
return -1;
|
||||||
|
|
||||||
|
bool saw_nonblank = false;
|
||||||
|
const char* p = env;
|
||||||
|
while (*p) {
|
||||||
|
if (*p == '&')
|
||||||
|
break;
|
||||||
|
if (isspace((unsigned char)*p)) {
|
||||||
|
p++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
saw_nonblank = true;
|
||||||
|
char token[64];
|
||||||
|
size_t len = 0;
|
||||||
|
while (*p && *p != '&' && !isspace((unsigned char)*p)) {
|
||||||
|
if (len < sizeof(token) - 1)
|
||||||
|
token[len++] = *p;
|
||||||
|
p++;
|
||||||
|
}
|
||||||
|
token[len] = '\0';
|
||||||
|
if (len > 0) {
|
||||||
|
int id = resolve(token);
|
||||||
|
if (id >= 0) {
|
||||||
|
if (specified)
|
||||||
|
*specified = true;
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (specified)
|
||||||
|
*specified = saw_nonblank;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
#define STR_HASH_SET_MIN_CAPACITY 16
|
#define STR_HASH_SET_MIN_CAPACITY 16
|
||||||
|
|
||||||
static size_t str_hash_set_hash(const char* key, size_t len) {
|
static size_t str_hash_set_hash(const char* key, size_t len) {
|
||||||
@@ -640,7 +682,8 @@ static bool is_synced_dir(const PathIndex* dirs, const char* rel) {
|
|||||||
like any other non-directory extra (never followed). */
|
like any other non-directory extra (never followed). */
|
||||||
static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
||||||
const PathIndex* dirs, DeleteBudget* budget,
|
const PathIndex* dirs, DeleteBudget* budget,
|
||||||
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
|
const FilterRuleList* protect_rules, bool parent_deletable,
|
||||||
bool* all_removed, DeletePathObserver observer,
|
bool* all_removed, DeletePathObserver observer,
|
||||||
void* observer_context) {
|
void* observer_context) {
|
||||||
/* openat(dirfd, ".") opens an independent file description: a dup() would
|
/* openat(dirfd, ".") opens an independent file description: a dup() would
|
||||||
@@ -687,12 +730,23 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
|
|||||||
free(child_rel);
|
free(child_rel);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (S_ISDIR(st.st_mode)) {
|
bool is_dir = S_ISDIR(st.st_mode);
|
||||||
|
if (protect_rules && filter_rules_apply_side(protect_rules, child_rel, entry->d_name, is_dir,
|
||||||
|
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
|
||||||
|
/* A first-match protect rule shields the extra; for a directory the whole
|
||||||
|
subtree is shielded (rsync prunes an excluded directory), so do not
|
||||||
|
descend. */
|
||||||
|
local_survives = true;
|
||||||
|
free(child_rel);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (is_dir) {
|
||||||
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
bool child_all_removed = false;
|
bool child_all_removed = false;
|
||||||
if (childfd >= 0) {
|
if (childfd >= 0) {
|
||||||
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count, deletable,
|
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count,
|
||||||
&child_all_removed, observer, observer_context))
|
protect_rules, deletable, &child_all_removed, observer,
|
||||||
|
observer_context))
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
close(childfd);
|
close(childfd);
|
||||||
} else if (errno != ENOENT) {
|
} else if (errno != ENOENT) {
|
||||||
@@ -760,7 +814,8 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
|
|||||||
reportable children (depth-first), matching the delete pass's ordering. */
|
reportable children (depth-first), matching the delete pass's ordering. */
|
||||||
static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
||||||
const PathIndex* dirs, ArrayList* out, size_t* recorded,
|
const PathIndex* dirs, ArrayList* out, size_t* recorded,
|
||||||
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
|
const FilterRuleList* protect_rules, bool parent_deletable,
|
||||||
bool* all_removed) {
|
bool* all_removed) {
|
||||||
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
if (scanfd < 0)
|
if (scanfd < 0)
|
||||||
@@ -794,12 +849,21 @@ static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* kee
|
|||||||
free(child_rel);
|
free(child_rel);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (S_ISDIR(st.st_mode)) {
|
bool is_dir = S_ISDIR(st.st_mode);
|
||||||
|
if (protect_rules && filter_rules_apply_side(protect_rules, child_rel, entry->d_name, is_dir,
|
||||||
|
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
|
||||||
|
/* Mirror the delete walk: a protected entry is never reported as a
|
||||||
|
would-delete and a protected directory's subtree is not enumerated. */
|
||||||
|
local_survives = true;
|
||||||
|
free(child_rel);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (is_dir) {
|
||||||
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
bool child_all_removed = false;
|
bool child_all_removed = false;
|
||||||
if (childfd >= 0) {
|
if (childfd >= 0) {
|
||||||
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
|
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
|
||||||
deletable, &child_all_removed))
|
protect_rules, deletable, &child_all_removed))
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
close(childfd);
|
close(childfd);
|
||||||
} else if (errno != ENOENT) {
|
} else if (errno != ENOENT) {
|
||||||
@@ -850,7 +914,7 @@ static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* kee
|
|||||||
|
|
||||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||||
ArrayList* out, size_t* count_out) {
|
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out) {
|
||||||
if (count_out)
|
if (count_out)
|
||||||
*count_out = 0;
|
*count_out = 0;
|
||||||
if (!manifest || !out)
|
if (!manifest || !out)
|
||||||
@@ -886,7 +950,7 @@ bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
|||||||
bool all_removed = false;
|
bool all_removed = false;
|
||||||
size_t recorded = 0;
|
size_t recorded = 0;
|
||||||
bool ok = list_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, out, &recorded, skips,
|
bool ok = list_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, out, &recorded, skips,
|
||||||
skip_count, false, &all_removed);
|
skip_count, protect_rules, false, &all_removed);
|
||||||
if (close(rootfd) != 0)
|
if (close(rootfd) != 0)
|
||||||
ok = false;
|
ok = false;
|
||||||
path_index_free(&keep);
|
path_index_free(&keep);
|
||||||
@@ -900,6 +964,7 @@ bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
|||||||
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
|
const FilterRuleList* protect_rules,
|
||||||
size_t* deleted_out, size_t* skipped_out,
|
size_t* deleted_out, size_t* skipped_out,
|
||||||
DeletePathObserver observer,
|
DeletePathObserver observer,
|
||||||
void* observer_context) {
|
void* observer_context) {
|
||||||
@@ -942,8 +1007,9 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
}
|
}
|
||||||
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
|
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
|
||||||
bool all_removed = false;
|
bool all_removed = false;
|
||||||
bool ok = delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips,
|
bool ok =
|
||||||
skip_count, false, &all_removed, observer, observer_context);
|
delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips, skip_count,
|
||||||
|
protect_rules, false, &all_removed, observer, observer_context);
|
||||||
if (close(rootfd) != 0)
|
if (close(rootfd) != 0)
|
||||||
ok = false;
|
ok = false;
|
||||||
path_index_free(&keep);
|
path_index_free(&keep);
|
||||||
@@ -961,13 +1027,15 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
size_t* deleted_out, size_t* skipped_out) {
|
const FilterRuleList* protect_rules, size_t* deleted_out,
|
||||||
|
size_t* skipped_out) {
|
||||||
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
|
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
|
||||||
skip_count, deleted_out, skipped_out, NULL, NULL);
|
skip_count, protect_rules, deleted_out, skipped_out, NULL,
|
||||||
|
NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
|
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
|
||||||
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL) ==
|
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL, NULL) ==
|
||||||
DELETE_WALK_OK;
|
DELETE_WALK_OK;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+21
-3
@@ -2,6 +2,7 @@
|
|||||||
#define UTILS_H
|
#define UTILS_H
|
||||||
|
|
||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
|
#include "filter.h"
|
||||||
#include <stddef.h>
|
#include <stddef.h>
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
@@ -81,6 +82,17 @@ bool path_index_has_descendant(const PathIndex* index, const char* path);
|
|||||||
|
|
||||||
char* str_dup(const char* string);
|
char* str_dup(const char* string);
|
||||||
char* output_escape(const char* string, bool eight_bit_output);
|
char* output_escape(const char* string, bool eight_bit_output);
|
||||||
|
|
||||||
|
/* Resolve the first supported name from a rsync algorithm-preference
|
||||||
|
* environment variable (RSYNC_COMPRESS_LIST / RSYNC_CHECKSUM_LIST). `resolve`
|
||||||
|
* maps a case-insensitive name to an algorithm id (>= 0) or -1 for an unknown
|
||||||
|
* name. rsync's syntax is a whitespace-separated list (comma/colon are NOT
|
||||||
|
* separators); the client-side half ends at '&'. Unknown entries are skipped
|
||||||
|
* and the first resolvable one wins. *specified is set true when the variable
|
||||||
|
* holds at least one non-blank character. Returns the first resolvable id, or
|
||||||
|
* -1 when the variable is unset/blank or names no supported algorithm. */
|
||||||
|
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified);
|
||||||
|
|
||||||
/* Upper bound on one line/token read from a local list file (--files-from,
|
/* Upper bound on one line/token read from a local list file (--files-from,
|
||||||
* --exclude-from/--include-from, .rsync-filter). Mirrors MAX_STRING_SIZE and
|
* --exclude-from/--include-from, .rsync-filter). Mirrors MAX_STRING_SIZE and
|
||||||
* stops a hostile multi-gigabyte line from forcing unbounded allocation. */
|
* stops a hostile multi-gigabyte line from forcing unbounded allocation. */
|
||||||
@@ -137,7 +149,8 @@ bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSki
|
|||||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
size_t* deleted_out, size_t* skipped_out);
|
const FilterRuleList* protect_rules, size_t* deleted_out,
|
||||||
|
size_t* skipped_out);
|
||||||
|
|
||||||
/* Optional per-deletion observer: called for each destination-relative path
|
/* Optional per-deletion observer: called for each destination-relative path
|
||||||
actually removed (a file, symlink, or directory), in removal order, so the
|
actually removed (a file, symlink, or directory), in removal order, so the
|
||||||
@@ -145,10 +158,15 @@ DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* m
|
|||||||
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
|
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
|
||||||
|
|
||||||
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
|
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
|
||||||
observer; the observer is invoked only for entries truly removed. */
|
* observer; the observer is invoked only for entries truly removed. When
|
||||||
|
* `protect_rules` is non-NULL its receiver-side verdict is evaluated for every
|
||||||
|
* candidate extra: a first-match PROTECT leaves the entry (and, for a
|
||||||
|
* directory, its whole subtree) in place, while RISK/NONE fall through to the
|
||||||
|
* ordinary skip-prefix/keep-set logic. */
|
||||||
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
|
const FilterRuleList* protect_rules,
|
||||||
size_t* deleted_out, size_t* skipped_out,
|
size_t* deleted_out, size_t* skipped_out,
|
||||||
DeletePathObserver observer,
|
DeletePathObserver observer,
|
||||||
void* observer_context);
|
void* observer_context);
|
||||||
@@ -159,7 +177,7 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
strings appended to `out` and receives their count in *count_out. */
|
strings appended to `out` and receives their count in *count_out. */
|
||||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||||
ArrayList* out, size_t* count_out);
|
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out);
|
||||||
bool delete_extras(const char* dest_root, const ArrayList* manifest);
|
bool delete_extras(const char* dest_root, const ArrayList* manifest);
|
||||||
/* Open the existing destination directory at `dest_root`, confined to the
|
/* Open the existing destination directory at `dest_root`, confined to the
|
||||||
authorized root with an O_NOFOLLOW component walk (the same confinement the
|
authorized root with an O_NOFOLLOW component walk (the same confinement the
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
OLDDEST
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
NEWCONTENT
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
NEWCONTENT
|
||||||
@@ -119,6 +119,13 @@ static void build_canonical_frame(void) {
|
|||||||
cfg->usermap[0].to = MAP_TO;
|
cfg->usermap[0].to = MAP_TO;
|
||||||
cfg->usermap[0].to_name = NULL;
|
cfg->usermap[0].to_name = NULL;
|
||||||
}
|
}
|
||||||
|
/* Force a non-empty receiver delete-protection block so the fuzzer mutates
|
||||||
|
* its rule count, action/sides codes and pattern strings. */
|
||||||
|
cfg->filters = array_list_create(free);
|
||||||
|
if (cfg->filters) {
|
||||||
|
array_list_add(cfg->filters, str_dup("P *.log"));
|
||||||
|
array_list_add(cfg->filters, str_dup("+r keep/**"));
|
||||||
|
}
|
||||||
if (!cfg->send_directory || !cfg->receive_root_directory || !cfg->usermap) {
|
if (!cfg->send_directory || !cfg->receive_root_directory || !cfg->usermap) {
|
||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -54,12 +54,16 @@ STDOUT_NONE = None
|
|||||||
STDOUT_ITEMIZE = "itemize"
|
STDOUT_ITEMIZE = "itemize"
|
||||||
STDOUT_OUTFMT = "outfmt"
|
STDOUT_OUTFMT = "outfmt"
|
||||||
STDOUT_STATS = "stats"
|
STDOUT_STATS = "stats"
|
||||||
|
STDOUT_PROGRESS = "progress"
|
||||||
|
|
||||||
# rsync --stats lines that are protocol-independent and must match exactly.
|
# rsync --stats lines that are protocol-independent and must match exactly.
|
||||||
# Deliberately excluded: the per-type "Number of files"/"Number of created
|
# `Number of files` and `Number of created files` carry rsync's per-type
|
||||||
# files" breakdown and Total bytes sent/received (documented residual, see the
|
# breakdown; protocol 2.28.0 reports the receiver-created split over
|
||||||
# `--stats` row in RSYNC_COMPAT.md).
|
# STATUS_STATS. Deliberately excluded: Total bytes sent/received (protocol
|
||||||
|
# framing differs, see the `--stats` row in RSYNC_COMPAT.md).
|
||||||
STATS_KEYS = (
|
STATS_KEYS = (
|
||||||
|
"Number of files",
|
||||||
|
"Number of created files",
|
||||||
"Number of deleted files",
|
"Number of deleted files",
|
||||||
"Number of regular files transferred",
|
"Number of regular files transferred",
|
||||||
"Total file size",
|
"Total file size",
|
||||||
@@ -70,6 +74,8 @@ STATS_KEYS = (
|
|||||||
)
|
)
|
||||||
|
|
||||||
_ITEMIZE_RE = re.compile(r"^(<|>|c|h|\.|\*)[fdLDS][.+\-][.+\-][.+\-][.+\-]")
|
_ITEMIZE_RE = re.compile(r"^(<|>|c|h|\.|\*)[fdLDS][.+\-][.+\-][.+\-][.+\-]")
|
||||||
|
_PROGRESS_TOTAL_RE = re.compile(r"to-chk=\d+/(\d+)")
|
||||||
|
_PROGRESS_XFR_RE = re.compile(r"xfr#(\d+)")
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -181,6 +187,24 @@ def corpus_empty_dir(root: str) -> None:
|
|||||||
_write(os.path.join(root, "nonempty", "f.txt"), b"f\n")
|
_write(os.path.join(root, "nonempty", "f.txt"), b"f\n")
|
||||||
|
|
||||||
|
|
||||||
|
def corpus_multidir(root: str) -> None:
|
||||||
|
"""Multi-directory tree for the --progress file-list naming/denominator.
|
||||||
|
|
||||||
|
Nested files, a directory-only branch, an empty directory and a symlink
|
||||||
|
exercise every file-list entry type rsync counts in `to-chk` but FastSync's
|
||||||
|
streaming scanner never emits as a transfer entry.
|
||||||
|
"""
|
||||||
|
clean_dir(root)
|
||||||
|
_write(os.path.join(root, "a.txt"), b"alpha\n")
|
||||||
|
_write(os.path.join(root, "b.txt"), b"bravo\n")
|
||||||
|
_write(os.path.join(root, "sub1", "c.txt"), b"charlie\n")
|
||||||
|
_write(os.path.join(root, "sub1", "deep", "d.txt"), b"delta\n")
|
||||||
|
_write(os.path.join(root, "sub2", "e.txt"), b"echo\n")
|
||||||
|
os.symlink("a.txt", os.path.join(root, "link1"))
|
||||||
|
os.makedirs(os.path.join(root, "emptydir"), exist_ok=True)
|
||||||
|
os.utime(os.path.join(root, "emptydir"), (_SRC_MTIME, _SRC_MTIME))
|
||||||
|
|
||||||
|
|
||||||
def corpus_relative(root: str) -> None:
|
def corpus_relative(root: str) -> None:
|
||||||
"""Tree for the -R/--files-from cases."""
|
"""Tree for the -R/--files-from cases."""
|
||||||
clean_dir(root)
|
clean_dir(root)
|
||||||
@@ -205,6 +229,18 @@ def corpus_iconv(root: str) -> None:
|
|||||||
os.utime(full, (_SRC_MTIME, _SRC_MTIME))
|
os.utime(full, (_SRC_MTIME, _SRC_MTIME))
|
||||||
|
|
||||||
|
|
||||||
|
# Payload for the --fuzzy basis corpus: large enough for the delta engine's
|
||||||
|
# 16 KiB minimum and with repeated content so a coinciding basis yields a
|
||||||
|
# non-zero (and identical) Matched data count in both tools.
|
||||||
|
FUZZY_PAYLOAD = (b"the quick brown fox jumps over the lazy dog\n" * 2000)[:65536]
|
||||||
|
|
||||||
|
|
||||||
|
def corpus_fuzzy(root: str) -> None:
|
||||||
|
"""A named regular file; the `fuzzy` seed adds the similar-suffix sibling."""
|
||||||
|
clean_dir(root)
|
||||||
|
_write(os.path.join(root, "report_v2.txt"), FUZZY_PAYLOAD)
|
||||||
|
|
||||||
|
|
||||||
CORPORA: Dict[str, Callable[[str], None]] = {
|
CORPORA: Dict[str, Callable[[str], None]] = {
|
||||||
"basic": corpus_basic,
|
"basic": corpus_basic,
|
||||||
"unicode": corpus_unicode,
|
"unicode": corpus_unicode,
|
||||||
@@ -213,8 +249,10 @@ CORPORA: Dict[str, Callable[[str], None]] = {
|
|||||||
"sparse": corpus_sparse,
|
"sparse": corpus_sparse,
|
||||||
"filters": corpus_filters,
|
"filters": corpus_filters,
|
||||||
"empty_dir": corpus_empty_dir,
|
"empty_dir": corpus_empty_dir,
|
||||||
|
"multidir": corpus_multidir,
|
||||||
"relative": corpus_relative,
|
"relative": corpus_relative,
|
||||||
"iconv": corpus_iconv,
|
"iconv": corpus_iconv,
|
||||||
|
"fuzzy": corpus_fuzzy,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -379,6 +417,37 @@ def normalize_stdout(text: str, mode: Optional[str]) -> object:
|
|||||||
if line.startswith(key + ":"):
|
if line.startswith(key + ":"):
|
||||||
found[key] = _parse_bytes(line.split(":", 1)[1])
|
found[key] = _parse_bytes(line.split(":", 1)[1])
|
||||||
return found
|
return found
|
||||||
|
if mode == STDOUT_PROGRESS:
|
||||||
|
# rsync prints the file-list entries in sorted depth-first order while
|
||||||
|
# FastSync's streaming scan emits them in readdir/BFS order; only the
|
||||||
|
# entry set and deterministic fields are compared. The transfer-root
|
||||||
|
# `./` line's trigger condition is a separate documented residual, and
|
||||||
|
# the per-frame rate/elapsed/xfr#/to-chk numerator are wall-clock- or
|
||||||
|
# order-dependent, so only the `to-chk` denominator and the name set are
|
||||||
|
# asserted.
|
||||||
|
names = []
|
||||||
|
totals = set()
|
||||||
|
max_xfr = 0
|
||||||
|
for line in (text or "").splitlines():
|
||||||
|
line = line.rstrip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
if "%" in line:
|
||||||
|
m = _PROGRESS_TOTAL_RE.search(line)
|
||||||
|
if m:
|
||||||
|
totals.add(int(m.group(1)))
|
||||||
|
mx = _PROGRESS_XFR_RE.search(line)
|
||||||
|
if mx:
|
||||||
|
max_xfr = max(max_xfr, int(mx.group(1)))
|
||||||
|
continue
|
||||||
|
if line == "sending incremental file list":
|
||||||
|
continue
|
||||||
|
if line.startswith("created directory "):
|
||||||
|
continue
|
||||||
|
if line == "./":
|
||||||
|
continue
|
||||||
|
names.append(line)
|
||||||
|
return {"names": sorted(names), "total": sorted(totals), "xfr": max_xfr}
|
||||||
# raw
|
# raw
|
||||||
return sorted(l.rstrip() for l in (text or "").splitlines() if l.strip())
|
return sorted(l.rstrip() for l in (text or "").splitlines() if l.strip())
|
||||||
|
|
||||||
@@ -390,6 +459,8 @@ def stdout_diff(rsync_out: str, fs_out: str, mode: Optional[str]) -> List[str]:
|
|||||||
return []
|
return []
|
||||||
if mode == STDOUT_STATS:
|
if mode == STDOUT_STATS:
|
||||||
return [f"stats rsync={r}", f"stats fastsync={f}"]
|
return [f"stats rsync={r}", f"stats fastsync={f}"]
|
||||||
|
if mode == STDOUT_PROGRESS:
|
||||||
|
return [f"progress rsync={r}", f"progress fastsync={f}"]
|
||||||
return list(difflib.unified_diff(
|
return list(difflib.unified_diff(
|
||||||
[str(x) for x in r], [str(x) for x in f],
|
[str(x) for x in r], [str(x) for x in f],
|
||||||
fromfile="rsync", tofile="fastsync", lineterm=""))
|
fromfile="rsync", tofile="fastsync", lineterm=""))
|
||||||
@@ -443,6 +514,11 @@ def run_differential( # noqa: PLR0913 (explicit scenario parameters)
|
|||||||
rroot, froot = os.path.join(rdst, rel), get_dest_received_dir(fdst, src)
|
rroot, froot = os.path.join(rdst, rel), get_dest_received_dir(fdst, src)
|
||||||
else:
|
else:
|
||||||
rroot, froot = rdst, get_dest_received_dir(fdst, src)
|
rroot, froot = rdst, get_dest_received_dir(fdst, src)
|
||||||
|
# rsync's destination root always exists (clean_dir created it). FastSync's
|
||||||
|
# logical transfer root is the mirror path below the destination argument,
|
||||||
|
# so pre-create it too: `Number of created files` counts the root only when
|
||||||
|
# it is genuinely absent, and the two tools must start from the same state.
|
||||||
|
os.makedirs(froot, exist_ok=True)
|
||||||
if seed:
|
if seed:
|
||||||
seed(src, rroot, froot)
|
seed(src, rroot, froot)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,213 @@
|
|||||||
|
"""Differential tests for the client CLI's codec defaults and env lists.
|
||||||
|
|
||||||
|
Track 3a of the rsync-parity plan pins two rsync 3.4.1 behaviors that are
|
||||||
|
resolved entirely on the client:
|
||||||
|
|
||||||
|
* the per-codec default ``--compress-level`` (zstd 3, zlib/zlibx 6, lz4
|
||||||
|
ignored) applied when the user omits ``--compress-level``/``--zl``, with an
|
||||||
|
explicit level clamped to the codec's range; and
|
||||||
|
* the ``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` preference lists that
|
||||||
|
rsync's ``auto`` consults before its compiled-in order (whitespace-separated,
|
||||||
|
unknown names skipped, first supported wins, all-unknown is exit 4).
|
||||||
|
|
||||||
|
The rsync side is observed through ``--debug=NSTR1``; FastSync publishes its
|
||||||
|
resolved codec/level through ``--debug=util``. The checksum side is confirmed
|
||||||
|
byte-for-byte through ``--out-format %C``. The rsync-based tests skip cleanly
|
||||||
|
when rsync is not installed.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(__file__))
|
||||||
|
from common import (
|
||||||
|
TEST_DATA_DIR,
|
||||||
|
run_client,
|
||||||
|
clean_dir,
|
||||||
|
get_dest_received_dir,
|
||||||
|
)
|
||||||
|
|
||||||
|
RSYNC = shutil.which("rsync")
|
||||||
|
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
|
||||||
|
|
||||||
|
CODEC_ROOT = os.path.join(TEST_DATA_DIR, "cli_differential")
|
||||||
|
|
||||||
|
_COMPRESS_RE = re.compile(r"compress(?:ion)?: (\w+) \(level (-?\d+)\)")
|
||||||
|
|
||||||
|
|
||||||
|
def _rsync(args):
|
||||||
|
env = dict(os.environ, LC_ALL="C")
|
||||||
|
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120)
|
||||||
|
|
||||||
|
|
||||||
|
def _scratch(tag):
|
||||||
|
path = os.path.join(CODEC_ROOT, tag)
|
||||||
|
clean_dir(path)
|
||||||
|
os.makedirs(path, exist_ok=True)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _make_corpus(root):
|
||||||
|
clean_dir(root)
|
||||||
|
os.makedirs(root, exist_ok=True)
|
||||||
|
with open(os.path.join(root, "big.bin"), "wb") as fh:
|
||||||
|
fh.write(b"FastSync codec payload " * 4096)
|
||||||
|
with open(os.path.join(root, "small.txt"), "wb") as fh:
|
||||||
|
fh.write(b"hello codec world\n" * 32)
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
def _rsync_compress_level(choice, level):
|
||||||
|
src = _make_corpus(_scratch(f"lvl_src_{choice}_{level}"))
|
||||||
|
dst = _scratch(f"lvl_rsync_{choice}_{level}")
|
||||||
|
args = ["-a", "-z", f"--zc={choice}"]
|
||||||
|
if level is not None:
|
||||||
|
args.append(f"--zl={level}")
|
||||||
|
args += ["--debug=NSTR1", src + "/", dst + "/"]
|
||||||
|
result = _rsync(args)
|
||||||
|
assert result.returncode == 0, result.stderr
|
||||||
|
match = _COMPRESS_RE.search(result.stdout + result.stderr)
|
||||||
|
assert match, (result.stdout, result.stderr)
|
||||||
|
return match.group(1), int(match.group(2))
|
||||||
|
|
||||||
|
|
||||||
|
def _fastsync_compress_level(choice, level, shared_server):
|
||||||
|
src = _make_corpus(_scratch(f"lvl_src_fs_{choice}_{level}"))
|
||||||
|
dst = _scratch(f"lvl_fs_{choice}_{level}")
|
||||||
|
args = ["-a", "-z", f"--zc={choice}"]
|
||||||
|
if level is not None:
|
||||||
|
args.append(f"--zl={level}")
|
||||||
|
args += ["-v", "--debug=util"]
|
||||||
|
result, _ = run_client(src, dst, flags=args, port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
match = _COMPRESS_RE.search(result.stdout)
|
||||||
|
assert match, result.stdout[:500]
|
||||||
|
return match.group(1), int(match.group(2))
|
||||||
|
|
||||||
|
|
||||||
|
class TestPerCodecCompressionLevelDefaults:
|
||||||
|
"""``--compress-level`` defaults and clamping match rsync per codec."""
|
||||||
|
|
||||||
|
# FastSync uses a positive lz4 placeholder because its "level > 0" gate
|
||||||
|
# enables compression; lz4_compress ignores the value, so rsync's level 0
|
||||||
|
# and FastSync's level 1 produce the same bytes.
|
||||||
|
CASES = [
|
||||||
|
("zstd", None, 3),
|
||||||
|
("zlib", None, 6),
|
||||||
|
("zlibx", None, 6),
|
||||||
|
("lz4", None, 1),
|
||||||
|
("zstd", 10, 10),
|
||||||
|
("zlib", 15, 9),
|
||||||
|
("zlib", 3, 3),
|
||||||
|
("lz4", 15, 1),
|
||||||
|
]
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
@pytest.mark.parametrize("choice,level,fs_level", CASES)
|
||||||
|
def test_level_matches_rsync(self, choice, level, fs_level, shared_server):
|
||||||
|
rsync_algo, rsync_level = _rsync_compress_level(choice, level)
|
||||||
|
fs_algo, fs_level_actual = _fastsync_compress_level(choice, level, shared_server)
|
||||||
|
assert rsync_algo == choice
|
||||||
|
assert fs_algo == choice
|
||||||
|
if choice == "lz4":
|
||||||
|
assert rsync_level == 0 and fs_level_actual > 0
|
||||||
|
else:
|
||||||
|
assert rsync_level == fs_level
|
||||||
|
assert fs_level_actual == fs_level
|
||||||
|
|
||||||
|
|
||||||
|
class TestEnvPreferenceLists:
|
||||||
|
"""``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` drive auto like rsync."""
|
||||||
|
|
||||||
|
# (env value, expected codec, rsync level, FastSync level)
|
||||||
|
COMPRESS_CASES = [
|
||||||
|
("zlib lz4", "zlib", 6, 6),
|
||||||
|
("lz4 zstd", "lz4", 0, 1),
|
||||||
|
("bogus zstd zlib", "zstd", 3, 3),
|
||||||
|
(" ", "zstd", 3, 3),
|
||||||
|
]
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
@pytest.mark.parametrize("env,algo,rsync_level,fs_level", COMPRESS_CASES)
|
||||||
|
def test_compress_list_matches_rsync(self, env, algo, rsync_level, fs_level, shared_server,
|
||||||
|
monkeypatch):
|
||||||
|
monkeypatch.setenv("RSYNC_COMPRESS_LIST", env)
|
||||||
|
src = _make_corpus(_scratch(f"envc_src_{algo}"))
|
||||||
|
rdst = _scratch(f"envc_rsync_{algo}")
|
||||||
|
rs = _rsync(["-a", "-z", "--debug=NSTR1", src + "/", rdst + "/"])
|
||||||
|
assert rs.returncode == 0, rs.stderr
|
||||||
|
rm = _COMPRESS_RE.search(rs.stdout + rs.stderr)
|
||||||
|
assert rm, (rs.stdout, rs.stderr)
|
||||||
|
assert rm.group(1) == algo
|
||||||
|
assert int(rm.group(2)) == rsync_level
|
||||||
|
|
||||||
|
fdst = _scratch(f"envc_fs_{algo}")
|
||||||
|
result, _ = run_client(src, fdst, flags=["-a", "-z", "-v", "--debug=util"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
fm = _COMPRESS_RE.search(result.stdout)
|
||||||
|
assert fm, result.stdout[:500]
|
||||||
|
assert fm.group(1) == algo
|
||||||
|
assert int(fm.group(2)) == fs_level
|
||||||
|
received = get_dest_received_dir(fdst, src)
|
||||||
|
assert _tree_bytes(received) == _tree_bytes(src)
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
@pytest.mark.parametrize("env,algo", [("md5", "md5"), ("sha1", "sha1"), ("xxh3 md5", "xxh3")])
|
||||||
|
def test_checksum_list_matches_rsync(self, env, algo, shared_server, monkeypatch):
|
||||||
|
monkeypatch.setenv("RSYNC_CHECKSUM_LIST", env)
|
||||||
|
src = _make_corpus(_scratch(f"envcc_src_{algo}"))
|
||||||
|
rdst = _scratch(f"envcc_rsync_{algo}")
|
||||||
|
rs = _rsync(["-a", "--checksum", "--out-format=%C %n", src + "/", rdst + "/"])
|
||||||
|
assert rs.returncode == 0, rs.stderr
|
||||||
|
rs_digests = _digests(rs.stdout)
|
||||||
|
|
||||||
|
fdst = _scratch(f"envcc_fs_{algo}")
|
||||||
|
result, _ = run_client(src, fdst, flags=["-a", "--checksum", "--out-format=%C %n"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
assert _digests(result.stdout) == rs_digests
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_all_unknown_lists_fail_like_rsync(self, shared_server, monkeypatch):
|
||||||
|
src = _make_corpus(_scratch("envbad_src"))
|
||||||
|
monkeypatch.setenv("RSYNC_COMPRESS_LIST", "bogus")
|
||||||
|
rs = _rsync(["-a", "-z", src + "/", _scratch("envbad_rsync_c") + "/"])
|
||||||
|
assert rs.returncode == 4, rs.stderr
|
||||||
|
result, _ = run_client(src, _scratch("envbad_fs_c"), flags=["-a", "-z"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
|
||||||
|
|
||||||
|
monkeypatch.setenv("RSYNC_CHECKSUM_LIST", "bogus")
|
||||||
|
rs = _rsync(["-a", src + "/", _scratch("envbad_rsync_s") + "/"])
|
||||||
|
assert rs.returncode == 4, rs.stderr
|
||||||
|
result, _ = run_client(src, _scratch("envbad_fs_s"), flags=["-a"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
|
||||||
|
|
||||||
|
|
||||||
|
def _tree_bytes(root):
|
||||||
|
out = {}
|
||||||
|
for dirpath, _dirs, files in os.walk(root):
|
||||||
|
for name in files:
|
||||||
|
path = os.path.join(dirpath, name)
|
||||||
|
with open(path, "rb") as fh:
|
||||||
|
out[os.path.relpath(path, root)] = fh.read()
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _digests(output):
|
||||||
|
out = {}
|
||||||
|
for line in output.splitlines():
|
||||||
|
parts = line.split()
|
||||||
|
if len(parts) == 2 and parts[0]:
|
||||||
|
out[parts[1]] = parts[0]
|
||||||
|
return out
|
||||||
@@ -9,6 +9,7 @@ directory when the shared test server is launched), so every scratch tree lives
|
|||||||
under ``TEST_DATA_DIR`` rather than pytest's ``tmp_path``.
|
under ``TEST_DATA_DIR`` rather than pytest's ``tmp_path``.
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
|
import random
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
@@ -68,6 +69,84 @@ def _tree_bytes(root):
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
_CC_DELTA_T0 = 1_600_000_000
|
||||||
|
_CC_DELTA_T1 = 1_600_000_100
|
||||||
|
|
||||||
|
_DELTA_STATS_KEYS = (
|
||||||
|
"Number of created files",
|
||||||
|
"Number of regular files transferred",
|
||||||
|
"Total transferred file size",
|
||||||
|
"Literal data",
|
||||||
|
"Matched data",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _pin_tree(root, mtime):
|
||||||
|
for dirpath, dirnames, filenames in os.walk(root):
|
||||||
|
for name in dirnames + filenames:
|
||||||
|
path = os.path.join(dirpath, name)
|
||||||
|
if not os.path.islink(path):
|
||||||
|
os.utime(path, (mtime, mtime))
|
||||||
|
os.utime(root, (mtime, mtime))
|
||||||
|
|
||||||
|
|
||||||
|
def _make_delta_basis(src, size=512 * 1024):
|
||||||
|
"""Build a source and a matching pre-modification basis tree.
|
||||||
|
|
||||||
|
The source's ``big.bin`` is then modified in a few disjoint places and given
|
||||||
|
a newer mtime so both tools take the delta path. Returns the basis dir.
|
||||||
|
"""
|
||||||
|
clean_dir(src)
|
||||||
|
original = random.Random(20240101).randbytes(size)
|
||||||
|
with open(os.path.join(src, "big.bin"), "wb") as fh:
|
||||||
|
fh.write(original)
|
||||||
|
with open(os.path.join(src, "small.txt"), "wb") as fh:
|
||||||
|
fh.write(b"hello world\n")
|
||||||
|
_pin_tree(src, _CC_DELTA_T0)
|
||||||
|
|
||||||
|
basis = src.rstrip("/") + "_basis"
|
||||||
|
clean_dir(basis)
|
||||||
|
shutil.copy2(os.path.join(src, "big.bin"), os.path.join(basis, "big.bin"))
|
||||||
|
shutil.copy2(os.path.join(src, "small.txt"), os.path.join(basis, "small.txt"))
|
||||||
|
_pin_tree(basis, _CC_DELTA_T0)
|
||||||
|
|
||||||
|
modified = bytearray(original)
|
||||||
|
for off in (0, size // 3, 2 * size // 3, size - 64):
|
||||||
|
for i in range(32):
|
||||||
|
modified[off + i] ^= 0x5A
|
||||||
|
with open(os.path.join(src, "big.bin"), "wb") as fh:
|
||||||
|
fh.write(bytes(modified))
|
||||||
|
os.utime(os.path.join(src, "big.bin"), (_CC_DELTA_T1, _CC_DELTA_T1))
|
||||||
|
return basis
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_from_basis(basis, target):
|
||||||
|
clean_dir(target)
|
||||||
|
for name in os.listdir(basis):
|
||||||
|
shutil.copy2(os.path.join(basis, name), os.path.join(target, name))
|
||||||
|
|
||||||
|
|
||||||
|
def _delta_stats(text):
|
||||||
|
found = {}
|
||||||
|
for line in text.splitlines():
|
||||||
|
for key in _DELTA_STATS_KEYS:
|
||||||
|
if line.startswith(key + ":"):
|
||||||
|
found[key] = line.split(":", 1)[1].strip()
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def _big_bin_outfmt(text):
|
||||||
|
"""The ``(c, C)`` pair from the ``big.bin`` out-format line (`%c|%C %n`)."""
|
||||||
|
for line in text.splitlines():
|
||||||
|
stripped = line.strip()
|
||||||
|
if "|" not in stripped or not stripped.endswith("big.bin"):
|
||||||
|
continue
|
||||||
|
c_field, rest = stripped.split("|", 1)
|
||||||
|
fields = rest.split()
|
||||||
|
return c_field.strip(), (fields[0] if fields else "")
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
class TestCodecChoiceMatrix:
|
class TestCodecChoiceMatrix:
|
||||||
"""The CLI accept/reject set and exit codes must match rsync 3.4.1."""
|
"""The CLI accept/reject set and exit codes must match rsync 3.4.1."""
|
||||||
|
|
||||||
@@ -191,6 +270,76 @@ class TestCodecTransferDifferential:
|
|||||||
assert _tree_bytes(received) == _tree_bytes(rsync_dst)
|
assert _tree_bytes(received) == _tree_bytes(rsync_dst)
|
||||||
|
|
||||||
|
|
||||||
|
class TestChecksumChoiceDeltaSurface:
|
||||||
|
"""--checksum-choice does not move the delta-transfer parity surface.
|
||||||
|
|
||||||
|
FastSync's delta BLOCK strong checksum is a fixed xxHash32, so the
|
||||||
|
negotiated algorithm only selects the whole-file comparison digest (and the
|
||||||
|
``%C`` transfer digest). A pre-seeded delta transfer must therefore land
|
||||||
|
byte-identical bytes and report the same counters for every choice, while
|
||||||
|
``%C`` -- the one token that tracks the choice -- stays byte-identical to
|
||||||
|
rsync. This pins the Track-3b reclassification in RSYNC_COMPAT.md.
|
||||||
|
"""
|
||||||
|
|
||||||
|
CHOICES = ["xxh64", "xxh128", "xxh3", "md5", "md4", "sha1",
|
||||||
|
"xxh64,sha1", "sha1,xxh64"]
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_delta_surface_invariant_to_checksum_choice(self, shared_server):
|
||||||
|
src = _scratch("ccdelta_src")
|
||||||
|
basis = _make_delta_basis(src)
|
||||||
|
source_bytes = _tree_bytes(src)
|
||||||
|
|
||||||
|
rsync_c, fastsync_c, digests = {}, {}, {}
|
||||||
|
rsync_stats, fastsync_stats = {}, {}
|
||||||
|
for choice in self.CHOICES:
|
||||||
|
tag = choice.replace(",", "_")
|
||||||
|
rsync_dst = _scratch(f"ccdelta_rs_{tag}")
|
||||||
|
fs_dst = _scratch(f"ccdelta_fs_{tag}")
|
||||||
|
fs_root = get_dest_received_dir(fs_dst, src)
|
||||||
|
_seed_from_basis(basis, rsync_dst)
|
||||||
|
_seed_from_basis(basis, fs_root)
|
||||||
|
|
||||||
|
# Pin the block size on both ends so the literal/matched split is
|
||||||
|
# comparable (rsync's adaptive default would otherwise differ from
|
||||||
|
# FastSync's 8192-byte default).
|
||||||
|
rsync_result = _rsync(["-a", "--no-whole-file", "-B8192", "--stats",
|
||||||
|
"--out-format=%c|%C %n", f"--cc={choice}",
|
||||||
|
src + "/", rsync_dst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
result, _ = run_client(
|
||||||
|
src, fs_dst,
|
||||||
|
flags=["-a", "--incremental", "--delta", "-B8192", "--stats",
|
||||||
|
"--out-format=%c|%C %n", f"--cc={choice}"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
|
||||||
|
assert _tree_bytes(rsync_dst) == source_bytes, choice
|
||||||
|
assert _tree_bytes(fs_root) == source_bytes, choice
|
||||||
|
assert _delta_stats(rsync_result.stdout) == _delta_stats(result.stdout), choice
|
||||||
|
|
||||||
|
rs_c, rs_C = _big_bin_outfmt(rsync_result.stdout)
|
||||||
|
fs_c, fs_C = _big_bin_outfmt(result.stdout)
|
||||||
|
assert rs_C == fs_C, f"{choice}: %C rsync={rs_C!r} fastsync={fs_C!r}"
|
||||||
|
rsync_c[choice] = rs_c
|
||||||
|
fastsync_c[choice] = fs_c
|
||||||
|
digests[choice] = fs_C
|
||||||
|
rsync_stats[choice] = _delta_stats(rsync_result.stdout)
|
||||||
|
fastsync_stats[choice] = _delta_stats(result.stdout)
|
||||||
|
|
||||||
|
# The choice is only observable in %C, and it is effective (the digests
|
||||||
|
# are not all the same algorithm's output).
|
||||||
|
assert len(set(digests.values())) > 1, digests
|
||||||
|
# The compared --stats counters are invariant across choices in each tool
|
||||||
|
# (and were asserted equal cross-tool inside the loop).
|
||||||
|
assert len({tuple(sorted(s.items())) for s in rsync_stats.values()}) == 1, rsync_stats
|
||||||
|
assert len({tuple(sorted(s.items())) for s in fastsync_stats.values()}) == 1, fastsync_stats
|
||||||
|
# The block-checksum token (%c) is invariant across choices in each tool.
|
||||||
|
assert len(set(rsync_c.values())) == 1, rsync_c
|
||||||
|
assert len(set(fastsync_c.values())) == 1, fastsync_c
|
||||||
|
|
||||||
|
|
||||||
class TestCodecNegotiationFallback:
|
class TestCodecNegotiationFallback:
|
||||||
"""FastSync's auto negotiation and deterministic fallback order."""
|
"""FastSync's auto negotiation and deterministic fallback order."""
|
||||||
|
|
||||||
|
|||||||
@@ -1,19 +1,16 @@
|
|||||||
"""Differential coverage for ``--delete-delay`` + ``--max-delete`` with a
|
"""Differential coverage for ``--delete-delay`` + ``--max-delete`` with a
|
||||||
refilled deferred directory.
|
refilled deferred directory.
|
||||||
|
|
||||||
FastSync snapshots a directory's extras at plan time (``defer_add``) and charges
|
FastSync snapshots a directory's extras at plan time (``defer_add``) but charges
|
||||||
``--max-delete`` then, and its deferred commit only removes the snapshot path, so
|
``--max-delete`` only when a path is actually removed, and its deferred commit
|
||||||
a directory refilled before the commit survives ``ENOTEMPTY``. rsync computes
|
re-scans a queued directory and removes content created after the plan -- the
|
||||||
the deferred deletions during the transfer, charges ``--max-delete`` on actual
|
same rules as rsync. These tests run both tools on the same fixture and assert
|
||||||
removals, and recursively removes a queued directory -- so content created after
|
both sides remove the late content (recursively) and bound the deletion with
|
||||||
the plan inside an extra directory is removed too.
|
``--max-delete`` identically.
|
||||||
|
|
||||||
These tests run both tools on the same fixture and pin the shared budget bound
|
They are not part of the fast PR gate because the rsync side needs a wide
|
||||||
(the later extra survives, both exit 25) plus the documented residual (the
|
real-time injection window (a throttled transfer), while the FastSync side uses
|
||||||
refilled directory's late content survives under FastSync, not rsync). They are
|
the existing byte-deterministic slicing proxy.
|
||||||
not part of the fast PR gate because the rsync side needs a wide real-time
|
|
||||||
injection window (a throttled transfer), while the FastSync side uses the
|
|
||||||
existing byte-deterministic slicing proxy.
|
|
||||||
|
|
||||||
The refilled directory sits at the transfer ROOT, whose delete plan is always
|
The refilled directory sits at the transfer ROOT, whose delete plan is always
|
||||||
processed before any subdirectory's, so the budget is deterministically charged
|
processed before any subdirectory's, so the budget is deterministically charged
|
||||||
@@ -101,7 +98,7 @@ def _rsync(args, timeout=120):
|
|||||||
|
|
||||||
|
|
||||||
class TestDeleteDelayRefilledDirVsRsync:
|
class TestDeleteDelayRefilledDirVsRsync:
|
||||||
"""The shared budget bound and the documented recursive-removal residual."""
|
"""Both tools charge --max-delete on actual removals and recurse."""
|
||||||
|
|
||||||
def _fastsync_refilled(self, tag, max_delete=None):
|
def _fastsync_refilled(self, tag, max_delete=None):
|
||||||
"""Run FastSync with the refill injected deterministically by the proxy
|
"""Run FastSync with the refill injected deterministically by the proxy
|
||||||
@@ -125,18 +122,18 @@ class TestDeleteDelayRefilledDirVsRsync:
|
|||||||
return result, received, late
|
return result, received, late
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
def test_max_delete_budget_bound_matches_and_residual_pinned(self):
|
def test_max_delete_budget_bound_matches(self):
|
||||||
# --- FastSync: budget charged at snapshot; late content preserved ---
|
# --- FastSync: the one actual removal is the late file; dirs survive ---
|
||||||
result, received, late = self._fastsync_refilled("budget_fs", max_delete=1)
|
result, received, late = self._fastsync_refilled("budget_fs", max_delete=1)
|
||||||
assert result.returncode == 25, (result.stderr or result.stdout)[:300]
|
assert result.returncode == 25, (result.stderr or result.stdout)[:300]
|
||||||
assert _deleted_count(result.stdout) == 0, result.stdout
|
assert _deleted_count(result.stdout) == 1, result.stdout
|
||||||
assert os.path.exists(late), "FastSync removed the late content of a snapshotted dir"
|
assert not os.path.exists(late), "FastSync kept the late content of a queued dir"
|
||||||
assert os.path.isdir(os.path.join(received, "xdir"))
|
assert os.path.isdir(os.path.join(received, "xdir"))
|
||||||
assert os.path.isdir(os.path.join(received, "b", "ydir")), (
|
assert os.path.isdir(os.path.join(received, "b", "ydir")), (
|
||||||
"FastSync did not charge the plan-time budget: b/ydir was removed"
|
"FastSync did not bound the deletion with --max-delete=1"
|
||||||
)
|
)
|
||||||
|
|
||||||
# --- rsync: budget charged on actual removals; dirs removed recursively ---
|
# --- rsync: same budget rule and recursive removal ---
|
||||||
source, rsync_dst = _seed_rsync("budget_rsync")
|
source, rsync_dst = _seed_rsync("budget_rsync")
|
||||||
|
|
||||||
def inject():
|
def inject():
|
||||||
@@ -151,26 +148,23 @@ class TestDeleteDelayRefilledDirVsRsync:
|
|||||||
)
|
)
|
||||||
t.join()
|
t.join()
|
||||||
assert rsync_result.returncode == 25, rsync_result.stderr
|
assert rsync_result.returncode == 25, rsync_result.stderr
|
||||||
# rsync removes the late content (recursive deferred removal); FastSync
|
assert _deleted_count(rsync_result.stdout) == _deleted_count(result.stdout)
|
||||||
# keeps it and charges the snapshot directive instead.
|
|
||||||
assert not os.path.exists(os.path.join(rsync_dst, "xdir", "new.txt")), (
|
assert not os.path.exists(os.path.join(rsync_dst, "xdir", "new.txt")), (
|
||||||
"rsync kept late content inside a queued directory"
|
"rsync kept late content inside a queued directory"
|
||||||
)
|
)
|
||||||
# The shared observable: the later extra survives in both tools under
|
|
||||||
# --max-delete=1, and both report the capped run with exit 25.
|
|
||||||
assert os.path.isdir(os.path.join(rsync_dst, "b", "ydir")), (
|
assert os.path.isdir(os.path.join(rsync_dst, "b", "ydir")), (
|
||||||
"rsync did not bound the deletion with --max-delete=1"
|
"rsync did not bound the deletion with --max-delete=1"
|
||||||
)
|
)
|
||||||
assert os.path.isdir(os.path.join(received, "b", "ydir"))
|
assert os.path.isdir(os.path.join(received, "b", "ydir"))
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
def test_refilled_extra_dir_recursive_removal_residual(self):
|
def test_refilled_extra_dir_recursive_removal_matches(self):
|
||||||
"""Without --max-delete the residual is a plain tree difference: rsync
|
"""Without --max-delete both tools remove the refilled extra directory
|
||||||
removes the refilled extra directory (and its late content), FastSync
|
(and its late content)."""
|
||||||
leaves the snapshot path in place on ENOTEMPTY."""
|
|
||||||
result, received, late = self._fastsync_refilled("recur_fs")
|
result, received, late = self._fastsync_refilled("recur_fs")
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
assert os.path.exists(late), "FastSync removed the refilled directory's late content"
|
assert not os.path.exists(late), "FastSync kept the refilled directory's late content"
|
||||||
|
assert not os.path.isdir(os.path.join(received, "xdir"))
|
||||||
|
|
||||||
source, rsync_dst = _seed_rsync("recur_rsync")
|
source, rsync_dst = _seed_rsync("recur_rsync")
|
||||||
|
|
||||||
|
|||||||
@@ -217,7 +217,21 @@ class _SlicingProxy:
|
|||||||
|
|
||||||
|
|
||||||
class TestDeleteTimingFinalStateParity:
|
class TestDeleteTimingFinalStateParity:
|
||||||
"""On a successful transfer the per-directory timings match rsync's result."""
|
"""On a successful transfer the per-directory timings match rsync's result.
|
||||||
|
|
||||||
|
Plain ``--delete`` has no rsync-incompatible spelling: it defaults to
|
||||||
|
delete-during on both tools, so it is compared against rsync's own default.
|
||||||
|
``--delete-commit`` is FastSync-only and selects the late whole-tree commit,
|
||||||
|
which is rsync's ``--delete-after`` timing.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# (fastsync flag, rsync flag)
|
||||||
|
PAIRS = [
|
||||||
|
("--delete", "--delete"),
|
||||||
|
("--delete-during", "--delete-during"),
|
||||||
|
("--delete-delay", "--delete-delay"),
|
||||||
|
("--delete-commit", "--delete-after"),
|
||||||
|
]
|
||||||
|
|
||||||
def _run_fastsync(self, tag, timing):
|
def _run_fastsync(self, tag, timing):
|
||||||
source, dest, received = _seed_pair(tag)
|
source, dest, received = _seed_pair(tag)
|
||||||
@@ -226,12 +240,12 @@ class TestDeleteTimingFinalStateParity:
|
|||||||
result, _ = run_client(source, dest, flags=[timing], port=server.port)
|
result, _ = run_client(source, dest, flags=[timing], port=server.port)
|
||||||
return result, received
|
return result, received
|
||||||
|
|
||||||
@pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
|
@pytest.mark.parametrize("fs_timing,rs_timing", PAIRS)
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
def test_success_final_state_matches_rsync(self, timing):
|
def test_success_final_state_matches_rsync(self, fs_timing, rs_timing):
|
||||||
# Worker-safe names: xdist may run both parametrizations concurrently, so
|
# Worker-safe names: xdist may run the parametrizations concurrently, so
|
||||||
# the timing is part of every fixture path.
|
# the flags are part of every fixture path.
|
||||||
label = timing.lstrip("-")
|
label = f"{fs_timing.lstrip('-')}_vs_{rs_timing.lstrip('-')}"
|
||||||
# Build the rsync fixture from the same seed so both sides start equal.
|
# Build the rsync fixture from the same seed so both sides start equal.
|
||||||
source, dest, received = _seed_pair(f"parity_rsync_{label}")
|
source, dest, received = _seed_pair(f"parity_rsync_{label}")
|
||||||
source2 = source
|
source2 = source
|
||||||
@@ -240,17 +254,18 @@ class TestDeleteTimingFinalStateParity:
|
|||||||
# rsync mirrors src/ into dst/; seed the same extra.
|
# rsync mirrors src/ into dst/; seed the same extra.
|
||||||
_write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n")
|
_write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n")
|
||||||
|
|
||||||
rsync_result = _rsync(["-a", timing, source2 + "/", rsync_dst + "/"])
|
rsync_result = _rsync(["-a", rs_timing, source2 + "/", rsync_dst + "/"])
|
||||||
assert rsync_result.returncode == 0, rsync_result.stderr
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
rsync_tree = _tree(rsync_dst)
|
rsync_tree = _tree(rsync_dst)
|
||||||
|
|
||||||
with ServerManager() as server:
|
with ServerManager() as server:
|
||||||
server.start(extra_args=["--allow-delete"])
|
server.start(extra_args=["--allow-delete"])
|
||||||
result, _ = run_client(source, dest, flags=[timing], port=server.port)
|
result, _ = run_client(source, dest, flags=[fs_timing], port=server.port)
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
fastsync_tree = _tree(received)
|
fastsync_tree = _tree(received)
|
||||||
assert fastsync_tree == rsync_tree, (
|
assert fastsync_tree == rsync_tree, (
|
||||||
f"{timing}: fastsync tree {fastsync_tree} != rsync tree {rsync_tree}"
|
f"{fs_timing} vs rsync {rs_timing}: fastsync tree {fastsync_tree} != "
|
||||||
|
f"rsync tree {rsync_tree}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -292,7 +307,14 @@ class TestDeleteTimingTypeConflictParity:
|
|||||||
|
|
||||||
|
|
||||||
class TestDeleteTimingFailure:
|
class TestDeleteTimingFailure:
|
||||||
"""A mid-transfer failure distinguishes during from delay."""
|
"""A mid-transfer failure distinguishes the during timings from the late
|
||||||
|
commit timings.
|
||||||
|
|
||||||
|
Plain ``--delete`` must behave like ``--delete-during`` (the rsync default),
|
||||||
|
removing the extras of the directories already reached; ``--delete-commit``
|
||||||
|
must behave like ``--delete-after`` and remove nothing until the transfer
|
||||||
|
has fully succeeded.
|
||||||
|
"""
|
||||||
|
|
||||||
@pytest.mark.parametrize("mt", [False, True])
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
def test_during_removes_delay_preserves_on_failure(self, mt):
|
def test_during_removes_delay_preserves_on_failure(self, mt):
|
||||||
@@ -301,8 +323,12 @@ class TestDeleteTimingFailure:
|
|||||||
assert os.path.exists(extra)
|
assert os.path.exists(extra)
|
||||||
with ServerManager() as server:
|
with ServerManager() as server:
|
||||||
server.start(extra_args=["--allow-delete"])
|
server.start(extra_args=["--allow-delete"])
|
||||||
for timing, expect_removed in (("--delete-during", True),
|
for timing, expect_removed in (
|
||||||
("--delete-delay", False)):
|
("--delete-during", True),
|
||||||
|
("--delete", True),
|
||||||
|
("--delete-delay", False),
|
||||||
|
("--delete-commit", False),
|
||||||
|
("--delete-after", False)):
|
||||||
# Re-seed the extra before each run.
|
# Re-seed the extra before each run.
|
||||||
_write(extra, b"stale extra\n")
|
_write(extra, b"stale extra\n")
|
||||||
proxy = _SlicingProxy(server.port, forward_limit=MID_TRANSFER_BYTES, throttle=PROXY_THROTTLE)
|
proxy = _SlicingProxy(server.port, forward_limit=MID_TRANSFER_BYTES, throttle=PROXY_THROTTLE)
|
||||||
@@ -320,10 +346,10 @@ class TestDeleteTimingFailure:
|
|||||||
class TestDeleteDelayDeletedCount:
|
class TestDeleteDelayDeletedCount:
|
||||||
"""The reported deleted count must reflect entries actually removed."""
|
"""The reported deleted count must reflect entries actually removed."""
|
||||||
|
|
||||||
def test_refilled_deferred_dir_is_not_counted(self):
|
def test_refilled_deferred_dir_is_recursively_removed_and_counted(self):
|
||||||
"""A directory snapshotted into a --delete-delay plan that is refilled
|
"""A directory snapshotted into a --delete-delay plan that is refilled
|
||||||
before the commit survives ENOTEMPTY and must NOT inflate "Number of
|
before the commit is re-scanned and removed recursively (rsync parity):
|
||||||
deleted files" (regression for delete_plan.c counting at snapshot)."""
|
the late file and the directory are both counted as deleted."""
|
||||||
source = os.path.join(TEST_DATA_DIR, "ddc_src")
|
source = os.path.join(TEST_DATA_DIR, "ddc_src")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "ddc_dst")
|
dest = os.path.join(TEST_DATA_DIR, "ddc_dst")
|
||||||
clean_dir(source)
|
clean_dir(source)
|
||||||
@@ -347,12 +373,13 @@ class TestDeleteDelayDeletedCount:
|
|||||||
proxy.finish()
|
proxy.finish()
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||||
assert proxy.hook_called.is_set(), "hook never fired"
|
assert proxy.hook_called.is_set(), "hook never fired"
|
||||||
assert os.path.exists(os.path.join(extra_dir, "new.txt")), "late file vanished"
|
assert not os.path.exists(os.path.join(extra_dir, "new.txt")), "late file survived"
|
||||||
|
assert not os.path.isdir(extra_dir), "refilled extra dir survived"
|
||||||
deleted = None
|
deleted = None
|
||||||
for line in result.stdout.splitlines():
|
for line in result.stdout.splitlines():
|
||||||
if line.startswith("Number of deleted files:"):
|
if line.startswith("Number of deleted files:"):
|
||||||
deleted = int(line.split(":", 1)[1].split()[0])
|
deleted = int(line.split(":", 1)[1].split()[0])
|
||||||
assert deleted == 0, (deleted, result.stdout)
|
assert deleted == 2, (deleted, result.stdout)
|
||||||
|
|
||||||
|
|
||||||
class TestDeleteDelayMaxDeleteParity:
|
class TestDeleteDelayMaxDeleteParity:
|
||||||
@@ -448,12 +475,13 @@ class TestDeleteDelayVsAfterSnapshot:
|
|||||||
|
|
||||||
|
|
||||||
class TestDeleteAfterThreadsKeepSet:
|
class TestDeleteAfterThreadsKeepSet:
|
||||||
"""Regression: -m/--threads with the default delete-after timing (plain
|
"""Regression: -j/--threads must still transmit the delete keep-set in every
|
||||||
--delete) must still transmit the keep-set manifest and remove destination
|
timing. PipelineContextSender.delete_suppressed was left uninitialized, so a
|
||||||
extras. PipelineContextSender.delete_suppressed was left uninitialized, so a
|
garbage true silently skipped the late keep-set manifest under --threads.
|
||||||
garbage true silently skipped the manifest under --threads."""
|
Plain --delete now uses the per-directory plans, while --delete-commit /
|
||||||
|
--delete-after keep exercising the late whole-tree manifest."""
|
||||||
|
|
||||||
@pytest.mark.parametrize("delete_flag", ["--delete", "--delete-after"])
|
@pytest.mark.parametrize("delete_flag", ["--delete", "--delete-commit", "--delete-after"])
|
||||||
def test_threads_delete_after_sends_keep_set(self, delete_flag):
|
def test_threads_delete_after_sends_keep_set(self, delete_flag):
|
||||||
source, dest, received = _seed_pair("mtkeep")
|
source, dest, received = _seed_pair("mtkeep")
|
||||||
extra = os.path.join(received, "d", "old_extra")
|
extra = os.path.join(received, "d", "old_extra")
|
||||||
@@ -464,14 +492,13 @@ class TestDeleteAfterThreadsKeepSet:
|
|||||||
port=server.port)
|
port=server.port)
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
assert not os.path.exists(extra), (
|
assert not os.path.exists(extra), (
|
||||||
f"{delete_flag} --threads did not remove an extra: keep-set manifest was suppressed"
|
f"{delete_flag} --threads did not remove an extra: delete keep-set was suppressed"
|
||||||
)
|
)
|
||||||
|
|
||||||
class TestDeleteDelayMaxDeleteRefilledDir:
|
class TestDeleteDelayMaxDeleteRefilledDir:
|
||||||
"""--delete-delay charges the --max-delete budget at plan/snapshot time, so a
|
"""--delete-delay charges the --max-delete budget on ACTUAL removals: the
|
||||||
refilled snapshotted directory that survives ENOTEMPTY still spends its slot
|
refilled directory's late content is removed first (consuming the one slot),
|
||||||
and a later extra is skipped, while the reported count stays at actual
|
so the directory itself and a later extra are skipped, matching rsync.
|
||||||
removals.
|
|
||||||
|
|
||||||
The refilled directory is at the destination ROOT (its plan is always sent
|
The refilled directory is at the destination ROOT (its plan is always sent
|
||||||
first) and the skipped extra is under a separate source directory, so the
|
first) and the skipped extra is under a separate source directory, so the
|
||||||
@@ -479,7 +506,7 @@ class TestDeleteDelayMaxDeleteRefilledDir:
|
|||||||
order. The refill is injected through the byte-barrier proxy so it is
|
order. The refill is injected through the byte-barrier proxy so it is
|
||||||
causally after the plan frame."""
|
causally after the plan frame."""
|
||||||
|
|
||||||
def test_budget_charged_at_plan_time(self):
|
def test_budget_charged_on_actual_removal(self):
|
||||||
source = os.path.join(TEST_DATA_DIR, "ddmb_src")
|
source = os.path.join(TEST_DATA_DIR, "ddmb_src")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "ddmb_dst")
|
dest = os.path.join(TEST_DATA_DIR, "ddmb_dst")
|
||||||
clean_dir(source)
|
clean_dir(source)
|
||||||
@@ -504,9 +531,9 @@ class TestDeleteDelayMaxDeleteRefilledDir:
|
|||||||
proxy.finish()
|
proxy.finish()
|
||||||
assert result.returncode == 25, (result.stderr or result.stdout)[:400]
|
assert result.returncode == 25, (result.stderr or result.stdout)[:400]
|
||||||
assert proxy.hook_called.is_set(), "hook never fired"
|
assert proxy.hook_called.is_set(), "hook never fired"
|
||||||
# The refilled directory still consumes the plan-time budget, so the
|
# The late content consumes the single budget slot; the refilled
|
||||||
# later extra is skipped...
|
# directory itself and the later extra are skipped.
|
||||||
assert os.path.exists(os.path.join(refilled_dir, "new.txt")), "late file vanished"
|
assert not os.path.exists(os.path.join(refilled_dir, "new.txt")), "late file survived"
|
||||||
assert os.path.isdir(later_dir), "later extra was not skipped by the plan-time budget"
|
assert os.path.isdir(later_dir), "later extra was not skipped by the budget"
|
||||||
# ...while the reported count reflects only actual removals (none here).
|
# The one actual removal is reported.
|
||||||
assert _deleted_count(result.stdout) == 0, result.stdout
|
assert _deleted_count(result.stdout) == 1, result.stdout
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ from common import ( # noqa: E402
|
|||||||
ServerManager,
|
ServerManager,
|
||||||
TEST_DATA_DIR,
|
TEST_DATA_DIR,
|
||||||
clean_dir,
|
clean_dir,
|
||||||
|
get_dest_received_dir,
|
||||||
)
|
)
|
||||||
from parity_caveats import ASPECTS, caveat_for # noqa: E402
|
from parity_caveats import ASPECTS, caveat_for # noqa: E402
|
||||||
import parity_harness as H # noqa: E402
|
import parity_harness as H # noqa: E402
|
||||||
@@ -116,12 +117,32 @@ def seed_delete_excluded(_src, rroot, froot):
|
|||||||
_mk(os.path.join(root, "keep.txt"), b"keep\n", _OLD_MTIME)
|
_mk(os.path.join(root, "keep.txt"), b"keep\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def seed_filter_protect(_src, rroot, froot):
|
||||||
|
"""Destination-only entries, including nested ones, for the receiver-side
|
||||||
|
`protect` rule: the `.log` extras must survive --delete, the rest go."""
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "extra.log"), b"dest-only log\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "other.txt"), b"dest-only other\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "extra2.log"), b"nested dest-only log\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "other2.txt"), b"nested dest-only other\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
def seed_max_delete(_src, rroot, froot):
|
def seed_max_delete(_src, rroot, froot):
|
||||||
for root in (rroot, froot):
|
for root in (rroot, froot):
|
||||||
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
|
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
|
||||||
_mk(os.path.join(root, "extra2.txt"), b"e2\n", _OLD_MTIME)
|
_mk(os.path.join(root, "extra2.txt"), b"e2\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def fuzzy_basis_seed(_src, rroot, froot):
|
||||||
|
"""Seed a same-suffix sibling whose name is one edit from the source and
|
||||||
|
whose content matches it, with a DIFFERENT mtime so rsync's exact
|
||||||
|
size+mtime pass cannot fire: both tools must select it via the
|
||||||
|
name-distance pass. Where the two tools' basis choices coincide the
|
||||||
|
block-level results are identical when the block size is pinned."""
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "report_v1.txt"), H.FUZZY_PAYLOAD, _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
def max_delete_count_check(_src, rroot, froot, _rs, _fs):
|
def max_delete_count_check(_src, rroot, froot, _rs, _fs):
|
||||||
"""The exact survivor set is order-dependent; the count must still match."""
|
"""The exact survivor set is order-dependent; the count must still match."""
|
||||||
r = H.snapshot(rroot)
|
r = H.snapshot(rroot)
|
||||||
@@ -173,6 +194,12 @@ _CASES = [
|
|||||||
stdout=H.STDOUT_OUTFMT, ref="--out-format %n %l"),
|
stdout=H.STDOUT_OUTFMT, ref="--out-format %n %l"),
|
||||||
H.Case("out_format_i_n", "basic", ["-a", "--out-format=%i %n"],
|
H.Case("out_format_i_n", "basic", ["-a", "--out-format=%i %n"],
|
||||||
stdout=H.STDOUT_OUTFMT, ref="--out-format %i %n"),
|
stdout=H.STDOUT_OUTFMT, ref="--out-format %i %n"),
|
||||||
|
H.Case("progress", "multidir", ["-a", "--progress"], stdout=H.STDOUT_PROGRESS,
|
||||||
|
ci=True, ref="--progress multi-directory file list"),
|
||||||
|
H.Case("progress_threads", "multidir", ["-a", "--progress"],
|
||||||
|
fastsync_flags=["-a", "--progress", "--threads"],
|
||||||
|
stdout=H.STDOUT_PROGRESS, ci=True,
|
||||||
|
ref="--progress multi-directory file list (--threads)"),
|
||||||
|
|
||||||
# --- transfer modifications -------------------------------------------
|
# --- transfer modifications -------------------------------------------
|
||||||
H.Case("update", "basic", ["-a", "--update"], seed=seed_update,
|
H.Case("update", "basic", ["-a", "--update"], seed=seed_update,
|
||||||
@@ -191,6 +218,18 @@ _CASES = [
|
|||||||
H.Case("chmod", "basic", ["-a", "--chmod=Fu+rwx"], compare_modes=True,
|
H.Case("chmod", "basic", ["-a", "--chmod=Fu+rwx"], compare_modes=True,
|
||||||
ci=True, ref="--chmod"),
|
ci=True, ref="--chmod"),
|
||||||
|
|
||||||
|
# --- delta / similar-file basis (--fuzzy) -----------------------------
|
||||||
|
# Basis choices coincide here (same-suffix sibling, name distance one edit,
|
||||||
|
# content identical); with the block size pinned both tools report the same
|
||||||
|
# Matched/Literal/transferred counters. The residual (FastSync's narrower
|
||||||
|
# delta size window) is covered by TestFuzzy in test_parity_quickwins.py.
|
||||||
|
H.Case("fuzzy_basis", "fuzzy",
|
||||||
|
["-a", "--no-whole-file", "--fuzzy", "--stats", "-B8192"],
|
||||||
|
fastsync_flags=["-a", "--incremental", "--delta", "--fuzzy",
|
||||||
|
"--stats", "--delta-block=8192"],
|
||||||
|
seed=fuzzy_basis_seed, stdout=H.STDOUT_STATS, ci=True,
|
||||||
|
ref="-y/--fuzzy similar-file basis"),
|
||||||
|
|
||||||
# --- deletion ---------------------------------------------------------
|
# --- deletion ---------------------------------------------------------
|
||||||
H.Case("delete", "basic", ["-a", "--delete"], seed=seed_extras,
|
H.Case("delete", "basic", ["-a", "--delete"], seed=seed_extras,
|
||||||
server_args=DELETE, ci=True, ref="--delete"),
|
server_args=DELETE, ci=True, ref="--delete"),
|
||||||
@@ -202,13 +241,36 @@ _CASES = [
|
|||||||
server_args=DELETE, ref="--delete-delay"),
|
server_args=DELETE, ref="--delete-delay"),
|
||||||
H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras,
|
H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras,
|
||||||
server_args=DELETE, ref="--delete-after"),
|
server_args=DELETE, ref="--delete-after"),
|
||||||
|
H.Case("delete_commit", "basic", ["-a", "--delete-after"], seed=seed_extras,
|
||||||
|
fastsync_flags=["-a", "--delete-commit"], server_args=DELETE,
|
||||||
|
ref="FastSync-only --delete-commit == rsync --delete-after"),
|
||||||
H.Case("delete_excluded", "filters",
|
H.Case("delete_excluded", "filters",
|
||||||
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
|
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
|
||||||
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
|
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
|
||||||
|
H.Case("exclude_protect_dest_only", "filters",
|
||||||
|
["-a", "--delete", "--exclude=*.log"],
|
||||||
|
seed=seed_delete_excluded, server_args=DELETE, ci=True,
|
||||||
|
ref="--delete protects a destination-only excluded entry like rsync"),
|
||||||
H.Case("max_delete", "basic", ["-a", "--delete", "--max-delete=1"],
|
H.Case("max_delete", "basic", ["-a", "--delete", "--max-delete=1"],
|
||||||
seed=seed_max_delete, server_args=DELETE,
|
seed=seed_max_delete, server_args=DELETE,
|
||||||
extra_check=max_delete_count_check, compare_tree=False,
|
extra_check=max_delete_count_check, compare_tree=False,
|
||||||
ref="--max-delete"),
|
ref="--max-delete"),
|
||||||
|
H.Case("filter_protect", "filters",
|
||||||
|
["-a", "--delete", "--filter=P *.log"],
|
||||||
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="--filter P/--protect receiver-side delete protection (default during)"),
|
||||||
|
H.Case("filter_protect_during", "filters",
|
||||||
|
["-a", "--delete-during", "--filter=P *.log"],
|
||||||
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="--filter P/--protect under --delete-during"),
|
||||||
|
H.Case("filter_protect_delay", "filters",
|
||||||
|
["-a", "--delete-delay", "--filter=P *.log"],
|
||||||
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="--filter P/--protect under --delete-delay"),
|
||||||
|
H.Case("filter_protect_after", "filters",
|
||||||
|
["-a", "--delete-after", "--filter=P *.log"],
|
||||||
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="--filter P/--protect under the whole-tree --delete-after commit"),
|
||||||
|
|
||||||
# --- relative / dirs --------------------------------------------------
|
# --- relative / dirs --------------------------------------------------
|
||||||
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
|
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
|
||||||
@@ -318,7 +380,11 @@ def _result_aspects(result):
|
|||||||
_STANDALONE_REFS = {
|
_STANDALONE_REFS = {
|
||||||
"incremental_modified": "-i/--itemize-changes + incremental second run",
|
"incremental_modified": "-i/--itemize-changes + incremental second run",
|
||||||
"compare_dest": "--compare-dest",
|
"compare_dest": "--compare-dest",
|
||||||
|
"copy_dest": "--copy-dest",
|
||||||
"link_dest": "--link-dest",
|
"link_dest": "--link-dest",
|
||||||
|
"link_dest_stats": "--link-dest + --stats",
|
||||||
|
"verify_basis": "--verify-basis (FastSync-only)",
|
||||||
|
"verify_basis_default": "--verify-basis (default quick-check vs rsync)",
|
||||||
"added_and_deleted": "--delete across two runs",
|
"added_and_deleted": "--delete across two runs",
|
||||||
"added_and_deleted_seed": "--delete across two runs",
|
"added_and_deleted_seed": "--delete across two runs",
|
||||||
"one_file_system": "-x/--one-file-system",
|
"one_file_system": "-x/--one-file-system",
|
||||||
@@ -379,14 +445,17 @@ def test_compare_dest_skips_basis(parity_server_factory):
|
|||||||
fdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_fdst")
|
fdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_fdst")
|
||||||
clean_dir(src)
|
clean_dir(src)
|
||||||
_mk(os.path.join(src, "f.txt"), b"basis-content\n")
|
_mk(os.path.join(src, "f.txt"), b"basis-content\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
|
||||||
server = parity_server_factory(SUPER)
|
server = parity_server_factory(SUPER)
|
||||||
rel = os.path.abspath(src).lstrip(os.sep)
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
# rsync resolves --compare-dest relative to the destination dir; FastSync
|
# rsync resolves --compare-dest relative to the destination dir; FastSync
|
||||||
# resolves it under the receive root and appends the mirrored source path.
|
# resolves it under the receive root and appends the mirrored source path.
|
||||||
|
# Both rely on rsync's size+mtime quick-check, so the basis mtime is pinned
|
||||||
|
# to the source's to keep the match deterministic across a second boundary.
|
||||||
def seed(_src, rroot, froot):
|
def seed(_src, rroot, froot):
|
||||||
_mk(os.path.join(rroot, "basis", "f.txt"), b"basis-content\n")
|
_mk(os.path.join(rroot, "basis", "f.txt"), b"basis-content\n", _OLD_MTIME)
|
||||||
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"basis-content\n")
|
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"basis-content\n", _OLD_MTIME)
|
||||||
|
|
||||||
def extra(_src, rroot, froot, _rs, _fs):
|
def extra(_src, rroot, froot, _rs, _fs):
|
||||||
out = []
|
out = []
|
||||||
@@ -414,12 +483,13 @@ def test_link_dest_hardlinks_basis(parity_server_factory):
|
|||||||
fdst = os.path.join(TEST_DATA_DIR, "parity_linkd_fdst")
|
fdst = os.path.join(TEST_DATA_DIR, "parity_linkd_fdst")
|
||||||
clean_dir(src)
|
clean_dir(src)
|
||||||
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
|
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
|
||||||
server = parity_server_factory(SUPER)
|
server = parity_server_factory(SUPER)
|
||||||
rel = os.path.abspath(src).lstrip(os.sep)
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
def seed(_src, rroot, froot):
|
def seed(_src, rroot, froot):
|
||||||
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n")
|
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n", _OLD_MTIME)
|
||||||
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n")
|
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n", _OLD_MTIME)
|
||||||
|
|
||||||
def extra(_src, rroot, froot, _rs, _fs):
|
def extra(_src, rroot, froot, _rs, _fs):
|
||||||
r_basis = os.stat(os.path.join(rroot, "basis", "f.txt")).st_ino
|
r_basis = os.stat(os.path.join(rroot, "basis", "f.txt")).st_ino
|
||||||
@@ -442,6 +512,134 @@ def test_link_dest_hardlinks_basis(parity_server_factory):
|
|||||||
_run_and_check(case_id, result)
|
_run_and_check(case_id, result)
|
||||||
|
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@parity
|
||||||
|
def test_link_dest_stats_matches_rsync(parity_server_factory):
|
||||||
|
"""A basis hit must not be counted as created or literal data: rsync reports
|
||||||
|
zero for both, so FastSync's receiver tallies must too (regression for the
|
||||||
|
basis materialization over-report)."""
|
||||||
|
case_id = "link_dest_stats"
|
||||||
|
src = os.path.join(TEST_DATA_DIR, "parity_linkds_src")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "parity_linkds_rdst")
|
||||||
|
fdst = os.path.join(TEST_DATA_DIR, "parity_linkds_fdst")
|
||||||
|
clean_dir(src)
|
||||||
|
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
|
||||||
|
server = parity_server_factory(SUPER)
|
||||||
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
|
def seed(_src, rroot, froot):
|
||||||
|
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
result = H.run_differential(
|
||||||
|
src, rdst, fdst,
|
||||||
|
["-a", "--link-dest=basis", "--stats"],
|
||||||
|
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental", "--stats"],
|
||||||
|
server, seed=seed, ignore_paths=("basis",), stdout=H.STDOUT_STATS)
|
||||||
|
_run_and_check(case_id, result, ref="--link-dest + --stats")
|
||||||
|
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@parity
|
||||||
|
def test_copy_dest_copies_basis(parity_server_factory):
|
||||||
|
"""--copy-dest: a basis match is materialized as an independent copy with the
|
||||||
|
source's attributes, matching rsync (copy then fix attributes)."""
|
||||||
|
case_id = "copy_dest"
|
||||||
|
src = os.path.join(TEST_DATA_DIR, "parity_copyd_src")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "parity_copyd_rdst")
|
||||||
|
fdst = os.path.join(TEST_DATA_DIR, "parity_copyd_fdst")
|
||||||
|
clean_dir(src)
|
||||||
|
_mk(os.path.join(src, "f.txt"), b"copy-basis-content\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), 1_600_000_000)
|
||||||
|
os.chmod(os.path.join(src, "f.txt"), 0o755)
|
||||||
|
server = parity_server_factory(SUPER)
|
||||||
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
|
def seed(_src, rroot, froot):
|
||||||
|
# Basis content matches the source; give the basis a different mode so a
|
||||||
|
# wrong "keep basis attributes" implementation is visible.
|
||||||
|
_mk(os.path.join(rroot, "basis", "f.txt"), b"copy-basis-content\n",
|
||||||
|
1_600_000_000)
|
||||||
|
os.chmod(os.path.join(rroot, "basis", "f.txt"), 0o644)
|
||||||
|
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"copy-basis-content\n",
|
||||||
|
1_600_000_000)
|
||||||
|
os.chmod(os.path.join(fdst, "basis", rel, "f.txt"), 0o644)
|
||||||
|
|
||||||
|
def extra(_src, rroot, froot, _rs, _fs):
|
||||||
|
out = []
|
||||||
|
bases = {"rsync": os.path.join(rroot, "basis", "f.txt"),
|
||||||
|
"fastsync": os.path.join(fdst, "basis", rel, "f.txt")}
|
||||||
|
for label, root in (("rsync", rroot), ("fastsync", froot)):
|
||||||
|
target = os.path.join(root, "f.txt")
|
||||||
|
if not os.path.exists(target):
|
||||||
|
out.append(f"{label}: f.txt missing")
|
||||||
|
continue
|
||||||
|
if os.stat(target).st_ino == os.stat(bases[label]).st_ino:
|
||||||
|
out.append(f"{label}: f.txt is hard-linked, not copied")
|
||||||
|
if (os.stat(target).st_mode & 0o777) != 0o755:
|
||||||
|
out.append(f"{label}: f.txt mode "
|
||||||
|
f"{oct(os.stat(target).st_mode & 0o777)} != 0o755")
|
||||||
|
return out
|
||||||
|
|
||||||
|
result = H.run_differential(
|
||||||
|
src, rdst, fdst,
|
||||||
|
["-a", "--copy-dest=basis"],
|
||||||
|
["-a", f"--copy-dest={os.path.join(fdst, 'basis')}", "--incremental"],
|
||||||
|
server, seed=seed, ignore_paths=("basis",), extra_check=extra,
|
||||||
|
compare_modes=True)
|
||||||
|
_run_and_check(case_id, result)
|
||||||
|
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@parity
|
||||||
|
def test_verify_basis_restores_strict_content(parity_server_factory):
|
||||||
|
"""Default matches rsync's metadata quick-check; FastSync-only
|
||||||
|
`--verify-basis` restores strict content equality and transfers the source
|
||||||
|
when a same-size/different-content basis would otherwise be trusted."""
|
||||||
|
case_id = "verify_basis"
|
||||||
|
src = os.path.join(TEST_DATA_DIR, "parity_vbasis_src")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "parity_vbasis_rdst")
|
||||||
|
fdst = os.path.join(TEST_DATA_DIR, "parity_vbasis_fdst")
|
||||||
|
clean_dir(src)
|
||||||
|
_mk(os.path.join(src, "f.txt"), b"AAAA\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
|
||||||
|
server = parity_server_factory(SUPER)
|
||||||
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
|
def seed(_src, rroot, froot):
|
||||||
|
# Same size and mtime as the source, different bytes: a metadata
|
||||||
|
# quick-check trusts it; --verify-basis must not.
|
||||||
|
for root, basis_rel in ((rroot, os.path.join("basis", "f.txt")),
|
||||||
|
(fdst, os.path.join("basis", rel, "f.txt"))):
|
||||||
|
_mk(os.path.join(root, basis_rel), b"BBBB\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
# Default: both tools trust the basis (rsync's quick check), so the
|
||||||
|
# destination carries the basis bytes and the trees match.
|
||||||
|
result = H.run_differential(
|
||||||
|
src, rdst, fdst,
|
||||||
|
["-a", "--link-dest=basis"],
|
||||||
|
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental"],
|
||||||
|
server, seed=seed, ignore_paths=("basis",))
|
||||||
|
_run_and_check(case_id + "_default", result)
|
||||||
|
|
||||||
|
# --verify-basis (FastSync only): the digest mismatch rejects the basis and
|
||||||
|
# the source is transferred, so the destination is the source bytes. rsync
|
||||||
|
# has no such flag; assert the FastSync outcome directly against the source.
|
||||||
|
fdst2 = os.path.join(TEST_DATA_DIR, "parity_vbasis_fdst2")
|
||||||
|
clean_dir(fdst2)
|
||||||
|
for root, basis_rel in ((fdst2, os.path.join("basis", rel, "f.txt")),):
|
||||||
|
_mk(os.path.join(root, basis_rel), b"BBBB\n", _OLD_MTIME)
|
||||||
|
result, _ = H.run_fastsync(src, fdst2,
|
||||||
|
["-a", f"--link-dest={os.path.join(fdst2, 'basis')}",
|
||||||
|
"--incremental", "--verify-basis"], server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
target = os.path.join(get_dest_received_dir(fdst2, src), "f.txt")
|
||||||
|
with open(target, "rb") as fh:
|
||||||
|
assert fh.read() == b"AAAA\n", \
|
||||||
|
"--verify-basis must reject the same-size/different-content basis"
|
||||||
|
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
@parity
|
@parity
|
||||||
def test_added_and_deleted_between_runs(parity_server_factory):
|
def test_added_and_deleted_between_runs(parity_server_factory):
|
||||||
|
|||||||
@@ -36,7 +36,7 @@ from common import ( # noqa: E402
|
|||||||
verify_transfer,
|
verify_transfer,
|
||||||
)
|
)
|
||||||
|
|
||||||
PROTOCOL_VERSION = b"2.27.0"
|
PROTOCOL_VERSION = b"2.28.0"
|
||||||
STATUS_MANIFEST = 5
|
STATUS_MANIFEST = 5
|
||||||
STATUS_OK = 0
|
STATUS_OK = 0
|
||||||
|
|
||||||
|
|||||||
@@ -584,53 +584,55 @@ class TestRemoteDryRun:
|
|||||||
assert _snapshot_tree(received) == before, f"{flags} mutated the destination"
|
assert _snapshot_tree(received) == before, f"{flags} mutated the destination"
|
||||||
|
|
||||||
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
|
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
|
||||||
def test_dry_run_delete_lines_over_report_residual(self):
|
def test_dry_run_delete_lines_match_rsync(self):
|
||||||
"""Documented residual (RSYNC_COMPAT.md `-n/--dry-run` row): FastSync's
|
"""`-n --delete` lists exactly the destination extras rsync would remove.
|
||||||
dry-run would-delete report includes the file that is merely being
|
|
||||||
updated (derived from the receiver's STATUS_STATS extras) and, unlike
|
Covers the three cases that a real run protects: the file being updated
|
||||||
rsync, also reports an excluded-but-protected extra. rsync `-n -i
|
(in the keep set), a filter-excluded source entry (protected prefix), and
|
||||||
--delete` lists only genuine extras. Pins the residual that keeps the
|
a --max-size-pruned source entry (always-protected prefix). Track 4a
|
||||||
row Divergent."""
|
adds a fourth: a destination-only entry matching the exclude rule is
|
||||||
|
re-derived on the receiver and also protected, so only the genuine
|
||||||
|
destination-only `extra.txt` appears.
|
||||||
|
"""
|
||||||
source = os.path.join(TEST_DATA_DIR, "dryrep_src")
|
source = os.path.join(TEST_DATA_DIR, "dryrep_src")
|
||||||
rdst = os.path.join(TEST_DATA_DIR, "dryrep_rdst")
|
rdst = os.path.join(TEST_DATA_DIR, "dryrep_rdst")
|
||||||
fdst = os.path.join(TEST_DATA_DIR, "dryrep_fdst")
|
fdst = os.path.join(TEST_DATA_DIR, "dryrep_fdst")
|
||||||
clean_dir(source)
|
clean_dir(source)
|
||||||
clean_dir(rdst)
|
clean_dir(rdst)
|
||||||
clean_dir(fdst)
|
clean_dir(fdst)
|
||||||
with open(os.path.join(source, "a.txt"), "wb") as fh:
|
for name, data in (("a.txt", b"new content\n"), ("keep.log", b"log\n"),
|
||||||
fh.write(b"new content\n")
|
("big.bin", b"B" * 2000)):
|
||||||
|
with open(os.path.join(source, name), "wb") as fh:
|
||||||
|
fh.write(data)
|
||||||
os.utime(os.path.join(source, "a.txt"), (1_700_000_000, 1_700_000_000))
|
os.utime(os.path.join(source, "a.txt"), (1_700_000_000, 1_700_000_000))
|
||||||
for root in (rdst, fdst):
|
received = get_dest_received_dir(fdst, source)
|
||||||
with open(os.path.join(root, "a.txt"), "wb") as fh:
|
os.makedirs(received, exist_ok=True)
|
||||||
fh.write(b"old\n")
|
for root in (rdst, received):
|
||||||
for name, data in (("extra.log", b"log\n"), ("extra.txt", b"extra\n")):
|
for name, data in (("a.txt", b"old\n"), ("keep.log", b"log\n"),
|
||||||
|
("big.bin", b"B" * 2000), ("extra.txt", b"extra\n"),
|
||||||
|
("stray.log", b"dest only\n")):
|
||||||
with open(os.path.join(root, name), "wb") as fh:
|
with open(os.path.join(root, name), "wb") as fh:
|
||||||
fh.write(data)
|
fh.write(data)
|
||||||
for p in (os.path.join(root, "a.txt"), os.path.join(root, "extra.log"),
|
os.utime(os.path.join(root, name), (1_500_000_000, 1_500_000_000))
|
||||||
os.path.join(root, "extra.txt")):
|
|
||||||
os.utime(p, (1_500_000_000, 1_500_000_000))
|
|
||||||
|
|
||||||
|
flags = ["-a", "-n", "-i", "--delete", "--exclude=*.log", "--max-size=1000"]
|
||||||
r = subprocess.run(["rsync", "-an", "-i", "--delete", "--exclude=*.log",
|
r = subprocess.run(["rsync", "-an", "-i", "--delete", "--exclude=*.log",
|
||||||
source + "/", rdst + "/"],
|
"--max-size=1000", source + "/", rdst + "/"],
|
||||||
capture_output=True, text=True,
|
capture_output=True, text=True,
|
||||||
env=dict(os.environ, LC_ALL="C"))
|
env=dict(os.environ, LC_ALL="C"))
|
||||||
assert r.returncode == 0, r.stderr
|
assert r.returncode == 0, r.stderr
|
||||||
rsync_del = {l.split(None, 1)[1] for l in r.stdout.splitlines()
|
rsync_del = sorted(l for l in r.stdout.splitlines() if l.startswith("*deleting"))
|
||||||
if l.startswith("*deleting")}
|
assert rsync_del == ["*deleting extra.txt"], f"unexpected rsync set: {rsync_del}"
|
||||||
assert rsync_del == {"extra.txt"}, f"unexpected rsync deleting set: {rsync_del}"
|
|
||||||
|
|
||||||
with ServerManager() as server:
|
with ServerManager() as server:
|
||||||
server.start(extra_args=["--allow-delete"])
|
server.start(extra_args=["--allow-delete"])
|
||||||
result, _ = run_client(source, fdst,
|
result, _ = run_client(source, fdst, flags=flags, port=server.port)
|
||||||
flags=["-a", "-n", "-i", "--delete", "--exclude=*.log"],
|
|
||||||
port=server.port)
|
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
fs_del = {l.split(None, 1)[1] for l in (result.stdout or "").splitlines()
|
fs_del = sorted(l for l in (result.stdout or "").splitlines()
|
||||||
if l.startswith("*deleting")}
|
if l.startswith("*deleting"))
|
||||||
# Documented over-report: the transferred/updated file and the excluded
|
assert fs_del == rsync_del, f"rsync={rsync_del}\nfastsync={fs_del}"
|
||||||
# extra appear in FastSync's would-delete set.
|
assert os.path.exists(os.path.join(received, "stray.log")), \
|
||||||
assert "a.txt" in fs_del, "residual changed: FastSync no longer over-reports the update"
|
"destination-only exclude match must be protected in the dry-run report"
|
||||||
assert "extra.log" in fs_del, "residual changed: FastSync no longer reports excluded extra"
|
|
||||||
|
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_remote_dry_run_quiet_is_silent(self, shared_server):
|
def test_remote_dry_run_quiet_is_silent(self, shared_server):
|
||||||
@@ -3980,15 +3982,16 @@ class TestDeleteTiming:
|
|||||||
assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n", \
|
assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n", \
|
||||||
f"{flag}: nested file was not written after the early deletion"
|
f"{flag}: nested file was not written after the early deletion"
|
||||||
|
|
||||||
@pytest.mark.parametrize("flag", ["--delete", "--delete-after"])
|
@pytest.mark.parametrize("flag", ["--delete-commit", "--delete-after"])
|
||||||
@pytest.mark.parametrize("mt", [False, True])
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
def test_late_flags_commit_only_after_success(self, flag, mt):
|
def test_late_flags_commit_only_after_success(self, flag, mt):
|
||||||
"""Plain --delete/--delete-after defer deletion until the whole transfer
|
"""--delete-commit/--delete-after defer deletion until the whole transfer
|
||||||
succeeds: a mid-transfer write failure must leave every extra in place
|
succeeds: a mid-transfer write failure must leave every extra in place
|
||||||
(commit-style safety). The -m receiver must also keep the extras: the
|
(commit-style safety). Plain --delete no longer defers (it defaults to
|
||||||
deferred keep-set is committed by the server only after the disk-writer
|
delete-during), so only the explicitly late timings are exercised here.
|
||||||
thread has finished, and a failing writer means the manifest is freed,
|
The --threads receiver must also keep the extras: the deferred keep-set is
|
||||||
never applied."""
|
committed by the server only after the disk-writer thread has finished,
|
||||||
|
and a failing writer means the manifest is freed, never applied."""
|
||||||
source = self._seed("late")
|
source = self._seed("late")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "deltiming_late_dst")
|
dest = os.path.join(TEST_DATA_DIR, "deltiming_late_dst")
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
@@ -4693,11 +4696,11 @@ class TestDeletePolicy:
|
|||||||
finally:
|
finally:
|
||||||
os.chmod(source, 0o755)
|
os.chmod(source, 0o755)
|
||||||
|
|
||||||
def test_delete_excluded_protection_is_sender_derived(self):
|
def test_delete_protection_reapplied_on_receiver(self):
|
||||||
"""Plain --delete protects destination mirrors of files the SOURCE scan
|
"""Plain --delete protects destination mirrors of files the SOURCE scan
|
||||||
excluded, but a destination-only file that merely matches an exclude
|
excluded, and (track 4a) also protects a destination-only file matching
|
||||||
rule is still an extra and is removed (protection never re-applies rules
|
an exclude rule because the compiled rule set is re-applied on the
|
||||||
to the destination)."""
|
receiver, matching rsync."""
|
||||||
source = os.path.join(TEST_DATA_DIR, "senderderived_src")
|
source = os.path.join(TEST_DATA_DIR, "senderderived_src")
|
||||||
clean_dir(source)
|
clean_dir(source)
|
||||||
self._write(os.path.join(source, "keep.txt"), b"kept\n")
|
self._write(os.path.join(source, "keep.txt"), b"kept\n")
|
||||||
@@ -4717,8 +4720,8 @@ class TestDeletePolicy:
|
|||||||
f"delete sync failed: {(result.stderr or result.stdout)[:300]}"
|
f"delete sync failed: {(result.stderr or result.stdout)[:300]}"
|
||||||
assert os.path.exists(os.path.join(received, "secret.log")), \
|
assert os.path.exists(os.path.join(received, "secret.log")), \
|
||||||
"source-excluded mirror was deleted under plain --delete"
|
"source-excluded mirror was deleted under plain --delete"
|
||||||
assert not os.path.exists(os.path.join(received, "stray.log")), \
|
assert os.path.exists(os.path.join(received, "stray.log")), \
|
||||||
"destination-only file matching the exclude rule was left (should be deleted)"
|
"destination-only file matching the exclude rule must be protected like rsync"
|
||||||
|
|
||||||
|
|
||||||
def _pin_mtime(path, ts):
|
def _pin_mtime(path, ts):
|
||||||
@@ -4738,11 +4741,12 @@ class TestBasisDestDirs:
|
|||||||
STAGING = ".fastsync-stage"
|
STAGING = ".fastsync-stage"
|
||||||
TS = 1577836800 # 2020-01-01 00:00:00 UTC, used to pin matching mtimes
|
TS = 1577836800 # 2020-01-01 00:00:00 UTC, used to pin matching mtimes
|
||||||
|
|
||||||
# fixture files: source and basis share the mtime pin, so a basis "match"
|
# fixture files: source and basis share the mtime pin, so the DEFAULT
|
||||||
# is decided purely by content (xxHash). unchanged.txt is byte-identical;
|
# (rsync-parity) quick-check is a size+mtime match and trusts the basis even
|
||||||
# changed.txt is byte-DIFFERENT but has the SAME SIZE as the source (and
|
# when the body differs. unchanged.txt is byte-identical; changed.txt is
|
||||||
# the same pinned mtime), which is what forces the content-hash gate;
|
# byte-DIFFERENT but has the SAME SIZE as the source (and the same pinned
|
||||||
# added.txt does not exist in the basis at all.
|
# mtime), which is what the FastSync-only --verify-basis content gate
|
||||||
|
# rejects; added.txt does not exist in the basis at all.
|
||||||
UNCHANGED = "unchanged.txt"
|
UNCHANGED = "unchanged.txt"
|
||||||
CHANGED = "changed.txt"
|
CHANGED = "changed.txt"
|
||||||
ADDED = "added.txt"
|
ADDED = "added.txt"
|
||||||
@@ -4782,18 +4786,19 @@ class TestBasisDestDirs:
|
|||||||
}
|
}
|
||||||
|
|
||||||
def _basis_tree(self, prefix):
|
def _basis_tree(self, prefix):
|
||||||
# unchanged.txt is identical to the source; changed.txt has the SAME
|
# unchanged.txt is identical to the source; changed.txt has a DIFFERENT
|
||||||
# byte size and pinned mtime but a different body (equal size forces
|
# size (and body) so the size leg of the quick-check fails and it is
|
||||||
# the xxHash gate); added.txt is missing from the basis.
|
# transferred normally; added.txt is missing from the basis.
|
||||||
return {
|
return {
|
||||||
self.UNCHANGED: b"stable content v1\n",
|
self.UNCHANGED: b"stable content v1\n",
|
||||||
self.CHANGED: b"CHANGED CONTENT NOW\n",
|
self.CHANGED: b"CHANGED CONTENT NOW AND LONGER\n",
|
||||||
}
|
}
|
||||||
|
|
||||||
def test_same_size_different_content_is_not_a_basis_match(self, shared_server):
|
def test_same_size_different_content_default_trusts_quick_check(self, shared_server):
|
||||||
# Core safety property: equal size + pinned mtime but different content
|
# Default rsync-parity behavior: equal size + pinned mtime is a basis
|
||||||
# must NEVER be hard-linked or copied from the basis -- the xxHash gate
|
# match, so the basis body is materialized/linked without reading it.
|
||||||
# rejects it and the sender's data is transferred instead.
|
# This mirrors rsync 3.4.1's quick check (differential-tested in
|
||||||
|
# test_differential_parity.py::test_verify_basis_restores_strict_content).
|
||||||
for flag, basis_dir in (("--link-dest", "szlb"), ("--copy-dest", "szcp"),
|
for flag, basis_dir in (("--link-dest", "szlb"), ("--copy-dest", "szcp"),
|
||||||
("--compare-dest", "szcmp")):
|
("--compare-dest", "szcmp")):
|
||||||
source = self._make_source("basis_same_size_src",
|
source = self._make_source("basis_same_size_src",
|
||||||
@@ -4805,7 +4810,36 @@ class TestBasisDestDirs:
|
|||||||
result, _ = run_client(source, dest, flags=[f"{flag}={basis_dir}"],
|
result, _ = run_client(source, dest, flags=[f"{flag}={basis_dir}"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode == 0, \
|
assert result.returncode == 0, \
|
||||||
f"{flag} same-size mismatch failed: {result.stderr[:300]}"
|
f"{flag} same-size quick-check failed: {result.stderr[:300]}"
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
dest_file = os.path.join(received, self.UNCHANGED)
|
||||||
|
if flag == "--compare-dest":
|
||||||
|
assert not os.path.exists(dest_file), \
|
||||||
|
f"{flag}: compare-dest must leave a matching file sparse"
|
||||||
|
else:
|
||||||
|
assert _read_file(dest_file) == b"SAME LENGTH BODY!", \
|
||||||
|
f"{flag}: default quick-check did not trust the basis body"
|
||||||
|
if flag == "--link-dest":
|
||||||
|
assert os.stat(dest_file).st_ino == os.stat(basis_file).st_ino, \
|
||||||
|
f"{flag}: basis was not hard-linked"
|
||||||
|
|
||||||
|
def test_verify_basis_rejects_same_size_different_content(self, shared_server):
|
||||||
|
# FastSync-only --verify-basis: the whole-file digest gate rejects the
|
||||||
|
# same-size/different-content basis, so the source data is transferred
|
||||||
|
# instead of the wrong basis bytes.
|
||||||
|
for flag, basis_dir in (("--link-dest", "vszlb"), ("--copy-dest", "vszcp"),
|
||||||
|
("--compare-dest", "vszcmp")):
|
||||||
|
source = self._make_source("basis_verify_src",
|
||||||
|
{self.UNCHANGED: b"same length body\n"})
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, f"basis_verify_dst_{basis_dir}")
|
||||||
|
clean_dir(dest)
|
||||||
|
basis_file = self._seed_basis_file(dest, source, basis_dir, self.UNCHANGED,
|
||||||
|
b"SAME LENGTH BODY!")
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=[f"{flag}={basis_dir}", "--verify-basis"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"{flag} --verify-basis failed: {result.stderr[:300]}"
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
dest_file = os.path.join(received, self.UNCHANGED)
|
dest_file = os.path.join(received, self.UNCHANGED)
|
||||||
assert _read_file(dest_file) == b"same length body\n", \
|
assert _read_file(dest_file) == b"same length body\n", \
|
||||||
@@ -4837,11 +4871,13 @@ class TestBasisDestDirs:
|
|||||||
self._source_tree("c")[self.ADDED], "added file not transferred"
|
self._source_tree("c")[self.ADDED], "added file not transferred"
|
||||||
|
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_dry_run_compare_dest_does_not_read_basis(self, shared_server):
|
def test_dry_run_compare_dest_quick_check_does_not_read_basis(self, shared_server):
|
||||||
# A dry-run --compare-dest must never read/hash the basis file: doing so
|
# A dry-run --compare-dest must never read/hash the basis file. Under
|
||||||
# is a 1-bit content oracle against the client-supplied digest. Even a
|
# the default metadata quick-check a matching basis is reported as a
|
||||||
# byte-identical basis with a matching size+mtime is therefore reported
|
# skip (matching rsync) without reading it; nothing is created. Under
|
||||||
# as would-transfer, and nothing is created.
|
# --verify-basis, which would require hashing, the dry-run cannot
|
||||||
|
# confirm the hit (that would be a 1-bit content oracle) and reports
|
||||||
|
# would-transfer instead.
|
||||||
source = self._make_source("basis_dry_src", {self.UNCHANGED: b"stable content v1\n"})
|
source = self._make_source("basis_dry_src", {self.UNCHANGED: b"stable content v1\n"})
|
||||||
dest = os.path.join(TEST_DATA_DIR, "basis_dry_dst")
|
dest = os.path.join(TEST_DATA_DIR, "basis_dry_dst")
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
@@ -4852,11 +4888,30 @@ class TestBasisDestDirs:
|
|||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode == 0, \
|
assert result.returncode == 0, \
|
||||||
f"dry-run compare-dest failed: {result.stderr[:300]}"
|
f"dry-run compare-dest failed: {result.stderr[:300]}"
|
||||||
assert self.UNCHANGED in result.stdout, (
|
assert self.UNCHANGED not in result.stdout, (
|
||||||
"dry-run compare-dest silently skipped: receiver read the basis content"
|
"dry-run compare-dest did not honor the metadata quick-check "
|
||||||
|
"(reported would-transfer for a matching basis)"
|
||||||
)
|
)
|
||||||
assert _snapshot_tree(dest) == before, "dry-run compare-dest mutated the destination"
|
assert _snapshot_tree(dest) == before, "dry-run compare-dest mutated the destination"
|
||||||
|
|
||||||
|
# --verify-basis: the hit needs the basis content, which a dry-run must
|
||||||
|
# not read, so the file is reported as would-transfer.
|
||||||
|
dest2 = os.path.join(TEST_DATA_DIR, "basis_dry_verify_dst")
|
||||||
|
clean_dir(dest2)
|
||||||
|
self._seed_basis(dest2, source, "drybasis", {self.UNCHANGED: b"stable content v1\n"})
|
||||||
|
before2 = _snapshot_tree(dest2)
|
||||||
|
result, _ = run_client(source, dest2,
|
||||||
|
flags=["--compare-dest=drybasis", "--dry-run",
|
||||||
|
"--verify-basis"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"dry-run --verify-basis compare-dest failed: {result.stderr[:300]}"
|
||||||
|
assert self.UNCHANGED in result.stdout, (
|
||||||
|
"dry-run --verify-basis must not read the basis to confirm a hit"
|
||||||
|
)
|
||||||
|
assert _snapshot_tree(dest2) == before2, \
|
||||||
|
"dry-run --verify-basis compare-dest mutated the destination"
|
||||||
|
|
||||||
def test_compare_dest_content_mismatch_forces_transfer(self, shared_server):
|
def test_compare_dest_content_mismatch_forces_transfer(self, shared_server):
|
||||||
# The basis holds a file with a DIFFERENT body: even though it shares
|
# The basis holds a file with a DIFFERENT body: even though it shares
|
||||||
# the mtime pin, the xxHash check fails and the data must be sent.
|
# the mtime pin, the xxHash check fails and the data must be sent.
|
||||||
@@ -5095,27 +5150,36 @@ class TestBasisDestDirs:
|
|||||||
assert os.stat(dest_file).st_ino != os.stat(basis_file).st_ino, \
|
assert os.stat(dest_file).st_ino != os.stat(basis_file).st_ino, \
|
||||||
"--ignore-times must not hard-link to a basis file"
|
"--ignore-times must not hard-link to a basis file"
|
||||||
|
|
||||||
def test_basis_refuses_file_above_whole_file_limit(self, shared_server):
|
def test_basis_handles_file_above_whole_file_limit(self, shared_server):
|
||||||
# Every whole-file payload path in FastSync (basis dirs included) is
|
# Track 5a: a basis hit streams the copy (and the --verify-basis digest
|
||||||
# bounded by MAX_RECEIVE_WHOLE_FILE_SIZE. rsync supports basis dirs for
|
# streams the basis), so a source larger than the whole-file payload
|
||||||
# arbitrary sizes; FastSync refuses such a run up front with a clear
|
# bound is supported for basis dirs exactly like rsync. A basis MISS
|
||||||
# diagnostic instead of letting the receiver abort the whole transfer
|
# still falls back to the normal transfer, which keeps its own bound.
|
||||||
# mid-stream with no client-side explanation.
|
|
||||||
source = self._make_source("basis_oversize_src", {"small.txt": b"ok\n"})
|
source = self._make_source("basis_oversize_src", {"small.txt": b"ok\n"})
|
||||||
big = os.path.join(source, "huge.bin")
|
big = os.path.join(source, "huge.bin")
|
||||||
with open(big, "wb") as fh:
|
with open(big, "wb") as fh:
|
||||||
os.ftruncate(fh.fileno(), 256 * 1024 * 1024 + 4096)
|
os.ftruncate(fh.fileno(), 256 * 1024 * 1024 + 4096)
|
||||||
dest = os.path.join(TEST_DATA_DIR, "basis_oversize_dst")
|
dest = os.path.join(TEST_DATA_DIR, "basis_oversize_dst")
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
result, _ = run_client(source, dest, flags=["--link-dest=nope"],
|
|
||||||
port=shared_server.port)
|
|
||||||
assert result.returncode != 0, \
|
|
||||||
"basis run with an over-limit file unexpectedly succeeded"
|
|
||||||
assert "larger than" in result.stderr, \
|
|
||||||
f"no clear over-limit diagnostic: {result.stderr[:300]}"
|
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
assert not os.path.exists(received), \
|
rel = os.path.relpath(received, dest)
|
||||||
"over-limit basis run transferred files before failing"
|
basis_big = os.path.join(dest, "ob", rel, "huge.bin")
|
||||||
|
os.makedirs(os.path.dirname(basis_big), exist_ok=True)
|
||||||
|
shutil.copyfile(big, basis_big)
|
||||||
|
os.utime(basis_big, (self.TS, self.TS))
|
||||||
|
os.utime(big, (self.TS, self.TS))
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=["--link-dest=ob", "--incremental"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"over-limit basis run failed: {result.stderr[:300]}"
|
||||||
|
dest_big = os.path.join(received, "huge.bin")
|
||||||
|
assert os.path.exists(dest_big), "over-limit basis hit was not materialized"
|
||||||
|
assert os.path.getsize(dest_big) == 256 * 1024 * 1024 + 4096
|
||||||
|
assert os.stat(dest_big).st_ino == os.stat(basis_big).st_ino, \
|
||||||
|
"over-limit --link-dest did not hard-link to the basis"
|
||||||
|
assert _read_file(os.path.join(received, "small.txt")) == b"ok\n"
|
||||||
|
|
||||||
|
|
||||||
def _random_payloads(size=2 * 1024 * 1024, changed=64 * 1024, seed=1234):
|
def _random_payloads(size=2 * 1024 * 1024, changed=64 * 1024, seed=1234):
|
||||||
|
|||||||
@@ -164,6 +164,64 @@ class TestInfoParity:
|
|||||||
f"rsync={entries(rsync_result.stdout)} fastsync={entries(result.stdout)}"
|
f"rsync={entries(rsync_result.stdout)} fastsync={entries(result.stdout)}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_info_name_root_line_matches_rsync(self, shared_server):
|
||||||
|
"""A fresh destination: rsync prints `created directory`, then the
|
||||||
|
transfer-root `./` name line before the entries; FastSync must emit the
|
||||||
|
same `./` line."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "inf_root_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "inf_root_dst")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "inf_root_rdst")
|
||||||
|
clean_dir(source)
|
||||||
|
_write(os.path.join(source, "f.bin"), b"payload\n")
|
||||||
|
clean_dir(dest)
|
||||||
|
shutil.rmtree(rdst, ignore_errors=True)
|
||||||
|
rsync_result = _rsync(["-a", "--info=name", source + "/", rdst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
result, _ = run_client(source, dest, flags=["-a", "--info=name"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
|
||||||
|
|
||||||
|
def names(text):
|
||||||
|
return [l for l in text.splitlines()
|
||||||
|
if l and not l.startswith("created directory")
|
||||||
|
and not (l.endswith("/") and l != "./")]
|
||||||
|
|
||||||
|
assert names(rsync_result.stdout) == ["./", "f.bin"], names(rsync_result.stdout)
|
||||||
|
assert names(result.stdout) == ["./", "f.bin"], names(result.stdout)
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_info_name2_uptodate_matches_rsync(self, shared_server):
|
||||||
|
"""--info=name2 prints `NAME is uptodate` for entries the receiver
|
||||||
|
already has, matching rsync byte-for-byte."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "inf_up_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "inf_up_dst")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "inf_up_rdst")
|
||||||
|
clean_dir(source)
|
||||||
|
os.makedirs(os.path.join(source, "sub"))
|
||||||
|
_write(os.path.join(source, "a.txt"), b"a\n")
|
||||||
|
_write(os.path.join(source, "sub", "b.txt"), b"b\n")
|
||||||
|
clean_dir(rdst)
|
||||||
|
assert _rsync(["-a", source + "/", rdst + "/"]).returncode == 0
|
||||||
|
rsync_result = _rsync(["-a", "--info=name2", source + "/", rdst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
|
||||||
|
clean_dir(dest)
|
||||||
|
seed, _ = run_client(source, dest, flags=["-a", "--incremental"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert seed.returncode == 0, (seed.stderr or seed.stdout)[:200]
|
||||||
|
result, _ = run_client(source, dest, flags=["-a", "--incremental", "--info=name2"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
|
||||||
|
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
|
||||||
|
if l.endswith("is uptodate"))
|
||||||
|
fast_lines = sorted(l for l in result.stdout.splitlines()
|
||||||
|
if l.endswith("is uptodate"))
|
||||||
|
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
|
||||||
|
assert fast_lines == ["a.txt is uptodate", "sub/b.txt is uptodate"], fast_lines
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_info_nonreg_matches_rsync(self, shared_server):
|
def test_info_nonreg_matches_rsync(self, shared_server):
|
||||||
@@ -420,14 +478,14 @@ class TestRemoteOptionDaemon:
|
|||||||
assert "remote-option" in (result.stderr + result.stdout)
|
assert "remote-option" in (result.stderr + result.stdout)
|
||||||
|
|
||||||
|
|
||||||
class TestFilterProtectDivergence:
|
class TestFilterProtect:
|
||||||
"""Documented residual: a protect rule that matches only a destination-only
|
"""Receiver-derived delete protection: a `protect`/`P` rule is compiled by
|
||||||
entry is not re-derived on the receiver (FastSync derives delete protection
|
the sender and sent on the config frame, so the receiver shields a
|
||||||
from the source scan), so rsync protects the extra but FastSync removes it."""
|
destination-only entry that never appeared on the sender, matching rsync."""
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_protect_dest_only_divergence(self, shared_server):
|
def test_protect_dest_only_matches_rsync(self, shared_server):
|
||||||
source = os.path.join(TEST_DATA_DIR, "fpd_src")
|
source = os.path.join(TEST_DATA_DIR, "fpd_src")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "fpd_dst")
|
dest = os.path.join(TEST_DATA_DIR, "fpd_dst")
|
||||||
rdst = os.path.join(TEST_DATA_DIR, "fpd_rdst")
|
rdst = os.path.join(TEST_DATA_DIR, "fpd_rdst")
|
||||||
@@ -440,6 +498,7 @@ class TestFilterProtectDivergence:
|
|||||||
rsync_result = _rsync(["-a", "--delete", "--filter=P *.log", source + "/", rdst + "/"])
|
rsync_result = _rsync(["-a", "--delete", "--filter=P *.log", source + "/", rdst + "/"])
|
||||||
assert rsync_result.returncode == 0, rsync_result.stderr
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
assert os.path.exists(os.path.join(rdst, "extra.log")), "rsync did not protect extra.log"
|
assert os.path.exists(os.path.join(rdst, "extra.log")), "rsync did not protect extra.log"
|
||||||
|
assert not os.path.exists(os.path.join(rdst, "other.txt")), "rsync did not delete other.txt"
|
||||||
|
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
@@ -451,9 +510,29 @@ class TestFilterProtectDivergence:
|
|||||||
flags=["-a", "--delete", "--filter=P *.log"],
|
flags=["-a", "--delete", "--filter=P *.log"],
|
||||||
port=server.port)
|
port=server.port)
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
|
||||||
# Pin the known divergence: FastSync deletes the destination-only file.
|
assert os.path.exists(os.path.join(received, "extra.log")), (
|
||||||
assert not os.path.exists(os.path.join(received, "extra.log")), (
|
"FastSync must protect a destination-only P match like rsync")
|
||||||
"FastSync now protects destination-only P matches; the --filter row may be "
|
|
||||||
"upgradable to full parity"
|
|
||||||
)
|
|
||||||
assert not os.path.exists(os.path.join(received, "other.txt"))
|
assert not os.path.exists(os.path.join(received, "other.txt"))
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_protect_dest_only_dry_run_enumeration(self, shared_server):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "fpd_nd_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "fpd_nd_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
_write(os.path.join(source, "keep.txt"), b"keep\n")
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
clean_dir(received)
|
||||||
|
_write(os.path.join(received, "keep.txt"), b"keep\n")
|
||||||
|
_write(os.path.join(received, "extra.log"), b"extra\n")
|
||||||
|
_write(os.path.join(received, "other.txt"), b"other\n")
|
||||||
|
with ServerManager() as server:
|
||||||
|
server.start(extra_args=["--allow-delete"])
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=["-a", "-n", "--delete", "--out-format=%n",
|
||||||
|
"--filter=P *.log"],
|
||||||
|
port=server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
assert "other.txt" in result.stdout, result.stdout
|
||||||
|
assert "extra.log" not in result.stdout, result.stdout
|
||||||
|
assert os.path.exists(os.path.join(received, "extra.log"))
|
||||||
|
assert os.path.exists(os.path.join(received, "other.txt"))
|
||||||
|
|||||||
@@ -289,6 +289,49 @@ def _make_one_file(root, name="f.bin", size=100):
|
|||||||
fh.write(bytes((i * 7 + 3) & 0xFF for i in range(size)))
|
fh.write(bytes((i * 7 + 3) & 0xFF for i in range(size)))
|
||||||
|
|
||||||
|
|
||||||
|
def _make_multidir_tree(root):
|
||||||
|
"""Multi-directory corpus for the --progress file-list tests: nested files,
|
||||||
|
a directory-only branch, an empty directory and a symlink."""
|
||||||
|
clean_dir(root)
|
||||||
|
for rel, data in (("a.txt", b"alpha\n"), ("b.txt", b"bravo\n"),
|
||||||
|
("sub1/c.txt", b"charlie\n"), ("sub1/deep/d.txt", b"delta\n"),
|
||||||
|
("sub2/e.txt", b"echo\n")):
|
||||||
|
path = os.path.join(root, rel)
|
||||||
|
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||||
|
with open(path, "wb") as fh:
|
||||||
|
fh.write(data)
|
||||||
|
os.symlink("a.txt", os.path.join(root, "link1"))
|
||||||
|
os.makedirs(os.path.join(root, "emptydir"), exist_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_progress(text):
|
||||||
|
"""Name lines and the `to-chk` denominators from a --progress run."""
|
||||||
|
names = []
|
||||||
|
totals = set()
|
||||||
|
for line in text.splitlines():
|
||||||
|
line = line.rstrip()
|
||||||
|
if not line or line == "sending incremental file list":
|
||||||
|
continue
|
||||||
|
if "%" in line:
|
||||||
|
match = re.search(r"to-chk=\d+/(\d+)", line)
|
||||||
|
if match:
|
||||||
|
totals.add(int(match.group(1)))
|
||||||
|
continue
|
||||||
|
if line == "./": # root-line trigger is a separate documented residual
|
||||||
|
continue
|
||||||
|
names.append(line)
|
||||||
|
return sorted(names), totals
|
||||||
|
|
||||||
|
|
||||||
|
def _pick_stats(text, keys):
|
||||||
|
out = {}
|
||||||
|
for line in text.splitlines():
|
||||||
|
for key in keys:
|
||||||
|
if line.startswith(key + ":"):
|
||||||
|
out[key] = line
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
class TestWireStatsParity:
|
class TestWireStatsParity:
|
||||||
"""Wire-counter output parity: --out-format %b/%c/%C, --progress and
|
"""Wire-counter output parity: --out-format %b/%c/%C, --progress and
|
||||||
--stats versus real rsync 3.4.1."""
|
--stats versus real rsync 3.4.1."""
|
||||||
@@ -484,6 +527,62 @@ class TestWireStatsParity:
|
|||||||
assert "to-chk=0/2" in fast_lines[-1], fast_lines[-1]
|
assert "to-chk=0/2" in fast_lines[-1], fast_lines[-1]
|
||||||
assert fast_lines[-1] == rsync_lines[-1], (rsync_lines[-1], fast_lines[-1])
|
assert fast_lines[-1] == rsync_lines[-1], (rsync_lines[-1], fast_lines[-1])
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
|
def test_progress_multidir_file_list_matches_rsync(self, shared_server, mt):
|
||||||
|
"""A multi-directory tree: the paths-only pre-count must reproduce
|
||||||
|
rsync's file-list set and `to-chk` denominator. Per-directory name
|
||||||
|
lines are emitted for directories, symlinks and the empty directory; the
|
||||||
|
name set and the denominator (every entry plus the transfer root) match
|
||||||
|
rsync, while the emitted *order* remains a documented residual (rsync
|
||||||
|
sorts depth-first, FastSync streams in readdir/BFS order)."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "wire_pgmd_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "wire_pgmd_dst")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "wire_pgmd_rdst")
|
||||||
|
_make_multidir_tree(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
clean_dir(rdst)
|
||||||
|
|
||||||
|
rsync_result = _rsync(["-a", "--progress", source + "/", rdst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
flags = ["-a", "--progress"] + (["--threads"] if mt else [])
|
||||||
|
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
|
||||||
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
|
|
||||||
|
rsync_names, rsync_totals = _parse_progress(rsync_result.stdout)
|
||||||
|
fast_names, fast_totals = _parse_progress(result.stdout)
|
||||||
|
assert sorted(rsync_names) == [
|
||||||
|
"a.txt", "b.txt", "emptydir/", "link1 -> a.txt", "sub1/",
|
||||||
|
"sub1/c.txt", "sub1/deep/", "sub1/deep/d.txt", "sub2/", "sub2/e.txt",
|
||||||
|
], rsync_names
|
||||||
|
assert fast_names == rsync_names, (rsync_names, fast_names)
|
||||||
|
# 10 entries + the transfer-root "." counted by rsync's file list.
|
||||||
|
assert rsync_totals == {11}, rsync_totals
|
||||||
|
assert fast_totals == rsync_totals, (rsync_totals, fast_totals)
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_progress_delete_during_reuses_pre_scan(self):
|
||||||
|
"""--delete-during + --progress reuses the keep-set pre-scan instead of
|
||||||
|
walking the tree a second time: the file-list total and directory name
|
||||||
|
lines are identical to a plain --progress run."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "wire_pgdel_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "wire_pgdel_dst")
|
||||||
|
_make_multidir_tree(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
server = ServerManager()
|
||||||
|
server.start(extra_args=["--allow-super", "--allow-delete"])
|
||||||
|
try:
|
||||||
|
result, _ = run_client(source, dest, flags=["-a", "--progress", "--delete-during"],
|
||||||
|
port=server.port)
|
||||||
|
finally:
|
||||||
|
server.stop()
|
||||||
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
|
names, totals = _parse_progress(result.stdout)
|
||||||
|
assert totals == {11}, totals
|
||||||
|
assert "sub1/" in names and "sub1/deep/" in names and "emptydir/" in names, names
|
||||||
|
assert "link1 -> a.txt" in names, names
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
@pytest.mark.parametrize("mt", [False, True])
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
@@ -496,6 +595,9 @@ class TestWireStatsParity:
|
|||||||
_make_one_file(source, "f.bin", 6000)
|
_make_one_file(source, "f.bin", 6000)
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
clean_dir(rdst)
|
clean_dir(rdst)
|
||||||
|
# Start both tools from the same state: rsync's destination root exists,
|
||||||
|
# so pre-create FastSync's mirrored logical root as well.
|
||||||
|
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
|
||||||
rsync_result = _rsync(["-a", "--stats", source + "/", rdst + "/"])
|
rsync_result = _rsync(["-a", "--stats", source + "/", rdst + "/"])
|
||||||
assert rsync_result.returncode == 0, rsync_result.stderr
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
flags = ["-a", "--stats"] + (["--threads"] if mt else [])
|
flags = ["-a", "--stats"] + (["--threads"] if mt else [])
|
||||||
@@ -526,17 +628,18 @@ class TestWireStatsParity:
|
|||||||
@requires_rsync
|
@requires_rsync
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_stats_file_count_breakdown_matches_rsync(self, shared_server):
|
def test_stats_file_count_breakdown_matches_rsync(self, shared_server):
|
||||||
"""`Number of files` now carries rsync's per-type breakdown: the scanner
|
"""`Number of files` and `Number of created files` both carry rsync's
|
||||||
accounts directory entries (captured for -a/-t/-p) plus reg/link/special
|
per-type breakdown (protocol 2.28.0 reports the receiver-created
|
||||||
from the transfer list. `Number of created files` still lacks the type
|
reg/dir/link/special split over STATUS_STATS)."""
|
||||||
breakdown (FastSync cannot tell which entries the receiver newly
|
|
||||||
created), so that residual is pinned separately."""
|
|
||||||
source = os.path.join(TEST_DATA_DIR, "wire_stc_src")
|
source = os.path.join(TEST_DATA_DIR, "wire_stc_src")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "wire_stc_dst")
|
dest = os.path.join(TEST_DATA_DIR, "wire_stc_dst")
|
||||||
rdst = os.path.join(TEST_DATA_DIR, "wire_stc_rdst")
|
rdst = os.path.join(TEST_DATA_DIR, "wire_stc_rdst")
|
||||||
_make_one_file(source, "f.bin", 6000)
|
_make_one_file(source, "f.bin", 6000)
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
clean_dir(rdst)
|
clean_dir(rdst)
|
||||||
|
# Start both tools from the same state: rsync's destination root exists,
|
||||||
|
# so pre-create FastSync's mirrored logical root as well.
|
||||||
|
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
|
||||||
rsync_result = _rsync(["-a", "--stats", source + "/", rdst + "/"])
|
rsync_result = _rsync(["-a", "--stats", source + "/", rdst + "/"])
|
||||||
assert rsync_result.returncode == 0, rsync_result.stderr
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
result, _ = run_client(source, dest, flags=["-a", "--stats"],
|
result, _ = run_client(source, dest, flags=["-a", "--stats"],
|
||||||
@@ -556,10 +659,71 @@ class TestWireStatsParity:
|
|||||||
|
|
||||||
assert re.match(r"Number of files: 2 \(reg: 1, dir: 1\)$", r_files), r_files
|
assert re.match(r"Number of files: 2 \(reg: 1, dir: 1\)$", r_files), r_files
|
||||||
assert r_files == f_files, (r_files, f_files)
|
assert r_files == f_files, (r_files, f_files)
|
||||||
# rsync always carries the created type breakdown; FastSync prints the
|
|
||||||
# bare transferred-regular count (documented residual).
|
|
||||||
assert re.match(r"Number of created files: 1 \(reg: 1\)$", r_created), r_created
|
assert re.match(r"Number of created files: 1 \(reg: 1\)$", r_created), r_created
|
||||||
assert re.fullmatch(r"Number of created files: 1", f_created), f_created
|
assert f_created == r_created, (r_created, f_created)
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
|
def test_stats_created_and_literal_fresh_update_delta(self, shared_server, mt):
|
||||||
|
"""The receiver-observed counters must match rsync for the three
|
||||||
|
transfer shapes: a fresh create (created breakdown + whole-file literal),
|
||||||
|
an update (created == 0, whole-file literal), and a delta update (only
|
||||||
|
the literal delta fragments are counted, not the whole file)."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "wire_stcd_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "wire_stcd_dst")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "wire_stcd_rdst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
clean_dir(rdst)
|
||||||
|
os.makedirs(source, exist_ok=True)
|
||||||
|
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
|
||||||
|
with open(os.path.join(source, "big.bin"), "wb") as fh:
|
||||||
|
fh.write(bytes(range(256)) * 4096) # 1 MiB
|
||||||
|
mt_flag = ["--threads"] if mt else []
|
||||||
|
|
||||||
|
def compare(tag):
|
||||||
|
# Pin the delta block size on both ends: rsync's adaptive block size
|
||||||
|
# would otherwise make the literal/matched split non-comparable.
|
||||||
|
rsync_result = _rsync(["-a", "--stats", "--no-whole-file", "-B8192",
|
||||||
|
source + "/", rdst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
result, _ = run_client(
|
||||||
|
source, dest,
|
||||||
|
flags=["-a", "--stats", "--incremental", "--delta", "-B8192"] + mt_flag,
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
|
keys = ("Number of created files", "Literal data", "Matched data",
|
||||||
|
"Total transferred file size")
|
||||||
|
r = _pick_stats(rsync_result.stdout, keys)
|
||||||
|
f = _pick_stats(result.stdout, keys)
|
||||||
|
assert r == f, f"{tag}: rsync={r} fastsync={f}"
|
||||||
|
return r
|
||||||
|
|
||||||
|
fresh = compare("fresh")
|
||||||
|
assert re.match(r"Number of created files: 1 \(reg: 1\)$",
|
||||||
|
fresh["Number of created files"]), fresh
|
||||||
|
|
||||||
|
# Update the source and re-run: the destination already exists.
|
||||||
|
sleep_mtime = os.path.getmtime(os.path.join(source, "big.bin")) + 2
|
||||||
|
with open(os.path.join(source, "big.bin"), "r+b") as fh:
|
||||||
|
fh.seek(100)
|
||||||
|
fh.write(b"XXXXXXXXXX")
|
||||||
|
os.utime(os.path.join(source, "big.bin"), (sleep_mtime, sleep_mtime))
|
||||||
|
update = compare("update")
|
||||||
|
assert update["Number of created files"] == "Number of created files: 0", update
|
||||||
|
|
||||||
|
# Second delta update: change bytes far apart, so rsync ships only the
|
||||||
|
# literal fragments and FastSync must report the same Literal data.
|
||||||
|
sleep_mtime = os.path.getmtime(os.path.join(source, "big.bin")) + 2
|
||||||
|
with open(os.path.join(source, "big.bin"), "r+b") as fh:
|
||||||
|
fh.seek(500000)
|
||||||
|
fh.write(b"YYYYYYYYYY")
|
||||||
|
os.utime(os.path.join(source, "big.bin"), (sleep_mtime, sleep_mtime))
|
||||||
|
delta = compare("delta")
|
||||||
|
assert delta["Number of created files"] == "Number of created files: 0", delta
|
||||||
|
lit = int(delta["Literal data"].split(":", 1)[1].strip().split()[0].replace(",", ""))
|
||||||
|
assert 0 < lit < 1024 * 1024, delta
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
|
|||||||
@@ -719,13 +719,18 @@ class TestVerifyAndFlip:
|
|||||||
source = self._src("cmpd")
|
source = self._src("cmpd")
|
||||||
dest = self._dst("cmpd")
|
dest = self._dst("cmpd")
|
||||||
rdst = self._dst("cmpd_r")
|
rdst = self._dst("cmpd_r")
|
||||||
|
# Pin the mtime so rsync's size+mtime quick-check (and FastSync's
|
||||||
|
# default) matches deterministically across a second boundary.
|
||||||
|
OLD = 1_500_000_000
|
||||||
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
||||||
fh.write(b"basis-content\n")
|
fh.write(b"basis-content\n")
|
||||||
|
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
|
||||||
# rsync resolves --compare-dest relative to the destination dir; its
|
# rsync resolves --compare-dest relative to the destination dir; its
|
||||||
# basis file sits at the transfer-relative path.
|
# basis file sits at the transfer-relative path.
|
||||||
os.makedirs(os.path.join(rdst, "basis"), exist_ok=True)
|
os.makedirs(os.path.join(rdst, "basis"), exist_ok=True)
|
||||||
with open(os.path.join(rdst, "basis", "f.txt"), "wb") as fh:
|
with open(os.path.join(rdst, "basis", "f.txt"), "wb") as fh:
|
||||||
fh.write(b"basis-content\n")
|
fh.write(b"basis-content\n")
|
||||||
|
os.utime(os.path.join(rdst, "basis", "f.txt"), (OLD, OLD))
|
||||||
rs = _rsync(["-a", "--compare-dest=basis", source + "/", rdst + "/"])
|
rs = _rsync(["-a", "--compare-dest=basis", source + "/", rdst + "/"])
|
||||||
assert rs.returncode == 0, rs.stderr
|
assert rs.returncode == 0, rs.stderr
|
||||||
assert not os.path.exists(os.path.join(rdst, "f.txt")), \
|
assert not os.path.exists(os.path.join(rdst, "f.txt")), \
|
||||||
@@ -738,6 +743,7 @@ class TestVerifyAndFlip:
|
|||||||
os.makedirs(basis, exist_ok=True)
|
os.makedirs(basis, exist_ok=True)
|
||||||
with open(os.path.join(basis, "f.txt"), "wb") as fh:
|
with open(os.path.join(basis, "f.txt"), "wb") as fh:
|
||||||
fh.write(b"basis-content\n")
|
fh.write(b"basis-content\n")
|
||||||
|
os.utime(os.path.join(basis, "f.txt"), (OLD, OLD))
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
result, _ = run_client(source, dest,
|
result, _ = run_client(source, dest,
|
||||||
flags=["--compare-dest=basis", "--incremental"],
|
flags=["--compare-dest=basis", "--incremental"],
|
||||||
@@ -751,14 +757,17 @@ class TestVerifyAndFlip:
|
|||||||
def test_link_dest_hardlinks_matches_rsync(self, shared_server):
|
def test_link_dest_hardlinks_matches_rsync(self, shared_server):
|
||||||
source = self._src("linkd")
|
source = self._src("linkd")
|
||||||
dest = self._dst("linkd")
|
dest = self._dst("linkd")
|
||||||
|
OLD = 1_500_000_000
|
||||||
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
||||||
fh.write(b"link-basis-content\n")
|
fh.write(b"link-basis-content\n")
|
||||||
|
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
|
||||||
rel = os.path.abspath(source).lstrip(os.sep)
|
rel = os.path.abspath(source).lstrip(os.sep)
|
||||||
basis = os.path.join(dest, "basis", rel)
|
basis = os.path.join(dest, "basis", rel)
|
||||||
os.makedirs(basis, exist_ok=True)
|
os.makedirs(basis, exist_ok=True)
|
||||||
basis_file = os.path.join(basis, "f.txt")
|
basis_file = os.path.join(basis, "f.txt")
|
||||||
with open(basis_file, "wb") as fh:
|
with open(basis_file, "wb") as fh:
|
||||||
fh.write(b"link-basis-content\n")
|
fh.write(b"link-basis-content\n")
|
||||||
|
os.utime(basis_file, (OLD, OLD))
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
result, _ = run_client(source, dest,
|
result, _ = run_client(source, dest,
|
||||||
flags=["--link-dest=basis", "--incremental"],
|
flags=["--link-dest=basis", "--incremental"],
|
||||||
@@ -771,12 +780,11 @@ class TestVerifyAndFlip:
|
|||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
def test_basis_dir_size_only_content_residual(self, shared_server):
|
def test_basis_dir_size_only_content_residual(self, shared_server):
|
||||||
"""Documented residual (RSYNC_COMPAT.md basis-dir rows): FastSync
|
"""rsync parity (default): a basis hit is decided by the metadata
|
||||||
xxHash-verifies a basis hit, while rsync's `--size-only` quick check
|
quick-check alone. With `--size-only`, a same-size, different-content
|
||||||
trusts the size alone. With a same-size, different-content basis,
|
basis is trusted, so rsync links the basis content and FastSync must now
|
||||||
rsync links/copies the wrong basis content while FastSync transfers the
|
do the same instead of xxHash-verifying it. `--verify-basis` restores
|
||||||
source. This test pins both observed behaviors (FastSync is stricter,
|
the stricter content equality (covered by the differential test)."""
|
||||||
so the rows are reclassified Divergent)."""
|
|
||||||
source = self._src("basissz")
|
source = self._src("basissz")
|
||||||
rdest = self._dst("basissz_r")
|
rdest = self._dst("basissz_r")
|
||||||
fdest = self._dst("basissz_f")
|
fdest = self._dst("basissz_f")
|
||||||
@@ -806,9 +814,123 @@ class TestVerifyAndFlip:
|
|||||||
flags=["-a", "--size-only", "--link-dest=basis", "--incremental"],
|
flags=["-a", "--size-only", "--link-dest=basis", "--incremental"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode == 0, result.stderr[:300]
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
|
with open(os.path.join(received, "f.txt"), "rb") as fh:
|
||||||
|
assert fh.read() == b"BBBB\n", \
|
||||||
|
"FastSync must trust the metadata quick-check exactly like rsync"
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
def test_verify_basis_restores_content_check(self, shared_server):
|
||||||
|
"""FastSync-only `--verify-basis`: a same-size, same-mtime basis with
|
||||||
|
different content is rejected by the whole-file digest, so the source is
|
||||||
|
transferred instead of installing the wrong basis bytes. The default
|
||||||
|
(no flag) installs the basis content, matching rsync."""
|
||||||
|
source = self._src("vbasis")
|
||||||
|
fdest = self._dst("vbasis_f")
|
||||||
|
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
||||||
|
fh.write(b"AAAA\n")
|
||||||
|
OLD = 1_400_000_000
|
||||||
|
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
|
||||||
|
rel = os.path.abspath(source).lstrip(os.sep)
|
||||||
|
basis = os.path.join(fdest, "basis", rel)
|
||||||
|
os.makedirs(basis, exist_ok=True)
|
||||||
|
with open(os.path.join(basis, "f.txt"), "wb") as fh:
|
||||||
|
fh.write(b"BBBB\n")
|
||||||
|
os.utime(os.path.join(basis, "f.txt"), (OLD, OLD))
|
||||||
|
received = get_dest_received_dir(fdest, source)
|
||||||
|
result, _ = run_client(source, fdest,
|
||||||
|
flags=["-a", "--link-dest=basis", "--incremental",
|
||||||
|
"--verify-basis"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
with open(os.path.join(received, "f.txt"), "rb") as fh:
|
with open(os.path.join(received, "f.txt"), "rb") as fh:
|
||||||
assert fh.read() == b"AAAA\n", \
|
assert fh.read() == b"AAAA\n", \
|
||||||
"FastSync must verify the basis content and transfer the source"
|
"--verify-basis must reject the same-size/different-content basis"
|
||||||
|
|
||||||
|
|
||||||
|
def _stat_bytes(output, key):
|
||||||
|
"""Parse a --stats byte counter (e.g. ``Matched data: 65,536 bytes``)."""
|
||||||
|
for line in output.splitlines():
|
||||||
|
if line.startswith(key + ":"):
|
||||||
|
raw = line.split(":", 1)[1].strip().split()[0]
|
||||||
|
return int(raw.replace(",", ""))
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class TestFuzzy:
|
||||||
|
"""Track 5b: `-y`/`--fuzzy` is an internal bandwidth optimization with a
|
||||||
|
byte-exact result. FastSync ports rsync 3.4.1's weighted-Levenshtein name
|
||||||
|
heuristic, so where both delta engines admit the candidate the tools pick
|
||||||
|
the same basis (the ``fuzzy_basis`` differential asserts the tree and the
|
||||||
|
Matched/Literal counters match with the block size pinned). The residual is
|
||||||
|
candidate ELIGIBILITY: FastSync's delta size gate (both files >= 16 KiB and
|
||||||
|
a <= 10x size ratio) is narrower than rsync's, which empirically uses a
|
||||||
|
fuzzy basis well beyond 10x and below 16 KiB. These tests pin the window
|
||||||
|
boundary and prove the byte-exact fallback on both sides of it."""
|
||||||
|
|
||||||
|
_BASE = b"the quick brown fox jumps over the lazy dog\n" * 4000
|
||||||
|
|
||||||
|
def _src(self, tag):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, f"fz_{tag}_src")
|
||||||
|
clean_dir(source)
|
||||||
|
return source
|
||||||
|
|
||||||
|
def _dst(self, tag):
|
||||||
|
d = os.path.join(TEST_DATA_DIR, f"fz_{tag}_dst")
|
||||||
|
clean_dir(d)
|
||||||
|
return d
|
||||||
|
|
||||||
|
def _run_both(self, shared_server, source, dest, rdst, payload, sibling,
|
||||||
|
rs_extra=(), fs_extra=()):
|
||||||
|
with open(os.path.join(source, "report_v2.txt"), "wb") as fh:
|
||||||
|
fh.write(payload)
|
||||||
|
for root in (rdst, get_dest_received_dir(dest, source)):
|
||||||
|
os.makedirs(root, exist_ok=True)
|
||||||
|
with open(os.path.join(root, "report_v1.txt"), "wb") as fh:
|
||||||
|
fh.write(sibling)
|
||||||
|
rs = _rsync(["-a", "--no-whole-file", "--fuzzy", "--stats"] +
|
||||||
|
list(rs_extra) + [source + "/", rdst + "/"])
|
||||||
|
assert rs.returncode == 0, rs.stderr[:300]
|
||||||
|
result, _ = run_client(
|
||||||
|
source, dest,
|
||||||
|
flags=["-a", "--incremental", "--delta", "--fuzzy", "--stats"] +
|
||||||
|
list(fs_extra),
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
|
_assert_same_tree(rdst, get_dest_received_dir(dest, source), "(--fuzzy)")
|
||||||
|
return rs, result
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
def test_fuzzy_above_size_window_declines_but_tree_exact(self, shared_server):
|
||||||
|
"""A sibling >10x the source is used by rsync but declined by FastSync's
|
||||||
|
delta size-ratio gate; both destinations stay byte-identical."""
|
||||||
|
n = 65536
|
||||||
|
payload = (self._BASE * ((n // len(self._BASE)) + 1))[:n]
|
||||||
|
sibling = (self._BASE * 200)[: n * 20]
|
||||||
|
source, dest, rdst = (self._src("big"), self._dst("big"),
|
||||||
|
self._dst("big_r"))
|
||||||
|
rs, result = self._run_both(shared_server, source, dest, rdst,
|
||||||
|
payload, sibling)
|
||||||
|
assert _stat_bytes(rs.stdout, "Matched data") > 0, \
|
||||||
|
"rsync should still use a >10x fuzzy basis"
|
||||||
|
assert _stat_bytes(result.stdout, "Matched data") == 0, \
|
||||||
|
"FastSync's 10x delta size-ratio gate must decline the oversized basis"
|
||||||
|
assert _stat_bytes(result.stdout, "Literal data") == n
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
def test_fuzzy_below_delta_minimum_declines_but_tree_exact(self, shared_server):
|
||||||
|
"""A sibling below the 16 KiB delta minimum is used by rsync but never
|
||||||
|
enters FastSync's delta/fuzzy path; both trees stay byte-identical."""
|
||||||
|
n = 8192
|
||||||
|
payload = (self._BASE * ((n // len(self._BASE)) + 1))[:n]
|
||||||
|
source, dest, rdst = (self._src("small"), self._dst("small"),
|
||||||
|
self._dst("small_r"))
|
||||||
|
rs, result = self._run_both(shared_server, source, dest, rdst,
|
||||||
|
payload, payload)
|
||||||
|
assert _stat_bytes(rs.stdout, "Matched data") > 0, \
|
||||||
|
"rsync applies --fuzzy below 16 KiB"
|
||||||
|
assert _stat_bytes(result.stdout, "Matched data") == 0, \
|
||||||
|
"FastSync's 16 KiB delta minimum must bypass the fuzzy basis"
|
||||||
|
assert _stat_bytes(result.stdout, "Literal data") == n
|
||||||
|
|
||||||
|
|
||||||
class TestIgnoreExistingShortCircuit:
|
class TestIgnoreExistingShortCircuit:
|
||||||
|
|||||||
@@ -94,14 +94,14 @@ def _seed_protocol_source(source):
|
|||||||
class TestProtocol:
|
class TestProtocol:
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_protocol_current_version_accepted(self, shared_server):
|
def test_protocol_current_version_accepted(self, shared_server):
|
||||||
"""--protocol=2.27.0 (the current PROTOCOL_VERSION) is accepted and the
|
"""--protocol=2.28.0 (the current PROTOCOL_VERSION) is accepted and the
|
||||||
transfer completes normally."""
|
transfer completes normally."""
|
||||||
source = os.path.join(TEST_DATA_DIR, "proto_ok_src")
|
source = os.path.join(TEST_DATA_DIR, "proto_ok_src")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "proto_ok_dst")
|
dest = os.path.join(TEST_DATA_DIR, "proto_ok_dst")
|
||||||
shutil.rmtree(dest, ignore_errors=True)
|
shutil.rmtree(dest, ignore_errors=True)
|
||||||
os.makedirs(dest)
|
os.makedirs(dest)
|
||||||
_seed_protocol_source(source)
|
_seed_protocol_source(source)
|
||||||
result, _ = run_client(source, dest, flags=["--protocol=2.27.0"],
|
result, _ = run_client(source, dest, flags=["--protocol=2.28.0"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode == 0, \
|
assert result.returncode == 0, \
|
||||||
f"--protocol current run failed: {(result.stderr or result.stdout)[:400]}"
|
f"--protocol current run failed: {(result.stderr or result.stdout)[:400]}"
|
||||||
@@ -118,8 +118,8 @@ class TestProtocol:
|
|||||||
shutil.rmtree(dest, ignore_errors=True)
|
shutil.rmtree(dest, ignore_errors=True)
|
||||||
os.makedirs(dest)
|
os.makedirs(dest)
|
||||||
_seed_protocol_source(source)
|
_seed_protocol_source(source)
|
||||||
for bad in ("2.22.0", "2.21.0", "2.20.0", "2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0",
|
for bad in ("2.27.0", "2.26.0", "2.25.0", "2.24.0", "2.23.0", "2.22.0", "2.21.0", "2.20.0",
|
||||||
"216", "31"):
|
"2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0", "216", "31"):
|
||||||
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
|
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode != 0, f"--protocol={bad} should be rejected"
|
assert result.returncode != 0, f"--protocol={bad} should be rejected"
|
||||||
|
|||||||
@@ -43,6 +43,7 @@
|
|||||||
#include "test_utils.h"
|
#include "test_utils.h"
|
||||||
#include "test_xattr.h"
|
#include "test_xattr.h"
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
#include <signal.h>
|
#include <signal.h>
|
||||||
|
|
||||||
// Define global test state variables
|
// Define global test state variables
|
||||||
@@ -52,6 +53,11 @@ bool current_test_failed = false;
|
|||||||
|
|
||||||
int main() {
|
int main() {
|
||||||
signal(SIGPIPE, SIG_IGN);
|
signal(SIGPIPE, SIG_IGN);
|
||||||
|
/* The codec/checksum resolvers consult rsync's preference-list environment
|
||||||
|
* variables; clear them so a developer's shell cannot change test outcomes.
|
||||||
|
* The env-specific tests set and restore their own values. */
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
unsetenv("RSYNC_CHECKSUM_LIST");
|
||||||
printf("\033[1;36m=== RUNNING UNIT TESTS ===\033[0m\n\n");
|
printf("\033[1;36m=== RUNNING UNIT TESTS ===\033[0m\n\n");
|
||||||
|
|
||||||
RUN_TEST(test_queue);
|
RUN_TEST(test_queue);
|
||||||
|
|||||||
@@ -265,6 +265,28 @@ static void test_checksum_digest_file_matches_oneshot(void) {
|
|||||||
free(data);
|
free(data);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* RSYNC_CHECKSUM_LIST precedence, syntax and fallback. */
|
||||||
|
static void test_checksum_choice_env_list() {
|
||||||
|
unsetenv("RSYNC_CHECKSUM_LIST");
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128);
|
||||||
|
EXPECT_EQ_INT((int)checksum_negotiate_default(), (int)CHECKSUM_ALGO_XXH128);
|
||||||
|
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "bogus md5 xxh3", 1);
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_MD5);
|
||||||
|
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "SHA1", 1);
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_SHA1);
|
||||||
|
|
||||||
|
/* Whitespace-separated only: comma is not a separator in rsync's syntax. */
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "md5,xxh3", 1);
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), -1);
|
||||||
|
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", " ", 1);
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128);
|
||||||
|
|
||||||
|
unsetenv("RSYNC_CHECKSUM_LIST");
|
||||||
|
}
|
||||||
|
|
||||||
void test_checksum(void) {
|
void test_checksum(void) {
|
||||||
test_checksum_xxh64_seed0();
|
test_checksum_xxh64_seed0();
|
||||||
test_checksum_xxh64_empty();
|
test_checksum_xxh64_empty();
|
||||||
@@ -281,4 +303,5 @@ void test_checksum(void) {
|
|||||||
test_checksum_truncated_buffer_rejected();
|
test_checksum_truncated_buffer_rejected();
|
||||||
test_checksum_null_empty_digest();
|
test_checksum_null_empty_digest();
|
||||||
test_checksum_digest_file_matches_oneshot();
|
test_checksum_digest_file_matches_oneshot();
|
||||||
|
test_checksum_choice_env_list();
|
||||||
}
|
}
|
||||||
|
|||||||
+218
-2
@@ -318,7 +318,7 @@ static void test_parse_args_protocol_accept_current() {
|
|||||||
Config* cfg = valid_client_config();
|
Config* cfg = valid_client_config();
|
||||||
EXPECT_NOT_NULL(cfg);
|
EXPECT_NOT_NULL(cfg);
|
||||||
char* argv_equals[] = {"fastsync", "--source-dir", "/src",
|
char* argv_equals[] = {"fastsync", "--source-dir", "/src",
|
||||||
"--dest-dir", "/dst", "--protocol=2.27.0"};
|
"--dest-dir", "/dst", "--protocol=2.28.0"};
|
||||||
int positional_args[2];
|
int positional_args[2];
|
||||||
int positional_count = 0;
|
int positional_count = 0;
|
||||||
EXPECT_EQ_INT(parse_args(cfg, 6, argv_equals, positional_args, &positional_count), 0);
|
EXPECT_EQ_INT(parse_args(cfg, 6, argv_equals, positional_args, &positional_count), 0);
|
||||||
@@ -328,7 +328,7 @@ static void test_parse_args_protocol_accept_current() {
|
|||||||
cfg = valid_client_config();
|
cfg = valid_client_config();
|
||||||
EXPECT_NOT_NULL(cfg);
|
EXPECT_NOT_NULL(cfg);
|
||||||
char* argv_space[] = {"fastsync", "--source-dir", "/src", "--dest-dir",
|
char* argv_space[] = {"fastsync", "--source-dir", "/src", "--dest-dir",
|
||||||
"/dst", "--protocol", "2.27.0"};
|
"/dst", "--protocol", "2.28.0"};
|
||||||
positional_count = 0;
|
positional_count = 0;
|
||||||
EXPECT_EQ_INT(parse_args(cfg, 7, argv_space, positional_args, &positional_count), 0);
|
EXPECT_EQ_INT(parse_args(cfg, 7, argv_space, positional_args, &positional_count), 0);
|
||||||
EXPECT_EQ_STR(cfg->version, PROTOCOL_VERSION);
|
EXPECT_EQ_STR(cfg->version, PROTOCOL_VERSION);
|
||||||
@@ -340,6 +340,7 @@ static void test_parse_args_protocol_accept_current() {
|
|||||||
static void test_parse_args_protocol_rejects_other_versions() {
|
static void test_parse_args_protocol_rejects_other_versions() {
|
||||||
static const char* const bad_versions[] = {"2.17", "2.16", "2.15.0", "2.16.0", "2.17.0",
|
static const char* const bad_versions[] = {"2.17", "2.16", "2.15.0", "2.16.0", "2.17.0",
|
||||||
"2.18.0", "2.19.0", "2.20.0", "2.21.0", "2.22.0",
|
"2.18.0", "2.19.0", "2.20.0", "2.21.0", "2.22.0",
|
||||||
|
"2.23.0", "2.24.0", "2.25.0", "2.26.0", "2.27.0",
|
||||||
"216", "31", "abc", ""};
|
"216", "31", "abc", ""};
|
||||||
for (size_t i = 0; i < sizeof(bad_versions) / sizeof(bad_versions[0]); i++) {
|
for (size_t i = 0; i < sizeof(bad_versions) / sizeof(bad_versions[0]); i++) {
|
||||||
Config* cfg = valid_client_config();
|
Config* cfg = valid_client_config();
|
||||||
@@ -1045,6 +1046,23 @@ static void test_parse_args_basis_dirs() {
|
|||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* --verify-basis (FastSync-only, long-only): default off; parses on as a plain
|
||||||
|
boolean and leaves the basis implications intact. */
|
||||||
|
static void test_parse_args_verify_basis() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
EXPECT_FALSE(cfg->verify_basis);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
cfg = config_create();
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
char* argv[] = {"fastsync", "--link-dest=prior", "--verify-basis", "/src", "/dst"};
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->verify_basis);
|
||||||
|
EXPECT_TRUE(config_has_basis(cfg));
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
/* Escaping or degenerate basis-dir values must be rejected up front (they would
|
/* Escaping or degenerate basis-dir values must be rejected up front (they would
|
||||||
resolve outside the destination root on the receiver); an absolute path is
|
resolve outside the destination root on the receiver); an absolute path is
|
||||||
accepted (rsync parity) and canonicalized with its leading '/' preserved. */
|
accepted (rsync parity) and canonicalized with its leading '/' preserved. */
|
||||||
@@ -1132,6 +1150,63 @@ static void test_parse_args_delete_timing_flags() {
|
|||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Plain --delete with no explicit timing defaults to delete-during, matching
|
||||||
|
* rsync's --del default (progressive deletion). --delete-commit is the
|
||||||
|
* FastSync-only long spelling that selects rsync's --delete-after timing (the
|
||||||
|
* late whole-tree commit), and an explicit timing always wins over the default.
|
||||||
|
*/
|
||||||
|
static void test_parse_args_delete_default_timing_and_commit() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* argv[] = {"fastsync", "--delete", "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->use_delete);
|
||||||
|
EXPECT_TRUE(cfg->delete_during);
|
||||||
|
EXPECT_FALSE(cfg->delete_before);
|
||||||
|
EXPECT_FALSE(cfg->delete_delay);
|
||||||
|
EXPECT_FALSE(cfg->delete_after);
|
||||||
|
cfg->send_directory = str_dup("/src");
|
||||||
|
cfg->receive_root_directory = str_dup("/dst");
|
||||||
|
EXPECT_TRUE(validate_config(cfg));
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* --delete-commit selects the late whole-tree commit (delete_after) and
|
||||||
|
implies --delete. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* argv_commit[] = {"fastsync", "--delete-commit", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv_commit, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->use_delete);
|
||||||
|
EXPECT_TRUE(cfg->delete_after);
|
||||||
|
EXPECT_FALSE(cfg->delete_before);
|
||||||
|
EXPECT_FALSE(cfg->delete_during);
|
||||||
|
EXPECT_FALSE(cfg->delete_delay);
|
||||||
|
cfg->send_directory = str_dup("/src");
|
||||||
|
cfg->receive_root_directory = str_dup("/dst");
|
||||||
|
EXPECT_TRUE(validate_config(cfg));
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* An explicit --delete-after alongside plain --delete keeps the late timing:
|
||||||
|
the default never overwrites an explicit timing. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* argv_after[] = {"fastsync", "--delete", "--delete-after", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, argv_after, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->use_delete);
|
||||||
|
EXPECT_TRUE(cfg->delete_after);
|
||||||
|
EXPECT_FALSE(cfg->delete_during);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* --delete-commit conflicts with a different timing. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* argv_conflict[] = {"fastsync", "--delete-commit", "--delete-during", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, argv_conflict, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_FALSE(validate_config(cfg));
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
/* Two different delete-timing flags on one command line are a conflict, not a
|
/* Two different delete-timing flags on one command line are a conflict, not a
|
||||||
* silent last-one-wins choice. */
|
* silent last-one-wins choice. */
|
||||||
static void test_parse_args_delete_timing_conflict_rejected() {
|
static void test_parse_args_delete_timing_conflict_rejected() {
|
||||||
@@ -1393,6 +1468,29 @@ static void test_parse_args_debug_info_levels() {
|
|||||||
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_STATS);
|
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_STATS);
|
||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* --info=name level 2 enables the "is uptodate" marker; a later level-1 or
|
||||||
|
level-0 token clears it again. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* name2_argv[] = {"fastsync", "--info=name2", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, name2_argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_NAME | LOG_INFO_NAME_UPTODATE);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
cfg = config_create();
|
||||||
|
char* name1_argv[] = {"fastsync", "--info=name2,name1", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, name1_argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_NAME);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
cfg = config_create();
|
||||||
|
char* name0_argv[] = {"fastsync", "--info=name2,name0", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, name0_argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->info_level, 0);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
cfg = config_create();
|
cfg = config_create();
|
||||||
char* bad_argv[] = {"fastsync", "--debug=123", "/src", "/dst"};
|
char* bad_argv[] = {"fastsync", "--debug=123", "/src", "/dst"};
|
||||||
positional_count = 0;
|
positional_count = 0;
|
||||||
@@ -2408,6 +2506,119 @@ static void test_parse_args_rejects_invalid_compression_choice() {
|
|||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* rsync gives each codec its own default --compress-level; an omitted level
|
||||||
|
* resolves to that default and an explicit one is clamped to the codec range. */
|
||||||
|
static void test_parse_args_per_codec_compression_level_defaults() {
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
struct {
|
||||||
|
const char* choice;
|
||||||
|
int level;
|
||||||
|
} cases[] = {
|
||||||
|
{"zstd", 3},
|
||||||
|
{"zlib", 6},
|
||||||
|
{"zlibx", 6},
|
||||||
|
{"lz4", 1},
|
||||||
|
};
|
||||||
|
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* argv[] = {"fastsync", "-z", "--compress-choice", (char*)cases[i].choice, "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 6, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->use_compression);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, cases[i].level);
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Bare -z resolves to the zstd default. */
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* bare[] = {"fastsync", "-z", "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, bare, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, 3);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* An explicit level wins unchanged for zstd... */
|
||||||
|
cfg = config_create();
|
||||||
|
char* zv[] = {"fastsync", "-z", "--compress-level", "10", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, zv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, 10);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* ...but zlib clamps an over-range level to 9 like rsync. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* zc[] = {"fastsync", "-z", "--compress-choice", "zlib", "--compress-level", "15",
|
||||||
|
"/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 8, zc, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, 9);
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* RSYNC_COMPRESS_LIST drives the bare -z ("auto") resolution. */
|
||||||
|
static void test_parse_args_compression_env_list() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* argv[] = {"fastsync", "-z", "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "zlib lz4", 1);
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZLIB);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, 6);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* An explicit --compress-choice beats the env list. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* explicit_argv[] = {"fastsync", "-z", "--compress-choice", "zstd", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 6, explicit_argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* A list with no supported name is rsync's failed negotiation (exit 4). */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "bogus", 1);
|
||||||
|
cfg = config_create();
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
|
||||||
|
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
|
||||||
|
config_delete(cfg);
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
}
|
||||||
|
|
||||||
|
/* RSYNC_CHECKSUM_LIST drives the default checksum choice. */
|
||||||
|
static void test_parse_args_checksum_env_list() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* argv[] = {"fastsync", "--checksum", "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "md5", 1);
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD5);
|
||||||
|
EXPECT_EQ_INT(cfg->checksum_transfer_algo, (int)CHECKSUM_ALGO_MD5);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* An explicit --cc wins. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* cc_argv[] = {"fastsync", "--checksum", "--cc=sha1", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, cc_argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_SHA1);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* A list with no supported name is rsync's failed negotiation (exit 4). */
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "bogus", 1);
|
||||||
|
cfg = config_create();
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
|
||||||
|
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
|
||||||
|
config_delete(cfg);
|
||||||
|
unsetenv("RSYNC_CHECKSUM_LIST");
|
||||||
|
}
|
||||||
|
|
||||||
/* Every value-taking table option accepts an inline "--opt=value" form. */
|
/* Every value-taking table option accepts an inline "--opt=value" form. */
|
||||||
static void test_parse_args_table_equals_size_options() {
|
static void test_parse_args_table_equals_size_options() {
|
||||||
Config* cfg = config_create();
|
Config* cfg = config_create();
|
||||||
@@ -4604,6 +4815,7 @@ void test_client_cli() {
|
|||||||
test_parse_args_relative_no_implied_mkpath();
|
test_parse_args_relative_no_implied_mkpath();
|
||||||
test_parse_args_delete_during_alias();
|
test_parse_args_delete_during_alias();
|
||||||
test_parse_args_delete_timing_flags();
|
test_parse_args_delete_timing_flags();
|
||||||
|
test_parse_args_delete_default_timing_and_commit();
|
||||||
test_parse_args_delete_timing_conflict_rejected();
|
test_parse_args_delete_timing_conflict_rejected();
|
||||||
test_parse_args_delete_timing_without_delete_rejected();
|
test_parse_args_delete_timing_without_delete_rejected();
|
||||||
test_parse_args_rejects_unimplemented_options();
|
test_parse_args_rejects_unimplemented_options();
|
||||||
@@ -4659,6 +4871,9 @@ void test_client_cli() {
|
|||||||
test_parse_args_compression_alias_equals();
|
test_parse_args_compression_alias_equals();
|
||||||
test_parse_args_rejects_invalid_compression_level_equals();
|
test_parse_args_rejects_invalid_compression_level_equals();
|
||||||
test_parse_args_rejects_invalid_compression_choice();
|
test_parse_args_rejects_invalid_compression_choice();
|
||||||
|
test_parse_args_per_codec_compression_level_defaults();
|
||||||
|
test_parse_args_compression_env_list();
|
||||||
|
test_parse_args_checksum_env_list();
|
||||||
test_parse_args_table_equals_size_options();
|
test_parse_args_table_equals_size_options();
|
||||||
test_parse_args_table_equals_string_and_int_options();
|
test_parse_args_table_equals_string_and_int_options();
|
||||||
test_parse_args_missing_argument_diagnostic();
|
test_parse_args_missing_argument_diagnostic();
|
||||||
@@ -4692,6 +4907,7 @@ void test_client_cli() {
|
|||||||
test_parse_args_filter_rules();
|
test_parse_args_filter_rules();
|
||||||
test_parse_args_from0_cvs_filter_file_flags();
|
test_parse_args_from0_cvs_filter_file_flags();
|
||||||
test_parse_args_basis_dirs();
|
test_parse_args_basis_dirs();
|
||||||
|
test_parse_args_verify_basis();
|
||||||
test_parse_args_basis_invalid_paths();
|
test_parse_args_basis_invalid_paths();
|
||||||
test_validate_config_basis_rejects_chunk_serialization();
|
test_validate_config_basis_rejects_chunk_serialization();
|
||||||
test_parse_args_delete_policy_flags();
|
test_parse_args_delete_policy_flags();
|
||||||
|
|||||||
@@ -4,6 +4,7 @@
|
|||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
#include <sys/wait.h>
|
#include <sys/wait.h>
|
||||||
@@ -390,6 +391,56 @@ static void test_codec_name_mapping() {
|
|||||||
EXPECT_TRUE(compression_algo_enabled(COMPRESSION_ALGO_ZSTD));
|
EXPECT_TRUE(compression_algo_enabled(COMPRESSION_ALGO_ZSTD));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* rsync 3.4.1's per-codec default levels and its clamping ranges. */
|
||||||
|
static void test_codec_level_defaults_and_clamp() {
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), ZSTD_CLEVEL_DEFAULT);
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), 3);
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIB), 6);
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIBX), 6);
|
||||||
|
/* lz4 has no tunable level; a positive placeholder keeps the codec engaged. */
|
||||||
|
EXPECT_TRUE(compression_default_level(COMPRESSION_ALGO_LZ4) > 0);
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_NONE), 0);
|
||||||
|
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 1), 1);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 22), 22);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 23), 22);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 0), 1);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 15), 9);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIBX, 15), 9);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 1), 1);
|
||||||
|
EXPECT_TRUE(compression_clamp_level(COMPRESSION_ALGO_LZ4, 20) > 0);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_NONE, 20), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* RSYNC_COMPRESS_LIST precedence, syntax and fallback. */
|
||||||
|
static void test_codec_choice_env_list() {
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
EXPECT_EQ_INT((int)compression_negotiate_default(), (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
|
||||||
|
/* Unknown entries are skipped; the first supported wins. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "bogus zlib lz4", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZLIB);
|
||||||
|
|
||||||
|
/* Case-insensitive. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "ZSTD", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
|
||||||
|
/* Whitespace-separated; the client half ends at '&'. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "lz4 zlib & zstd", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_LZ4);
|
||||||
|
|
||||||
|
/* Blank falls back to the compiled-in order. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", " ", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
|
||||||
|
/* rsync's syntax has no comma/colon separator: this is one unknown name. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "bogus,lz4", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), -1);
|
||||||
|
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
}
|
||||||
|
|
||||||
/* The process-global codec selects what the legacy wrappers produce. */
|
/* The process-global codec selects what the legacy wrappers produce. */
|
||||||
static void test_codec_global_selection() {
|
static void test_codec_global_selection() {
|
||||||
Data* original = data_create_empty(64);
|
Data* original = data_create_empty(64);
|
||||||
@@ -423,5 +474,7 @@ void test_compression() {
|
|||||||
test_chunk_compress_decompress_roundtrip();
|
test_chunk_compress_decompress_roundtrip();
|
||||||
test_codec_roundtrips();
|
test_codec_roundtrips();
|
||||||
test_codec_name_mapping();
|
test_codec_name_mapping();
|
||||||
|
test_codec_level_defaults_and_clamp();
|
||||||
|
test_codec_choice_env_list();
|
||||||
test_codec_global_selection();
|
test_codec_global_selection();
|
||||||
}
|
}
|
||||||
|
|||||||
+106
-7
@@ -1,6 +1,7 @@
|
|||||||
#include "test_config.h"
|
#include "test_config.h"
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
|
#include "filter.h"
|
||||||
#include "identity.h"
|
#include "identity.h"
|
||||||
#include "multiprocessing.h"
|
#include "multiprocessing.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
@@ -2119,6 +2120,48 @@ static void test_config_receive_rejects_oversized_string_budget() {
|
|||||||
config_delete(over_bytes);
|
config_delete(over_bytes);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Pre-auth bounds for the receiver-side filter rule block (protocol 2.28.0).
|
||||||
|
A peer may send `protect`/`risk` rules; the receiver must reject an over-cap
|
||||||
|
count or an over-long pattern before evaluating anything, so a crafted config
|
||||||
|
cannot drive unbounded glob work or install a rule that silently never
|
||||||
|
matches. */
|
||||||
|
static void test_config_receive_rejects_bad_protect_rules() {
|
||||||
|
if (is_running_under_valgrind())
|
||||||
|
return;
|
||||||
|
|
||||||
|
/* Over-cap rule count: one more than MAX_FILTER_RULES rules. */
|
||||||
|
Config* c = config_create();
|
||||||
|
EXPECT_NOT_NULL(c);
|
||||||
|
c->send_directory = str_dup("/src");
|
||||||
|
c->receive_root_directory = str_dup("/dst");
|
||||||
|
c->filters = array_list_create(free);
|
||||||
|
EXPECT_NOT_NULL(c->filters);
|
||||||
|
for (int i = 0; i <= MAX_FILTER_RULES; i++)
|
||||||
|
EXPECT_TRUE(array_list_add(c->filters, str_dup("- *.tmp")));
|
||||||
|
EXPECT_TRUE(roundtrip_config_rejected(c));
|
||||||
|
config_delete(c);
|
||||||
|
|
||||||
|
/* A pattern longer than the receiver's evaluation bound is rejected by the
|
||||||
|
sender (mirroring the receiver's guard) instead of being sent as an inert
|
||||||
|
rule. */
|
||||||
|
c = config_create();
|
||||||
|
EXPECT_NOT_NULL(c);
|
||||||
|
c->send_directory = str_dup("/src");
|
||||||
|
c->receive_root_directory = str_dup("/dst");
|
||||||
|
c->filters = array_list_create(free);
|
||||||
|
EXPECT_NOT_NULL(c->filters);
|
||||||
|
size_t big = MAX_PROTECT_PATTERN_LEN + 1;
|
||||||
|
char* long_rule = malloc(big + 3);
|
||||||
|
EXPECT_NOT_NULL(long_rule);
|
||||||
|
long_rule[0] = '-';
|
||||||
|
long_rule[1] = ' ';
|
||||||
|
memset(long_rule + 2, 'x', big);
|
||||||
|
long_rule[big + 2] = '\0';
|
||||||
|
EXPECT_TRUE(array_list_add(c->filters, long_rule));
|
||||||
|
EXPECT_TRUE(roundtrip_config_rejected(c));
|
||||||
|
config_delete(c);
|
||||||
|
}
|
||||||
|
|
||||||
/* identity_copy_as_refused() is the pure, pre-snapshot refusal predicate: a
|
/* identity_copy_as_refused() is the pure, pre-snapshot refusal predicate: a
|
||||||
--copy-as is refused when the receiver is not root OR the effective super
|
--copy-as is refused when the receiver is not root OR the effective super
|
||||||
mode is OFF (an operator veto), and never when --copy-as is unset. */
|
mode is OFF (an operator veto), and never when --copy-as is unset. */
|
||||||
@@ -2590,6 +2633,46 @@ static bool basis_equal(const Config* a, const Config* b) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The receiver reconstructs its delete-protection list from the sender's
|
||||||
|
* compiled base rules, so compare the received list against a fresh
|
||||||
|
* filter_base_build() of the sender's raw --filter texts. */
|
||||||
|
static bool filter_rules_equal(const Config* a, const FilterRuleList* got) {
|
||||||
|
int count = a->filters ? a->filters->size : 0;
|
||||||
|
const char** texts = NULL;
|
||||||
|
if (count > 0) {
|
||||||
|
texts = calloc((size_t)count, sizeof(char*));
|
||||||
|
if (!texts)
|
||||||
|
return false;
|
||||||
|
for (int i = 0; i < count; i++)
|
||||||
|
texts[i] = (const char*)a->filters->items[i];
|
||||||
|
}
|
||||||
|
char err[160];
|
||||||
|
FilterRuleList* expected =
|
||||||
|
filter_base_build(texts, count, a->cvs_exclude, a->delete_excluded, err, sizeof(err));
|
||||||
|
free(texts);
|
||||||
|
if (!expected)
|
||||||
|
return false;
|
||||||
|
bool equal = true;
|
||||||
|
int want = expected->count;
|
||||||
|
int have = got ? got->count : 0;
|
||||||
|
if (want != have) {
|
||||||
|
equal = false;
|
||||||
|
} else {
|
||||||
|
for (int i = 0; i < want; i++) {
|
||||||
|
const FilterRule* x = expected->items[i];
|
||||||
|
const FilterRule* y = got->items[i];
|
||||||
|
if (x->action != y->action || x->sides != y->sides || x->anchored != y->anchored ||
|
||||||
|
x->dir_only != y->dir_only || x->negate != y->negate ||
|
||||||
|
!str_opt_equal(x->owner, y->owner) || !str_opt_equal(x->pattern, y->pattern)) {
|
||||||
|
equal = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
filter_rule_list_free(expected);
|
||||||
|
return equal;
|
||||||
|
}
|
||||||
|
|
||||||
#define CONFIG_CMP_BOOL(a, b, name) ((a)->name == (b)->name)
|
#define CONFIG_CMP_BOOL(a, b, name) ((a)->name == (b)->name)
|
||||||
#define CONFIG_CMP_INT(a, b, name) ((a)->name == (b)->name)
|
#define CONFIG_CMP_INT(a, b, name) ((a)->name == (b)->name)
|
||||||
#define CONFIG_CMP_RAW(a, b, name) ((a)->name == (b)->name)
|
#define CONFIG_CMP_RAW(a, b, name) ((a)->name == (b)->name)
|
||||||
@@ -2615,6 +2698,7 @@ static bool basis_equal(const Config* a, const Config* b) {
|
|||||||
#define CONFIG_CMP_COPY_AS_ID(a, b, name) (!(a)->copy_as_set || (a)->name == (b)->name)
|
#define CONFIG_CMP_COPY_AS_ID(a, b, name) (!(a)->copy_as_set || (a)->name == (b)->name)
|
||||||
#define CONFIG_CMP_BLOCK_SKIP_SUFFIXES(a, b, name) skip_suffixes_equal((a), (b))
|
#define CONFIG_CMP_BLOCK_SKIP_SUFFIXES(a, b, name) skip_suffixes_equal((a), (b))
|
||||||
#define CONFIG_CMP_BLOCK_BASIS(a, b, name) basis_equal((a), (b))
|
#define CONFIG_CMP_BLOCK_BASIS(a, b, name) basis_equal((a), (b))
|
||||||
|
#define CONFIG_CMP_BLOCK_PROTECT_RULES(a, b, name) filter_rules_equal((a), (b)->name)
|
||||||
#define CONFIG_CMP_BLOCK_IDMAP(a, b, name) \
|
#define CONFIG_CMP_BLOCK_IDMAP(a, b, name) \
|
||||||
idmap_equal((a)->name, (a)->name##_count, (b)->name, (b)->name##_count)
|
idmap_equal((a)->name, (a)->name##_count, (b)->name, (b)->name##_count)
|
||||||
|
|
||||||
@@ -2784,6 +2868,7 @@ static void golden_config_populate(Config* c) {
|
|||||||
c->skip_compress_suffixes[1] = str_dup(".xz");
|
c->skip_compress_suffixes[1] = str_dup(".xz");
|
||||||
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_COMPARE, "compare"), 0);
|
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_COMPARE, "compare"), 0);
|
||||||
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "link"), 0);
|
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "link"), 0);
|
||||||
|
c->verify_basis = true;
|
||||||
c->fuzzy = true;
|
c->fuzzy = true;
|
||||||
c->checksum_algo = CHECKSUM_ALGO_MD5;
|
c->checksum_algo = CHECKSUM_ALGO_MD5;
|
||||||
c->checksum_seed = 0x1122334455667788ULL;
|
c->checksum_seed = 0x1122334455667788ULL;
|
||||||
@@ -2829,17 +2914,29 @@ static void golden_config_populate(Config* c) {
|
|||||||
c->copy_as_set = true;
|
c->copy_as_set = true;
|
||||||
c->copy_as_uid = 111;
|
c->copy_as_uid = 111;
|
||||||
c->copy_as_gid = 222;
|
c->copy_as_gid = 222;
|
||||||
|
/* Compile-through delete-protection rules (protocol 2.28.0). The golden
|
||||||
|
* sender serializes its compiled base rules, so populate a diverse set that
|
||||||
|
* exercises both sides, negate, anchoring and dir-only. */
|
||||||
|
c->filters = array_list_create(free);
|
||||||
|
array_list_add(c->filters, str_dup("P *.log"));
|
||||||
|
array_list_add(c->filters, str_dup("+r **/*.txt"));
|
||||||
|
array_list_add(c->filters, str_dup("H,!secret"));
|
||||||
|
array_list_add(c->filters, str_dup("- /sub/dir/"));
|
||||||
}
|
}
|
||||||
|
|
||||||
/* The pinned golden frame (protocol 2.27.0). The values below are the only
|
/* The pinned golden frame (protocol 2.28.0). The values below are the only
|
||||||
* thing that ties the generated table to the historical wire format; update
|
* thing that ties the generated table to the historical wire format; update
|
||||||
* them ONLY with a PROTOCOL_VERSION bump and a documented reason. The 2.24.0
|
* them ONLY with a PROTOCOL_VERSION bump and a documented reason. The 2.24.0
|
||||||
* delete-plan wave changed only the version string; 2.25.0 appended the
|
* delete-plan wave changed only the version string; 2.25.0 appended the
|
||||||
* report_stats bool, 2.26.0 appended the compression_algo int, and 2.27.0
|
* report_stats bool, 2.26.0 appended the compression_algo int, 2.27.0 appended
|
||||||
* appended the report_deletes bool. The byte-exact values are recomputed for
|
* the report_deletes bool, and 2.28.0 changed only the version string and
|
||||||
* the merged layout. */
|
* appended the receiver-side delete-protection rule block (the STATUS_STATS
|
||||||
#define GOLDEN_WIRE_LEN 709
|
* body also grew, but that is not part of this frame). Track 5a appends the
|
||||||
#define GOLDEN_WIRE_HASH 14423869696887880000ULL
|
* FastSync-only verify_basis bool to the basis block WITHOUT a version bump
|
||||||
|
* (project decision), so the frame grew by one int to 886 bytes. The
|
||||||
|
* byte-exact values are recomputed for the merged layout. */
|
||||||
|
#define GOLDEN_WIRE_LEN 886
|
||||||
|
#define GOLDEN_WIRE_HASH 5809509022716816757ULL
|
||||||
|
|
||||||
static unsigned long long fnv1a_64(const unsigned char* buf, size_t len) {
|
static unsigned long long fnv1a_64(const unsigned char* buf, size_t len) {
|
||||||
unsigned long long h = 1469598103934665603ULL;
|
unsigned long long h = 1469598103934665603ULL;
|
||||||
@@ -2921,7 +3018,7 @@ static unsigned long long capture_wire_hash(const Config* cfg, size_t* out_len)
|
|||||||
return h;
|
return h;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Byte-for-byte wire compatibility guard (protocol 2.27.0). The expected hash
|
/* Byte-for-byte wire compatibility guard (protocol 2.28.0). The expected hash
|
||||||
* pins the pre-X-macro byte stream; the refactor MUST NOT change it. */
|
* pins the pre-X-macro byte stream; the refactor MUST NOT change it. */
|
||||||
static void test_config_wire_golden() {
|
static void test_config_wire_golden() {
|
||||||
if (is_running_under_valgrind())
|
if (is_running_under_valgrind())
|
||||||
@@ -2988,6 +3085,7 @@ static void test_config_wire_golden_receive() {
|
|||||||
recv->groupmap[0].to_name != NULL && strcmp(recv->groupmap[0].to_name, "root") == 0;
|
recv->groupmap[0].to_name != NULL && strcmp(recv->groupmap[0].to_name, "root") == 0;
|
||||||
ok = ok && recv->basis_count == 2 && recv->basis_dirs[0].type == BASIS_DEST_COMPARE &&
|
ok = ok && recv->basis_count == 2 && recv->basis_dirs[0].type == BASIS_DEST_COMPARE &&
|
||||||
recv->basis_dirs[1].type == BASIS_DEST_LINK;
|
recv->basis_dirs[1].type == BASIS_DEST_LINK;
|
||||||
|
ok = ok && recv->verify_basis;
|
||||||
ok = ok && recv->module != NULL && strcmp(recv->module, "goldenmod") == 0;
|
ok = ok && recv->module != NULL && strcmp(recv->module, "goldenmod") == 0;
|
||||||
ok = ok && recv->copy_as_set && recv->copy_as_uid == 111 && recv->copy_as_gid == 222;
|
ok = ok && recv->copy_as_set && recv->copy_as_uid == 111 && recv->copy_as_gid == 222;
|
||||||
}
|
}
|
||||||
@@ -3248,6 +3346,7 @@ void test_config() {
|
|||||||
test_config_wire_golden_receive();
|
test_config_wire_golden_receive();
|
||||||
test_config_wire_receive_bounds();
|
test_config_wire_receive_bounds();
|
||||||
test_config_receive_rejects_overcap_counts();
|
test_config_receive_rejects_overcap_counts();
|
||||||
|
test_config_receive_rejects_bad_protect_rules();
|
||||||
test_config_wire_roundtrip_all_fields();
|
test_config_wire_roundtrip_all_fields();
|
||||||
test_config_preserve_attribute_wire_roundtrip();
|
test_config_preserve_attribute_wire_roundtrip();
|
||||||
}
|
}
|
||||||
|
|||||||
+78
-29
@@ -17,16 +17,16 @@
|
|||||||
* caller/receiver entry point) describing `dir` with no kept children. */
|
* caller/receiver entry point) describing `dir` with no kept children. */
|
||||||
static void send_plan_frame(int fd, const char* dir) {
|
static void send_plan_frame(int fd, const char* dir) {
|
||||||
EXPECT_TRUE(send_int(fd, 0)); /* has_config */
|
EXPECT_TRUE(send_int(fd, 0)); /* has_config */
|
||||||
|
EXPECT_TRUE(send_int(fd, 1)); /* apply: a real plan */
|
||||||
EXPECT_TRUE(send_wire_str(fd, dir));
|
EXPECT_TRUE(send_wire_str(fd, dir));
|
||||||
EXPECT_TRUE(send_int(fd, 0)); /* kept child dirs */
|
EXPECT_TRUE(send_int(fd, 0)); /* kept child dirs */
|
||||||
EXPECT_TRUE(send_int(fd, 0)); /* kept child files */
|
EXPECT_TRUE(send_int(fd, 0)); /* kept child files */
|
||||||
}
|
}
|
||||||
|
|
||||||
/* --delete-delay: a directory snapshotted into the plan that is refilled before
|
/* --delete-delay: a directory snapshotted into the plan that is refilled before
|
||||||
* the commit must NOT be counted as deleted once its unlink fails ENOTEMPTY.
|
* the commit is re-scanned and removed recursively (rsync parity). Regression
|
||||||
* Regression for delete_plan.c counting at snapshot (defer_add) instead of at
|
* for the old single-unlink ENOTEMPTY path that left the directory behind. */
|
||||||
* the actual removal. */
|
static void test_delete_delay_refilled_dir_removed_recursively(void) {
|
||||||
static void test_delete_delay_refilled_dir_not_counted(void) {
|
|
||||||
char root[] = "/tmp/fastsync_dp_refill_XXXXXX";
|
char root[] = "/tmp/fastsync_dp_refill_XXXXXX";
|
||||||
EXPECT_TRUE(mkdtemp(root) != NULL);
|
EXPECT_TRUE(mkdtemp(root) != NULL);
|
||||||
char extra[1024];
|
char extra[1024];
|
||||||
@@ -57,16 +57,14 @@ static void test_delete_delay_refilled_dir_not_counted(void) {
|
|||||||
close(fd);
|
close(fd);
|
||||||
|
|
||||||
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_OK);
|
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_OK);
|
||||||
/* ENOTEMPTY: the directory survives, so it must not be reported as deleted. */
|
/* The late content and the directory itself are both removed. */
|
||||||
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
|
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 2);
|
||||||
struct stat st;
|
struct stat st;
|
||||||
EXPECT_EQ_INT(lstat(extra, &st), 0);
|
EXPECT_TRUE(lstat(extra, &st) != 0);
|
||||||
|
|
||||||
delete_plan_session_destroy(session);
|
delete_plan_session_destroy(session);
|
||||||
close(p[0]);
|
close(p[0]);
|
||||||
close(p[1]);
|
close(p[1]);
|
||||||
unlink(refill);
|
|
||||||
rmdir(extra);
|
|
||||||
rmdir(root);
|
rmdir(root);
|
||||||
config_delete(config);
|
config_delete(config);
|
||||||
}
|
}
|
||||||
@@ -106,8 +104,9 @@ static void test_delete_delay_removed_file_counted(void) {
|
|||||||
config_delete(config);
|
config_delete(config);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* --max-delete still bounds the deferred plan; the actual (removed) count must
|
/* --max-delete is charged on actual removals, not at plan/snapshot time: after
|
||||||
* not exceed the limit even though more extras existed. */
|
* receiving the plans the budget is untouched, and only the commit removes up to
|
||||||
|
* the limit. */
|
||||||
static void test_delete_delay_max_delete_bounds_actual(void) {
|
static void test_delete_delay_max_delete_bounds_actual(void) {
|
||||||
char root[] = "/tmp/fastsync_dp_max_XXXXXX";
|
char root[] = "/tmp/fastsync_dp_max_XXXXXX";
|
||||||
EXPECT_TRUE(mkdtemp(root) != NULL);
|
EXPECT_TRUE(mkdtemp(root) != NULL);
|
||||||
@@ -133,8 +132,11 @@ static void test_delete_delay_max_delete_bounds_actual(void) {
|
|||||||
EXPECT_NOT_NULL(session);
|
EXPECT_NOT_NULL(session);
|
||||||
send_plan_frame(p[1], ".");
|
send_plan_frame(p[1], ".");
|
||||||
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
|
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
|
||||||
EXPECT_TRUE(delete_plan_session_limit_reached(session));
|
/* Nothing removed yet, so the budget is not consumed at snapshot time. */
|
||||||
|
EXPECT_FALSE(delete_plan_session_limit_reached(session));
|
||||||
|
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
|
||||||
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
|
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
|
||||||
|
EXPECT_TRUE(delete_plan_session_limit_reached(session));
|
||||||
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
|
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
|
||||||
|
|
||||||
delete_plan_session_destroy(session);
|
delete_plan_session_destroy(session);
|
||||||
@@ -149,13 +151,12 @@ static void test_delete_delay_max_delete_bounds_actual(void) {
|
|||||||
config_delete(config);
|
config_delete(config);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* --max-delete is charged at plan/snapshot time, not at actual removal: a
|
/* --max-delete is charged on ACTUAL removals: the refilled directory's late
|
||||||
* deferred entry that survives ENOTEMPTY still consumes its budget slot, so a
|
* content is removed first (consuming the single budget slot), so the directory
|
||||||
* later directory's extra is skipped even though nothing was actually removed.
|
* itself and the later extra are skipped, matching rsync. The two plans are
|
||||||
* The reported count stays 0 (actual removals) while the run is partial. The
|
* sent as separate frames for "a" then "b", so the ordering that decides which
|
||||||
* two plans are sent as separate frames for "a" then "b", so the ordering that
|
* entry gets the budget is deterministic (unlike readdir order). */
|
||||||
* decides which entry gets the budget is deterministic (unlike readdir order). */
|
static void test_delete_delay_actual_removal_charges_budget(void) {
|
||||||
static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
|
|
||||||
char root[] = "/tmp/fastsync_dp_planbudget_XXXXXX";
|
char root[] = "/tmp/fastsync_dp_planbudget_XXXXXX";
|
||||||
EXPECT_TRUE(mkdtemp(root) != NULL);
|
EXPECT_TRUE(mkdtemp(root) != NULL);
|
||||||
char adir[1024], bdir[1024], xdir[1024], ydir[1024];
|
char adir[1024], bdir[1024], xdir[1024], ydir[1024];
|
||||||
@@ -180,18 +181,16 @@ static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
|
|||||||
|
|
||||||
DeletePlanSession* session = delete_plan_session_create(config);
|
DeletePlanSession* session = delete_plan_session_create(config);
|
||||||
EXPECT_NOT_NULL(session);
|
EXPECT_NOT_NULL(session);
|
||||||
/* Plan "a" first: its empty extra dir snapshots and charges the budget. */
|
/* Both plan snapshots are taken; neither consumes budget yet. */
|
||||||
send_plan_frame(p[1], "a");
|
send_plan_frame(p[1], "a");
|
||||||
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
|
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
|
||||||
EXPECT_FALSE(delete_plan_session_limit_reached(session));
|
EXPECT_FALSE(delete_plan_session_limit_reached(session));
|
||||||
/* Plan "b": the budget is already spent at snapshot time, so b/y is skipped
|
|
||||||
even though a/x has not (and will not) be removed. */
|
|
||||||
send_plan_frame(p[1], "b");
|
send_plan_frame(p[1], "b");
|
||||||
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
|
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
|
||||||
EXPECT_TRUE(delete_plan_session_limit_reached(session));
|
EXPECT_FALSE(delete_plan_session_limit_reached(session));
|
||||||
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
|
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
|
||||||
|
|
||||||
/* Refill a/x so its deferred rmdir fails ENOTEMPTY. */
|
/* Refill a/x after its plan: the recursive commit must remove this content. */
|
||||||
char refill[1200];
|
char refill[1200];
|
||||||
snprintf(refill, sizeof(refill), "%s/new.txt", xdir);
|
snprintf(refill, sizeof(refill), "%s/new.txt", xdir);
|
||||||
int fd = open(refill, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
int fd = open(refill, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
||||||
@@ -199,16 +198,17 @@ static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
|
|||||||
close(fd);
|
close(fd);
|
||||||
|
|
||||||
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
|
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
|
||||||
/* Nothing was actually removed, and the plan-time budget still stopped b/y. */
|
/* The one budget slot removed the late content; the two directories survive. */
|
||||||
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
|
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
|
||||||
|
EXPECT_TRUE(delete_plan_session_limit_reached(session));
|
||||||
struct stat st;
|
struct stat st;
|
||||||
|
EXPECT_TRUE(lstat(refill, &st) != 0);
|
||||||
EXPECT_EQ_INT(lstat(xdir, &st), 0);
|
EXPECT_EQ_INT(lstat(xdir, &st), 0);
|
||||||
EXPECT_EQ_INT(lstat(ydir, &st), 0);
|
EXPECT_EQ_INT(lstat(ydir, &st), 0);
|
||||||
|
|
||||||
delete_plan_session_destroy(session);
|
delete_plan_session_destroy(session);
|
||||||
close(p[0]);
|
close(p[0]);
|
||||||
close(p[1]);
|
close(p[1]);
|
||||||
unlink(refill);
|
|
||||||
rmdir(xdir);
|
rmdir(xdir);
|
||||||
rmdir(ydir);
|
rmdir(ydir);
|
||||||
rmdir(adir);
|
rmdir(adir);
|
||||||
@@ -217,9 +217,58 @@ static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
|
|||||||
config_delete(config);
|
config_delete(config);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Send a config-only carrier frame (apply=false): the per-run config block with
|
||||||
|
* one --delete-missing-args exact path, and no directory walk. */
|
||||||
|
static void send_config_only_frame(int fd, const char* missing_path) {
|
||||||
|
EXPECT_TRUE(send_int(fd, 1)); /* has_config */
|
||||||
|
EXPECT_TRUE(send_int(fd, 0)); /* protected prefixes */
|
||||||
|
EXPECT_TRUE(send_int(fd, 0)); /* size-skipped */
|
||||||
|
EXPECT_TRUE(send_int(fd, 1)); /* missing args */
|
||||||
|
EXPECT_TRUE(send_wire_str(fd, missing_path));
|
||||||
|
EXPECT_TRUE(send_int(fd, 0)); /* apply = false */
|
||||||
|
EXPECT_TRUE(send_wire_str(fd, "."));
|
||||||
|
EXPECT_TRUE(send_int(fd, 0));
|
||||||
|
EXPECT_TRUE(send_int(fd, 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The config-only carrier frame (apply=false) still applies the
|
||||||
|
* --delete-missing-args exact deletions even though it walks no directory. This
|
||||||
|
* is the fix for a --files-from list that synchronizes no directory. */
|
||||||
|
static void test_config_only_frame_applies_missing_args(void) {
|
||||||
|
char root[] = "/tmp/fastsync_dp_cfgonly_XXXXXX";
|
||||||
|
EXPECT_TRUE(mkdtemp(root) != NULL);
|
||||||
|
char gone[1024];
|
||||||
|
snprintf(gone, sizeof(gone), "%s/gone.txt", root);
|
||||||
|
int fd = open(gone, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
||||||
|
EXPECT_TRUE(fd >= 0);
|
||||||
|
close(fd);
|
||||||
|
|
||||||
|
Config* config = config_create();
|
||||||
|
EXPECT_NOT_NULL(config);
|
||||||
|
config->receive_root_directory = str_dup(root);
|
||||||
|
config->delete_missing_args = true;
|
||||||
|
|
||||||
|
int p[2];
|
||||||
|
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
|
||||||
|
|
||||||
|
DeletePlanSession* session = delete_plan_session_create(config);
|
||||||
|
EXPECT_NOT_NULL(session);
|
||||||
|
send_config_only_frame(p[1], "gone.txt");
|
||||||
|
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
|
||||||
|
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
|
||||||
|
EXPECT_TRUE(lstat(gone, &(struct stat){0}) != 0);
|
||||||
|
|
||||||
|
delete_plan_session_destroy(session);
|
||||||
|
close(p[0]);
|
||||||
|
close(p[1]);
|
||||||
|
rmdir(root);
|
||||||
|
config_delete(config);
|
||||||
|
}
|
||||||
|
|
||||||
void test_delete_plan(void) {
|
void test_delete_plan(void) {
|
||||||
test_delete_delay_refilled_dir_not_counted();
|
test_delete_delay_refilled_dir_removed_recursively();
|
||||||
test_delete_delay_removed_file_counted();
|
test_delete_delay_removed_file_counted();
|
||||||
test_delete_delay_max_delete_bounds_actual();
|
test_delete_delay_max_delete_bounds_actual();
|
||||||
test_delete_delay_refilled_dir_charges_budget_at_plan();
|
test_delete_delay_actual_removal_charges_budget();
|
||||||
|
test_config_only_frame_applies_missing_args();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,7 @@
|
|||||||
#include "file_receive.h"
|
#include "file_receive.h"
|
||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
|
#include "charset.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "test_utils.h"
|
#include "test_utils.h"
|
||||||
@@ -1807,6 +1808,159 @@ static void test_receive_incremental_check_empty_path() {
|
|||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Pure policy helpers behind the basis quick-check / --verify-basis decision. */
|
||||||
|
static void test_file_basis_quick_match_decision() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
EXPECT_NOT_NULL(cfg);
|
||||||
|
EXPECT_FALSE(file_basis_content_required(cfg));
|
||||||
|
cfg->verify_basis = true;
|
||||||
|
EXPECT_TRUE(file_basis_content_required(cfg));
|
||||||
|
cfg->verify_basis = false;
|
||||||
|
|
||||||
|
struct stat st;
|
||||||
|
memset(&st, 0, sizeof(st));
|
||||||
|
st.st_mtime = 1500000000;
|
||||||
|
#ifdef __linux__
|
||||||
|
st.st_mtim.tv_nsec = 500;
|
||||||
|
#endif
|
||||||
|
/* Equal size is required by the caller; this leg is the mtime / --size-only
|
||||||
|
rule. Equal mtime matches, a different mtime misses by default. */
|
||||||
|
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000000, 500));
|
||||||
|
EXPECT_FALSE(file_basis_quick_match(cfg, &st, 1500000001, 500));
|
||||||
|
cfg->size_only = true;
|
||||||
|
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000001, 500));
|
||||||
|
cfg->size_only = false;
|
||||||
|
cfg->modify_window = 2;
|
||||||
|
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000002, 500));
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* End-to-end handshake decision for a same-size, same-mtime, DIFFERENT-content
|
||||||
|
basis. Default (rsync parity): the metadata quick-check is trusted, the
|
||||||
|
receiver answers STATUS_OK and materializes the basis bytes. --verify-basis:
|
||||||
|
the whole-file digest is required, the basis is rejected and the receiver
|
||||||
|
asks for the source (STATUS_NEXT + full transfer). */
|
||||||
|
static void test_receive_incremental_check_basis_quick_check_and_verify() {
|
||||||
|
const char* root = "test_basis_quick_root";
|
||||||
|
const char* basis_dir = "test_basis_quick_root/basis";
|
||||||
|
const char* basis_file = "test_basis_quick_root/basis/f.txt";
|
||||||
|
unlink(basis_file);
|
||||||
|
unlink("test_basis_quick_root/f.txt");
|
||||||
|
rmdir(basis_dir);
|
||||||
|
rmdir(root);
|
||||||
|
EXPECT_EQ_INT(mkdir(root, 0755), 0);
|
||||||
|
EXPECT_EQ_INT(mkdir(basis_dir, 0755), 0);
|
||||||
|
|
||||||
|
const char* src_bytes = "AAAA";
|
||||||
|
const unsigned long long size = 4;
|
||||||
|
const time_t mtime = 1500000000;
|
||||||
|
{
|
||||||
|
FILE* fh = fopen(basis_file, "wb");
|
||||||
|
EXPECT_NOT_NULL(fh);
|
||||||
|
// cppcheck-suppress knownConditionTrueFalse
|
||||||
|
if (fh) {
|
||||||
|
EXPECT_EQ_INT((int)fwrite("BBBB", 1, (size_t)size, fh), (int)size);
|
||||||
|
fclose(fh);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
struct timespec ts[2] = {{mtime, 0}, {mtime, 0}};
|
||||||
|
EXPECT_EQ_INT(utimensat(AT_FDCWD, basis_file, ts, 0), 0);
|
||||||
|
|
||||||
|
char root_abs[PATH_MAX];
|
||||||
|
EXPECT_NOT_NULL(realpath(root, root_abs));
|
||||||
|
int root_fd = open(root_abs, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
|
||||||
|
EXPECT_TRUE(root_fd >= 0);
|
||||||
|
// cppcheck-suppress knownConditionTrueFalse
|
||||||
|
if (root_fd < 0) {
|
||||||
|
unlink(basis_file);
|
||||||
|
rmdir(basis_dir);
|
||||||
|
rmdir(root);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
EXPECT_TRUE(utils_set_authorized_root(root_fd, root_abs));
|
||||||
|
|
||||||
|
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
|
size_t digest_len = 0;
|
||||||
|
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, src_bytes, size, digest, sizeof(digest),
|
||||||
|
&digest_len));
|
||||||
|
|
||||||
|
/* Route protocol I/O through the explicit descriptors (a previous test group
|
||||||
|
may have left io_set_fds() bound to its own pipe). */
|
||||||
|
io_set_fds(-1, -1);
|
||||||
|
io_set_bwlimit(0);
|
||||||
|
|
||||||
|
for (int verify = 0; verify <= 1; verify++) {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
EXPECT_NOT_NULL(cfg);
|
||||||
|
cfg->receive_root_directory = str_dup(root_abs);
|
||||||
|
cfg->checksum = false;
|
||||||
|
cfg->checksum_algo = CHECKSUM_ALGO_XXH64;
|
||||||
|
cfg->checksum_seed = 0;
|
||||||
|
cfg->use_incremental = true;
|
||||||
|
cfg->use_delta = false;
|
||||||
|
cfg->use_metadata = false;
|
||||||
|
cfg->verify_basis = (verify != 0);
|
||||||
|
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_LINK, "basis"), 0);
|
||||||
|
|
||||||
|
int p[2];
|
||||||
|
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
|
||||||
|
EXPECT_TRUE(send_wire_str(p[1], "f.txt"));
|
||||||
|
unsigned long long check_size = size;
|
||||||
|
long long check_mtime = (long long)mtime;
|
||||||
|
long long check_mtime_nsec = 0;
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &check_size, sizeof(check_size)));
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &check_mtime, sizeof(check_mtime)));
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &check_mtime_nsec, sizeof(check_mtime_nsec)));
|
||||||
|
/* Only --verify-basis needs the digest (cfg->checksum is false) and the
|
||||||
|
pre-staged fallback full transfer the receiver will request. */
|
||||||
|
if (verify) {
|
||||||
|
uint8_t wire_len = (uint8_t)digest_len;
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
|
||||||
|
char* payload_bytes = str_dup(src_bytes);
|
||||||
|
EXPECT_NOT_NULL(payload_bytes);
|
||||||
|
Data* payload = data_create(payload_bytes, size);
|
||||||
|
EXPECT_NOT_NULL(payload);
|
||||||
|
// cppcheck-suppress knownConditionTrueFalse
|
||||||
|
if (payload)
|
||||||
|
EXPECT_TRUE(send_data(p[1], payload));
|
||||||
|
data_destroy(payload);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool skipped = false;
|
||||||
|
File* file = receive_incremental_check(p[0], cfg, &skipped);
|
||||||
|
EXPECT_NOT_NULL(file);
|
||||||
|
// cppcheck-suppress knownConditionTrueFalse
|
||||||
|
if (file) {
|
||||||
|
EXPECT_FALSE(skipped);
|
||||||
|
Status reply = STATUS_ERROR;
|
||||||
|
EXPECT_TRUE(receive_status(p[1], &reply));
|
||||||
|
if (verify) {
|
||||||
|
EXPECT_EQ_INT((int)reply, (int)STATUS_NEXT);
|
||||||
|
EXPECT_NOT_NULL(file->data->data);
|
||||||
|
EXPECT_TRUE(file->data->data != NULL && memcmp(file->data->data, src_bytes, size) == 0);
|
||||||
|
} else {
|
||||||
|
EXPECT_EQ_INT((int)reply, (int)STATUS_OK);
|
||||||
|
EXPECT_TRUE(file->skip);
|
||||||
|
/* The default quick-check hit materializes from the basis PATH at
|
||||||
|
install time (streaming), so no content is buffered on the File. */
|
||||||
|
EXPECT_NOT_NULL(file->basis_link);
|
||||||
|
EXPECT_NULL(file->data->data);
|
||||||
|
}
|
||||||
|
file_destroy(file);
|
||||||
|
}
|
||||||
|
close(p[0]);
|
||||||
|
close(p[1]);
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
utils_set_authorized_root(-1, NULL);
|
||||||
|
close(root_fd);
|
||||||
|
unlink(basis_file);
|
||||||
|
rmdir(basis_dir);
|
||||||
|
rmdir(root);
|
||||||
|
}
|
||||||
|
|
||||||
/* -K/--keep-dirlinks secure open: with an authorized root, a destination path
|
/* -K/--keep-dirlinks secure open: with an authorized root, a destination path
|
||||||
* component that is a symlink to an IN-ROOT directory is used as that directory
|
* component that is a symlink to an IN-ROOT directory is used as that directory
|
||||||
* (its referent is opened through a relative O_NOFOLLOW walk from the root fd,
|
* (its referent is opened through a relative O_NOFOLLOW walk from the root fd,
|
||||||
@@ -2140,6 +2294,8 @@ void test_file() {
|
|||||||
test_dir_time_list();
|
test_dir_time_list();
|
||||||
test_dir_time_list_cap();
|
test_dir_time_list_cap();
|
||||||
test_receive_incremental_check_empty_path();
|
test_receive_incremental_check_empty_path();
|
||||||
|
test_file_basis_quick_match_decision();
|
||||||
|
test_receive_incremental_check_basis_quick_check_and_verify();
|
||||||
test_keep_dirlinks_secure_open();
|
test_keep_dirlinks_secure_open();
|
||||||
test_inplace_overwrite_clears_special_mode_bits();
|
test_inplace_overwrite_clears_special_mode_bits();
|
||||||
test_inplace_overwrite_metadata_strips_special_bits();
|
test_inplace_overwrite_metadata_strips_special_bits();
|
||||||
|
|||||||
@@ -86,9 +86,43 @@ static void test_dest_state_roundtrip() {
|
|||||||
close(fds[1]);
|
close(fds[1]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static void test_stats_roundtrip() {
|
||||||
|
/* STATUS_STATS grew from three counters (2.25.0) to eight (2.28.0); the codec
|
||||||
|
* must carry every field, including the receiver-observed literal/created
|
||||||
|
* counters, across the wire in order. */
|
||||||
|
int fds[2];
|
||||||
|
if (socketpair(AF_UNIX, SOCK_STREAM, 0, fds) != 0)
|
||||||
|
return;
|
||||||
|
ReceiverStats out;
|
||||||
|
memset(&out, 0, sizeof(out));
|
||||||
|
out.matched_data = 111111111ULL;
|
||||||
|
out.deleted_files = 7;
|
||||||
|
out.would_delete_count = 3;
|
||||||
|
out.literal_bytes = 222222222ULL;
|
||||||
|
out.created_reg = 5;
|
||||||
|
out.created_dir = 4;
|
||||||
|
out.created_link = 2;
|
||||||
|
out.created_special = 1;
|
||||||
|
ReceiverStats in;
|
||||||
|
memset(&in, 0, sizeof(in));
|
||||||
|
EXPECT_TRUE(format_stats_send(fds[0], &out));
|
||||||
|
EXPECT_TRUE(format_stats_receive(fds[1], &in));
|
||||||
|
EXPECT_TRUE(in.matched_data == out.matched_data);
|
||||||
|
EXPECT_TRUE(in.deleted_files == out.deleted_files);
|
||||||
|
EXPECT_TRUE(in.would_delete_count == out.would_delete_count);
|
||||||
|
EXPECT_TRUE(in.literal_bytes == out.literal_bytes);
|
||||||
|
EXPECT_TRUE(in.created_reg == out.created_reg);
|
||||||
|
EXPECT_TRUE(in.created_dir == out.created_dir);
|
||||||
|
EXPECT_TRUE(in.created_link == out.created_link);
|
||||||
|
EXPECT_TRUE(in.created_special == out.created_special);
|
||||||
|
close(fds[0]);
|
||||||
|
close(fds[1]);
|
||||||
|
}
|
||||||
|
|
||||||
void test_format(void) {
|
void test_format(void) {
|
||||||
test_human_size_decimal();
|
test_human_size_decimal();
|
||||||
test_big_num_grouping();
|
test_big_num_grouping();
|
||||||
test_datetime_format();
|
test_datetime_format();
|
||||||
test_dest_state_roundtrip();
|
test_dest_state_roundtrip();
|
||||||
|
test_stats_roundtrip();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,10 +18,11 @@
|
|||||||
|
|
||||||
/* P8 config-frame tail: super_mode (4) + copy-as presence (4) + uid (4) + gid (4). */
|
/* P8 config-frame tail: super_mode (4) + copy-as presence (4) + uid (4) + gid (4). */
|
||||||
#define P8_TAIL_BYTES 16
|
#define P8_TAIL_BYTES 16
|
||||||
/* Bytes after the P8 tail: report_dest_info (4), report_stats (4, wire-stats
|
/* Bytes after the P8 tail: report_dest_info (4), report_stats (4),
|
||||||
* wave) and compression_algo (4, codec wave). The P8 fields sit this many
|
* report_deletes (4, --info=del wave), compression_algo (4, codec wave) and the
|
||||||
* bytes before the end of the frame. */
|
* receiver delete-protection count (4, protocol 2.28.0). The P8 fields sit
|
||||||
#define POST_P8_TAIL_BYTES 12
|
* this many bytes before the end of the frame. */
|
||||||
|
#define POST_P8_TAIL_BYTES 20
|
||||||
|
|
||||||
/* Smoke test for chunk_deserialize fuzz target */
|
/* Smoke test for chunk_deserialize fuzz target */
|
||||||
static void test_fuzz_chunk_deserialize() {
|
static void test_fuzz_chunk_deserialize() {
|
||||||
|
|||||||
@@ -1240,6 +1240,55 @@ static int collect_scan_info_parallel(ParallelScanner* scanner, const char* root
|
|||||||
return failed ? -1 : count;
|
return failed ? -1 : count;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The --progress paths-only pre-count relies on `list_dirs` emitting every
|
||||||
|
* directory (including empty ones) exactly once, alongside the files and
|
||||||
|
* symlinks the streaming scanner already emits. */
|
||||||
|
static void test_scanner_list_dirs_counts_every_entry() {
|
||||||
|
const char* root = "test_scan_listdirs";
|
||||||
|
EXPECT_EQ_INT(mkdir(root, 0755), 0);
|
||||||
|
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub1", 0755), 0);
|
||||||
|
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub1/deep", 0755), 0);
|
||||||
|
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub2", 0755), 0);
|
||||||
|
EXPECT_EQ_INT(mkdir("test_scan_listdirs/emptydir", 0755), 0);
|
||||||
|
create_test_file("test_scan_listdirs/a.txt", "a");
|
||||||
|
create_test_file("test_scan_listdirs/sub1/c.txt", "c");
|
||||||
|
create_test_file("test_scan_listdirs/sub1/deep/d.txt", "d");
|
||||||
|
create_test_file("test_scan_listdirs/sub2/e.txt", "e");
|
||||||
|
EXPECT_EQ_INT(symlink("a.txt", "test_scan_listdirs/link1"), 0);
|
||||||
|
|
||||||
|
ScannerOptions options = {0};
|
||||||
|
options.list_dirs = true;
|
||||||
|
options.emit_empty_dirs = true;
|
||||||
|
options.follow_symlinks = true; /* -l/--links: carry symlinks, don't skip */
|
||||||
|
ScanInfo infos[16];
|
||||||
|
int count = collect_scan_info(root, &options, infos, 16);
|
||||||
|
EXPECT_EQ_INT(count, 9);
|
||||||
|
int dirs = 0;
|
||||||
|
for (int i = 0; i < count; i++) {
|
||||||
|
if (infos[i].is_dir)
|
||||||
|
dirs++;
|
||||||
|
}
|
||||||
|
EXPECT_EQ_INT(dirs, 4);
|
||||||
|
EXPECT_TRUE(scan_info_present(infos, count, "sub1", true, NULL));
|
||||||
|
EXPECT_TRUE(scan_info_present(infos, count, "sub1/deep", true, NULL));
|
||||||
|
EXPECT_TRUE(scan_info_present(infos, count, "sub2", true, NULL));
|
||||||
|
EXPECT_TRUE(scan_info_present(infos, count, "emptydir", true, NULL));
|
||||||
|
EXPECT_TRUE(scan_info_present(infos, count, "a.txt", false, ""));
|
||||||
|
EXPECT_TRUE(scan_info_present(infos, count, "sub1/c.txt", false, ""));
|
||||||
|
EXPECT_TRUE(scan_info_present(infos, count, "link1", false, ""));
|
||||||
|
|
||||||
|
unlink("test_scan_listdirs/a.txt");
|
||||||
|
unlink("test_scan_listdirs/sub1/c.txt");
|
||||||
|
unlink("test_scan_listdirs/sub1/deep/d.txt");
|
||||||
|
unlink("test_scan_listdirs/sub2/e.txt");
|
||||||
|
unlink("test_scan_listdirs/link1");
|
||||||
|
rmdir("test_scan_listdirs/sub1/deep");
|
||||||
|
rmdir("test_scan_listdirs/sub1");
|
||||||
|
rmdir("test_scan_listdirs/sub2");
|
||||||
|
rmdir("test_scan_listdirs/emptydir");
|
||||||
|
rmdir(root);
|
||||||
|
}
|
||||||
|
|
||||||
/* -d without --files-from emits exactly the source-root directory (empty) and
|
/* -d without --files-from emits exactly the source-root directory (empty) and
|
||||||
* never descends. */
|
* never descends. */
|
||||||
static void test_dirs_no_descent() {
|
static void test_dirs_no_descent() {
|
||||||
@@ -1667,6 +1716,7 @@ void test_scanner() {
|
|||||||
test_per_dir_filter_override(true);
|
test_per_dir_filter_override(true);
|
||||||
test_dirs_no_descent();
|
test_dirs_no_descent();
|
||||||
test_dirs_files_from();
|
test_dirs_files_from();
|
||||||
|
test_scanner_list_dirs_counts_every_entry();
|
||||||
test_files_from_relative_send_path();
|
test_files_from_relative_send_path();
|
||||||
test_scanner_captures_directory_times();
|
test_scanner_captures_directory_times();
|
||||||
test_scanner_chunk_ownership();
|
test_scanner_chunk_ownership();
|
||||||
|
|||||||
+148
-13
@@ -3,6 +3,8 @@
|
|||||||
#include "config.h"
|
#include "config.h"
|
||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
|
#include "file_receive.h"
|
||||||
|
#include "format.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "test_utils.h"
|
#include "test_utils.h"
|
||||||
@@ -134,6 +136,124 @@ static void test_receive_files_single_file() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Protocol 2.28.0: a fresh single-file transfer over the wire reports the
|
||||||
|
* receiver-observed literal bytes and the created-regular counter through the
|
||||||
|
* terminal STATUS_STATS frame, and an update reports created_reg == 0. */
|
||||||
|
static void test_receive_stats_frame_created_and_literal() {
|
||||||
|
const char* content = "stats frame content";
|
||||||
|
size_t len = strlen(content);
|
||||||
|
char root_template[] = "/tmp/fastsync_stats_XXXXXX";
|
||||||
|
char* root = mkdtemp(root_template);
|
||||||
|
EXPECT_NOT_NULL(root);
|
||||||
|
|
||||||
|
int p[2];
|
||||||
|
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
|
||||||
|
io_set_fds(p[0], p[1]);
|
||||||
|
io_set_bwlimit(0);
|
||||||
|
|
||||||
|
Config* cfg = config_create();
|
||||||
|
EXPECT_NOT_NULL(cfg);
|
||||||
|
free(cfg->version);
|
||||||
|
cfg->version = str_dup(PROTOCOL_VERSION);
|
||||||
|
cfg->send_directory = str_dup("/src");
|
||||||
|
cfg->receive_root_directory = str_dup(root);
|
||||||
|
cfg->save_to_disk = true;
|
||||||
|
cfg->report_stats = true;
|
||||||
|
|
||||||
|
pid_t pid = fork();
|
||||||
|
if (pid == 0) {
|
||||||
|
close(p[1]);
|
||||||
|
io_set_fds(p[0], p[0]);
|
||||||
|
int ret = receiver_receive_files(cfg, p[0]);
|
||||||
|
close(p[0]);
|
||||||
|
config_delete(cfg);
|
||||||
|
_exit(ret == 0 ? 0 : 1);
|
||||||
|
}
|
||||||
|
close(p[0]);
|
||||||
|
io_set_fds(p[1], p[1]);
|
||||||
|
send_status(p[1], STATUS_NEXT);
|
||||||
|
File* file = file_create("created.bin");
|
||||||
|
EXPECT_NOT_NULL(file);
|
||||||
|
file->data->data = malloc(len);
|
||||||
|
EXPECT_NOT_NULL(file->data->data);
|
||||||
|
memcpy(file->data->data, content, len);
|
||||||
|
file->data->size = len;
|
||||||
|
send_str(p[1], file->path);
|
||||||
|
send_data(p[1], file->data);
|
||||||
|
file_destroy(file);
|
||||||
|
send_status(p[1], STATUS_FINISHED);
|
||||||
|
|
||||||
|
Status status;
|
||||||
|
EXPECT_TRUE(receive_status(p[1], &status));
|
||||||
|
EXPECT_EQ_INT(status, STATUS_STATS);
|
||||||
|
ReceiverStats stats;
|
||||||
|
EXPECT_TRUE(format_stats_receive(p[1], &stats));
|
||||||
|
int would = 0;
|
||||||
|
EXPECT_TRUE(receive_int(p[1], &would));
|
||||||
|
EXPECT_EQ_INT(would, 0);
|
||||||
|
EXPECT_TRUE(stats.created_reg == 1);
|
||||||
|
EXPECT_TRUE(stats.created_dir == 0);
|
||||||
|
EXPECT_TRUE(stats.created_link == 0);
|
||||||
|
EXPECT_TRUE(stats.created_special == 0);
|
||||||
|
EXPECT_TRUE(stats.literal_bytes == (unsigned long long)len);
|
||||||
|
EXPECT_TRUE(stats.matched_data == 0);
|
||||||
|
|
||||||
|
Status final;
|
||||||
|
EXPECT_TRUE(receive_status(p[1], &final));
|
||||||
|
EXPECT_EQ_INT(final, STATUS_OK);
|
||||||
|
int wstatus;
|
||||||
|
waitpid(pid, &wstatus, 0);
|
||||||
|
close(p[1]);
|
||||||
|
config_delete(cfg);
|
||||||
|
EXPECT_TRUE(WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 0);
|
||||||
|
|
||||||
|
/* Second run against the now-existing destination: no created file. */
|
||||||
|
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
|
||||||
|
io_set_fds(p[0], p[1]);
|
||||||
|
io_set_bwlimit(0);
|
||||||
|
cfg = config_create();
|
||||||
|
EXPECT_NOT_NULL(cfg);
|
||||||
|
free(cfg->version);
|
||||||
|
cfg->version = str_dup(PROTOCOL_VERSION);
|
||||||
|
cfg->send_directory = str_dup("/src");
|
||||||
|
cfg->receive_root_directory = str_dup(root);
|
||||||
|
cfg->save_to_disk = true;
|
||||||
|
cfg->report_stats = true;
|
||||||
|
pid = fork();
|
||||||
|
if (pid == 0) {
|
||||||
|
close(p[1]);
|
||||||
|
io_set_fds(p[0], p[0]);
|
||||||
|
int ret = receiver_receive_files(cfg, p[0]);
|
||||||
|
close(p[0]);
|
||||||
|
config_delete(cfg);
|
||||||
|
_exit(ret == 0 ? 0 : 1);
|
||||||
|
}
|
||||||
|
close(p[0]);
|
||||||
|
io_set_fds(p[1], p[1]);
|
||||||
|
send_status(p[1], STATUS_NEXT);
|
||||||
|
file = file_create("created.bin");
|
||||||
|
EXPECT_NOT_NULL(file);
|
||||||
|
file->data->data = malloc(len);
|
||||||
|
EXPECT_NOT_NULL(file->data->data);
|
||||||
|
memcpy(file->data->data, content, len);
|
||||||
|
file->data->size = len;
|
||||||
|
send_str(p[1], file->path);
|
||||||
|
send_data(p[1], file->data);
|
||||||
|
file_destroy(file);
|
||||||
|
send_status(p[1], STATUS_FINISHED);
|
||||||
|
EXPECT_TRUE(receive_status(p[1], &status));
|
||||||
|
EXPECT_EQ_INT(status, STATUS_STATS);
|
||||||
|
EXPECT_TRUE(format_stats_receive(p[1], &stats));
|
||||||
|
EXPECT_TRUE(receive_int(p[1], &would));
|
||||||
|
EXPECT_TRUE(stats.created_reg == 0);
|
||||||
|
EXPECT_TRUE(stats.literal_bytes == (unsigned long long)len);
|
||||||
|
EXPECT_TRUE(receive_status(p[1], &final));
|
||||||
|
waitpid(pid, &wstatus, 0);
|
||||||
|
close(p[1]);
|
||||||
|
config_delete(cfg);
|
||||||
|
EXPECT_TRUE(WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 0);
|
||||||
|
}
|
||||||
|
|
||||||
/* Test receive_files with STATUS_ABORT */
|
/* Test receive_files with STATUS_ABORT */
|
||||||
static void test_receive_files_abort() {
|
static void test_receive_files_abort() {
|
||||||
Config* cfg = config_create();
|
Config* cfg = config_create();
|
||||||
@@ -943,14 +1063,18 @@ static void test_incremental_check_fifo_destination_does_not_hang() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/* A server-contacting --dry-run with an alternate basis dir must never read or
|
/* A server-contacting --dry-run with an alternate basis dir must never read or
|
||||||
hash the basis file. An exact (size+mtime+content) basis match would
|
hash the basis file. Under the default metadata quick-check a hit needs no
|
||||||
otherwise let a client probe the basis bytes against its own supplied digest
|
basis bytes, so a compare-dest match is reported as a skip (STATUS_OK) just
|
||||||
(a 1-bit content oracle). The dry-run decision is metadata-only, so even a
|
like a real run -- and still no content is read. Under --verify-basis a hit
|
||||||
byte-identical basis is reported as would-transfer, not a compare-dest skip. */
|
would require hashing the basis against the client-supplied digest (a 1-bit
|
||||||
static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
content oracle), which a dry-run must never do, so even a byte-identical
|
||||||
|
basis is reported as would-transfer. The destination is never materialized
|
||||||
|
in either arm. */
|
||||||
|
static void run_dry_run_basis_check(bool verify, Status expected) {
|
||||||
Config* cfg = config_create();
|
Config* cfg = config_create();
|
||||||
EXPECT_NOT_NULL(cfg);
|
EXPECT_NOT_NULL(cfg);
|
||||||
cfg->dry_run = true;
|
cfg->dry_run = true;
|
||||||
|
cfg->verify_basis = verify;
|
||||||
char* root = make_check_root("dryb");
|
char* root = make_check_root("dryb");
|
||||||
EXPECT_NOT_NULL(root);
|
EXPECT_NOT_NULL(root);
|
||||||
cfg->receive_root_directory = str_dup(root);
|
cfg->receive_root_directory = str_dup(root);
|
||||||
@@ -966,8 +1090,8 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
|||||||
EXPECT_EQ_INT(stat(basis_path, &bst), 0);
|
EXPECT_EQ_INT(stat(basis_path, &bst), 0);
|
||||||
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_COMPARE, "basis"), 0);
|
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_COMPARE, "basis"), 0);
|
||||||
|
|
||||||
/* The (correct) source digest for the basis bytes: an unfixed dry-run would
|
/* The (correct) source digest for the basis bytes: a buggy dry-run that read
|
||||||
read+hash the basis and treat this as an exact compare-dest hit. */
|
and hashed the basis would treat this as an exact compare-dest hit. */
|
||||||
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
size_t digest_len = 0;
|
size_t digest_len = 0;
|
||||||
EXPECT_TRUE(checksum_digest((ChecksumAlgo)cfg->checksum_algo, cfg->checksum_seed, content,
|
EXPECT_TRUE(checksum_digest((ChecksumAlgo)cfg->checksum_algo, cfg->checksum_seed, content,
|
||||||
@@ -986,7 +1110,8 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
|||||||
bool skipped = false;
|
bool skipped = false;
|
||||||
bool would_transfer = false;
|
bool would_transfer = false;
|
||||||
File* file = receive_incremental_check_ex(p[0], cfg, &skipped, &would_transfer);
|
File* file = receive_incremental_check_ex(p[0], cfg, &skipped, &would_transfer);
|
||||||
bool ok = file == NULL && !skipped && would_transfer;
|
bool ok = file == NULL && skipped == (expected == STATUS_OK) &&
|
||||||
|
would_transfer == (expected != STATUS_OK);
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
close(p[0]);
|
close(p[0]);
|
||||||
@@ -1004,13 +1129,16 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
|||||||
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
|
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
|
||||||
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
|
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
|
||||||
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
|
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
|
||||||
uint8_t wire_len = (uint8_t)digest_len;
|
/* The digest is only on the wire when --checksum or --verify-basis needs it
|
||||||
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
|
(cfg->checksum is false here); the default quick-check arm sends none. */
|
||||||
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
|
if (verify) {
|
||||||
|
uint8_t wire_len = (uint8_t)digest_len;
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
|
||||||
|
}
|
||||||
Status s;
|
Status s;
|
||||||
EXPECT_TRUE(receive_status(p[1], &s));
|
EXPECT_TRUE(receive_status(p[1], &s));
|
||||||
/* A skip here would mean the receiver read+hashed the basis file. */
|
EXPECT_EQ_INT(s, expected);
|
||||||
EXPECT_EQ_INT(s, STATUS_DRY_RUN_TRANSFER);
|
|
||||||
|
|
||||||
int status;
|
int status;
|
||||||
waitpid(pid, &status, 0);
|
waitpid(pid, &status, 0);
|
||||||
@@ -1028,6 +1156,11 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
||||||
|
run_dry_run_basis_check(false, STATUS_OK);
|
||||||
|
run_dry_run_basis_check(true, STATUS_DRY_RUN_TRANSFER);
|
||||||
|
}
|
||||||
|
|
||||||
/* B1: a FIFO planted in a --link-dest basis directory must not block
|
/* B1: a FIFO planted in a --link-dest basis directory must not block
|
||||||
* basis_open_regular() either; the basis match is simply declined. */
|
* basis_open_regular() either; the basis match is simply declined. */
|
||||||
static void test_incremental_check_basis_fifo_does_not_hang() {
|
static void test_incremental_check_basis_fifo_does_not_hang() {
|
||||||
@@ -1158,6 +1291,7 @@ static void test_dry_run_delete_plan_commit_does_not_delete() {
|
|||||||
EXPECT_TRUE(send_int(p[1], 0)); /* size-skipped prefixes */
|
EXPECT_TRUE(send_int(p[1], 0)); /* size-skipped prefixes */
|
||||||
EXPECT_TRUE(send_int(p[1], 1)); /* missing-args exact deletions */
|
EXPECT_TRUE(send_int(p[1], 1)); /* missing-args exact deletions */
|
||||||
EXPECT_TRUE(send_str(p[1], "victim.txt"));
|
EXPECT_TRUE(send_str(p[1], "victim.txt"));
|
||||||
|
EXPECT_TRUE(send_int(p[1], 1)); /* apply: a real plan */
|
||||||
EXPECT_TRUE(send_str(p[1], ".")); /* receive root plan */
|
EXPECT_TRUE(send_str(p[1], ".")); /* receive root plan */
|
||||||
EXPECT_TRUE(send_int(p[1], 0)); /* kept child directories */
|
EXPECT_TRUE(send_int(p[1], 0)); /* kept child directories */
|
||||||
EXPECT_TRUE(send_int(p[1], 0)); /* kept child files */
|
EXPECT_TRUE(send_int(p[1], 0)); /* kept child files */
|
||||||
@@ -1183,6 +1317,7 @@ void test_server() {
|
|||||||
if (!is_running_under_valgrind()) {
|
if (!is_running_under_valgrind()) {
|
||||||
test_receive_files_finished();
|
test_receive_files_finished();
|
||||||
test_receive_files_single_file();
|
test_receive_files_single_file();
|
||||||
|
test_receive_stats_frame_created_and_literal();
|
||||||
test_receive_files_abort();
|
test_receive_files_abort();
|
||||||
test_receive_manifest_rejects_traversal();
|
test_receive_manifest_rejects_traversal();
|
||||||
test_receive_incremental_check_rejects_invalid_nanoseconds();
|
test_receive_incremental_check_rejects_invalid_nanoseconds();
|
||||||
|
|||||||
@@ -10,6 +10,7 @@
|
|||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
|
#include <strings.h>
|
||||||
#include <sys/socket.h>
|
#include <sys/socket.h>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
#include <threads.h>
|
#include <threads.h>
|
||||||
@@ -137,7 +138,7 @@ static void test_walker_removes_extras_keeps_manifest_and_protected() {
|
|||||||
DeleteSkipEntry skip = {"prot", false};
|
DeleteSkipEntry skip = {"prot", false};
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, NULL, 100000, &skip, 1, &deleted, NULL);
|
delete_extras_limited(root, manifest, NULL, 100000, &skip, 1, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_FALSE(file_exists(root, "a.txt"));
|
EXPECT_FALSE(file_exists(root, "a.txt"));
|
||||||
EXPECT_TRUE(file_exists(root, "keep.txt"));
|
EXPECT_TRUE(file_exists(root, "keep.txt"));
|
||||||
@@ -172,7 +173,7 @@ static void test_walker_keeps_nested_manifest_dirs() {
|
|||||||
EXPECT_NOT_NULL(manifest);
|
EXPECT_NOT_NULL(manifest);
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, &deleted, NULL);
|
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_FALSE(file_exists(root, "extra.txt"));
|
EXPECT_FALSE(file_exists(root, "extra.txt"));
|
||||||
EXPECT_TRUE(file_exists(root, "keepdir/deep/keep.txt"));
|
EXPECT_TRUE(file_exists(root, "keepdir/deep/keep.txt"));
|
||||||
@@ -203,7 +204,7 @@ static void test_walker_max_delete_partial_deletes_up_to_cap() {
|
|||||||
size_t deleted = 999;
|
size_t deleted = 999;
|
||||||
size_t skipped = 0;
|
size_t skipped = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, &deleted, &skipped);
|
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, NULL, &deleted, &skipped);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_LIMIT_REACHED);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_LIMIT_REACHED);
|
||||||
EXPECT_EQ_INT((int)deleted, 2);
|
EXPECT_EQ_INT((int)deleted, 2);
|
||||||
EXPECT_EQ_INT((int)skipped, 1);
|
EXPECT_EQ_INT((int)skipped, 1);
|
||||||
@@ -224,7 +225,8 @@ static void test_walker_max_delete_exact_bound_deletes() {
|
|||||||
ArrayList* manifest = make_manifest_strings(keeps, 0);
|
ArrayList* manifest = make_manifest_strings(keeps, 0);
|
||||||
EXPECT_NOT_NULL(manifest);
|
EXPECT_NOT_NULL(manifest);
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result = delete_extras_limited(root, manifest, NULL, 2, NULL, 0, &deleted, NULL);
|
DeleteWalkResult result =
|
||||||
|
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_EQ_INT((int)deleted, 2);
|
EXPECT_EQ_INT((int)deleted, 2);
|
||||||
EXPECT_FALSE(file_exists(root, "a.txt"));
|
EXPECT_FALSE(file_exists(root, "a.txt"));
|
||||||
@@ -257,7 +259,7 @@ static void test_walker_removes_extraneous_symlinks() {
|
|||||||
EXPECT_NOT_NULL(manifest);
|
EXPECT_NOT_NULL(manifest);
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, &deleted, NULL);
|
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_FALSE(file_exists(root, "link_file"));
|
EXPECT_FALSE(file_exists(root, "link_file"));
|
||||||
EXPECT_FALSE(file_exists(root, "link_dir"));
|
EXPECT_FALSE(file_exists(root, "link_dir"));
|
||||||
@@ -293,7 +295,7 @@ static void test_walker_confines_deletion_to_synced_dirs() {
|
|||||||
EXPECT_TRUE(array_list_add(dirs, str_dup("inscope")));
|
EXPECT_TRUE(array_list_add(dirs, str_dup("inscope")));
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, dirs, 100000, NULL, 0, &deleted, NULL);
|
delete_extras_limited(root, manifest, dirs, 100000, NULL, 0, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_TRUE(file_exists(root, "rootextra.txt"));
|
EXPECT_TRUE(file_exists(root, "rootextra.txt"));
|
||||||
EXPECT_FALSE(file_exists(root, "inscope/extra.txt"));
|
EXPECT_FALSE(file_exists(root, "inscope/extra.txt"));
|
||||||
@@ -323,6 +325,45 @@ static void test_walker_unlimited_deletes_all() {
|
|||||||
free(root);
|
free(root);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Receiver-side filter protection (protocol 2.28.0): a compiled protect rule
|
||||||
|
shields a DESTINATION-ONLY extra that never appeared on the sender, a risk
|
||||||
|
rule cancels an earlier/later protect (first match wins), and a dir-only
|
||||||
|
protect rule shields the whole subtree. */
|
||||||
|
static void test_walker_protect_rules_shield_dest_only() {
|
||||||
|
char* root = make_walk_root("protectrules");
|
||||||
|
EXPECT_NOT_NULL(root);
|
||||||
|
EXPECT_TRUE(write_file_at(root, "keep.txt", "kept"));
|
||||||
|
EXPECT_TRUE(write_file_at(root, "extra.log", "risk cancels protect"));
|
||||||
|
EXPECT_TRUE(write_file_at(root, "safe.log", "protected"));
|
||||||
|
EXPECT_TRUE(write_file_at(root, "other.txt", "deleted"));
|
||||||
|
EXPECT_EQ_INT(make_subdir(root, "prot"), 0);
|
||||||
|
EXPECT_TRUE(write_file_at(root, "prot/inside.txt", "shielded subtree"));
|
||||||
|
EXPECT_TRUE(write_file_at(root, "prot/deep.log", "shielded subtree"));
|
||||||
|
|
||||||
|
const char* keeps[] = {"keep.txt"};
|
||||||
|
ArrayList* manifest = make_manifest_strings(keeps, 1);
|
||||||
|
EXPECT_NOT_NULL(manifest);
|
||||||
|
const char* rule_text[] = {"R extra.log", "P *.log", "P prot/"};
|
||||||
|
char err[160];
|
||||||
|
FilterRuleList* rules = filter_base_build(rule_text, 3, false, false, err, sizeof(err));
|
||||||
|
EXPECT_NOT_NULL(rules);
|
||||||
|
size_t deleted = 0;
|
||||||
|
DeleteWalkResult result =
|
||||||
|
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, rules, &deleted, NULL);
|
||||||
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
|
EXPECT_TRUE(file_exists(root, "keep.txt"));
|
||||||
|
EXPECT_FALSE(file_exists(root, "extra.log")); /* risk wins the first match */
|
||||||
|
EXPECT_TRUE(file_exists(root, "safe.log")); /* protect shields the extra */
|
||||||
|
EXPECT_FALSE(file_exists(root, "other.txt"));
|
||||||
|
EXPECT_TRUE(dir_exists(root, "prot"));
|
||||||
|
EXPECT_TRUE(file_exists(root, "prot/inside.txt"));
|
||||||
|
EXPECT_TRUE(file_exists(root, "prot/deep.log"));
|
||||||
|
filter_rule_list_free(rules);
|
||||||
|
array_list_delete(manifest);
|
||||||
|
remove_walk_tree(root);
|
||||||
|
free(root);
|
||||||
|
}
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
bool eight_bit_output;
|
bool eight_bit_output;
|
||||||
const char* expected;
|
const char* expected;
|
||||||
@@ -578,7 +619,43 @@ static void test_getdelim_bounded() {
|
|||||||
fclose(fp);
|
fclose(fp);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static int test_env_resolver(const char* name) {
|
||||||
|
if (strcasecmp(name, "alpha") == 0)
|
||||||
|
return 10;
|
||||||
|
if (strcasecmp(name, "beta") == 0)
|
||||||
|
return 20;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_env_choice_first_parsing() {
|
||||||
|
const char* var = "FASTSYNC_TEST_CHOICE_LIST";
|
||||||
|
bool specified = true;
|
||||||
|
unsetenv(var);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
|
||||||
|
EXPECT_FALSE(specified);
|
||||||
|
|
||||||
|
/* Unknown entries are skipped, case-insensitive, first supported wins. */
|
||||||
|
setenv(var, "bogus BETA alpha", 1);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 20);
|
||||||
|
EXPECT_TRUE(specified);
|
||||||
|
|
||||||
|
/* The client half ends at '&'. */
|
||||||
|
setenv(var, "alpha & beta", 1);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 10);
|
||||||
|
|
||||||
|
/* Blank means "unspecified"; all-unknown means "specified but no match". */
|
||||||
|
setenv(var, " ", 1);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
|
||||||
|
EXPECT_FALSE(specified);
|
||||||
|
setenv(var, "nope,alpha", 1);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
|
||||||
|
EXPECT_TRUE(specified);
|
||||||
|
|
||||||
|
unsetenv(var);
|
||||||
|
}
|
||||||
|
|
||||||
void test_shared_utils() {
|
void test_shared_utils() {
|
||||||
|
test_env_choice_first_parsing();
|
||||||
test_path_index_bounded();
|
test_path_index_bounded();
|
||||||
test_path_index_semantics();
|
test_path_index_semantics();
|
||||||
test_getdelim_bounded();
|
test_getdelim_bounded();
|
||||||
@@ -589,6 +666,7 @@ void test_shared_utils() {
|
|||||||
test_walker_removes_extraneous_symlinks();
|
test_walker_removes_extraneous_symlinks();
|
||||||
test_walker_confines_deletion_to_synced_dirs();
|
test_walker_confines_deletion_to_synced_dirs();
|
||||||
test_walker_unlimited_deletes_all();
|
test_walker_unlimited_deletes_all();
|
||||||
|
test_walker_protect_rules_shield_dest_only();
|
||||||
test_loopback_helpers();
|
test_loopback_helpers();
|
||||||
test_fd_peer_ip();
|
test_fd_peer_ip();
|
||||||
|
|
||||||
|
|||||||
+2
-1
@@ -210,7 +210,8 @@ static void test_xattr_receive_drops_acl_without_preserve_acls() {
|
|||||||
|
|
||||||
/* MINOR-2: a --link-dest / -H copy fallback (linkat refused) must still apply
|
/* MINOR-2: a --link-dest / -H copy fallback (linkat refused) must still apply
|
||||||
* the per-file xattrs and --fake-super stat. A DIRECTORY basis forces linkat
|
* the per-file xattrs and --fake-super stat. A DIRECTORY basis forces linkat
|
||||||
* to fail with EPERM, exercising the byte-copy fallback deterministically.
|
* to fail with EPERM, exercising the byte-copy fallback deterministically (the
|
||||||
|
* basis is not a regular file, so the fallback uses the caller's bytes).
|
||||||
* Guarded on filesystem xattr support. */
|
* Guarded on filesystem xattr support. */
|
||||||
static void test_link_copy_fallback_preserves_xattrs() {
|
static void test_link_copy_fallback_preserves_xattrs() {
|
||||||
const char* dest = "test_link_xattr_dest.txt";
|
const char* dest = "test_link_xattr_dest.txt";
|
||||||
|
|||||||
Reference in New Issue
Block a user