TapTap/FastSync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

feat(parity): rsync parity tracks 1-6 (protocol 2.28.0) #303

Merged
TapTap merged 14 commits from feat/parity-2.28 into dev 2026-09-19 17:14:37 +02:00
Conversation 0 Commits 14 Files Changed 61
+4400 -677
61 changed files with 4400 additions and 677 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+31
View File
@@ -4,6 +4,37 @@ All notable changes to FastSync are documented here. Versions match
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
run the same version because the handshake is strict.
## [Unreleased]
### Changed
- **`--delete` now defaults to delete-during (rsync `--del`) timing.** With no
explicit timing flag, a plain `--delete` removes each directory's extras as
that directory is processed instead of committing one whole-tree deletion only
after the entire transfer succeeds. This matches rsync, frees destination
space progressively, and avoids the whole-old+new-tree peak that could
`ENOSPC` a tight destination. The client maps the default onto the existing
`delete_during` wire boolean, so `PROTOCOL_VERSION` stays `2.28.0`.
- Added the FastSync-only long option **`--delete-commit`** (implies
`--delete`): it selects the old late whole-tree commit and is timing-identical
to `--delete-after` (the same `delete_after` wire boolean). Explicit timing
flags always win over the default, at most one timing flag may be given, and a
timing flag combined with `--no-delete` is still rejected.
- The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag:
the one-shot per-run config block (protected prefixes, size-pruned mirrors,
`--delete-missing-args` exact paths) is now always transmitted first on a
config-only carrier (`apply=false`), fixing a latent bug where a
`--delete-missing-args` run whose `--files-from` list synchronized no directory
never sent its exact deletions.
### Migration
- Scripts that relied on plain `--delete` deleting nothing until the transfer
fully succeeded must pass **`--delete-commit`** (or `--delete-after`) to keep
that behavior. Plain `--delete` now removes reached directories' extras during
the transfer, exactly like rsync's default; on a completed run the final tree
is unchanged.
## [2.26.0] - 2026-09-17
### Added
CMakeLists.txt
+1 -1
View File
@@ -1,6 +1,6 @@
cmake_minimum_required(VERSION 3.22)
project(FastFileTransfer VERSION 2.27.0)
project(FastFileTransfer VERSION 2.28.0)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_C_STANDARD 11)
HANDOFF.md
+115 -4
View File
@@ -52,9 +52,10 @@
`tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
over daemon/TCP (no argv channel in FastSync's binary config handshake;
rsync-daemon differential pins the rsync behavior) and receiver-side
`protect`/`risk` re-derivation for destination-only entries (would need a
receiver filter engine; differential pins the divergence). The options pass
stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
`protect`/`risk` re-derivation for destination-only entries (would need a
receiver filter engine; differential pins the divergence — **reversed by
track 4a below**, which adds that engine). The options pass
stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
`--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
@@ -74,13 +75,123 @@
heuristic with a 10× size window, not rsync's matcher), but its residual is the
candidate-selection heuristic itself: the final tree is byte-exact by design, so
it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
differential. The parity-review pass then moved `--delete-delay` to ⚠️ (the
differential. (Track 5b later found the name heuristic is rsync's own and moved
the row ❌ → ⚠️, leaving only the narrower delta size window; see entry 15.) The parity-review pass then moved `--delete-delay` to ⚠️ (the
plan-time `--max-delete` charge and non-recursive deferred removal differ from
rsync when a snapshotted entry fails removal). Differential-gate allowlist
entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
integrated stats+options+fs branch stands at **111 ✅ / 13 ⚠️ / 33 ❌ = 157**;
full suite + ASan + clang-format + cppcheck clean.
11. **No-wire parity track 1** on `feat/parity-2.28` (no protocol change):
`-n --delete` now sends the same filter-excluded + size-pruned protected
prefixes and synchronized-directory scope as a real run (dry-run would-delete
matches rsync for source-derived protections; the destination-only exclude
residual was later closed by track 4a, readdir ordering remains);
`--delete-delay` now charges
`--max-delete` on actual removals and re-scans a queued directory at commit
to remove content created after the plan, with an independent deferred-list
cap (only partial-delete ordering remains); and `--info=name2` emits `NAME is
uptodate` plus the leading `./` root name line for `--info=name` (only the
root-line trigger condition and receiver-side `skip` wording remain). Matrix
now **111 ✅ / 14 ⚠️ / 32 ❌ = 157**; differential + unit tests added in
`test_features.py`, `test_option_parity.py`, `test_delete_plan.c`,
`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`.
12. **No-wire parity track 2b** on `feat/parity-2.28` (no protocol change):
`--progress`/`-P`/`--info=progress` (when not `--quiet`) now run an opt-in
paths-only metadata pre-count (no file reads/hashing) that supplies rsync's
full file-list total for the `to-chk` denominator and the directory names,
and emits per-directory/symlink/special name lines, in both the sequential
and `--threads` paths. `--delete-during`/`--delete-delay` reuse their
keep-set pre-scan instead of a second walk; non-progress runs are
unaffected. Differential tests (`progress`/`progress_threads` over a new
`multidir` corpus) match rsync's name set and `to-chk` denominator on a
fresh transfer, and the single-file byte-identical test still passes;
emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream)
plus re-run over-naming (unconditional `./`, ancestor dirs named with a
transferred child, and no quick-check for symlinks/empty dirs) remain the
caveats, so the row stays ⚠️ and the matrix is unchanged at
**111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
13. **Wire parity track 4a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
`2.28.0`): the receiver now has a delete-time filter engine. The sender
compiles its root-level selection rules exactly as the scanner does
(`filter_base_build`) and streams them as one bounded, self-describing
config-frame block (action, sides, anchored, dir-only, negate, owner,
pattern; bounded rule count and pattern bytes, unknown action/sides is a
protocol error). The receiver reconstructs `protect_rules` and applies them
first-match-wins to each extraneous destination path in every delete timing
(the whole-tree commit walker, the `--delete-during`/`--delete-delay`
per-directory plans, and the `-n` would-delete enumeration), so a
`P *.log` rule protects a destination-only `extra.log` like rsync (with
`risk` cancelling); the sender-derived protected-prefix behavior is
preserved when no rules are sent and `--delete-excluded` semantics are
unchanged. Per-directory merge (`:`/`.`) receiver re-derivation remains the
residual. `TestFilterProtect` (real + dry-run) plus differential cases
`filter_protect`, `filter_protect_during`, `filter_protect_delay` added and
the `--filter=RULE` row moves ❌ → ✅: matrix now
**115 ✅ / 11 ⚠️ / 31 ❌ = 157**; unit tests, the three named integration
files, clang-format and cppcheck clean.
14. **Wire parity track 5a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
`2.28.0` by project decision): the three basis-dir options now default to
rsync's metadata quick-check (equal size + equal mtime, or size alone under
`--size-only`; `-I` disables matching) instead of FastSync's historical
xxHash64 content equality, so a same-size/different-content basis is trusted
exactly as rsync trusts it. A new FastSync-only, long-only `--verify-basis`
flag restores the strict whole-file content equality; its bool is appended to
the basis block of the config frame (golden wire frame 882 → 886 bytes).
`--verify-basis` streams the confined basis descriptor to hash it, and a
basis hit is no longer capped at the 256 MiB whole-file payload bound:
`--copy-dest` streams the basis through a bounded buffer and `--link-dest`'s
copy fallback streams from the basis, so an over-limit hit materializes (a
basis MISS still falls back to the normal transfer and keeps its own bound).
A `--copy-dest` hit re-applies the SOURCE attributes (the sender transmits
the source metadata with the basis check frame), matching rsync's
"copy then fix attributes"; a `--link-dest` success keeps the shared inode's
attributes (writing through it would mutate the basis). Differential cases
`copy_dest` and `verify_basis` added; `test_basis_dir_size_only_content_residual`
converted to a passing parity assertion; `TestBasisDestDirs` updated for the
new default + `--verify-basis`; unit tests cover the quick-check/verify
decision and the same-size/different-content handshake. The
`--compare-dest`/`--copy-dest`/`--link-dest` rows move ❌ → ⚠️ (relative-DIR
resolution base and over-limit MISS refusal): matrix now
**116 ✅ / 13 ⚠️ / 28 ❌ = 157**.
15. **No-wire parity track 5b** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
`2.28.0` by project decision): `-y`/`--fuzzy` reclassified ❌ → ⚠️. A probe
against real rsync 3.4.1 (pinned `-B8192`, repeated-content 64 KiB corpus)
showed the name heuristic is already rsync's (`util1.c fuzzy_distance` /
`find_filename_suffix` + the exact size+mtime pass) and the output is always
byte-exact; the only residual is candidate ELIGIBILITY, because FastSync's
`delta_should_attempt` gate caps the size ratio at 10× and requires both
files ≥ 16 KiB while rsync will reuse a basis from 0.25× to 10000× and below
16 KiB. The choice is observable only as `--stats` bandwidth counters. Added
differential case `fuzzy_basis` (same-suffix sibling, one name edit,
identical content, block size pinned) asserting tree **and** normalized
`--stats` parity where the choices coincide, plus `TestFuzzy` pinning the
window boundary on both sides (>10× and <16 KiB siblings declined by
FastSync while rsync uses them, both trees byte-identical). Matrix now
**116 ✅ / 14 ⚠️ / 27 ❌ = 157**.
16. **Lockstep delete-default track 6** on `feat/parity-2.28` (`PROTOCOL_VERSION`
stays `2.28.0`): plain `--delete` now defaults to rsync's delete-during
(`--del`) timing, normalized on the client onto the existing `delete_during`
wire bool. The old late whole-tree commit is opt-in via `--delete-after` or
the FastSync-only long `--delete-commit` (identical `delete_after` timing).
`-d/--dirs` still falls back to the end commit, `--delay-updates` still
deletes before publication, and `--files-from`/`-R` scope is unchanged. The
`STATUS_DELETE_PLAN` frame gained a one-int `apply` flag so the per-run
config block (including `--delete-missing-args` exact paths) is always
transmitted, on a config-only carrier when the scope allows no directory
plan — fixing a latent bug with a file-only `--files-from` list. Differential
cases `delete`/`delete_commit`/`filter_protect_after` plus the extended
`test_delete_timing_parity.py` (plain `--delete` mid-abort removes reached
extras, `--delete-commit` defers) pass; full `-m "not setpriv"` suite,
clang-format and cppcheck clean. Matrix unchanged at
**116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay
⚠️ for the abort boundary; `--delete-after` stays ✅).
## Next steps
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
2. **Deferred security items** (documented, not implemented):
README.md
+18 -9
View File
@@ -113,13 +113,18 @@ matrix is classified as parity, caveat, or divergent in
(`-B1000`, `-essh`, `-MOPT`, `--opt=value`) are accepted, matching rsync.
- `-r`, `-b`, `-L`, and `-B` are parsed with the rsync short names.
- `--stats` prints the counters FastSync can observe plus the receiver-only
counters (`Matched data`, deleted files) reported over the wire; rsync's
per-type `Number of files` breakdown is not reproduced. `--progress` prints
rsync-style per-file blocks (without rsync's leading `./` line).
counters reported over the wire (`Matched data`, deleted files, and the
created/literal counters); `Number of files` and `Number of created files`
carry rsync's per-type breakdown. `--progress` prints rsync-style per-file
blocks including the leading `./` line, and (when progress is requested) a
paths-only pre-count supplies rsync's `to-chk` denominator.
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums, negotiated with
`auto`; `zlibx` behaves as `zlib`, and the transfer checksum is not separately
selectable.
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums. `auto` honors
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` and otherwise follows rsync's
compiled-in order. An omitted `--compress-level` uses the codec's rsync
default (zstd 3, zlib/zlibx 6, lz4 ignored); `zlib`/`zlibx` share the
literal-only zlib path (rsync's zlibx semantics), and the transfer checksum is
not separately selectable.
The detailed flag matrix is maintained in
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It reports each row as **parity**,
@@ -202,11 +207,13 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`) |
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination |
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win) |
| `--delete` | Delete files on receiver not present in source (default timing: delete-after, i.e. only after the whole transfer succeeded). Scoped to the synchronized directories, so `--files-from` subsets are safe |
| `--verify-basis` | FastSync-only: require a basis hit (`--compare-dest`/`--copy-dest`/`--link-dest`) to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync) |
| `--delete` | Delete files on receiver not present in source (default timing: delete-during, matching rsync, so destination space is freed progressively). Scoped to the synchronized directories, so `--files-from` subsets are safe |
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`) |
| `--delete-after` | Explicit delete-after timing (implies `--delete`) |
| `--delete-commit` | FastSync-only: keep the pre-2.28 atomic timing — delete only after the whole transfer succeeded (identical timing to `--delete-after`) |
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication) |
@@ -505,8 +512,8 @@ features without changing the meaning of ordinary compatibility options.
|---|---|
| `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. |
| `-z [level]`, `--compress [level]` | Enable streaming compression (default `zstd`), levels 1-22. |
| `--compress-level <n>` | Set the compression level. |
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlibx` behaves as `zlib`. |
| `--compress-level <n>` | Set the compression level (1-22). Omitted, each codec uses its rsync default: zstd 3, zlib/zlibx 6, lz4 ignored. |
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlib`/`zlibx` share the same literal-only zlib path. |
| `--zl <n>` | Alias for `--compress-level`. |
| `--skip-compress <list>` | Skip compression for `/`- or `,`-separated suffixes; defaults to rsync 3.4.1's built-in list. Incompatible with `--chunk-serialization`. |
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
@@ -564,6 +571,7 @@ remote SSH argv is already built injection-safe.
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`). |
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination. |
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win). |
| `--verify-basis` | FastSync-only: require a basis hit to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync). |
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
| `--append-verify` | Like `--append`, but verifies the retained prefix checksum first (falls back to a full transfer on mismatch). |
@@ -571,6 +579,7 @@ remote SSH argv is already built injection-safe.
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
| `--delete-during`, `--del` | Delete extras once the keep-set manifest is known, before data is applied (implies `--delete`; early mode, same engine behaviour as `--delete-before`). |
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`; commit mode, same behaviour as `--delete-after`). |
| `--delete-commit` | FastSync-only: atomic delete-after timing (only after the whole transfer succeeded). |
| `--delete-after` | Explicit delete-after timing: delete only after the transfer succeeded (implies `--delete`). |
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected). |
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync. |
RSYNC_COMPAT.md
+81 -45
View File
File diff suppressed because one or more lines are too long
src/client/change_list.c
+71 -5
View File
@@ -75,6 +75,14 @@ bool change_list_enabled(const Config* config) {
(config->info_level & LOG_INFO_NAME) != 0);
}
/* Emitted once, lazily, ahead of the first --info=name entry: rsync prints the
* transfer-root `./` name line when the root directory is (re)created. */
static bool name_root_printed = false;
void change_reset_name_root(void) {
name_root_printed = false;
}
/* ---- Itemize code ---- */
/* Format the permission bits as an `ls -l` string, e.g. `-rw-r--r--`. */
@@ -212,6 +220,23 @@ static char* change_render_name(const ChangeEvent* event) {
return line.data;
}
/* rsync's `--info=name2` line for an unchanged entry: `NAME is uptodate`. */
static char* change_render_name_uptodate(const ChangeEvent* event) {
char* name = change_render_name(event);
if (name == NULL)
return NULL;
size_t length = strlen(name);
char* line = malloc(length + sizeof(" is uptodate"));
if (line == NULL) {
free(name);
return NULL;
}
memcpy(line, name, length);
memcpy(line + length, " is uptodate", sizeof(" is uptodate"));
free(name);
return line;
}
/* ---- --out-format / --log-file-format ---- */
/* rsync 3.4.1's `%C` uses the negotiated TRANSFER checksum (the first name of a
@@ -461,10 +486,24 @@ static void print_escaped_line(FILE* stream, const char* line, bool eight_bit_ou
void change_emit(const Config* config, const ChangeEvent* event) {
if (event == NULL || !change_list_enabled(config))
return;
if (event->decision == CHANGE_UP_TO_DATE)
return;
bool to_stdout = config->itemize_changes || config->out_format != NULL;
bool to_log = config->log_file != NULL && config->log_file_format != NULL;
bool progress_active = config->show_progress || (config->info_level & LOG_INFO_PROGRESS);
if (event->decision == CHANGE_UP_TO_DATE) {
/* --info=name2 prints `NAME is uptodate` for entries the receiver already
had. An itemize/out-format run reports them through its own format (or
not at all), the progress stream has no frame for them, and neither the
itemize nor the log-file stream previously reported an up-to-date entry,
so nothing else here changes. */
if (!to_stdout && (config->info_level & LOG_INFO_NAME_UPTODATE) != 0 && !progress_active) {
char* line = change_render_name_uptodate(event);
if (line != NULL) {
print_escaped_line(stdout, line, config->eight_bit_output);
free(line);
}
}
return;
}
if (to_stdout) {
char* line = config->out_format != NULL
? change_render_format(config->out_format, config, event)
@@ -473,11 +512,15 @@ void change_emit(const Config* config, const ChangeEvent* event) {
print_escaped_line(stdout, line, config->eight_bit_output);
free(line);
}
} else if ((config->info_level & LOG_INFO_NAME) != 0 &&
!(config->show_progress || (config->info_level & LOG_INFO_PROGRESS))) {
} else if ((config->info_level & LOG_INFO_NAME) != 0 && !progress_active) {
/* --info=name without -i/--out-format: print the updated entry's name. The
--progress path owns the name line when progress output is active (it
emits the same names before the progress frames), so do not duplicate. */
emits the same names before the progress frames), so do not duplicate.
The transfer-root `./` line precedes the first such name. */
if (!name_root_printed) {
name_root_printed = true;
fputs("./\n", stdout);
}
char* line = change_render_name(event);
if (line != NULL) {
print_escaped_line(stdout, line, config->eight_bit_output);
@@ -643,6 +686,29 @@ void change_emit_file_sent(const Config* config, const File* file) {
change_emit_file_sent_bytes(config, file, payload, 0);
}
void change_emit_file_uptodate(const Config* config, const File* file) {
if (file == NULL || !change_list_enabled(config))
return;
ChangeEvent event;
memset(&event, 0, sizeof(event));
event.decision = CHANGE_UP_TO_DATE;
event.is_directory = false;
event.is_symlink = file->is_symlink;
event.is_special = file->is_special;
event.is_hardlink = file->link_group != 0 && !file->link_first;
event.symlink_target = file->symlink_target;
event.hardlink_target = file->hardlink_target;
event.size = file->data != NULL ? file->data->size : 0;
event.dest = file->dest_state;
char* name = NULL;
char* path = NULL;
fill_event_from_file(config, file, &event, &name, &path);
if (name != NULL && path != NULL)
change_emit(config, &event);
free(name);
free(path);
}
void change_emit_dir_sent(const Config* config, const File* file) {
if (file == NULL || !change_list_enabled(config))
return;
src/client/change_list.h
+9
View File
@@ -102,4 +102,13 @@ void change_emit_file_sent(const Config* config, const File* file);
/* Build and emit a CHANGE_SENT event for an explicit directory entry (-d). */
void change_emit_dir_sent(const Config* config, const File* file);
/* Build and emit a CHANGE_UP_TO_DATE event for a file the receiver already had.
* With --info=name2 it renders rsync's "NAME is uptodate" line (no output
* otherwise). */
void change_emit_file_uptodate(const Config* config, const File* file);
/* Reset the lazy transfer-root `./` line emitted ahead of the first
* --info=name entry. Call once at the start of a transfer. */
void change_reset_name_root(void);
#endif
src/client/client_cli.c
+101 -12
View File
@@ -161,10 +161,16 @@ static int set_compression_choice(Config* config, const char* value) {
return -1;
}
int algo;
if (strcasecmp(value, "auto") == 0)
algo = (int)compression_negotiate_default();
else
if (strcasecmp(value, "auto") == 0) {
algo = compression_choice_resolve();
if (algo < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
config->cli_exit_code = 4;
return -1;
}
} else {
algo = compression_algo_from_name(value);
}
if (algo < 0) {
log_message(LOG_LEVEL_ERROR,
"--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, "
@@ -228,16 +234,25 @@ static int set_checksum_choice(Config* config, const char* value) {
config->cli_exit_code = 4;
return -1;
}
ChecksumAlgo negotiated = checksum_negotiate_default();
int negotiated = -1;
if (rc1 == 1 || rc2 == 1) {
negotiated = checksum_choice_resolve();
if (negotiated < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
config->cli_exit_code = 4;
return -1;
}
}
if (rc1 == 1)
transfer = (int)negotiated;
transfer = negotiated;
if (!name2)
pre = transfer;
else if (rc2 == 1)
pre = (int)negotiated;
pre = negotiated;
config->checksum_algo = pre;
config->checksum_transfer_algo = transfer;
config->checksum_choice_set = true;
/* rsync: "none" for the transfer checksum forces --whole-file. */
if (transfer == (int)CHECKSUM_ALGO_NONE)
config->whole_file = true;
@@ -614,9 +629,19 @@ static int parse_info_flags(const char* value, Config* config) {
}
if (strcmp(name, "copy") == 0)
flag = LOG_INFO_COPY;
else if (strcmp(name, "name") == 0)
flag = LOG_INFO_NAME;
else if (strcmp(name, "misc") == 0)
else if (strcmp(name, "name") == 0) {
/* name level 2 adds rsync's "is uptodate" lines. */
if (level == 0)
parsed &= ~(uint32_t)(LOG_INFO_NAME | LOG_INFO_NAME_UPTODATE);
else {
parsed |= LOG_INFO_NAME;
if (level >= 2)
parsed |= LOG_INFO_NAME_UPTODATE;
else
parsed &= ~(uint32_t)LOG_INFO_NAME_UPTODATE;
}
continue;
} else if (strcmp(name, "misc") == 0)
flag = LOG_INFO_MISC;
else if (strcmp(name, "skip") == 0)
flag = LOG_INFO_SKIP;
@@ -972,6 +997,12 @@ static const OptionEntry OPTION_TABLE[] = {
{"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)},
{"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)},
{"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)},
/* FastSync-only long spelling of the late whole-tree commit, which selects
the same timing as rsync's --delete-after in FastSync (the whole-tree
keep-set manifest is committed only after the entire transfer succeeded).
Plain --delete now defaults to delete-during, so this restores the old
FastSync behavior; it maps onto the same delete_after wire field. */
{"--delete-commit", NULL, OPT_FLAG, offsetof(Config, delete_after)},
{"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)},
{"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)},
{"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)},
@@ -1029,6 +1060,11 @@ static const OptionEntry OPTION_TABLE[] = {
* --remote-option is parsed. --trust-sender is a local receiver policy and
* never travels to the remote peer. */
{"--trust-sender", NULL, OPT_FLAG, offsetof(Config, trust_sender)},
/* FastSync-only (not an rsync option): require a basis-hit's content to
* match the source by whole-file digest instead of trusting rsync's
* size+mtime quick-check. Long-only; crosses the wire so the receiver
* performs the extra read/hash. */
{"--verify-basis", NULL, OPT_FLAG, offsetof(Config, verify_basis)},
};
/* Only boolean options with no required argument are safe to negate. */
@@ -1071,6 +1107,7 @@ static const NegatableOption NEGATABLE_OPTIONS[] = {
{"xattrs", "X", offsetof(Config, preserve_xattrs)},
{"acls", "A", offsetof(Config, preserve_acls)},
{"fake-super", NULL, offsetof(Config, fake_super)},
{"verify-basis", NULL, offsetof(Config, verify_basis)},
};
static bool opt_is(const char* arg, const char* name, const char* alias) {
@@ -1454,6 +1491,8 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
ctx->exit_code = -1;
return true;
}
if (entry->offset == offsetof(Config, compression_level))
config->compression_level_set = true;
if (entry->offset == offsetof(Config, chmod_spec)) {
mode_t ignored;
if (!chmod_apply(0, config->chmod_spec, &ignored)) {
@@ -1480,7 +1519,9 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX)
config->per_dir_filter_count++;
/* A delete-timing flag selects when --delete removes extras, so it
implies --delete exactly like the rsync options do. */
implies --delete exactly like the rsync options do. --delete-commit (the
FastSync-only late-commit spelling) is mapped onto delete_after and so is
covered here too. */
if (entry->offset == offsetof(Config, delete_before) ||
entry->offset == offsetof(Config, delete_during) ||
entry->offset == offsetof(Config, delete_delay) ||
@@ -1738,6 +1779,7 @@ static bool cli_handle_transfer_flags(CliParseCtx* ctx) {
return true;
}
config->compression_level = (int)level;
config->compression_level_set = true;
log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level);
ctx->i++;
}
@@ -2533,6 +2575,18 @@ static bool cli_handle_outbuf_option(CliParseCtx* ctx) {
* -1 on error. */
static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) {
set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
/* rsync's plain --delete defaults to delete-during (--del): each directory's
extras are removed as that directory is processed, so space is freed
progressively and a tight destination never has to hold the whole old+new
tree at once. The late whole-tree commit FastSync historically used is
still selected explicitly by --delete-after or by the FastSync-only long
spelling --delete-commit (an exact alias for --delete-after). Resolve the
default on the client, before validation and before the config crosses the
wire, so exactly one timing flag is ever set; an explicit timing (including
--delete-commit) always wins. */
if (config->use_delete && !config->delete_before && !config->delete_during &&
!config->delete_delay && !config->delete_after)
config->delete_during = true;
if (config->compress_choice) {
int algo = compression_algo_from_name(config->compress_choice);
if (algo >= 0) {
@@ -2540,8 +2594,43 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE);
}
}
if (config->use_compression && config->compression_algo == (int)COMPRESSION_ALGO_NONE)
config->compression_algo = (int)compression_negotiate_default();
/* A bare -z (no --compress-choice) resolves like rsync's "auto": the
* RSYNC_COMPRESS_LIST preference list first, then the compiled-in order. A
* list that names no supported codec is rsync's failed negotiation (exit 4). */
if (config->use_compression && !config->compress_choice) {
int resolved = compression_choice_resolve();
if (resolved < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
config->cli_exit_code = 4;
return -1;
}
config->compression_algo = resolved;
if (resolved == (int)COMPRESSION_ALGO_NONE)
config->use_compression = false;
}
/* Apply rsync's per-codec compression level: an explicit --compress-level is
* clamped to the codec's range, otherwise the codec's own default is used. */
if (config->use_compression) {
CompressionAlgo algo = (CompressionAlgo)config->compression_algo;
config->compression_level = config->compression_level_set
? compression_clamp_level(algo, config->compression_level)
: compression_default_level(algo);
log_debug_message(LOG_DEBUG_UTIL, "Client compression: %s (level %d)",
compression_algo_name(algo), config->compression_level);
}
/* The negotiated checksum is always resolved (rsync negotiates one for the
* delta strong sum even without --checksum): RSYNC_CHECKSUM_LIST first, then
* the compiled-in order. An explicit --checksum-choice already set it. */
if (!config->checksum_choice_set) {
int resolved = checksum_choice_resolve();
if (resolved < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
config->cli_exit_code = 4;
return -1;
}
config->checksum_algo = resolved;
config->checksum_transfer_algo = resolved;
}
/* rsync parity: "none" as the pre-transfer checksum cannot be combined with
* --checksum (exit 4). The check runs here because --checksum may appear on
* either side of --checksum-choice. */
src/client/client_send.c
+408 -126
View File
@@ -93,12 +93,11 @@ static const char* stats_bytes(const Config* config, unsigned long long bytes, c
return buffer;
}
/* Build rsync's `Number of files` parenthetical: each non-zero category, in
reg/dir/link/special order. Empty when the flist counted nothing. */
static void stats_type_breakdown(const TransferStats* stats, char* out, size_t out_size) {
unsigned long long total =
stats->flist_reg + stats->flist_dir + stats->flist_link + stats->flist_special;
if (total == 0) {
/* Build rsync's per-type parenthetical: each non-zero category, in
reg/dir/link/special order. Empty when every count is zero. */
static void type_breakdown(unsigned long long reg, unsigned long long dir, unsigned long long link,
unsigned long long special, char* out, size_t out_size) {
if (reg + dir + link + special == 0) {
out[0] = '\0';
return;
}
@@ -107,10 +106,7 @@ static void stats_type_breakdown(const TransferStats* stats, char* out, size_t o
const struct {
const char* name;
unsigned long long count;
} parts[4] = {{"reg", stats->flist_reg},
{"dir", stats->flist_dir},
{"link", stats->flist_link},
{"special", stats->flist_special}};
} parts[4] = {{"reg", reg}, {"dir", dir}, {"link", link}, {"special", special}};
bool first = true;
for (size_t i = 0; i < 4; i++) {
if (parts[i].count == 0)
@@ -127,6 +123,12 @@ static void stats_type_breakdown(const TransferStats* stats, char* out, size_t o
out[used] = '\0';
}
/* Build rsync's `Number of files` parenthetical from the scan's flist counts. */
static void stats_type_breakdown(const TransferStats* stats, char* out, size_t out_size) {
type_breakdown(stats->flist_reg, stats->flist_dir, stats->flist_link, stats->flist_special, out,
out_size);
}
/* Print the rsync `--stats` block on stdout. The source-side flist and
transferred counters come from `stats` (filled while scanning/sending), the
receiver-only counters from the STATUS_STATS frame, and the wire byte totals
@@ -151,6 +153,7 @@ static void report_transfer_stats(const Config* config, const TransferStats* sta
char total_buffer[32];
char transferred_buffer[32];
char literal_buffer[32];
char matched_buffer[32];
char sent_buffer[32];
char recv_buffer[32];
char rate_buffer[32] = {0};
@@ -159,8 +162,15 @@ static void report_transfer_stats(const Config* config, const TransferStats* sta
stats_bytes(config, stats->total_file_size, total_buffer, sizeof(total_buffer));
const char* transferred = stats_bytes(config, stats->transferred_file_size, transferred_buffer,
sizeof(transferred_buffer));
const char* literal =
stats_bytes(config, stats->literal_data, literal_buffer, sizeof(literal_buffer));
/* Protocol 2.28.0: the receiver reports the bytes it literally stored, which
is exact for a delta transfer (the sender's own literal_data counts each
stored file's whole source size and is only an upper bound). Fall back to
the sender total when the receiver reported no delta/literal accounting
(e.g. a local no-server path). */
unsigned long long literal_bytes = (recv->literal_bytes != 0 || recv->matched_data != 0)
? recv->literal_bytes
: stats->literal_data;
const char* literal = stats_bytes(config, literal_bytes, literal_buffer, sizeof(literal_buffer));
const char* sent_s = stats_bytes(config, sent, sent_buffer, sizeof(sent_buffer));
const char* recv_s = stats_bytes(config, received, recv_buffer, sizeof(recv_buffer));
const char* rate_str = rate_buffer;
@@ -177,21 +187,30 @@ static void report_transfer_stats(const Config* config, const TransferStats* sta
stats_type_breakdown(stats, breakdown, sizeof(breakdown));
unsigned long long flist_total =
stats->flist_reg + stats->flist_dir + stats->flist_link + stats->flist_special;
char created_breakdown[128];
type_breakdown(recv->created_reg, recv->created_dir, recv->created_link, recv->created_special,
created_breakdown, sizeof(created_breakdown));
unsigned long long created_total =
recv->created_reg + recv->created_dir + recv->created_link + recv->created_special;
printf("\n");
if (breakdown[0] != '\0')
printf("Number of files: %llu %s\n", flist_total, breakdown);
else
printf("Number of files: %llu\n", flist_total);
/* FastSync cannot tell which entries the receiver newly created, so it
reports the transferred regular files (which are created on a fresh
destination). See RSYNC_COMPAT.md for the documented residual. */
printf("Number of created files: %llu\n", stats->transferred_regular);
/* Protocol 2.28.0: the receiver reports which destination entries it newly
created, split by type, so this line matches rsync exactly. */
if (created_breakdown[0] != '\0')
printf("Number of created files: %llu %s\n", created_total, created_breakdown);
else
printf("Number of created files: %llu\n", created_total);
printf("Number of deleted files: %llu\n", recv->deleted_files);
printf("Number of regular files transferred: %llu\n", stats->transferred_regular);
printf("Total file size: %s bytes\n", total);
printf("Total transferred file size: %s bytes\n", transferred);
printf("Literal data: %s bytes\n", literal);
printf("Matched data: %llu bytes\n", recv->matched_data);
const char* matched =
stats_bytes(config, recv->matched_data, matched_buffer, sizeof(matched_buffer));
printf("Matched data: %s bytes\n", matched);
printf("File list size: 0\n");
printf("File list generation time: 0.000 seconds\n");
printf("File list transfer time: 0.000 seconds\n");
@@ -256,10 +275,61 @@ static void transfer_stats_note_transferred(TransferStats* stats, const File* fi
static const char* delete_display_path(const Config* config, const char* path);
/* Paths-only pre-count of the source file list, built once at transfer start
* when progress output is requested. rsync's `to-chk` denominator is the whole
* file list -- every regular file, directory, symlink and special plus the
* transfer root -- while the streaming scan never emits directories. A
* metadata-only walk (no file reads, no hashing) supplies that total and the
* directory names, so the opt-in pass leaves non-progress runs untouched. */
typedef struct {
unsigned long long total;
ArrayList* dir_paths; /* owned char* in transfer-relative display form */
} ProgressPrecount;
static bool g_progress_active;
static unsigned long long g_progress_xferred;
static unsigned long long g_progress_seen;
static unsigned long long g_progress_index;
static unsigned long long g_progress_total;
static struct timespec g_progress_file_start;
static ProgressPrecount g_progress_precount;
static PathIndex g_progress_dir_index;
static bool g_progress_dir_index_valid;
static StrHashSet g_progress_emitted;
static bool g_progress_emitted_valid;
static ArrayList* g_progress_emitted_keys;
static bool progress_requested(const Config* config) {
return config != NULL && !config->quiet &&
(config->show_progress || (config->info_level & LOG_INFO_PROGRESS) != 0);
}
static void progress_precount_dispose(ProgressPrecount* p) {
if (p->dir_paths != NULL) {
array_list_delete(p->dir_paths);
p->dir_paths = NULL;
}
p->total = 0;
}
static void client_progress_cleanup(void) {
if (g_progress_dir_index_valid) {
path_index_free(&g_progress_dir_index);
g_progress_dir_index_valid = false;
}
if (g_progress_emitted_valid) {
str_hash_set_free(&g_progress_emitted);
g_progress_emitted_valid = false;
}
if (g_progress_emitted_keys != NULL) {
array_list_delete(g_progress_emitted_keys);
g_progress_emitted_keys = NULL;
}
progress_precount_dispose(&g_progress_precount);
g_progress_active = false;
g_progress_total = 0;
g_progress_index = 0;
g_progress_xferred = 0;
}
static void progress_first_frame(unsigned long long size, char* out, size_t out_size) {
char ofs_buf[32];
@@ -296,12 +366,12 @@ static void progress_final_frame(unsigned long long size, char* out, size_t out_
unsigned long long remain = (unsigned long long)(diff_ms / 1000);
snprintf(rembuf, sizeof(rembuf), "%4u:%02u:%02u", (unsigned)(remain / 3600),
(unsigned)((remain / 60) % 60), (unsigned)(remain % 60));
/* rsync's `to-chk` denominator is the whole file list, which includes the
transfer-root directory FastSync never emits as a transfer entry. Count
that root entry so a single-file transfer matches rsync exactly. */
unsigned long long total = g_progress_seen + 1;
unsigned long long to_chk =
g_progress_seen > g_progress_xferred ? g_progress_seen - g_progress_xferred : 0;
/* rsync's `to-chk` denominator is the whole file list (the pre-count); the
numerator falls as each entry is processed, root first. Without a
pre-count (the paths-only walk failed) fall back to the transferred-file
count so the single-file layout stays intact. */
unsigned long long total = g_progress_total > 0 ? g_progress_total : g_progress_xferred + 1;
unsigned long long to_chk = total > g_progress_index ? total - g_progress_index - 1 : 0;
snprintf(out, out_size, "\r%15s %3d%% %7.2f%s %s (xfr#%llu, to-chk=%llu/%llu)\n", ofs_buf, 100,
rate, units, rembuf, g_progress_xferred, to_chk, total);
}
@@ -349,11 +419,73 @@ static void print_delete_reports(const Config* config, const ArrayList* paths) {
fflush(stdout);
}
static void client_progress_emit_ancestors(const Config* config, const char* rel) {
if (!g_progress_dir_index_valid || !g_progress_emitted_valid || g_progress_emitted_keys == NULL ||
rel == NULL)
return;
size_t rel_len = strlen(rel);
for (size_t i = 0; i < rel_len; i++) {
if (rel[i] != '/')
continue;
char* prefix = malloc(i + 1);
if (prefix == NULL)
return;
memcpy(prefix, rel, i);
prefix[i] = '\0';
if (path_index_contains(&g_progress_dir_index, prefix) &&
!str_hash_set_lookup(&g_progress_emitted, prefix)) {
char* key = str_dup(prefix);
if (key != NULL && array_list_add(g_progress_emitted_keys, key)) {
str_hash_set_insert_ref(&g_progress_emitted, key);
char* escaped = output_escape(prefix, config->eight_bit_output);
printf("%s/\n", escaped ? escaped : prefix);
free(escaped);
g_progress_index++;
} else {
free(key);
}
}
free(prefix);
}
}
/* rsync's --info=name/progress line for one entry: transfer-relative name (a
* trailing slash for directories) plus the ` -> target` symlink suffix. */
static char* progress_entry_line(const File* file, const char* rel) {
const char* arrow = NULL;
const char* target = NULL;
if (file->is_symlink && file->symlink_target != NULL) {
arrow = " -> ";
target = file->symlink_target;
} else if (file->link_group != 0 && !file->link_first && file->hardlink_target != NULL) {
arrow = " => ";
target = file->hardlink_target;
}
size_t rel_len = strlen(rel);
bool dir_slash = file->is_dir && (rel_len == 0 || rel[rel_len - 1] != '/');
size_t extra = (dir_slash ? 1u : 0u) + (target != NULL ? 4u + strlen(target) : 0u);
char* line = malloc(rel_len + extra + 1);
if (line == NULL)
return NULL;
memcpy(line, rel, rel_len);
size_t off = rel_len;
if (dir_slash)
line[off++] = '/';
if (target != NULL) {
memcpy(line + off, arrow, 4);
off += 4;
memcpy(line + off, target, strlen(target));
off += strlen(target);
}
line[off] = '\0';
return line;
}
static void client_progress_begin(const Config* config) {
g_progress_active =
(config->show_progress || info_flag_enabled(config, LOG_INFO_PROGRESS)) && !config->quiet;
change_reset_name_root();
g_progress_active = progress_requested(config);
g_progress_xferred = 0;
g_progress_seen = 0;
g_progress_index = 1; /* the transfer root is file-list entry #0 */
if (!g_progress_active) {
/* `--info=flist` prints rsync's file-list header even without progress. */
if (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST)) {
@@ -375,12 +507,14 @@ static void client_progress_begin(const Config* config) {
static void client_progress_file(const Config* config, const File* file) {
if (!g_progress_active || file == NULL || !file->data)
return;
g_progress_seen++;
g_progress_xferred++;
unsigned long long size = file->data->size;
if (!config->itemize_changes && config->out_format == NULL) {
const char* name = delete_display_path(config, file_wire_path(file));
printf("%s\n", name ? name : "");
const char* rel = delete_display_path(config, file_wire_path(file));
client_progress_emit_ancestors(config, rel);
char* escaped = output_escape(rel, config->eight_bit_output);
printf("%s\n", escaped ? escaped : (rel ? rel : ""));
free(escaped);
}
clock_gettime(CLOCK_MONOTONIC, &g_progress_file_start);
char frame[160];
@@ -388,9 +522,41 @@ static void client_progress_file(const Config* config, const File* file) {
fputs(frame, stdout);
progress_final_frame(size, frame, sizeof(frame));
fputs(frame, stdout);
g_progress_index++;
fflush(stdout);
}
/* Emit the name line for a transferred non-regular entry (directory, symlink,
* special or hard-link sibling): rsync prints these in the file list but has no
* progress frame for them. */
static void client_progress_name(const Config* config, const File* file) {
if (!g_progress_active || file == NULL)
return;
const char* rel = delete_display_path(config, file_wire_path(file));
if (!config->itemize_changes && config->out_format == NULL) {
client_progress_emit_ancestors(config, rel);
char* line = progress_entry_line(file, rel ? rel : "");
if (line != NULL) {
char* escaped = output_escape(line, config->eight_bit_output);
printf("%s\n", escaped ? escaped : line);
free(escaped);
free(line);
fflush(stdout);
}
}
g_progress_index++;
}
/* An entry the receiver already had prints no name under --progress but still
* occupies a file-list slot in the `to-chk` numerator. */
static void client_progress_uptodate(const Config* config, const File* file) {
(void)config;
(void)file;
if (!g_progress_active)
return;
g_progress_index++;
}
/* Compiled scanner inputs that are shared read-only across scanner instances
* and, in -m mode, across worker threads. `base_filters` owns the compiled
* command-line + -C rules; the FileListSet allow-set lives in the Config.
@@ -535,6 +701,126 @@ static void prepared_scanner_destroy(PreparedScanner* prepared) {
prepared->relative_prefix = NULL;
}
static bool progress_precount_add_dir(ProgressPrecount* p, const char* path) {
if (path == NULL || path[0] == '\0')
return true;
char* dup = str_dup(path);
if (dup == NULL)
return false;
if (array_list_add(p->dir_paths, dup))
return true;
free(dup);
return false;
}
/* Metadata-only walk collecting the full file-list total and every directory
* name. It uses its own scanner (fresh filter compilation and hard-link table)
* so the data pass's link-group state is never perturbed. */
static bool progress_precount_scan(const Config* config, ProgressPrecount* out) {
out->dir_paths = array_list_create(free);
if (out->dir_paths == NULL)
return false;
out->total = 0;
PreparedScanner prepared;
memset(&prepared, 0, sizeof(prepared));
if (!prepare_scanner(config, 0, &prepared)) {
progress_precount_dispose(out);
return false;
}
ScannerOptions local = prepared.options;
local.list_dirs = true;
local.note_nonreg = false;
local.use_metadata = false;
local.preserve_xattrs = false;
local.preserve_acls = false;
local.checksum = false;
local.capture_dir_times = false;
local.excluded_paths = NULL;
local.size_skipped_paths = NULL;
local.synced_dirs = NULL;
local.plan_dirs = NULL;
local.dir_entries = NULL;
local.dir_entries_mutex = NULL;
local.hardlinks = NULL;
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
bool ok = scanner != NULL;
if (scanner != NULL) {
Chunk* chunk;
while (ok && (chunk = directory_scanner_next(scanner)) != NULL) {
out->total += (unsigned long long)chunk->element_count;
for (int i = 0; i < chunk->element_count && ok; i++) {
const File* f = chunk->items[i];
if (f != NULL && f->is_dir)
ok = progress_precount_add_dir(out, delete_display_path(config, file_wire_path(f)));
}
chunk_destroy(chunk);
}
if (ok && directory_scanner_failed(scanner))
ok = false;
directory_scanner_destroy(scanner);
}
prepared_scanner_destroy(&prepared);
if (!ok) {
progress_precount_dispose(out);
return false;
}
out->total += 1; /* the transfer root "." */
return true;
}
/* Reuse the --delete-during/--delete-delay keep-set pre-scan: its traversed
* directory list already holds every directory and `non_dir_count` the entries
* counted during that same pass, so progress costs no second walk. */
static bool progress_precount_from_plan_dirs(const Config* config, const ArrayList* plan_dirs,
unsigned long long non_dir_count,
ProgressPrecount* out) {
out->dir_paths = array_list_create(free);
if (out->dir_paths == NULL)
return false;
out->total = non_dir_count + 1;
for (int i = 0; i < plan_dirs->size; i++) {
const char* path = (const char*)plan_dirs->items[i];
const char* rel = config->send_directory != NULL
? utils_strip_transfer_root(path, config->send_directory)
: path;
if (!progress_precount_add_dir(out, rel)) {
progress_precount_dispose(out);
return false;
}
}
out->total += (unsigned long long)out->dir_paths->size;
return true;
}
/* Build the optional progress pre-count. A failed pre-count is non-fatal: the
* transfer proceeds and the progress denominator falls back to the transferred
* file count. */
static void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
unsigned long long plan_non_dir_count) {
client_progress_cleanup();
g_progress_active = progress_requested(config);
if (!g_progress_active)
return;
bool ok = plan_dirs != NULL ? progress_precount_from_plan_dirs(
config, plan_dirs, plan_non_dir_count, &g_progress_precount)
: progress_precount_scan(config, &g_progress_precount);
if (!ok) {
g_progress_total = 0;
return;
}
g_progress_total = g_progress_precount.total;
if (g_progress_precount.dir_paths != NULL && g_progress_precount.dir_paths->size > 0 &&
path_index_build(&g_progress_dir_index,
(const char* const*)g_progress_precount.dir_paths->items,
(size_t)g_progress_precount.dir_paths->size))
g_progress_dir_index_valid = true;
if (str_hash_set_init(&g_progress_emitted, (size_t)(g_progress_precount.dir_paths != NULL
? g_progress_precount.dir_paths->size + 1
: 1)))
g_progress_emitted_valid = true;
g_progress_emitted_keys = array_list_create(free);
}
/* -R/--relative implied directories: rsync transmits the metadata of the
* parent directories implied by the source path (every prefix component above
* the source root) so the receiver applies their attributes to the created
@@ -761,53 +1047,6 @@ static bool files_from_list_check(const Config* config, ArrayList* missing_dest,
return true;
}
/* Basis directories are honored by the receiver's per-file incremental check,
which (like every whole-file payload path in FastSync) is bounded by
MAX_RECEIVE_WHOLE_FILE_SIZE. rsync would apply basis dirs to files of any
size; FastSync cannot, so when basis dirs are requested this preflight scan
refuses the run up front with a clear diagnostic instead of letting the
receiver abort the whole transfer mid-stream with no client explanation.
Returns true when the tree can be transferred. */
static bool basis_oversize_preflight(const Config* config) {
PreparedScanner prepared;
if (!prepare_scanner(config, 0, &prepared))
return false;
DirectoryScanner* scanner =
directory_scanner_create_with_options(config->send_directory, &prepared.options);
if (!scanner) {
prepared_scanner_destroy(&prepared);
return false;
}
bool ok = true;
Chunk* chunk;
while ((chunk = directory_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count; i++) {
File* f = chunk->items[i];
if (f == NULL || f->is_dir || f->data == NULL || f->data->size <= MAX_RECEIVE_WHOLE_FILE_SIZE)
continue;
char* escaped = output_escape(file_wire_path(f), config->eight_bit_output);
log_message(LOG_LEVEL_ERROR,
"%s is %llu bytes, larger than the %llu-byte whole-file transfer limit; "
"--compare-dest/--copy-dest/--link-dest cannot sync files above this limit",
escaped ? escaped : "<allocation failed>", (unsigned long long)f->data->size,
(unsigned long long)MAX_RECEIVE_WHOLE_FILE_SIZE);
free(escaped);
ok = false;
break;
}
chunk_destroy(chunk);
if (!ok)
break;
}
if (directory_scanner_failed(scanner) || directory_scanner_had_io_error(scanner))
ok = false;
/* The scanner borrows prepared.options' base_filters/hardlinks pointers, so
prepared must outlive the scanner. */
directory_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
return ok;
}
/* Read the daemon's MOTD frame and, unless --no-motd, display it on stdout.
*
* The daemon sends the MOTD as the first thing after the config-frame STATUS_OK
@@ -1549,9 +1788,12 @@ static bool send_delete_manifest_early(Client* client, ArrayList* manifest,
directory and *io_error_out reports it (the caller still performs the
deletion but reports the run as errored). */
static bool scan_paths_only(const Config* config, const ScannerOptions* options,
ArrayList* manifest, DeletePlanSender* plans, bool* io_error_out) {
ArrayList* manifest, DeletePlanSender* plans, bool* io_error_out,
unsigned long long* non_dir_count_out) {
if (io_error_out)
*io_error_out = false;
if (non_dir_count_out)
*non_dir_count_out = 0;
ScannerOptions local = *options;
/* The pre-scan is a paths-only pass with no client output; it must not emit
--info=nonreg lines (the data pass does that once). */
@@ -1562,6 +1804,13 @@ static bool scan_paths_only(const Config* config, const ScannerOptions* options,
bool ok = true;
Chunk* chunk;
while ((chunk = directory_scanner_next(scanner)) != NULL) {
if (non_dir_count_out) {
for (int i = 0; i < chunk->element_count; i++) {
const File* f = chunk->items[i];
if (f && !f->is_dir)
(*non_dir_count_out)++;
}
}
if (manifest && !add_chunk_to_manifest(manifest, chunk)) {
ok = false;
chunk_destroy(chunk);
@@ -1643,11 +1892,13 @@ static int incremental_check(Client* client, File* file, const Config* config,
return -1;
if (!send_n_data(client->file_descriptor, &mtime_nsec, sizeof(mtime_nsec)))
return -1;
/* With alternate basis directories the receiver must be able to verify the
* content of every candidate basis file, so the sender supplies its whole-file
* digest (computed with the negotiated --checksum-choice algorithm and
* --checksum-seed) for every file even when --checksum was not requested. */
if (config->checksum || config_has_basis(config)) {
/* The whole-file digest (negotiated --checksum-choice algorithm and
* --checksum-seed) is only needed when it drives a decision: --checksum's
* per-file quick check, or a --verify-basis content equality. Under the
* default metadata quick-check the receiver never reads it, so the sender
* skips the full-file read/hash exactly as rsync does for a plain
* --link-dest run. */
if (config->checksum || config->verify_basis) {
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
size_t digest_len = 0;
if (!file_checksum(file, (ChecksumAlgo)config->checksum_algo, config->checksum_seed, digest,
@@ -1658,6 +1909,15 @@ static int incremental_check(Client* client, File* file, const Config* config,
!send_n_data(client->file_descriptor, digest, wire_len))
return -1;
}
/* Basis directories: the receiver materializes a hit from the basis without a
* data frame, so it would otherwise only have the basis inode's metadata.
* Transmit the SOURCE metadata with the check (rsync's copy-then-fix) so a
* --copy-dest hit / --link-dest copy fallback applies the source's
* attributes. Symmetric with incremental_check_receive_request. */
if (config_has_basis(config) && config->use_metadata) {
if (!metadata_send(client->file_descriptor, file->metadata))
return -1;
}
Status s;
if (!receive_status(client->file_descriptor, &s))
return -1;
@@ -1897,12 +2157,6 @@ static int send_dry_run_remote(Config* config) {
}
if (missing_args)
array_list_delete(missing_args);
/* Alternate basis dirs force the whole-file per-file check on the real
receiver; refuse an oversize source up front exactly as send_files does so
dry-run reports the same clear diagnostic instead of aborting mid-stream. */
if (config_has_basis(config) && !basis_oversize_preflight(config))
return 1;
/* A live session may follow, so arm graceful abort handling. */
client_set_abort_armed(true);
Client* client = connect_transfer_client(config);
@@ -1928,11 +2182,41 @@ static int send_dry_run_remote(Config* config) {
DirectoryScanner* scanner = NULL;
ArrayList* dry_manifest = NULL;
ArrayList* dry_dirs = NULL;
ArrayList* dry_excluded = NULL;
ArrayList* dry_size_skipped = NULL;
if (!config_send(client->file_descriptor, config))
goto dry_fail;
receive_daemon_motd(client, config);
if (!prepare_scanner(config, 0, &prepared))
goto dry_fail;
/* -n --delete: build the same keep-set manifest, protected prefixes, and
synchronized-directory scope a real run would send, so the receiver's
read-only extras walk enumerates exactly the deletions a real run makes. */
if (config->use_delete) {
dry_manifest = array_list_create(free);
dry_dirs = array_list_create(free);
dry_size_skipped = array_list_create(free);
if (!dry_manifest || !dry_dirs || !dry_size_skipped)
goto dry_fail;
if (!config->delete_excluded) {
dry_excluded = array_list_create(free);
if (!dry_excluded)
goto dry_fail;
prepared.options.excluded_paths = dry_excluded;
}
prepared.options.size_skipped_paths = dry_size_skipped;
/* A --files-from subset confines the extras walk to the directories the
scan synchronized; a full recursive transfer marks the root itself. */
if (config->files_from_set == NULL) {
char* root_marker = delete_scope_root_marker(config);
if (!root_marker || !array_list_add(dry_dirs, root_marker)) {
free(root_marker);
goto dry_fail;
}
} else {
prepared.options.synced_dirs = dry_dirs;
}
}
scanner = directory_scanner_create_with_options(config->send_directory, &prepared.options);
if (!scanner)
goto dry_fail;
@@ -1940,26 +2224,6 @@ static int send_dry_run_remote(Config* config) {
int file_count = 0;
unsigned long long total_bytes = 0;
char size_buffer[32];
/* -n --delete: build the same keep-set manifest a real run would send so the
receiver can enumerate (read-only) the destination extras. Filter-excluded
and size-pruned protections are not propagated here, so a filtered dry-run
may over-report; the no-filter case is exact. */
dry_manifest = config->use_delete ? array_list_create(free) : NULL;
if (config->use_delete && !dry_manifest)
goto dry_fail;
/* Scope the receiver-side extras walk to the receive root (the "." sentinel),
exactly as the recursive transfer path does. */
if (config->use_delete) {
dry_dirs = array_list_create(free);
char* root_marker = dry_dirs ? str_dup(".") : NULL;
if (!dry_dirs || !root_marker || !array_list_add(dry_dirs, root_marker)) {
free(root_marker);
if (dry_dirs)
array_list_delete(dry_dirs);
dry_dirs = NULL;
goto dry_fail;
}
}
if (!config->quiet)
printf("Dry run: files to be transferred\n");
Chunk* chunk;
@@ -2034,8 +2298,8 @@ static int send_dry_run_remote(Config* config) {
the terminal FINISHED. */
bool early_delete = config->use_delete && config_delete_timing_early(config);
if (dry_manifest) {
if (send_delete_manifest(client->file_descriptor, dry_manifest, NULL, NULL, NULL, dry_dirs) !=
0)
if (send_delete_manifest(client->file_descriptor, dry_manifest, dry_excluded, dry_size_skipped,
NULL, dry_dirs) != 0)
goto dry_fail;
if (early_delete) {
Status ack;
@@ -2091,6 +2355,10 @@ dry_fail:
array_list_delete(dry_manifest);
if (dry_dirs)
array_list_delete(dry_dirs);
if (dry_excluded)
array_list_delete(dry_excluded);
if (dry_size_skipped)
array_list_delete(dry_size_skipped);
if (scanner)
directory_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
@@ -2383,6 +2651,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (!send_directory_entry(client, f, config))
return -1;
change_emit_dir_sent(config, f);
client_progress_name(config, f);
continue;
}
/* --hard-links/-H sibling: a later member of a hard-link group that has no
@@ -2396,6 +2665,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
!send_wire_str(client->file_descriptor, f->hardlink_target))
return -1;
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
}
/* Symlink entry (-l / -k keep-as-symlink): only the target rides the wire. */
@@ -2403,6 +2673,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (!send_symlink_entry(client, f, config))
return -1;
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
}
/* --devices/--specials: a device/special node is recreated on the receiver,
@@ -2411,6 +2682,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (!file_send_special(f, client->file_descriptor, config->use_metadata))
return -1;
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
}
bool stream = f->data->data == NULL && f->data->size > 0;
@@ -2423,6 +2695,8 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
int rc = send_single_file(client, f, config, config->use_incremental, use_sendfile);
if (rc == 1) {
source_file_destroy(source);
change_emit_file_uptodate(config, f);
client_progress_uptodate(config, f);
continue;
}
if (rc < 0) {
@@ -2924,11 +3198,6 @@ int send_files(Config* config) {
array_list_delete(missing_args);
return 1;
}
if (config_has_basis(config) && !basis_oversize_preflight(config)) {
if (missing_args)
array_list_delete(missing_args);
return 1;
}
/* From here on a server session may be live, so Ctrl-C/SIGTERM should set the
abort flag (and be forwarded as STATUS_ABORT) instead of terminating. */
@@ -2969,6 +3238,7 @@ int send_files(Config* config) {
bool delete_per_dir = config->use_delete && config_delete_timing_per_dir(config) && !config->dirs;
bool send_failed = false;
bool had_scan_io = false;
unsigned long long per_dir_non_dir_count = 0;
PreparedScanner prepared;
memset(&prepared, 0, sizeof(prepared));
if (!config_send(client->file_descriptor, config))
@@ -3017,7 +3287,8 @@ int send_files(Config* config) {
prepared.options.synced_dirs = synced_dirs;
}
}
/* The late-timing modes (plain --delete / --delete-after) build the manifest
/* The late-timing modes (--delete-after/--delete-commit and a plain --delete
that fell back from per-dir mode because of -d/--dirs) build the manifest
while streaming and send it after the last data frame. --delete-before
sends a whole-tree keep-set up front; --delete-during/--delete-delay build a
per-directory plan set up front (paths only) and stream the plans alongside
@@ -3030,7 +3301,7 @@ int send_files(Config* config) {
if (!early_manifest)
goto send_fail;
bool prescan_ok =
scan_paths_only(config, &prepared.options, early_manifest, NULL, &had_scan_io);
scan_paths_only(config, &prepared.options, early_manifest, NULL, &had_scan_io, NULL);
bool early_ok = false;
bool skip_delete = false;
if (prescan_ok) {
@@ -3072,7 +3343,8 @@ int send_files(Config* config) {
if (!plan_sender || !plan_dirs)
goto send_fail;
prepared.options.plan_dirs = plan_dirs;
bool prescan_ok = scan_paths_only(config, &prepared.options, NULL, plan_sender, &had_scan_io);
bool prescan_ok = scan_paths_only(config, &prepared.options, NULL, plan_sender, &had_scan_io,
&per_dir_non_dir_count);
bool plans_ok = false;
bool skip_delete = false;
if (prescan_ok) {
@@ -3110,6 +3382,11 @@ int send_files(Config* config) {
if (!manifest)
goto send_fail;
}
/* --progress/--info=progress: pre-count the file list for rsync's to-chk
denominator. When a --delete-during/--delete-delay pre-scan already ran,
reuse its traversed directory list instead of walking the tree again. */
if (progress_requested(config))
client_progress_prepare(config, plan_dirs, per_dir_non_dir_count);
/* Phase 6: compute the client-only stop deadline once at transfer start. The
early-delete pre-scan above deliberately ignores it so the keep-set (and
its committed deletion) is always complete and correct. */
@@ -3331,6 +3608,7 @@ send_fail:
if (scanner)
directory_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
client_progress_cleanup();
disconnect_transfer_client(client);
protocol_session_unbind();
client_set_abort_armed(false);
@@ -3358,11 +3636,6 @@ int send_files_multithreaded(Config** config_ptr) {
array_list_delete(missing_args);
return 1;
}
if (config_has_basis(config) && !basis_oversize_preflight(config)) {
if (missing_args)
array_list_delete(missing_args);
return 1;
}
/* Armed only once a session may go live (see send_files). */
client_set_abort_armed(true);
@@ -3414,6 +3687,7 @@ int send_files_multithreaded(Config** config_ptr) {
stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins, config->stop_at_set,
config->stop_at, now_mono);
bool collect_excluded = config->use_delete && !config->delete_excluded;
unsigned long long pre_scan_non_dir = 0;
if (config->use_delete) {
if (collect_excluded) {
context->excluded_paths = array_list_create(free);
@@ -3472,8 +3746,9 @@ int send_files_multithreaded(Config** config_ptr) {
prepared_ok = prepared_ok && context->manifest != NULL;
}
bool prebuilt =
prepared_ok && scan_paths_only(config, &prepared.options, context->manifest,
context->delete_plans, &context->scan_had_io_error);
prepared_ok &&
scan_paths_only(config, &prepared.options, context->manifest, context->delete_plans,
&context->scan_had_io_error, &pre_scan_non_dir);
prepared_scanner_destroy(&prepared);
if (per_dir && prebuilt) {
const char* walk_root = delete_plan_walk_root(config, context->synced_dirs);
@@ -3536,6 +3811,11 @@ int send_files_multithreaded(Config** config_ptr) {
pipeline_context_sender_destroy(context);
return 1;
}
/* --progress/--info=progress: pre-count the file list for rsync's to-chk
denominator, reusing a --delete-during/--delete-delay pre-scan when one
already ran. */
if (progress_requested(config))
client_progress_prepare(config, context->plan_dirs, pre_scan_non_dir);
thrd_t scanner, loader, sender;
bool scanner_created = false;
@@ -3561,6 +3841,7 @@ int send_files_multithreaded(Config** config_ptr) {
if (scanner_created)
thrd_join(scanner, NULL);
pipeline_context_sender_destroy(context);
client_progress_cleanup();
return 1;
}
@@ -3580,6 +3861,7 @@ int send_files_multithreaded(Config** config_ptr) {
transfer (exit 23). A --max-delete-capped commit is a successful transfer
that rsync reports with exit code 25. */
pipeline_context_sender_destroy(context);
client_progress_cleanup();
client_set_abort_armed(false);
if (!sender_ok)
return 1;
src/client/usage.c
+9 -4
View File
@@ -62,8 +62,7 @@ void print_usage(void) {
printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n");
printf(" files (different container format); do not mix the two tools.\n");
printf(" --delete Delete files on receiver not in source\n");
printf(" (default timing: delete only after the whole\n");
printf(" transfer has succeeded)\n");
printf(" (default timing: delete-during, like rsync --del)\n");
printf(" --delete-before Delete extras before the transfer starts\n");
printf(" (implies --delete)\n");
printf(" --delete-during Delete a directory's extras as that directory is\n");
@@ -72,7 +71,9 @@ void print_usage(void) {
printf(" --delete-delay Record the extras during the scan but remove them\n");
printf(" only after a successful transfer (implies --delete)\n");
printf(" --delete-after Delete only after the whole transfer succeeded\n");
printf(" (the default --delete timing; implies --delete)\n");
printf(" (implies --delete)\n");
printf(" --delete-commit FastSync-only: restore the late whole-tree commit\n");
printf(" (identical to --delete-after; implies --delete)\n");
printf(" --delete-excluded Also delete destination files that were excluded on\n");
printf(" the source (default protects them, matching rsync)\n");
printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n");
@@ -93,7 +94,8 @@ void print_usage(void) {
printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n");
printf(" otherwise recreates them, like rsync)\n");
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
printf(" --no-delete (in either order) is rejected as a config error.\n");
printf(" --no-delete (in either order) is rejected as a config error, as is more\n");
printf(" than one timing flag.\n");
printf(" --ignore-existing Skip files that already exist on receiver\n");
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
@@ -138,6 +140,9 @@ void print_usage(void) {
printf(" into the destination instead of transferring its data\n");
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
printf(" into the destination (repeatable; earlier DIRs win)\n");
printf(" --verify-basis FastSync-only: require a basis hit's content to match the\n");
printf(" source by whole-file digest instead of trusting rsync's\n");
printf(" size+mtime (or --size-only) quick-check\n");
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
printf(" --checksum compares. Accepted: xxh128 (default), xxh3, xxh64\n");
printf(" (aka xxhash), md5, md4, sha1, or none. A two-name\n");
src/server/receiver.c
+14 -5
View File
@@ -305,14 +305,16 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
/* Runs the whole receive loop. The delete manifest may legitimately arrive
either FIRST (--delete-before / --delete-during: the sender transmits the
validated keep-set before any file data) or LAST (plain --delete /
--delete-after / --delete-delay: the manifest closes the data stream). In
validated keep-set before any file data) or LAST (--delete-after /
--delete-commit / --delete-delay: the manifest closes the data stream). In
the early modes the receiver deletes as soon as the manifest has been read
and acknowledges with STATUS_OK so the sender only starts streaming once the
deletion has committed (or failed); in the late modes the manifest is held
and the deletion is committed only after the terminal STATUS_FINISHED proves
the whole transfer succeeded. See receiver_process_pending() for how the -m
receiver defers that commit until its disk writer has drained. */
the whole transfer succeeded. A plain --delete defaults to the per-directory
delete-during plan mode (no manifest at all). See
receiver_process_pending() for how the -m receiver defers that commit until
its disk writer has drained. */
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
Status status;
@@ -613,6 +615,8 @@ typedef struct {
static bool receiver_save_file(File* file, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
FileSaveResult result = FILE_SAVE_ERROR;
bool created = false;
unsigned created_dirs = 0;
if (context->config->dry_run) {
/* Defense in depth: a dry-run receiver mutates nothing even if a data
frame reaches the sink (the sender is not supposed to send one). */
@@ -622,12 +626,17 @@ static bool receiver_save_file(File* file, void* context_pointer) {
--remove-source-files sender keeps its source. */
result = FILE_SAVE_SKIPPED;
} else {
result = file_save_to_disk_full(context->config->receive_root_directory, file, context->config);
result = file_save_to_disk_full_ex(context->config->receive_root_directory, file,
context->config, &created, &created_dirs);
}
/* Wire-stats tally: bytes reconstructed from the basis file (delta matches)
count as matched data in the end-of-transfer report. */
if (result != FILE_SAVE_ERROR && file->matched_bytes > 0)
context->stats.matched_data += file->matched_bytes;
/* Protocol 2.28.0: receiver-observed literal bytes and the created-entry
breakdown (regular/dir/link/special) for the `--stats` report. */
if (result == FILE_SAVE_WRITTEN)
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
/* A directory's metadata is deferred, never applied inline: collect it now
and apply it at the end. -O/--omit-dir-times and --preserve_perms/-times
are honored by dir_metadata_list_apply's caller (see
src/server/receiver_pipeline.c
+12 -1
View File
@@ -247,12 +247,23 @@ int write_thread(void* pipeline_context) {
}
size_t file_bytes = file->data ? file->data->size : 0;
FileSaveResult result = FILE_SAVE_SKIPPED;
bool created = false;
unsigned created_dirs = 0;
/* Server-contacting --dry-run: never write. The receiver thread does not
enqueue anything on the dry-run path, but this keeps the writer thread
provably mutation-free if a data frame ever reached it. */
bool dry_run = context->config->dry_run;
if (save_to_disk && !dry_run) {
result = file_save_to_disk_full(root_directory, file, context->config);
result =
file_save_to_disk_full_ex(root_directory, file, context->config, &created, &created_dirs);
if (result == FILE_SAVE_WRITTEN) {
/* Protocol 2.28.0: fold the receiver-observed literal bytes and the
created-entry type into the shared stats block under its mutex (the
receive thread also writes stats.matched_data). */
mtx_lock(&context->mutex);
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
mtx_unlock(&context->mutex);
}
if (result == FILE_SAVE_ERROR) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
src/shared/checksum.c
+28 -12
View File
@@ -1,4 +1,5 @@
#include "checksum.h"
#include "utils.h"
#include <fcntl.h>
#include <openssl/evp.h>
#include <string.h>
@@ -223,23 +224,31 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
if (fd < 0)
return false;
bool ok = checksum_digest_fd(algo, seed, fd, out, out_capacity, out_len);
close(fd);
return ok;
}
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
size_t* out_len) {
if (fd < 0 || !out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
return false;
if (algo == CHECKSUM_ALGO_NONE) {
/* No checksum requested: nothing to read; an empty digest succeeds. */
close(fd);
*out_len = 0;
return true;
}
uint8_t buffer[64 * 1024];
bool ok = false;
lseek(fd, 0, SEEK_SET);
if (algo == CHECKSUM_ALGO_MD5 || algo == CHECKSUM_ALGO_SHA1) {
const EVP_MD* md = algo == CHECKSUM_ALGO_MD5 ? EVP_md5() : EVP_sha1();
EVP_MD_CTX* ctx = EVP_MD_CTX_new();
if (!ctx) {
close(fd);
if (!ctx)
return false;
}
unsigned int digest_len = 0;
if (EVP_DigestInit_ex(ctx, md, NULL) == 1) {
ok = true;
@@ -258,7 +267,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
ok = false;
}
EVP_MD_CTX_free(ctx);
close(fd);
return ok;
}
@@ -275,7 +283,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
md4_final(&ctx, out);
*out_len = 16;
}
close(fd);
return ok;
}
@@ -285,16 +292,13 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
XXH64_reset(&xxh64, seed);
} else if (algo == CHECKSUM_ALGO_XXH3 || algo == CHECKSUM_ALGO_XXH128) {
xxh3 = XXH3_createState();
if (!xxh3) {
close(fd);
if (!xxh3)
return false;
}
if (algo == CHECKSUM_ALGO_XXH3)
XXH3_64bits_reset_withSeed(xxh3, seed);
else
XXH3_128bits_reset_withSeed(xxh3, seed);
} else {
close(fd);
return false;
}
@@ -328,7 +332,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
}
if (xxh3)
XXH3_freeState(xxh3);
close(fd);
return ok;
}
@@ -396,7 +399,7 @@ uint8_t checksum_digest_len(ChecksumAlgo algo) {
return 0;
}
ChecksumAlgo checksum_negotiate_default(void) {
static ChecksumAlgo compiled_checksum_preference_first(void) {
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
* resolves to xxh128. */
static const ChecksumAlgo preference[] = {
@@ -409,3 +412,16 @@ ChecksumAlgo checksum_negotiate_default(void) {
}
return CHECKSUM_ALGO_XXH64;
}
int checksum_choice_resolve(void) {
bool specified = false;
int env = env_choice_first("RSYNC_CHECKSUM_LIST", checksum_algo_from_name, &specified);
if (specified)
return env; /* -1 = the list named no supported checksum */
return (int)compiled_checksum_preference_first();
}
ChecksumAlgo checksum_negotiate_default(void) {
int resolved = checksum_choice_resolve();
return resolved >= 0 ? (ChecksumAlgo)resolved : compiled_checksum_preference_first();
}
src/shared/checksum.h
+14
View File
@@ -51,6 +51,13 @@ bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t
bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, uint8_t* out,
size_t out_capacity, size_t* out_len);
/* Descriptor form of the streaming digest: rewinds `fd` to the start and hashes
* to EOF without closing it. Used by the --verify-basis path to hash an
* already-open, root-confined basis descriptor. Same contract as
* checksum_digest_file. */
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
size_t* out_len);
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
* Accepts "xxh64"/"xxhash", "xxh3", "xxh128", "md5", "md4", "sha1", "none".
* "auto" is not an algorithm here; the caller resolves it to the negotiated
@@ -72,4 +79,11 @@ uint8_t checksum_digest_len(ChecksumAlgo algo);
* xxh128 xxh3 xxh64 md5 md4 sha1 none). Used to resolve "auto". */
ChecksumAlgo checksum_negotiate_default(void);
/* Resolve "auto" the way rsync does: the first supported name in
* RSYNC_CHECKSUM_LIST (whitespace-separated, client half ends at '&'), then the
* compiled-in preference order when the variable is unset/blank. Returns -1
* when the variable is set but names no supported checksum (rsync's failed
* negotiation), otherwise a valid ChecksumAlgo id. */
int checksum_choice_resolve(void);
#endif /* CHECKSUM_H */
src/shared/compression.c
+53 -1
View File
@@ -2,6 +2,7 @@
#include "data.h"
#include "log.h"
#include "protocol.h"
#include "utils.h"
#include <limits.h>
#include <lz4.h>
#include <stdatomic.h>
@@ -119,7 +120,7 @@ bool compression_algo_enabled(CompressionAlgo algo) {
return algo != COMPRESSION_ALGO_NONE;
}
CompressionAlgo compression_negotiate_default(void) {
static CompressionAlgo compiled_preference_first(void) {
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
* resolves to zstd. */
static const CompressionAlgo preference[] = {
@@ -133,6 +134,57 @@ CompressionAlgo compression_negotiate_default(void) {
return COMPRESSION_ALGO_ZSTD;
}
int compression_choice_resolve(void) {
bool specified = false;
int env = env_choice_first("RSYNC_COMPRESS_LIST", compression_algo_from_name, &specified);
if (specified)
return env; /* -1 = the list named no supported codec */
return (int)compiled_preference_first();
}
CompressionAlgo compression_negotiate_default(void) {
int resolved = compression_choice_resolve();
return resolved >= 0 ? (CompressionAlgo)resolved : compiled_preference_first();
}
int compression_default_level(CompressionAlgo algo) {
switch (algo) {
case COMPRESSION_ALGO_ZSTD:
return ZSTD_CLEVEL_DEFAULT;
case COMPRESSION_ALGO_ZLIB:
case COMPRESSION_ALGO_ZLIBX:
return 6; /* rsync resolves zlib's Z_DEFAULT_COMPRESSION (-1) to 6 */
case COMPRESSION_ALGO_LZ4:
return 1; /* rsync lz4 level is 0/ignored; positive keeps the gate on */
case COMPRESSION_ALGO_NONE:
return 0;
}
return 0;
}
int compression_clamp_level(CompressionAlgo algo, int level) {
switch (algo) {
case COMPRESSION_ALGO_ZSTD:
if (level < 1)
return 1;
if (level > 22)
return 22;
return level;
case COMPRESSION_ALGO_ZLIB:
case COMPRESSION_ALGO_ZLIBX:
if (level < 1)
return 1;
if (level > 9)
return 9;
return level;
case COMPRESSION_ALGO_LZ4:
return 1; /* ignored by lz4_compress; keeps the "compress" gate on */
case COMPRESSION_ALGO_NONE:
return 0;
}
return level;
}
void compression_set_algo(CompressionAlgo algo) {
if (compression_algo_valid((int)algo))
atomic_store(&g_compression_algo, (int)algo);
src/shared/compression.h
+20
View File
@@ -35,6 +35,26 @@ bool compression_algo_valid(int algo);
* "auto". */
CompressionAlgo compression_negotiate_default(void);
/* Resolve "auto" the way rsync does: the first supported name in
* RSYNC_COMPRESS_LIST (whitespace-separated, client half ends at '&'), then the
* compiled-in preference order when the variable is unset/blank. Returns -1
* when the variable is set but names no supported codec (rsync's failed
* negotiation), otherwise a valid CompressionAlgo id. */
int compression_choice_resolve(void);
/* rsync 3.4.1's per-codec default level, applied when the user did not pass
* --compress-level/--zl. zstd uses ZSTD_CLEVEL_DEFAULT (3) and zlib/zlibx the
* resolved Z_DEFAULT_COMPRESSION (6). lz4 has no tunable level in rsync
* (always the default acceleration); FastSync returns a positive placeholder so
* its "level > 0" compression gate stays engaged, and lz4_compress ignores the
* value, so the output is identical to rsync's. none is 0. */
int compression_default_level(CompressionAlgo algo);
/* Clamp an explicit --compress-level to the codec's accepted range the way
* rsync's init_compression_level() does: zstd 1..22, zlib/zlibx 1..9, lz4
* ignored (fixed positive placeholder), none 0. */
int compression_clamp_level(CompressionAlgo algo, int level);
/* True when the algorithm actually compresses (i.e. is not NONE). */
bool compression_algo_enabled(CompressionAlgo algo);
src/shared/config.c
+131 -2
View File
@@ -70,6 +70,8 @@ static void config_set_defaults(Config* config) {
config->ignore_missing_args = false;
config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
config->cli_exit_code = 0;
config->compression_level_set = false;
config->checksum_choice_set = false;
config->filters = NULL;
config->files_from = NULL;
config->files_from_set = NULL;
@@ -789,6 +791,8 @@ void config_delete(Config* config) {
if (config->filters) {
array_list_delete(config->filters);
}
filter_rule_list_free(config->protect_rules);
config->protect_rules = NULL;
/* A --delay-updates staging tree is transient receiver state: remove any
leftovers on every exit path (success already emptied it). */
if (config->delay_context)
@@ -1024,6 +1028,124 @@ static bool receive_basis_entries(int fd, Config* c, ConfigStringBudget* budget)
return true;
}
/* Receiver-side delete-protection rules (protocol 2.28.0). The sender compiles
* its command-line selection rules exactly as the scanner does and streams the
* result as one bounded, self-describing block (count + per-rule records); the
* receiver reconstructs a FilterRuleList for the --delete extras walk. owner
* and pattern are charged through the shared ConfigStringBudget and the block
* additionally enforces MAX_FILTER_RULES / MAX_FILTER_BYTES. */
static bool send_protect_entries(int fd, const Config* c) {
int count = c->filters ? c->filters->size : 0;
const char** texts = NULL;
if (count > 0) {
texts = malloc((size_t)count * sizeof(char*));
if (!texts)
return false;
for (int i = 0; i < count; i++)
texts[i] = (const char*)c->filters->items[i];
}
char err[160];
FilterRuleList* rules =
filter_base_build(texts, count, c->cvs_exclude, c->delete_excluded, err, sizeof(err));
free(texts);
if (!rules) {
log_message(LOG_LEVEL_ERROR, "invalid filter rule: %s", err);
return false;
}
bool ok = send_int(fd, rules->count);
for (int i = 0; ok && i < rules->count; i++) {
const FilterRule* r = rules->items[i];
/* Mirror the receiver's limit so the peer never receives a rule it will
reject as a protocol error. */
if (r->pattern && strlen(r->pattern) > MAX_PROTECT_PATTERN_LEN) {
log_message(LOG_LEVEL_ERROR, "filter pattern exceeds %d bytes", MAX_PROTECT_PATTERN_LEN);
filter_rule_list_free(rules);
return false;
}
ok = send_int(fd, (int)r->action) && send_int(fd, (int)r->sides) &&
send_int(fd, r->anchored ? 1 : 0) && send_int(fd, r->dir_only ? 1 : 0) &&
send_int(fd, r->negate ? 1 : 0) && send_str(fd, r->owner ? r->owner : "") &&
send_str(fd, r->pattern ? r->pattern : "");
}
filter_rule_list_free(rules);
return ok;
}
static bool receive_protect_entries(int fd, Config* c, ConfigStringBudget* budget) {
int count;
if (!receive_int(fd, &count))
return false;
if (count < 0 || count > MAX_FILTER_RULES)
return false;
if (count == 0)
return true;
FilterRuleList* list = filter_rule_list_create();
if (!list)
return false;
size_t pattern_bytes = 0;
for (int i = 0; i < count; i++) {
int action;
int sides;
bool anchored;
bool dir_only;
bool negate;
if (!receive_int(fd, &action) ||
(action != FILTER_ACTION_EXCLUDE && action != FILTER_ACTION_INCLUDE) ||
!receive_int(fd, &sides) || sides < (int)FILTER_SIDE_SENDER ||
sides > (int)(FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER) ||
!receive_wire_bool(fd, &anchored) || !receive_wire_bool(fd, &dir_only) ||
!receive_wire_bool(fd, &negate))
goto fail;
char* owner = config_receive_str(fd, budget);
if (!owner)
goto fail;
char* pattern = config_receive_str(fd, budget);
if (!pattern || pattern[0] == '\0') {
free(owner);
free(pattern);
goto fail;
}
/* A pattern too long to be evaluated by glob_match against a PATH_MAX path
would silently fail to match and leave a protect rule inert (fail-open:
the entry is then deleted). Reject it up front as a protocol error
rather than accept a rule that can never shield anything. */
if (strlen(pattern) > MAX_PROTECT_PATTERN_LEN) {
free(owner);
free(pattern);
goto fail;
}
size_t bytes = strlen(owner) + strlen(pattern);
if (bytes > MAX_FILTER_BYTES - pattern_bytes) {
free(owner);
free(pattern);
goto fail;
}
pattern_bytes += bytes;
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule) {
free(owner);
free(pattern);
goto fail;
}
rule->action = (FilterAction)action;
rule->sides = (unsigned)sides;
rule->anchored = anchored;
rule->dir_only = dir_only;
rule->negate = negate;
rule->owner = owner;
rule->pattern = pattern;
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
goto fail;
}
}
c->protect_rules = list;
return true;
fail:
filter_rule_list_free(list);
return false;
}
static bool send_identity_entries(int fd, const IdentityMap* map, int count) {
for (int i = 0; i < count; i++) {
if (!send_int(fd, map[i].from) || !send_int(fd, map[i].from_hi) || !send_int(fd, map[i].to) ||
@@ -1151,6 +1273,9 @@ fail:
#define CONFIG_RECV_BLOCK_IDMAP(name) \
receive_identity_entries(fd, budget, c->name##_count, &c->name)
#define CONFIG_SEND_BLOCK_PROTECT_RULES(name) send_protect_entries(fd, c)
#define CONFIG_RECV_BLOCK_PROTECT_RULES(name) receive_protect_entries(fd, c, budget)
/* One table entry, applied in sequence. XSEND/XRECV are statement macros so
* consecutive entries read as a plain sequence of assignments. */
#define XSEND(name, ctype, def, kind) ok = ok && (CONFIG_SEND_##kind(name));
@@ -1188,6 +1313,7 @@ CONFIG_DEFINE_SEND(send_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
CONFIG_DEFINE_SEND(send_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
CONFIG_DEFINE_SEND(send_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
CONFIG_DEFINE_SEND(send_codec_options, CONFIG_WIRE_CODEC_FIELDS)
CONFIG_DEFINE_SEND(send_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
CONFIG_DEFINE_RECV(receive_core_fields, CONFIG_WIRE_CORE_FIELDS)
CONFIG_DEFINE_RECV(receive_delta_fields, CONFIG_WIRE_DELTA_FIELDS)
@@ -1208,6 +1334,7 @@ CONFIG_DEFINE_RECV(receive_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
CONFIG_DEFINE_RECV(receive_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
CONFIG_DEFINE_RECV(receive_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
CONFIG_DEFINE_RECV(receive_codec_options, CONFIG_WIRE_CODEC_FIELDS)
CONFIG_DEFINE_RECV(receive_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
#undef XSEND
#undef XRECV
@@ -1326,7 +1453,8 @@ bool config_send_wire_block(int file_descriptor, const Config* config) {
send_privilege_options(file_descriptor, config) &&
send_copy_as_options(file_descriptor, config) &&
send_output_options(file_descriptor, config) &&
send_codec_options(file_descriptor, config);
send_codec_options(file_descriptor, config) &&
send_protect_options(file_descriptor, config);
}
bool config_send(int file_descriptor, const Config* config) {
@@ -1398,7 +1526,8 @@ Config* config_receive_with_validate(int file_descriptor, ConfigValidateFunc val
!receive_privilege_options(file_descriptor, config, &budget) ||
!receive_copy_as_options(file_descriptor, config, &budget) ||
!receive_output_options(file_descriptor, config, &budget) ||
!receive_codec_options(file_descriptor, config, &budget))
!receive_codec_options(file_descriptor, config, &budget) ||
!receive_protect_options(file_descriptor, config, &budget))
goto error;
/* Validate/normalize the negotiated codec. compress_choice is the human
* spelling (NULL or "" when -z was not given); compression_algo is the
src/shared/config.h
+81 -10
View File
@@ -4,6 +4,7 @@
#include "array_list.h"
#include "checksum.h"
#include "compression.h"
#include "filter.h"
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
@@ -82,7 +83,7 @@ typedef struct {
typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF = 2 } SuperMode;
/* ===========================================================================
* Config wire-field table (single source of truth for protocol 2.27.0).
* Config wire-field table (single source of truth for protocol 2.28.0).
*
* Every field below crosses the wire. The table is the ONLY place a
* serialized field is named: config.h expands CONFIG_WIRE_FIELDS() to declare
@@ -197,9 +198,18 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
X(skip_compress_count, int, 0, INT_SKIPCOUNT) \
X(skip_compress_suffixes, char**, NULL, BLOCK_SKIP_SUFFIXES)
/* FastSync-only --verify-basis (protocol 2.28.0, no version bump by project
* decision): restores the stricter content equality on a basis hit. By
* default a basis hit is accepted on rsync's metadata quick-check alone (equal
* size plus equal mtime, or size alone under --size-only); with this flag the
* receiver ALSO requires the basis bytes' whole-file digest (the negotiated
* --checksum-choice algorithm) to equal the sender's, exactly FastSync's
* historical behavior. It is a receiver policy and crosses the wire so the
* receiver knows whether to read and hash the basis content. */
#define CONFIG_WIRE_BASIS_FIELDS(X) \
X(basis_count, int, 0, INT_BASISCOUNT) \
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS)
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS) \
X(verify_basis, bool, false, BOOL)
#define CONFIG_WIRE_FUZZY_FIELDS(X) X(fuzzy, bool, false, BOOL)
@@ -293,6 +303,20 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
#define CONFIG_WIRE_CODEC_FIELDS(X) \
X(compression_algo, int, COMPRESSION_ALGO_ZSTD, INT_COMPRESSION_ALGO)
/* Receiver-side delete-protection filter rules (protocol 2.28.0). The sender
* compiles its root-level selection rules exactly as the scanner does
* (filter_base_build over --filter/-f/--exclude/--include/-C) and streams them
* as one self-describing, bounded block (count followed by per-rule records).
* The receiver reconstructs `protect_rules` and evaluates them against
* DESTINATION-ONLY entries during the --delete extras walk, so a
* `protect`/`P` rule protects an extra that never appeared on the sender
* (rsync re-derives deletion protection from the filter list; FastSync
* historically derived it only from the source scan). `protect_rules` is NULL
* on the sender and is owned/freed by the receiver Config. Bounded by
* MAX_FILTER_RULES and MAX_FILTER_BYTES; an unknown action/sides is a protocol
* error. */
#define CONFIG_WIRE_PROTECT_FIELDS(X) X(protect_rules, FilterRuleList*, NULL, BLOCK_PROTECT_RULES)
/* All serialized fields, in exact wire order. Concatenating the per-segment
* lists here is what keeps the declaration order = the wire order. */
#define CONFIG_WIRE_FIELDS(X) \
@@ -315,7 +339,8 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
CONFIG_WIRE_PRIVILEGE_FIELDS(X) \
CONFIG_WIRE_COPY_AS_FIELDS(X) \
CONFIG_WIRE_OUTPUT_FIELDS(X) \
CONFIG_WIRE_CODEC_FIELDS(X)
CONFIG_WIRE_CODEC_FIELDS(X) \
CONFIG_WIRE_PROTECT_FIELDS(X)
typedef struct Config {
/* -j/--threads=N: number of parallel scanner worker threads for the -m
@@ -419,6 +444,12 @@ typedef struct Config {
* for an unsupported checksum/compress algorithm) so main() can mirror it. */
int checksum_transfer_algo;
int cli_exit_code;
/* Client-only "the user explicitly chose" bits. They let the per-codec
* default level / checksum list be applied only when the corresponding
* rsync option was omitted (an explicit --compress-level / --checksum-choice
* always wins). Never serialized. */
bool compression_level_set;
bool checksum_choice_set;
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
// never serialized to the wire (the receiver must not learn them).
@@ -611,10 +642,14 @@ typedef struct Config {
source directory is streamed in directory order, and the receiver removes
each directory's extras when its plan arrives (during) or snapshots them
and removes them only after a successful transfer (delay). delete_after
(and plain --delete) keep the whole-tree commit mode: extras are removed
from a fresh end-of-transfer destination scan only after the whole transfer
succeeded. See config_delete_timing_early()/config_delete_timing_per_dir()
below. */
keeps the whole-tree commit mode: extras are removed from a fresh
end-of-transfer destination scan only after the whole transfer succeeded.
A plain --delete with no explicit timing flag defaults to delete_during on
the client (cli_finalize_config), matching rsync's --del default; the old
late-commit behavior is selected explicitly by --delete-after or the
FastSync-only long spelling --delete-commit (an exact alias for
--delete-after, mapped onto the same wire field). See
config_delete_timing_early()/config_delete_timing_per_dir() below. */
/* partial_dir */
// PR #174: Partial transfer resumption
/* suffix */
@@ -1006,11 +1041,44 @@ typedef struct Config {
* the sender can print rsync's `deleting PATH` lines for a real deletion. No
* change to the fixed STATUS_STATS record itself; only a new trailing config
* bool, which still requires the version bump for the strict lockstep. */
#define PROTOCOL_VERSION "2.27.0"
/* (8) --stats receiver-observed counters (protocol 2.28.0): the fixed
* STATUS_STATS record grows from three counters to eight. The receiver now
* reports the bytes it literally stored (`literal_data`) and the count of
* destination entries it newly CREATED, split by type
* (reg/dir/link/special), so the sender can print rsync's exact
* `Number of created files: N (reg: X, dir: Y, link: Z, special: W)` line and
* an exact `Literal data` total even for delta transfers. The config-frame
* LAYOUT is unchanged (no new config field), but the STATUS_STATS body grows,
* so a 2.27 peer that does not consume the five new fixed-width counters would
* desynchronize on the trailing would-delete path list; the strict
* same-version handshake (config_receive rejects a mismatched version before
* parsing anything else) keeps mixed deployments from ever reaching that
* state. */
/* (9) Receiver-side delete protection (still protocol 2.28.0): the config frame
* gains one trailing self-describing block carrying the sender's compiled base
* filter rules so the receiver can protect DESTINATION-ONLY entries from
* --delete with `protect`/`risk` rules (rsync parity). The block appends after
* compression_algo; see CONFIG_WIRE_PROTECT_FIELDS. */
#define PROTOCOL_VERSION "2.28.0"
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
#define MAX_BASIS_DIRS 64
/* Bounds on the received receiver-side delete-protection rule block. The rule
* count and the aggregate pattern+owner bytes are each capped so a hostile
* peer cannot pin unbounded pre-auth memory; both are validated strictly on
* receive (alongside the per-string ConfigStringBudget). */
/* A peer may supply protect rules; cap the list so a crafted config cannot make
* the receiver's delete walk evaluate an unbounded number of glob patterns per
* destination entry (glob_match is O(pattern x path)). 1024 is far above any
* legitimate selection. */
#define MAX_FILTER_RULES 1024
#define MAX_FILTER_BYTES (256 * 1024)
/* glob_match's DP is capped at 64 Mi work units; a pattern longer than this
* could exceed the cap against a PATH_MAX path and silently stop matching,
* leaving a protect rule inert. Reject such a rule at receive time. */
#define MAX_PROTECT_PATTERN_LEN 8192
/* Upper bound on the number of --skip-compress suffixes accepted from the wire.
* Each suffix is an independent wire string (up to MAX_STRING_SIZE = 64 KiB), so
* without this a hostile pre-auth client could otherwise retain
@@ -1112,8 +1180,11 @@ bool config_delete_timing_early(const Config* config);
* commits them only after a fully-successful transfer (delay). */
bool config_delete_timing_per_dir(const Config* config);
/* Delete-timing sanity: with deletion enabled at most one timing flag may be
* set (none = the default delete-after commit timing); without deletion no
* timing flag may be set (each timing flag implies --delete). */
* set; without deletion no timing flag may be set (each timing flag implies
* --delete). A plain --delete is normalized to delete_during by
* cli_finalize_config on the client, so a transmitted use_delete config always
* carries exactly one timing; the zero-timing case remains valid only for a
* config that has not been through the CLI. */
bool config_has_valid_delete_timing(const Config* config);
/* Single source of truth for the cross-field ("combination") invariants a
src/shared/delete_plan.c
+150 -30
View File
@@ -331,6 +331,8 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
return -1;
sender->config_sent = true;
}
if (!send_int(fd, 1)) /* apply = true */
return -1;
if (!send_wire_str(fd, node->dir))
return -1;
if (send_str_section(fd, node->dirs) != 0 || send_str_section(fd, node->files) != 0)
@@ -339,6 +341,29 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
return 0;
}
/* Transmit the one-shot per-run config block (protected prefixes, size-pruned
* mirrors, --delete-missing-args exact paths) on its own carrier frame, with
* apply=false so the receiver consumes the config but walks nothing. This is
* how the config still reaches the receiver when the scope allows no directory
* plan at all (a --files-from list of bare files synchronizes no directory):
* without it, the missing-args exact deletions would be lost. Idempotent. */
static int send_config_only(int fd, DeletePlanSender* sender) {
if (!sender || sender->config_sent)
return 0;
if (!send_status(fd, STATUS_DELETE_PLAN) || !send_int(fd, 1))
return -1;
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
send_str_section(fd, sender->size_skipped) != 0 ||
send_str_section(fd, sender->missing_args) != 0)
return -1;
sender->config_sent = true;
if (!send_int(fd, 0)) /* apply = false */
return -1;
if (!send_wire_str(fd, ".") || !send_int(fd, 0) || !send_int(fd, 0))
return -1;
return 0;
}
static int send_prefix_plan(int fd, DeletePlanSender* sender, const char* dir) {
PlanNode* node = plan_find(sender, dir);
if (!node || node->sent)
@@ -354,6 +379,10 @@ int delete_plan_send_root(int fd, DeletePlanSender* sender) {
const char* root = sender->walk_root ? sender->walk_root : ".";
if (!plan_ensure(sender, root))
return -1;
/* Put the config block on the wire first, on its own carrier frame, so the
receiver always sees it even when the scope permits no directory plan. */
if (send_config_only(fd, sender) != 0)
return -1;
return send_prefix_plan(fd, sender, root);
}
@@ -412,11 +441,17 @@ struct DeletePlanSession {
bool dry_run;
size_t max_delete;
size_t deleted;
/* Removals charged against --max-delete. In --delete-delay mode a path is
planned (and the budget consumed) while scanning, but `deleted` advances
only when the commit actually unlinks it, so an entry that survives the
commit (a directory refilled mid-transfer -> ENOTEMPTY) is not reported. */
/* Removals charged against --max-delete. The budget is charged on ACTUAL
removals (an unlink/rmdir that succeeded), matching rsync: a snapshotted
entry that fails removal consumes nothing, so a later extra is still
deleted. `planned` and `deleted` advance together for the inline paths and
`apply_missing`; `deleted` is the reported count. */
size_t planned;
/* Hard bound on the deferred snapshot list. Because the budget is no longer
charged at snapshot time, this independent cap keeps a huge destination
from growing the list without limit (it matches the receiver's overall
deletion bound). */
size_t defer_cap;
size_t skipped;
bool limit_hit;
bool limit_logged;
@@ -448,6 +483,7 @@ DeletePlanSession* delete_plan_session_create(const Config* config) {
config->max_delete >= 0 && (size_t)config->max_delete < DELETE_PLAN_SERVER_LIMIT;
session->max_delete =
user_limited ? (size_t)config->max_delete : (size_t)DELETE_PLAN_SERVER_LIMIT;
session->defer_cap = DELETE_PLAN_SERVER_LIMIT;
session->protected_prefixes = array_list_create(free);
session->size_skipped = array_list_create(free);
session->missing = array_list_create(free);
@@ -539,12 +575,18 @@ static int open_plan_dir(const Config* config, const char* dir) {
typedef struct PlanSkips {
DeleteSkipEntry* entries;
int count;
/* Receiver-side delete-protection rules received on the config frame (NULL
when the sender sent none). Evaluated per extra so a protect/risk rule is
honored under --delete-during/--delete-delay exactly like the whole-tree
commit walker. */
const FilterRuleList* protect_rules;
} PlanSkips;
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
PlanSkips* out) {
out->entries = NULL;
out->count = 0;
out->protect_rules = config->protect_rules;
int count = (config->delay_updates ? 1 : 0) + config->basis_count +
session->protected_prefixes->size + session->size_skipped->size;
if (count == 0)
@@ -592,10 +634,15 @@ static void log_deleted(const char* rel) {
free(escaped);
}
/* Append a snapshot path for --delete-delay. The budget is charged here, but
* `deleted` is not: the path counts only once apply_deferred_path truly
* unlinks it. */
/* Append a snapshot path for --delete-delay. The budget is NOT charged here:
* the remover charges --max-delete only when a path is actually unlinked (see
* apply_deferred_path), so a snapshotted entry that survives ENOTEMPTY cannot
* deny budget to a later extra. The independent `defer_cap` bounds the list. */
static bool defer_add(DeletePlanSession* session, const char* rel) {
if ((size_t)session->deferred->size >= session->defer_cap) {
note_skipped(session);
return true;
}
char* copy = str_dup(rel);
if (!copy)
return false;
@@ -603,7 +650,6 @@ static bool defer_add(DeletePlanSession* session, const char* rel) {
free(copy);
return false;
}
session->planned++;
return true;
}
@@ -636,16 +682,16 @@ static bool process_extra_dir(int dirfd, const char* name, const char* child_rel
return false;
if (survives)
return true;
if (!budget_available(session)) {
note_skipped(session);
return true;
}
if (session->defer && !force_now) {
if (!defer_add(session, child_rel))
return false;
*removed = true;
return true;
}
if (!budget_available(session)) {
note_skipped(session);
return true;
}
if (unlinkat(dirfd, name, AT_REMOVEDIR) == 0) {
session->deleted++;
session->planned++;
@@ -665,13 +711,13 @@ static bool process_extra_dir(int dirfd, const char* name, const char* child_rel
static bool process_extra_file(int dirfd, const char* name, const char* child_rel, bool force_now,
DeletePlanSession* session) {
if (session->defer && !force_now) {
return defer_add(session, child_rel);
}
if (!budget_available(session)) {
note_skipped(session);
return true;
}
if (session->defer && !force_now) {
return defer_add(session, child_rel);
}
if (unlinkat(dirfd, name, 0) == 0) {
session->deleted++;
session->planned++;
@@ -722,6 +768,10 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
bool is_dir = S_ISDIR(st.st_mode);
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entry->d_name);
bool in_keep_files = !is_dir && list_contains_str(keep_files, entry->d_name);
bool rule_protected =
skips->protect_rules &&
filter_rules_apply_side(skips->protect_rules, child_rel, entry->d_name, is_dir,
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT;
if (in_keep_dirs) {
local_survives = true;
} else if (keep_dirs && !is_dir && list_contains_str(keep_dirs, entry->d_name)) {
@@ -739,11 +789,18 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
else if (!removed)
local_survives = true;
} else if (is_dir) {
bool removed = false;
if (!process_extra_dir(dirfd, entry->d_name, child_rel, force_now, skips, session, &removed))
operation_ok = false;
else if (!removed)
if (rule_protected) {
local_survives = true;
} else {
bool removed = false;
if (!process_extra_dir(dirfd, entry->d_name, child_rel, force_now, skips, session,
&removed))
operation_ok = false;
else if (!removed)
local_survives = true;
}
} else if (rule_protected) {
local_survives = true;
} else {
if (!process_extra_file(dirfd, entry->d_name, child_rel, force_now, session))
operation_ok = false;
@@ -822,6 +879,14 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
}
session->config_seen = true;
}
/* apply=false is the config-only carrier frame: the receiver consumes the
config (and the missing-args exact deletions) but must not walk any
directory. Every real plan carries apply=true. */
int apply;
if (!receive_int(fd, &apply) || (apply != 0 && apply != 1)) {
send_status(fd, STATUS_ERROR);
return -1;
}
char* dir = receive_wire_str(fd);
ArrayList* dirs = array_list_create(free);
ArrayList* files = array_list_create(free);
@@ -839,7 +904,7 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
if (!session->dry_run && enabled) {
if (!session->defer && !apply_missing(session, config))
ok = false;
if (ok && !apply_plan_dir(session, config, dir, dirs, files))
if (ok && apply && !apply_plan_dir(session, config, dir, dirs, files))
ok = false;
}
free(dir);
@@ -858,7 +923,9 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
}
/* Apply one snapshotted --delete-delay path (post-order: children precede their
* parent directory). */
* parent directory). A directory that is still present is re-scanned so content
* created after the plan is removed too; every actual removal charges
* --max-delete. */
static bool apply_deferred_path(DeletePlanSession* session, const Config* config, const char* rel) {
char* full = path_cat(config->receive_root_directory, rel);
if (!full)
@@ -872,25 +939,78 @@ static bool apply_deferred_path(DeletePlanSession* session, const Config* config
}
struct stat st;
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0) {
bool absent = errno == ENOENT;
bool absent = errno == ENOENT || errno == ENOTDIR;
close(parent_fd);
free(leaf);
return absent;
}
int rc;
if (S_ISDIR(st.st_mode))
rc = unlinkat(parent_fd, leaf, AT_REMOVEDIR);
else
rc = unlinkat(parent_fd, leaf, 0);
bool ok = rc == 0 || errno == ENOENT || errno == ENOTEMPTY || errno == EEXIST;
if (rc == 0) {
if (S_ISDIR(st.st_mode)) {
if (!budget_available(session)) {
note_skipped(session);
close(parent_fd);
free(leaf);
return true;
}
int dirfd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (dirfd < 0) {
bool absent = errno == ENOENT || errno == ENOTDIR;
close(parent_fd);
free(leaf);
return absent;
}
PlanSkips skips;
if (!build_plan_skips(config, session, &skips)) {
close(dirfd);
close(parent_fd);
free(leaf);
return false;
}
bool survives = false;
bool ok = process_children(dirfd, rel, NULL, NULL, false, true, &skips, session, &survives);
free(skips.entries);
close(dirfd);
if (!ok) {
close(parent_fd);
free(leaf);
return false;
}
if (!survives) {
if (!budget_available(session)) {
note_skipped(session);
} else if (unlinkat(parent_fd, leaf, AT_REMOVEDIR) == 0) {
session->deleted++;
session->planned++;
log_deleted(rel);
notify_deleted(session, rel);
} else if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST) {
close(parent_fd);
free(leaf);
return false;
}
}
close(parent_fd);
free(leaf);
return true;
}
if (!budget_available(session)) {
note_skipped(session);
close(parent_fd);
free(leaf);
return true;
}
if (unlinkat(parent_fd, leaf, 0) == 0) {
session->deleted++;
session->planned++;
log_deleted(rel);
notify_deleted(session, rel);
} else if (errno != ENOENT) {
close(parent_fd);
free(leaf);
return false;
}
close(parent_fd);
free(leaf);
return ok;
return true;
}
void delete_plan_session_set_delete_observer(DeletePlanSession* session,
src/shared/delete_plan.h
+5 -2
View File
@@ -48,11 +48,14 @@ void delete_plan_sender_finalize(DeletePlanSender* sender, const ArrayList* sync
Directory keep entries do not count, so an I/O error that hid every file
still refuses to delete. */
bool delete_plan_sender_empty(const DeletePlanSender* sender);
/* Attach the global config sections advertised on the first plan frame. */
/* Attach the global config sections advertised on the first plan frame. The
* block is always transmitted by delete_plan_send_root(), on a config-only
* carrier frame when the scope allows no directory plan. */
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
const ArrayList* size_skipped, const ArrayList* missing_args);
/* Send the root plan (even before any data, so root extras are handled like
* rsync's first generator directory). Returns -1 on I/O error. */
* rsync's first generator directory), after transmitting the per-run config
* block on its own carrier frame. Returns -1 on I/O error. */
int delete_plan_send_root(int fd, DeletePlanSender* sender);
/* Send the plans for every ancestor of `path` (root-first) and, when is_dir,
* for `path` itself; already-sent plans are skipped. */
src/shared/file.c
+365 -22
View File
@@ -15,6 +15,7 @@
#include <unistd.h>
#include "data.h"
#include "checksum.h"
#include "delta.h"
#include "file.h"
#include "file_store.h"
@@ -24,6 +25,13 @@
#include "utils.h"
#include "protocol.h"
#include "xattr.h"
#include <fcntl.h>
#include <unistd.h>
/* Files larger than this are not loaded whole for transfer (the sender streams
* them); a whole-file digest is computed from the path instead. Kept in sync
* with the sender's streaming threshold. */
#define STREAM_THRESHOLD (64ULL * 1024 * 1024)
static bool write_all(int fd, const void* data, unsigned long long size) {
const unsigned char* p = data;
@@ -39,6 +47,31 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
return true;
}
/* Streaming copy of an open source descriptor into the just-created destination
`fd` (already at offset 0). Used by the --copy-dest basis install so a basis
larger than any in-memory whole-file bound still materializes without
buffering the entire file. `expected_size` is the caller-verified basis
size; the copy must produce exactly that many bytes (a short source is a hard
error, never a silently truncated destination). The final ftruncate drops
any residual tail a raced-in longer source might have left. */
static bool copy_fd_all(int dst_fd, int src_fd, unsigned long long expected_size) {
unsigned char buf[1 << 20];
unsigned long long done = 0;
while (done < expected_size) {
unsigned long long remaining = expected_size - done;
size_t want = remaining < sizeof(buf) ? (size_t)remaining : sizeof(buf);
ssize_t n = read(src_fd, buf, want);
if (n < 0 && errno == EINTR)
continue;
if (n <= 0)
return false;
if (!write_all(dst_fd, buf, (unsigned long long)n))
return false;
done += (unsigned long long)n;
}
return ftruncate(dst_fd, (off_t)expected_size) == 0;
}
/* Preallocate `size` bytes on `fd` before any data is written (--preallocate).
* fallocate(2) reserves real disk blocks, so an out-of-space condition
* (ENOSPC/EDQUOT) surfaces up front instead of partway through a transfer;
@@ -140,6 +173,12 @@ bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, s
if (file->data->size == 0) {
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
}
/* A streamed source (data not loaded) may exceed any in-memory whole-file
bound; hash it from the file path in bounded buffers instead of forcing a
full load. This is the same digest the receiver recomputes on the basis. */
if (!file->data->data && file->path && file->data->size > STREAM_THRESHOLD &&
checksum_digest_file(algo, seed, file->path, out, out_capacity, out_len))
return true;
if (!file->data->data && !file_load_data(file))
return false;
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
@@ -176,6 +215,7 @@ File* file_create(const char* path) {
file->is_dir = false;
file->dir_time_only = false;
file->basis_link = NULL;
file->basis_copy = NULL;
file->link_group = 0;
file->link_first = false;
file->hardlink_target = NULL;
@@ -187,6 +227,7 @@ File* file_create(const char* path) {
file->xattrs = NULL;
file->dest_state = (OutputDestState){0};
file->matched_bytes = 0;
file->literal_bytes = 0;
return file;
}
@@ -204,6 +245,8 @@ void file_destroy(void* item) {
file->send_path = NULL;
free(file->basis_link);
file->basis_link = NULL;
free(file->basis_copy);
file->basis_copy = NULL;
free(file->hardlink_target);
file->hardlink_target = NULL;
free(file->symlink_target);
@@ -614,6 +657,103 @@ static int open_dir_beneath_root(const char* resolved, const char* root) {
}
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs) {
return file_open_secure_parent_counted(path, leaf_out, create_dirs, NULL, NULL);
}
/* The logical transfer root expressed in the same coordinate as the secure
* parent walk's `rel_buf` (relative to the authorized root, with a leading
* '/'), used as the floor at or below which a created directory is a real
* file-list entry. The on-disk transfer root is the receive root joined to the
* wire path; the mirror scaffolding above it (the absolute source path below
* the destination root) is not an rsync entry. Returns an allocated string or
* NULL (count every created component). */
static char* transfer_root_floor(const Config* config) {
if (!config || !config->send_directory || config->send_directory[0] == '\0')
return NULL;
const char* spec = config->send_directory;
const char* after = spec;
if (spec[0] == '.' && spec[1] == '/') {
after = spec + 2;
} else {
const char* cut = strstr(spec, "/./");
if (cut)
after = cut + 3;
}
while (*after == '/')
after++;
char* wire_root = str_dup(after);
if (!wire_root)
return NULL;
size_t wlen = strlen(wire_root);
while (wlen > 0 && wire_root[wlen - 1] == '/')
wire_root[--wlen] = '\0';
if (wlen == 0) {
free(wire_root);
return NULL;
}
char* disk_root = config->receive_root_directory
? path_cat(config->receive_root_directory, wire_root)
: str_dup(wire_root);
free(wire_root);
if (!disk_root)
return NULL;
const char* root_path = utils_get_authorized_root_path();
const char* floor = disk_root;
if (root_path && root_path[0] == '/') {
size_t rl = strlen(root_path);
while (rl > 0 && root_path[rl - 1] == '/')
rl--;
if (strncmp(disk_root, root_path, rl) == 0 && (disk_root[rl] == '/' || disk_root[rl] == '\0'))
floor = disk_root + rl;
}
while (*floor == '/')
floor++;
char* out = str_dup(floor);
free(disk_root);
if (!out)
return NULL;
if (out[0] == '\0') {
free(out);
return NULL;
}
return out;
}
/* A created parent component counts toward `Number of created files` only when
* its receive-root-relative path is at or below the logical transfer root
* (`count_floor`). The transfer root itself corresponds to rsync's `.` entry
* (created on a fresh destination, pre-existing otherwise); the mirror
* scaffolding above it is FastSync's absolute-path layout, not an rsync entry. */
static bool created_dir_counts(const char* count_floor, const char* rel_buf,
const char* component) {
if (!count_floor)
return true;
char candidate[PATH_MAX];
int n = snprintf(candidate, sizeof(candidate), "%s/%s", rel_buf, component);
if (n < 0 || (size_t)n >= sizeof(candidate))
return false;
const char* cand = candidate;
while (*cand == '/')
cand++;
size_t fl = strlen(count_floor);
if (strncmp(cand, count_floor, fl) != 0)
return false;
return cand[fl] == '\0' || cand[fl] == '/';
}
/* Public wrapper for the receiver's created-directory accounting: the logical
* transfer root expressed receive-root-relative, or NULL when the wire paths
* carry no mirror scaffolding above it (--relative and --files-from, whose
* paths are already relative to the transfer root). The caller frees a
* non-NULL result. */
char* file_transfer_root_floor(const Config* config) {
if (!config || config->relative || config->files_from_set != NULL)
return NULL;
return transfer_root_floor(config);
}
int file_open_secure_parent_counted(const char* path, char** leaf_out, bool create_dirs,
unsigned* dirs_created, const char* count_floor) {
char* copy = str_dup(path);
if (!copy)
return -1;
@@ -674,6 +814,14 @@ int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs)
if (next < 0 && create_dirs && errno == ENOENT) {
bool created = mkdirat(fd, component, (mode_t)(0777 & ~(mode_t)file_process_umask())) == 0;
if (created || errno == EEXIST) {
/* Protocol 2.28.0: only directories the logical file list would
create count toward `Number of created files`; the mirror
scaffolding above the transfer root (e.g. the absolute source path
under the destination root) is not an rsync entry. `count_floor`
is a receive-root-relative prefix that must be reached before a
created component is counted. */
if (created && dirs_created && created_dir_counts(count_floor, rel_buf, component))
(*dirs_created)++;
/* P7 Wave E: --copy-as owns EVERY entry, including the intermediate
directories this walk creates implicitly. Its target ids are a
global policy, so they are available here without per-entry source
@@ -1026,15 +1174,14 @@ static void restore_extra_fd(int fd, const FileMetadata* metadata, const FileXat
}
}
static bool file_to_disk_secure_impl(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace,
bool use_fsync, const char* temp_dir,
const FileXattrList* xattrs, bool fake_super,
bool keep_partial) {
static bool
file_to_disk_secure_impl(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace, bool use_fsync,
const char* temp_dir, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, unsigned* dirs_created, const char* count_floor) {
char* leaf = NULL;
int dirfd = file_open_secure_parent(path, &leaf, true);
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
if (dirfd < 0)
return false;
int fd = -1;
@@ -1325,7 +1472,7 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
"non-atomic copy into the destination directory");
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, update, no_replace, use_fsync, NULL, xattrs, fake_super,
keep_partial);
keep_partial, dirs_created, count_floor);
}
return ok;
}
@@ -1334,7 +1481,8 @@ bool file_to_disk_secure(const char* path, const void* data, unsigned long long
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, false, false, false, temp_dir, NULL, false, false);
policy, false, false, false, temp_dir, NULL, false, false, NULL,
NULL);
}
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
@@ -1342,7 +1490,8 @@ bool file_to_disk_secure_update(const char* path, const void* data, unsigned lon
const FileMetadata* metadata, FileAttrPolicy policy,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, true, false, false, temp_dir, NULL, false, false);
policy, true, false, false, temp_dir, NULL, false, false, NULL,
NULL);
}
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
@@ -1350,7 +1499,8 @@ bool file_to_disk_secure_with_fsync(const char* path, const void* data,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool use_fsync, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, false, false, use_fsync, temp_dir, NULL, false, false);
policy, false, false, use_fsync, temp_dir, NULL, false, false,
NULL, NULL);
}
bool file_to_disk_secure_no_replace(const char* path, const void* data,
@@ -1358,7 +1508,8 @@ bool file_to_disk_secure_no_replace(const char* path, const void* data,
const FileMetadata* metadata, FileAttrPolicy policy,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
policy, false, true, false, temp_dir, NULL, false, false);
policy, false, true, false, temp_dir, NULL, false, false, NULL,
NULL);
}
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
@@ -1371,9 +1522,21 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
bool no_replace, bool use_fsync, const FileXattrList* xattrs,
bool fake_super, bool keep_partial, const char* temp_dir) {
return file_to_disk_secure_attrs_counted(path, data, data_size, inplace, sparse, preallocate,
metadata, policy, update, no_replace, use_fsync, xattrs,
fake_super, keep_partial, temp_dir, NULL, NULL);
}
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, const char* temp_dir,
unsigned* dirs_created, const char* count_floor) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, update, no_replace, use_fsync, temp_dir, xattrs,
fake_super, keep_partial);
fake_super, keep_partial, dirs_created, count_floor);
}
/* Atomic --link-dest install. The destination is replaced (via a temporary
@@ -1391,16 +1554,176 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
* basis). Likewise `xattrs`/`fake_super` are applied only on the copy
* fallback, so a fallback copy preserves the per-file attributes instead of
* silently dropping them. */
/* Streaming --copy-dest basis install: atomically materialize `path` from the
* bytes of `basis_path` without holding the file in memory, so a basis larger
* than any whole-file bound still works. Mirrors the ordinary secure store
* path (confined parent walk, temp + rename, --update/--ignore-existing/
* --preallocate/--temp-dir) but sources the data from the basis descriptor
* rather than a caller buffer, and applies the SOURCE metadata (rsync copies
* then fixes attributes). A hard-link install that falls back to a byte copy
* also routes through here when the caller supplies the basis path. */
static bool file_copy_basis_stream_impl(const char* path, const char* basis_path,
unsigned long long expected_size, bool preallocate,
const FileMetadata* metadata, FileAttrPolicy policy,
bool update, bool no_replace, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir, unsigned* dirs_created,
const char* count_floor) {
if (!path || !basis_path)
return false;
char* leaf = NULL;
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
if (dirfd < 0)
return false;
char* basis_leaf = NULL;
int basis_dirfd = file_open_secure_parent(basis_path, &basis_leaf, false);
int src_fd = -1;
if (basis_dirfd >= 0 && basis_leaf != NULL) {
/* O_NONBLOCK rejects a raced-in FIFO without blocking; the S_ISREG gate
below is the real type check. */
src_fd = openat(basis_dirfd, basis_leaf, O_RDONLY | O_CLOEXEC | O_NOFOLLOW | O_NONBLOCK);
struct stat src_st;
if (src_fd >= 0 && (fstat(src_fd, &src_st) != 0 || !S_ISREG(src_st.st_mode))) {
close(src_fd);
src_fd = -1;
}
}
if (basis_dirfd >= 0)
close(basis_dirfd);
free(basis_leaf);
if (src_fd < 0) {
close(dirfd);
free(leaf);
return false;
}
struct stat destination_stat;
bool destination_is_regular = fstatat(dirfd, leaf, &destination_stat, AT_SYMLINK_NOFOLLOW) == 0 &&
S_ISREG(destination_stat.st_mode);
if (update && metadata && destination_is_regular && stat_is_newer(&destination_stat, metadata)) {
close(src_fd);
close(dirfd);
free(leaf);
return true;
}
if (no_replace && file_path_exists_secure(path)) {
close(src_fd);
close(dirfd);
free(leaf);
return true;
}
int scratch_dirfd = -1;
if (temp_dir) {
scratch_dirfd = file_open_temp_dir(temp_dir);
if (scratch_dirfd < 0) {
int saved_errno = errno;
log_message(LOG_LEVEL_ERROR,
"--temp-dir '%s' could not be opened (rsync requires it to already exist): %s",
temp_dir, strerror(saved_errno));
close(src_fd);
close(dirfd);
free(leaf);
return false;
}
}
int target_dirfd = scratch_dirfd >= 0 ? scratch_dirfd : dirfd;
int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%llu", leaf, (long)getpid(), ~0ULL);
char* tmp = NULL;
bool ok = false;
if (tmp_size >= 0)
tmp = malloc((size_t)tmp_size + 1);
if (tmp) {
for (unsigned int i = 0; i < 100 && !ok; ++i) {
if (scratch_dirfd >= 0)
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%llu", leaf, (long)getpid(),
next_temp_sequence());
else
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
int fd =
openat(target_dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
if (fd < 0) {
if (errno != EEXIST)
break;
continue;
}
bool wrote = true;
if (preallocate && expected_size > 0 && preallocate_fd(fd, expected_size) != 0)
wrote = false;
if (wrote)
wrote = copy_fd_all(fd, src_fd, expected_size);
if (wrote && metadata) {
if (!policy.perms &&
fchmod(fd, file_mode_base(metadata, destination_is_regular,
destination_is_regular ? destination_stat.st_mode & 0777
: 0)) != 0)
wrote = false;
if (wrote)
wrote = file_restore_metadata_fd(fd, metadata, policy);
} else if (wrote && fchmod(fd, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH) != 0) {
wrote = false;
}
if (wrote)
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
if (wrote && use_fsync)
wrote = fsync(fd) == 0;
if (close(fd) != 0)
wrote = false;
if (wrote && renameat(target_dirfd, tmp, dirfd, leaf) != 0)
wrote = false;
if (!wrote)
unlinkat(target_dirfd, tmp, 0);
ok = wrote;
}
free(tmp);
}
if (!ok && scratch_dirfd >= 0) {
/* Retry once with no scratch dir (rsync's EXDEV fallback). */
close(scratch_dirfd);
close(src_fd);
close(dirfd);
free(leaf);
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata,
policy, update, no_replace, use_fsync, xattrs, fake_super,
NULL, dirs_created, count_floor);
}
if (scratch_dirfd >= 0)
close(scratch_dirfd);
close(src_fd);
close(dirfd);
free(leaf);
return ok;
}
/* --copy-dest basis install (streaming). Applies the source metadata and the
per-file xattrs / --fake-super record. */
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
unsigned long long expected_size, bool preallocate,
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir) {
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata, policy,
update, false, use_fsync, xattrs, fake_super, temp_dir, NULL,
NULL);
}
static bool file_to_disk_secure_link_impl(const char* path, const char* basis_path,
const void* data, unsigned long long data_size,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir) {
const char* temp_dir, unsigned* dirs_created,
const char* count_floor) {
if (!path || !basis_path)
return false;
/* The caller-supplied buffer is no longer used: the copy fallback streams
from the basis path (which may hold an over-limit file). Kept in the
signature for the existing API. */
(void)data;
char* leaf = NULL;
int dirfd = file_open_secure_parent(path, &leaf, true);
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
if (dirfd < 0)
return false;
@@ -1482,10 +1805,17 @@ static bool file_to_disk_secure_link_impl(const char* path, const char* basis_pa
close(dirfd);
free(leaf);
/* The basis file could not be linked in (missing, cross-device, refused
by the filesystem). Write a byte-identical local copy instead. */
return file_to_disk_secure_attrs(path, data, data_size, false, false, preallocate, metadata,
policy, false, false, use_fsync, xattrs, fake_super, false,
temp_dir);
by the filesystem). Stream a byte-identical local copy from the basis
itself (never the possibly-absent caller buffer) so an over-limit basis
still materializes. When the basis path is not a readable regular file
(e.g. a directory raced in), fall back to the caller-supplied bytes. */
if (file_copy_basis_stream_impl(path, basis_path, data_size, preallocate, metadata, policy,
false, false, use_fsync, xattrs, fake_super, temp_dir,
dirs_created, count_floor))
return true;
return file_to_disk_secure_attrs_counted(
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
xattrs, fake_super, false, temp_dir, dirs_created, count_floor);
}
if (scratch_dirfd >= 0)
@@ -1500,7 +1830,7 @@ bool file_to_disk_secure_link(const char* path, const char* basis_path, const vo
const FileMetadata* metadata, FileAttrPolicy policy, bool use_fsync,
const char* temp_dir) {
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
policy, use_fsync, NULL, false, temp_dir);
policy, use_fsync, NULL, false, temp_dir, NULL, NULL);
}
bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, const void* data,
@@ -1508,8 +1838,21 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
const FileMetadata* metadata, FileAttrPolicy policy,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir) {
return file_to_disk_secure_link_attrs_counted(path, basis_path, data, data_size, preallocate,
metadata, policy, use_fsync, xattrs, fake_super,
temp_dir, NULL, NULL);
}
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
const void* data, unsigned long long data_size,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir, unsigned* dirs_created,
const char* count_floor) {
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
policy, use_fsync, xattrs, fake_super, temp_dir);
policy, use_fsync, xattrs, fake_super, temp_dir,
dirs_created, count_floor);
}
bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
src/shared/file.h
+37
View File
@@ -87,6 +87,11 @@ bool file_path_exists_secure(const char* path);
bool file_stat_secure(const char* path, struct stat* st);
bool file_destination_is_newer_secure(const char* path, const FileMetadata* metadata);
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
/* Protocol 2.28.0 variant: also increments *dirs_created for every missing
* parent directory this walk creates that lies strictly below `count_floor`
* (a receive-root-relative path, or NULL to count all of them). */
int file_open_secure_parent_counted(const char* path, char** leaf_out, bool create_dirs,
unsigned* dirs_created, const char* count_floor);
bool file_ensure_directory_secure(const char* path);
bool file_directory_exists_secure(const char* path);
bool file_rename_secure(const char* old_path, const char* new_path);
@@ -158,5 +163,37 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
const FileMetadata* metadata, FileAttrPolicy policy,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir);
/* Streaming --copy-dest install: atomically materialize `path` by copying the
* bytes of `basis_path` through a bounded buffer (no whole-file buffering, so
* an arbitrarily large basis works), applying the SOURCE metadata and the
* per-file xattrs / --fake-super record. `update` honors a newer destination;
* a --temp-dir scratch location falls back to a direct write on EXDEV. */
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
unsigned long long expected_size, bool preallocate,
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir);
/* Protocol 2.28.0 receiver-stat variants: like the two above but additionally
* report through `dirs_created` (when non-NULL) how many parent directories the
* confined secure walk had to create that lie strictly below `count_floor` (a
* receive-root-relative prefix, or NULL for all). Used to reproduce rsync's
* `Number of created files` directory count on a fresh destination. */
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, const char* temp_dir,
unsigned* dirs_created, const char* count_floor);
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
const void* data, unsigned long long data_size,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir, unsigned* dirs_created,
const char* count_floor);
/* The logical transfer root expressed receive-root-relative, or NULL when the
* wire paths carry no mirror scaffolding above it. Caller frees non-NULL. */
char* file_transfer_root_floor(const Config* config);
#endif
src/shared/file_receive.c
+233 -113
View File
@@ -37,7 +37,12 @@
#define MANIFEST_ENTRY_OVERHEAD (sizeof(char*) + 16)
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config) {
return file_save_to_disk_full(root_directory, file, config) != FILE_SAVE_ERROR;
return file_save_to_disk_full_ex(root_directory, file, config, NULL, NULL) != FILE_SAVE_ERROR;
}
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
const Config* config) {
return file_save_to_disk_full_ex(root_directory, file, config, NULL, NULL);
}
/* --delay-updates receiver path: write the file into a private staging tree
@@ -94,6 +99,12 @@ static FileSaveResult file_stage_delayed_update(const char* root_directory,
if (file->basis_link) {
ok = file_to_disk_secure_link(staged_path, file->basis_link, file->data->data, file->data->size,
config->preallocate, metadata, policy, config->use_fsync, NULL);
} else if (file->basis_copy) {
/* --copy-dest basis hit: stream the basis into the staging tree (bounded
buffers, so an over-limit basis still stages). */
ok = file_copy_basis_stream_attrs(staged_path, file->basis_copy, file->data->size,
config->preallocate, metadata, policy, config->update,
config->use_fsync, file->xattrs, config->fake_super, NULL);
} else {
ok =
file_to_disk_secure_attrs(staged_path, file->data->data, file->data->size, false, sparse,
@@ -208,13 +219,14 @@ static FileSaveResult hardlink_sibling_absent_first(const char* destination_path
--existing/--ignore-existing/--update policies are decided against the final
destination like every normal write. */
static FileSaveResult file_save_hardlink_sibling(const char* root_directory, const File* file,
const Config* config) {
const Config* config, bool* created) {
Config* cfg = (Config*)config;
if (!root_directory || !file || !file->path || !file->hardlink_target)
return FILE_SAVE_ERROR;
char* destination_path = path_cat(root_directory, file->path);
if (!destination_path)
return FILE_SAVE_ERROR;
bool existed = file_path_exists_secure(destination_path);
if (cfg->existing && !file_path_exists_secure(destination_path)) {
free(destination_path);
@@ -278,6 +290,8 @@ static FileSaveResult file_save_hardlink_sibling(const char* root_directory, con
free(staged_first);
free(staged_sibling);
free(destination_path);
if (ok && created && !existed)
*created = true;
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
}
@@ -324,6 +338,8 @@ static FileSaveResult file_save_hardlink_sibling(const char* root_directory, con
free(content);
free(first_disk);
free(destination_path);
if (ok && created && !existed)
*created = true;
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
}
@@ -360,7 +376,7 @@ bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode) {
* non-device entry must carry an empty rdev.
*/
static FileSaveResult file_save_special_to_disk(const char* root_directory, const File* file,
const Config* config) {
const Config* config, bool* created) {
/* The empty-path and structural checks stay unconditional; the redundant
".." list-path re-check is skipped under --trust-sender exactly like the
receive layer (confinement is deferred to the secure parent walk below,
@@ -412,6 +428,7 @@ static FileSaveResult file_save_special_to_disk(const char* root_directory, cons
char* destination = path_cat(root_directory, file->path);
if (!destination)
return FILE_SAVE_ERROR;
bool existed = file_path_exists_secure(destination);
char* leaf = NULL;
int parent_fd = file_open_secure_parent(destination, &leaf, true);
if (parent_fd < 0) {
@@ -532,6 +549,8 @@ static FileSaveResult file_save_special_to_disk(const char* root_directory, cons
free(destination);
/* A failed required --copy-as ownership marks the node as failed; every other
* identity policy stays best-effort. */
if (owner_ok && created && !existed)
*created = true;
return owner_ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
}
@@ -610,8 +629,13 @@ static FileSaveResult file_save_write_device(const char* root_directory, const F
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_SKIPPED;
}
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
const Config* config) {
FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File* file,
const Config* config, bool* created,
unsigned* created_dirs) {
if (created)
*created = false;
if (created_dirs)
*created_dirs = 0;
/* Central no-mutation guard: a server-contacting --dry-run (or a local batch
apply that somehow carries dry_run) must never touch the destination, no
matter which caller reached this primitive. The per-caller guards remain,
@@ -634,7 +658,8 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
char* destination_path = NULL;
char *backup_path = NULL, *parent_copy = NULL;
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data) ||
if (!file || !file->path || !file->data ||
(file->data->size != 0 && !file->data->data && !file->basis_link && !file->basis_copy) ||
(!file_get_trust_sender() && has_path_traversal(file->path)) ||
(backup_enabled &&
(!backup_suffix || backup_suffix[0] == '\0' || strchr(backup_suffix, '/') != NULL ||
@@ -658,7 +683,7 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
/* Device/special node (--devices/--specials): recreate the node instead of
writing content (privilege-gated, confined, rdev-validated). */
if (file->is_special)
return file_save_special_to_disk(root_directory, file, config);
return file_save_special_to_disk(root_directory, file, config, created);
/* --write-devices: write straight into an existing device node. Writing
into a device is a super-user activity, so --no-super must suppress it just
like device-node creation; the default AUTO/--super attempt it (the wide
@@ -689,6 +714,7 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
char* dir_path = path_cat(root_directory, file->path);
if (!dir_path)
return FILE_SAVE_ERROR;
bool dir_existed = file_path_exists_secure(dir_path);
bool ok = file_ensure_directory_secure(dir_path);
/* P7 Wave E: apply the negotiated ownership to the directory ITSELF (not
just the files inside it). --copy-as and every explicit identity policy
@@ -712,6 +738,8 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
free(leaf);
}
free(dir_path);
if (ok && created && !dir_existed)
*created = true;
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
}
@@ -728,6 +756,7 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
char* link_path = path_cat(root_directory, file->path);
if (!link_path)
return FILE_SAVE_ERROR;
bool link_existed = file_path_exists_secure(link_path);
/* The link value is stored verbatim (rsync -l parity: absolute and
".."-bearing targets are preserved; the scanner's --safe-links /
--copy-unsafe-links decide which links are sent at all). --munge-links
@@ -768,6 +797,8 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
ok = file_restore_symlink_metadata(link_path, file->metadata, link_policy,
config->omit_link_times);
}
if (ok && created && !link_existed)
*created = true;
free(link_path);
return ok ? FILE_SAVE_WRITTEN : FILE_SAVE_ERROR;
}
@@ -777,7 +808,7 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
byte-identical copy of) the group's first member. Handled entirely here,
before the normal data-write paths (which would create an empty file). */
if (file->link_group != 0 && !file->link_first && file->hardlink_target != NULL) {
return file_save_hardlink_sibling(root_directory, file, config);
return file_save_hardlink_sibling(root_directory, file, config, created);
}
/* These options arrive from the client. --backup-dir, --partial-dir and
@@ -808,12 +839,18 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
free(disk_path);
return FILE_SAVE_ERROR;
}
/* Snapshot the final destination's existence BEFORE any backup/force/partial
step can move or remove it, so the receiver can report rsync's
`Number of created files` (protocol 2.28.0). */
bool dest_existed = file_path_exists_secure(destination_path);
/* --delay-updates diverts the whole write into the staging tree; the rest of
this function is the immediate-install path. */
if (config && config->delay_updates) {
FileSaveResult result =
file_stage_delayed_update(root_directory, destination_path, file, (Config*)config);
if (result == FILE_SAVE_WRITTEN && created && !dest_existed)
*created = true;
free(confined_backup);
free(confined_partial);
free(destination_path);
@@ -939,19 +976,30 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
existing/ignore-existing/update/backup preamble above has already made the
policy decision. */
bool ok;
char* count_floor = file_transfer_root_floor(config);
if (config && file->basis_link) {
ok = file_to_disk_secure_link_attrs(
ok = file_to_disk_secure_link_attrs_counted(
disk_path, file->basis_link, file->data->data, file->data->size, config->preallocate,
metadata, policy, config->use_fsync, file->xattrs, config->fake_super, confined_temp);
metadata, policy, config->use_fsync, file->xattrs, config->fake_super, confined_temp,
created_dirs, count_floor);
} else if (config && file->basis_copy) {
/* --copy-dest: stream the basis bytes through a bounded buffer so a basis
larger than any whole-file bound still materializes. The source
metadata was transmitted with the check frame. */
ok = file_copy_basis_stream_attrs(
disk_path, file->basis_copy, file->data->size, config->preallocate, metadata, policy,
config->update, config->use_fsync, file->xattrs, config->fake_super, confined_temp);
} else {
/* The plain no-replace / update / with-fsync engines, plus per-file xattr
(-X/-A) and --fake-super application on the written fd. */
ok = file_to_disk_secure_attrs(
ok = file_to_disk_secure_attrs_counted(
disk_path, file->data->data, file->data->size, inplace, sparse,
config && config->preallocate, metadata, policy, config && config->update,
config && config->ignore_existing, config && config->use_fsync, file->xattrs,
config ? config->fake_super : false, config ? config->partial : false, confined_temp);
config ? config->fake_super : false, config ? config->partial : false, confined_temp,
created_dirs, count_floor);
}
free(count_floor);
free(confined_temp);
confined_temp = NULL;
if (!ok)
@@ -972,6 +1020,8 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
free(confined_partial);
free(destination_path);
free(disk_path);
if (created && !dest_existed)
*created = true;
return FILE_SAVE_WRITTEN;
fail:
@@ -984,6 +1034,37 @@ fail:
return FILE_SAVE_ERROR;
}
void receiver_stats_note_saved(ReceiverStats* stats, const File* file, bool created,
unsigned created_dirs) {
if (!stats || !file)
return;
/* A basis-dir hit (--link-dest/--copy-dest) materializes bytes the sender
* never transferred. rsync reports no literal data and no created entry for
* such a file, and does not count the parent directories it creates only to
* hold it, so exclude the whole entry from the receiver tallies. */
bool basis_sourced = file->basis_link != NULL || file->basis_copy != NULL;
if (basis_sourced)
return;
bool is_sibling = file->link_group != 0 && !file->link_first;
if (!file->is_dir && !file->is_symlink && !file->is_special && !is_sibling) {
unsigned long long literal = file->literal_bytes;
if (literal == 0 && file->matched_bytes == 0)
literal = file->data ? file->data->size : 0;
stats->literal_bytes += literal;
}
stats->created_dir += created_dirs;
if (!created)
return;
if (file->is_dir)
stats->created_dir++;
else if (file->is_symlink)
stats->created_link++;
else if (file->is_special)
stats->created_special++;
else
stats->created_reg++;
}
/* Receive a file's xattr block (when the config enables xattr transport) and
* attach it to `file`. Returns false on a malformed/oversized frame. */
static bool receive_file_xattrs(File* file, int fd, const Config* config) {
@@ -1094,11 +1175,15 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
return NULL;
}
/* Wire-stats tally: bytes taken straight from the basis file (matched
delta blocks). Computed before the delta is destroyed. */
delta blocks) and bytes shipped literally (protocol 2.28.0). Computed
before the delta is destroyed. */
unsigned long long matched = 0;
unsigned long long literal = 0;
for (uint32_t k = 0; k < delta->instruction_count; k++) {
if (delta->instructions[k].type == DELTA_INSTR_BLOCK_MATCH)
matched += delta->instructions[k].match.length;
else if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
literal += delta->instructions[k].literal.length;
}
void* new_data = delta_apply(old_data, old_size, delta, config->delta_block_size);
delta_destroy(delta);
@@ -1119,6 +1204,7 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
return NULL;
}
file->matched_bytes = matched;
file->literal_bytes = literal;
if (config->use_metadata) {
int meta_ok = 1;
@@ -1233,16 +1319,17 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
/* ---- Alternate basis directories (--compare-dest / --copy-dest / --link-dest) ----
* The receiver consults the ordered basis-dir list only when the destination
* entry is NOT already up to date. An "exact match" requires an equal size,
* an equal mtime (unless --size-only), and an equal content xxHash64, so a
* hard link / local copy is only ever made from byte-identical content. */
* entry is NOT already up to date. By default an "exact match" is rsync's
* metadata quick-check: an equal size and an equal mtime (unless --size-only).
* The FastSync-only --verify-basis additionally requires an equal whole-file
* content digest, so a hard link / local copy is only then made from
* byte-verified content. */
typedef struct BasisMatch {
bool hit;
BasisDestType type;
char* basis_path; /* owned absolute path of the matched basis file */
struct stat st; /* fstat() of the matched basis file */
Data* content; /* owned basis bytes (or empty Data), NULL when not loaded */
} BasisMatch;
static void basis_match_free(BasisMatch* match) {
@@ -1250,8 +1337,6 @@ static void basis_match_free(BasisMatch* match) {
return;
free(match->basis_path);
match->basis_path = NULL;
data_destroy(match->content);
match->content = NULL;
match->hit = false;
match->type = BASIS_DEST_NONE;
}
@@ -1283,32 +1368,10 @@ static bool basis_open_regular(const char* path, unsigned long long expected_siz
return true;
}
/* Read the whole remaining content of an open descriptor. A zero-length file
yields an empty Data (data pointer NULL). */
static Data* basis_read_content(int fd, unsigned long long size) {
if (size == 0)
return data_create_reserve(0);
if (size > MAX_RECEIVE_WHOLE_FILE_SIZE || size > SIZE_MAX)
return NULL;
void* buf = protocol_alloc((size_t)size);
if (!buf)
return NULL;
size_t got = 0;
while (got < (size_t)size) {
ssize_t n = read(fd, (char*)buf + got, (size_t)size - got);
if (n <= 0) {
free(buf);
return NULL;
}
got += (size_t)n;
}
return data_create(buf, (size_t)size);
}
/* --ignore-times forces every file to be updated, so no basis hit is ever
declared (matching rsync, where -I prevents link-dest from linking). */
static bool basis_quick_matches(const Config* config, const struct stat* st, time_t check_mtime,
long check_mtime_nsec) {
bool file_basis_quick_match(const Config* config, const struct stat* st, time_t check_mtime,
long check_mtime_nsec) {
if (config->size_only)
return true;
long mtime_nsec = 0;
@@ -1319,28 +1382,39 @@ static bool basis_quick_matches(const Config* config, const struct stat* st, tim
config->modify_window);
}
/* Search the basis-dir list in command-line order and return the first exact
match. When load_content is true the matched bytes are kept in out->content
so the caller can materialize the file without re-reading it.
/* True when a basis hit must be confirmed by a whole-file content digest
(--verify-basis). False is the rsync-parity default: the metadata
quick-check alone decides a hit. */
bool file_basis_content_required(const Config* config) {
return config != NULL && config->verify_basis;
}
An exact match ALSO requires the basis bytes' digest to equal the source's,
so `hash_content` gates the content read/hash itself. A server-contacting
--dry-run passes hash_content=false: no basis file may be read or hashed
(that would be a 1-bit content oracle against a client-supplied digest), so a
metadata-only pass can never confirm a hit and declines it. The real path
always passes hash_content=true, keeping its behavior byte-for-byte. */
/* Search the basis-dir list in command-line order and return the first match.
By default (no --verify-basis) rsync's metadata quick-check is sufficient:
basis_open_regular has already required an equal size, and
file_basis_quick_match applies rsync's mtime (or --size-only) rule.
--verify-basis additionally requires the basis bytes' whole-file digest to
equal the sender's, restoring FastSync's historical content equality; that
digest is computed by streaming the open basis descriptor, so an arbitrarily
large basis is verified without buffering it. A copy/link install re-reads
the basis from its path in bounded buffers, so no content buffer is kept.
`hash_content` gates content READS under --verify-basis: a server-contacting
--dry-run passes false because hashing a basis against a client-supplied
digest would be a 1-bit content oracle. Without --verify-basis a dry-run can
still confirm the metadata-only hit without reading any basis bytes, matching
rsync's read-only quick-check. */
static bool basis_match_find(const Config* config, const char* check_path,
unsigned long long check_size, time_t check_mtime,
long check_mtime_nsec, const uint8_t* check_digest,
size_t check_digest_len, bool load_content, bool hash_content,
BasisMatch* out) {
size_t check_digest_len, bool hash_content, BasisMatch* out) {
memset(out, 0, sizeof(*out));
if (!config || !config_has_basis(config) || config->ignore_times)
return false;
/* Dry-run: never read/hash basis content. A hit cannot be decided from
metadata alone, so report no match (the caller treats it as would-transfer)
without touching the file's contents. */
if (!hash_content)
/* --verify-basis needs the basis content; a content-blind (dry-run) pass can
never confirm it and must not read the file, so decline without touching
the basis bytes. */
if (file_basis_content_required(config) && !hash_content)
return false;
for (int i = 0; i < config->basis_count; i++) {
const BasisDest* entry = &config->basis_dirs[i];
@@ -1359,29 +1433,26 @@ static bool basis_match_find(const Config* config, const char* check_path,
int fd;
struct stat st;
if (basis_open_regular(candidate, check_size, &fd, &st)) {
if (basis_quick_matches(config, &st, check_mtime, check_mtime_nsec)) {
Data* content = basis_read_content(fd, check_size);
if (content) {
if (file_basis_quick_match(config, &st, check_mtime, check_mtime_nsec)) {
bool hit = true;
if (file_basis_content_required(config)) {
uint8_t basis_digest[CHECKSUM_MAX_DIGEST_LEN];
size_t basis_len = 0;
bool hashed = checksum_digest((ChecksumAlgo)config->checksum_algo, config->checksum_seed,
content->data, content->size, basis_digest,
sizeof(basis_digest), &basis_len);
if (hashed && basis_len == check_digest_len && check_digest_len > 0 &&
memcmp(basis_digest, check_digest, check_digest_len) == 0) {
out->hit = true;
out->type = entry->type;
out->basis_path = candidate;
candidate = NULL; /* ownership transferred to out */
out->st = st;
out->content = load_content ? content : NULL;
if (!load_content)
data_destroy(content);
close(fd);
return true;
}
bool hashed =
checksum_digest_fd((ChecksumAlgo)config->checksum_algo, config->checksum_seed, fd,
basis_digest, sizeof(basis_digest), &basis_len);
hit = hashed && basis_len == check_digest_len && check_digest_len > 0 &&
memcmp(basis_digest, check_digest, check_digest_len) == 0;
}
if (hit) {
out->hit = true;
out->type = entry->type;
out->basis_path = candidate;
candidate = NULL; /* ownership transferred to out */
out->st = st;
close(fd);
return true;
}
data_destroy(content);
}
close(fd);
}
@@ -1806,6 +1877,12 @@ typedef struct {
long long check_mtime_nsec;
uint8_t check_digest[CHECKSUM_MAX_DIGEST_LEN];
size_t check_digest_len;
/* Source metadata carried alongside the check frame whenever a basis dir is
configured (rsync keeps the whole file list; FastSync's sender-driven
incremental path otherwise never transmits metadata for a SKIPPED file).
A basis materialization applies these SOURCE attributes instead of the
basis inode's, matching rsync's "copy then fix attributes". */
FileMetadata* source_metadata;
bool dest_exists; /* any destination entry exists (lstat succeeded) */
bool has_old_file;
int old_fd;
@@ -1838,6 +1915,8 @@ static void incremental_check_state_cleanup(IncrementalCheckState* state) {
if (state->old_fd >= 0)
close(state->old_fd);
state->old_fd = -1;
file_metadata_destroy(state->source_metadata);
state->source_metadata = NULL;
free(state->full_path);
state->full_path = NULL;
free(state->check_path);
@@ -1862,7 +1941,7 @@ static IncrementalCheckOutcome incremental_check_receive_request(IncrementalChec
send_error_detail(fd, "invalid check mtime nanoseconds");
return INCREMENTAL_ERROR;
}
if ((config->checksum || config_has_basis(config))) {
if ((config->checksum || config->verify_basis)) {
uint8_t wire_len;
if (!receive_n_data(fd, &wire_len, sizeof(wire_len)) || wire_len == 0 ||
wire_len > CHECKSUM_MAX_DIGEST_LEN ||
@@ -1874,8 +1953,24 @@ static IncrementalCheckOutcome incremental_check_receive_request(IncrementalChec
if (!receive_n_data(fd, state->check_digest, state->check_digest_len))
return INCREMENTAL_ERROR;
}
/* The sender transmits the source metadata with every basis-configured check
so a basis hit can be materialized with the SOURCE's attributes (rsync
copies/copies-then-fixes; the receiver would otherwise only have the basis
inode's stat). The block is symmetric and consumed unconditionally here,
whether or not this file ends up as a basis hit. */
if (config_has_basis(config) && config->use_metadata) {
int meta_ok = 1;
state->source_metadata = metadata_receive(fd, &meta_ok);
if (!meta_ok)
return INCREMENTAL_ERROR;
}
if (state->check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
/* A basis-configured run may materialize a file larger than the whole-file
payload bound: a basis hit is streamed from the basis path (bounded
buffers), so the check size is not itself an allocation. Every other
path (delta/append/full) still applies MAX_RECEIVE_WHOLE_FILE_SIZE, and a
miss simply falls through to the normal transfer with its own bound. */
if (!config_has_basis(config) && state->check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
send_error_detail(fd, "check size exceeds receiver limit");
return INCREMENTAL_ERROR;
}
@@ -1965,6 +2060,20 @@ incremental_check_ignore_existing(const IncrementalCheckState* state) {
return INCREMENTAL_SKIP;
}
/* Metadata for a materialized basis hit: prefer the SOURCE metadata the sender
transmitted with the check frame (rsync copies then fixes the destination to
the source's attributes); fall back to the basis inode's own stat when
metadata was not negotiated. Consumes state->source_metadata on success. */
static FileMetadata* basis_take_metadata(IncrementalCheckState* state,
const struct stat* basis_st) {
if (state->source_metadata) {
FileMetadata* meta = state->source_metadata;
state->source_metadata = NULL;
return meta;
}
return file_metadata_create(NULL, basis_st, false, false);
}
/* --link-dest relink of an already up-to-date destination. rsync hard-links a
destination entry to a matching basis even when the entry is already correct,
so a run over an existing tree still maximizes sharing with the basis. Only a
@@ -1982,7 +2091,7 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
BasisMatch basis;
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
true, true, &basis);
true, &basis);
/* Only a link-dest hit relinks; a copy-dest/compare-dest hit (or a miss) lets
the up-to-date check below keep the existing destination. */
if (!basis.hit || basis.type != BASIS_DEST_LINK) {
@@ -1995,11 +2104,16 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
return INCREMENTAL_CONTINUE;
}
File* materialized = file_create(state->check_path);
if (materialized && basis.content) {
if (materialized) {
data_destroy(materialized->data);
materialized->data = basis.content;
basis.content = NULL;
materialized->metadata = file_metadata_create(NULL, &basis.st, false, false);
materialized->data = data_create_reserve((size_t)state->check_size);
if (!materialized->data) {
file_destroy(materialized);
materialized = NULL;
}
}
if (materialized) {
materialized->metadata = basis_take_metadata(state, &basis.st);
materialized->skip = true;
materialized->basis_link = basis.basis_path;
basis.basis_path = NULL;
@@ -2007,9 +2121,6 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
file_destroy(materialized);
materialized = NULL;
}
} else {
file_destroy(materialized);
materialized = NULL;
}
if (materialized) {
if (!send_status(state->fd, STATUS_OK)) {
@@ -2110,12 +2221,13 @@ static IncrementalCheckOutcome incremental_check_quick_skip(IncrementalCheckStat
materialize nothing (no basis link/copy, no append/delta/full transfer) and
the sender must send no data, so answer STATUS_DRY_RUN_TRANSFER and stop.
The basis lookup is deliberately content-blind: a real run would only accept
a --compare-dest exact hit after hashing the basis file and comparing it with
the client-supplied digest, which in a dry-run is a 1-bit content oracle.
Under dry_run no basis bytes may be read, so an otherwise-matching entry is
treated as would-transfer instead of a skip. Everything read here (the
destination file's metadata, basis candidates' metadata) is read-only. */
The basis lookup is content-blind: under the default metadata quick-check a
hit needs no basis bytes and is honored here just as in a real run; under
--verify-basis a real run hashes the basis against the client-supplied digest,
which in a dry-run is a 1-bit content oracle, so no basis bytes may be read
and an otherwise-matching entry is reported as would-transfer. Everything
read here (the destination file's metadata, basis candidates' metadata) is
read-only. */
static IncrementalCheckOutcome incremental_check_dry_run_shortcut(IncrementalCheckState* state,
bool* skipped,
bool* would_transfer) {
@@ -2126,12 +2238,14 @@ static IncrementalCheckOutcome incremental_check_dry_run_shortcut(IncrementalChe
bool skip_via_compare = false;
if (config_has_basis(config) && !config->ignore_times) {
BasisMatch basis;
/* hash_content=false: a dry-run must not read or hash the basis file. No
content comparison is possible, so no compare-dest hit can be confirmed
and an otherwise-matching file is reported as would-transfer. */
/* hash_content=false: a dry-run must not read or hash the basis file, so
under --verify-basis no compare-dest hit can be confirmed and an
otherwise-matching file is reported as would-transfer. Without
--verify-basis the metadata quick-check confirms it without touching any
basis bytes. */
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
false, false, &basis);
false, &basis);
if (basis.hit && basis.type == BASIS_DEST_COMPARE && !state->has_old_file)
skip_via_compare = true;
basis_match_free(&basis);
@@ -2159,7 +2273,7 @@ static IncrementalCheckOutcome incremental_check_try_basis(IncrementalCheckState
BasisMatch basis;
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
true, true, &basis);
true, &basis);
if (basis.hit) {
if (basis.type == BASIS_DEST_COMPARE) {
basis_match_free(&basis);
@@ -2169,24 +2283,30 @@ static IncrementalCheckOutcome incremental_check_try_basis(IncrementalCheckState
return INCREMENTAL_SKIP;
}
} else {
/* Copy/link installs source their bytes from the basis PATH at install
time (bounded buffers), so no whole-file content buffer is needed here
even for an over-limit basis. */
File* materialized = file_create(state->check_path);
if (materialized && basis.content) {
if (materialized) {
data_destroy(materialized->data);
materialized->data = basis.content;
basis.content = NULL;
materialized->metadata = file_metadata_create(NULL, &basis.st, false, false);
materialized->skip = true; /* receiver must not ack this as a data file */
if (basis.type == BASIS_DEST_LINK) {
materialized->basis_link = basis.basis_path;
basis.basis_path = NULL;
materialized->data = data_create_reserve((size_t)state->check_size);
if (!materialized->data) {
file_destroy(materialized);
materialized = NULL;
}
}
if (materialized) {
materialized->metadata = basis_take_metadata(state, &basis.st);
materialized->skip = true; /* receiver must not ack this as a data file */
if (basis.type == BASIS_DEST_LINK)
materialized->basis_link = basis.basis_path;
else
materialized->basis_copy = basis.basis_path;
basis.basis_path = NULL;
if (!materialized->metadata) {
file_destroy(materialized);
materialized = NULL;
}
} else {
file_destroy(materialized);
materialized = NULL;
}
if (materialized) {
if (!send_status(fd, STATUS_OK)) {
@@ -3263,7 +3383,7 @@ static bool delete_extras_budgeted_observed(const Config* config, DeleteManifest
size_t skipped = 0;
DeleteWalkResult result = delete_extras_limited_observed(
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips, used,
&deleted, &skipped, observer, observer_context);
config->protect_rules, &deleted, &skipped, observer, observer_context);
if (owned_prefixes) {
for (int i = 0; i < config->basis_count; i++)
free(owned_prefixes[i]);
@@ -3444,7 +3564,7 @@ static bool delete_missing_args_budgeted_observed(const Config* config, DeleteMa
PrefixedDeleteObserver nested = {observer, observer_context, rel};
DeleteWalkResult walk =
no_keeps ? delete_extras_limited_observed(full, no_keeps, NULL, remaining, NULL, 0,
&contents_deleted, &contents_skipped,
NULL, &contents_deleted, &contents_skipped,
observer ? prefixed_delete_observer : NULL,
observer ? &nested : NULL)
: DELETE_WALK_ERROR;
@@ -3555,7 +3675,7 @@ bool manifest_would_delete_list(const Config* config, DeleteManifest* manifest,
used = idx;
}
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
skips, used, out, count_out);
skips, used, config->protect_rules, out, count_out);
if (owned_prefixes) {
for (int i = 0; i < config->basis_count; i++)
free(owned_prefixes[i]);
src/shared/file_receive.h
+26
View File
@@ -24,6 +24,16 @@ File* file_receive_hardlink(int file_descriptor);
File* file_receive_symlink(int file_descriptor, const Config* config);
File* file_receive_special(int file_descriptor);
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
/* Testable basis quick-check / verification policy. file_basis_quick_match is
* rsync's metadata quick-check for a basis candidate (equal size is required
* separately by the caller; this adds the --size-only / mtime / --modify-window
* leg). file_basis_content_required reports whether a hit must ALSO be
* confirmed by a whole-file content digest (--verify-basis; false is the
* default rsync-parity behavior). */
bool file_basis_quick_match(const Config* config, const struct stat* st, time_t check_mtime,
long check_mtime_nsec);
bool file_basis_content_required(const Config* config);
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
* true only on the server-contacting --dry-run path when the file is not up to
@@ -179,6 +189,22 @@ typedef enum { FILE_SAVE_ERROR = 0, FILE_SAVE_WRITTEN = 1, FILE_SAVE_SKIPPED = 2
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
const Config* config);
/* Protocol 2.28.0 variant: also reports through `created` (when non-NULL)
* whether the destination entry did not exist before this save, and through
* `created_dirs` how many parent directories the confined walk created, so the
* receiver can build rsync's `Number of created files` breakdown. The plain
* file_save_to_disk_full() is this with both out-params NULL. */
FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File* file,
const Config* config, bool* created,
unsigned* created_dirs);
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
/* Protocol 2.28.0 receiver counter accumulator: fold one successfully saved
* entry into `stats`, adding its receiver-observed literal bytes and, when
* `created`, the matching created-by-type counter (regular file / symlink /
* special) plus `created_dirs` implicitly-created parent directories.
* Non-first hardlink siblings contribute no literal bytes. */
void receiver_stats_note_saved(ReceiverStats* stats, const File* file, bool created,
unsigned created_dirs);
#endif
src/shared/file_types.h
+10
View File
@@ -57,6 +57,11 @@ typedef struct {
* equals the incoming file, and `data` is kept as the cross-filesystem
* fallback (a local copy) if the hard link cannot be created. */
char* basis_link;
/* Receiver-only, --copy-dest: when set (and basis_link is NULL), stream the
* basis file's bytes into the destination instead of `data`/`data->size`.
* This lets a basis larger than any whole-file bound materialize without
* buffering it; the source metadata on `metadata` is applied afterwards. */
char* basis_copy;
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
* id shared by every member of one source inode (0 = not part of a group).
* The FIRST member (link_first == true) carries its data on the wire and is
@@ -97,6 +102,11 @@ typedef struct {
* basis file (matched delta blocks) for this entry. 0 when the file was sent
* whole. Accumulated into ReceiverStats.matched_data by the receiver sink. */
unsigned long long matched_bytes;
/* Receiver-only (protocol 2.28.0) wire-stats tally: the literal delta fragment
* bytes this entry carried (DELTA_INSTR_LITERAL). 0 when the file was sent
* whole; the sink then falls back to the whole payload size. Accumulated
* into ReceiverStats.literal_bytes. */
unsigned long long literal_bytes;
} File;
/* The path that should be sent on the wire and used for the receiver-side
src/shared/filter.h
+1 -1
View File
@@ -53,7 +53,7 @@ typedef struct {
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
} FilterRule;
typedef struct {
typedef struct FilterRuleList {
FilterRule** items; /* owned array of rule pointers */
int count;
int capacity;
src/shared/format.c
+15 -13
View File
@@ -105,25 +105,27 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
bool format_stats_send(int fd, const ReceiverStats* stats) {
if (!stats)
return false;
unsigned long long matched = stats->matched_data;
unsigned long long deleted = stats->deleted_files;
unsigned long long would = stats->would_delete_count;
return send_n_data(fd, &matched, sizeof(matched)) && send_n_data(fd, &deleted, sizeof(deleted)) &&
send_n_data(fd, &would, sizeof(would));
unsigned long long fields[8] = {
stats->matched_data, stats->deleted_files, stats->would_delete_count, stats->literal_bytes,
stats->created_reg, stats->created_dir, stats->created_link, stats->created_special,
};
return send_n_data(fd, fields, sizeof(fields));
}
bool format_stats_receive(int fd, ReceiverStats* stats) {
if (!stats)
return false;
unsigned long long matched = 0;
unsigned long long deleted = 0;
unsigned long long would = 0;
if (!receive_n_data(fd, &matched, sizeof(matched)) ||
!receive_n_data(fd, &deleted, sizeof(deleted)) || !receive_n_data(fd, &would, sizeof(would)))
unsigned long long fields[8] = {0};
if (!receive_n_data(fd, fields, sizeof(fields)))
return false;
memset(stats, 0, sizeof(*stats));
stats->matched_data = matched;
stats->deleted_files = deleted;
stats->would_delete_count = would;
stats->matched_data = fields[0];
stats->deleted_files = fields[1];
stats->would_delete_count = fields[2];
stats->literal_bytes = fields[3];
stats->created_reg = fields[4];
stats->created_dir = fields[5];
stats->created_link = fields[6];
stats->created_special = fields[7];
return true;
}
src/shared/format.h
+16 -4
View File
@@ -57,14 +57,26 @@ bool format_dest_state_send(int fd, const OutputDestState* state);
bool format_dest_state_receive(int fd, OutputDestState* state);
/* End-of-transfer receiver counters reported through STATUS_STATS (protocol
* 2.25.0) when the wire config carries report_stats. `would_delete_count` is
* the number of destination-relative paths the receiver would have deleted in a
* -n/--dry-run --delete run; that many wire strings immediately follow the
* fixed record (sent/read by the caller). */
* 2.25.0, extended in 2.28.0) when the wire config carries report_stats.
* `would_delete_count` is the number of destination-relative paths the receiver
* would have deleted in a -n/--dry-run --delete run; that many wire strings
* immediately follow the fixed record (sent/read by the caller).
*
* Protocol 2.28.0 adds the receiver-observed counters the sender cannot see:
* `literal_bytes` is the file data the receiver actually stored literally
* (whole files plus the literal fragments of a delta) and the four `created_*`
* counters split the destination entries the receiver newly created by type,
* reproducing rsync's `Number of created files` breakdown and an exact
* `Literal data` for a delta run. */
typedef struct {
unsigned long long matched_data;
unsigned long long deleted_files;
unsigned long long would_delete_count;
unsigned long long literal_bytes;
unsigned long long created_reg;
unsigned long long created_dir;
unsigned long long created_link;
unsigned long long created_special;
} ReceiverStats;
/* Fixed-width STATUS_STATS counter record. The status frame and the optional
src/shared/log.h
+5
View File
@@ -33,6 +33,11 @@ typedef enum {
LOG_INFO_FLIST = 1u << 7,
LOG_INFO_NONREG = 1u << 8,
LOG_INFO_PROGRESS = 1u << 9,
/* Marker for `--info=name2` and higher: also print rsync's
"NAME is uptodate" line for entries the receiver already has. It rides in
the info_level bitset (there is no separate Config field) and is never set
by --info=all (which selects level 1). */
LOG_INFO_NAME_UPTODATE = 1u << 10,
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS | LOG_INFO_DEL |
LOG_INFO_REMOVE | LOG_INFO_NAME | LOG_INFO_FLIST | LOG_INFO_NONREG |
LOG_INFO_PROGRESS,
src/shared/protocol.h
+3 -1
View File
@@ -191,7 +191,9 @@ enum NET_STATUS {
* (--delete-delay). Payload: an int32 has_config flag (1 on the first plan
* of the run, 0 afterwards); when set, the three global config sections
* (protected-prefix count+paths, size-skipped count+paths, missing-args
* count+paths); then the destination-relative directory path wire string
* count+paths); then an int32 apply flag (1 for a real plan, 0 for a
* config-only carrier frame that must not walk a directory); then the
* destination-relative directory path wire string
* ("." for the receive root); then the child-directory count + names and the
* child-file count + names that must be kept. Appended after
* STATUS_DEST_INFO so no existing status is renumbered. */
src/shared/utils.c
+82 -14
View File
@@ -2,6 +2,7 @@
#include "array_list.h"
#include "log.h"
#include <arpa/inet.h>
#include <ctype.h>
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
@@ -142,6 +143,47 @@ char* str_dup(const char* string) {
return new_string;
}
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified) {
if (specified)
*specified = false;
if (!env_name || !resolve)
return -1;
const char* env = getenv(env_name);
if (!env)
return -1;
bool saw_nonblank = false;
const char* p = env;
while (*p) {
if (*p == '&')
break;
if (isspace((unsigned char)*p)) {
p++;
continue;
}
saw_nonblank = true;
char token[64];
size_t len = 0;
while (*p && *p != '&' && !isspace((unsigned char)*p)) {
if (len < sizeof(token) - 1)
token[len++] = *p;
p++;
}
token[len] = '\0';
if (len > 0) {
int id = resolve(token);
if (id >= 0) {
if (specified)
*specified = true;
return id;
}
}
}
if (specified)
*specified = saw_nonblank;
return -1;
}
#define STR_HASH_SET_MIN_CAPACITY 16
static size_t str_hash_set_hash(const char* key, size_t len) {
@@ -640,7 +682,8 @@ static bool is_synced_dir(const PathIndex* dirs, const char* rel) {
like any other non-directory extra (never followed). */
static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
const PathIndex* dirs, DeleteBudget* budget,
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, bool parent_deletable,
bool* all_removed, DeletePathObserver observer,
void* observer_context) {
/* openat(dirfd, ".") opens an independent file description: a dup() would
@@ -687,12 +730,23 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
free(child_rel);
continue;
}
if (S_ISDIR(st.st_mode)) {
bool is_dir = S_ISDIR(st.st_mode);
if (protect_rules && filter_rules_apply_side(protect_rules, child_rel, entry->d_name, is_dir,
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
/* A first-match protect rule shields the extra; for a directory the whole
subtree is shielded (rsync prunes an excluded directory), so do not
descend. */
local_survives = true;
free(child_rel);
continue;
}
if (is_dir) {
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count, deletable,
&child_all_removed, observer, observer_context))
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count,
protect_rules, deletable, &child_all_removed, observer,
observer_context))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
@@ -760,7 +814,8 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
reportable children (depth-first), matching the delete pass's ordering. */
static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
const PathIndex* dirs, ArrayList* out, size_t* recorded,
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, bool parent_deletable,
bool* all_removed) {
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (scanfd < 0)
@@ -794,12 +849,21 @@ static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* kee
free(child_rel);
continue;
}
if (S_ISDIR(st.st_mode)) {
bool is_dir = S_ISDIR(st.st_mode);
if (protect_rules && filter_rules_apply_side(protect_rules, child_rel, entry->d_name, is_dir,
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
/* Mirror the delete walk: a protected entry is never reported as a
would-delete and a protected directory's subtree is not enumerated. */
local_survives = true;
free(child_rel);
continue;
}
if (is_dir) {
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
deletable, &child_all_removed))
protect_rules, deletable, &child_all_removed))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
@@ -850,7 +914,7 @@ static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* kee
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
ArrayList* out, size_t* count_out) {
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out) {
if (count_out)
*count_out = 0;
if (!manifest || !out)
@@ -886,7 +950,7 @@ bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
bool all_removed = false;
size_t recorded = 0;
bool ok = list_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, out, &recorded, skips,
skip_count, false, &all_removed);
skip_count, protect_rules, false, &all_removed);
if (close(rootfd) != 0)
ok = false;
path_index_free(&keep);
@@ -900,6 +964,7 @@ bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer,
void* observer_context) {
@@ -942,8 +1007,9 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
}
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
bool all_removed = false;
bool ok = delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips,
skip_count, false, &all_removed, observer, observer_context);
bool ok =
delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips, skip_count,
protect_rules, false, &all_removed, observer, observer_context);
if (close(rootfd) != 0)
ok = false;
path_index_free(&keep);
@@ -961,13 +1027,15 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
size_t* deleted_out, size_t* skipped_out) {
const FilterRuleList* protect_rules, size_t* deleted_out,
size_t* skipped_out) {
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
skip_count, deleted_out, skipped_out, NULL, NULL);
skip_count, protect_rules, deleted_out, skipped_out, NULL,
NULL);
}
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL) ==
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL, NULL) ==
DELETE_WALK_OK;
}
src/shared/utils.h
+21 -3
View File
@@ -2,6 +2,7 @@
#define UTILS_H
#include "array_list.h"
#include "filter.h"
#include <stddef.h>
#include <stdbool.h>
#include <stdio.h>
@@ -81,6 +82,17 @@ bool path_index_has_descendant(const PathIndex* index, const char* path);
char* str_dup(const char* string);
char* output_escape(const char* string, bool eight_bit_output);
/* Resolve the first supported name from a rsync algorithm-preference
* environment variable (RSYNC_COMPRESS_LIST / RSYNC_CHECKSUM_LIST). `resolve`
* maps a case-insensitive name to an algorithm id (>= 0) or -1 for an unknown
* name. rsync's syntax is a whitespace-separated list (comma/colon are NOT
* separators); the client-side half ends at '&'. Unknown entries are skipped
* and the first resolvable one wins. *specified is set true when the variable
* holds at least one non-blank character. Returns the first resolvable id, or
* -1 when the variable is unset/blank or names no supported algorithm. */
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified);
/* Upper bound on one line/token read from a local list file (--files-from,
* --exclude-from/--include-from, .rsync-filter). Mirrors MAX_STRING_SIZE and
* stops a hostile multi-gigabyte line from forcing unbounded allocation. */
@@ -137,7 +149,8 @@ bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSki
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
size_t* deleted_out, size_t* skipped_out);
const FilterRuleList* protect_rules, size_t* deleted_out,
size_t* skipped_out);
/* Optional per-deletion observer: called for each destination-relative path
actually removed (a file, symlink, or directory), in removal order, so the
@@ -145,10 +158,15 @@ DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* m
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
observer; the observer is invoked only for entries truly removed. */
* observer; the observer is invoked only for entries truly removed. When
* `protect_rules` is non-NULL its receiver-side verdict is evaluated for every
* candidate extra: a first-match PROTECT leaves the entry (and, for a
* directory, its whole subtree) in place, while RISK/NONE fall through to the
* ordinary skip-prefix/keep-set logic. */
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer,
void* observer_context);
@@ -159,7 +177,7 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
strings appended to `out` and receives their count in *count_out. */
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
ArrayList* out, size_t* count_out);
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out);
bool delete_extras(const char* dest_root, const ArrayList* manifest);
/* Open the existing destination directory at `dest_root`, confined to the
authorized root with an O_NOFOLLOW component walk (the same confinement the
test_data-manual/dst/f.bin
+1
View File
@@ -0,0 +1 @@
OLDDEST
test_data-manual/dst/test_data-manual/src/f.bin
+1
View File
@@ -0,0 +1 @@
NEWCONTENT
test_data-manual/src/f.bin
+1
View File
@@ -0,0 +1 @@
NEWCONTENT
tests/fuzz/fuzz_config_receive.c
+7
View File
@@ -119,6 +119,13 @@ static void build_canonical_frame(void) {
cfg->usermap[0].to = MAP_TO;
cfg->usermap[0].to_name = NULL;
}
/* Force a non-empty receiver delete-protection block so the fuzzer mutates
* its rule count, action/sides codes and pattern strings. */
cfg->filters = array_list_create(free);
if (cfg->filters) {
array_list_add(cfg->filters, str_dup("P *.log"));
array_list_add(cfg->filters, str_dup("+r keep/**"));
}
if (!cfg->send_directory || !cfg->receive_root_directory || !cfg->usermap) {
config_delete(cfg);
return;
tests/integration/parity_harness.py
+79 -3
View File
@@ -54,12 +54,16 @@ STDOUT_NONE = None
STDOUT_ITEMIZE = "itemize"
STDOUT_OUTFMT = "outfmt"
STDOUT_STATS = "stats"
STDOUT_PROGRESS = "progress"
# rsync --stats lines that are protocol-independent and must match exactly.
# Deliberately excluded: the per-type "Number of files"/"Number of created
# files" breakdown and Total bytes sent/received (documented residual, see the
# `--stats` row in RSYNC_COMPAT.md).
# `Number of files` and `Number of created files` carry rsync's per-type
# breakdown; protocol 2.28.0 reports the receiver-created split over
# STATUS_STATS. Deliberately excluded: Total bytes sent/received (protocol
# framing differs, see the `--stats` row in RSYNC_COMPAT.md).
STATS_KEYS = (
"Number of files",
"Number of created files",
"Number of deleted files",
"Number of regular files transferred",
"Total file size",
@@ -70,6 +74,8 @@ STATS_KEYS = (
)
_ITEMIZE_RE = re.compile(r"^(<|>|c|h|\.|\*)[fdLDS][.+\-][.+\-][.+\-][.+\-]")
_PROGRESS_TOTAL_RE = re.compile(r"to-chk=\d+/(\d+)")
_PROGRESS_XFR_RE = re.compile(r"xfr#(\d+)")
@dataclass
@@ -181,6 +187,24 @@ def corpus_empty_dir(root: str) -> None:
_write(os.path.join(root, "nonempty", "f.txt"), b"f\n")
def corpus_multidir(root: str) -> None:
"""Multi-directory tree for the --progress file-list naming/denominator.
Nested files, a directory-only branch, an empty directory and a symlink
exercise every file-list entry type rsync counts in `to-chk` but FastSync's
streaming scanner never emits as a transfer entry.
"""
clean_dir(root)
_write(os.path.join(root, "a.txt"), b"alpha\n")
_write(os.path.join(root, "b.txt"), b"bravo\n")
_write(os.path.join(root, "sub1", "c.txt"), b"charlie\n")
_write(os.path.join(root, "sub1", "deep", "d.txt"), b"delta\n")
_write(os.path.join(root, "sub2", "e.txt"), b"echo\n")
os.symlink("a.txt", os.path.join(root, "link1"))
os.makedirs(os.path.join(root, "emptydir"), exist_ok=True)
os.utime(os.path.join(root, "emptydir"), (_SRC_MTIME, _SRC_MTIME))
def corpus_relative(root: str) -> None:
"""Tree for the -R/--files-from cases."""
clean_dir(root)
@@ -205,6 +229,18 @@ def corpus_iconv(root: str) -> None:
os.utime(full, (_SRC_MTIME, _SRC_MTIME))
# Payload for the --fuzzy basis corpus: large enough for the delta engine's
# 16 KiB minimum and with repeated content so a coinciding basis yields a
# non-zero (and identical) Matched data count in both tools.
FUZZY_PAYLOAD = (b"the quick brown fox jumps over the lazy dog\n" * 2000)[:65536]
def corpus_fuzzy(root: str) -> None:
"""A named regular file; the `fuzzy` seed adds the similar-suffix sibling."""
clean_dir(root)
_write(os.path.join(root, "report_v2.txt"), FUZZY_PAYLOAD)
CORPORA: Dict[str, Callable[[str], None]] = {
"basic": corpus_basic,
"unicode": corpus_unicode,
@@ -213,8 +249,10 @@ CORPORA: Dict[str, Callable[[str], None]] = {
"sparse": corpus_sparse,
"filters": corpus_filters,
"empty_dir": corpus_empty_dir,
"multidir": corpus_multidir,
"relative": corpus_relative,
"iconv": corpus_iconv,
"fuzzy": corpus_fuzzy,
}
@@ -379,6 +417,37 @@ def normalize_stdout(text: str, mode: Optional[str]) -> object:
if line.startswith(key + ":"):
found[key] = _parse_bytes(line.split(":", 1)[1])
return found
if mode == STDOUT_PROGRESS:
# rsync prints the file-list entries in sorted depth-first order while
# FastSync's streaming scan emits them in readdir/BFS order; only the
# entry set and deterministic fields are compared. The transfer-root
# `./` line's trigger condition is a separate documented residual, and
# the per-frame rate/elapsed/xfr#/to-chk numerator are wall-clock- or
# order-dependent, so only the `to-chk` denominator and the name set are
# asserted.
names = []
totals = set()
max_xfr = 0
for line in (text or "").splitlines():
line = line.rstrip()
if not line:
continue
if "%" in line:
m = _PROGRESS_TOTAL_RE.search(line)
if m:
totals.add(int(m.group(1)))
mx = _PROGRESS_XFR_RE.search(line)
if mx:
max_xfr = max(max_xfr, int(mx.group(1)))
continue
if line == "sending incremental file list":
continue
if line.startswith("created directory "):
continue
if line == "./":
continue
names.append(line)
return {"names": sorted(names), "total": sorted(totals), "xfr": max_xfr}
# raw
return sorted(l.rstrip() for l in (text or "").splitlines() if l.strip())
@@ -390,6 +459,8 @@ def stdout_diff(rsync_out: str, fs_out: str, mode: Optional[str]) -> List[str]:
return []
if mode == STDOUT_STATS:
return [f"stats rsync={r}", f"stats fastsync={f}"]
if mode == STDOUT_PROGRESS:
return [f"progress rsync={r}", f"progress fastsync={f}"]
return list(difflib.unified_diff(
[str(x) for x in r], [str(x) for x in f],
fromfile="rsync", tofile="fastsync", lineterm=""))
@@ -443,6 +514,11 @@ def run_differential( # noqa: PLR0913 (explicit scenario parameters)
rroot, froot = os.path.join(rdst, rel), get_dest_received_dir(fdst, src)
else:
rroot, froot = rdst, get_dest_received_dir(fdst, src)
# rsync's destination root always exists (clean_dir created it). FastSync's
# logical transfer root is the mirror path below the destination argument,
# so pre-create it too: `Number of created files` counts the root only when
# it is genuinely absent, and the two tools must start from the same state.
os.makedirs(froot, exist_ok=True)
if seed:
seed(src, rroot, froot)
tests/integration/test_client_cli.py
+213
View File
@@ -0,0 +1,213 @@
"""Differential tests for the client CLI's codec defaults and env lists.
Track 3a of the rsync-parity plan pins two rsync 3.4.1 behaviors that are
resolved entirely on the client:
* the per-codec default ``--compress-level`` (zstd 3, zlib/zlibx 6, lz4
ignored) applied when the user omits ``--compress-level``/``--zl``, with an
explicit level clamped to the codec's range; and
* the ``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` preference lists that
rsync's ``auto`` consults before its compiled-in order (whitespace-separated,
unknown names skipped, first supported wins, all-unknown is exit 4).
The rsync side is observed through ``--debug=NSTR1``; FastSync publishes its
resolved codec/level through ``--debug=util``. The checksum side is confirmed
byte-for-byte through ``--out-format %C``. The rsync-based tests skip cleanly
when rsync is not installed.
"""
import os
import re
import shutil
import subprocess
import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import (
TEST_DATA_DIR,
run_client,
clean_dir,
get_dest_received_dir,
)
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
CODEC_ROOT = os.path.join(TEST_DATA_DIR, "cli_differential")
_COMPRESS_RE = re.compile(r"compress(?:ion)?: (\w+) \(level (-?\d+)\)")
def _rsync(args):
env = dict(os.environ, LC_ALL="C")
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120)
def _scratch(tag):
path = os.path.join(CODEC_ROOT, tag)
clean_dir(path)
os.makedirs(path, exist_ok=True)
return path
def _make_corpus(root):
clean_dir(root)
os.makedirs(root, exist_ok=True)
with open(os.path.join(root, "big.bin"), "wb") as fh:
fh.write(b"FastSync codec payload " * 4096)
with open(os.path.join(root, "small.txt"), "wb") as fh:
fh.write(b"hello codec world\n" * 32)
return root
def _rsync_compress_level(choice, level):
src = _make_corpus(_scratch(f"lvl_src_{choice}_{level}"))
dst = _scratch(f"lvl_rsync_{choice}_{level}")
args = ["-a", "-z", f"--zc={choice}"]
if level is not None:
args.append(f"--zl={level}")
args += ["--debug=NSTR1", src + "/", dst + "/"]
result = _rsync(args)
assert result.returncode == 0, result.stderr
match = _COMPRESS_RE.search(result.stdout + result.stderr)
assert match, (result.stdout, result.stderr)
return match.group(1), int(match.group(2))
def _fastsync_compress_level(choice, level, shared_server):
src = _make_corpus(_scratch(f"lvl_src_fs_{choice}_{level}"))
dst = _scratch(f"lvl_fs_{choice}_{level}")
args = ["-a", "-z", f"--zc={choice}"]
if level is not None:
args.append(f"--zl={level}")
args += ["-v", "--debug=util"]
result, _ = run_client(src, dst, flags=args, port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
match = _COMPRESS_RE.search(result.stdout)
assert match, result.stdout[:500]
return match.group(1), int(match.group(2))
class TestPerCodecCompressionLevelDefaults:
"""``--compress-level`` defaults and clamping match rsync per codec."""
# FastSync uses a positive lz4 placeholder because its "level > 0" gate
# enables compression; lz4_compress ignores the value, so rsync's level 0
# and FastSync's level 1 produce the same bytes.
CASES = [
("zstd", None, 3),
("zlib", None, 6),
("zlibx", None, 6),
("lz4", None, 1),
("zstd", 10, 10),
("zlib", 15, 9),
("zlib", 3, 3),
("lz4", 15, 1),
]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("choice,level,fs_level", CASES)
def test_level_matches_rsync(self, choice, level, fs_level, shared_server):
rsync_algo, rsync_level = _rsync_compress_level(choice, level)
fs_algo, fs_level_actual = _fastsync_compress_level(choice, level, shared_server)
assert rsync_algo == choice
assert fs_algo == choice
if choice == "lz4":
assert rsync_level == 0 and fs_level_actual > 0
else:
assert rsync_level == fs_level
assert fs_level_actual == fs_level
class TestEnvPreferenceLists:
"""``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` drive auto like rsync."""
# (env value, expected codec, rsync level, FastSync level)
COMPRESS_CASES = [
("zlib lz4", "zlib", 6, 6),
("lz4 zstd", "lz4", 0, 1),
("bogus zstd zlib", "zstd", 3, 3),
(" ", "zstd", 3, 3),
]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("env,algo,rsync_level,fs_level", COMPRESS_CASES)
def test_compress_list_matches_rsync(self, env, algo, rsync_level, fs_level, shared_server,
monkeypatch):
monkeypatch.setenv("RSYNC_COMPRESS_LIST", env)
src = _make_corpus(_scratch(f"envc_src_{algo}"))
rdst = _scratch(f"envc_rsync_{algo}")
rs = _rsync(["-a", "-z", "--debug=NSTR1", src + "/", rdst + "/"])
assert rs.returncode == 0, rs.stderr
rm = _COMPRESS_RE.search(rs.stdout + rs.stderr)
assert rm, (rs.stdout, rs.stderr)
assert rm.group(1) == algo
assert int(rm.group(2)) == rsync_level
fdst = _scratch(f"envc_fs_{algo}")
result, _ = run_client(src, fdst, flags=["-a", "-z", "-v", "--debug=util"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fm = _COMPRESS_RE.search(result.stdout)
assert fm, result.stdout[:500]
assert fm.group(1) == algo
assert int(fm.group(2)) == fs_level
received = get_dest_received_dir(fdst, src)
assert _tree_bytes(received) == _tree_bytes(src)
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("env,algo", [("md5", "md5"), ("sha1", "sha1"), ("xxh3 md5", "xxh3")])
def test_checksum_list_matches_rsync(self, env, algo, shared_server, monkeypatch):
monkeypatch.setenv("RSYNC_CHECKSUM_LIST", env)
src = _make_corpus(_scratch(f"envcc_src_{algo}"))
rdst = _scratch(f"envcc_rsync_{algo}")
rs = _rsync(["-a", "--checksum", "--out-format=%C %n", src + "/", rdst + "/"])
assert rs.returncode == 0, rs.stderr
rs_digests = _digests(rs.stdout)
fdst = _scratch(f"envcc_fs_{algo}")
result, _ = run_client(src, fdst, flags=["-a", "--checksum", "--out-format=%C %n"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert _digests(result.stdout) == rs_digests
@requires_rsync
@pytest.mark.ci
def test_all_unknown_lists_fail_like_rsync(self, shared_server, monkeypatch):
src = _make_corpus(_scratch("envbad_src"))
monkeypatch.setenv("RSYNC_COMPRESS_LIST", "bogus")
rs = _rsync(["-a", "-z", src + "/", _scratch("envbad_rsync_c") + "/"])
assert rs.returncode == 4, rs.stderr
result, _ = run_client(src, _scratch("envbad_fs_c"), flags=["-a", "-z"],
port=shared_server.port)
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
monkeypatch.setenv("RSYNC_CHECKSUM_LIST", "bogus")
rs = _rsync(["-a", src + "/", _scratch("envbad_rsync_s") + "/"])
assert rs.returncode == 4, rs.stderr
result, _ = run_client(src, _scratch("envbad_fs_s"), flags=["-a"],
port=shared_server.port)
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
def _tree_bytes(root):
out = {}
for dirpath, _dirs, files in os.walk(root):
for name in files:
path = os.path.join(dirpath, name)
with open(path, "rb") as fh:
out[os.path.relpath(path, root)] = fh.read()
return out
def _digests(output):
out = {}
for line in output.splitlines():
parts = line.split()
if len(parts) == 2 and parts[0]:
out[parts[1]] = parts[0]
return out
tests/integration/test_codecs.py
+149
View File
@@ -9,6 +9,7 @@ directory when the shared test server is launched), so every scratch tree lives
under ``TEST_DATA_DIR`` rather than pytest's ``tmp_path``.
"""
import os
import random
import shutil
import subprocess
import sys
@@ -68,6 +69,84 @@ def _tree_bytes(root):
return out
_CC_DELTA_T0 = 1_600_000_000
_CC_DELTA_T1 = 1_600_000_100
_DELTA_STATS_KEYS = (
"Number of created files",
"Number of regular files transferred",
"Total transferred file size",
"Literal data",
"Matched data",
)
def _pin_tree(root, mtime):
for dirpath, dirnames, filenames in os.walk(root):
for name in dirnames + filenames:
path = os.path.join(dirpath, name)
if not os.path.islink(path):
os.utime(path, (mtime, mtime))
os.utime(root, (mtime, mtime))
def _make_delta_basis(src, size=512 * 1024):
"""Build a source and a matching pre-modification basis tree.
The source's ``big.bin`` is then modified in a few disjoint places and given
a newer mtime so both tools take the delta path. Returns the basis dir.
"""
clean_dir(src)
original = random.Random(20240101).randbytes(size)
with open(os.path.join(src, "big.bin"), "wb") as fh:
fh.write(original)
with open(os.path.join(src, "small.txt"), "wb") as fh:
fh.write(b"hello world\n")
_pin_tree(src, _CC_DELTA_T0)
basis = src.rstrip("/") + "_basis"
clean_dir(basis)
shutil.copy2(os.path.join(src, "big.bin"), os.path.join(basis, "big.bin"))
shutil.copy2(os.path.join(src, "small.txt"), os.path.join(basis, "small.txt"))
_pin_tree(basis, _CC_DELTA_T0)
modified = bytearray(original)
for off in (0, size // 3, 2 * size // 3, size - 64):
for i in range(32):
modified[off + i] ^= 0x5A
with open(os.path.join(src, "big.bin"), "wb") as fh:
fh.write(bytes(modified))
os.utime(os.path.join(src, "big.bin"), (_CC_DELTA_T1, _CC_DELTA_T1))
return basis
def _seed_from_basis(basis, target):
clean_dir(target)
for name in os.listdir(basis):
shutil.copy2(os.path.join(basis, name), os.path.join(target, name))
def _delta_stats(text):
found = {}
for line in text.splitlines():
for key in _DELTA_STATS_KEYS:
if line.startswith(key + ":"):
found[key] = line.split(":", 1)[1].strip()
return found
def _big_bin_outfmt(text):
"""The ``(c, C)`` pair from the ``big.bin`` out-format line (`%c|%C %n`)."""
for line in text.splitlines():
stripped = line.strip()
if "|" not in stripped or not stripped.endswith("big.bin"):
continue
c_field, rest = stripped.split("|", 1)
fields = rest.split()
return c_field.strip(), (fields[0] if fields else "")
return None, None
class TestCodecChoiceMatrix:
"""The CLI accept/reject set and exit codes must match rsync 3.4.1."""
@@ -191,6 +270,76 @@ class TestCodecTransferDifferential:
assert _tree_bytes(received) == _tree_bytes(rsync_dst)
class TestChecksumChoiceDeltaSurface:
"""--checksum-choice does not move the delta-transfer parity surface.
FastSync's delta BLOCK strong checksum is a fixed xxHash32, so the
negotiated algorithm only selects the whole-file comparison digest (and the
``%C`` transfer digest). A pre-seeded delta transfer must therefore land
byte-identical bytes and report the same counters for every choice, while
``%C`` -- the one token that tracks the choice -- stays byte-identical to
rsync. This pins the Track-3b reclassification in RSYNC_COMPAT.md.
"""
CHOICES = ["xxh64", "xxh128", "xxh3", "md5", "md4", "sha1",
"xxh64,sha1", "sha1,xxh64"]
@requires_rsync
@pytest.mark.ci
def test_delta_surface_invariant_to_checksum_choice(self, shared_server):
src = _scratch("ccdelta_src")
basis = _make_delta_basis(src)
source_bytes = _tree_bytes(src)
rsync_c, fastsync_c, digests = {}, {}, {}
rsync_stats, fastsync_stats = {}, {}
for choice in self.CHOICES:
tag = choice.replace(",", "_")
rsync_dst = _scratch(f"ccdelta_rs_{tag}")
fs_dst = _scratch(f"ccdelta_fs_{tag}")
fs_root = get_dest_received_dir(fs_dst, src)
_seed_from_basis(basis, rsync_dst)
_seed_from_basis(basis, fs_root)
# Pin the block size on both ends so the literal/matched split is
# comparable (rsync's adaptive default would otherwise differ from
# FastSync's 8192-byte default).
rsync_result = _rsync(["-a", "--no-whole-file", "-B8192", "--stats",
"--out-format=%c|%C %n", f"--cc={choice}",
src + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(
src, fs_dst,
flags=["-a", "--incremental", "--delta", "-B8192", "--stats",
"--out-format=%c|%C %n", f"--cc={choice}"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert _tree_bytes(rsync_dst) == source_bytes, choice
assert _tree_bytes(fs_root) == source_bytes, choice
assert _delta_stats(rsync_result.stdout) == _delta_stats(result.stdout), choice
rs_c, rs_C = _big_bin_outfmt(rsync_result.stdout)
fs_c, fs_C = _big_bin_outfmt(result.stdout)
assert rs_C == fs_C, f"{choice}: %C rsync={rs_C!r} fastsync={fs_C!r}"
rsync_c[choice] = rs_c
fastsync_c[choice] = fs_c
digests[choice] = fs_C
rsync_stats[choice] = _delta_stats(rsync_result.stdout)
fastsync_stats[choice] = _delta_stats(result.stdout)
# The choice is only observable in %C, and it is effective (the digests
# are not all the same algorithm's output).
assert len(set(digests.values())) > 1, digests
# The compared --stats counters are invariant across choices in each tool
# (and were asserted equal cross-tool inside the loop).
assert len({tuple(sorted(s.items())) for s in rsync_stats.values()}) == 1, rsync_stats
assert len({tuple(sorted(s.items())) for s in fastsync_stats.values()}) == 1, fastsync_stats
# The block-checksum token (%c) is invariant across choices in each tool.
assert len(set(rsync_c.values())) == 1, rsync_c
assert len(set(fastsync_c.values())) == 1, fastsync_c
class TestCodecNegotiationFallback:
"""FastSync's auto negotiation and deterministic fallback order."""
tests/integration/test_delete_delay_budget_parity.py
+22 -28
View File
@@ -1,19 +1,16 @@
"""Differential coverage for ``--delete-delay`` + ``--max-delete`` with a
refilled deferred directory.
FastSync snapshots a directory's extras at plan time (``defer_add``) and charges
``--max-delete`` then, and its deferred commit only removes the snapshot path, so
a directory refilled before the commit survives ``ENOTEMPTY``. rsync computes
the deferred deletions during the transfer, charges ``--max-delete`` on actual
removals, and recursively removes a queued directory -- so content created after
the plan inside an extra directory is removed too.
FastSync snapshots a directory's extras at plan time (``defer_add``) but charges
``--max-delete`` only when a path is actually removed, and its deferred commit
re-scans a queued directory and removes content created after the plan -- the
same rules as rsync. These tests run both tools on the same fixture and assert
both sides remove the late content (recursively) and bound the deletion with
``--max-delete`` identically.
These tests run both tools on the same fixture and pin the shared budget bound
(the later extra survives, both exit 25) plus the documented residual (the
refilled directory's late content survives under FastSync, not rsync). They are
not part of the fast PR gate because the rsync side needs a wide real-time
injection window (a throttled transfer), while the FastSync side uses the
existing byte-deterministic slicing proxy.
They are not part of the fast PR gate because the rsync side needs a wide
real-time injection window (a throttled transfer), while the FastSync side uses
the existing byte-deterministic slicing proxy.
The refilled directory sits at the transfer ROOT, whose delete plan is always
processed before any subdirectory's, so the budget is deterministically charged
@@ -101,7 +98,7 @@ def _rsync(args, timeout=120):
class TestDeleteDelayRefilledDirVsRsync:
"""The shared budget bound and the documented recursive-removal residual."""
"""Both tools charge --max-delete on actual removals and recurse."""
def _fastsync_refilled(self, tag, max_delete=None):
"""Run FastSync with the refill injected deterministically by the proxy
@@ -125,18 +122,18 @@ class TestDeleteDelayRefilledDirVsRsync:
return result, received, late
@requires_rsync
def test_max_delete_budget_bound_matches_and_residual_pinned(self):
# --- FastSync: budget charged at snapshot; late content preserved ---
def test_max_delete_budget_bound_matches(self):
# --- FastSync: the one actual removal is the late file; dirs survive ---
result, received, late = self._fastsync_refilled("budget_fs", max_delete=1)
assert result.returncode == 25, (result.stderr or result.stdout)[:300]
assert _deleted_count(result.stdout) == 0, result.stdout
assert os.path.exists(late), "FastSync removed the late content of a snapshotted dir"
assert _deleted_count(result.stdout) == 1, result.stdout
assert not os.path.exists(late), "FastSync kept the late content of a queued dir"
assert os.path.isdir(os.path.join(received, "xdir"))
assert os.path.isdir(os.path.join(received, "b", "ydir")), (
"FastSync did not charge the plan-time budget: b/ydir was removed"
"FastSync did not bound the deletion with --max-delete=1"
)
# --- rsync: budget charged on actual removals; dirs removed recursively ---
# --- rsync: same budget rule and recursive removal ---
source, rsync_dst = _seed_rsync("budget_rsync")
def inject():
@@ -151,26 +148,23 @@ class TestDeleteDelayRefilledDirVsRsync:
)
t.join()
assert rsync_result.returncode == 25, rsync_result.stderr
# rsync removes the late content (recursive deferred removal); FastSync
# keeps it and charges the snapshot directive instead.
assert _deleted_count(rsync_result.stdout) == _deleted_count(result.stdout)
assert not os.path.exists(os.path.join(rsync_dst, "xdir", "new.txt")), (
"rsync kept late content inside a queued directory"
)
# The shared observable: the later extra survives in both tools under
# --max-delete=1, and both report the capped run with exit 25.
assert os.path.isdir(os.path.join(rsync_dst, "b", "ydir")), (
"rsync did not bound the deletion with --max-delete=1"
)
assert os.path.isdir(os.path.join(received, "b", "ydir"))
@requires_rsync
def test_refilled_extra_dir_recursive_removal_residual(self):
"""Without --max-delete the residual is a plain tree difference: rsync
removes the refilled extra directory (and its late content), FastSync
leaves the snapshot path in place on ENOTEMPTY."""
def test_refilled_extra_dir_recursive_removal_matches(self):
"""Without --max-delete both tools remove the refilled extra directory
(and its late content)."""
result, received, late = self._fastsync_refilled("recur_fs")
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert os.path.exists(late), "FastSync removed the refilled directory's late content"
assert not os.path.exists(late), "FastSync kept the refilled directory's late content"
assert not os.path.isdir(os.path.join(received, "xdir"))
source, rsync_dst = _seed_rsync("recur_rsync")
tests/integration/test_delete_timing_parity.py
+61 -34
View File
@@ -217,7 +217,21 @@ class _SlicingProxy:
class TestDeleteTimingFinalStateParity:
"""On a successful transfer the per-directory timings match rsync's result."""
"""On a successful transfer the per-directory timings match rsync's result.
Plain ``--delete`` has no rsync-incompatible spelling: it defaults to
delete-during on both tools, so it is compared against rsync's own default.
``--delete-commit`` is FastSync-only and selects the late whole-tree commit,
which is rsync's ``--delete-after`` timing.
"""
# (fastsync flag, rsync flag)
PAIRS = [
("--delete", "--delete"),
("--delete-during", "--delete-during"),
("--delete-delay", "--delete-delay"),
("--delete-commit", "--delete-after"),
]
def _run_fastsync(self, tag, timing):
source, dest, received = _seed_pair(tag)
@@ -226,12 +240,12 @@ class TestDeleteTimingFinalStateParity:
result, _ = run_client(source, dest, flags=[timing], port=server.port)
return result, received
@pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
@pytest.mark.parametrize("fs_timing,rs_timing", PAIRS)
@requires_rsync
def test_success_final_state_matches_rsync(self, timing):
# Worker-safe names: xdist may run both parametrizations concurrently, so
# the timing is part of every fixture path.
label = timing.lstrip("-")
def test_success_final_state_matches_rsync(self, fs_timing, rs_timing):
# Worker-safe names: xdist may run the parametrizations concurrently, so
# the flags are part of every fixture path.
label = f"{fs_timing.lstrip('-')}_vs_{rs_timing.lstrip('-')}"
# Build the rsync fixture from the same seed so both sides start equal.
source, dest, received = _seed_pair(f"parity_rsync_{label}")
source2 = source
@@ -240,17 +254,18 @@ class TestDeleteTimingFinalStateParity:
# rsync mirrors src/ into dst/; seed the same extra.
_write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n")
rsync_result = _rsync(["-a", timing, source2 + "/", rsync_dst + "/"])
rsync_result = _rsync(["-a", rs_timing, source2 + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_tree = _tree(rsync_dst)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=[timing], port=server.port)
result, _ = run_client(source, dest, flags=[fs_timing], port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fastsync_tree = _tree(received)
assert fastsync_tree == rsync_tree, (
f"{timing}: fastsync tree {fastsync_tree} != rsync tree {rsync_tree}"
f"{fs_timing} vs rsync {rs_timing}: fastsync tree {fastsync_tree} != "
f"rsync tree {rsync_tree}"
)
@@ -292,7 +307,14 @@ class TestDeleteTimingTypeConflictParity:
class TestDeleteTimingFailure:
"""A mid-transfer failure distinguishes during from delay."""
"""A mid-transfer failure distinguishes the during timings from the late
commit timings.
Plain ``--delete`` must behave like ``--delete-during`` (the rsync default),
removing the extras of the directories already reached; ``--delete-commit``
must behave like ``--delete-after`` and remove nothing until the transfer
has fully succeeded.
"""
@pytest.mark.parametrize("mt", [False, True])
def test_during_removes_delay_preserves_on_failure(self, mt):
@@ -301,8 +323,12 @@ class TestDeleteTimingFailure:
assert os.path.exists(extra)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
for timing, expect_removed in (("--delete-during", True),
("--delete-delay", False)):
for timing, expect_removed in (
("--delete-during", True),
("--delete", True),
("--delete-delay", False),
("--delete-commit", False),
("--delete-after", False)):
# Re-seed the extra before each run.
_write(extra, b"stale extra\n")
proxy = _SlicingProxy(server.port, forward_limit=MID_TRANSFER_BYTES, throttle=PROXY_THROTTLE)
@@ -320,10 +346,10 @@ class TestDeleteTimingFailure:
class TestDeleteDelayDeletedCount:
"""The reported deleted count must reflect entries actually removed."""
def test_refilled_deferred_dir_is_not_counted(self):
def test_refilled_deferred_dir_is_recursively_removed_and_counted(self):
"""A directory snapshotted into a --delete-delay plan that is refilled
before the commit survives ENOTEMPTY and must NOT inflate "Number of
deleted files" (regression for delete_plan.c counting at snapshot)."""
before the commit is re-scanned and removed recursively (rsync parity):
the late file and the directory are both counted as deleted."""
source = os.path.join(TEST_DATA_DIR, "ddc_src")
dest = os.path.join(TEST_DATA_DIR, "ddc_dst")
clean_dir(source)
@@ -347,12 +373,13 @@ class TestDeleteDelayDeletedCount:
proxy.finish()
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert proxy.hook_called.is_set(), "hook never fired"
assert os.path.exists(os.path.join(extra_dir, "new.txt")), "late file vanished"
assert not os.path.exists(os.path.join(extra_dir, "new.txt")), "late file survived"
assert not os.path.isdir(extra_dir), "refilled extra dir survived"
deleted = None
for line in result.stdout.splitlines():
if line.startswith("Number of deleted files:"):
deleted = int(line.split(":", 1)[1].split()[0])
assert deleted == 0, (deleted, result.stdout)
assert deleted == 2, (deleted, result.stdout)
class TestDeleteDelayMaxDeleteParity:
@@ -448,12 +475,13 @@ class TestDeleteDelayVsAfterSnapshot:
class TestDeleteAfterThreadsKeepSet:
"""Regression: -m/--threads with the default delete-after timing (plain
--delete) must still transmit the keep-set manifest and remove destination
extras. PipelineContextSender.delete_suppressed was left uninitialized, so a
garbage true silently skipped the manifest under --threads."""
"""Regression: -j/--threads must still transmit the delete keep-set in every
timing. PipelineContextSender.delete_suppressed was left uninitialized, so a
garbage true silently skipped the late keep-set manifest under --threads.
Plain --delete now uses the per-directory plans, while --delete-commit /
--delete-after keep exercising the late whole-tree manifest."""
@pytest.mark.parametrize("delete_flag", ["--delete", "--delete-after"])
@pytest.mark.parametrize("delete_flag", ["--delete", "--delete-commit", "--delete-after"])
def test_threads_delete_after_sends_keep_set(self, delete_flag):
source, dest, received = _seed_pair("mtkeep")
extra = os.path.join(received, "d", "old_extra")
@@ -464,14 +492,13 @@ class TestDeleteAfterThreadsKeepSet:
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert not os.path.exists(extra), (
f"{delete_flag} --threads did not remove an extra: keep-set manifest was suppressed"
f"{delete_flag} --threads did not remove an extra: delete keep-set was suppressed"
)
class TestDeleteDelayMaxDeleteRefilledDir:
"""--delete-delay charges the --max-delete budget at plan/snapshot time, so a
refilled snapshotted directory that survives ENOTEMPTY still spends its slot
and a later extra is skipped, while the reported count stays at actual
removals.
"""--delete-delay charges the --max-delete budget on ACTUAL removals: the
refilled directory's late content is removed first (consuming the one slot),
so the directory itself and a later extra are skipped, matching rsync.
The refilled directory is at the destination ROOT (its plan is always sent
first) and the skipped extra is under a separate source directory, so the
@@ -479,7 +506,7 @@ class TestDeleteDelayMaxDeleteRefilledDir:
order. The refill is injected through the byte-barrier proxy so it is
causally after the plan frame."""
def test_budget_charged_at_plan_time(self):
def test_budget_charged_on_actual_removal(self):
source = os.path.join(TEST_DATA_DIR, "ddmb_src")
dest = os.path.join(TEST_DATA_DIR, "ddmb_dst")
clean_dir(source)
@@ -504,9 +531,9 @@ class TestDeleteDelayMaxDeleteRefilledDir:
proxy.finish()
assert result.returncode == 25, (result.stderr or result.stdout)[:400]
assert proxy.hook_called.is_set(), "hook never fired"
# The refilled directory still consumes the plan-time budget, so the
# later extra is skipped...
assert os.path.exists(os.path.join(refilled_dir, "new.txt")), "late file vanished"
assert os.path.isdir(later_dir), "later extra was not skipped by the plan-time budget"
# ...while the reported count reflects only actual removals (none here).
assert _deleted_count(result.stdout) == 0, result.stdout
# The late content consumes the single budget slot; the refilled
# directory itself and the later extra are skipped.
assert not os.path.exists(os.path.join(refilled_dir, "new.txt")), "late file survived"
assert os.path.isdir(later_dir), "later extra was not skipped by the budget"
# The one actual removal is reported.
assert _deleted_count(result.stdout) == 1, result.stdout
tests/integration/test_differential_parity.py
+202 -4
View File
@@ -28,6 +28,7 @@ from common import ( # noqa: E402
ServerManager,
TEST_DATA_DIR,
clean_dir,
get_dest_received_dir,
)
from parity_caveats import ASPECTS, caveat_for # noqa: E402
import parity_harness as H # noqa: E402
@@ -116,12 +117,32 @@ def seed_delete_excluded(_src, rroot, froot):
_mk(os.path.join(root, "keep.txt"), b"keep\n", _OLD_MTIME)
def seed_filter_protect(_src, rroot, froot):
"""Destination-only entries, including nested ones, for the receiver-side
`protect` rule: the `.log` extras must survive --delete, the rest go."""
for root in (rroot, froot):
_mk(os.path.join(root, "extra.log"), b"dest-only log\n", _OLD_MTIME)
_mk(os.path.join(root, "other.txt"), b"dest-only other\n", _OLD_MTIME)
_mk(os.path.join(root, "sub", "extra2.log"), b"nested dest-only log\n", _OLD_MTIME)
_mk(os.path.join(root, "sub", "other2.txt"), b"nested dest-only other\n", _OLD_MTIME)
def seed_max_delete(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
_mk(os.path.join(root, "extra2.txt"), b"e2\n", _OLD_MTIME)
def fuzzy_basis_seed(_src, rroot, froot):
"""Seed a same-suffix sibling whose name is one edit from the source and
whose content matches it, with a DIFFERENT mtime so rsync's exact
size+mtime pass cannot fire: both tools must select it via the
name-distance pass. Where the two tools' basis choices coincide the
block-level results are identical when the block size is pinned."""
for root in (rroot, froot):
_mk(os.path.join(root, "report_v1.txt"), H.FUZZY_PAYLOAD, _OLD_MTIME)
def max_delete_count_check(_src, rroot, froot, _rs, _fs):
"""The exact survivor set is order-dependent; the count must still match."""
r = H.snapshot(rroot)
@@ -173,6 +194,12 @@ _CASES = [
stdout=H.STDOUT_OUTFMT, ref="--out-format %n %l"),
H.Case("out_format_i_n", "basic", ["-a", "--out-format=%i %n"],
stdout=H.STDOUT_OUTFMT, ref="--out-format %i %n"),
H.Case("progress", "multidir", ["-a", "--progress"], stdout=H.STDOUT_PROGRESS,
ci=True, ref="--progress multi-directory file list"),
H.Case("progress_threads", "multidir", ["-a", "--progress"],
fastsync_flags=["-a", "--progress", "--threads"],
stdout=H.STDOUT_PROGRESS, ci=True,
ref="--progress multi-directory file list (--threads)"),
# --- transfer modifications -------------------------------------------
H.Case("update", "basic", ["-a", "--update"], seed=seed_update,
@@ -191,6 +218,18 @@ _CASES = [
H.Case("chmod", "basic", ["-a", "--chmod=Fu+rwx"], compare_modes=True,
ci=True, ref="--chmod"),
# --- delta / similar-file basis (--fuzzy) -----------------------------
# Basis choices coincide here (same-suffix sibling, name distance one edit,
# content identical); with the block size pinned both tools report the same
# Matched/Literal/transferred counters. The residual (FastSync's narrower
# delta size window) is covered by TestFuzzy in test_parity_quickwins.py.
H.Case("fuzzy_basis", "fuzzy",
["-a", "--no-whole-file", "--fuzzy", "--stats", "-B8192"],
fastsync_flags=["-a", "--incremental", "--delta", "--fuzzy",
"--stats", "--delta-block=8192"],
seed=fuzzy_basis_seed, stdout=H.STDOUT_STATS, ci=True,
ref="-y/--fuzzy similar-file basis"),
# --- deletion ---------------------------------------------------------
H.Case("delete", "basic", ["-a", "--delete"], seed=seed_extras,
server_args=DELETE, ci=True, ref="--delete"),
@@ -202,13 +241,36 @@ _CASES = [
server_args=DELETE, ref="--delete-delay"),
H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras,
server_args=DELETE, ref="--delete-after"),
H.Case("delete_commit", "basic", ["-a", "--delete-after"], seed=seed_extras,
fastsync_flags=["-a", "--delete-commit"], server_args=DELETE,
ref="FastSync-only --delete-commit == rsync --delete-after"),
H.Case("delete_excluded", "filters",
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
H.Case("exclude_protect_dest_only", "filters",
["-a", "--delete", "--exclude=*.log"],
seed=seed_delete_excluded, server_args=DELETE, ci=True,
ref="--delete protects a destination-only excluded entry like rsync"),
H.Case("max_delete", "basic", ["-a", "--delete", "--max-delete=1"],
seed=seed_max_delete, server_args=DELETE,
extra_check=max_delete_count_check, compare_tree=False,
ref="--max-delete"),
H.Case("filter_protect", "filters",
["-a", "--delete", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect receiver-side delete protection (default during)"),
H.Case("filter_protect_during", "filters",
["-a", "--delete-during", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect under --delete-during"),
H.Case("filter_protect_delay", "filters",
["-a", "--delete-delay", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect under --delete-delay"),
H.Case("filter_protect_after", "filters",
["-a", "--delete-after", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect under the whole-tree --delete-after commit"),
# --- relative / dirs --------------------------------------------------
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
@@ -318,7 +380,11 @@ def _result_aspects(result):
_STANDALONE_REFS = {
"incremental_modified": "-i/--itemize-changes + incremental second run",
"compare_dest": "--compare-dest",
"copy_dest": "--copy-dest",
"link_dest": "--link-dest",
"link_dest_stats": "--link-dest + --stats",
"verify_basis": "--verify-basis (FastSync-only)",
"verify_basis_default": "--verify-basis (default quick-check vs rsync)",
"added_and_deleted": "--delete across two runs",
"added_and_deleted_seed": "--delete across two runs",
"one_file_system": "-x/--one-file-system",
@@ -379,14 +445,17 @@ def test_compare_dest_skips_basis(parity_server_factory):
fdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"basis-content\n")
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
# rsync resolves --compare-dest relative to the destination dir; FastSync
# resolves it under the receive root and appends the mirrored source path.
# Both rely on rsync's size+mtime quick-check, so the basis mtime is pinned
# to the source's to keep the match deterministic across a second boundary.
def seed(_src, rroot, froot):
_mk(os.path.join(rroot, "basis", "f.txt"), b"basis-content\n")
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"basis-content\n")
_mk(os.path.join(rroot, "basis", "f.txt"), b"basis-content\n", _OLD_MTIME)
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"basis-content\n", _OLD_MTIME)
def extra(_src, rroot, froot, _rs, _fs):
out = []
@@ -414,12 +483,13 @@ def test_link_dest_hardlinks_basis(parity_server_factory):
fdst = os.path.join(TEST_DATA_DIR, "parity_linkd_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n")
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n")
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n", _OLD_MTIME)
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n", _OLD_MTIME)
def extra(_src, rroot, froot, _rs, _fs):
r_basis = os.stat(os.path.join(rroot, "basis", "f.txt")).st_ino
@@ -442,6 +512,134 @@ def test_link_dest_hardlinks_basis(parity_server_factory):
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_link_dest_stats_matches_rsync(parity_server_factory):
"""A basis hit must not be counted as created or literal data: rsync reports
zero for both, so FastSync's receiver tallies must too (regression for the
basis materialization over-report)."""
case_id = "link_dest_stats"
src = os.path.join(TEST_DATA_DIR, "parity_linkds_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_linkds_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_linkds_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n", _OLD_MTIME)
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n", _OLD_MTIME)
result = H.run_differential(
src, rdst, fdst,
["-a", "--link-dest=basis", "--stats"],
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental", "--stats"],
server, seed=seed, ignore_paths=("basis",), stdout=H.STDOUT_STATS)
_run_and_check(case_id, result, ref="--link-dest + --stats")
@requires_rsync
@parity
def test_copy_dest_copies_basis(parity_server_factory):
"""--copy-dest: a basis match is materialized as an independent copy with the
source's attributes, matching rsync (copy then fix attributes)."""
case_id = "copy_dest"
src = os.path.join(TEST_DATA_DIR, "parity_copyd_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_copyd_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_copyd_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"copy-basis-content\n")
_pin(os.path.join(src, "f.txt"), 1_600_000_000)
os.chmod(os.path.join(src, "f.txt"), 0o755)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
# Basis content matches the source; give the basis a different mode so a
# wrong "keep basis attributes" implementation is visible.
_mk(os.path.join(rroot, "basis", "f.txt"), b"copy-basis-content\n",
1_600_000_000)
os.chmod(os.path.join(rroot, "basis", "f.txt"), 0o644)
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"copy-basis-content\n",
1_600_000_000)
os.chmod(os.path.join(fdst, "basis", rel, "f.txt"), 0o644)
def extra(_src, rroot, froot, _rs, _fs):
out = []
bases = {"rsync": os.path.join(rroot, "basis", "f.txt"),
"fastsync": os.path.join(fdst, "basis", rel, "f.txt")}
for label, root in (("rsync", rroot), ("fastsync", froot)):
target = os.path.join(root, "f.txt")
if not os.path.exists(target):
out.append(f"{label}: f.txt missing")
continue
if os.stat(target).st_ino == os.stat(bases[label]).st_ino:
out.append(f"{label}: f.txt is hard-linked, not copied")
if (os.stat(target).st_mode & 0o777) != 0o755:
out.append(f"{label}: f.txt mode "
f"{oct(os.stat(target).st_mode & 0o777)} != 0o755")
return out
result = H.run_differential(
src, rdst, fdst,
["-a", "--copy-dest=basis"],
["-a", f"--copy-dest={os.path.join(fdst, 'basis')}", "--incremental"],
server, seed=seed, ignore_paths=("basis",), extra_check=extra,
compare_modes=True)
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_verify_basis_restores_strict_content(parity_server_factory):
"""Default matches rsync's metadata quick-check; FastSync-only
`--verify-basis` restores strict content equality and transfers the source
when a same-size/different-content basis would otherwise be trusted."""
case_id = "verify_basis"
src = os.path.join(TEST_DATA_DIR, "parity_vbasis_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_vbasis_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_vbasis_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"AAAA\n")
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
# Same size and mtime as the source, different bytes: a metadata
# quick-check trusts it; --verify-basis must not.
for root, basis_rel in ((rroot, os.path.join("basis", "f.txt")),
(fdst, os.path.join("basis", rel, "f.txt"))):
_mk(os.path.join(root, basis_rel), b"BBBB\n", _OLD_MTIME)
# Default: both tools trust the basis (rsync's quick check), so the
# destination carries the basis bytes and the trees match.
result = H.run_differential(
src, rdst, fdst,
["-a", "--link-dest=basis"],
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental"],
server, seed=seed, ignore_paths=("basis",))
_run_and_check(case_id + "_default", result)
# --verify-basis (FastSync only): the digest mismatch rejects the basis and
# the source is transferred, so the destination is the source bytes. rsync
# has no such flag; assert the FastSync outcome directly against the source.
fdst2 = os.path.join(TEST_DATA_DIR, "parity_vbasis_fdst2")
clean_dir(fdst2)
for root, basis_rel in ((fdst2, os.path.join("basis", rel, "f.txt")),):
_mk(os.path.join(root, basis_rel), b"BBBB\n", _OLD_MTIME)
result, _ = H.run_fastsync(src, fdst2,
["-a", f"--link-dest={os.path.join(fdst2, 'basis')}",
"--incremental", "--verify-basis"], server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
target = os.path.join(get_dest_received_dir(fdst2, src), "f.txt")
with open(target, "rb") as fh:
assert fh.read() == b"AAAA\n", \
"--verify-basis must reject the same-size/different-content basis"
@requires_rsync
@parity
def test_added_and_deleted_between_runs(parity_server_factory):
tests/integration/test_fault_injection.py
+1 -1
View File
@@ -36,7 +36,7 @@ from common import ( # noqa: E402
verify_transfer,
)
PROTOCOL_VERSION = b"2.27.0"
PROTOCOL_VERSION = b"2.28.0"
STATUS_MANIFEST = 5
STATUS_OK = 0
tests/integration/test_features.py
+140 -76
View File
@@ -584,53 +584,55 @@ class TestRemoteDryRun:
assert _snapshot_tree(received) == before, f"{flags} mutated the destination"
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
def test_dry_run_delete_lines_over_report_residual(self):
"""Documented residual (RSYNC_COMPAT.md `-n/--dry-run` row): FastSync's
dry-run would-delete report includes the file that is merely being
updated (derived from the receiver's STATUS_STATS extras) and, unlike
rsync, also reports an excluded-but-protected extra. rsync `-n -i
--delete` lists only genuine extras. Pins the residual that keeps the
row Divergent."""
def test_dry_run_delete_lines_match_rsync(self):
"""`-n --delete` lists exactly the destination extras rsync would remove.
Covers the three cases that a real run protects: the file being updated
(in the keep set), a filter-excluded source entry (protected prefix), and
a --max-size-pruned source entry (always-protected prefix). Track 4a
adds a fourth: a destination-only entry matching the exclude rule is
re-derived on the receiver and also protected, so only the genuine
destination-only `extra.txt` appears.
"""
source = os.path.join(TEST_DATA_DIR, "dryrep_src")
rdst = os.path.join(TEST_DATA_DIR, "dryrep_rdst")
fdst = os.path.join(TEST_DATA_DIR, "dryrep_fdst")
clean_dir(source)
clean_dir(rdst)
clean_dir(fdst)
with open(os.path.join(source, "a.txt"), "wb") as fh:
fh.write(b"new content\n")
for name, data in (("a.txt", b"new content\n"), ("keep.log", b"log\n"),
("big.bin", b"B" * 2000)):
with open(os.path.join(source, name), "wb") as fh:
fh.write(data)
os.utime(os.path.join(source, "a.txt"), (1_700_000_000, 1_700_000_000))
for root in (rdst, fdst):
with open(os.path.join(root, "a.txt"), "wb") as fh:
fh.write(b"old\n")
for name, data in (("extra.log", b"log\n"), ("extra.txt", b"extra\n")):
received = get_dest_received_dir(fdst, source)
os.makedirs(received, exist_ok=True)
for root in (rdst, received):
for name, data in (("a.txt", b"old\n"), ("keep.log", b"log\n"),
("big.bin", b"B" * 2000), ("extra.txt", b"extra\n"),
("stray.log", b"dest only\n")):
with open(os.path.join(root, name), "wb") as fh:
fh.write(data)
for p in (os.path.join(root, "a.txt"), os.path.join(root, "extra.log"),
os.path.join(root, "extra.txt")):
os.utime(p, (1_500_000_000, 1_500_000_000))
os.utime(os.path.join(root, name), (1_500_000_000, 1_500_000_000))
flags = ["-a", "-n", "-i", "--delete", "--exclude=*.log", "--max-size=1000"]
r = subprocess.run(["rsync", "-an", "-i", "--delete", "--exclude=*.log",
source + "/", rdst + "/"],
"--max-size=1000", source + "/", rdst + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r.returncode == 0, r.stderr
rsync_del = {l.split(None, 1)[1] for l in r.stdout.splitlines()
if l.startswith("*deleting")}
assert rsync_del == {"extra.txt"}, f"unexpected rsync deleting set: {rsync_del}"
rsync_del = sorted(l for l in r.stdout.splitlines() if l.startswith("*deleting"))
assert rsync_del == ["*deleting extra.txt"], f"unexpected rsync set: {rsync_del}"
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, fdst,
flags=["-a", "-n", "-i", "--delete", "--exclude=*.log"],
port=server.port)
result, _ = run_client(source, fdst, flags=flags, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fs_del = {l.split(None, 1)[1] for l in (result.stdout or "").splitlines()
if l.startswith("*deleting")}
# Documented over-report: the transferred/updated file and the excluded
# extra appear in FastSync's would-delete set.
assert "a.txt" in fs_del, "residual changed: FastSync no longer over-reports the update"
assert "extra.log" in fs_del, "residual changed: FastSync no longer reports excluded extra"
fs_del = sorted(l for l in (result.stdout or "").splitlines()
if l.startswith("*deleting"))
assert fs_del == rsync_del, f"rsync={rsync_del}\nfastsync={fs_del}"
assert os.path.exists(os.path.join(received, "stray.log")), \
"destination-only exclude match must be protected in the dry-run report"
@pytest.mark.ci
def test_remote_dry_run_quiet_is_silent(self, shared_server):
@@ -3980,15 +3982,16 @@ class TestDeleteTiming:
assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n", \
f"{flag}: nested file was not written after the early deletion"
@pytest.mark.parametrize("flag", ["--delete", "--delete-after"])
@pytest.mark.parametrize("flag", ["--delete-commit", "--delete-after"])
@pytest.mark.parametrize("mt", [False, True])
def test_late_flags_commit_only_after_success(self, flag, mt):
"""Plain --delete/--delete-after defer deletion until the whole transfer
"""--delete-commit/--delete-after defer deletion until the whole transfer
succeeds: a mid-transfer write failure must leave every extra in place
(commit-style safety). The -m receiver must also keep the extras: the
deferred keep-set is committed by the server only after the disk-writer
thread has finished, and a failing writer means the manifest is freed,
never applied."""
(commit-style safety). Plain --delete no longer defers (it defaults to
delete-during), so only the explicitly late timings are exercised here.
The --threads receiver must also keep the extras: the deferred keep-set is
committed by the server only after the disk-writer thread has finished,
and a failing writer means the manifest is freed, never applied."""
source = self._seed("late")
dest = os.path.join(TEST_DATA_DIR, "deltiming_late_dst")
clean_dir(dest)
@@ -4693,11 +4696,11 @@ class TestDeletePolicy:
finally:
os.chmod(source, 0o755)
def test_delete_excluded_protection_is_sender_derived(self):
def test_delete_protection_reapplied_on_receiver(self):
"""Plain --delete protects destination mirrors of files the SOURCE scan
excluded, but a destination-only file that merely matches an exclude
rule is still an extra and is removed (protection never re-applies rules
to the destination)."""
excluded, and (track 4a) also protects a destination-only file matching
an exclude rule because the compiled rule set is re-applied on the
receiver, matching rsync."""
source = os.path.join(TEST_DATA_DIR, "senderderived_src")
clean_dir(source)
self._write(os.path.join(source, "keep.txt"), b"kept\n")
@@ -4717,8 +4720,8 @@ class TestDeletePolicy:
f"delete sync failed: {(result.stderr or result.stdout)[:300]}"
assert os.path.exists(os.path.join(received, "secret.log")), \
"source-excluded mirror was deleted under plain --delete"
assert not os.path.exists(os.path.join(received, "stray.log")), \
"destination-only file matching the exclude rule was left (should be deleted)"
assert os.path.exists(os.path.join(received, "stray.log")), \
"destination-only file matching the exclude rule must be protected like rsync"
def _pin_mtime(path, ts):
@@ -4738,11 +4741,12 @@ class TestBasisDestDirs:
STAGING = ".fastsync-stage"
TS = 1577836800 # 2020-01-01 00:00:00 UTC, used to pin matching mtimes
# fixture files: source and basis share the mtime pin, so a basis "match"
# is decided purely by content (xxHash). unchanged.txt is byte-identical;
# changed.txt is byte-DIFFERENT but has the SAME SIZE as the source (and
# the same pinned mtime), which is what forces the content-hash gate;
# added.txt does not exist in the basis at all.
# fixture files: source and basis share the mtime pin, so the DEFAULT
# (rsync-parity) quick-check is a size+mtime match and trusts the basis even
# when the body differs. unchanged.txt is byte-identical; changed.txt is
# byte-DIFFERENT but has the SAME SIZE as the source (and the same pinned
# mtime), which is what the FastSync-only --verify-basis content gate
# rejects; added.txt does not exist in the basis at all.
UNCHANGED = "unchanged.txt"
CHANGED = "changed.txt"
ADDED = "added.txt"
@@ -4782,18 +4786,19 @@ class TestBasisDestDirs:
}
def _basis_tree(self, prefix):
# unchanged.txt is identical to the source; changed.txt has the SAME
# byte size and pinned mtime but a different body (equal size forces
# the xxHash gate); added.txt is missing from the basis.
# unchanged.txt is identical to the source; changed.txt has a DIFFERENT
# size (and body) so the size leg of the quick-check fails and it is
# transferred normally; added.txt is missing from the basis.
return {
self.UNCHANGED: b"stable content v1\n",
self.CHANGED: b"CHANGED CONTENT NOW\n",
self.CHANGED: b"CHANGED CONTENT NOW AND LONGER\n",
}
def test_same_size_different_content_is_not_a_basis_match(self, shared_server):
# Core safety property: equal size + pinned mtime but different content
# must NEVER be hard-linked or copied from the basis -- the xxHash gate
# rejects it and the sender's data is transferred instead.
def test_same_size_different_content_default_trusts_quick_check(self, shared_server):
# Default rsync-parity behavior: equal size + pinned mtime is a basis
# match, so the basis body is materialized/linked without reading it.
# This mirrors rsync 3.4.1's quick check (differential-tested in
# test_differential_parity.py::test_verify_basis_restores_strict_content).
for flag, basis_dir in (("--link-dest", "szlb"), ("--copy-dest", "szcp"),
("--compare-dest", "szcmp")):
source = self._make_source("basis_same_size_src",
@@ -4805,7 +4810,36 @@ class TestBasisDestDirs:
result, _ = run_client(source, dest, flags=[f"{flag}={basis_dir}"],
port=shared_server.port)
assert result.returncode == 0, \
f"{flag} same-size mismatch failed: {result.stderr[:300]}"
f"{flag} same-size quick-check failed: {result.stderr[:300]}"
received = get_dest_received_dir(dest, source)
dest_file = os.path.join(received, self.UNCHANGED)
if flag == "--compare-dest":
assert not os.path.exists(dest_file), \
f"{flag}: compare-dest must leave a matching file sparse"
else:
assert _read_file(dest_file) == b"SAME LENGTH BODY!", \
f"{flag}: default quick-check did not trust the basis body"
if flag == "--link-dest":
assert os.stat(dest_file).st_ino == os.stat(basis_file).st_ino, \
f"{flag}: basis was not hard-linked"
def test_verify_basis_rejects_same_size_different_content(self, shared_server):
# FastSync-only --verify-basis: the whole-file digest gate rejects the
# same-size/different-content basis, so the source data is transferred
# instead of the wrong basis bytes.
for flag, basis_dir in (("--link-dest", "vszlb"), ("--copy-dest", "vszcp"),
("--compare-dest", "vszcmp")):
source = self._make_source("basis_verify_src",
{self.UNCHANGED: b"same length body\n"})
dest = os.path.join(TEST_DATA_DIR, f"basis_verify_dst_{basis_dir}")
clean_dir(dest)
basis_file = self._seed_basis_file(dest, source, basis_dir, self.UNCHANGED,
b"SAME LENGTH BODY!")
result, _ = run_client(source, dest,
flags=[f"{flag}={basis_dir}", "--verify-basis"],
port=shared_server.port)
assert result.returncode == 0, \
f"{flag} --verify-basis failed: {result.stderr[:300]}"
received = get_dest_received_dir(dest, source)
dest_file = os.path.join(received, self.UNCHANGED)
assert _read_file(dest_file) == b"same length body\n", \
@@ -4837,11 +4871,13 @@ class TestBasisDestDirs:
self._source_tree("c")[self.ADDED], "added file not transferred"
@pytest.mark.ci
def test_dry_run_compare_dest_does_not_read_basis(self, shared_server):
# A dry-run --compare-dest must never read/hash the basis file: doing so
# is a 1-bit content oracle against the client-supplied digest. Even a
# byte-identical basis with a matching size+mtime is therefore reported
# as would-transfer, and nothing is created.
def test_dry_run_compare_dest_quick_check_does_not_read_basis(self, shared_server):
# A dry-run --compare-dest must never read/hash the basis file. Under
# the default metadata quick-check a matching basis is reported as a
# skip (matching rsync) without reading it; nothing is created. Under
# --verify-basis, which would require hashing, the dry-run cannot
# confirm the hit (that would be a 1-bit content oracle) and reports
# would-transfer instead.
source = self._make_source("basis_dry_src", {self.UNCHANGED: b"stable content v1\n"})
dest = os.path.join(TEST_DATA_DIR, "basis_dry_dst")
clean_dir(dest)
@@ -4852,11 +4888,30 @@ class TestBasisDestDirs:
port=shared_server.port)
assert result.returncode == 0, \
f"dry-run compare-dest failed: {result.stderr[:300]}"
assert self.UNCHANGED in result.stdout, (
"dry-run compare-dest silently skipped: receiver read the basis content"
assert self.UNCHANGED not in result.stdout, (
"dry-run compare-dest did not honor the metadata quick-check "
"(reported would-transfer for a matching basis)"
)
assert _snapshot_tree(dest) == before, "dry-run compare-dest mutated the destination"
# --verify-basis: the hit needs the basis content, which a dry-run must
# not read, so the file is reported as would-transfer.
dest2 = os.path.join(TEST_DATA_DIR, "basis_dry_verify_dst")
clean_dir(dest2)
self._seed_basis(dest2, source, "drybasis", {self.UNCHANGED: b"stable content v1\n"})
before2 = _snapshot_tree(dest2)
result, _ = run_client(source, dest2,
flags=["--compare-dest=drybasis", "--dry-run",
"--verify-basis"],
port=shared_server.port)
assert result.returncode == 0, \
f"dry-run --verify-basis compare-dest failed: {result.stderr[:300]}"
assert self.UNCHANGED in result.stdout, (
"dry-run --verify-basis must not read the basis to confirm a hit"
)
assert _snapshot_tree(dest2) == before2, \
"dry-run --verify-basis compare-dest mutated the destination"
def test_compare_dest_content_mismatch_forces_transfer(self, shared_server):
# The basis holds a file with a DIFFERENT body: even though it shares
# the mtime pin, the xxHash check fails and the data must be sent.
@@ -5095,27 +5150,36 @@ class TestBasisDestDirs:
assert os.stat(dest_file).st_ino != os.stat(basis_file).st_ino, \
"--ignore-times must not hard-link to a basis file"
def test_basis_refuses_file_above_whole_file_limit(self, shared_server):
# Every whole-file payload path in FastSync (basis dirs included) is
# bounded by MAX_RECEIVE_WHOLE_FILE_SIZE. rsync supports basis dirs for
# arbitrary sizes; FastSync refuses such a run up front with a clear
# diagnostic instead of letting the receiver abort the whole transfer
# mid-stream with no client-side explanation.
def test_basis_handles_file_above_whole_file_limit(self, shared_server):
# Track 5a: a basis hit streams the copy (and the --verify-basis digest
# streams the basis), so a source larger than the whole-file payload
# bound is supported for basis dirs exactly like rsync. A basis MISS
# still falls back to the normal transfer, which keeps its own bound.
source = self._make_source("basis_oversize_src", {"small.txt": b"ok\n"})
big = os.path.join(source, "huge.bin")
with open(big, "wb") as fh:
os.ftruncate(fh.fileno(), 256 * 1024 * 1024 + 4096)
dest = os.path.join(TEST_DATA_DIR, "basis_oversize_dst")
clean_dir(dest)
result, _ = run_client(source, dest, flags=["--link-dest=nope"],
port=shared_server.port)
assert result.returncode != 0, \
"basis run with an over-limit file unexpectedly succeeded"
assert "larger than" in result.stderr, \
f"no clear over-limit diagnostic: {result.stderr[:300]}"
received = get_dest_received_dir(dest, source)
assert not os.path.exists(received), \
"over-limit basis run transferred files before failing"
rel = os.path.relpath(received, dest)
basis_big = os.path.join(dest, "ob", rel, "huge.bin")
os.makedirs(os.path.dirname(basis_big), exist_ok=True)
shutil.copyfile(big, basis_big)
os.utime(basis_big, (self.TS, self.TS))
os.utime(big, (self.TS, self.TS))
result, _ = run_client(source, dest,
flags=["--link-dest=ob", "--incremental"],
port=shared_server.port)
assert result.returncode == 0, \
f"over-limit basis run failed: {result.stderr[:300]}"
dest_big = os.path.join(received, "huge.bin")
assert os.path.exists(dest_big), "over-limit basis hit was not materialized"
assert os.path.getsize(dest_big) == 256 * 1024 * 1024 + 4096
assert os.stat(dest_big).st_ino == os.stat(basis_big).st_ino, \
"over-limit --link-dest did not hard-link to the basis"
assert _read_file(os.path.join(received, "small.txt")) == b"ok\n"
def _random_payloads(size=2 * 1024 * 1024, changed=64 * 1024, seed=1234):
tests/integration/test_option_parity.py
+89 -10
View File
@@ -164,6 +164,64 @@ class TestInfoParity:
f"rsync={entries(rsync_result.stdout)} fastsync={entries(result.stdout)}"
)
@requires_rsync
@pytest.mark.ci
def test_info_name_root_line_matches_rsync(self, shared_server):
"""A fresh destination: rsync prints `created directory`, then the
transfer-root `./` name line before the entries; FastSync must emit the
same `./` line."""
source = os.path.join(TEST_DATA_DIR, "inf_root_src")
dest = os.path.join(TEST_DATA_DIR, "inf_root_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_root_rdst")
clean_dir(source)
_write(os.path.join(source, "f.bin"), b"payload\n")
clean_dir(dest)
shutil.rmtree(rdst, ignore_errors=True)
rsync_result = _rsync(["-a", "--info=name", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--info=name"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
def names(text):
return [l for l in text.splitlines()
if l and not l.startswith("created directory")
and not (l.endswith("/") and l != "./")]
assert names(rsync_result.stdout) == ["./", "f.bin"], names(rsync_result.stdout)
assert names(result.stdout) == ["./", "f.bin"], names(result.stdout)
@requires_rsync
@pytest.mark.ci
def test_info_name2_uptodate_matches_rsync(self, shared_server):
"""--info=name2 prints `NAME is uptodate` for entries the receiver
already has, matching rsync byte-for-byte."""
source = os.path.join(TEST_DATA_DIR, "inf_up_src")
dest = os.path.join(TEST_DATA_DIR, "inf_up_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_up_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub"))
_write(os.path.join(source, "a.txt"), b"a\n")
_write(os.path.join(source, "sub", "b.txt"), b"b\n")
clean_dir(rdst)
assert _rsync(["-a", source + "/", rdst + "/"]).returncode == 0
rsync_result = _rsync(["-a", "--info=name2", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
clean_dir(dest)
seed, _ = run_client(source, dest, flags=["-a", "--incremental"],
port=shared_server.port)
assert seed.returncode == 0, (seed.stderr or seed.stdout)[:200]
result, _ = run_client(source, dest, flags=["-a", "--incremental", "--info=name2"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.endswith("is uptodate"))
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.endswith("is uptodate"))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
assert fast_lines == ["a.txt is uptodate", "sub/b.txt is uptodate"], fast_lines
@requires_rsync
@pytest.mark.ci
def test_info_nonreg_matches_rsync(self, shared_server):
@@ -420,14 +478,14 @@ class TestRemoteOptionDaemon:
assert "remote-option" in (result.stderr + result.stdout)
class TestFilterProtectDivergence:
"""Documented residual: a protect rule that matches only a destination-only
entry is not re-derived on the receiver (FastSync derives delete protection
from the source scan), so rsync protects the extra but FastSync removes it."""
class TestFilterProtect:
"""Receiver-derived delete protection: a `protect`/`P` rule is compiled by
the sender and sent on the config frame, so the receiver shields a
destination-only entry that never appeared on the sender, matching rsync."""
@requires_rsync
@pytest.mark.ci
def test_protect_dest_only_divergence(self, shared_server):
def test_protect_dest_only_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "fpd_src")
dest = os.path.join(TEST_DATA_DIR, "fpd_dst")
rdst = os.path.join(TEST_DATA_DIR, "fpd_rdst")
@@ -440,6 +498,7 @@ class TestFilterProtectDivergence:
rsync_result = _rsync(["-a", "--delete", "--filter=P *.log", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
assert os.path.exists(os.path.join(rdst, "extra.log")), "rsync did not protect extra.log"
assert not os.path.exists(os.path.join(rdst, "other.txt")), "rsync did not delete other.txt"
clean_dir(dest)
received = get_dest_received_dir(dest, source)
@@ -451,9 +510,29 @@ class TestFilterProtectDivergence:
flags=["-a", "--delete", "--filter=P *.log"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
# Pin the known divergence: FastSync deletes the destination-only file.
assert not os.path.exists(os.path.join(received, "extra.log")), (
"FastSync now protects destination-only P matches; the --filter row may be "
"upgradable to full parity"
)
assert os.path.exists(os.path.join(received, "extra.log")), (
"FastSync must protect a destination-only P match like rsync")
assert not os.path.exists(os.path.join(received, "other.txt"))
@pytest.mark.ci
def test_protect_dest_only_dry_run_enumeration(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "fpd_nd_src")
dest = os.path.join(TEST_DATA_DIR, "fpd_nd_dst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
received = get_dest_received_dir(dest, source)
clean_dir(received)
_write(os.path.join(received, "keep.txt"), b"keep\n")
_write(os.path.join(received, "extra.log"), b"extra\n")
_write(os.path.join(received, "other.txt"), b"other\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest,
flags=["-a", "-n", "--delete", "--out-format=%n",
"--filter=P *.log"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert "other.txt" in result.stdout, result.stdout
assert "extra.log" not in result.stdout, result.stdout
assert os.path.exists(os.path.join(received, "extra.log"))
assert os.path.exists(os.path.join(received, "other.txt"))
tests/integration/test_output_parity.py
+172 -8
View File
@@ -289,6 +289,49 @@ def _make_one_file(root, name="f.bin", size=100):
fh.write(bytes((i * 7 + 3) & 0xFF for i in range(size)))
def _make_multidir_tree(root):
"""Multi-directory corpus for the --progress file-list tests: nested files,
a directory-only branch, an empty directory and a symlink."""
clean_dir(root)
for rel, data in (("a.txt", b"alpha\n"), ("b.txt", b"bravo\n"),
("sub1/c.txt", b"charlie\n"), ("sub1/deep/d.txt", b"delta\n"),
("sub2/e.txt", b"echo\n")):
path = os.path.join(root, rel)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(data)
os.symlink("a.txt", os.path.join(root, "link1"))
os.makedirs(os.path.join(root, "emptydir"), exist_ok=True)
def _parse_progress(text):
"""Name lines and the `to-chk` denominators from a --progress run."""
names = []
totals = set()
for line in text.splitlines():
line = line.rstrip()
if not line or line == "sending incremental file list":
continue
if "%" in line:
match = re.search(r"to-chk=\d+/(\d+)", line)
if match:
totals.add(int(match.group(1)))
continue
if line == "./": # root-line trigger is a separate documented residual
continue
names.append(line)
return sorted(names), totals
def _pick_stats(text, keys):
out = {}
for line in text.splitlines():
for key in keys:
if line.startswith(key + ":"):
out[key] = line
return out
class TestWireStatsParity:
"""Wire-counter output parity: --out-format %b/%c/%C, --progress and
--stats versus real rsync 3.4.1."""
@@ -484,6 +527,62 @@ class TestWireStatsParity:
assert "to-chk=0/2" in fast_lines[-1], fast_lines[-1]
assert fast_lines[-1] == rsync_lines[-1], (rsync_lines[-1], fast_lines[-1])
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_progress_multidir_file_list_matches_rsync(self, shared_server, mt):
"""A multi-directory tree: the paths-only pre-count must reproduce
rsync's file-list set and `to-chk` denominator. Per-directory name
lines are emitted for directories, symlinks and the empty directory; the
name set and the denominator (every entry plus the transfer root) match
rsync, while the emitted *order* remains a documented residual (rsync
sorts depth-first, FastSync streams in readdir/BFS order)."""
source = os.path.join(TEST_DATA_DIR, "wire_pgmd_src")
dest = os.path.join(TEST_DATA_DIR, "wire_pgmd_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_pgmd_rdst")
_make_multidir_tree(source)
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-a", "--progress", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
flags = ["-a", "--progress"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
rsync_names, rsync_totals = _parse_progress(rsync_result.stdout)
fast_names, fast_totals = _parse_progress(result.stdout)
assert sorted(rsync_names) == [
"a.txt", "b.txt", "emptydir/", "link1 -> a.txt", "sub1/",
"sub1/c.txt", "sub1/deep/", "sub1/deep/d.txt", "sub2/", "sub2/e.txt",
], rsync_names
assert fast_names == rsync_names, (rsync_names, fast_names)
# 10 entries + the transfer-root "." counted by rsync's file list.
assert rsync_totals == {11}, rsync_totals
assert fast_totals == rsync_totals, (rsync_totals, fast_totals)
@pytest.mark.ci
def test_progress_delete_during_reuses_pre_scan(self):
"""--delete-during + --progress reuses the keep-set pre-scan instead of
walking the tree a second time: the file-list total and directory name
lines are identical to a plain --progress run."""
source = os.path.join(TEST_DATA_DIR, "wire_pgdel_src")
dest = os.path.join(TEST_DATA_DIR, "wire_pgdel_dst")
_make_multidir_tree(source)
clean_dir(dest)
server = ServerManager()
server.start(extra_args=["--allow-super", "--allow-delete"])
try:
result, _ = run_client(source, dest, flags=["-a", "--progress", "--delete-during"],
port=server.port)
finally:
server.stop()
assert result.returncode == 0, result.stderr[:300]
names, totals = _parse_progress(result.stdout)
assert totals == {11}, totals
assert "sub1/" in names and "sub1/deep/" in names and "emptydir/" in names, names
assert "link1 -> a.txt" in names, names
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
@@ -496,6 +595,9 @@ class TestWireStatsParity:
_make_one_file(source, "f.bin", 6000)
clean_dir(dest)
clean_dir(rdst)
# Start both tools from the same state: rsync's destination root exists,
# so pre-create FastSync's mirrored logical root as well.
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
rsync_result = _rsync(["-a", "--stats", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
flags = ["-a", "--stats"] + (["--threads"] if mt else [])
@@ -526,17 +628,18 @@ class TestWireStatsParity:
@requires_rsync
@pytest.mark.ci
def test_stats_file_count_breakdown_matches_rsync(self, shared_server):
"""`Number of files` now carries rsync's per-type breakdown: the scanner
accounts directory entries (captured for -a/-t/-p) plus reg/link/special
from the transfer list. `Number of created files` still lacks the type
breakdown (FastSync cannot tell which entries the receiver newly
created), so that residual is pinned separately."""
"""`Number of files` and `Number of created files` both carry rsync's
per-type breakdown (protocol 2.28.0 reports the receiver-created
reg/dir/link/special split over STATUS_STATS)."""
source = os.path.join(TEST_DATA_DIR, "wire_stc_src")
dest = os.path.join(TEST_DATA_DIR, "wire_stc_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_stc_rdst")
_make_one_file(source, "f.bin", 6000)
clean_dir(dest)
clean_dir(rdst)
# Start both tools from the same state: rsync's destination root exists,
# so pre-create FastSync's mirrored logical root as well.
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
rsync_result = _rsync(["-a", "--stats", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--stats"],
@@ -556,10 +659,71 @@ class TestWireStatsParity:
assert re.match(r"Number of files: 2 \(reg: 1, dir: 1\)$", r_files), r_files
assert r_files == f_files, (r_files, f_files)
# rsync always carries the created type breakdown; FastSync prints the
# bare transferred-regular count (documented residual).
assert re.match(r"Number of created files: 1 \(reg: 1\)$", r_created), r_created
assert re.fullmatch(r"Number of created files: 1", f_created), f_created
assert f_created == r_created, (r_created, f_created)
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_stats_created_and_literal_fresh_update_delta(self, shared_server, mt):
"""The receiver-observed counters must match rsync for the three
transfer shapes: a fresh create (created breakdown + whole-file literal),
an update (created == 0, whole-file literal), and a delta update (only
the literal delta fragments are counted, not the whole file)."""
source = os.path.join(TEST_DATA_DIR, "wire_stcd_src")
dest = os.path.join(TEST_DATA_DIR, "wire_stcd_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_stcd_rdst")
clean_dir(source)
clean_dir(dest)
clean_dir(rdst)
os.makedirs(source, exist_ok=True)
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
with open(os.path.join(source, "big.bin"), "wb") as fh:
fh.write(bytes(range(256)) * 4096) # 1 MiB
mt_flag = ["--threads"] if mt else []
def compare(tag):
# Pin the delta block size on both ends: rsync's adaptive block size
# would otherwise make the literal/matched split non-comparable.
rsync_result = _rsync(["-a", "--stats", "--no-whole-file", "-B8192",
source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(
source, dest,
flags=["-a", "--stats", "--incremental", "--delta", "-B8192"] + mt_flag,
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
keys = ("Number of created files", "Literal data", "Matched data",
"Total transferred file size")
r = _pick_stats(rsync_result.stdout, keys)
f = _pick_stats(result.stdout, keys)
assert r == f, f"{tag}: rsync={r} fastsync={f}"
return r
fresh = compare("fresh")
assert re.match(r"Number of created files: 1 \(reg: 1\)$",
fresh["Number of created files"]), fresh
# Update the source and re-run: the destination already exists.
sleep_mtime = os.path.getmtime(os.path.join(source, "big.bin")) + 2
with open(os.path.join(source, "big.bin"), "r+b") as fh:
fh.seek(100)
fh.write(b"XXXXXXXXXX")
os.utime(os.path.join(source, "big.bin"), (sleep_mtime, sleep_mtime))
update = compare("update")
assert update["Number of created files"] == "Number of created files: 0", update
# Second delta update: change bytes far apart, so rsync ships only the
# literal fragments and FastSync must report the same Literal data.
sleep_mtime = os.path.getmtime(os.path.join(source, "big.bin")) + 2
with open(os.path.join(source, "big.bin"), "r+b") as fh:
fh.seek(500000)
fh.write(b"YYYYYYYYYY")
os.utime(os.path.join(source, "big.bin"), (sleep_mtime, sleep_mtime))
delta = compare("delta")
assert delta["Number of created files"] == "Number of created files: 0", delta
lit = int(delta["Literal data"].split(":", 1)[1].strip().split()[0].replace(",", ""))
assert 0 < lit < 1024 * 1024, delta
@requires_rsync
@pytest.mark.ci
tests/integration/test_parity_quickwins.py
+129 -7
View File
@@ -719,13 +719,18 @@ class TestVerifyAndFlip:
source = self._src("cmpd")
dest = self._dst("cmpd")
rdst = self._dst("cmpd_r")
# Pin the mtime so rsync's size+mtime quick-check (and FastSync's
# default) matches deterministically across a second boundary.
OLD = 1_500_000_000
with open(os.path.join(source, "f.txt"), "wb") as fh:
fh.write(b"basis-content\n")
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
# rsync resolves --compare-dest relative to the destination dir; its
# basis file sits at the transfer-relative path.
os.makedirs(os.path.join(rdst, "basis"), exist_ok=True)
with open(os.path.join(rdst, "basis", "f.txt"), "wb") as fh:
fh.write(b"basis-content\n")
os.utime(os.path.join(rdst, "basis", "f.txt"), (OLD, OLD))
rs = _rsync(["-a", "--compare-dest=basis", source + "/", rdst + "/"])
assert rs.returncode == 0, rs.stderr
assert not os.path.exists(os.path.join(rdst, "f.txt")), \
@@ -738,6 +743,7 @@ class TestVerifyAndFlip:
os.makedirs(basis, exist_ok=True)
with open(os.path.join(basis, "f.txt"), "wb") as fh:
fh.write(b"basis-content\n")
os.utime(os.path.join(basis, "f.txt"), (OLD, OLD))
received = get_dest_received_dir(dest, source)
result, _ = run_client(source, dest,
flags=["--compare-dest=basis", "--incremental"],
@@ -751,14 +757,17 @@ class TestVerifyAndFlip:
def test_link_dest_hardlinks_matches_rsync(self, shared_server):
source = self._src("linkd")
dest = self._dst("linkd")
OLD = 1_500_000_000
with open(os.path.join(source, "f.txt"), "wb") as fh:
fh.write(b"link-basis-content\n")
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
rel = os.path.abspath(source).lstrip(os.sep)
basis = os.path.join(dest, "basis", rel)
os.makedirs(basis, exist_ok=True)
basis_file = os.path.join(basis, "f.txt")
with open(basis_file, "wb") as fh:
fh.write(b"link-basis-content\n")
os.utime(basis_file, (OLD, OLD))
received = get_dest_received_dir(dest, source)
result, _ = run_client(source, dest,
flags=["--link-dest=basis", "--incremental"],
@@ -771,12 +780,11 @@ class TestVerifyAndFlip:
@requires_rsync
def test_basis_dir_size_only_content_residual(self, shared_server):
"""Documented residual (RSYNC_COMPAT.md basis-dir rows): FastSync
xxHash-verifies a basis hit, while rsync's `--size-only` quick check
trusts the size alone. With a same-size, different-content basis,
rsync links/copies the wrong basis content while FastSync transfers the
source. This test pins both observed behaviors (FastSync is stricter,
so the rows are reclassified Divergent)."""
"""rsync parity (default): a basis hit is decided by the metadata
quick-check alone. With `--size-only`, a same-size, different-content
basis is trusted, so rsync links the basis content and FastSync must now
do the same instead of xxHash-verifying it. `--verify-basis` restores
the stricter content equality (covered by the differential test)."""
source = self._src("basissz")
rdest = self._dst("basissz_r")
fdest = self._dst("basissz_f")
@@ -806,9 +814,123 @@ class TestVerifyAndFlip:
flags=["-a", "--size-only", "--link-dest=basis", "--incremental"],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
with open(os.path.join(received, "f.txt"), "rb") as fh:
assert fh.read() == b"BBBB\n", \
"FastSync must trust the metadata quick-check exactly like rsync"
@requires_rsync
def test_verify_basis_restores_content_check(self, shared_server):
"""FastSync-only `--verify-basis`: a same-size, same-mtime basis with
different content is rejected by the whole-file digest, so the source is
transferred instead of installing the wrong basis bytes. The default
(no flag) installs the basis content, matching rsync."""
source = self._src("vbasis")
fdest = self._dst("vbasis_f")
with open(os.path.join(source, "f.txt"), "wb") as fh:
fh.write(b"AAAA\n")
OLD = 1_400_000_000
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
rel = os.path.abspath(source).lstrip(os.sep)
basis = os.path.join(fdest, "basis", rel)
os.makedirs(basis, exist_ok=True)
with open(os.path.join(basis, "f.txt"), "wb") as fh:
fh.write(b"BBBB\n")
os.utime(os.path.join(basis, "f.txt"), (OLD, OLD))
received = get_dest_received_dir(fdest, source)
result, _ = run_client(source, fdest,
flags=["-a", "--link-dest=basis", "--incremental",
"--verify-basis"],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
with open(os.path.join(received, "f.txt"), "rb") as fh:
assert fh.read() == b"AAAA\n", \
"FastSync must verify the basis content and transfer the source"
"--verify-basis must reject the same-size/different-content basis"
def _stat_bytes(output, key):
"""Parse a --stats byte counter (e.g. ``Matched data: 65,536 bytes``)."""
for line in output.splitlines():
if line.startswith(key + ":"):
raw = line.split(":", 1)[1].strip().split()[0]
return int(raw.replace(",", ""))
return None
class TestFuzzy:
"""Track 5b: `-y`/`--fuzzy` is an internal bandwidth optimization with a
byte-exact result. FastSync ports rsync 3.4.1's weighted-Levenshtein name
heuristic, so where both delta engines admit the candidate the tools pick
the same basis (the ``fuzzy_basis`` differential asserts the tree and the
Matched/Literal counters match with the block size pinned). The residual is
candidate ELIGIBILITY: FastSync's delta size gate (both files >= 16 KiB and
a <= 10x size ratio) is narrower than rsync's, which empirically uses a
fuzzy basis well beyond 10x and below 16 KiB. These tests pin the window
boundary and prove the byte-exact fallback on both sides of it."""
_BASE = b"the quick brown fox jumps over the lazy dog\n" * 4000
def _src(self, tag):
source = os.path.join(TEST_DATA_DIR, f"fz_{tag}_src")
clean_dir(source)
return source
def _dst(self, tag):
d = os.path.join(TEST_DATA_DIR, f"fz_{tag}_dst")
clean_dir(d)
return d
def _run_both(self, shared_server, source, dest, rdst, payload, sibling,
rs_extra=(), fs_extra=()):
with open(os.path.join(source, "report_v2.txt"), "wb") as fh:
fh.write(payload)
for root in (rdst, get_dest_received_dir(dest, source)):
os.makedirs(root, exist_ok=True)
with open(os.path.join(root, "report_v1.txt"), "wb") as fh:
fh.write(sibling)
rs = _rsync(["-a", "--no-whole-file", "--fuzzy", "--stats"] +
list(rs_extra) + [source + "/", rdst + "/"])
assert rs.returncode == 0, rs.stderr[:300]
result, _ = run_client(
source, dest,
flags=["-a", "--incremental", "--delta", "--fuzzy", "--stats"] +
list(fs_extra),
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
_assert_same_tree(rdst, get_dest_received_dir(dest, source), "(--fuzzy)")
return rs, result
@requires_rsync
def test_fuzzy_above_size_window_declines_but_tree_exact(self, shared_server):
"""A sibling >10x the source is used by rsync but declined by FastSync's
delta size-ratio gate; both destinations stay byte-identical."""
n = 65536
payload = (self._BASE * ((n // len(self._BASE)) + 1))[:n]
sibling = (self._BASE * 200)[: n * 20]
source, dest, rdst = (self._src("big"), self._dst("big"),
self._dst("big_r"))
rs, result = self._run_both(shared_server, source, dest, rdst,
payload, sibling)
assert _stat_bytes(rs.stdout, "Matched data") > 0, \
"rsync should still use a >10x fuzzy basis"
assert _stat_bytes(result.stdout, "Matched data") == 0, \
"FastSync's 10x delta size-ratio gate must decline the oversized basis"
assert _stat_bytes(result.stdout, "Literal data") == n
@requires_rsync
def test_fuzzy_below_delta_minimum_declines_but_tree_exact(self, shared_server):
"""A sibling below the 16 KiB delta minimum is used by rsync but never
enters FastSync's delta/fuzzy path; both trees stay byte-identical."""
n = 8192
payload = (self._BASE * ((n // len(self._BASE)) + 1))[:n]
source, dest, rdst = (self._src("small"), self._dst("small"),
self._dst("small_r"))
rs, result = self._run_both(shared_server, source, dest, rdst,
payload, payload)
assert _stat_bytes(rs.stdout, "Matched data") > 0, \
"rsync applies --fuzzy below 16 KiB"
assert _stat_bytes(result.stdout, "Matched data") == 0, \
"FastSync's 16 KiB delta minimum must bypass the fuzzy basis"
assert _stat_bytes(result.stdout, "Literal data") == n
class TestIgnoreExistingShortCircuit:
tests/integration/test_preflight.py
+4 -4
View File
@@ -94,14 +94,14 @@ def _seed_protocol_source(source):
class TestProtocol:
@pytest.mark.ci
def test_protocol_current_version_accepted(self, shared_server):
"""--protocol=2.27.0 (the current PROTOCOL_VERSION) is accepted and the
"""--protocol=2.28.0 (the current PROTOCOL_VERSION) is accepted and the
transfer completes normally."""
source = os.path.join(TEST_DATA_DIR, "proto_ok_src")
dest = os.path.join(TEST_DATA_DIR, "proto_ok_dst")
shutil.rmtree(dest, ignore_errors=True)
os.makedirs(dest)
_seed_protocol_source(source)
result, _ = run_client(source, dest, flags=["--protocol=2.27.0"],
result, _ = run_client(source, dest, flags=["--protocol=2.28.0"],
port=shared_server.port)
assert result.returncode == 0, \
f"--protocol current run failed: {(result.stderr or result.stdout)[:400]}"
@@ -118,8 +118,8 @@ class TestProtocol:
shutil.rmtree(dest, ignore_errors=True)
os.makedirs(dest)
_seed_protocol_source(source)
for bad in ("2.22.0", "2.21.0", "2.20.0", "2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0",
"216", "31"):
for bad in ("2.27.0", "2.26.0", "2.25.0", "2.24.0", "2.23.0", "2.22.0", "2.21.0", "2.20.0",
"2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0", "216", "31"):
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
port=shared_server.port)
assert result.returncode != 0, f"--protocol={bad} should be rejected"
tests/runner.c
+6
View File
@@ -43,6 +43,7 @@
#include "test_utils.h"
#include "test_xattr.h"
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
// Define global test state variables
@@ -52,6 +53,11 @@ bool current_test_failed = false;
int main() {
signal(SIGPIPE, SIG_IGN);
/* The codec/checksum resolvers consult rsync's preference-list environment
* variables; clear them so a developer's shell cannot change test outcomes.
* The env-specific tests set and restore their own values. */
unsetenv("RSYNC_COMPRESS_LIST");
unsetenv("RSYNC_CHECKSUM_LIST");
printf("\033[1;36m=== RUNNING UNIT TESTS ===\033[0m\n\n");
RUN_TEST(test_queue);
tests/test_checksum.c
+23
View File
@@ -265,6 +265,28 @@ static void test_checksum_digest_file_matches_oneshot(void) {
free(data);
}
/* RSYNC_CHECKSUM_LIST precedence, syntax and fallback. */
static void test_checksum_choice_env_list() {
unsetenv("RSYNC_CHECKSUM_LIST");
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128);
EXPECT_EQ_INT((int)checksum_negotiate_default(), (int)CHECKSUM_ALGO_XXH128);
setenv("RSYNC_CHECKSUM_LIST", "bogus md5 xxh3", 1);
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_MD5);
setenv("RSYNC_CHECKSUM_LIST", "SHA1", 1);
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_SHA1);
/* Whitespace-separated only: comma is not a separator in rsync's syntax. */
setenv("RSYNC_CHECKSUM_LIST", "md5,xxh3", 1);
EXPECT_EQ_INT(checksum_choice_resolve(), -1);
setenv("RSYNC_CHECKSUM_LIST", " ", 1);
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128);
unsetenv("RSYNC_CHECKSUM_LIST");
}
void test_checksum(void) {
test_checksum_xxh64_seed0();
test_checksum_xxh64_empty();
@@ -281,4 +303,5 @@ void test_checksum(void) {
test_checksum_truncated_buffer_rejected();
test_checksum_null_empty_digest();
test_checksum_digest_file_matches_oneshot();
test_checksum_choice_env_list();
}
tests/test_client_cli.c
+218 -2
View File
@@ -318,7 +318,7 @@ static void test_parse_args_protocol_accept_current() {
Config* cfg = valid_client_config();
EXPECT_NOT_NULL(cfg);
char* argv_equals[] = {"fastsync", "--source-dir", "/src",
"--dest-dir", "/dst", "--protocol=2.27.0"};
"--dest-dir", "/dst", "--protocol=2.28.0"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, argv_equals, positional_args, &positional_count), 0);
@@ -328,7 +328,7 @@ static void test_parse_args_protocol_accept_current() {
cfg = valid_client_config();
EXPECT_NOT_NULL(cfg);
char* argv_space[] = {"fastsync", "--source-dir", "/src", "--dest-dir",
"/dst", "--protocol", "2.27.0"};
"/dst", "--protocol", "2.28.0"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 7, argv_space, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->version, PROTOCOL_VERSION);
@@ -340,6 +340,7 @@ static void test_parse_args_protocol_accept_current() {
static void test_parse_args_protocol_rejects_other_versions() {
static const char* const bad_versions[] = {"2.17", "2.16", "2.15.0", "2.16.0", "2.17.0",
"2.18.0", "2.19.0", "2.20.0", "2.21.0", "2.22.0",
"2.23.0", "2.24.0", "2.25.0", "2.26.0", "2.27.0",
"216", "31", "abc", ""};
for (size_t i = 0; i < sizeof(bad_versions) / sizeof(bad_versions[0]); i++) {
Config* cfg = valid_client_config();
@@ -1045,6 +1046,23 @@ static void test_parse_args_basis_dirs() {
config_delete(cfg);
}
/* --verify-basis (FastSync-only, long-only): default off; parses on as a plain
boolean and leaves the basis implications intact. */
static void test_parse_args_verify_basis() {
Config* cfg = config_create();
EXPECT_FALSE(cfg->verify_basis);
config_delete(cfg);
cfg = config_create();
int positional_args[2];
int positional_count = 0;
char* argv[] = {"fastsync", "--link-dest=prior", "--verify-basis", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->verify_basis);
EXPECT_TRUE(config_has_basis(cfg));
config_delete(cfg);
}
/* Escaping or degenerate basis-dir values must be rejected up front (they would
resolve outside the destination root on the receiver); an absolute path is
accepted (rsync parity) and canonicalized with its leading '/' preserved. */
@@ -1132,6 +1150,63 @@ static void test_parse_args_delete_timing_flags() {
config_delete(cfg);
}
/* Plain --delete with no explicit timing defaults to delete-during, matching
* rsync's --del default (progressive deletion). --delete-commit is the
* FastSync-only long spelling that selects rsync's --delete-after timing (the
* late whole-tree commit), and an explicit timing always wins over the default.
*/
static void test_parse_args_delete_default_timing_and_commit() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--delete", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_during);
EXPECT_FALSE(cfg->delete_before);
EXPECT_FALSE(cfg->delete_delay);
EXPECT_FALSE(cfg->delete_after);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg);
/* --delete-commit selects the late whole-tree commit (delete_after) and
implies --delete. */
cfg = config_create();
char* argv_commit[] = {"fastsync", "--delete-commit", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_commit, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_after);
EXPECT_FALSE(cfg->delete_before);
EXPECT_FALSE(cfg->delete_during);
EXPECT_FALSE(cfg->delete_delay);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg);
/* An explicit --delete-after alongside plain --delete keeps the late timing:
the default never overwrites an explicit timing. */
cfg = config_create();
char* argv_after[] = {"fastsync", "--delete", "--delete-after", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv_after, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_after);
EXPECT_FALSE(cfg->delete_during);
config_delete(cfg);
/* --delete-commit conflicts with a different timing. */
cfg = config_create();
char* argv_conflict[] = {"fastsync", "--delete-commit", "--delete-during", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv_conflict, positional_args, &positional_count), 0);
EXPECT_FALSE(validate_config(cfg));
config_delete(cfg);
}
/* Two different delete-timing flags on one command line are a conflict, not a
* silent last-one-wins choice. */
static void test_parse_args_delete_timing_conflict_rejected() {
@@ -1393,6 +1468,29 @@ static void test_parse_args_debug_info_levels() {
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_STATS);
config_delete(cfg);
/* --info=name level 2 enables the "is uptodate" marker; a later level-1 or
level-0 token clears it again. */
cfg = config_create();
char* name2_argv[] = {"fastsync", "--info=name2", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, name2_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_NAME | LOG_INFO_NAME_UPTODATE);
config_delete(cfg);
cfg = config_create();
char* name1_argv[] = {"fastsync", "--info=name2,name1", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, name1_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_NAME);
config_delete(cfg);
cfg = config_create();
char* name0_argv[] = {"fastsync", "--info=name2,name0", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, name0_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, 0);
config_delete(cfg);
cfg = config_create();
char* bad_argv[] = {"fastsync", "--debug=123", "/src", "/dst"};
positional_count = 0;
@@ -2408,6 +2506,119 @@ static void test_parse_args_rejects_invalid_compression_choice() {
config_delete(cfg);
}
/* rsync gives each codec its own default --compress-level; an omitted level
* resolves to that default and an explicit one is clamped to the codec range. */
static void test_parse_args_per_codec_compression_level_defaults() {
unsetenv("RSYNC_COMPRESS_LIST");
struct {
const char* choice;
int level;
} cases[] = {
{"zstd", 3},
{"zlib", 6},
{"zlibx", 6},
{"lz4", 1},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-z", "--compress-choice", (char*)cases[i].choice, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_compression);
EXPECT_EQ_INT(cfg->compression_level, cases[i].level);
config_delete(cfg);
}
/* Bare -z resolves to the zstd default. */
Config* cfg = config_create();
char* bare[] = {"fastsync", "-z", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, bare, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_level, 3);
config_delete(cfg);
/* An explicit level wins unchanged for zstd... */
cfg = config_create();
char* zv[] = {"fastsync", "-z", "--compress-level", "10", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, zv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_level, 10);
config_delete(cfg);
/* ...but zlib clamps an over-range level to 9 like rsync. */
cfg = config_create();
char* zc[] = {"fastsync", "-z", "--compress-choice", "zlib", "--compress-level", "15",
"/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 8, zc, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_level, 9);
config_delete(cfg);
}
/* RSYNC_COMPRESS_LIST drives the bare -z ("auto") resolution. */
static void test_parse_args_compression_env_list() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-z", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
setenv("RSYNC_COMPRESS_LIST", "zlib lz4", 1);
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZLIB);
EXPECT_EQ_INT(cfg->compression_level, 6);
config_delete(cfg);
/* An explicit --compress-choice beats the env list. */
cfg = config_create();
char* explicit_argv[] = {"fastsync", "-z", "--compress-choice", "zstd", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, explicit_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZSTD);
config_delete(cfg);
/* A list with no supported name is rsync's failed negotiation (exit 4). */
setenv("RSYNC_COMPRESS_LIST", "bogus", 1);
cfg = config_create();
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
config_delete(cfg);
unsetenv("RSYNC_COMPRESS_LIST");
}
/* RSYNC_CHECKSUM_LIST drives the default checksum choice. */
static void test_parse_args_checksum_env_list() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--checksum", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
setenv("RSYNC_CHECKSUM_LIST", "md5", 1);
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD5);
EXPECT_EQ_INT(cfg->checksum_transfer_algo, (int)CHECKSUM_ALGO_MD5);
config_delete(cfg);
/* An explicit --cc wins. */
cfg = config_create();
char* cc_argv[] = {"fastsync", "--checksum", "--cc=sha1", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, cc_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_SHA1);
config_delete(cfg);
/* A list with no supported name is rsync's failed negotiation (exit 4). */
setenv("RSYNC_CHECKSUM_LIST", "bogus", 1);
cfg = config_create();
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
config_delete(cfg);
unsetenv("RSYNC_CHECKSUM_LIST");
}
/* Every value-taking table option accepts an inline "--opt=value" form. */
static void test_parse_args_table_equals_size_options() {
Config* cfg = config_create();
@@ -4604,6 +4815,7 @@ void test_client_cli() {
test_parse_args_relative_no_implied_mkpath();
test_parse_args_delete_during_alias();
test_parse_args_delete_timing_flags();
test_parse_args_delete_default_timing_and_commit();
test_parse_args_delete_timing_conflict_rejected();
test_parse_args_delete_timing_without_delete_rejected();
test_parse_args_rejects_unimplemented_options();
@@ -4659,6 +4871,9 @@ void test_client_cli() {
test_parse_args_compression_alias_equals();
test_parse_args_rejects_invalid_compression_level_equals();
test_parse_args_rejects_invalid_compression_choice();
test_parse_args_per_codec_compression_level_defaults();
test_parse_args_compression_env_list();
test_parse_args_checksum_env_list();
test_parse_args_table_equals_size_options();
test_parse_args_table_equals_string_and_int_options();
test_parse_args_missing_argument_diagnostic();
@@ -4692,6 +4907,7 @@ void test_client_cli() {
test_parse_args_filter_rules();
test_parse_args_from0_cvs_filter_file_flags();
test_parse_args_basis_dirs();
test_parse_args_verify_basis();
test_parse_args_basis_invalid_paths();
test_validate_config_basis_rejects_chunk_serialization();
test_parse_args_delete_policy_flags();
tests/test_compression.c
+53
View File
@@ -4,6 +4,7 @@
#include "data.h"
#include "file.h"
#include "utils.h"
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/wait.h>
@@ -390,6 +391,56 @@ static void test_codec_name_mapping() {
EXPECT_TRUE(compression_algo_enabled(COMPRESSION_ALGO_ZSTD));
}
/* rsync 3.4.1's per-codec default levels and its clamping ranges. */
static void test_codec_level_defaults_and_clamp() {
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), ZSTD_CLEVEL_DEFAULT);
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), 3);
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIB), 6);
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIBX), 6);
/* lz4 has no tunable level; a positive placeholder keeps the codec engaged. */
EXPECT_TRUE(compression_default_level(COMPRESSION_ALGO_LZ4) > 0);
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_NONE), 0);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 1), 1);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 22), 22);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 23), 22);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 0), 1);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 15), 9);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIBX, 15), 9);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 1), 1);
EXPECT_TRUE(compression_clamp_level(COMPRESSION_ALGO_LZ4, 20) > 0);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_NONE, 20), 0);
}
/* RSYNC_COMPRESS_LIST precedence, syntax and fallback. */
static void test_codec_choice_env_list() {
unsetenv("RSYNC_COMPRESS_LIST");
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
EXPECT_EQ_INT((int)compression_negotiate_default(), (int)COMPRESSION_ALGO_ZSTD);
/* Unknown entries are skipped; the first supported wins. */
setenv("RSYNC_COMPRESS_LIST", "bogus zlib lz4", 1);
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZLIB);
/* Case-insensitive. */
setenv("RSYNC_COMPRESS_LIST", "ZSTD", 1);
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
/* Whitespace-separated; the client half ends at '&'. */
setenv("RSYNC_COMPRESS_LIST", "lz4 zlib & zstd", 1);
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_LZ4);
/* Blank falls back to the compiled-in order. */
setenv("RSYNC_COMPRESS_LIST", " ", 1);
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
/* rsync's syntax has no comma/colon separator: this is one unknown name. */
setenv("RSYNC_COMPRESS_LIST", "bogus,lz4", 1);
EXPECT_EQ_INT(compression_choice_resolve(), -1);
unsetenv("RSYNC_COMPRESS_LIST");
}
/* The process-global codec selects what the legacy wrappers produce. */
static void test_codec_global_selection() {
Data* original = data_create_empty(64);
@@ -423,5 +474,7 @@ void test_compression() {
test_chunk_compress_decompress_roundtrip();
test_codec_roundtrips();
test_codec_name_mapping();
test_codec_level_defaults_and_clamp();
test_codec_choice_env_list();
test_codec_global_selection();
}
tests/test_config.c
+106 -7
View File
@@ -1,6 +1,7 @@
#include "test_config.h"
#include "config.h"
#include "delta.h"
#include "filter.h"
#include "identity.h"
#include "multiprocessing.h"
#include "protocol.h"
@@ -2119,6 +2120,48 @@ static void test_config_receive_rejects_oversized_string_budget() {
config_delete(over_bytes);
}
/* Pre-auth bounds for the receiver-side filter rule block (protocol 2.28.0).
A peer may send `protect`/`risk` rules; the receiver must reject an over-cap
count or an over-long pattern before evaluating anything, so a crafted config
cannot drive unbounded glob work or install a rule that silently never
matches. */
static void test_config_receive_rejects_bad_protect_rules() {
if (is_running_under_valgrind())
return;
/* Over-cap rule count: one more than MAX_FILTER_RULES rules. */
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->filters = array_list_create(free);
EXPECT_NOT_NULL(c->filters);
for (int i = 0; i <= MAX_FILTER_RULES; i++)
EXPECT_TRUE(array_list_add(c->filters, str_dup("- *.tmp")));
EXPECT_TRUE(roundtrip_config_rejected(c));
config_delete(c);
/* A pattern longer than the receiver's evaluation bound is rejected by the
sender (mirroring the receiver's guard) instead of being sent as an inert
rule. */
c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->filters = array_list_create(free);
EXPECT_NOT_NULL(c->filters);
size_t big = MAX_PROTECT_PATTERN_LEN + 1;
char* long_rule = malloc(big + 3);
EXPECT_NOT_NULL(long_rule);
long_rule[0] = '-';
long_rule[1] = ' ';
memset(long_rule + 2, 'x', big);
long_rule[big + 2] = '\0';
EXPECT_TRUE(array_list_add(c->filters, long_rule));
EXPECT_TRUE(roundtrip_config_rejected(c));
config_delete(c);
}
/* identity_copy_as_refused() is the pure, pre-snapshot refusal predicate: a
--copy-as is refused when the receiver is not root OR the effective super
mode is OFF (an operator veto), and never when --copy-as is unset. */
@@ -2590,6 +2633,46 @@ static bool basis_equal(const Config* a, const Config* b) {
return true;
}
/* The receiver reconstructs its delete-protection list from the sender's
* compiled base rules, so compare the received list against a fresh
* filter_base_build() of the sender's raw --filter texts. */
static bool filter_rules_equal(const Config* a, const FilterRuleList* got) {
int count = a->filters ? a->filters->size : 0;
const char** texts = NULL;
if (count > 0) {
texts = calloc((size_t)count, sizeof(char*));
if (!texts)
return false;
for (int i = 0; i < count; i++)
texts[i] = (const char*)a->filters->items[i];
}
char err[160];
FilterRuleList* expected =
filter_base_build(texts, count, a->cvs_exclude, a->delete_excluded, err, sizeof(err));
free(texts);
if (!expected)
return false;
bool equal = true;
int want = expected->count;
int have = got ? got->count : 0;
if (want != have) {
equal = false;
} else {
for (int i = 0; i < want; i++) {
const FilterRule* x = expected->items[i];
const FilterRule* y = got->items[i];
if (x->action != y->action || x->sides != y->sides || x->anchored != y->anchored ||
x->dir_only != y->dir_only || x->negate != y->negate ||
!str_opt_equal(x->owner, y->owner) || !str_opt_equal(x->pattern, y->pattern)) {
equal = false;
break;
}
}
}
filter_rule_list_free(expected);
return equal;
}
#define CONFIG_CMP_BOOL(a, b, name) ((a)->name == (b)->name)
#define CONFIG_CMP_INT(a, b, name) ((a)->name == (b)->name)
#define CONFIG_CMP_RAW(a, b, name) ((a)->name == (b)->name)
@@ -2615,6 +2698,7 @@ static bool basis_equal(const Config* a, const Config* b) {
#define CONFIG_CMP_COPY_AS_ID(a, b, name) (!(a)->copy_as_set || (a)->name == (b)->name)
#define CONFIG_CMP_BLOCK_SKIP_SUFFIXES(a, b, name) skip_suffixes_equal((a), (b))
#define CONFIG_CMP_BLOCK_BASIS(a, b, name) basis_equal((a), (b))
#define CONFIG_CMP_BLOCK_PROTECT_RULES(a, b, name) filter_rules_equal((a), (b)->name)
#define CONFIG_CMP_BLOCK_IDMAP(a, b, name) \
idmap_equal((a)->name, (a)->name##_count, (b)->name, (b)->name##_count)
@@ -2784,6 +2868,7 @@ static void golden_config_populate(Config* c) {
c->skip_compress_suffixes[1] = str_dup(".xz");
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_COMPARE, "compare"), 0);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "link"), 0);
c->verify_basis = true;
c->fuzzy = true;
c->checksum_algo = CHECKSUM_ALGO_MD5;
c->checksum_seed = 0x1122334455667788ULL;
@@ -2829,17 +2914,29 @@ static void golden_config_populate(Config* c) {
c->copy_as_set = true;
c->copy_as_uid = 111;
c->copy_as_gid = 222;
/* Compile-through delete-protection rules (protocol 2.28.0). The golden
* sender serializes its compiled base rules, so populate a diverse set that
* exercises both sides, negate, anchoring and dir-only. */
c->filters = array_list_create(free);
array_list_add(c->filters, str_dup("P *.log"));
array_list_add(c->filters, str_dup("+r **/*.txt"));
array_list_add(c->filters, str_dup("H,!secret"));
array_list_add(c->filters, str_dup("- /sub/dir/"));
}
/* The pinned golden frame (protocol 2.27.0). The values below are the only
/* The pinned golden frame (protocol 2.28.0). The values below are the only
* thing that ties the generated table to the historical wire format; update
* them ONLY with a PROTOCOL_VERSION bump and a documented reason. The 2.24.0
* delete-plan wave changed only the version string; 2.25.0 appended the
* report_stats bool, 2.26.0 appended the compression_algo int, and 2.27.0
* appended the report_deletes bool. The byte-exact values are recomputed for
* the merged layout. */
#define GOLDEN_WIRE_LEN 709
#define GOLDEN_WIRE_HASH 14423869696887880000ULL
* report_stats bool, 2.26.0 appended the compression_algo int, 2.27.0 appended
* the report_deletes bool, and 2.28.0 changed only the version string and
* appended the receiver-side delete-protection rule block (the STATUS_STATS
* body also grew, but that is not part of this frame). Track 5a appends the
* FastSync-only verify_basis bool to the basis block WITHOUT a version bump
* (project decision), so the frame grew by one int to 886 bytes. The
* byte-exact values are recomputed for the merged layout. */
#define GOLDEN_WIRE_LEN 886
#define GOLDEN_WIRE_HASH 5809509022716816757ULL
static unsigned long long fnv1a_64(const unsigned char* buf, size_t len) {
unsigned long long h = 1469598103934665603ULL;
@@ -2921,7 +3018,7 @@ static unsigned long long capture_wire_hash(const Config* cfg, size_t* out_len)
return h;
}
/* Byte-for-byte wire compatibility guard (protocol 2.27.0). The expected hash
/* Byte-for-byte wire compatibility guard (protocol 2.28.0). The expected hash
* pins the pre-X-macro byte stream; the refactor MUST NOT change it. */
static void test_config_wire_golden() {
if (is_running_under_valgrind())
@@ -2988,6 +3085,7 @@ static void test_config_wire_golden_receive() {
recv->groupmap[0].to_name != NULL && strcmp(recv->groupmap[0].to_name, "root") == 0;
ok = ok && recv->basis_count == 2 && recv->basis_dirs[0].type == BASIS_DEST_COMPARE &&
recv->basis_dirs[1].type == BASIS_DEST_LINK;
ok = ok && recv->verify_basis;
ok = ok && recv->module != NULL && strcmp(recv->module, "goldenmod") == 0;
ok = ok && recv->copy_as_set && recv->copy_as_uid == 111 && recv->copy_as_gid == 222;
}
@@ -3248,6 +3346,7 @@ void test_config() {
test_config_wire_golden_receive();
test_config_wire_receive_bounds();
test_config_receive_rejects_overcap_counts();
test_config_receive_rejects_bad_protect_rules();
test_config_wire_roundtrip_all_fields();
test_config_preserve_attribute_wire_roundtrip();
}
tests/test_delete_plan.c
+78 -29
View File
@@ -17,16 +17,16 @@
* caller/receiver entry point) describing `dir` with no kept children. */
static void send_plan_frame(int fd, const char* dir) {
EXPECT_TRUE(send_int(fd, 0)); /* has_config */
EXPECT_TRUE(send_int(fd, 1)); /* apply: a real plan */
EXPECT_TRUE(send_wire_str(fd, dir));
EXPECT_TRUE(send_int(fd, 0)); /* kept child dirs */
EXPECT_TRUE(send_int(fd, 0)); /* kept child files */
}
/* --delete-delay: a directory snapshotted into the plan that is refilled before
* the commit must NOT be counted as deleted once its unlink fails ENOTEMPTY.
* Regression for delete_plan.c counting at snapshot (defer_add) instead of at
* the actual removal. */
static void test_delete_delay_refilled_dir_not_counted(void) {
* the commit is re-scanned and removed recursively (rsync parity). Regression
* for the old single-unlink ENOTEMPTY path that left the directory behind. */
static void test_delete_delay_refilled_dir_removed_recursively(void) {
char root[] = "/tmp/fastsync_dp_refill_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
char extra[1024];
@@ -57,16 +57,14 @@ static void test_delete_delay_refilled_dir_not_counted(void) {
close(fd);
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_OK);
/* ENOTEMPTY: the directory survives, so it must not be reported as deleted. */
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
/* The late content and the directory itself are both removed. */
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 2);
struct stat st;
EXPECT_EQ_INT(lstat(extra, &st), 0);
EXPECT_TRUE(lstat(extra, &st) != 0);
delete_plan_session_destroy(session);
close(p[0]);
close(p[1]);
unlink(refill);
rmdir(extra);
rmdir(root);
config_delete(config);
}
@@ -106,8 +104,9 @@ static void test_delete_delay_removed_file_counted(void) {
config_delete(config);
}
/* --max-delete still bounds the deferred plan; the actual (removed) count must
* not exceed the limit even though more extras existed. */
/* --max-delete is charged on actual removals, not at plan/snapshot time: after
* receiving the plans the budget is untouched, and only the commit removes up to
* the limit. */
static void test_delete_delay_max_delete_bounds_actual(void) {
char root[] = "/tmp/fastsync_dp_max_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
@@ -133,8 +132,11 @@ static void test_delete_delay_max_delete_bounds_actual(void) {
EXPECT_NOT_NULL(session);
send_plan_frame(p[1], ".");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
EXPECT_TRUE(delete_plan_session_limit_reached(session));
/* Nothing removed yet, so the budget is not consumed at snapshot time. */
EXPECT_FALSE(delete_plan_session_limit_reached(session));
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
EXPECT_TRUE(delete_plan_session_limit_reached(session));
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
delete_plan_session_destroy(session);
@@ -149,13 +151,12 @@ static void test_delete_delay_max_delete_bounds_actual(void) {
config_delete(config);
}
/* --max-delete is charged at plan/snapshot time, not at actual removal: a
* deferred entry that survives ENOTEMPTY still consumes its budget slot, so a
* later directory's extra is skipped even though nothing was actually removed.
* The reported count stays 0 (actual removals) while the run is partial. The
* two plans are sent as separate frames for "a" then "b", so the ordering that
* decides which entry gets the budget is deterministic (unlike readdir order). */
static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
/* --max-delete is charged on ACTUAL removals: the refilled directory's late
* content is removed first (consuming the single budget slot), so the directory
* itself and the later extra are skipped, matching rsync. The two plans are
* sent as separate frames for "a" then "b", so the ordering that decides which
* entry gets the budget is deterministic (unlike readdir order). */
static void test_delete_delay_actual_removal_charges_budget(void) {
char root[] = "/tmp/fastsync_dp_planbudget_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
char adir[1024], bdir[1024], xdir[1024], ydir[1024];
@@ -180,18 +181,16 @@ static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
DeletePlanSession* session = delete_plan_session_create(config);
EXPECT_NOT_NULL(session);
/* Plan "a" first: its empty extra dir snapshots and charges the budget. */
/* Both plan snapshots are taken; neither consumes budget yet. */
send_plan_frame(p[1], "a");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
EXPECT_FALSE(delete_plan_session_limit_reached(session));
/* Plan "b": the budget is already spent at snapshot time, so b/y is skipped
even though a/x has not (and will not) be removed. */
send_plan_frame(p[1], "b");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
EXPECT_TRUE(delete_plan_session_limit_reached(session));
EXPECT_FALSE(delete_plan_session_limit_reached(session));
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
/* Refill a/x so its deferred rmdir fails ENOTEMPTY. */
/* Refill a/x after its plan: the recursive commit must remove this content. */
char refill[1200];
snprintf(refill, sizeof(refill), "%s/new.txt", xdir);
int fd = open(refill, O_WRONLY | O_CREAT | O_TRUNC, 0600);
@@ -199,16 +198,17 @@ static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
close(fd);
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
/* Nothing was actually removed, and the plan-time budget still stopped b/y. */
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
/* The one budget slot removed the late content; the two directories survive. */
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
EXPECT_TRUE(delete_plan_session_limit_reached(session));
struct stat st;
EXPECT_TRUE(lstat(refill, &st) != 0);
EXPECT_EQ_INT(lstat(xdir, &st), 0);
EXPECT_EQ_INT(lstat(ydir, &st), 0);
delete_plan_session_destroy(session);
close(p[0]);
close(p[1]);
unlink(refill);
rmdir(xdir);
rmdir(ydir);
rmdir(adir);
@@ -217,9 +217,58 @@ static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
config_delete(config);
}
/* Send a config-only carrier frame (apply=false): the per-run config block with
* one --delete-missing-args exact path, and no directory walk. */
static void send_config_only_frame(int fd, const char* missing_path) {
EXPECT_TRUE(send_int(fd, 1)); /* has_config */
EXPECT_TRUE(send_int(fd, 0)); /* protected prefixes */
EXPECT_TRUE(send_int(fd, 0)); /* size-skipped */
EXPECT_TRUE(send_int(fd, 1)); /* missing args */
EXPECT_TRUE(send_wire_str(fd, missing_path));
EXPECT_TRUE(send_int(fd, 0)); /* apply = false */
EXPECT_TRUE(send_wire_str(fd, "."));
EXPECT_TRUE(send_int(fd, 0));
EXPECT_TRUE(send_int(fd, 0));
}
/* The config-only carrier frame (apply=false) still applies the
* --delete-missing-args exact deletions even though it walks no directory. This
* is the fix for a --files-from list that synchronizes no directory. */
static void test_config_only_frame_applies_missing_args(void) {
char root[] = "/tmp/fastsync_dp_cfgonly_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
char gone[1024];
snprintf(gone, sizeof(gone), "%s/gone.txt", root);
int fd = open(gone, O_WRONLY | O_CREAT | O_TRUNC, 0600);
EXPECT_TRUE(fd >= 0);
close(fd);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->receive_root_directory = str_dup(root);
config->delete_missing_args = true;
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
DeletePlanSession* session = delete_plan_session_create(config);
EXPECT_NOT_NULL(session);
send_config_only_frame(p[1], "gone.txt");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
EXPECT_TRUE(lstat(gone, &(struct stat){0}) != 0);
delete_plan_session_destroy(session);
close(p[0]);
close(p[1]);
rmdir(root);
config_delete(config);
}
void test_delete_plan(void) {
test_delete_delay_refilled_dir_not_counted();
test_delete_delay_refilled_dir_removed_recursively();
test_delete_delay_removed_file_counted();
test_delete_delay_max_delete_bounds_actual();
test_delete_delay_refilled_dir_charges_budget_at_plan();
test_delete_delay_actual_removal_charges_budget();
test_config_only_frame_applies_missing_args();
}
tests/test_file.c
+156
View File
@@ -6,6 +6,7 @@
#include "file_receive.h"
#include "data.h"
#include "config.h"
#include "charset.h"
#include "utils.h"
#include "protocol.h"
#include "test_utils.h"
@@ -1807,6 +1808,159 @@ static void test_receive_incremental_check_empty_path() {
config_delete(cfg);
}
/* Pure policy helpers behind the basis quick-check / --verify-basis decision. */
static void test_file_basis_quick_match_decision() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
EXPECT_FALSE(file_basis_content_required(cfg));
cfg->verify_basis = true;
EXPECT_TRUE(file_basis_content_required(cfg));
cfg->verify_basis = false;
struct stat st;
memset(&st, 0, sizeof(st));
st.st_mtime = 1500000000;
#ifdef __linux__
st.st_mtim.tv_nsec = 500;
#endif
/* Equal size is required by the caller; this leg is the mtime / --size-only
rule. Equal mtime matches, a different mtime misses by default. */
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000000, 500));
EXPECT_FALSE(file_basis_quick_match(cfg, &st, 1500000001, 500));
cfg->size_only = true;
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000001, 500));
cfg->size_only = false;
cfg->modify_window = 2;
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000002, 500));
config_delete(cfg);
}
/* End-to-end handshake decision for a same-size, same-mtime, DIFFERENT-content
basis. Default (rsync parity): the metadata quick-check is trusted, the
receiver answers STATUS_OK and materializes the basis bytes. --verify-basis:
the whole-file digest is required, the basis is rejected and the receiver
asks for the source (STATUS_NEXT + full transfer). */
static void test_receive_incremental_check_basis_quick_check_and_verify() {
const char* root = "test_basis_quick_root";
const char* basis_dir = "test_basis_quick_root/basis";
const char* basis_file = "test_basis_quick_root/basis/f.txt";
unlink(basis_file);
unlink("test_basis_quick_root/f.txt");
rmdir(basis_dir);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(basis_dir, 0755), 0);
const char* src_bytes = "AAAA";
const unsigned long long size = 4;
const time_t mtime = 1500000000;
{
FILE* fh = fopen(basis_file, "wb");
EXPECT_NOT_NULL(fh);
// cppcheck-suppress knownConditionTrueFalse
if (fh) {
EXPECT_EQ_INT((int)fwrite("BBBB", 1, (size_t)size, fh), (int)size);
fclose(fh);
}
}
struct timespec ts[2] = {{mtime, 0}, {mtime, 0}};
EXPECT_EQ_INT(utimensat(AT_FDCWD, basis_file, ts, 0), 0);
char root_abs[PATH_MAX];
EXPECT_NOT_NULL(realpath(root, root_abs));
int root_fd = open(root_abs, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
EXPECT_TRUE(root_fd >= 0);
// cppcheck-suppress knownConditionTrueFalse
if (root_fd < 0) {
unlink(basis_file);
rmdir(basis_dir);
rmdir(root);
return;
}
EXPECT_TRUE(utils_set_authorized_root(root_fd, root_abs));
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
size_t digest_len = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, src_bytes, size, digest, sizeof(digest),
&digest_len));
/* Route protocol I/O through the explicit descriptors (a previous test group
may have left io_set_fds() bound to its own pipe). */
io_set_fds(-1, -1);
io_set_bwlimit(0);
for (int verify = 0; verify <= 1; verify++) {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->receive_root_directory = str_dup(root_abs);
cfg->checksum = false;
cfg->checksum_algo = CHECKSUM_ALGO_XXH64;
cfg->checksum_seed = 0;
cfg->use_incremental = true;
cfg->use_delta = false;
cfg->use_metadata = false;
cfg->verify_basis = (verify != 0);
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_LINK, "basis"), 0);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
EXPECT_TRUE(send_wire_str(p[1], "f.txt"));
unsigned long long check_size = size;
long long check_mtime = (long long)mtime;
long long check_mtime_nsec = 0;
EXPECT_TRUE(send_n_data(p[1], &check_size, sizeof(check_size)));
EXPECT_TRUE(send_n_data(p[1], &check_mtime, sizeof(check_mtime)));
EXPECT_TRUE(send_n_data(p[1], &check_mtime_nsec, sizeof(check_mtime_nsec)));
/* Only --verify-basis needs the digest (cfg->checksum is false) and the
pre-staged fallback full transfer the receiver will request. */
if (verify) {
uint8_t wire_len = (uint8_t)digest_len;
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
char* payload_bytes = str_dup(src_bytes);
EXPECT_NOT_NULL(payload_bytes);
Data* payload = data_create(payload_bytes, size);
EXPECT_NOT_NULL(payload);
// cppcheck-suppress knownConditionTrueFalse
if (payload)
EXPECT_TRUE(send_data(p[1], payload));
data_destroy(payload);
}
bool skipped = false;
File* file = receive_incremental_check(p[0], cfg, &skipped);
EXPECT_NOT_NULL(file);
// cppcheck-suppress knownConditionTrueFalse
if (file) {
EXPECT_FALSE(skipped);
Status reply = STATUS_ERROR;
EXPECT_TRUE(receive_status(p[1], &reply));
if (verify) {
EXPECT_EQ_INT((int)reply, (int)STATUS_NEXT);
EXPECT_NOT_NULL(file->data->data);
EXPECT_TRUE(file->data->data != NULL && memcmp(file->data->data, src_bytes, size) == 0);
} else {
EXPECT_EQ_INT((int)reply, (int)STATUS_OK);
EXPECT_TRUE(file->skip);
/* The default quick-check hit materializes from the basis PATH at
install time (streaming), so no content is buffered on the File. */
EXPECT_NOT_NULL(file->basis_link);
EXPECT_NULL(file->data->data);
}
file_destroy(file);
}
close(p[0]);
close(p[1]);
config_delete(cfg);
}
utils_set_authorized_root(-1, NULL);
close(root_fd);
unlink(basis_file);
rmdir(basis_dir);
rmdir(root);
}
/* -K/--keep-dirlinks secure open: with an authorized root, a destination path
* component that is a symlink to an IN-ROOT directory is used as that directory
* (its referent is opened through a relative O_NOFOLLOW walk from the root fd,
@@ -2140,6 +2294,8 @@ void test_file() {
test_dir_time_list();
test_dir_time_list_cap();
test_receive_incremental_check_empty_path();
test_file_basis_quick_match_decision();
test_receive_incremental_check_basis_quick_check_and_verify();
test_keep_dirlinks_secure_open();
test_inplace_overwrite_clears_special_mode_bits();
test_inplace_overwrite_metadata_strips_special_bits();
tests/test_format.c
+34
View File
@@ -86,9 +86,43 @@ static void test_dest_state_roundtrip() {
close(fds[1]);
}
static void test_stats_roundtrip() {
/* STATUS_STATS grew from three counters (2.25.0) to eight (2.28.0); the codec
* must carry every field, including the receiver-observed literal/created
* counters, across the wire in order. */
int fds[2];
if (socketpair(AF_UNIX, SOCK_STREAM, 0, fds) != 0)
return;
ReceiverStats out;
memset(&out, 0, sizeof(out));
out.matched_data = 111111111ULL;
out.deleted_files = 7;
out.would_delete_count = 3;
out.literal_bytes = 222222222ULL;
out.created_reg = 5;
out.created_dir = 4;
out.created_link = 2;
out.created_special = 1;
ReceiverStats in;
memset(&in, 0, sizeof(in));
EXPECT_TRUE(format_stats_send(fds[0], &out));
EXPECT_TRUE(format_stats_receive(fds[1], &in));
EXPECT_TRUE(in.matched_data == out.matched_data);
EXPECT_TRUE(in.deleted_files == out.deleted_files);
EXPECT_TRUE(in.would_delete_count == out.would_delete_count);
EXPECT_TRUE(in.literal_bytes == out.literal_bytes);
EXPECT_TRUE(in.created_reg == out.created_reg);
EXPECT_TRUE(in.created_dir == out.created_dir);
EXPECT_TRUE(in.created_link == out.created_link);
EXPECT_TRUE(in.created_special == out.created_special);
close(fds[0]);
close(fds[1]);
}
void test_format(void) {
test_human_size_decimal();
test_big_num_grouping();
test_datetime_format();
test_dest_state_roundtrip();
test_stats_roundtrip();
}
tests/test_fuzz_smoke.c
+5 -4
View File
@@ -18,10 +18,11 @@
/* P8 config-frame tail: super_mode (4) + copy-as presence (4) + uid (4) + gid (4). */
#define P8_TAIL_BYTES 16
/* Bytes after the P8 tail: report_dest_info (4), report_stats (4, wire-stats
* wave) and compression_algo (4, codec wave). The P8 fields sit this many
* bytes before the end of the frame. */
#define POST_P8_TAIL_BYTES 12
/* Bytes after the P8 tail: report_dest_info (4), report_stats (4),
* report_deletes (4, --info=del wave), compression_algo (4, codec wave) and the
* receiver delete-protection count (4, protocol 2.28.0). The P8 fields sit
* this many bytes before the end of the frame. */
#define POST_P8_TAIL_BYTES 20
/* Smoke test for chunk_deserialize fuzz target */
static void test_fuzz_chunk_deserialize() {
tests/test_scanner.c
+50
View File
@@ -1240,6 +1240,55 @@ static int collect_scan_info_parallel(ParallelScanner* scanner, const char* root
return failed ? -1 : count;
}
/* The --progress paths-only pre-count relies on `list_dirs` emitting every
* directory (including empty ones) exactly once, alongside the files and
* symlinks the streaming scanner already emits. */
static void test_scanner_list_dirs_counts_every_entry() {
const char* root = "test_scan_listdirs";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub1", 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub1/deep", 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub2", 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_listdirs/emptydir", 0755), 0);
create_test_file("test_scan_listdirs/a.txt", "a");
create_test_file("test_scan_listdirs/sub1/c.txt", "c");
create_test_file("test_scan_listdirs/sub1/deep/d.txt", "d");
create_test_file("test_scan_listdirs/sub2/e.txt", "e");
EXPECT_EQ_INT(symlink("a.txt", "test_scan_listdirs/link1"), 0);
ScannerOptions options = {0};
options.list_dirs = true;
options.emit_empty_dirs = true;
options.follow_symlinks = true; /* -l/--links: carry symlinks, don't skip */
ScanInfo infos[16];
int count = collect_scan_info(root, &options, infos, 16);
EXPECT_EQ_INT(count, 9);
int dirs = 0;
for (int i = 0; i < count; i++) {
if (infos[i].is_dir)
dirs++;
}
EXPECT_EQ_INT(dirs, 4);
EXPECT_TRUE(scan_info_present(infos, count, "sub1", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "sub1/deep", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "sub2", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "emptydir", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "a.txt", false, ""));
EXPECT_TRUE(scan_info_present(infos, count, "sub1/c.txt", false, ""));
EXPECT_TRUE(scan_info_present(infos, count, "link1", false, ""));
unlink("test_scan_listdirs/a.txt");
unlink("test_scan_listdirs/sub1/c.txt");
unlink("test_scan_listdirs/sub1/deep/d.txt");
unlink("test_scan_listdirs/sub2/e.txt");
unlink("test_scan_listdirs/link1");
rmdir("test_scan_listdirs/sub1/deep");
rmdir("test_scan_listdirs/sub1");
rmdir("test_scan_listdirs/sub2");
rmdir("test_scan_listdirs/emptydir");
rmdir(root);
}
/* -d without --files-from emits exactly the source-root directory (empty) and
* never descends. */
static void test_dirs_no_descent() {
@@ -1667,6 +1716,7 @@ void test_scanner() {
test_per_dir_filter_override(true);
test_dirs_no_descent();
test_dirs_files_from();
test_scanner_list_dirs_counts_every_entry();
test_files_from_relative_send_path();
test_scanner_captures_directory_times();
test_scanner_chunk_ownership();
tests/test_server.c
+148 -13
View File
@@ -3,6 +3,8 @@
#include "config.h"
#include "delta.h"
#include "file.h"
#include "file_receive.h"
#include "format.h"
#include "log.h"
#include "protocol.h"
#include "test_utils.h"
@@ -134,6 +136,124 @@ static void test_receive_files_single_file() {
}
}
/* Protocol 2.28.0: a fresh single-file transfer over the wire reports the
* receiver-observed literal bytes and the created-regular counter through the
* terminal STATUS_STATS frame, and an update reports created_reg == 0. */
static void test_receive_stats_frame_created_and_literal() {
const char* content = "stats frame content";
size_t len = strlen(content);
char root_template[] = "/tmp/fastsync_stats_XXXXXX";
char* root = mkdtemp(root_template);
EXPECT_NOT_NULL(root);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
free(cfg->version);
cfg->version = str_dup(PROTOCOL_VERSION);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup(root);
cfg->save_to_disk = true;
cfg->report_stats = true;
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
int ret = receiver_receive_files(cfg, p[0]);
close(p[0]);
config_delete(cfg);
_exit(ret == 0 ? 0 : 1);
}
close(p[0]);
io_set_fds(p[1], p[1]);
send_status(p[1], STATUS_NEXT);
File* file = file_create("created.bin");
EXPECT_NOT_NULL(file);
file->data->data = malloc(len);
EXPECT_NOT_NULL(file->data->data);
memcpy(file->data->data, content, len);
file->data->size = len;
send_str(p[1], file->path);
send_data(p[1], file->data);
file_destroy(file);
send_status(p[1], STATUS_FINISHED);
Status status;
EXPECT_TRUE(receive_status(p[1], &status));
EXPECT_EQ_INT(status, STATUS_STATS);
ReceiverStats stats;
EXPECT_TRUE(format_stats_receive(p[1], &stats));
int would = 0;
EXPECT_TRUE(receive_int(p[1], &would));
EXPECT_EQ_INT(would, 0);
EXPECT_TRUE(stats.created_reg == 1);
EXPECT_TRUE(stats.created_dir == 0);
EXPECT_TRUE(stats.created_link == 0);
EXPECT_TRUE(stats.created_special == 0);
EXPECT_TRUE(stats.literal_bytes == (unsigned long long)len);
EXPECT_TRUE(stats.matched_data == 0);
Status final;
EXPECT_TRUE(receive_status(p[1], &final));
EXPECT_EQ_INT(final, STATUS_OK);
int wstatus;
waitpid(pid, &wstatus, 0);
close(p[1]);
config_delete(cfg);
EXPECT_TRUE(WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 0);
/* Second run against the now-existing destination: no created file. */
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
cfg = config_create();
EXPECT_NOT_NULL(cfg);
free(cfg->version);
cfg->version = str_dup(PROTOCOL_VERSION);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup(root);
cfg->save_to_disk = true;
cfg->report_stats = true;
pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
int ret = receiver_receive_files(cfg, p[0]);
close(p[0]);
config_delete(cfg);
_exit(ret == 0 ? 0 : 1);
}
close(p[0]);
io_set_fds(p[1], p[1]);
send_status(p[1], STATUS_NEXT);
file = file_create("created.bin");
EXPECT_NOT_NULL(file);
file->data->data = malloc(len);
EXPECT_NOT_NULL(file->data->data);
memcpy(file->data->data, content, len);
file->data->size = len;
send_str(p[1], file->path);
send_data(p[1], file->data);
file_destroy(file);
send_status(p[1], STATUS_FINISHED);
EXPECT_TRUE(receive_status(p[1], &status));
EXPECT_EQ_INT(status, STATUS_STATS);
EXPECT_TRUE(format_stats_receive(p[1], &stats));
EXPECT_TRUE(receive_int(p[1], &would));
EXPECT_TRUE(stats.created_reg == 0);
EXPECT_TRUE(stats.literal_bytes == (unsigned long long)len);
EXPECT_TRUE(receive_status(p[1], &final));
waitpid(pid, &wstatus, 0);
close(p[1]);
config_delete(cfg);
EXPECT_TRUE(WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 0);
}
/* Test receive_files with STATUS_ABORT */
static void test_receive_files_abort() {
Config* cfg = config_create();
@@ -943,14 +1063,18 @@ static void test_incremental_check_fifo_destination_does_not_hang() {
}
/* A server-contacting --dry-run with an alternate basis dir must never read or
hash the basis file. An exact (size+mtime+content) basis match would
otherwise let a client probe the basis bytes against its own supplied digest
(a 1-bit content oracle). The dry-run decision is metadata-only, so even a
byte-identical basis is reported as would-transfer, not a compare-dest skip. */
static void test_incremental_check_dry_run_basis_does_not_read_content() {
hash the basis file. Under the default metadata quick-check a hit needs no
basis bytes, so a compare-dest match is reported as a skip (STATUS_OK) just
like a real run -- and still no content is read. Under --verify-basis a hit
would require hashing the basis against the client-supplied digest (a 1-bit
content oracle), which a dry-run must never do, so even a byte-identical
basis is reported as would-transfer. The destination is never materialized
in either arm. */
static void run_dry_run_basis_check(bool verify, Status expected) {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->dry_run = true;
cfg->verify_basis = verify;
char* root = make_check_root("dryb");
EXPECT_NOT_NULL(root);
cfg->receive_root_directory = str_dup(root);
@@ -966,8 +1090,8 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
EXPECT_EQ_INT(stat(basis_path, &bst), 0);
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_COMPARE, "basis"), 0);
/* The (correct) source digest for the basis bytes: an unfixed dry-run would
read+hash the basis and treat this as an exact compare-dest hit. */
/* The (correct) source digest for the basis bytes: a buggy dry-run that read
and hashed the basis would treat this as an exact compare-dest hit. */
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
size_t digest_len = 0;
EXPECT_TRUE(checksum_digest((ChecksumAlgo)cfg->checksum_algo, cfg->checksum_seed, content,
@@ -986,7 +1110,8 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
bool skipped = false;
bool would_transfer = false;
File* file = receive_incremental_check_ex(p[0], cfg, &skipped, &would_transfer);
bool ok = file == NULL && !skipped && would_transfer;
bool ok = file == NULL && skipped == (expected == STATUS_OK) &&
would_transfer == (expected != STATUS_OK);
file_destroy(file);
config_delete(cfg);
close(p[0]);
@@ -1004,13 +1129,16 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
uint8_t wire_len = (uint8_t)digest_len;
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
/* The digest is only on the wire when --checksum or --verify-basis needs it
(cfg->checksum is false here); the default quick-check arm sends none. */
if (verify) {
uint8_t wire_len = (uint8_t)digest_len;
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
}
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
/* A skip here would mean the receiver read+hashed the basis file. */
EXPECT_EQ_INT(s, STATUS_DRY_RUN_TRANSFER);
EXPECT_EQ_INT(s, expected);
int status;
waitpid(pid, &status, 0);
@@ -1028,6 +1156,11 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
}
}
static void test_incremental_check_dry_run_basis_does_not_read_content() {
run_dry_run_basis_check(false, STATUS_OK);
run_dry_run_basis_check(true, STATUS_DRY_RUN_TRANSFER);
}
/* B1: a FIFO planted in a --link-dest basis directory must not block
* basis_open_regular() either; the basis match is simply declined. */
static void test_incremental_check_basis_fifo_does_not_hang() {
@@ -1158,6 +1291,7 @@ static void test_dry_run_delete_plan_commit_does_not_delete() {
EXPECT_TRUE(send_int(p[1], 0)); /* size-skipped prefixes */
EXPECT_TRUE(send_int(p[1], 1)); /* missing-args exact deletions */
EXPECT_TRUE(send_str(p[1], "victim.txt"));
EXPECT_TRUE(send_int(p[1], 1)); /* apply: a real plan */
EXPECT_TRUE(send_str(p[1], ".")); /* receive root plan */
EXPECT_TRUE(send_int(p[1], 0)); /* kept child directories */
EXPECT_TRUE(send_int(p[1], 0)); /* kept child files */
@@ -1183,6 +1317,7 @@ void test_server() {
if (!is_running_under_valgrind()) {
test_receive_files_finished();
test_receive_files_single_file();
test_receive_stats_frame_created_and_literal();
test_receive_files_abort();
test_receive_manifest_rejects_traversal();
test_receive_incremental_check_rejects_invalid_nanoseconds();
tests/test_shared_utils.c
+84 -6
View File
@@ -10,6 +10,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <threads.h>
@@ -137,7 +138,7 @@ static void test_walker_removes_extras_keeps_manifest_and_protected() {
DeleteSkipEntry skip = {"prot", false};
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 100000, &skip, 1, &deleted, NULL);
delete_extras_limited(root, manifest, NULL, 100000, &skip, 1, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_FALSE(file_exists(root, "a.txt"));
EXPECT_TRUE(file_exists(root, "keep.txt"));
@@ -172,7 +173,7 @@ static void test_walker_keeps_nested_manifest_dirs() {
EXPECT_NOT_NULL(manifest);
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, &deleted, NULL);
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_FALSE(file_exists(root, "extra.txt"));
EXPECT_TRUE(file_exists(root, "keepdir/deep/keep.txt"));
@@ -203,7 +204,7 @@ static void test_walker_max_delete_partial_deletes_up_to_cap() {
size_t deleted = 999;
size_t skipped = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, &deleted, &skipped);
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, NULL, &deleted, &skipped);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_LIMIT_REACHED);
EXPECT_EQ_INT((int)deleted, 2);
EXPECT_EQ_INT((int)skipped, 1);
@@ -224,7 +225,8 @@ static void test_walker_max_delete_exact_bound_deletes() {
ArrayList* manifest = make_manifest_strings(keeps, 0);
EXPECT_NOT_NULL(manifest);
size_t deleted = 0;
DeleteWalkResult result = delete_extras_limited(root, manifest, NULL, 2, NULL, 0, &deleted, NULL);
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_EQ_INT((int)deleted, 2);
EXPECT_FALSE(file_exists(root, "a.txt"));
@@ -257,7 +259,7 @@ static void test_walker_removes_extraneous_symlinks() {
EXPECT_NOT_NULL(manifest);
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, &deleted, NULL);
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_FALSE(file_exists(root, "link_file"));
EXPECT_FALSE(file_exists(root, "link_dir"));
@@ -293,7 +295,7 @@ static void test_walker_confines_deletion_to_synced_dirs() {
EXPECT_TRUE(array_list_add(dirs, str_dup("inscope")));
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, dirs, 100000, NULL, 0, &deleted, NULL);
delete_extras_limited(root, manifest, dirs, 100000, NULL, 0, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_TRUE(file_exists(root, "rootextra.txt"));
EXPECT_FALSE(file_exists(root, "inscope/extra.txt"));
@@ -323,6 +325,45 @@ static void test_walker_unlimited_deletes_all() {
free(root);
}
/* Receiver-side filter protection (protocol 2.28.0): a compiled protect rule
shields a DESTINATION-ONLY extra that never appeared on the sender, a risk
rule cancels an earlier/later protect (first match wins), and a dir-only
protect rule shields the whole subtree. */
static void test_walker_protect_rules_shield_dest_only() {
char* root = make_walk_root("protectrules");
EXPECT_NOT_NULL(root);
EXPECT_TRUE(write_file_at(root, "keep.txt", "kept"));
EXPECT_TRUE(write_file_at(root, "extra.log", "risk cancels protect"));
EXPECT_TRUE(write_file_at(root, "safe.log", "protected"));
EXPECT_TRUE(write_file_at(root, "other.txt", "deleted"));
EXPECT_EQ_INT(make_subdir(root, "prot"), 0);
EXPECT_TRUE(write_file_at(root, "prot/inside.txt", "shielded subtree"));
EXPECT_TRUE(write_file_at(root, "prot/deep.log", "shielded subtree"));
const char* keeps[] = {"keep.txt"};
ArrayList* manifest = make_manifest_strings(keeps, 1);
EXPECT_NOT_NULL(manifest);
const char* rule_text[] = {"R extra.log", "P *.log", "P prot/"};
char err[160];
FilterRuleList* rules = filter_base_build(rule_text, 3, false, false, err, sizeof(err));
EXPECT_NOT_NULL(rules);
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, rules, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_TRUE(file_exists(root, "keep.txt"));
EXPECT_FALSE(file_exists(root, "extra.log")); /* risk wins the first match */
EXPECT_TRUE(file_exists(root, "safe.log")); /* protect shields the extra */
EXPECT_FALSE(file_exists(root, "other.txt"));
EXPECT_TRUE(dir_exists(root, "prot"));
EXPECT_TRUE(file_exists(root, "prot/inside.txt"));
EXPECT_TRUE(file_exists(root, "prot/deep.log"));
filter_rule_list_free(rules);
array_list_delete(manifest);
remove_walk_tree(root);
free(root);
}
typedef struct {
bool eight_bit_output;
const char* expected;
@@ -578,7 +619,43 @@ static void test_getdelim_bounded() {
fclose(fp);
}
static int test_env_resolver(const char* name) {
if (strcasecmp(name, "alpha") == 0)
return 10;
if (strcasecmp(name, "beta") == 0)
return 20;
return -1;
}
static void test_env_choice_first_parsing() {
const char* var = "FASTSYNC_TEST_CHOICE_LIST";
bool specified = true;
unsetenv(var);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
EXPECT_FALSE(specified);
/* Unknown entries are skipped, case-insensitive, first supported wins. */
setenv(var, "bogus BETA alpha", 1);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 20);
EXPECT_TRUE(specified);
/* The client half ends at '&'. */
setenv(var, "alpha & beta", 1);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 10);
/* Blank means "unspecified"; all-unknown means "specified but no match". */
setenv(var, " ", 1);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
EXPECT_FALSE(specified);
setenv(var, "nope,alpha", 1);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
EXPECT_TRUE(specified);
unsetenv(var);
}
void test_shared_utils() {
test_env_choice_first_parsing();
test_path_index_bounded();
test_path_index_semantics();
test_getdelim_bounded();
@@ -589,6 +666,7 @@ void test_shared_utils() {
test_walker_removes_extraneous_symlinks();
test_walker_confines_deletion_to_synced_dirs();
test_walker_unlimited_deletes_all();
test_walker_protect_rules_shield_dest_only();
test_loopback_helpers();
test_fd_peer_ip();
tests/test_xattr.c
+2 -1
View File
@@ -210,7 +210,8 @@ static void test_xattr_receive_drops_acl_without_preserve_acls() {
/* MINOR-2: a --link-dest / -H copy fallback (linkat refused) must still apply
* the per-file xattrs and --fake-super stat. A DIRECTORY basis forces linkat
* to fail with EPERM, exercising the byte-copy fallback deterministically.
* to fail with EPERM, exercising the byte-copy fallback deterministically (the
* basis is not a regular file, so the fallback uses the caller's bytes).
* Guarded on filesystem xattr support. */
static void test_link_copy_fallback_preserves_xattrs() {
const char* dest = "test_link_xattr_dest.txt";