fix(delete-before): reuse pre-scan file list in single-threaded data pass
This commit is contained in:
+2
-2
@@ -193,7 +193,7 @@ Every one of those has an entry below with its remaining caveats.
|
||||
| Flag | Rsync Description | FastSync Status | Notes |
|
||||
|------|-------------------|-----------------|-------|
|
||||
| `--delete` | Delete extraneous files from dest | ✅ Parity | `use_delete` config field. Deletion is always derived from the keep-set the sender actually transmitted (the per-directory `STATUS_DELETE_PLAN` set by default, or the whole-tree manifest for the late timings — never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. **Lockstep track 6 (protocol 2.28.0): plain `--delete` with no explicit timing flag now defaults to `--delete-during`**, exactly like rsync's `--del` (the client normalizes it to the existing `delete_during` wire bool; no new wire field). This frees destination space progressively during the transfer and avoids the whole-old+new-tree peak that could `ENOSPC` a tight destination. The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`. **Abort/ordering parity (parity-2.29):** the complete per-directory plan set is transmitted before the first data frame, so a mid-transfer abort has already applied every planned removal exactly like rsync's generator (which runs ahead of its throttled sender); `-d/--dirs` uses the same per-directory plans (the generator records only the directories whose direct children it enumerated, so an untraversed subdirectory's mirror is shielded); and the sorted depth-first traversal makes the removal order — and therefore the survivor set under a partial `--max-delete` — match rsync exactly (`test_delete_boundary_parity.py`, `test_parity_order.py`). By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent on the wire (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing |
|
||||
| `--delete-before` | Delete before transfer | ⚠️ Caveat | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. **Phase-0 divergence (sharpened):** rsync builds the full file list first, so a source file created after that scan is NOT transferred and its destination extra is deleted; FastSync's single-threaded data pass re-scans the source, so the late file IS transferred (a safe superset), while FastSync `--threads` pipelines the scan and matches rsync |
|
||||
| `--delete-before` | Delete before transfer | ✅ Parity | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. **Phase-0 divergence closed (no-wire):** the single-threaded data pass now replays the exact file list the pre-scan built for the keep-set instead of re-reading the source, so a source file created after that scan is NOT transferred and its destination extra is deleted, exactly like rsync's single file list (and like the `--threads` path). The pre-scan captures the deferred directory times and the `--stats` directory count because no later scan runs (`test_delete_timing_parity.py::TestDeleteBeforeLateFileParity`, differential vs rsync 3.4.1) |
|
||||
| `--del`, `--delete-during` | Delete during transfer | ✅ Parity | Both spellings accepted; imply `--delete`, and since lockstep track 6 this is also the default timing of a plain `--delete`. **Protocol 2.24.0 implements per-directory delete plans:** as the sender reaches each source directory it streams a `STATUS_DELETE_PLAN` for that directory and the receiver removes that directory's extras (verified with a byte-slicing proxy). The one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) rides a dedicated config-only carrier frame with an `apply=false` flag, so it reaches the receiver even when the scope allows no directory plan at all (a `--files-from` list of bare files synchronizes no directory). **Abort/ordering parity (parity-2.29):** the complete plan set is transmitted before the first data frame, so on a mid-transfer abort every planned extra has already been removed exactly like rsync's generator (which runs ahead of its throttled sender); `-d/--dirs` no longer falls back to the end-of-transfer commit but records only the directories whose direct children it enumerated; and the sorted depth-first traversal makes the removal order — and the partial-`--max-delete` survivor set — identical to rsync (`test_delete_boundary_parity.py`, `test_parity_order.py`). `-R` plans are scoped to the transferred prefix subtree |
|
||||
| `--delete-delay` | Find deletions during, delete after | ✅ Parity | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The **reported** deleted count advances only on an actual removal. **Fixed (no-wire):** the `--max-delete` budget is now charged on ACTUAL removals (an unlink/rmdir that succeeded), not at plan/snapshot time, and a queued directory is re-scanned at commit and removed recursively (content created after the plan included), matching rsync: a snapshotted entry that fails or is skipped consumes no budget, so a later extra rsync would delete is still deleted. The deferred snapshot list keeps an independent hard cap (`DELETE_PLAN_SERVER_LIMIT`) so it cannot grow without bound now that the budget is no longer charged while scanning. A `--max-delete=2` partial delete reports exactly 2 and exits 25 in both tools, and the refilled-directory differential (late content removed, directory removed, budget shared) now matches rsync 3.4.1 on both sides (`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`). Unit tests cover recursive removal, actual-removal charging, and the bounded deferred list. **Ordering parity (parity-2.29):** the sorted depth-first traversal plus the up-front plan set make the order in which extras are removed — and therefore the survivor set under a partial `--max-delete` — match rsync exactly (differential `test_parity_order.py::test_delete_delay_deletion_order_matches_rsync` and `::test_partial_max_delete_survivor_order_matches_rsync`) |
|
||||
| `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. Selects the late whole-tree commit: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing. Since lockstep track 6 a plain `--delete` defaults to delete-during (rsync's `--del`); `--delete-after` — or the FastSync-only `--delete-commit` spelling, which selects the identical timing — is the explicit way to keep the old commit-style behavior |
|
||||
@@ -956,7 +956,7 @@ These are the last compatibility items and the closing phase toward rsync flag p
|
||||
|
||||
**Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity.
|
||||
|
||||
**Honest status after the parity 2.29 cycle (protocol 2.28.0, no wire change), updated by the parity cycle 2.29 pass, the audit-cycle follow-ups, and the triage cycle.** ✅ Parity 117 / ⚠️ Caveat 13 / ❌ Divergent 27 = 157 rows. The 2.29 cycle closed the scanner-order, delete-timing, relative-basis, and fuzzy-eligibility residuals (moving `-n`/`--delete`/`--del`/`--delete-delay` to ✅) and improved the `--info`/`--stats`/`--debug` partial rows; the triage cycle moved `-F` and `-i`/`--itemize-changes` ✅ → ⚠️ for their documented residuals. The remaining ⚠️ rows are `--info`, `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `-i`, `--delete-before`, `--filter`, `-F`, the three basis-dir options, and `-y/--fuzzy`. Earlier: **Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP
|
||||
**Honest status after the parity 2.29 cycle (protocol 2.28.0, no wire change), updated by the parity cycle 2.29 pass, the audit-cycle follow-ups, the triage cycle, and the no-wire `--delete-before` phase-0 close.** ✅ Parity 118 / ⚠️ Caveat 12 / ❌ Divergent 27 = 157 rows. The 2.29 cycle closed the scanner-order, delete-timing, relative-basis, and fuzzy-eligibility residuals (moving `-n`/`--delete`/`--del`/`--delete-delay` to ✅) and improved the `--info`/`--stats`/`--debug` partial rows; the triage cycle moved `-F` and `-i`/`--itemize-changes` ✅ → ⚠️ for their documented residuals. The `--delete-before` close (no-wire) makes the single-threaded data pass replay the pre-scan file list, so a source file created after the scan is neither transferred nor kept, matching rsync (`--delete-before` ⚠️ → ✅). The remaining ⚠️ rows are `--info`, `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `-i`, `--filter`, `-F`, the three basis-dir options, and `-y/--fuzzy`. Earlier: **Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP
|
||||
and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel /
|
||||
receiver filter engine); the wire parity-track-4a pass later added that
|
||||
receiver filter engine, flipping `--filter=RULE` back to ✅ (see above; the
|
||||
|
||||
@@ -381,21 +381,28 @@ bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* s
|
||||
paths, loading and sending nothing. --delete-before/--delete-during need the
|
||||
complete keep-set manifest before the first data byte, so it is built by a
|
||||
dedicated pre-scan pass and transmitted early; the data pass then re-scans
|
||||
with a fresh scanner. A source I/O error is fatal unless the options carry
|
||||
--ignore-errors, in which case the scan continues past the unreadable
|
||||
directory and *io_error_out reports it (the caller still performs the
|
||||
deletion but reports the run as errored). */
|
||||
with a fresh scanner. --delete-before additionally replays this very scan as
|
||||
its data pass (rsync's single file list), so `chunks_out` (optional) retains
|
||||
the scanned Chunk objects for the caller to send instead of destroying them;
|
||||
the caller owns the list and must give it a chunk_destroy destructor. A
|
||||
source I/O error is fatal unless the options carry --ignore-errors, in which
|
||||
case the scan continues past the unreadable directory and *io_error_out
|
||||
reports it (the caller still performs the deletion but reports the run as
|
||||
errored). */
|
||||
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
|
||||
DeletePlanSender* plans, bool* io_error_out,
|
||||
unsigned long long* non_dir_count_out) {
|
||||
unsigned long long* non_dir_count_out, ArrayList* chunks_out,
|
||||
bool emit_nonreg) {
|
||||
if (io_error_out)
|
||||
*io_error_out = false;
|
||||
if (non_dir_count_out)
|
||||
*non_dir_count_out = 0;
|
||||
ScannerOptions local = *options;
|
||||
/* The pre-scan is a paths-only pass with no client output; it must not emit
|
||||
--info=nonreg lines (the data pass does that once). */
|
||||
local.note_nonreg = false;
|
||||
/* The pre-scan is normally a paths-only pass with no client output: it must
|
||||
not emit --info=nonreg lines because the data pass re-scans and emits them
|
||||
once. When the caller replays this scan as the data pass (--delete-before)
|
||||
there is no later scan, so it opts in and the lines are emitted here. */
|
||||
local.note_nonreg = emit_nonreg && options->note_nonreg;
|
||||
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
|
||||
if (!scanner)
|
||||
return false;
|
||||
@@ -430,7 +437,16 @@ bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayL
|
||||
break;
|
||||
}
|
||||
}
|
||||
chunk_destroy(chunk);
|
||||
if (chunks_out) {
|
||||
/* Retain the chunk for the caller's data pass; ownership moves with it. */
|
||||
if (!array_list_add(chunks_out, chunk)) {
|
||||
ok = false;
|
||||
chunk_destroy(chunk);
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
chunk_destroy(chunk);
|
||||
}
|
||||
}
|
||||
if (ok) {
|
||||
/* Keep every traversed source directory, including empty ones, so a plan
|
||||
|
||||
+66
-21
@@ -1394,6 +1394,10 @@ typedef struct {
|
||||
Client* client;
|
||||
DirectoryScanner* scanner;
|
||||
ArrayList* manifest;
|
||||
/* --delete-before: the pre-scan that built the keep-set, retained as the data
|
||||
pass's file list (owning Chunk*; consumed chunks are NULLed as they are
|
||||
sent). NULL in every other mode, where the data pass scans normally. */
|
||||
ArrayList* prescan_chunks;
|
||||
DeletePlanSender* plan_sender;
|
||||
ArrayList* remove_sources;
|
||||
ArrayList* dir_entries;
|
||||
@@ -1478,12 +1482,23 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
||||
if (state->delete_early) {
|
||||
/* Pass 1: collect the complete keep-set (paths only, no data loaded) and
|
||||
transmit it now, before any file data. The receiver removes extras and
|
||||
acks; the transfer aborts here if the deletion could not commit. */
|
||||
acks; the transfer aborts here if the deletion could not commit. The
|
||||
scanned chunks are retained so the data pass can replay this exact list
|
||||
instead of re-reading the source (rsync builds one file list and never
|
||||
transfers a file created after it). */
|
||||
ArrayList* early_manifest = array_list_create(free);
|
||||
if (!early_manifest)
|
||||
ArrayList* prescan_chunks = array_list_create(chunk_destroy);
|
||||
if (!early_manifest || !prescan_chunks) {
|
||||
array_list_delete(early_manifest);
|
||||
array_list_delete(prescan_chunks);
|
||||
return false;
|
||||
}
|
||||
/* No later scan runs for --delete-before, so this pass must also capture the
|
||||
deferred directory times and the --stats directory count. */
|
||||
state->prepared.options.dir_entries = state->dir_entries;
|
||||
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
|
||||
bool prescan_ok = scan_paths_only(config, &state->prepared.options, early_manifest, NULL,
|
||||
&state->had_scan_io, NULL);
|
||||
&state->had_scan_io, NULL, prescan_chunks, true);
|
||||
bool early_ok = false;
|
||||
bool skip_delete = false;
|
||||
if (prescan_ok) {
|
||||
@@ -1514,8 +1529,13 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
||||
state->prepared.options.excluded_paths = NULL;
|
||||
state->prepared.options.size_skipped_paths = NULL;
|
||||
state->prepared.options.synced_dirs = NULL;
|
||||
if (!prescan_ok || (!early_ok && !skip_delete))
|
||||
if (!prescan_ok || (!early_ok && !skip_delete)) {
|
||||
array_list_delete(prescan_chunks);
|
||||
return false;
|
||||
}
|
||||
/* Adopt the captured scan as the data pass's file list (including when an
|
||||
I/O error suppressed only the deletion: the list is still complete). */
|
||||
state->prescan_chunks = prescan_chunks;
|
||||
} else if (state->delete_per_dir) {
|
||||
/* --delete-during/--delete-delay: build one plan per source directory from a
|
||||
path-only pre-scan and transmit the COMPLETE plan set now, before any data,
|
||||
@@ -1527,8 +1547,9 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
||||
if (!state->plan_sender || !state->plan_dirs)
|
||||
return false;
|
||||
state->prepared.options.plan_dirs = state->plan_dirs;
|
||||
bool prescan_ok = scan_paths_only(config, &state->prepared.options, NULL, state->plan_sender,
|
||||
&state->had_scan_io, &state->per_dir_non_dir_count);
|
||||
bool prescan_ok =
|
||||
scan_paths_only(config, &state->prepared.options, NULL, state->plan_sender,
|
||||
&state->had_scan_io, &state->per_dir_non_dir_count, NULL, false);
|
||||
bool plans_ok = false;
|
||||
bool skip_delete = false;
|
||||
if (prescan_ok) {
|
||||
@@ -1592,24 +1613,42 @@ static bool send_files_run(Config* config, SendFilesState* state) {
|
||||
state->stop = stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins,
|
||||
config->cli.stop_at_set, config->stop_at, now_mono);
|
||||
state->prepared.options.stop_condition = &state->stop;
|
||||
/* The early-delete pre-scan above already ran; only the data pass should feed
|
||||
the directory-time list (otherwise every directory would be captured
|
||||
twice). */
|
||||
state->prepared.options.dir_entries = state->dir_entries;
|
||||
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
|
||||
state->scanner =
|
||||
directory_scanner_create_with_options(config->send_directory, &state->prepared.options);
|
||||
if (!state->scanner)
|
||||
return false;
|
||||
/* --delete-before reuses the pre-scan that built the keep-set as the data
|
||||
pass's file list, so a source file created after that scan is neither
|
||||
transferred nor kept (rsync builds one file list). That pre-scan captured
|
||||
the deferred directory times and the --stats directory count because no
|
||||
later scan runs; every other mode opens a fresh data scanner here. */
|
||||
if (state->prescan_chunks == NULL) {
|
||||
state->prepared.options.dir_entries = state->dir_entries;
|
||||
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
|
||||
state->scanner =
|
||||
directory_scanner_create_with_options(config->send_directory, &state->prepared.options);
|
||||
if (!state->scanner)
|
||||
return false;
|
||||
}
|
||||
|
||||
Chunk* current_chunk;
|
||||
int prescan_index = 0;
|
||||
memset(&state->transfer_stats, 0, sizeof(state->transfer_stats));
|
||||
state->start = time(NULL);
|
||||
client_progress_begin(config);
|
||||
/* True when the stop deadline cut the scan short so the keep-set manifest is
|
||||
only a prefix of the source. */
|
||||
bool send_failed = false;
|
||||
while ((current_chunk = directory_scanner_next(state->scanner)) != NULL) {
|
||||
while (true) {
|
||||
if (state->prescan_chunks != NULL) {
|
||||
if (prescan_index >= state->prescan_chunks->size)
|
||||
break;
|
||||
/* Move ownership out of the retained list so chunk_destroy below (and the
|
||||
cleanup tail for an early exit) never double-frees it. */
|
||||
current_chunk = (Chunk*)state->prescan_chunks->items[prescan_index];
|
||||
state->prescan_chunks->items[prescan_index] = NULL;
|
||||
prescan_index++;
|
||||
} else {
|
||||
current_chunk = directory_scanner_next(state->scanner);
|
||||
if (current_chunk == NULL)
|
||||
break;
|
||||
}
|
||||
/* Graceful abort (Ctrl-C/SIGTERM): notify the receiver and clean up. The
|
||||
session is active (config_send already succeeded); a send failure here is
|
||||
fine because the client is exiting anyway. */
|
||||
@@ -1667,10 +1706,14 @@ static bool send_files_run(Config* config, SendFilesState* state) {
|
||||
* Returns the rsync-compatible exit code. */
|
||||
static int send_files_finalize(const Config* config, SendFilesState* state) {
|
||||
Client* client = state->client;
|
||||
if (directory_scanner_failed(state->scanner))
|
||||
return 1;
|
||||
if (directory_scanner_had_io_error(state->scanner))
|
||||
state->had_scan_io = true;
|
||||
/* A --delete-before run replays the pre-scan and owns no data scanner; its
|
||||
I/O-error verdict was already recorded by that pre-scan. */
|
||||
if (state->scanner != NULL) {
|
||||
if (directory_scanner_failed(state->scanner))
|
||||
return 1;
|
||||
if (directory_scanner_had_io_error(state->scanner))
|
||||
state->had_scan_io = true;
|
||||
}
|
||||
/* An abort that arrived after the last chunk must still stop the completion
|
||||
tail (manifest/finalize) rather than let it run to success. */
|
||||
if (client_abort_pending()) {
|
||||
@@ -1774,6 +1817,8 @@ static int send_files_finalize(const Config* config, SendFilesState* state) {
|
||||
static void send_files_cleanup(SendFilesState* state) {
|
||||
if (state->manifest)
|
||||
array_list_delete(state->manifest);
|
||||
if (state->prescan_chunks)
|
||||
array_list_delete(state->prescan_chunks);
|
||||
if (state->plan_sender)
|
||||
delete_plan_sender_destroy(state->plan_sender);
|
||||
if (state->excluded)
|
||||
@@ -1993,7 +2038,7 @@ int send_files_multithreaded(Config* config) {
|
||||
bool prebuilt =
|
||||
prepared_ok &&
|
||||
scan_paths_only(config, &prepared.options, context->manifest, context->delete_plans,
|
||||
&context->scan_had_io_error, &pre_scan_non_dir);
|
||||
&context->scan_had_io_error, &pre_scan_non_dir, NULL, false);
|
||||
prepared_scanner_destroy(&prepared);
|
||||
if (per_dir && prebuilt) {
|
||||
const char* walk_root = delete_plan_walk_root(config, context->synced_dirs);
|
||||
|
||||
@@ -44,7 +44,8 @@ const char* delete_plan_walk_root(const Config* config, const ArrayList* synced_
|
||||
bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* skipped_out);
|
||||
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
|
||||
DeletePlanSender* plans, bool* io_error_out,
|
||||
unsigned long long* non_dir_count_out);
|
||||
unsigned long long* non_dir_count_out, ArrayList* chunks_out,
|
||||
bool emit_nonreg);
|
||||
|
||||
/* client_report.c */
|
||||
void log_server_rejection(const char* context);
|
||||
|
||||
@@ -111,13 +111,20 @@ class _SlicingProxy:
|
||||
"""
|
||||
|
||||
def __init__(self, target_port, forward_limit=None, hook=None, hook_after=0,
|
||||
throttle=0.0, wait_for_reply=False):
|
||||
throttle=0.0, wait_for_reply=False, hook_after_config_ack=False):
|
||||
self.target = ("127.0.0.1", target_port)
|
||||
self.forward_limit = forward_limit
|
||||
self.hook = hook
|
||||
self.hook_after = hook_after
|
||||
self.throttle = throttle
|
||||
self.wait_for_reply = wait_for_reply
|
||||
# When set, the hook fires on the FIRST client->server bytes that follow
|
||||
# the config-frame ack, BEFORE they are forwarded. For --delete-before
|
||||
# those bytes are the keep-set manifest, so this runs the hook after the
|
||||
# client's source pre-scan but before the receiver's delete ack releases
|
||||
# the client into its data pass -- a deterministic late-file window.
|
||||
self.hook_after_config_ack = hook_after_config_ack
|
||||
self.config_acked = False
|
||||
self.server_replied = threading.Event()
|
||||
self.hook_called = threading.Event()
|
||||
self.listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
@@ -165,6 +172,13 @@ class _SlicingProxy:
|
||||
socks = []
|
||||
break
|
||||
data = data[:room]
|
||||
if (self.hook_after_config_ack and self.config_acked and self.hook is not None
|
||||
and not self.hook_called.is_set()):
|
||||
# The first client bytes after the config ack are the
|
||||
# pre-scan keep-set manifest: run the injection before
|
||||
# forwarding so it is causally after the source scan.
|
||||
self.hook()
|
||||
self.hook_called.set()
|
||||
backend.sendall(data)
|
||||
forwarded += len(data)
|
||||
self._maybe_hook(forwarded)
|
||||
@@ -177,6 +191,7 @@ class _SlicingProxy:
|
||||
client.sendall(data)
|
||||
# Any server reply proves the receiver consumed the
|
||||
# frames that precede it, so the hook barrier is met.
|
||||
self.config_acked = True
|
||||
self.server_replied.set()
|
||||
self._maybe_hook(forwarded)
|
||||
except OSError:
|
||||
@@ -198,8 +213,11 @@ class _SlicingProxy:
|
||||
def _maybe_hook(self, forwarded):
|
||||
"""Fire the one-shot hook once its barrier is satisfied: enough client
|
||||
bytes have been forwarded and, when ``wait_for_reply`` is set, the
|
||||
server has sent a reply proving it processed the preceding frames."""
|
||||
if self.hook is None or self.hook_called.is_set():
|
||||
server has sent a reply proving it processed the preceding frames.
|
||||
|
||||
``hook_after_config_ack`` uses its own barrier (see ``_serve``), so the
|
||||
byte/reply heuristic is bypassed entirely."""
|
||||
if self.hook is None or self.hook_called.is_set() or self.hook_after_config_ack:
|
||||
return
|
||||
if forwarded < self.hook_after:
|
||||
return
|
||||
@@ -537,3 +555,60 @@ class TestDeleteDelayMaxDeleteRefilledDir:
|
||||
assert os.path.isdir(later_dir), "later extra was not skipped by the budget"
|
||||
# The one actual removal is reported.
|
||||
assert _deleted_count(result.stdout) == 1, result.stdout
|
||||
|
||||
|
||||
class TestDeleteBeforeLateFileParity:
|
||||
"""rsync builds its file list once, so a source file created after that scan
|
||||
is NOT transferred and its destination extra is deleted. FastSync's
|
||||
single-threaded --delete-before used to re-scan the source in its data pass
|
||||
and would transfer the late file (a safe superset); it now replays the
|
||||
pre-scan file list instead, matching rsync.
|
||||
|
||||
The late file is injected through the config-ack barrier: the first client
|
||||
bytes after the config ack are the pre-scan keep-set manifest, so the hook
|
||||
runs causally after the source scan and before the receiver's delete ack
|
||||
releases the client into its data pass -- deterministic, no timing guess.
|
||||
"""
|
||||
|
||||
@requires_rsync
|
||||
def test_late_source_file_not_transferred_and_extra_deleted(self):
|
||||
source = os.path.join(TEST_DATA_DIR, "dblate_src")
|
||||
dest = os.path.join(TEST_DATA_DIR, "dblate_dst")
|
||||
rsync_dst = os.path.join(TEST_DATA_DIR, "dblate_rsync_dst")
|
||||
clean_dir(source)
|
||||
clean_dir(dest)
|
||||
clean_dir(rsync_dst)
|
||||
_write(os.path.join(source, "d", "keep.txt"), b"kept payload\n")
|
||||
# Both destinations carry the would-be late file as an extra.
|
||||
for root in (dest, rsync_dst):
|
||||
_write(os.path.join(get_dest_received_dir(root, source), "d", "late.txt"),
|
||||
b"stale extra\n")
|
||||
|
||||
# rsync reference: the same source with no late file; the extra is removed
|
||||
# and nothing is transferred for the (never-scanned) late path.
|
||||
rsync_result = _rsync(["-a", "--delete-before", source + "/", rsync_dst + "/"])
|
||||
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||
rsync_tree = _tree(rsync_dst)
|
||||
assert "d/late.txt" not in rsync_tree
|
||||
|
||||
received = get_dest_received_dir(dest, source)
|
||||
late_source = os.path.join(source, "d", "late.txt")
|
||||
|
||||
def hook():
|
||||
# Runs after the pre-scan and before the data pass begins.
|
||||
_write(late_source, b"created after the scan\n")
|
||||
|
||||
with ServerManager() as server:
|
||||
server.start(extra_args=["--allow-delete"])
|
||||
proxy = _SlicingProxy(server.port, hook=hook, hook_after_config_ack=True)
|
||||
result, _ = run_client(source, dest, flags=["--delete-before"], port=proxy.port)
|
||||
proxy.finish()
|
||||
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||
assert proxy.hook_called.is_set(), "late-file hook never fired"
|
||||
assert os.path.exists(late_source), "the source late file unexpectedly vanished"
|
||||
assert not os.path.exists(os.path.join(received, "d", "late.txt")), (
|
||||
"late source file was transferred: the single-threaded data pass re-scanned"
|
||||
)
|
||||
assert _tree(received) == rsync_tree, (
|
||||
f"fastsync tree {_tree(received)} != rsync tree {rsync_tree}"
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user