4 Commits
Author SHA1 Message Date
TapTap 741d1f3b93 test: review follow-up coverage for --fuzzy
CI / lint (pull_request) Failing after 37s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Unit (CLI): --fuzzy --no-incremental (either order) stays a valid plain-mode
config -- no forced handshake, no delta implication; --fuzzy --no-delta stays
covered.

Integration (TestFuzzy):
- worthless basis: a sibling that passes the name+size gates but shares no
  blocks makes the sender reply STATUS_NEXT; the whole file is consumed inside
  the delta handshake with byte-exact output (no protocol desync).
- non-displacement: an existing exact-path destination file INSIDE the delta
  size bounds (same size, different content, older mtime) is used as the delta
  basis instead of a byte-identical similar sibling (whole-file wire cost).
- asymmetric bases: basis larger than source (prefix reuse) and basis smaller
  than source (appended tail as literals) both reconstruct byte-exactly with a
  small delta.
- --no-fuzzy end-to-end equals the no-flag whole-file behavior.
CountingProxy closes its listener socket (fd hygiene).
2026-09-06 23:05:43 +02:00
TapTap c379e2dbdd docs: fuzzy oracle note and --no-incremental asymmetry
Review follow-ups on the --fuzzy row: note that the receiver's block
signature is derived from a sibling file it may not otherwise send, exposing
destination sibling files to the sender at block granularity (the same
known-plaintext information class as the ordinary delta path); and document
that --fuzzy honors an explicit --no-incremental (unlike the basis-dir
options, which force it), with the delta implication suppressed accordingly.
2026-09-06 23:05:39 +02:00
TapTap c1aabc68de feat(cli): --fuzzy honors an explicit --no-incremental
The --fuzzy implication previously forced use_incremental back on even when
the user passed --no-incremental, while --no-delta and -W were honored.
Track a --no-incremental latch (like the no_delta latch): with it set, do not
force the handshake on, and because delta needs the handshake, also suppress
the delta implication so no invalid '--delta requires --incremental' config
results.  A --fuzzy --no-incremental run is therefore a plain default-mode
transfer (fuzzy inert), consistent with -W/--no-delta.  Documented in the
usage text (this deliberately differs from the basis-dir options, which still
force incremental unconditionally).
2026-09-06 23:05:35 +02:00
TapTap be37a509a5 perf(receiver): bound the fuzzy edit-distance cost, harden the open
Review follow-ups on the --fuzzy candidate scan:
- Allocate the two DP rows once per directory scan instead of once per
  candidate (4096-entry directories no longer do thousands of malloc pairs).
- Pre-prune before the DP with two cheap lower bounds on the edit distance:
  the name length gap and the count of characters of one basename absent from
  the other; a candidate whose gate (distance*2 <= longer) already fails on
  the max of those bounds is skipped without running the DP.
- Trim the common prefix and non-overlapping common suffix before the DP so
  it only runs over the differing middles.
- Note that the 4096 readdir cap bounds iterations, not per-entry DP cost,
  and that the seen set is filesystem-order dependent (winner stays
  deterministic via the total comparator).
- Open the chosen candidate with O_NONBLOCK so a name raced to a FIFO cannot
  block the receive thread forever in open(2); the existing fstat S_ISREG gate
  still rejects non-regular files.  (The pre-existing basis_open_regular has
  the same latent FIFO pattern and is intentionally left unchanged.)
- Free old_data in receive_delta_file's defensive NULL guard.
2026-09-06 23:05:32 +02:00
7 changed files with 281 additions and 33 deletions

No files matched your search

+1 -1
View File
@@ -213,7 +213,7 @@ order-independent because it runs over the fully parsed config.
| `--compare-dest=DIR` | Compare dest files relative to DIR | ✅ Implemented | DIR is a receiver-side basis relative to the destination root (confined below it; absolute/`..`/`.` rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Divergences: when the destination already holds a *different* version rsync deletes it but FastSync instead transfers the data (keeps the mirror complete; never deletes without `--delete`); attribute-only differences on a match are not re-applied (data is skipped so the sender never sends metadata); content is verified by xxHash64, stricter than rsync's default quick check. Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) |
| `--copy-dest=DIR` | Include copies of unchanged files | ✅ Implemented | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. Divergences: a basis-hit destination keeps the basis file's own mode/uid/gid and mtime (the sender sends no metadata on a skip), so with `--size-only` its mtime can differ from the source and attribute-only differences are copied with the basis attributes rather than rsync's "copy + fix attributes". Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `--link-dest=DIR` | Hardlink to files when unchanged | ✅ Implemented | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. Divergences and caveats: an already up-to-date destination file is not re-linked to a basis file (only files that would otherwise be written are linked); a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); with `--size-only` the linked mtime can differ from the source; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ✅ Implemented | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (deterministic, simpler than rsync's deliberately-fuzzy matching, and documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×) rather than rsync's ~1.5× size window; the name gate is a Levenshtein edit distance between the basenames accepted only when ≤ half the length of the longer basename; the single best candidate (smallest distance, tie-break size closest to the incoming file then lexicographically smaller basename) is read; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: rsync's own matching uses a fuzzy name/size rule set; FastSync implements the closest safe deterministic approximation above. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant); `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file) |
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ✅ Implemented | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (deterministic, simpler than rsync's deliberately-fuzzy matching, and documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×) rather than rsync's ~1.5× size window; the name gate is a Levenshtein edit distance between the basenames accepted only when ≤ half the length of the longer basename; the single best candidate (smallest distance, tie-break size closest to the incoming file then lexicographically smaller basename) is read; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: rsync's own matching uses a fuzzy name/size rule set; FastSync implements the closest safe deterministic approximation above. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file) |
## 12. Compression
+17 -8
View File
@@ -610,9 +610,11 @@ static int apply_table_option(Config* config, const OptionEntry* entry, const ch
int parse_args(Config* config, int argc, char* argv[], int* positional_args,
int* positional_count) {
bool verbose = false;
/* --no-delta seen on the command line: the user explicitly switched the
delta machinery off, so the --fuzzy implication must not override it. */
/* Explicit --no-delta / --no-incremental seen on the command line: the user
switched part of the delta machinery off, so the --fuzzy implication must
not silently turn it back on. */
bool no_delta = false;
bool no_incremental = false;
protocol_set_8_bit_output(config->eight_bit_output);
/* Apply output controls before processing other options so their order is irrelevant. */
@@ -644,6 +646,8 @@ int parse_args(Config* config, int argc, char* argv[], int* positional_args,
if (strncmp(argv[i], "--no-", strlen("--no-")) == 0) {
if (strcmp(argv[i], "--no-delta") == 0)
no_delta = true;
else if (strcmp(argv[i], "--no-incremental") == 0)
no_incremental = true;
if (apply_negation(config, argv[i]) != 0)
return -1;
continue;
@@ -1077,14 +1081,19 @@ int parse_args(Config* config, int argc, char* argv[], int* positional_args,
/* -y/--fuzzy reuses an existing similar-named destination file as the delta
* basis, so it is meaningless without the receiver-driven delta path:
* imply --incremental, and --delta unless --whole-file (or an explicit
* --no-delta) switched the delta machinery off. FastSync has delta OFF by
* default (unlike rsync), so a bare --fuzzy must turn it on or it would be
* a silent no-op. --whole-file/--no-delta after --fuzzy therefore leave
* fuzzy inert, matching rsync where --fuzzy only affects delta transfers. */
* imply --incremental and --delta unless --whole-file or an explicit
* --no-delta / --no-incremental switched the machinery off. FastSync has
* delta OFF by default (unlike rsync), so a bare --fuzzy must turn it on or
* it would be a silent no-op. -W/--no-delta/--no-incremental therefore
* leave fuzzy inert, matching rsync where --whole-file makes fuzzy
* irrelevant (note: unlike the basis-dir options, --fuzzy honors an
* explicit --no-incremental instead of forcing the handshake back on). */
if (config->fuzzy) {
if (!no_incremental)
config->use_incremental = true;
if (!config->whole_file && !no_delta)
/* Delta needs the incremental per-file handshake, so an explicit
* --no-incremental also suppresses the delta implication. */
if (!config->whole_file && !no_delta && !no_incremental)
config->use_delta = true;
}
+2 -1
View File
@@ -84,7 +84,8 @@ void print_usage(void) {
printf(" -y, --fuzzy Use a similar-named file already in the destination\n");
printf(" directory as the delta basis when the destination has no\n");
printf(" usable file at the exact path (saves bandwidth; implies\n");
printf(" --incremental and --delta; inert with --whole-file)\n");
printf(" --incremental and --delta; inert with --whole-file,\n");
printf(" --no-delta, or --no-incremental)\n");
printf(" --no-fuzzy Disable --fuzzy\n");
printf(" --delta-block <n> Delta block size in bytes (default: %d)\n",
DELTA_BLOCK_SIZE_DEFAULT);
+121 -23
View File
@@ -336,6 +336,7 @@ fail:
static File* receive_delta_file(int fd, const Config* config, const char* check_path,
void* old_data, unsigned long long old_size, bool* failed) {
if (!old_data) {
free(old_data); /* defensive: old_data is always non-NULL today */
*failed = true;
return NULL;
}
@@ -699,7 +700,12 @@ static bool basis_match_find(const Config* config, const char* check_path,
* ------------------------------------------------------------------------- */
/* A directory scan is linear in the number of entries; the fuzzy search stops
* after this many so a pathological huge directory cannot stall a transfer. */
* after this many so a pathological huge directory cannot stall a transfer.
* The cap bounds the readdir() ITERATIONS, not the per-entry work: every
* entry that survives the (cheap) size and pre-name gates still runs an
* edit-distance DP, so the per-entry DP cost is separately bounded below by
* pre-pruning on the name length gap and the absent-character bound, and by
* trimming the common prefix/suffix before the DP runs on the middles only. */
#define FUZZY_MAX_DIRECTORY_SCAN 4096
/* Names longer than this never take part in fuzzy matching: the edit-distance
* DP below is O(len^2), so over-long names are bounded out of the search. */
@@ -712,24 +718,80 @@ typedef struct {
unsigned long long size_gap;
} FuzzyCandidate;
/* Levenshtein edit distance, or SIZE_MAX when the operands are too long or the
* DP could not be allocated. */
static size_t fuzzy_edit_distance(const char* a, size_t la, const char* b, size_t lb) {
if (la > FUZZY_NAME_LIMIT || lb > FUZZY_NAME_LIMIT)
return SIZE_MAX;
size_t* prev = malloc((lb + 1) * sizeof(size_t));
size_t* cur = malloc((lb + 1) * sizeof(size_t));
if (!prev || !cur) {
free(prev);
free(cur);
return SIZE_MAX;
/* Two-row DP scratch, allocated once per directory scan (not per candidate) so
* a 4096-entry directory never performs 4096 malloc/free pairs. */
typedef struct {
size_t* prev;
size_t* cur;
} FuzzyEditBuffer;
static bool fuzzy_edit_buffer_init(FuzzyEditBuffer* buf) {
buf->prev = malloc((FUZZY_NAME_LIMIT + 1) * sizeof(size_t));
buf->cur = malloc((FUZZY_NAME_LIMIT + 1) * sizeof(size_t));
if (!buf->prev || !buf->cur) {
free(buf->prev);
free(buf->cur);
buf->prev = NULL;
buf->cur = NULL;
return false;
}
for (size_t j = 0; j <= lb; j++)
return true;
}
static void fuzzy_edit_buffer_destroy(FuzzyEditBuffer* buf) {
free(buf->prev);
free(buf->cur);
buf->prev = NULL;
buf->cur = NULL;
}
/* Cheap lower bounds used to reject a candidate BEFORE the DP:
* - any edit script must at least absorb the length gap: d >= |la - lb|;
* - any character of `a` that does not occur in `b` at all must be deleted or
* substituted at its own position: d >= (count of such characters).
* The acceptance gate is d*2 <= longer, so a candidate whose max of these two
* bounds already violates it can be skipped without computing the distance. */
static size_t fuzzy_absent_char_bound(const char* a, size_t la, const char* b, size_t lb) {
if (lb == 0)
return la;
bool present[256] = {false};
for (size_t i = 0; i < lb; i++)
present[(uint8_t)b[i]] = true;
size_t absent = 0;
for (size_t i = 0; i < la; i++)
if (!present[(uint8_t)a[i]])
absent++;
return absent;
}
/* Levenshtein edit distance between the two basenames. A shared prefix and a
* (non-overlapping) shared suffix can always be aligned at no cost, so the DP
* only runs over the differing middles; its two rows come from `buf` (allocated
* once by the caller). Callers enforce la, lb <= FUZZY_NAME_LIMIT. */
static size_t fuzzy_edit_distance(FuzzyEditBuffer* buf, const char* a, size_t la, const char* b,
size_t lb) {
size_t p = 0;
while (p < la && p < lb && a[p] == b[p])
p++;
size_t s = 0;
while (s < la - p && s < lb - p && a[la - 1 - s] == b[lb - 1 - s])
s++;
size_t ma = la - p - s;
size_t mb = lb - p - s;
if (ma == 0)
return mb;
if (mb == 0)
return ma;
const char* A = a + p;
const char* B = b + p;
size_t* prev = buf->prev;
size_t* cur = buf->cur;
for (size_t j = 0; j <= mb; j++)
prev[j] = j;
for (size_t i = 1; i <= la; i++) {
for (size_t i = 1; i <= ma; i++) {
cur[0] = i;
for (size_t j = 1; j <= lb; j++) {
size_t cost = a[i - 1] == b[j - 1] ? 0 : 1;
for (size_t j = 1; j <= mb; j++) {
size_t cost = A[i - 1] == B[j - 1] ? 0 : 1;
size_t del = prev[j] + 1;
size_t ins = cur[j - 1] + 1;
size_t sub = prev[j - 1] + cost;
@@ -740,10 +802,7 @@ static size_t fuzzy_edit_distance(const char* a, size_t la, const char* b, size_
prev = cur;
cur = tmp;
}
size_t distance = prev[lb];
free(prev);
free(cur);
return distance;
return prev[mb];
}
/* Deterministic ordering of two fuzzy candidates: smallest edit distance,
@@ -783,6 +842,14 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
return NULL;
}
size_t target_len = strlen(leaf);
/* A target basename longer than FUZZY_NAME_LIMIT can never pass the name gate
(every candidate name is bounded by the same limit), so skip the scan. */
if (target_len > FUZZY_NAME_LIMIT) {
close(dir_fd);
free(leaf);
free(full_path);
return NULL;
}
int scanfd = dup(dir_fd);
if (scanfd < 0) {
@@ -800,10 +867,24 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
return NULL;
}
/* The DP scratch rows are allocated once per scan (not once per candidate). */
FuzzyEditBuffer ebuf;
if (!fuzzy_edit_buffer_init(&ebuf)) {
closedir(dir);
close(dir_fd);
free(leaf);
free(full_path);
return NULL;
}
FuzzyCandidate best;
memset(&best, 0, sizeof(best));
const struct dirent* entry;
size_t scanned = 0;
/* readdir() yields entries in filesystem-dependent order, so the SET of
candidates seen is order-dependent; the winner is still deterministic
because every candidate is compared with the total ordering in
fuzzy_candidate_better (acceptable for a heuristic). */
while (scanned < FUZZY_MAX_DIRECTORY_SCAN && (entry = readdir(dir)) != NULL) {
scanned++;
const char* name = entry->d_name;
@@ -817,9 +898,21 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
if (cand_size == 0 || cand_size > MAX_RECEIVE_WHOLE_FILE_SIZE ||
!delta_should_attempt(cand_size, check_size, config->delta_max_file_size))
continue;
size_t distance = fuzzy_edit_distance(leaf, target_len, name, name_len);
/* Cheap pre-name gates run BEFORE the edit-distance DP. The edit distance
is bounded below by the length gap |la-lb| and by the number of
characters of one basename that are absent from the other (each such
position costs at least one op), so a candidate whose acceptance gate
(distance*2 <= longer) already fails on the max of those bounds is
skipped without running the DP. */
size_t longer = target_len > name_len ? target_len : name_len;
if (distance == SIZE_MAX || distance * 2 > longer)
size_t bound = longer - (target_len < name_len ? target_len : name_len);
size_t absent = fuzzy_absent_char_bound(leaf, target_len, name, name_len);
if (absent > bound)
bound = absent;
if (bound * 2 > longer)
continue;
size_t distance = fuzzy_edit_distance(&ebuf, leaf, target_len, name, name_len);
if (distance * 2 > longer)
continue;
FuzzyCandidate cand;
memcpy(cand.name, name, name_len + 1);
@@ -831,10 +924,15 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
}
closedir(dir);
free(leaf);
fuzzy_edit_buffer_destroy(&ebuf);
void* basis = NULL;
if (best.name[0]) {
int fd = openat(dir_fd, best.name, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
/* O_NONBLOCK: a name raced to a FIFO between the fstatat gate and this open
would otherwise block the receive thread forever on open(2); with it the
open fails (ENXIO) and the fstat/S_ISREG gate below would reject it too.
A regular file opened with O_NONBLOCK is unaffected. */
int fd = openat(dir_fd, best.name, O_RDONLY | O_NOFOLLOW | O_NONBLOCK | O_CLOEXEC);
if (fd >= 0) {
struct stat st;
if (fstat(fd, &st) == 0 && S_ISREG(st.st_mode) &&
+2
View File
@@ -106,6 +106,7 @@ class CountingProxy:
server_sock = socket.create_connection(("127.0.0.1", self.target_port),
timeout=10)
except OSError:
self._listener.close()
return
c2s, s2c = [0], [0]
a = threading.Thread(target=self._pump, args=(client_sock, server_sock, c2s))
@@ -116,6 +117,7 @@ class CountingProxy:
b.join()
self.client_to_server = c2s[0]
self.server_to_client = s2c[0]
self._listener.close()
thread = threading.Thread(target=serve)
thread.start()
+111
View File
@@ -2513,6 +2513,8 @@ class TestFuzzy:
OLD_NAME = "report-2025.dat"
NEW_NAME = "report-2026.dat"
TS = 1577836800 # 2020-01-01, used to pin stale destination mtimes
SIZE = 2 * 1024 * 1024
def _client_via_proxy(self, source, dest, flags, proxy):
cmd = (CLIENT_CMD + ["--source-dir", source, "--dest-dir", dest,
@@ -2721,3 +2723,112 @@ class TestFuzzy:
assert _read_file(os.path.join(received, self.NEW_NAME)) == new_bytes
assert _read_file(os.path.join(received, self.OLD_NAME)) == old_bytes
@staticmethod
def _rand_bytes(size, seed):
return random.Random(seed).randbytes(size)
def _replace_source_file(self, source, old_name, new_name, new_bytes):
"""Remove old_name from source and add new_name with new_bytes."""
os.unlink(os.path.join(source, old_name))
with open(os.path.join(source, new_name), "wb") as fh:
fh.write(new_bytes)
def test_worthless_fuzzy_basis_falls_back_inside_handshake(self, shared_server):
# The sibling passes the name AND size gates but shares no blocks with
# the incoming file, so the sender's delta is not worthwhile: it replies
# STATUS_NEXT and the receiver consumes the WHOLE file inside the delta
# handshake. This proves a bad fuzzy basis cannot desync the protocol
# or corrupt the result.
source, dest = self._prepare("worthless")
basis = self._rand_bytes(self.SIZE, 424242)
target = self._rand_bytes(self.SIZE, 777777)
self._seed_dest(source, dest, {self.OLD_NAME: basis}, shared_server.port)
self._replace_source_file(source, self.OLD_NAME, self.NEW_NAME, target)
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
assert result.returncode == 0, \
f"--fuzzy worthless-basis run failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, self.NEW_NAME)) == target, \
"whole-file fallback after a worthless fuzzy basis is not byte-exact"
assert proxy.client_to_server > self.SIZE // 2, \
"a worthless basis should have made the sender fall back to the whole file"
def test_existing_dest_file_preferred_over_fuzzy_sibling(self, shared_server):
# Non-displacement: the destination holds a file at the exact path that
# is inside the delta size bounds (same size, different content, older
# mtime). FastSync must delta against THAT file -- even though it
# shares nothing with the source -- and must NOT reuse a similar-named
# sibling that is byte-identical to the source.
source, dest = self._prepare("nondisp")
sibling = self._rand_bytes(self.SIZE, 111) # will equal the incoming file
stale = self._rand_bytes(self.SIZE, 333) # worthless exact-path file
self._seed_dest(source, dest,
{self.OLD_NAME: sibling, self.NEW_NAME: stale},
shared_server.port)
# Force the exact-path destination file's mtime into the past so the
# quick check deterministically decides to transfer it.
os.utime(os.path.join(get_dest_received_dir(dest, source), self.NEW_NAME),
(self.TS, self.TS))
os.unlink(os.path.join(source, self.OLD_NAME))
with open(os.path.join(source, self.NEW_NAME), "wb") as fh:
fh.write(sibling)
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
assert result.returncode == 0, \
f"--fuzzy non-displacement run failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, self.NEW_NAME)) == sibling
assert proxy.client_to_server > self.SIZE // 2, \
"the exact-path destination file must be the delta basis, not the fuzzy sibling"
def test_fuzzy_basis_larger_than_source(self, shared_server):
# The similar sibling is LARGER than the incoming file (within the delta
# engine's 10x ratio); the new file is an exact prefix of the basis, so
# every block matches and only a tiny delta travels.
source, dest = self._prepare("largerbasis")
big = self._rand_bytes(1536 * 1024, 1)
prefix = big[:1024 * 1024]
self._seed_dest(source, dest, {self.OLD_NAME: big}, shared_server.port)
self._replace_source_file(source, self.OLD_NAME, self.NEW_NAME, prefix)
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
assert result.returncode == 0, \
f"--fuzzy larger-basis run failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, self.NEW_NAME)) == prefix, \
"shrunken file reconstructed from a larger fuzzy basis is not byte-exact"
assert proxy.client_to_server < len(prefix) // 4, \
"larger fuzzy basis should have carried most of the file as block matches"
def test_fuzzy_basis_smaller_than_source(self, shared_server):
# The similar sibling is SMALLER than the incoming file; the new file
# appends data past the basis, so the appended tail travels as literals
# while the shared prefix is block-matched.
source, dest = self._prepare("smallerbasis")
base = self._rand_bytes(self.SIZE, 2)
tail = self._rand_bytes(64 * 1024, 3)
new_bytes = base + tail
self._seed_dest(source, dest, {self.OLD_NAME: base}, shared_server.port)
self._replace_source_file(source, self.OLD_NAME, self.NEW_NAME, new_bytes)
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
assert result.returncode == 0, \
f"--fuzzy smaller-basis run failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, self.NEW_NAME)) == new_bytes, \
"grown file reconstructed from a smaller fuzzy basis is not byte-exact"
assert proxy.client_to_server < len(new_bytes) // 4, \
"smaller fuzzy basis should have block-matched the shared prefix"
def test_no_fuzzy_end_to_end_equals_no_flag(self, shared_server):
# --no-fuzzy must not enable anything: a run with it behaves exactly
# like a run without it (whole-file transfer, byte-exact output).
source, dest = self._prepare("nofuzzye2e")
old_bytes, new_bytes = _random_payloads()
self._seed_dest(source, dest, {self.OLD_NAME: old_bytes}, shared_server.port)
self._replace_source_file(source, self.OLD_NAME, self.NEW_NAME, new_bytes)
result, proxy = self._run_measured(source, dest, ["--no-fuzzy"], shared_server.port)
assert result.returncode == 0, \
f"--no-fuzzy run failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, self.NEW_NAME)) == new_bytes
assert proxy.client_to_server > len(new_bytes) // 2, \
"--no-fuzzy should leave the default whole-file behavior intact"
+27
View File
@@ -1246,6 +1246,32 @@ static void test_parse_args_fuzzy_respects_no_delta() {
}
}
/* An explicit --no-incremental is respected by the --fuzzy implication in
* either argument order (unlike the basis-dir options, --fuzzy does not force
* the incremental handshake back on). Because delta needs the handshake, the
* delta implication is suppressed too, so the run is a plain (default-mode)
* transfer rather than an invalid "--delta requires --incremental" config. */
static void test_parse_args_fuzzy_respects_no_incremental() {
static const char* const combos[][2] = {
{"--fuzzy", "--no-incremental"},
{"--no-incremental", "--fuzzy"},
};
for (size_t i = 0; i < sizeof(combos) / sizeof(combos[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)combos[i][0], (char*)combos[i][1], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->fuzzy);
EXPECT_FALSE(cfg->use_incremental);
EXPECT_FALSE(cfg->use_delta);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg);
}
}
/* --fuzzy requires the delta machinery, which the chunk-serialization (-s) and
* sendfile (-f) modes reject -- mirroring the --delta constraint checks. */
static void test_validate_config_fuzzy_incompatible_modes() {
@@ -1831,6 +1857,7 @@ void test_client_cli() {
test_parse_args_fuzzy_negation();
test_parse_args_fuzzy_with_whole_file();
test_parse_args_fuzzy_respects_no_delta();
test_parse_args_fuzzy_respects_no_incremental();
test_validate_config_fuzzy_incompatible_modes();
test_parse_args_one_file_system();
test_parse_args_compression_aliases();