Compare commits
4
Commits
2bc43d6084
...
741d1f3b93
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
741d1f3b93 | ||
|
|
c379e2dbdd | ||
|
|
c1aabc68de | ||
|
|
be37a509a5 |
No files matched your search
+1
-1
@@ -213,7 +213,7 @@ order-independent because it runs over the fully parsed config.
|
||||
| `--compare-dest=DIR` | Compare dest files relative to DIR | ✅ Implemented | DIR is a receiver-side basis relative to the destination root (confined below it; absolute/`..`/`.` rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Divergences: when the destination already holds a *different* version rsync deletes it but FastSync instead transfers the data (keeps the mirror complete; never deletes without `--delete`); attribute-only differences on a match are not re-applied (data is skipped so the sender never sends metadata); content is verified by xxHash64, stricter than rsync's default quick check. Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) |
|
||||
| `--copy-dest=DIR` | Include copies of unchanged files | ✅ Implemented | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. Divergences: a basis-hit destination keeps the basis file's own mode/uid/gid and mtime (the sender sends no metadata on a skip), so with `--size-only` its mtime can differ from the source and attribute-only differences are copied with the basis attributes rather than rsync's "copy + fix attributes". Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
||||
| `--link-dest=DIR` | Hardlink to files when unchanged | ✅ Implemented | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. Divergences and caveats: an already up-to-date destination file is not re-linked to a basis file (only files that would otherwise be written are linked); a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); with `--size-only` the linked mtime can differ from the source; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
||||
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ✅ Implemented | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (deterministic, simpler than rsync's deliberately-fuzzy matching, and documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×) rather than rsync's ~1.5× size window; the name gate is a Levenshtein edit distance between the basenames accepted only when ≤ half the length of the longer basename; the single best candidate (smallest distance, tie-break size closest to the incoming file then lexicographically smaller basename) is read; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: rsync's own matching uses a fuzzy name/size rule set; FastSync implements the closest safe deterministic approximation above. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant); `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file) |
|
||||
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ✅ Implemented | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (deterministic, simpler than rsync's deliberately-fuzzy matching, and documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×) rather than rsync's ~1.5× size window; the name gate is a Levenshtein edit distance between the basenames accepted only when ≤ half the length of the longer basename; the single best candidate (smallest distance, tie-break size closest to the incoming file then lexicographically smaller basename) is read; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: rsync's own matching uses a fuzzy name/size rule set; FastSync implements the closest safe deterministic approximation above. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file) |
|
||||
|
||||
## 12. Compression
|
||||
|
||||
|
||||
+17
-8
@@ -610,9 +610,11 @@ static int apply_table_option(Config* config, const OptionEntry* entry, const ch
|
||||
int parse_args(Config* config, int argc, char* argv[], int* positional_args,
|
||||
int* positional_count) {
|
||||
bool verbose = false;
|
||||
/* --no-delta seen on the command line: the user explicitly switched the
|
||||
delta machinery off, so the --fuzzy implication must not override it. */
|
||||
/* Explicit --no-delta / --no-incremental seen on the command line: the user
|
||||
switched part of the delta machinery off, so the --fuzzy implication must
|
||||
not silently turn it back on. */
|
||||
bool no_delta = false;
|
||||
bool no_incremental = false;
|
||||
protocol_set_8_bit_output(config->eight_bit_output);
|
||||
|
||||
/* Apply output controls before processing other options so their order is irrelevant. */
|
||||
@@ -644,6 +646,8 @@ int parse_args(Config* config, int argc, char* argv[], int* positional_args,
|
||||
if (strncmp(argv[i], "--no-", strlen("--no-")) == 0) {
|
||||
if (strcmp(argv[i], "--no-delta") == 0)
|
||||
no_delta = true;
|
||||
else if (strcmp(argv[i], "--no-incremental") == 0)
|
||||
no_incremental = true;
|
||||
if (apply_negation(config, argv[i]) != 0)
|
||||
return -1;
|
||||
continue;
|
||||
@@ -1077,14 +1081,19 @@ int parse_args(Config* config, int argc, char* argv[], int* positional_args,
|
||||
|
||||
/* -y/--fuzzy reuses an existing similar-named destination file as the delta
|
||||
* basis, so it is meaningless without the receiver-driven delta path:
|
||||
* imply --incremental, and --delta unless --whole-file (or an explicit
|
||||
* --no-delta) switched the delta machinery off. FastSync has delta OFF by
|
||||
* default (unlike rsync), so a bare --fuzzy must turn it on or it would be
|
||||
* a silent no-op. --whole-file/--no-delta after --fuzzy therefore leave
|
||||
* fuzzy inert, matching rsync where --fuzzy only affects delta transfers. */
|
||||
* imply --incremental and --delta unless --whole-file or an explicit
|
||||
* --no-delta / --no-incremental switched the machinery off. FastSync has
|
||||
* delta OFF by default (unlike rsync), so a bare --fuzzy must turn it on or
|
||||
* it would be a silent no-op. -W/--no-delta/--no-incremental therefore
|
||||
* leave fuzzy inert, matching rsync where --whole-file makes fuzzy
|
||||
* irrelevant (note: unlike the basis-dir options, --fuzzy honors an
|
||||
* explicit --no-incremental instead of forcing the handshake back on). */
|
||||
if (config->fuzzy) {
|
||||
if (!no_incremental)
|
||||
config->use_incremental = true;
|
||||
if (!config->whole_file && !no_delta)
|
||||
/* Delta needs the incremental per-file handshake, so an explicit
|
||||
* --no-incremental also suppresses the delta implication. */
|
||||
if (!config->whole_file && !no_delta && !no_incremental)
|
||||
config->use_delta = true;
|
||||
}
|
||||
|
||||
|
||||
+2
-1
@@ -84,7 +84,8 @@ void print_usage(void) {
|
||||
printf(" -y, --fuzzy Use a similar-named file already in the destination\n");
|
||||
printf(" directory as the delta basis when the destination has no\n");
|
||||
printf(" usable file at the exact path (saves bandwidth; implies\n");
|
||||
printf(" --incremental and --delta; inert with --whole-file)\n");
|
||||
printf(" --incremental and --delta; inert with --whole-file,\n");
|
||||
printf(" --no-delta, or --no-incremental)\n");
|
||||
printf(" --no-fuzzy Disable --fuzzy\n");
|
||||
printf(" --delta-block <n> Delta block size in bytes (default: %d)\n",
|
||||
DELTA_BLOCK_SIZE_DEFAULT);
|
||||
|
||||
+121
-23
@@ -336,6 +336,7 @@ fail:
|
||||
static File* receive_delta_file(int fd, const Config* config, const char* check_path,
|
||||
void* old_data, unsigned long long old_size, bool* failed) {
|
||||
if (!old_data) {
|
||||
free(old_data); /* defensive: old_data is always non-NULL today */
|
||||
*failed = true;
|
||||
return NULL;
|
||||
}
|
||||
@@ -699,7 +700,12 @@ static bool basis_match_find(const Config* config, const char* check_path,
|
||||
* ------------------------------------------------------------------------- */
|
||||
|
||||
/* A directory scan is linear in the number of entries; the fuzzy search stops
|
||||
* after this many so a pathological huge directory cannot stall a transfer. */
|
||||
* after this many so a pathological huge directory cannot stall a transfer.
|
||||
* The cap bounds the readdir() ITERATIONS, not the per-entry work: every
|
||||
* entry that survives the (cheap) size and pre-name gates still runs an
|
||||
* edit-distance DP, so the per-entry DP cost is separately bounded below by
|
||||
* pre-pruning on the name length gap and the absent-character bound, and by
|
||||
* trimming the common prefix/suffix before the DP runs on the middles only. */
|
||||
#define FUZZY_MAX_DIRECTORY_SCAN 4096
|
||||
/* Names longer than this never take part in fuzzy matching: the edit-distance
|
||||
* DP below is O(len^2), so over-long names are bounded out of the search. */
|
||||
@@ -712,24 +718,80 @@ typedef struct {
|
||||
unsigned long long size_gap;
|
||||
} FuzzyCandidate;
|
||||
|
||||
/* Levenshtein edit distance, or SIZE_MAX when the operands are too long or the
|
||||
* DP could not be allocated. */
|
||||
static size_t fuzzy_edit_distance(const char* a, size_t la, const char* b, size_t lb) {
|
||||
if (la > FUZZY_NAME_LIMIT || lb > FUZZY_NAME_LIMIT)
|
||||
return SIZE_MAX;
|
||||
size_t* prev = malloc((lb + 1) * sizeof(size_t));
|
||||
size_t* cur = malloc((lb + 1) * sizeof(size_t));
|
||||
if (!prev || !cur) {
|
||||
free(prev);
|
||||
free(cur);
|
||||
return SIZE_MAX;
|
||||
/* Two-row DP scratch, allocated once per directory scan (not per candidate) so
|
||||
* a 4096-entry directory never performs 4096 malloc/free pairs. */
|
||||
typedef struct {
|
||||
size_t* prev;
|
||||
size_t* cur;
|
||||
} FuzzyEditBuffer;
|
||||
|
||||
static bool fuzzy_edit_buffer_init(FuzzyEditBuffer* buf) {
|
||||
buf->prev = malloc((FUZZY_NAME_LIMIT + 1) * sizeof(size_t));
|
||||
buf->cur = malloc((FUZZY_NAME_LIMIT + 1) * sizeof(size_t));
|
||||
if (!buf->prev || !buf->cur) {
|
||||
free(buf->prev);
|
||||
free(buf->cur);
|
||||
buf->prev = NULL;
|
||||
buf->cur = NULL;
|
||||
return false;
|
||||
}
|
||||
for (size_t j = 0; j <= lb; j++)
|
||||
return true;
|
||||
}
|
||||
|
||||
static void fuzzy_edit_buffer_destroy(FuzzyEditBuffer* buf) {
|
||||
free(buf->prev);
|
||||
free(buf->cur);
|
||||
buf->prev = NULL;
|
||||
buf->cur = NULL;
|
||||
}
|
||||
|
||||
/* Cheap lower bounds used to reject a candidate BEFORE the DP:
|
||||
* - any edit script must at least absorb the length gap: d >= |la - lb|;
|
||||
* - any character of `a` that does not occur in `b` at all must be deleted or
|
||||
* substituted at its own position: d >= (count of such characters).
|
||||
* The acceptance gate is d*2 <= longer, so a candidate whose max of these two
|
||||
* bounds already violates it can be skipped without computing the distance. */
|
||||
static size_t fuzzy_absent_char_bound(const char* a, size_t la, const char* b, size_t lb) {
|
||||
if (lb == 0)
|
||||
return la;
|
||||
bool present[256] = {false};
|
||||
for (size_t i = 0; i < lb; i++)
|
||||
present[(uint8_t)b[i]] = true;
|
||||
size_t absent = 0;
|
||||
for (size_t i = 0; i < la; i++)
|
||||
if (!present[(uint8_t)a[i]])
|
||||
absent++;
|
||||
return absent;
|
||||
}
|
||||
|
||||
/* Levenshtein edit distance between the two basenames. A shared prefix and a
|
||||
* (non-overlapping) shared suffix can always be aligned at no cost, so the DP
|
||||
* only runs over the differing middles; its two rows come from `buf` (allocated
|
||||
* once by the caller). Callers enforce la, lb <= FUZZY_NAME_LIMIT. */
|
||||
static size_t fuzzy_edit_distance(FuzzyEditBuffer* buf, const char* a, size_t la, const char* b,
|
||||
size_t lb) {
|
||||
size_t p = 0;
|
||||
while (p < la && p < lb && a[p] == b[p])
|
||||
p++;
|
||||
size_t s = 0;
|
||||
while (s < la - p && s < lb - p && a[la - 1 - s] == b[lb - 1 - s])
|
||||
s++;
|
||||
size_t ma = la - p - s;
|
||||
size_t mb = lb - p - s;
|
||||
if (ma == 0)
|
||||
return mb;
|
||||
if (mb == 0)
|
||||
return ma;
|
||||
const char* A = a + p;
|
||||
const char* B = b + p;
|
||||
size_t* prev = buf->prev;
|
||||
size_t* cur = buf->cur;
|
||||
for (size_t j = 0; j <= mb; j++)
|
||||
prev[j] = j;
|
||||
for (size_t i = 1; i <= la; i++) {
|
||||
for (size_t i = 1; i <= ma; i++) {
|
||||
cur[0] = i;
|
||||
for (size_t j = 1; j <= lb; j++) {
|
||||
size_t cost = a[i - 1] == b[j - 1] ? 0 : 1;
|
||||
for (size_t j = 1; j <= mb; j++) {
|
||||
size_t cost = A[i - 1] == B[j - 1] ? 0 : 1;
|
||||
size_t del = prev[j] + 1;
|
||||
size_t ins = cur[j - 1] + 1;
|
||||
size_t sub = prev[j - 1] + cost;
|
||||
@@ -740,10 +802,7 @@ static size_t fuzzy_edit_distance(const char* a, size_t la, const char* b, size_
|
||||
prev = cur;
|
||||
cur = tmp;
|
||||
}
|
||||
size_t distance = prev[lb];
|
||||
free(prev);
|
||||
free(cur);
|
||||
return distance;
|
||||
return prev[mb];
|
||||
}
|
||||
|
||||
/* Deterministic ordering of two fuzzy candidates: smallest edit distance,
|
||||
@@ -783,6 +842,14 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
||||
return NULL;
|
||||
}
|
||||
size_t target_len = strlen(leaf);
|
||||
/* A target basename longer than FUZZY_NAME_LIMIT can never pass the name gate
|
||||
(every candidate name is bounded by the same limit), so skip the scan. */
|
||||
if (target_len > FUZZY_NAME_LIMIT) {
|
||||
close(dir_fd);
|
||||
free(leaf);
|
||||
free(full_path);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
int scanfd = dup(dir_fd);
|
||||
if (scanfd < 0) {
|
||||
@@ -800,10 +867,24 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* The DP scratch rows are allocated once per scan (not once per candidate). */
|
||||
FuzzyEditBuffer ebuf;
|
||||
if (!fuzzy_edit_buffer_init(&ebuf)) {
|
||||
closedir(dir);
|
||||
close(dir_fd);
|
||||
free(leaf);
|
||||
free(full_path);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
FuzzyCandidate best;
|
||||
memset(&best, 0, sizeof(best));
|
||||
const struct dirent* entry;
|
||||
size_t scanned = 0;
|
||||
/* readdir() yields entries in filesystem-dependent order, so the SET of
|
||||
candidates seen is order-dependent; the winner is still deterministic
|
||||
because every candidate is compared with the total ordering in
|
||||
fuzzy_candidate_better (acceptable for a heuristic). */
|
||||
while (scanned < FUZZY_MAX_DIRECTORY_SCAN && (entry = readdir(dir)) != NULL) {
|
||||
scanned++;
|
||||
const char* name = entry->d_name;
|
||||
@@ -817,9 +898,21 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
||||
if (cand_size == 0 || cand_size > MAX_RECEIVE_WHOLE_FILE_SIZE ||
|
||||
!delta_should_attempt(cand_size, check_size, config->delta_max_file_size))
|
||||
continue;
|
||||
size_t distance = fuzzy_edit_distance(leaf, target_len, name, name_len);
|
||||
/* Cheap pre-name gates run BEFORE the edit-distance DP. The edit distance
|
||||
is bounded below by the length gap |la-lb| and by the number of
|
||||
characters of one basename that are absent from the other (each such
|
||||
position costs at least one op), so a candidate whose acceptance gate
|
||||
(distance*2 <= longer) already fails on the max of those bounds is
|
||||
skipped without running the DP. */
|
||||
size_t longer = target_len > name_len ? target_len : name_len;
|
||||
if (distance == SIZE_MAX || distance * 2 > longer)
|
||||
size_t bound = longer - (target_len < name_len ? target_len : name_len);
|
||||
size_t absent = fuzzy_absent_char_bound(leaf, target_len, name, name_len);
|
||||
if (absent > bound)
|
||||
bound = absent;
|
||||
if (bound * 2 > longer)
|
||||
continue;
|
||||
size_t distance = fuzzy_edit_distance(&ebuf, leaf, target_len, name, name_len);
|
||||
if (distance * 2 > longer)
|
||||
continue;
|
||||
FuzzyCandidate cand;
|
||||
memcpy(cand.name, name, name_len + 1);
|
||||
@@ -831,10 +924,15 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
||||
}
|
||||
closedir(dir);
|
||||
free(leaf);
|
||||
fuzzy_edit_buffer_destroy(&ebuf);
|
||||
|
||||
void* basis = NULL;
|
||||
if (best.name[0]) {
|
||||
int fd = openat(dir_fd, best.name, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
|
||||
/* O_NONBLOCK: a name raced to a FIFO between the fstatat gate and this open
|
||||
would otherwise block the receive thread forever on open(2); with it the
|
||||
open fails (ENXIO) and the fstat/S_ISREG gate below would reject it too.
|
||||
A regular file opened with O_NONBLOCK is unaffected. */
|
||||
int fd = openat(dir_fd, best.name, O_RDONLY | O_NOFOLLOW | O_NONBLOCK | O_CLOEXEC);
|
||||
if (fd >= 0) {
|
||||
struct stat st;
|
||||
if (fstat(fd, &st) == 0 && S_ISREG(st.st_mode) &&
|
||||
|
||||
@@ -106,6 +106,7 @@ class CountingProxy:
|
||||
server_sock = socket.create_connection(("127.0.0.1", self.target_port),
|
||||
timeout=10)
|
||||
except OSError:
|
||||
self._listener.close()
|
||||
return
|
||||
c2s, s2c = [0], [0]
|
||||
a = threading.Thread(target=self._pump, args=(client_sock, server_sock, c2s))
|
||||
@@ -116,6 +117,7 @@ class CountingProxy:
|
||||
b.join()
|
||||
self.client_to_server = c2s[0]
|
||||
self.server_to_client = s2c[0]
|
||||
self._listener.close()
|
||||
|
||||
thread = threading.Thread(target=serve)
|
||||
thread.start()
|
||||
|
||||
@@ -2513,6 +2513,8 @@ class TestFuzzy:
|
||||
|
||||
OLD_NAME = "report-2025.dat"
|
||||
NEW_NAME = "report-2026.dat"
|
||||
TS = 1577836800 # 2020-01-01, used to pin stale destination mtimes
|
||||
SIZE = 2 * 1024 * 1024
|
||||
|
||||
def _client_via_proxy(self, source, dest, flags, proxy):
|
||||
cmd = (CLIENT_CMD + ["--source-dir", source, "--dest-dir", dest,
|
||||
@@ -2721,3 +2723,112 @@ class TestFuzzy:
|
||||
assert _read_file(os.path.join(received, self.NEW_NAME)) == new_bytes
|
||||
assert _read_file(os.path.join(received, self.OLD_NAME)) == old_bytes
|
||||
|
||||
@staticmethod
|
||||
def _rand_bytes(size, seed):
|
||||
return random.Random(seed).randbytes(size)
|
||||
|
||||
def _replace_source_file(self, source, old_name, new_name, new_bytes):
|
||||
"""Remove old_name from source and add new_name with new_bytes."""
|
||||
os.unlink(os.path.join(source, old_name))
|
||||
with open(os.path.join(source, new_name), "wb") as fh:
|
||||
fh.write(new_bytes)
|
||||
|
||||
def test_worthless_fuzzy_basis_falls_back_inside_handshake(self, shared_server):
|
||||
# The sibling passes the name AND size gates but shares no blocks with
|
||||
# the incoming file, so the sender's delta is not worthwhile: it replies
|
||||
# STATUS_NEXT and the receiver consumes the WHOLE file inside the delta
|
||||
# handshake. This proves a bad fuzzy basis cannot desync the protocol
|
||||
# or corrupt the result.
|
||||
source, dest = self._prepare("worthless")
|
||||
basis = self._rand_bytes(self.SIZE, 424242)
|
||||
target = self._rand_bytes(self.SIZE, 777777)
|
||||
self._seed_dest(source, dest, {self.OLD_NAME: basis}, shared_server.port)
|
||||
self._replace_source_file(source, self.OLD_NAME, self.NEW_NAME, target)
|
||||
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
|
||||
assert result.returncode == 0, \
|
||||
f"--fuzzy worthless-basis run failed: {(result.stderr or result.stdout)[:300]}"
|
||||
received = get_dest_received_dir(dest, source)
|
||||
assert _read_file(os.path.join(received, self.NEW_NAME)) == target, \
|
||||
"whole-file fallback after a worthless fuzzy basis is not byte-exact"
|
||||
assert proxy.client_to_server > self.SIZE // 2, \
|
||||
"a worthless basis should have made the sender fall back to the whole file"
|
||||
|
||||
def test_existing_dest_file_preferred_over_fuzzy_sibling(self, shared_server):
|
||||
# Non-displacement: the destination holds a file at the exact path that
|
||||
# is inside the delta size bounds (same size, different content, older
|
||||
# mtime). FastSync must delta against THAT file -- even though it
|
||||
# shares nothing with the source -- and must NOT reuse a similar-named
|
||||
# sibling that is byte-identical to the source.
|
||||
source, dest = self._prepare("nondisp")
|
||||
sibling = self._rand_bytes(self.SIZE, 111) # will equal the incoming file
|
||||
stale = self._rand_bytes(self.SIZE, 333) # worthless exact-path file
|
||||
self._seed_dest(source, dest,
|
||||
{self.OLD_NAME: sibling, self.NEW_NAME: stale},
|
||||
shared_server.port)
|
||||
# Force the exact-path destination file's mtime into the past so the
|
||||
# quick check deterministically decides to transfer it.
|
||||
os.utime(os.path.join(get_dest_received_dir(dest, source), self.NEW_NAME),
|
||||
(self.TS, self.TS))
|
||||
os.unlink(os.path.join(source, self.OLD_NAME))
|
||||
with open(os.path.join(source, self.NEW_NAME), "wb") as fh:
|
||||
fh.write(sibling)
|
||||
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
|
||||
assert result.returncode == 0, \
|
||||
f"--fuzzy non-displacement run failed: {(result.stderr or result.stdout)[:300]}"
|
||||
received = get_dest_received_dir(dest, source)
|
||||
assert _read_file(os.path.join(received, self.NEW_NAME)) == sibling
|
||||
assert proxy.client_to_server > self.SIZE // 2, \
|
||||
"the exact-path destination file must be the delta basis, not the fuzzy sibling"
|
||||
|
||||
def test_fuzzy_basis_larger_than_source(self, shared_server):
|
||||
# The similar sibling is LARGER than the incoming file (within the delta
|
||||
# engine's 10x ratio); the new file is an exact prefix of the basis, so
|
||||
# every block matches and only a tiny delta travels.
|
||||
source, dest = self._prepare("largerbasis")
|
||||
big = self._rand_bytes(1536 * 1024, 1)
|
||||
prefix = big[:1024 * 1024]
|
||||
self._seed_dest(source, dest, {self.OLD_NAME: big}, shared_server.port)
|
||||
self._replace_source_file(source, self.OLD_NAME, self.NEW_NAME, prefix)
|
||||
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
|
||||
assert result.returncode == 0, \
|
||||
f"--fuzzy larger-basis run failed: {(result.stderr or result.stdout)[:300]}"
|
||||
received = get_dest_received_dir(dest, source)
|
||||
assert _read_file(os.path.join(received, self.NEW_NAME)) == prefix, \
|
||||
"shrunken file reconstructed from a larger fuzzy basis is not byte-exact"
|
||||
assert proxy.client_to_server < len(prefix) // 4, \
|
||||
"larger fuzzy basis should have carried most of the file as block matches"
|
||||
|
||||
def test_fuzzy_basis_smaller_than_source(self, shared_server):
|
||||
# The similar sibling is SMALLER than the incoming file; the new file
|
||||
# appends data past the basis, so the appended tail travels as literals
|
||||
# while the shared prefix is block-matched.
|
||||
source, dest = self._prepare("smallerbasis")
|
||||
base = self._rand_bytes(self.SIZE, 2)
|
||||
tail = self._rand_bytes(64 * 1024, 3)
|
||||
new_bytes = base + tail
|
||||
self._seed_dest(source, dest, {self.OLD_NAME: base}, shared_server.port)
|
||||
self._replace_source_file(source, self.OLD_NAME, self.NEW_NAME, new_bytes)
|
||||
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
|
||||
assert result.returncode == 0, \
|
||||
f"--fuzzy smaller-basis run failed: {(result.stderr or result.stdout)[:300]}"
|
||||
received = get_dest_received_dir(dest, source)
|
||||
assert _read_file(os.path.join(received, self.NEW_NAME)) == new_bytes, \
|
||||
"grown file reconstructed from a smaller fuzzy basis is not byte-exact"
|
||||
assert proxy.client_to_server < len(new_bytes) // 4, \
|
||||
"smaller fuzzy basis should have block-matched the shared prefix"
|
||||
|
||||
def test_no_fuzzy_end_to_end_equals_no_flag(self, shared_server):
|
||||
# --no-fuzzy must not enable anything: a run with it behaves exactly
|
||||
# like a run without it (whole-file transfer, byte-exact output).
|
||||
source, dest = self._prepare("nofuzzye2e")
|
||||
old_bytes, new_bytes = _random_payloads()
|
||||
self._seed_dest(source, dest, {self.OLD_NAME: old_bytes}, shared_server.port)
|
||||
self._replace_source_file(source, self.OLD_NAME, self.NEW_NAME, new_bytes)
|
||||
result, proxy = self._run_measured(source, dest, ["--no-fuzzy"], shared_server.port)
|
||||
assert result.returncode == 0, \
|
||||
f"--no-fuzzy run failed: {(result.stderr or result.stdout)[:300]}"
|
||||
received = get_dest_received_dir(dest, source)
|
||||
assert _read_file(os.path.join(received, self.NEW_NAME)) == new_bytes
|
||||
assert proxy.client_to_server > len(new_bytes) // 2, \
|
||||
"--no-fuzzy should leave the default whole-file behavior intact"
|
||||
|
||||
@@ -1246,6 +1246,32 @@ static void test_parse_args_fuzzy_respects_no_delta() {
|
||||
}
|
||||
}
|
||||
|
||||
/* An explicit --no-incremental is respected by the --fuzzy implication in
|
||||
* either argument order (unlike the basis-dir options, --fuzzy does not force
|
||||
* the incremental handshake back on). Because delta needs the handshake, the
|
||||
* delta implication is suppressed too, so the run is a plain (default-mode)
|
||||
* transfer rather than an invalid "--delta requires --incremental" config. */
|
||||
static void test_parse_args_fuzzy_respects_no_incremental() {
|
||||
static const char* const combos[][2] = {
|
||||
{"--fuzzy", "--no-incremental"},
|
||||
{"--no-incremental", "--fuzzy"},
|
||||
};
|
||||
for (size_t i = 0; i < sizeof(combos) / sizeof(combos[0]); i++) {
|
||||
Config* cfg = config_create();
|
||||
char* argv[] = {"fastsync", (char*)combos[i][0], (char*)combos[i][1], "/src", "/dst"};
|
||||
int positional_args[2];
|
||||
int positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
|
||||
EXPECT_TRUE(cfg->fuzzy);
|
||||
EXPECT_FALSE(cfg->use_incremental);
|
||||
EXPECT_FALSE(cfg->use_delta);
|
||||
cfg->send_directory = str_dup("/src");
|
||||
cfg->receive_root_directory = str_dup("/dst");
|
||||
EXPECT_TRUE(validate_config(cfg));
|
||||
config_delete(cfg);
|
||||
}
|
||||
}
|
||||
|
||||
/* --fuzzy requires the delta machinery, which the chunk-serialization (-s) and
|
||||
* sendfile (-f) modes reject -- mirroring the --delta constraint checks. */
|
||||
static void test_validate_config_fuzzy_incompatible_modes() {
|
||||
@@ -1831,6 +1857,7 @@ void test_client_cli() {
|
||||
test_parse_args_fuzzy_negation();
|
||||
test_parse_args_fuzzy_with_whole_file();
|
||||
test_parse_args_fuzzy_respects_no_delta();
|
||||
test_parse_args_fuzzy_respects_no_incremental();
|
||||
test_validate_config_fuzzy_incompatible_modes();
|
||||
test_parse_args_one_file_system();
|
||||
test_parse_args_compression_aliases();
|
||||
|
||||
Reference in new issue
Block a user