Merge feat/p3-checksum-choice: --checksum-choice/--cc and --checksum-seed
This commit is contained in:
+3
-3
@@ -288,8 +288,8 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
|
||||
|
||||
| Flag | Rsync Description | FastSync Status | Notes |
|
||||
|------|-------------------|-----------------|-------|
|
||||
| `--checksum` | Skip based on checksum | ✅ Implemented | With `--incremental`, compares xxHash64 content checksums; `-c` remains compression |
|
||||
| `--checksum-choice=STR` | Choose checksum algorithm | ❌ Not Implemented | xxHash used internally |
|
||||
| `--checksum` | Skip based on checksum | ✅ Implemented | With `--incremental`, compares per-file whole-file content digests to skip unchanged files. The digest algorithm is `xxh64` with seed 0 by default and is selectable via `--checksum-choice`/`--cc` (xxh64/xxhash or md5) and `--checksum-seed=NUM` (see those rows); `-c` remains compression |
|
||||
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ✅ Implemented | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and by the basis-dir content verification. FastSync genuinely supports `xxh64` (the default, exact xxHash64, seeded by `--checksum-seed`) and `md5` (via OpenSSL EVP); `xxhash` is accepted as rsync's spelling of xxHash64. Any other name (md4/sha1/sha256/crc32/none/…) is rejected with a clear error at parse time — never a silent no-op. `--cc` is the alias (`--cc=ALG` and space forms both parse). The algorithm id and seed cross the wire with the config frame, so the receiver hashes its on-disk old file with the SAME algorithm+seed the sender used and both agree on a match; the sender's digest and the receiver's comparison live in the per-file `STATUS_CHECK` handshake, which now carries a length-prefixed, bounded (1..16 byte) digest instead of a fixed 64-bit value, and the receiver pins the received length to the negotiated algorithm's digest length (defense-in-depth: a mismatched/malicious length only forces a safe re-transfer). Note: `md5` is a FIPS-non-approved algorithm, so under an OpenSSL build with FIPS mode enabled `--checksum-choice=md5` fails loudly rather than silently falling back. Protocol/layout: `PROTOCOL_VERSION` bumped **2.9.0 → 2.10.0** (peers must match). Defaults preserve the pre-existing behavior byte-for-byte (xxh64, seed 0). Like rsync, the choice only takes effect where a whole-file digest is actually computed (`--checksum` on, or a basis-dir flag); it does not itself enable `--checksum`. Closely-related divergence: the delta BLOCK strong checksum (§11 delta) stays xxHash32 — `--checksum-choice` selects only the whole-file digest, matching rsync where the per-block checksum is independent of the whole-file checksum choice |
|
||||
| `--compare-dest=DIR` | Compare dest files relative to DIR | ✅ Implemented | DIR is a receiver-side basis relative to the destination root (confined below it; absolute/`..`/`.` rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Divergences: when the destination already holds a *different* version rsync deletes it but FastSync instead transfers the data (keeps the mirror complete; never deletes without `--delete`); attribute-only differences on a match are not re-applied (data is skipped so the sender never sends metadata); content is verified by xxHash64, stricter than rsync's default quick check. Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) |
|
||||
| `--copy-dest=DIR` | Include copies of unchanged files | ✅ Implemented | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. Divergences: a basis-hit destination keeps the basis file's own mode/uid/gid and mtime (the sender sends no metadata on a skip), so with `--size-only` its mtime can differ from the source and attribute-only differences are copied with the basis attributes rather than rsync's "copy + fix attributes". Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
||||
| `--link-dest=DIR` | Hardlink to files when unchanged | ✅ Implemented | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. Divergences and caveats: an already up-to-date destination file is not re-linked to a basis file (only files that would otherwise be written are linked); a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); with `--size-only` the linked mtime can differ from the source; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
||||
@@ -363,7 +363,7 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
|
||||
| `--fsync` | Fsync every written file before publication | ✅ Implemented | |
|
||||
| `--protocol=NUM` | Force older protocol version | ❌ Not Implemented | |
|
||||
| `--iconv=CONVERT_SPEC` | Charset conversion | ❌ Not Implemented | |
|
||||
| `--checksum-seed=NUM` | Set checksum seed | ❌ Not Implemented | |
|
||||
| `--checksum-seed=NUM` | Set checksum seed | ✅ Implemented | Sets the seed for FastSync's whole-file xxHash64 digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). An explicit seed deterministically changes every computed digest on BOTH endpoints (sender and receiver share the seed via the config frame, protocol 2.10.0), so identical runs with the same seed skip the same files and a changed seed changes the digests — the explicit-seed path that makes xxHash comparisons deterministic. `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta`. Divergence from rsync: the default is seed 0, and FastSync never randomizes the seed (rsync uses a random per-transfer seed when `--checksum-seed` is unset); FastSync's unset default therefore reproduces its historical byte-for-byte behavior |
|
||||
| `--secluded-args` | Use protocol to send args | 🔄 Compatibility No-op | Accepted for CLI compatibility; it does not change FastSync transport or protocol behavior. `-s` remains chunk serialization. |
|
||||
| `--no-OPTION` | Turn off implied option | ✅ Supported | Supported boolean FastSync options and archive-implied options; unsafe or value-taking options are rejected. |
|
||||
|
||||
|
||||
+59
-2
@@ -94,6 +94,47 @@ static int set_compression_choice(Config* config, const char* value) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Validate and store the --checksum-choice/--cc algorithm. Only the algorithms
|
||||
* the engine genuinely supports are accepted (xxHash64 and md5); anything else
|
||||
* is a clear error, never a silent no-op. "xxhash" is accepted as rsync's
|
||||
* spelling of xxHash64. */
|
||||
static int set_checksum_choice(Config* config, const char* value) {
|
||||
int algo = checksum_algo_from_name(value);
|
||||
if (algo < 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "--checksum-choice must be xxh64 (or xxhash) or md5 (got '%s')",
|
||||
value);
|
||||
return -1;
|
||||
}
|
||||
config->checksum_algo = algo;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* parse_ull_arg is defined later in this file; declared here for the seed
|
||||
parser below. */
|
||||
static int parse_ull_arg(const char* val, unsigned long long* out, const char* optname);
|
||||
|
||||
/* Parse --checksum-seed=NUM as a strict decimal 0..UINT64_MAX. A blank value,
|
||||
* a sign, or any non-digit (which parse_ull_arg's strtoull would silently
|
||||
* coerce) is rejected: an explicit seed must be an exact unsigned integer or
|
||||
* the run fails with a clear error rather than quietly ignoring the value. */
|
||||
static int set_checksum_seed(Config* config, const char* value) {
|
||||
if (!value || *value == '\0') {
|
||||
log_message(LOG_LEVEL_ERROR, "--checksum-seed must be a non-negative integer");
|
||||
return -1;
|
||||
}
|
||||
for (const char* p = value; *p; p++) {
|
||||
if (*p < '0' || *p > '9') {
|
||||
log_message(LOG_LEVEL_ERROR, "--checksum-seed must be a non-negative integer");
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
unsigned long long seed;
|
||||
if (parse_ull_arg(value, &seed, "--checksum-seed") != 0)
|
||||
return -1;
|
||||
config->checksum_seed = seed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int set_compression_threads_option(int* dest, const char* value) {
|
||||
if (set_positive_int_option(dest, value, "--compress-threads") != 0)
|
||||
return -1;
|
||||
@@ -1019,8 +1060,24 @@ int parse_args(Config* config, int argc, char* argv[], int* positional_args,
|
||||
log_message(LOG_LEVEL_ERROR, "missing argument for %s", argv[i]);
|
||||
return -1;
|
||||
}
|
||||
log_message(LOG_LEVEL_ERROR, "%s is not supported yet (xxHash64 is used)", argv[i]);
|
||||
return -1;
|
||||
if (set_checksum_choice(config, argv[++i]) != 0)
|
||||
return -1;
|
||||
} else if (strncmp(argv[i], "--checksum-choice=", 18) == 0) {
|
||||
if (set_checksum_choice(config, argv[i] + 18) != 0)
|
||||
return -1;
|
||||
} else if (strncmp(argv[i], "--cc=", 5) == 0) {
|
||||
if (set_checksum_choice(config, argv[i] + 5) != 0)
|
||||
return -1;
|
||||
} else if (strncmp(argv[i], "--checksum-seed=", 16) == 0) {
|
||||
if (set_checksum_seed(config, argv[i] + 16) != 0)
|
||||
return -1;
|
||||
} else if (opt_is(argv[i], "--checksum-seed", NULL)) {
|
||||
if (i + 1 >= argc) {
|
||||
log_message(LOG_LEVEL_ERROR, "missing argument for --checksum-seed");
|
||||
return -1;
|
||||
}
|
||||
if (set_checksum_seed(config, argv[++i]) != 0)
|
||||
return -1;
|
||||
} else if (strncmp(argv[i], "--compare-dest=", 15) == 0) {
|
||||
if (set_basis_dest_option(config, BASIS_DEST_COMPARE, argv[i] + 15, "--compare-dest") != 0)
|
||||
return -1;
|
||||
|
||||
@@ -843,12 +843,18 @@ static int incremental_check(Client* client, File* file, const Config* config,
|
||||
if (!send_n_data(client->file_descriptor, &mtime_nsec, sizeof(mtime_nsec)))
|
||||
return -1;
|
||||
/* With alternate basis directories the receiver must be able to verify the
|
||||
* content of every candidate basis file, so the sender supplies its xxHash64
|
||||
* for every file even when --checksum was not requested. */
|
||||
* content of every candidate basis file, so the sender supplies its whole-file
|
||||
* digest (computed with the negotiated --checksum-choice algorithm and
|
||||
* --checksum-seed) for every file even when --checksum was not requested. */
|
||||
if (config->checksum || config_has_basis(config)) {
|
||||
uint64_t checksum;
|
||||
if (!file_checksum(file, &checksum) ||
|
||||
!send_n_data(client->file_descriptor, &checksum, sizeof(checksum)))
|
||||
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t digest_len = 0;
|
||||
if (!file_checksum(file, (ChecksumAlgo)config->checksum_algo, config->checksum_seed, digest,
|
||||
sizeof(digest), &digest_len))
|
||||
return -1;
|
||||
uint8_t wire_len = (uint8_t)digest_len;
|
||||
if (!send_n_data(client->file_descriptor, &wire_len, sizeof(wire_len)) ||
|
||||
!send_n_data(client->file_descriptor, digest, wire_len))
|
||||
return -1;
|
||||
}
|
||||
Status s;
|
||||
@@ -897,7 +903,8 @@ static int incremental_check(Client* client, File* file, const Config* config,
|
||||
}
|
||||
|
||||
static int send_delta(Client* client, File* file, DeltaSignature* sig, Config* config) {
|
||||
Delta* delta = delta_compute(file->data->data, file->data->size, sig, config->delta_block_size);
|
||||
Delta* delta = delta_compute_seeded(file->data->data, file->data->size, sig,
|
||||
config->delta_block_size, (uint32_t)config->checksum_seed);
|
||||
/* The receiver is blocked after sending the signature. Every local
|
||||
fallback therefore needs the explicit NEXT response before full data. */
|
||||
if (!delta)
|
||||
|
||||
+6
-2
@@ -97,8 +97,12 @@ void print_usage(void) {
|
||||
printf(" into the destination instead of transferring its data\n");
|
||||
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
|
||||
printf(" into the destination (repeatable; earlier DIRs win)\n");
|
||||
printf(" --checksum-choice, --cc <alg> Checksum algorithm (not supported yet; xxHash64 is "
|
||||
"used)\n");
|
||||
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
|
||||
printf(" --checksum compares (xxh64/xxhash or md5; default xxh64 with\n");
|
||||
printf(" seed 0). The seed comes from --checksum-seed\n");
|
||||
printf(" --checksum-seed <num> Seed for the whole-file xxHash64 digest (and the delta\n");
|
||||
printf(" block strong hash, low 32 bits); md5 ignores the seed. The\n");
|
||||
printf(" digest algorithm and seed must match on sender and receiver\n");
|
||||
printf(" --delta Delta transfer for changed files (requires --incremental)\n");
|
||||
printf(" -W, --whole-file Transfer changed files without delta processing\n");
|
||||
printf(" -y, --fuzzy Use a similar-named file already in the destination\n");
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
#include "checksum.h"
|
||||
#include <openssl/evp.h>
|
||||
#include <string.h>
|
||||
#include <strings.h>
|
||||
|
||||
/* delta.c owns the single XXH_IMPLEMENTATION that provides the xxHash symbols
|
||||
* for the whole binary; this TU only needs the declarations. */
|
||||
#include <xxhash.h>
|
||||
|
||||
bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t size, uint8_t* out,
|
||||
size_t out_capacity, size_t* out_len) {
|
||||
if (!out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
|
||||
return false;
|
||||
if (data == NULL && size != 0)
|
||||
return false;
|
||||
|
||||
if (algo == CHECKSUM_ALGO_XXH64) {
|
||||
uint64_t digest = XXH64(data, size, seed);
|
||||
memcpy(out, &digest, sizeof(digest));
|
||||
*out_len = sizeof(digest);
|
||||
return true;
|
||||
}
|
||||
|
||||
if (algo == CHECKSUM_ALGO_MD5) {
|
||||
/* md5 takes no seed; the caller's seed is deliberately ignored (documented
|
||||
* in RSYNC_COMPAT.md). OpenSSL's one-shot EVP_Digest needs a non-NULL
|
||||
* buffer even for an empty input, so map a NULL data + size==0 to an empty
|
||||
* buffer. */
|
||||
static const uint8_t empty = 0;
|
||||
const void* input = data ? data : ∅
|
||||
unsigned int digest_len = 0;
|
||||
if (EVP_Digest(input, size, out, &digest_len, EVP_md5(), NULL) != 1)
|
||||
return false;
|
||||
if (digest_len > out_capacity)
|
||||
return false;
|
||||
*out_len = digest_len;
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
int checksum_algo_from_name(const char* name) {
|
||||
if (!name)
|
||||
return -1;
|
||||
if (strcasecmp(name, "xxh64") == 0 || strcasecmp(name, "xxhash") == 0)
|
||||
return (int)CHECKSUM_ALGO_XXH64;
|
||||
if (strcasecmp(name, "md5") == 0)
|
||||
return (int)CHECKSUM_ALGO_MD5;
|
||||
return -1;
|
||||
}
|
||||
|
||||
const char* checksum_algo_name(ChecksumAlgo algo) {
|
||||
switch (algo) {
|
||||
case CHECKSUM_ALGO_XXH64:
|
||||
return "xxh64";
|
||||
case CHECKSUM_ALGO_MD5:
|
||||
return "md5";
|
||||
}
|
||||
return "<unknown>";
|
||||
}
|
||||
|
||||
bool checksum_algo_valid(int algo) {
|
||||
return algo == (int)CHECKSUM_ALGO_XXH64 || algo == (int)CHECKSUM_ALGO_MD5;
|
||||
}
|
||||
|
||||
uint8_t checksum_digest_len(ChecksumAlgo algo) {
|
||||
switch (algo) {
|
||||
case CHECKSUM_ALGO_XXH64:
|
||||
return 8;
|
||||
case CHECKSUM_ALGO_MD5:
|
||||
return 16;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
#ifndef CHECKSUM_H
|
||||
#define CHECKSUM_H
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
|
||||
/* Whole-file content-digest algorithms selectable with --checksum-choice and
|
||||
* seeded with --checksum-seed. The ids are the values actually placed on the
|
||||
* wire (config frame), so they must be kept stable and validated on receive.
|
||||
* CHECKSUM_ALGO_XXH64 == 0 is the default and is byte-for-byte what FastSync
|
||||
* computed before these options existed (xxHash64 with seed 0). */
|
||||
typedef enum { CHECKSUM_ALGO_XXH64 = 0, CHECKSUM_ALGO_MD5 = 1 } ChecksumAlgo;
|
||||
|
||||
/* md5 digest is 16 bytes, the longest supported. */
|
||||
#define CHECKSUM_MAX_DIGEST_LEN 16
|
||||
|
||||
/* Compute the whole-file digest of the first `size` bytes of `data`.
|
||||
*
|
||||
* - CHECKSUM_ALGO_XXH64: xxHash64(data, size, seed) (full 64-bit seed).
|
||||
* - CHECKSUM_ALGO_MD5: md5(data, size) via OpenSSL EVP.
|
||||
* md5 has no seed, so `seed` is ignored (documented).
|
||||
* - `size == 0` hashes the empty input (plus its seed), not a NULL input.
|
||||
*
|
||||
* Writes up to `out_capacity` bytes into `out`, storing the digest length in
|
||||
* *out_len. Returns false on NULL out* or when the digest would not fit.
|
||||
* Never writes more than CHECKSUM_MAX_DIGEST_LEN bytes. */
|
||||
bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t size, uint8_t* out,
|
||||
size_t out_capacity, size_t* out_len);
|
||||
|
||||
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
|
||||
* Accepts "xxh64" and "xxhash" (both map to CHECKSUM_ALGO_XXH64, rsync's
|
||||
* xxhash spelling) and "md5". Returns -1 for any unsupported name. */
|
||||
int checksum_algo_from_name(const char* name);
|
||||
|
||||
/* Canonical name of an algorithm (used in CLI error messages). */
|
||||
const char* checksum_algo_name(ChecksumAlgo algo);
|
||||
|
||||
/* True when `algo` is a supported id (used by config receive validation). */
|
||||
bool checksum_algo_valid(int algo);
|
||||
|
||||
/* Digest length in bytes for an algorithm (xxx64 = 8, md5 = 16). */
|
||||
uint8_t checksum_digest_len(ChecksumAlgo algo);
|
||||
|
||||
#endif /* CHECKSUM_H */
|
||||
+24
-3
@@ -129,6 +129,8 @@ static void config_set_defaults(Config* config) {
|
||||
config->daemon_config = NULL;
|
||||
config->server_mode = false;
|
||||
config->checksum = false;
|
||||
config->checksum_algo = CHECKSUM_ALGO_XXH64;
|
||||
config->checksum_seed = 0;
|
||||
config->compress_choice = NULL;
|
||||
config->chmod_spec = NULL;
|
||||
config->skip_compress_suffixes = NULL;
|
||||
@@ -175,7 +177,7 @@ static bool validate_received_config(const Config* config) {
|
||||
!(config->delay_updates && delay_updates_staging_name_conflict(config->backup_dir)) &&
|
||||
valid_wire_bool(config->partial) && valid_wire_bool(config->delete_before) &&
|
||||
valid_wire_bool(config->checksum) && valid_wire_bool(config->eight_bit_output) &&
|
||||
config_has_valid_delete_timing(config) &&
|
||||
checksum_algo_valid(config->checksum_algo) && config_has_valid_delete_timing(config) &&
|
||||
!(config->skip_compress_set && config->use_chunk_serialization) &&
|
||||
/* --append / --append-verify tail resume needs the per-file check,
|
||||
which chunk serialization -s disables: reject on the receiver too
|
||||
@@ -470,6 +472,15 @@ static bool send_fuzzy_option(int fd, const Config* c) {
|
||||
return send_int(fd, c->fuzzy);
|
||||
}
|
||||
|
||||
/* --checksum-choice/--cc + --checksum-seed. The algorithm id and seed travel
|
||||
* with the config so the receiver hashes the on-disk old file with the same
|
||||
* parameters the sender used for its digest (see checksum.h). Trailing fields
|
||||
* on the config frame; protocol 2.10.0. */
|
||||
static bool send_checksum_options(int fd, const Config* c) {
|
||||
return send_int(fd, c->checksum_algo) &&
|
||||
send_n_data(fd, &c->checksum_seed, sizeof(c->checksum_seed));
|
||||
}
|
||||
|
||||
static bool receive_core_fields(int fd, Config* c) {
|
||||
int value;
|
||||
if (!receive_wire_bool(fd, &c->eight_bit_output))
|
||||
@@ -654,13 +665,22 @@ static bool receive_fuzzy_option(int fd, Config* c) {
|
||||
return receive_wire_bool(fd, &c->fuzzy);
|
||||
}
|
||||
|
||||
static bool receive_checksum_options(int fd, Config* c) {
|
||||
int algo;
|
||||
if (!receive_int(fd, &algo) || !checksum_algo_valid(algo))
|
||||
return false;
|
||||
c->checksum_algo = algo;
|
||||
return receive_n_data(fd, &c->checksum_seed, sizeof(c->checksum_seed));
|
||||
}
|
||||
|
||||
bool config_send(int file_descriptor, const Config* config) {
|
||||
protocol_session_set_max_alloc(NULL, config->max_alloc);
|
||||
if (!send_core_fields(file_descriptor, config) || !send_delta_fields(file_descriptor, config) ||
|
||||
!send_file_options(file_descriptor, config) ||
|
||||
!send_selection_options(file_descriptor, config) ||
|
||||
!send_resume_options(file_descriptor, config) ||
|
||||
!send_basis_options(file_descriptor, config) || !send_fuzzy_option(file_descriptor, config))
|
||||
!send_basis_options(file_descriptor, config) || !send_fuzzy_option(file_descriptor, config) ||
|
||||
!send_checksum_options(file_descriptor, config))
|
||||
return false;
|
||||
Status status;
|
||||
if (!receive_status(file_descriptor, &status))
|
||||
@@ -694,7 +714,8 @@ Config* config_receive(int file_descriptor) {
|
||||
!receive_selection_options(file_descriptor, config) ||
|
||||
!receive_resume_options(file_descriptor, config) ||
|
||||
!receive_basis_options(file_descriptor, config) ||
|
||||
!receive_fuzzy_option(file_descriptor, config))
|
||||
!receive_fuzzy_option(file_descriptor, config) ||
|
||||
!receive_checksum_options(file_descriptor, config))
|
||||
goto error;
|
||||
if (config->compress_choice[0] != '\0' && strcmp(config->compress_choice, "zstd") != 0 &&
|
||||
strcmp(config->compress_choice, "none") != 0) {
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
#define CONFIG_H
|
||||
|
||||
#include "array_list.h"
|
||||
#include "checksum.h"
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
@@ -235,6 +236,18 @@ typedef struct Config {
|
||||
char* compress_choice;
|
||||
char* chmod_spec;
|
||||
|
||||
/* --checksum-choice / --cc and --checksum-seed. checksum_algo is the id of
|
||||
* the whole-file content-digest algorithm used by the per-file --incremental
|
||||
* handshake (sender computes it, receiver compares it to skip unchanged
|
||||
* files) and by the basis-dir content verification. checksum_seed is passed
|
||||
* to xxHash64 (and to the delta block strong hash, low 32 bits); md5 has no
|
||||
* seed so it is ignored there. Both cross the wire: the receiver MUST hash
|
||||
* the on-disk old file with the same algorithm and seed to reach a matching
|
||||
* digest. Defaults (XXH64 / seed 0) reproduce the pre-existing behavior
|
||||
* byte-for-byte. */
|
||||
int checksum_algo; /* ChecksumAlgo, default CHECKSUM_ALGO_XXH64 */
|
||||
uint64_t checksum_seed; /* default 0 */
|
||||
|
||||
char** skip_compress_suffixes;
|
||||
int skip_compress_count;
|
||||
bool skip_compress_set;
|
||||
|
||||
+19
-5
@@ -29,12 +29,21 @@ uint32_t delta_xxhash32(const void* data, uint32_t len) {
|
||||
return XXH32(data, len, 0);
|
||||
}
|
||||
|
||||
uint32_t delta_xxhash32_seeded(const void* data, uint32_t len, uint32_t seed) {
|
||||
return XXH32(data, len, seed);
|
||||
}
|
||||
|
||||
uint64_t delta_xxhash64(const void* data, size_t len) {
|
||||
return XXH64(data, len, 0);
|
||||
}
|
||||
|
||||
DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_file_size,
|
||||
uint32_t block_size) {
|
||||
return delta_signature_create_seeded(old_file_data, old_file_size, block_size, 0);
|
||||
}
|
||||
|
||||
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
|
||||
uint32_t block_size, uint32_t seed) {
|
||||
if (old_file_data == NULL || old_file_size == 0 || block_size == 0)
|
||||
return NULL;
|
||||
|
||||
@@ -67,7 +76,7 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
|
||||
uint32_t len =
|
||||
(uint32_t)((old_file_size - offset < block_size) ? (old_file_size - offset) : block_size);
|
||||
sig->blocks[i].adler32 = delta_adler32(data + offset, len);
|
||||
sig->blocks[i].xxhash = delta_xxhash32(data + offset, len);
|
||||
sig->blocks[i].xxhash = delta_xxhash32_seeded(data + offset, len, seed);
|
||||
}
|
||||
|
||||
return sig;
|
||||
@@ -297,7 +306,7 @@ static uint32_t* delta_build_index(const DeltaSignature* sig, uint32_t bucket_co
|
||||
* O(1) per window); otherwise an exact linear scan is used. */
|
||||
static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uint32_t adler,
|
||||
bool full_window, const DeltaSignature* sig, const uint32_t* heads,
|
||||
const uint32_t* next, uint32_t mask) {
|
||||
const uint32_t* next, uint32_t mask, uint32_t seed) {
|
||||
if (!full_window || sig->block_count == 0)
|
||||
return DELTA_NO_BLOCK;
|
||||
|
||||
@@ -309,7 +318,7 @@ static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uin
|
||||
if (sig->blocks[j].adler32 != adler)
|
||||
continue;
|
||||
if (!have_xxh) {
|
||||
window_xxh = delta_xxhash32(window, window_len);
|
||||
window_xxh = delta_xxhash32_seeded(window, window_len, seed);
|
||||
have_xxh = true;
|
||||
}
|
||||
if (window_xxh == sig->blocks[j].xxhash)
|
||||
@@ -321,7 +330,7 @@ static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uin
|
||||
/* Fallback used when the index could not be allocated. */
|
||||
for (uint32_t j = 0; j < sig->block_count; j++) {
|
||||
if (sig->blocks[j].adler32 == adler) {
|
||||
uint32_t window_xxh = delta_xxhash32(window, window_len);
|
||||
uint32_t window_xxh = delta_xxhash32_seeded(window, window_len, seed);
|
||||
if (window_xxh == sig->blocks[j].xxhash)
|
||||
return j;
|
||||
}
|
||||
@@ -331,6 +340,11 @@ static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uin
|
||||
|
||||
Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const DeltaSignature* sig,
|
||||
uint32_t block_size) {
|
||||
return delta_compute_seeded(new_file_data, new_file_size, sig, block_size, 0);
|
||||
}
|
||||
|
||||
Delta* delta_compute_seeded(const void* new_file_data, uint64_t new_file_size,
|
||||
const DeltaSignature* sig, uint32_t block_size, uint32_t seed) {
|
||||
if (!new_file_data || !sig || !sig->blocks || new_file_size == 0 || block_size == 0 ||
|
||||
block_size > DELTA_BLOCK_SIZE_MAX || sig->block_size != block_size)
|
||||
return NULL;
|
||||
@@ -397,7 +411,7 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
|
||||
|
||||
bool matched = false;
|
||||
uint32_t match_block = delta_find_match(new_data + i, window_len, adler, full_window, sig,
|
||||
index, chain_next, mask);
|
||||
index, chain_next, mask, seed);
|
||||
if (match_block != DELTA_NO_BLOCK) {
|
||||
if (has_literal) {
|
||||
if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, i)) {
|
||||
|
||||
@@ -56,12 +56,22 @@ typedef struct {
|
||||
|
||||
DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_file_size,
|
||||
uint32_t block_size);
|
||||
/* Seeded equivalent of delta_signature_create: the per-block strong (xxHash32)
|
||||
* checksum uses `seed` (the low 32 bits of --checksum-seed). Passing seed 0 is
|
||||
* identical to the unseeded function. */
|
||||
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
|
||||
uint32_t block_size, uint32_t seed);
|
||||
Data* delta_signature_serialize(const DeltaSignature* sig);
|
||||
DeltaSignature* delta_signature_deserialize(const Data* data);
|
||||
void delta_signature_destroy(DeltaSignature* sig);
|
||||
|
||||
Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const DeltaSignature* sig,
|
||||
uint32_t block_size);
|
||||
/* Seeded equivalent of delta_compute: the per-window strong (xxHash32) check
|
||||
* uses `seed` (the low 32 bits of --checksum-seed). The receiver's signature
|
||||
* must have been built with the same seed for matching. */
|
||||
Delta* delta_compute_seeded(const void* new_file_data, uint64_t new_file_size,
|
||||
const DeltaSignature* sig, uint32_t block_size, uint32_t seed);
|
||||
Data* delta_serialize(const Delta* delta);
|
||||
Delta* delta_deserialize(const Data* data);
|
||||
void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta, uint32_t block_size);
|
||||
@@ -72,6 +82,7 @@ bool delta_is_worthwhile(const Delta* delta, uint64_t new_file_size);
|
||||
|
||||
uint32_t delta_adler32(const void* data, uint32_t len);
|
||||
uint32_t delta_xxhash32(const void* data, uint32_t len);
|
||||
uint32_t delta_xxhash32_seeded(const void* data, uint32_t len, uint32_t seed);
|
||||
uint64_t delta_xxhash64(const void* data, size_t len);
|
||||
|
||||
#endif
|
||||
|
||||
+6
-6
@@ -43,17 +43,17 @@ static unsigned long long next_temp_sequence(void) {
|
||||
return atomic_fetch_add_explicit(&sequence, 1, memory_order_relaxed);
|
||||
}
|
||||
|
||||
bool file_checksum(File* file, uint64_t* checksum) {
|
||||
if (!file || !checksum || !file->data)
|
||||
bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, size_t out_capacity,
|
||||
size_t* out_len) {
|
||||
if (!file || !out || !out_len || !file->data)
|
||||
return false;
|
||||
if (file->data->size == 0) {
|
||||
*checksum = delta_xxhash64("", 0);
|
||||
return true;
|
||||
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
|
||||
}
|
||||
if (!file->data->data && !file_load_data(file))
|
||||
return false;
|
||||
*checksum = delta_xxhash64(file->data->data, file->data->size);
|
||||
return true;
|
||||
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
|
||||
out_len);
|
||||
}
|
||||
|
||||
File* file_create(const char* path) {
|
||||
|
||||
+7
-1
@@ -4,6 +4,7 @@
|
||||
#include "file_send.h"
|
||||
#include "file_receive.h"
|
||||
#include "file_types.h"
|
||||
#include "checksum.h"
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <sys/stat.h>
|
||||
@@ -14,7 +15,12 @@
|
||||
File* file_create(const char* path);
|
||||
void file_destroy(void* item);
|
||||
bool file_load_data(File* file);
|
||||
bool file_checksum(File* file, uint64_t* checksum);
|
||||
/* Compute the whole-file content digest of `file` with the negotiated
|
||||
* --checksum-choice algorithm and --checksum-seed. Writes the digest into
|
||||
* `out` (capacity `out_capacity`) and its length into `*out_len`. Returns
|
||||
* false on read/allocation failure or when the digest would not fit. */
|
||||
bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, size_t out_capacity,
|
||||
size_t* out_len);
|
||||
size_t file_content_to_buffer(File* file);
|
||||
FileMetadata* file_metadata_create(const struct stat* stats);
|
||||
void file_metadata_destroy(void* metadata);
|
||||
|
||||
+35
-17
@@ -355,7 +355,8 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
|
||||
return NULL;
|
||||
}
|
||||
|
||||
DeltaSignature* sig = delta_signature_create(old_data, old_size, config->delta_block_size);
|
||||
DeltaSignature* sig = delta_signature_create_seeded(old_data, old_size, config->delta_block_size,
|
||||
(uint32_t)config->checksum_seed);
|
||||
if (!sig) {
|
||||
free(old_data);
|
||||
*failed = true;
|
||||
@@ -630,8 +631,8 @@ static bool basis_quick_matches(const Config* config, const struct stat* st, tim
|
||||
so the caller can materialize the file without re-reading it. */
|
||||
static bool basis_match_find(const Config* config, const char* check_path,
|
||||
unsigned long long check_size, time_t check_mtime,
|
||||
long check_mtime_nsec, uint64_t check_checksum, bool load_content,
|
||||
BasisMatch* out) {
|
||||
long check_mtime_nsec, const uint8_t* check_digest,
|
||||
size_t check_digest_len, bool load_content, BasisMatch* out) {
|
||||
memset(out, 0, sizeof(*out));
|
||||
if (!config || !config_has_basis(config) || config->ignore_times)
|
||||
return false;
|
||||
@@ -651,10 +652,13 @@ static bool basis_match_find(const Config* config, const char* check_path,
|
||||
if (basis_quick_matches(config, &st, check_mtime, check_mtime_nsec)) {
|
||||
Data* content = basis_read_content(fd, check_size);
|
||||
if (content) {
|
||||
uint64_t basis_hash = check_size == 0 ? delta_xxhash64("", 0)
|
||||
: content->data ? delta_xxhash64(content->data, content->size)
|
||||
: 0;
|
||||
if (basis_hash == check_checksum) {
|
||||
uint8_t basis_digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t basis_len = 0;
|
||||
bool hashed = checksum_digest((ChecksumAlgo)config->checksum_algo, config->checksum_seed,
|
||||
content->data, content->size, basis_digest,
|
||||
sizeof(basis_digest), &basis_len);
|
||||
if (hashed && basis_len == check_digest_len && check_digest_len > 0 &&
|
||||
memcmp(basis_digest, check_digest, check_digest_len) == 0) {
|
||||
out->hit = true;
|
||||
out->type = entry->type;
|
||||
out->basis_path = candidate;
|
||||
@@ -1041,7 +1045,8 @@ File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
|
||||
unsigned long long check_size;
|
||||
long long check_mtime;
|
||||
long long check_mtime_nsec;
|
||||
uint64_t check_checksum = 0;
|
||||
uint8_t check_digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t check_digest_len = 0;
|
||||
if (!receive_n_data(fd, &check_size, sizeof(check_size)) ||
|
||||
!receive_n_data(fd, &check_mtime, sizeof(check_mtime))) {
|
||||
free(check_path);
|
||||
@@ -1053,10 +1058,20 @@ File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return NULL;
|
||||
}
|
||||
if ((config->checksum || config_has_basis(config)) &&
|
||||
!receive_n_data(fd, &check_checksum, sizeof(check_checksum))) {
|
||||
free(check_path);
|
||||
return NULL;
|
||||
if ((config->checksum || config_has_basis(config))) {
|
||||
uint8_t wire_len;
|
||||
if (!receive_n_data(fd, &wire_len, sizeof(wire_len)) || wire_len == 0 ||
|
||||
wire_len > CHECKSUM_MAX_DIGEST_LEN ||
|
||||
wire_len != checksum_digest_len((ChecksumAlgo)config->checksum_algo)) {
|
||||
free(check_path);
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return NULL;
|
||||
}
|
||||
check_digest_len = wire_len;
|
||||
if (!receive_n_data(fd, check_digest, check_digest_len)) {
|
||||
free(check_path);
|
||||
return NULL;
|
||||
}
|
||||
}
|
||||
|
||||
if (check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
|
||||
@@ -1142,10 +1157,13 @@ File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
|
||||
skipped and the transfer proceeds with the full new contents. */
|
||||
bool match = false;
|
||||
if (checksum_needs_read) {
|
||||
if (old_size == 0)
|
||||
match = delta_xxhash64("", 0) == check_checksum;
|
||||
else
|
||||
match = old_data != NULL && delta_xxhash64(old_data, (size_t)old_size) == check_checksum;
|
||||
uint8_t old_digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t old_len = 0;
|
||||
bool hashed = checksum_digest((ChecksumAlgo)config->checksum_algo, config->checksum_seed,
|
||||
old_size == 0 ? "" : old_data, (size_t)old_size, old_digest,
|
||||
sizeof(old_digest), &old_len);
|
||||
match = hashed && old_len == check_digest_len && check_digest_len > 0 &&
|
||||
memcmp(old_digest, check_digest, check_digest_len) == 0;
|
||||
} else if (size_equal && !config->ignore_times) {
|
||||
match = config->size_only || match_by_metadata;
|
||||
}
|
||||
@@ -1169,7 +1187,7 @@ File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
|
||||
if (config_has_basis(config)) {
|
||||
BasisMatch basis;
|
||||
basis_match_find(config, check_path, check_size, (time_t)check_mtime, (long)check_mtime_nsec,
|
||||
check_checksum, true, &basis);
|
||||
check_digest, check_digest_len, true, &basis);
|
||||
if (basis.hit) {
|
||||
if (basis.type == BASIS_DEST_COMPARE) {
|
||||
/* compare-dest never copies: an exact match only suppresses the data
|
||||
|
||||
@@ -550,6 +550,147 @@ class TestIncremental:
|
||||
assert not mismatches, f"Mismatch: {mismatches}"
|
||||
|
||||
|
||||
class TestChecksumChoice:
|
||||
"""--checksum-choice/--cc and --checksum-seed: the whole-file digest used by
|
||||
the --incremental/--checksum handshake is selectable and seedable. The
|
||||
receiver hashes the on-disk old file with the SAME algorithm+seed, so an
|
||||
unchanged file is skipped and a changed file (even with identical size and
|
||||
mtime) is transferred -- and the transfer always lands byte-exact.
|
||||
FastSync accepts xxh64 (default, seed-aware) and md5; names it does not
|
||||
implement are rejected, never silently ignored."""
|
||||
|
||||
def test_unsupported_algorithm_is_rejected(self, shared_server):
|
||||
result, _ = run_client(
|
||||
SOURCE_DIR, DEST_DIR,
|
||||
flags=["--checksum", "--checksum-choice=sha256"],
|
||||
port=shared_server.port,
|
||||
)
|
||||
assert result.returncode != 0, "sha256 must be rejected, not silently ignored"
|
||||
|
||||
@pytest.mark.parametrize("algo", ["xxh64", "md5"])
|
||||
@pytest.mark.parametrize("mt", [False, True])
|
||||
def test_unchanged_skipped_and_bytes_preserved(self, shared_server, algo, mt):
|
||||
clean_dir(DEST_DIR)
|
||||
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-M"], port=shared_server.port)
|
||||
assert result.returncode == 0, f"seed sync failed: {result.stderr[:200]}"
|
||||
|
||||
flags = (["-M", "--incremental", "--checksum", f"--checksum-choice={algo}"] +
|
||||
(["-m"] if mt else []))
|
||||
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
|
||||
assert result.returncode == 0, f"checksum {algo} run failed: {result.stderr[:200]}"
|
||||
|
||||
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
|
||||
mismatches, missing = verify_transfer(SOURCE_DIR, received)
|
||||
assert not missing, f"Missing: {missing}"
|
||||
assert not mismatches, f"Mismatch: {mismatches}"
|
||||
|
||||
# A changed source file with the SAME size and mtime must still be
|
||||
# detected (and re-transferred byte-exactly) because the whole-file digest
|
||||
# differs -- the explicit reason --checksum exists. This exercises the
|
||||
# sender/receiver digest agreement for a non-default algorithm.
|
||||
@pytest.mark.parametrize("algo", ["xxh64", "md5"])
|
||||
@pytest.mark.parametrize("mt", [False, True])
|
||||
def test_changed_same_size_mtime_redetected(self, shared_server, algo, mt):
|
||||
clean_dir(DEST_DIR)
|
||||
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-M"], port=shared_server.port)
|
||||
assert result.returncode == 0
|
||||
|
||||
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
|
||||
source_file = os.path.join(SOURCE_DIR, "small.txt") # "hello world\n" (12 bytes)
|
||||
received_file = os.path.join(received, "small.txt")
|
||||
source_stat = os.stat(source_file)
|
||||
with open(received_file, "wb") as f:
|
||||
f.write(b"DDDDDDDDDDDD") # same size, different content
|
||||
os.utime(received_file, (source_stat.st_atime, source_stat.st_mtime))
|
||||
|
||||
flags = (["-M", "--incremental", "--checksum", f"--checksum-choice={algo}"] +
|
||||
(["-m"] if mt else []))
|
||||
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
|
||||
assert result.returncode == 0, f"checksum {algo} redetect failed: {result.stderr[:200]}"
|
||||
with open(received_file, "rb") as f:
|
||||
assert f.read() == b"hello world\n"
|
||||
|
||||
@pytest.mark.parametrize("algo", ["xxh64", "md5"])
|
||||
def test_unchanged_run_transfers_almost_no_data(self, shared_server, algo):
|
||||
# A fully-unchanged --checksum run skips every file: only the config + a
|
||||
# small handshake travels, not the payloads. Proxy byte counts are not
|
||||
# available for -m (multithreaded connections), so single-thread only.
|
||||
clean_dir(DEST_DIR)
|
||||
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-M"], port=shared_server.port)
|
||||
assert result.returncode == 0
|
||||
|
||||
flags = ["-M", "--incremental", "--checksum", f"--checksum-choice={algo}"]
|
||||
proxy = CountingProxy(shared_server.port)
|
||||
cmd = (CLIENT_CMD + ["--source-dir", SOURCE_DIR, "--dest-dir", DEST_DIR,
|
||||
"--save-to-disk", "--server-port", str(proxy.port)] + flags)
|
||||
result = proxy.run(cmd)
|
||||
assert result.returncode == 0, f"checksum {algo} skip run failed: {result.stderr[:200]}"
|
||||
assert proxy.client_to_server < 100000, \
|
||||
f"unchanged --checksum run sent {proxy.client_to_server} bytes; expected a skip"
|
||||
|
||||
@pytest.mark.parametrize("mt", [False, True])
|
||||
def test_seed_is_deterministic_and_preserves_content(self, shared_server, mt):
|
||||
clean_dir(DEST_DIR)
|
||||
flags = ["-M", "--incremental", "--checksum",
|
||||
"--checksum-choice=xxh64", "--checksum-seed=987654"] + (["-m"] if mt else [])
|
||||
first, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
|
||||
assert first.returncode == 0, f"seeded run failed: {first.stderr[:200]}"
|
||||
|
||||
# A second run with the SAME seed and unchanged content skips everything
|
||||
# deterministically (same digests both sides).
|
||||
second, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
|
||||
assert second.returncode == 0, f"deterministic rerun failed: {second.stderr[:200]}"
|
||||
|
||||
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
|
||||
mismatches, missing = verify_transfer(SOURCE_DIR, received)
|
||||
assert not missing and not mismatches, f"missing={missing} mismatches={mismatches}"
|
||||
|
||||
# A changed file with the same size and mtime is still caught and fixed
|
||||
# (a non-zero seed does not weaken the comparison).
|
||||
source_file = os.path.join(SOURCE_DIR, "medium.txt")
|
||||
received_file = os.path.join(received, "medium.txt")
|
||||
source_stat = os.stat(source_file)
|
||||
with open(received_file, "wb") as f:
|
||||
f.write(b"z" * os.path.getsize(source_file))
|
||||
os.utime(received_file, (source_stat.st_atime, source_stat.st_mtime))
|
||||
third, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
|
||||
assert third.returncode == 0, f"seeded redetect failed: {third.stderr[:200]}"
|
||||
with open(received_file, "rb") as f:
|
||||
assert f.read() == open(source_file, "rb").read()
|
||||
|
||||
# --checksum-seed also feeds the delta path's per-block strong checksum on
|
||||
# both ends (receiver signature and sender window hash use the same seed),
|
||||
# so a seeded delta transfer still lands byte-exact.
|
||||
@pytest.mark.parametrize("mt", [False, True])
|
||||
def test_seed_delta_block_hash_transfers_byte_exact(self, shared_server, mt):
|
||||
source = os.path.join(TEST_DATA_DIR, f"ccseed_{'m' if mt else 's'}_src")
|
||||
dest = os.path.join(TEST_DATA_DIR, f"ccseed_{'m' if mt else 's'}_dst")
|
||||
clean_dir(source)
|
||||
clean_dir(dest)
|
||||
big = os.path.join(source, "big.bin")
|
||||
with open(big, "wb") as f:
|
||||
f.write(bytes(range(256)) * 200) # 51200 bytes > delta 16K floor
|
||||
result, _ = run_client(source, dest, flags=["-M"], port=shared_server.port)
|
||||
assert result.returncode == 0, f"seed delta seed failed: {result.stderr[:200]}"
|
||||
|
||||
# Edit a region so the receiver must match a changed block with the seed.
|
||||
with open(big, "r+b") as f:
|
||||
f.seek(1000)
|
||||
f.write(b"\x00" * 64)
|
||||
# Force an mtime mismatch: the incremental quick-check skips files whose
|
||||
# stored mtime second equals the source's, which can collide when the
|
||||
# edit and the prior sync share a second. Setting an old dest mtime
|
||||
# guarantees the delta path is exercised deterministically.
|
||||
os.utime(os.path.join(get_dest_received_dir(dest, source), "big.bin"), (0, 0))
|
||||
flags = (["-M", "--incremental", "--delta", "--checksum-seed=314159"] +
|
||||
(["-m"] if mt else []))
|
||||
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
|
||||
assert result.returncode == 0, f"seed delta run failed: {result.stderr[:200]}"
|
||||
received = get_dest_received_dir(dest, source)
|
||||
assert _read_file(os.path.join(received, "big.bin")) == _read_file(big), \
|
||||
"seeded delta transfer is not byte-exact"
|
||||
|
||||
|
||||
class TestUpdate:
|
||||
def test_update_skips_older_destination_and_allows_equal_or_newer_source(self, shared_server):
|
||||
clean_dir(DEST_DIR)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#include "test_array_list.h"
|
||||
#include "test_chunk.h"
|
||||
#include "test_change_list.h"
|
||||
#include "test_checksum.h"
|
||||
#include "test_client_cli.h"
|
||||
#include "test_compression.h"
|
||||
#include "test_config.h"
|
||||
@@ -46,6 +47,7 @@ int main() {
|
||||
RUN_TEST(test_config);
|
||||
RUN_TEST(test_compression);
|
||||
RUN_TEST(test_scanner);
|
||||
RUN_TEST(test_checksum);
|
||||
RUN_TEST(test_delta);
|
||||
RUN_TEST(test_data);
|
||||
RUN_TEST(test_protocol);
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
#include "test_checksum.h"
|
||||
#include "checksum.h"
|
||||
#include "test_utils.h"
|
||||
#include <string.h>
|
||||
|
||||
/* Known xxHash64 vector (seed 0) for the empty string and a literal.
|
||||
* The md5 vectors are the standard NIST/RFC1321 test strings. These pin the
|
||||
* digest selection to genuinely distinct algorithm outputs so a --checksum-
|
||||
* choice change is observable, not a silent no-op. */
|
||||
|
||||
static void test_checksum_xxh64_seed0() {
|
||||
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t len = 0;
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "hello", 5, out, sizeof(out), &len));
|
||||
EXPECT_TRUE(len == (size_t)8);
|
||||
/* Hard-coded: XXH64("hello", 5, 0). */
|
||||
const uint8_t expect[8] = {0xa3, 0x6d, 0x9f, 0x88, 0x7d, 0x82, 0xc7, 0x26};
|
||||
for (int i = 0; i < 8; i++)
|
||||
EXPECT_EQ_INT(out[i], expect[i]);
|
||||
}
|
||||
|
||||
static void test_checksum_xxh64_empty() {
|
||||
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t len = 0;
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "", 0, out, sizeof(out), &len));
|
||||
EXPECT_TRUE(len == (size_t)8);
|
||||
/* XXH64("", 0, 0). */
|
||||
const uint8_t expect[8] = {0x99, 0xe9, 0xd8, 0x51, 0x37, 0xdb, 0x46, 0xef};
|
||||
for (int i = 0; i < 8; i++)
|
||||
EXPECT_EQ_INT(out[i], expect[i]);
|
||||
}
|
||||
|
||||
/* A nonzero seed must change the xxh64 digest: the algorithm is genuinely
|
||||
* seed-aware, deterministic, and distinct from seed 0. */
|
||||
static void test_checksum_xxh64_seed_changes_digest() {
|
||||
uint8_t a[CHECKSUM_MAX_DIGEST_LEN], b[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t alen = 0, blen = 0;
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 7, "payload", 7, a, sizeof(a), &alen));
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "payload", 7, b, sizeof(b), &blen));
|
||||
EXPECT_TRUE(alen == blen);
|
||||
EXPECT_TRUE(memcmp(a, b, alen) != 0);
|
||||
}
|
||||
|
||||
static void test_checksum_xxh64_seed_deterministic() {
|
||||
uint8_t a[CHECKSUM_MAX_DIGEST_LEN], b[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t alen = 0, blen = 0;
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 12345, "same", 4, a, sizeof(a), &alen));
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 12345, "same", 4, b, sizeof(b), &blen));
|
||||
EXPECT_TRUE(alen == blen);
|
||||
EXPECT_TRUE(memcmp(a, b, alen) == 0);
|
||||
}
|
||||
|
||||
static void test_checksum_md5_vectors() {
|
||||
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t len = 0;
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "", 0, out, sizeof(out), &len));
|
||||
EXPECT_TRUE(len == (size_t)16);
|
||||
const uint8_t expect_empty[16] = {0xd4, 0x1d, 0x8c, 0xd9, 0x8f, 0x00, 0xb2, 0x04,
|
||||
0xe9, 0x80, 0x09, 0x98, 0xec, 0xf8, 0x42, 0x7e};
|
||||
EXPECT_TRUE(memcmp(out, expect_empty, 16) == 0);
|
||||
|
||||
/* MD5("abc") */
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "abc", 3, out, sizeof(out), &len));
|
||||
const uint8_t expect_abc[16] = {0x90, 0x01, 0x50, 0x98, 0x3c, 0xd2, 0x4f, 0xb0,
|
||||
0xd6, 0x96, 0x3f, 0x7d, 0x28, 0xe1, 0x7f, 0x72};
|
||||
EXPECT_TRUE(memcmp(out, expect_abc, 16) == 0);
|
||||
}
|
||||
|
||||
/* md5 is 16 bytes and differs from the 8-byte xxh64 for the same input, so the
|
||||
* choice is observably different both in length and in content. */
|
||||
static void test_checksum_algo_lengths_distinct() {
|
||||
EXPECT_EQ_INT((int)checksum_digest_len(CHECKSUM_ALGO_XXH64), 8);
|
||||
EXPECT_EQ_INT((int)checksum_digest_len(CHECKSUM_ALGO_MD5), 16);
|
||||
|
||||
uint8_t x[CHECKSUM_MAX_DIGEST_LEN], m[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t xl = 0, ml = 0;
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "same content", 12, x, sizeof(x), &xl));
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "same content", 12, m, sizeof(m), &ml));
|
||||
EXPECT_TRUE(xl == (size_t)8);
|
||||
EXPECT_TRUE(ml == (size_t)16);
|
||||
EXPECT_TRUE(memcmp(x, m, 8) != 0);
|
||||
}
|
||||
|
||||
/* md5 has no seed: two distinct seeds give the same md5 digest (documented);
|
||||
* the seed is only honored by xxh64 and the delta block hash (low 32 bits). */
|
||||
static void test_checksum_md5_seed_ignored() {
|
||||
uint8_t a[CHECKSUM_MAX_DIGEST_LEN], b[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t alen = 0, blen = 0;
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "data", 4, a, sizeof(a), &alen));
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 99, "data", 4, b, sizeof(b), &blen));
|
||||
EXPECT_TRUE(memcmp(a, b, alen) == 0);
|
||||
}
|
||||
|
||||
static void test_checksum_algo_name_mapping() {
|
||||
EXPECT_EQ_INT(checksum_algo_from_name("xxh64"), (int)CHECKSUM_ALGO_XXH64);
|
||||
EXPECT_EQ_INT(checksum_algo_from_name("XXH64"), (int)CHECKSUM_ALGO_XXH64);
|
||||
EXPECT_EQ_INT(checksum_algo_from_name("xxhash"), (int)CHECKSUM_ALGO_XXH64);
|
||||
EXPECT_EQ_INT(checksum_algo_from_name("XXHASH"), (int)CHECKSUM_ALGO_XXH64);
|
||||
EXPECT_EQ_INT(checksum_algo_from_name("md5"), (int)CHECKSUM_ALGO_MD5);
|
||||
EXPECT_EQ_INT(checksum_algo_from_name("MD5"), (int)CHECKSUM_ALGO_MD5);
|
||||
EXPECT_TRUE(checksum_algo_from_name("sha256") < 0);
|
||||
EXPECT_TRUE(checksum_algo_from_name("crc32") < 0);
|
||||
EXPECT_TRUE(checksum_algo_from_name("none") < 0);
|
||||
EXPECT_TRUE(checksum_algo_from_name("xxh3") < 0);
|
||||
EXPECT_TRUE(checksum_algo_from_name("") < 0);
|
||||
EXPECT_TRUE(checksum_algo_from_name(NULL) < 0);
|
||||
|
||||
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_XXH64));
|
||||
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_MD5));
|
||||
EXPECT_FALSE(checksum_algo_valid(99));
|
||||
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_XXH64), "xxh64");
|
||||
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_MD5), "md5");
|
||||
}
|
||||
|
||||
static void test_checksum_truncated_buffer_rejected() {
|
||||
uint8_t small[4];
|
||||
size_t len = 0;
|
||||
/* The digest cannot fit in a 4-byte buffer. */
|
||||
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "x", 1, small, sizeof(small), &len));
|
||||
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "x", 1, small, sizeof(small), &len));
|
||||
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, NULL, 5, small, sizeof(small), &len));
|
||||
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "x", 1, NULL, 0, &len));
|
||||
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "x", 1, small, sizeof(small), NULL));
|
||||
}
|
||||
|
||||
/* A NULL data pointer with size 0 is the empty input, not an error. */
|
||||
static void test_checksum_null_empty_digest() {
|
||||
uint8_t a[CHECKSUM_MAX_DIGEST_LEN], b[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t alen = 0, blen = 0;
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, NULL, 0, a, sizeof(a), &alen));
|
||||
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "", 0, b, sizeof(b), &blen));
|
||||
EXPECT_TRUE(alen == blen);
|
||||
EXPECT_TRUE(memcmp(a, b, alen) == 0);
|
||||
}
|
||||
|
||||
void test_checksum(void) {
|
||||
test_checksum_xxh64_seed0();
|
||||
test_checksum_xxh64_empty();
|
||||
test_checksum_xxh64_seed_changes_digest();
|
||||
test_checksum_xxh64_seed_deterministic();
|
||||
test_checksum_md5_vectors();
|
||||
test_checksum_algo_lengths_distinct();
|
||||
test_checksum_md5_seed_ignored();
|
||||
test_checksum_algo_name_mapping();
|
||||
test_checksum_truncated_buffer_rejected();
|
||||
test_checksum_null_empty_digest();
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
#ifndef TEST_CHECKSUM_H
|
||||
#define TEST_CHECKSUM_H
|
||||
|
||||
void test_checksum(void);
|
||||
|
||||
#endif
|
||||
+83
-3
@@ -1,4 +1,5 @@
|
||||
#include "test_client_cli.h"
|
||||
#include "checksum.h"
|
||||
#include "client_validation.h"
|
||||
#include "chmod.h"
|
||||
#include "config.h"
|
||||
@@ -785,8 +786,7 @@ static void test_parse_args_rejects_unimplemented_options() {
|
||||
"--ipv4",
|
||||
"--daemon",
|
||||
"--config",
|
||||
"--server",
|
||||
"--checksum-choice"};
|
||||
"--server"};
|
||||
|
||||
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
|
||||
Config* cfg = config_create();
|
||||
@@ -951,7 +951,8 @@ static void test_parse_args_no_preserve_blocks_implicit_metadata() {
|
||||
}
|
||||
}
|
||||
|
||||
/* Checksum-choice spellings are recognized and rejected until algorithms are implemented. */
|
||||
/* --checksum-choice and its --cc alias select the whole-file digest algorithm
|
||||
(default xxh64; both "xxh64" and the rsync "xxhash" spelling accepted). */
|
||||
static void test_parse_args_checksum_choice_aliases() {
|
||||
static const char* const options[] = {"--checksum-choice", "--cc"};
|
||||
|
||||
@@ -960,12 +961,88 @@ static void test_parse_args_checksum_choice_aliases() {
|
||||
char* argv[] = {"fastsync", (char*)options[i], "xxh64", "/src", "/dst"};
|
||||
int positional_args[2];
|
||||
int positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
|
||||
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_XXH64);
|
||||
config_delete(cfg);
|
||||
}
|
||||
}
|
||||
|
||||
/* Both the "--checksum-choice=ALG" and "--cc=ALG" inline forms parse. */
|
||||
static void test_parse_args_checksum_choice_equals_forms() {
|
||||
Config* cfg = config_create();
|
||||
char* argv[] = {"fastsync", "--checksum-choice=md5", "/src", "/dst"};
|
||||
int positional_args[2];
|
||||
int positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
|
||||
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD5);
|
||||
config_delete(cfg);
|
||||
|
||||
cfg = config_create();
|
||||
char* argv2[] = {"fastsync", "--cc=xxhash", "/src", "/dst"};
|
||||
positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 4, argv2, positional_args, &positional_count), 0);
|
||||
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_XXH64);
|
||||
config_delete(cfg);
|
||||
}
|
||||
|
||||
/* An algorithm FastSync does not support must be rejected, never a silent
|
||||
no-op. */
|
||||
static void test_parse_args_checksum_choice_rejects_unsupported() {
|
||||
static const char* const bad[] = {"md4", "sha256", "crc32", "none", "bogus"};
|
||||
for (size_t i = 0; i < sizeof(bad) / sizeof(bad[0]); i++) {
|
||||
Config* cfg = config_create();
|
||||
char* argv[] = {"fastsync", "--checksum-choice", (char*)bad[i], "/src", "/dst"};
|
||||
int positional_args[2];
|
||||
int positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
|
||||
config_delete(cfg);
|
||||
}
|
||||
}
|
||||
|
||||
/* --checksum-seed parses as a 64-bit non-negative integer (space and = forms);
|
||||
invalid values are rejected. */
|
||||
static void test_parse_args_checksum_seed() {
|
||||
Config* cfg = config_create();
|
||||
char* argv[] = {"fastsync", "--checksum-seed=42", "/src", "/dst"};
|
||||
int positional_args[2];
|
||||
int positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
|
||||
EXPECT_TRUE(cfg->checksum_seed == 42ULL);
|
||||
config_delete(cfg);
|
||||
|
||||
cfg = config_create();
|
||||
char* argv2[] = {"fastsync", "--checksum-seed", "12345", "/src", "/dst"};
|
||||
positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 5, argv2, positional_args, &positional_count), 0);
|
||||
EXPECT_TRUE(cfg->checksum_seed == 12345ULL);
|
||||
config_delete(cfg);
|
||||
|
||||
/* 0 is a valid (and default) seed. */
|
||||
cfg = config_create();
|
||||
char* argv3[] = {"fastsync", "--checksum-seed=0", "/src", "/dst"};
|
||||
positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 4, argv3, positional_args, &positional_count), 0);
|
||||
EXPECT_TRUE(cfg->checksum_seed == 0ULL);
|
||||
config_delete(cfg);
|
||||
|
||||
/* Non-numeric and negative seeds are rejected. */
|
||||
static const char* const bad[] = {"abc", "-5", "1.5", ""};
|
||||
for (size_t i = 0; i < sizeof(bad) / sizeof(bad[0]); i++) {
|
||||
cfg = config_create();
|
||||
char* argv4[] = {"fastsync", "--checksum-seed", (char*)bad[i], "/src", "/dst"};
|
||||
positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 5, argv4, positional_args, &positional_count), -1);
|
||||
config_delete(cfg);
|
||||
}
|
||||
|
||||
/* Missing value is rejected. */
|
||||
cfg = config_create();
|
||||
char* argv5[] = {"fastsync", "--checksum-seed"};
|
||||
positional_count = 0;
|
||||
EXPECT_EQ_INT(parse_args(cfg, 2, argv5, positional_args, &positional_count), -1);
|
||||
config_delete(cfg);
|
||||
}
|
||||
|
||||
static void test_parse_args_rejects_unsafe_negation() {
|
||||
static const char* const options[] = {"--no-archive", "--no-timeout", "--no-unknown"};
|
||||
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
|
||||
@@ -2101,6 +2178,9 @@ void test_client_cli() {
|
||||
test_parse_args_log_file_format();
|
||||
test_parse_args_checksum_choice_aliases();
|
||||
test_parse_args_checksum_choice_requires_value();
|
||||
test_parse_args_checksum_choice_equals_forms();
|
||||
test_parse_args_checksum_choice_rejects_unsupported();
|
||||
test_parse_args_checksum_seed();
|
||||
test_parse_args_temp_dir();
|
||||
test_parse_args_delay_updates();
|
||||
test_validate_config_delay_updates_rejects_inplace();
|
||||
|
||||
+64
-2
@@ -809,11 +809,11 @@ static void test_config_is_remote_dest() {
|
||||
are negotiated to the receiver so it knows to reply STATUS_APPEND on a
|
||||
shorter destination. */
|
||||
static void test_config_append_wire_roundtrip() {
|
||||
if (is_running_under_valgrind())
|
||||
return;
|
||||
struct {
|
||||
bool append, append_verify;
|
||||
} cases[] = {{true, false}, {false, true}, {true, true}, {false, false}};
|
||||
if (is_running_under_valgrind())
|
||||
return;
|
||||
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
|
||||
int p[2];
|
||||
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
|
||||
@@ -848,6 +848,66 @@ static void test_config_append_wire_roundtrip() {
|
||||
}
|
||||
}
|
||||
|
||||
/* --checksum-choice/--cc and --checksum-seed cross the wire intact so the
|
||||
receiver hashes the on-disk old file with the same algorithm and seed. */
|
||||
static void test_config_checksum_options_wire_roundtrip() {
|
||||
struct {
|
||||
int algo;
|
||||
unsigned long long seed;
|
||||
} cases[] = {
|
||||
{CHECKSUM_ALGO_XXH64, 0},
|
||||
{CHECKSUM_ALGO_XXH64, 42},
|
||||
{CHECKSUM_ALGO_MD5, 7},
|
||||
{CHECKSUM_ALGO_MD5, 0},
|
||||
};
|
||||
if (is_running_under_valgrind())
|
||||
return;
|
||||
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
|
||||
int p[2];
|
||||
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
|
||||
pid_t pid = fork();
|
||||
if (pid == 0) {
|
||||
close(p[1]);
|
||||
io_set_fds(p[0], p[0]);
|
||||
Config* recv = config_receive(p[0]);
|
||||
bool ok = recv != NULL && recv->checksum_algo == cases[i].algo &&
|
||||
recv->checksum_seed == cases[i].seed;
|
||||
config_delete(recv);
|
||||
close(p[0]);
|
||||
_exit(ok ? 0 : 1);
|
||||
} else {
|
||||
close(p[0]);
|
||||
io_set_fds(p[1], p[1]);
|
||||
Config* send_cfg = config_create();
|
||||
EXPECT_NOT_NULL(send_cfg);
|
||||
send_cfg->send_directory = str_dup("/src");
|
||||
send_cfg->receive_root_directory = str_dup("/dst");
|
||||
send_cfg->checksum_algo = cases[i].algo;
|
||||
send_cfg->checksum_seed = cases[i].seed;
|
||||
bool sent = config_send(p[1], send_cfg);
|
||||
int status;
|
||||
waitpid(pid, &status, 0);
|
||||
close(p[1]);
|
||||
config_delete(send_cfg);
|
||||
EXPECT_TRUE(sent);
|
||||
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* An out-of-range algorithm id on the wire must be rejected on receive, never
|
||||
accepted as-is (prevents mixing unsupported digests on a path). */
|
||||
static void test_config_receive_rejects_invalid_checksum_algo() {
|
||||
if (is_running_under_valgrind())
|
||||
return;
|
||||
Config* c = config_create();
|
||||
EXPECT_NOT_NULL(c);
|
||||
c->send_directory = str_dup("/src");
|
||||
c->receive_root_directory = str_dup("/dst");
|
||||
c->checksum_algo = 99;
|
||||
EXPECT_FALSE(roundtrip_config_ok(c));
|
||||
config_delete(c);
|
||||
}
|
||||
void test_config() {
|
||||
test_config_lifecycle();
|
||||
test_config_ssh_dest();
|
||||
@@ -870,6 +930,8 @@ void test_config() {
|
||||
test_config_basis_roundtrip();
|
||||
test_config_basis_wire_rejects_escaping();
|
||||
test_config_basis_normalization();
|
||||
test_config_checksum_options_wire_roundtrip();
|
||||
test_config_receive_rejects_invalid_checksum_algo();
|
||||
}
|
||||
test_config_delete_timing_early_helper();
|
||||
test_config_is_remote_dest();
|
||||
|
||||
@@ -634,12 +634,65 @@ static void test_delta_hash_index_large_mostly_matching() {
|
||||
free(new_data);
|
||||
}
|
||||
|
||||
/* --checksum-seed: the delta strong (block) hash is genuinely seed-aware. A
|
||||
* nonzero seed changes the per-block xxHash32, and a signature + delta computed
|
||||
* with the same seed still reconstruct the file exactly (symmetric), while a
|
||||
* mismatched seed produces a delta that does not match the signature blocks. */
|
||||
static void test_delta_xxhash32_seeded() {
|
||||
const char* data = "seedme";
|
||||
uint32_t a = delta_xxhash32(data, 6);
|
||||
uint32_t b = delta_xxhash32_seeded(data, 6, 42);
|
||||
uint32_t c = delta_xxhash32_seeded(data, 6, 42);
|
||||
EXPECT_TRUE(a != b);
|
||||
EXPECT_EQ_INT((int)b, (int)c);
|
||||
/* Unseeded == seeded with 0 (default reproduces today's behavior). */
|
||||
EXPECT_EQ_INT((int)delta_xxhash32(data, 6), (int)delta_xxhash32_seeded(data, 6, 0));
|
||||
}
|
||||
|
||||
static void test_delta_seeded_signature_compute_matches() {
|
||||
uint32_t block_size = 1024;
|
||||
/* Identical old/new data with a non-zero seed: the receiver builds a seeded
|
||||
signature and the sender computes a seeded delta over the same bytes, so
|
||||
every block matches and applying the delta rebuilds the file exactly. */
|
||||
char data[4096];
|
||||
for (int i = 0; i < 4096; i++)
|
||||
data[i] = (char)(i % 256);
|
||||
|
||||
DeltaSignature* sig = delta_signature_create_seeded(data, 4096, block_size, 99);
|
||||
EXPECT_NOT_NULL(sig);
|
||||
Delta* delta = delta_compute_seeded(data, 4096, sig, block_size, 99);
|
||||
EXPECT_NOT_NULL(delta);
|
||||
void* rebuilt = delta_apply(data, 4096, delta, block_size);
|
||||
EXPECT_NOT_NULL(rebuilt);
|
||||
EXPECT_TRUE(memcmp(rebuilt, data, 4096) == 0);
|
||||
free(rebuilt);
|
||||
delta_destroy(delta);
|
||||
delta_signature_destroy(sig);
|
||||
|
||||
/* A MISMATCHED seed means the sender's window xxHash32 never equals the
|
||||
receiver's signature-block xxHash32: no block can match, so the delta is
|
||||
not worthwhile / has no block matches. This proves the seed really gates
|
||||
the block comparison rather than being an inert parameter. */
|
||||
sig = delta_signature_create_seeded(data, 4096, block_size, 99);
|
||||
EXPECT_NOT_NULL(sig);
|
||||
delta = delta_compute_seeded(data, 4096, sig, block_size, 7);
|
||||
EXPECT_NOT_NULL(delta);
|
||||
bool any_match = false;
|
||||
for (uint32_t i = 0; i < delta->instruction_count; i++)
|
||||
if (delta->instructions[i].type == DELTA_INSTR_BLOCK_MATCH)
|
||||
any_match = true;
|
||||
EXPECT_FALSE(any_match);
|
||||
delta_destroy(delta);
|
||||
delta_signature_destroy(sig);
|
||||
}
|
||||
|
||||
void test_delta() {
|
||||
test_adler32_basic();
|
||||
test_adler32_different_data();
|
||||
test_xxhash32_basic();
|
||||
test_xxhash32_different_data();
|
||||
test_xxhash64_different_data();
|
||||
test_delta_xxhash32_seeded();
|
||||
test_signature_roundtrip();
|
||||
test_delta_identical_files();
|
||||
test_delta_small_edit();
|
||||
@@ -653,4 +706,5 @@ void test_delta() {
|
||||
test_delta_apply_rejects_output_overflow();
|
||||
test_delta_hash_index_matches_linear_reference();
|
||||
test_delta_hash_index_large_mostly_matching();
|
||||
test_delta_seeded_signature_compute_matches();
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user