diff --git a/README.md b/README.md index 43fb369..802975e 100644 --- a/README.md +++ b/README.md @@ -117,9 +117,12 @@ matrix is classified as parity, caveat, or divergent in per-type `Number of files` breakdown is not reproduced. `--progress` prints rsync-style per-file blocks (without rsync's leading `./` line). - Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and - `xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums, negotiated with - `auto`; `zlibx` behaves as `zlib`, and the transfer checksum is not separately - selectable. + `xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums. `auto` honors + `RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` and otherwise follows rsync's + compiled-in order. An omitted `--compress-level` uses the codec's rsync + default (zstd 3, zlib/zlibx 6, lz4 ignored); `zlib`/`zlibx` share the + literal-only zlib path (rsync's zlibx semantics), and the transfer checksum is + not separately selectable. The detailed flag matrix is maintained in [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It reports each row as **parity**, @@ -505,8 +508,8 @@ features without changing the meaning of ordinary compatibility options. |---|---| | `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. | | `-z [level]`, `--compress [level]` | Enable streaming compression (default `zstd`), levels 1-22. | -| `--compress-level ` | Set the compression level. | -| `--zc ` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlibx` behaves as `zlib`. | +| `--compress-level ` | Set the compression level (1-22). Omitted, each codec uses its rsync default: zstd 3, zlib/zlibx 6, lz4 ignored. | +| `--zc ` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlib`/`zlibx` share the same literal-only zlib path. | | `--zl ` | Alias for `--compress-level`. | | `--skip-compress ` | Skip compression for `/`- or `,`-separated suffixes; defaults to rsync 3.4.1's built-in list. Incompatible with `--chunk-serialization`. | | `--compress-threads ` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. | diff --git a/RSYNC_COMPAT.md b/RSYNC_COMPAT.md index 3cba6e3..93e30ee 100644 --- a/RSYNC_COMPAT.md +++ b/RSYNC_COMPAT.md @@ -6,8 +6,8 @@ This document maps rsync's full feature set to FastSync's current implementation | Status | Count | Description | |--------|-------|-------------| -| ✅ Parity | 111 | Reproduces rsync's semantics for this option's scope | -| ⚠️ Caveat | 14 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) | +| ✅ Parity | 113 | Reproduces rsync's semantics for this option's scope | +| ⚠️ Caveat | 12 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) | | ❌ Divergent | 32 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call | | **Total** | **157** | One row per rsync option/feature group; a row may name several spellings | @@ -41,6 +41,8 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**. **Parity track 2b (no-wire, on `feat/parity-2.28`).** An opt-in, paths-only metadata pre-count for `--progress`/`-P`/`--info=progress` (only when not `--quiet`) now gives the `to-chk` denominator rsync's full file-list total (every regular file, directory, symlink and special plus the transfer root) and emits the per-directory/symlink/special name lines, in both the sequential and `--threads` paths; `--delete-during`/`--delete-delay` reuse their keep-set pre-scan so no second walk happens, and non-progress runs are unaffected. A fresh multi-directory differential against rsync 3.4.1 matches the name set and the `to-chk` denominator, while a single-file transfer stays byte-identical; the remaining caveats are emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream, so the `to-chk` numerator and interleaving differ) and re-run/receiver-state-driven over-naming (unconditional `./`, ancestor dirs emitted with a transferred child, and no quick-check for symlinks/empty dirs), keeping the row ⚠️. The matrix is unchanged at **111 ✅ / 14 ⚠️ / 32 ❌ = 157**. +**Parity track 3a (no-wire, on `feat/parity-2.28`).** Three codec refinements, none of which change the wire layout (`PROTOCOL_VERSION` stays 2.28.0): (1) an omitted `--compress-level` now resolves to rsync 3.4.1's per-codec default (zstd 3, zlib/zlibx 6, lz4 ignored) and an explicit level is clamped per codec (zstd 1-22, zlib/zlibx 1-9), verified against `rsync --debug=NSTR1`; (2) `auto` now honors `RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` (rsync's whitespace-separated preference syntax, unknown names skipped, first supported wins, all-unknown is exit 4) before the compiled-in order, with an explicit `--zc`/`--cc` still winning; and (3) `--compress-choice=zlibx` is reclassified out of the caveat list — FastSync's zlib stream already carries only the literal/delta bytes (rsync's zlibx semantics) and is observably identical to `--zc=zlib`, so the zlib/zlibx aliasing is only an implementation detail. The matrix is now **113 ✅ / 12 ⚠️ / 32 ❌ = 157**. + **Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the remaining gaps the rsync-parity wave left open (delete timing, wire counters and output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side @@ -649,7 +651,7 @@ targets verbatim, matching rsync. | Flag | Rsync Description | FastSync Status | Notes | |------|-------------------|-----------------|-------| | `--checksum` | Skip based on checksum | ✅ Parity | `-c`/`--checksum` compares per-file whole-file content digests to skip unchanged files. **As of protocol 2.23.0 the short `-c` implies the checksum quick-check**, so a plain `-c` run verifies content rather than only affecting the `--incremental` handshake. The digest algorithm is `xxh128` by default (protocol 2.26.0's negotiated default) and is selectable via `--checksum-choice`/`--cc` (`xxh128`/`xxh3`/`xxh64`/`xxhash`/`md5`/`md4`/`sha1`/`none`/`auto`, plus rsync's two-name form) and `--checksum-seed=NUM` (see those rows) | -| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ⚠️ Caveat | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and basis-dir verification. **Protocol 2.26.0 accepts rsync 3.4.1's full set** — `xxh128` (the negotiated default), `xxh3`, `xxh64`, `xxhash`, `md5`, `md4`, `sha1`, `none`, `auto`, and the two-name `transfer,pre-transfer` form — with rsync's exit-4 rejection of an unknown name and of `none` on the transfer side when `--checksum` is on. `--cc=ALG` and space forms both parse. The algorithm id and seed cross the wire; the receiver hashes its old file with the same algorithm+seed and the per-file `STATUS_CHECK` handshake carries a bounded digest pinned to the negotiated length. `checksum_digest_file` now streams **every** supported algorithm (md4 via the self-contained RFC 1320 code, sha1/md5 via EVP, none as an empty digest), so the streaming path matches its contract, and `--out-format %C` uses the selected **transfer** half of a two-name choice and renders each algorithm byte-for-byte like rsync (xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, none a blank 2-char column) — differential-tested across all algorithms. **Remaining divergences:** rsync uses this choice for the block checksum on the wire too, while FastSync selects only the whole-file comparison digest and keeps the delta BLOCK strong checksum at xxHash32; the `RSYNC_CHECKSUM_LIST` environment variable is not consulted; and `auto` always resolves deterministically to the first supported entry in rsync's preference order rather than probing the peer | +| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ⚠️ Caveat | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and basis-dir verification. **Protocol 2.26.0 accepts rsync 3.4.1's full set** — `xxh128` (the negotiated default), `xxh3`, `xxh64`, `xxhash`, `md5`, `md4`, `sha1`, `none`, `auto`, and the two-name `transfer,pre-transfer` form — with rsync's exit-4 rejection of an unknown name and of `none` on the transfer side when `--checksum` is on. `--cc=ALG` and space forms both parse. The algorithm id and seed cross the wire; the receiver hashes its old file with the same algorithm+seed and the per-file `STATUS_CHECK` handshake carries a bounded digest pinned to the negotiated length. `checksum_digest_file` now streams **every** supported algorithm (md4 via the self-contained RFC 1320 code, sha1/md5 via EVP, none as an empty digest), so the streaming path matches its contract, and `--out-format %C` uses the selected **transfer** half of a two-name choice and renders each algorithm byte-for-byte like rsync (xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, none a blank 2-char column) — differential-tested across all algorithms. **Remaining divergence:** rsync uses this choice for the block checksum on the wire too, while FastSync selects only the whole-file comparison digest and keeps the delta BLOCK strong checksum at xxHash32. `auto` now consults `RSYNC_CHECKSUM_LIST` (rsync's whitespace-separated preference list; unknown names skipped, first supported wins, all-unknown is exit 4) before the compiled-in order; because both peers run the identical build this deterministic resolution needs no rsync peer probe, and an explicit `--cc` still wins. The list is differential-tested through `--out-format %C` (byte-identical digests to rsync for md5/sha1/xxh3) | | `--compare-dest=DIR` | Compare dest files relative to DIR | ❌ Divergent | DIR is a receiver-side basis; protocol 2.26.0 uses an absolute path verbatim (rsync semantics) and resolves a relative path below the destination root (`..` components are rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. **Reclassified Divergent (differential evidence):** FastSync verifies a basis hit's content with xxHash64 while rsync's `--size-only` quick check trusts size (and mtime) alone, so with a same-size/different-content basis rsync skips/links the *wrong* basis content while FastSync transfers the source — a deliberate safety-stricter behavior that cannot match rsync (see `test_basis_dir_size_only_content_residual` in `tests/integration/test_parity_quickwins.py`). Attribute-only differences on a match are also not re-applied (data is skipped so the sender never sends metadata). Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) | | `--copy-dest=DIR` | Include copies of unchanged files | ❌ Divergent | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. **Reclassified Divergent** for the same basis-hit verification divergence as `--compare-dest`: rsync's `--size-only` size/quick-check match rings a same-size/different-content file as unchanged and copies the wrong basis bytes, while FastSync's xxHash verification transfers the source (differential test `test_basis_dir_size_only_content_residual`); a basis-hit also keeps whatever metadata the copy derived from the basis rather than rsync's "copy + fix attributes" in attribute-only cases. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 | | `--link-dest=DIR` | Hardlink to files when unchanged | ❌ Divergent | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. **Reclassified Divergent** for the shared basis-hit divergence: with `--size-only` a same-size/different-content basis is linked by rsync (installing wrong content) but FastSync detects the xxHash mismatch and transfers the source (differential test `test_basis_dir_size_only_content_residual`). Other inherent caveats: protocol 2.26.0 re-links an already up-to-date destination file to the basis; a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 | @@ -659,9 +661,9 @@ targets verbatim, matching rsync. | Flag | Rsync Description | FastSync Status | Notes | |------|-------------------|-----------------|-------| -| `-z`, `--compress` | Compress file data | ⚠️ Caveat | Streaming compression. **Protocol 2.26.0 implements rsync 3.4.1's codec set** (`zstd` default, `lz4`, `zlib`, `zlibx`, `none`), selectable via `--compress-choice`/`--zc` and negotiated with `auto`. `-z` is the compression short form; `-c` is rsync's `--checksum`. `--skip-compress` applies rsync 3.4.1's default suffix list when no list is given. **Remaining codec divergence:** `zlibx` is treated as `zlib`, per-codec level defaults are not mirrored, and `auto` does not probe the peer | -| `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ⚠️ Caveat | Protocol 2.26.0 accepts rsync 3.4.1's compiled-in choices — `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, `auto` — and rejects an unknown name with exit 4 like rsync. The negotiated codec id crosses the wire (`compression_algo`), so the receiver decodes with the sender's codec. `--zc` is the alias. **Remaining divergences:** `zlibx` behaves as `zlib` (there is no separate zlibx path), the per-codec compression-level defaults differ from rsync's, and `auto` always resolves to the first supported entry in rsync's preference order rather than probing the peer (no `RSYNC_COMPRESS_LIST` handling) | -| `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Parity | 1-22, default 5 | +| `-z`, `--compress` | Compress file data | ✅ Parity | Streaming compression. **Protocol 2.26.0 implements rsync 3.4.1's codec set** (`zstd` default, `lz4`, `zlib`, `zlibx`, `none`), selectable via `--compress-choice`/`--zc` and negotiated with `auto`. `-z` is the compression short form; `-c` is rsync's `--checksum`. `--skip-compress` applies rsync 3.4.1's default suffix list when no list is given. **Track 3a closes the codec caveats:** `zlibx` is no longer a divergence — FastSync's zlib stream already carries only the delta/token (literal) bytes, which is exactly rsync's zlibx semantics, so `--zc=zlib` and `--zc=zlibx` land the same tree/stdout/exit (differential `test_compress_codec_matches_rsync_bytes`) and the zlib/zlibx aliasing is only an implementation detail. Each codec now uses rsync's own default `--compress-level` (zstd 3, zlib/zlibx 6, lz4 ignored) and `auto` consults `RSYNC_COMPRESS_LIST` before the compiled-in order; the deterministic same-build resolution needs no peer probe | +| `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ✅ Parity | Protocol 2.26.0 accepts rsync 3.4.1's compiled-in choices — `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, `auto` — and rejects an unknown name with exit 4 like rsync. The negotiated codec id crosses the wire (`compression_algo`), so the receiver decodes with the sender's codec. `--zc` is the alias. `auto` now resolves through `RSYNC_COMPRESS_LIST` (whitespace-separated; unknown names skipped, first supported wins, all-unknown is exit 4) and then the compiled-in order, and an explicit `--zc` wins; the deterministic same-build resolution needs no peer probe. `zlib`/`zlibx` share FastSync's literal-only zlib path, which is rsync's zlibx behavior and is observably identical for both, so `zlibx` is not a divergence (the aliasing is an implementation detail) | +| `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Parity | Accepted range 1-22. When omitted, rsync 3.4.1's **per-codec default** applies: zstd 3 (`ZSTD_CLEVEL_DEFAULT`), zlib/zlibx 6 (`Z_DEFAULT_COMPRESSION` resolved), lz4 ignored (no tunable level; FastSync keeps a positive gate value and `lz4_compress` ignores it, so the bytes match rsync). An explicit level is clamped per codec like rsync's `init_compression_level()`: zstd 1-22, zlib/zlibx 1-9, lz4 ignored. Verified against `rsync --debug=NSTR1`, which reports the same effective level per codec | | `--compress-threads=NUM` | Set compression threads | ✅ Parity | `compression_threads` config field (client-only; does not cross the wire). Sets the number of worker threads used by the zstd compression pool to NUM (1..64; 0/garbage/oversized rejected up front). Accepted in both `--compress-threads=NUM` and two-argument `--compress-threads NUM` forms. Composes with `-z`/compression; under the `-j`/`--threads` multithreaded pipeline it parallelizes compressed chunk encoding. See test_tcp.py `-z --compress-threads=2` and test_client_cli.c | | `--skip-compress=LIST` | Skip compress for suffixes | ✅ Parity | Comma-separated (or `/`-separated, as in rsync) case-insensitive suffix list; a leading dot is optional; an empty list skips none. **When the option is omitted, rsync 3.4.1's built-in default suffix list applies** (`3g2 3gp 7z aac … zip zst`); an explicit list replaces that default entirely, matching rsync. A user-supplied list is a client-side compression choice; incompatible with FastSync chunk serialization (`-s`) | @@ -1111,10 +1113,11 @@ wire protocol three times (full rationale in `src/shared/config.h`): `sha1`, `none`, `auto`, plus the two-name `transfer,pre-transfer` form. Unknown names and `none` on the transfer side under `--checksum` exit 4 like rsync. -- **Remaining codec residuals:** `zlibx` behaves as `zlib`; the transfer - checksum is not independently selectable (only the whole-file comparison - digest is); per-codec level defaults differ; and `RSYNC_CHECKSUM_LIST`/ - `RSYNC_COMPRESS_LIST` are not consulted. +- **Codec residuals:** the transfer checksum is not independently selectable + (only the whole-file comparison digest is); `zlib`/`zlibx` share FastSync's + literal-only zlib path (rsync's zlibx behavior, observably identical for both, + so `zlibx` is not a divergence). Per-codec compression-level defaults and + `RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` are implemented (track 3a). ### Selection, paths, and filters diff --git a/src/client/client_cli.c b/src/client/client_cli.c index 05fb6df..14271ee 100644 --- a/src/client/client_cli.c +++ b/src/client/client_cli.c @@ -161,10 +161,16 @@ static int set_compression_choice(Config* config, const char* value) { return -1; } int algo; - if (strcasecmp(value, "auto") == 0) - algo = (int)compression_negotiate_default(); - else + if (strcasecmp(value, "auto") == 0) { + algo = compression_choice_resolve(); + if (algo < 0) { + log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm"); + config->cli_exit_code = 4; + return -1; + } + } else { algo = compression_algo_from_name(value); + } if (algo < 0) { log_message(LOG_LEVEL_ERROR, "--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, " @@ -228,16 +234,25 @@ static int set_checksum_choice(Config* config, const char* value) { config->cli_exit_code = 4; return -1; } - ChecksumAlgo negotiated = checksum_negotiate_default(); + int negotiated = -1; + if (rc1 == 1 || rc2 == 1) { + negotiated = checksum_choice_resolve(); + if (negotiated < 0) { + log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm"); + config->cli_exit_code = 4; + return -1; + } + } if (rc1 == 1) - transfer = (int)negotiated; + transfer = negotiated; if (!name2) pre = transfer; else if (rc2 == 1) - pre = (int)negotiated; + pre = negotiated; config->checksum_algo = pre; config->checksum_transfer_algo = transfer; + config->checksum_choice_set = true; /* rsync: "none" for the transfer checksum forces --whole-file. */ if (transfer == (int)CHECKSUM_ALGO_NONE) config->whole_file = true; @@ -1464,6 +1479,8 @@ static bool cli_handle_table_option(CliParseCtx* ctx) { ctx->exit_code = -1; return true; } + if (entry->offset == offsetof(Config, compression_level)) + config->compression_level_set = true; if (entry->offset == offsetof(Config, chmod_spec)) { mode_t ignored; if (!chmod_apply(0, config->chmod_spec, &ignored)) { @@ -1748,6 +1765,7 @@ static bool cli_handle_transfer_flags(CliParseCtx* ctx) { return true; } config->compression_level = (int)level; + config->compression_level_set = true; log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level); ctx->i++; } @@ -2550,8 +2568,43 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE); } } - if (config->use_compression && config->compression_algo == (int)COMPRESSION_ALGO_NONE) - config->compression_algo = (int)compression_negotiate_default(); + /* A bare -z (no --compress-choice) resolves like rsync's "auto": the + * RSYNC_COMPRESS_LIST preference list first, then the compiled-in order. A + * list that names no supported codec is rsync's failed negotiation (exit 4). */ + if (config->use_compression && !config->compress_choice) { + int resolved = compression_choice_resolve(); + if (resolved < 0) { + log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm"); + config->cli_exit_code = 4; + return -1; + } + config->compression_algo = resolved; + if (resolved == (int)COMPRESSION_ALGO_NONE) + config->use_compression = false; + } + /* Apply rsync's per-codec compression level: an explicit --compress-level is + * clamped to the codec's range, otherwise the codec's own default is used. */ + if (config->use_compression) { + CompressionAlgo algo = (CompressionAlgo)config->compression_algo; + config->compression_level = config->compression_level_set + ? compression_clamp_level(algo, config->compression_level) + : compression_default_level(algo); + log_debug_message(LOG_DEBUG_UTIL, "Client compression: %s (level %d)", + compression_algo_name(algo), config->compression_level); + } + /* The negotiated checksum is always resolved (rsync negotiates one for the + * delta strong sum even without --checksum): RSYNC_CHECKSUM_LIST first, then + * the compiled-in order. An explicit --checksum-choice already set it. */ + if (!config->checksum_choice_set) { + int resolved = checksum_choice_resolve(); + if (resolved < 0) { + log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm"); + config->cli_exit_code = 4; + return -1; + } + config->checksum_algo = resolved; + config->checksum_transfer_algo = resolved; + } /* rsync parity: "none" as the pre-transfer checksum cannot be combined with * --checksum (exit 4). The check runs here because --checksum may appear on * either side of --checksum-choice. */ diff --git a/src/shared/checksum.c b/src/shared/checksum.c index 81dcb54..d5401bc 100644 --- a/src/shared/checksum.c +++ b/src/shared/checksum.c @@ -1,4 +1,5 @@ #include "checksum.h" +#include "utils.h" #include #include #include @@ -396,7 +397,7 @@ uint8_t checksum_digest_len(ChecksumAlgo algo) { return 0; } -ChecksumAlgo checksum_negotiate_default(void) { +static ChecksumAlgo compiled_checksum_preference_first(void) { /* rsync 3.4.1 default preference order; every entry is compiled in, so this * resolves to xxh128. */ static const ChecksumAlgo preference[] = { @@ -409,3 +410,16 @@ ChecksumAlgo checksum_negotiate_default(void) { } return CHECKSUM_ALGO_XXH64; } + +int checksum_choice_resolve(void) { + bool specified = false; + int env = env_choice_first("RSYNC_CHECKSUM_LIST", checksum_algo_from_name, &specified); + if (specified) + return env; /* -1 = the list named no supported checksum */ + return (int)compiled_checksum_preference_first(); +} + +ChecksumAlgo checksum_negotiate_default(void) { + int resolved = checksum_choice_resolve(); + return resolved >= 0 ? (ChecksumAlgo)resolved : compiled_checksum_preference_first(); +} diff --git a/src/shared/checksum.h b/src/shared/checksum.h index 36deebc..a273ab3 100644 --- a/src/shared/checksum.h +++ b/src/shared/checksum.h @@ -72,4 +72,11 @@ uint8_t checksum_digest_len(ChecksumAlgo algo); * xxh128 xxh3 xxh64 md5 md4 sha1 none). Used to resolve "auto". */ ChecksumAlgo checksum_negotiate_default(void); +/* Resolve "auto" the way rsync does: the first supported name in + * RSYNC_CHECKSUM_LIST (whitespace-separated, client half ends at '&'), then the + * compiled-in preference order when the variable is unset/blank. Returns -1 + * when the variable is set but names no supported checksum (rsync's failed + * negotiation), otherwise a valid ChecksumAlgo id. */ +int checksum_choice_resolve(void); + #endif /* CHECKSUM_H */ diff --git a/src/shared/compression.c b/src/shared/compression.c index 428b67b..c1fb5c9 100644 --- a/src/shared/compression.c +++ b/src/shared/compression.c @@ -2,6 +2,7 @@ #include "data.h" #include "log.h" #include "protocol.h" +#include "utils.h" #include #include #include @@ -119,7 +120,7 @@ bool compression_algo_enabled(CompressionAlgo algo) { return algo != COMPRESSION_ALGO_NONE; } -CompressionAlgo compression_negotiate_default(void) { +static CompressionAlgo compiled_preference_first(void) { /* rsync 3.4.1 default preference order; every entry is compiled in, so this * resolves to zstd. */ static const CompressionAlgo preference[] = { @@ -133,6 +134,57 @@ CompressionAlgo compression_negotiate_default(void) { return COMPRESSION_ALGO_ZSTD; } +int compression_choice_resolve(void) { + bool specified = false; + int env = env_choice_first("RSYNC_COMPRESS_LIST", compression_algo_from_name, &specified); + if (specified) + return env; /* -1 = the list named no supported codec */ + return (int)compiled_preference_first(); +} + +CompressionAlgo compression_negotiate_default(void) { + int resolved = compression_choice_resolve(); + return resolved >= 0 ? (CompressionAlgo)resolved : compiled_preference_first(); +} + +int compression_default_level(CompressionAlgo algo) { + switch (algo) { + case COMPRESSION_ALGO_ZSTD: + return ZSTD_CLEVEL_DEFAULT; + case COMPRESSION_ALGO_ZLIB: + case COMPRESSION_ALGO_ZLIBX: + return 6; /* rsync resolves zlib's Z_DEFAULT_COMPRESSION (-1) to 6 */ + case COMPRESSION_ALGO_LZ4: + return 1; /* rsync lz4 level is 0/ignored; positive keeps the gate on */ + case COMPRESSION_ALGO_NONE: + return 0; + } + return 0; +} + +int compression_clamp_level(CompressionAlgo algo, int level) { + switch (algo) { + case COMPRESSION_ALGO_ZSTD: + if (level < 1) + return 1; + if (level > 22) + return 22; + return level; + case COMPRESSION_ALGO_ZLIB: + case COMPRESSION_ALGO_ZLIBX: + if (level < 1) + return 1; + if (level > 9) + return 9; + return level; + case COMPRESSION_ALGO_LZ4: + return 1; /* ignored by lz4_compress; keeps the "compress" gate on */ + case COMPRESSION_ALGO_NONE: + return 0; + } + return level; +} + void compression_set_algo(CompressionAlgo algo) { if (compression_algo_valid((int)algo)) atomic_store(&g_compression_algo, (int)algo); diff --git a/src/shared/compression.h b/src/shared/compression.h index 3e5d928..ca874b8 100644 --- a/src/shared/compression.h +++ b/src/shared/compression.h @@ -35,6 +35,26 @@ bool compression_algo_valid(int algo); * "auto". */ CompressionAlgo compression_negotiate_default(void); +/* Resolve "auto" the way rsync does: the first supported name in + * RSYNC_COMPRESS_LIST (whitespace-separated, client half ends at '&'), then the + * compiled-in preference order when the variable is unset/blank. Returns -1 + * when the variable is set but names no supported codec (rsync's failed + * negotiation), otherwise a valid CompressionAlgo id. */ +int compression_choice_resolve(void); + +/* rsync 3.4.1's per-codec default level, applied when the user did not pass + * --compress-level/--zl. zstd uses ZSTD_CLEVEL_DEFAULT (3) and zlib/zlibx the + * resolved Z_DEFAULT_COMPRESSION (6). lz4 has no tunable level in rsync + * (always the default acceleration); FastSync returns a positive placeholder so + * its "level > 0" compression gate stays engaged, and lz4_compress ignores the + * value, so the output is identical to rsync's. none is 0. */ +int compression_default_level(CompressionAlgo algo); + +/* Clamp an explicit --compress-level to the codec's accepted range the way + * rsync's init_compression_level() does: zstd 1..22, zlib/zlibx 1..9, lz4 + * ignored (fixed positive placeholder), none 0. */ +int compression_clamp_level(CompressionAlgo algo, int level); + /* True when the algorithm actually compresses (i.e. is not NONE). */ bool compression_algo_enabled(CompressionAlgo algo); diff --git a/src/shared/config.c b/src/shared/config.c index 6b065c1..da7694f 100644 --- a/src/shared/config.c +++ b/src/shared/config.c @@ -70,6 +70,8 @@ static void config_set_defaults(Config* config) { config->ignore_missing_args = false; config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT; config->cli_exit_code = 0; + config->compression_level_set = false; + config->checksum_choice_set = false; config->filters = NULL; config->files_from = NULL; config->files_from_set = NULL; diff --git a/src/shared/config.h b/src/shared/config.h index 6cb9efe..dbb5bf7 100644 --- a/src/shared/config.h +++ b/src/shared/config.h @@ -419,6 +419,12 @@ typedef struct Config { * for an unsupported checksum/compress algorithm) so main() can mirror it. */ int checksum_transfer_algo; int cli_exit_code; + /* Client-only "the user explicitly chose" bits. They let the per-codec + * default level / checksum list be applied only when the corresponding + * rsync option was omitted (an explicit --compress-level / --checksum-choice + * always wins). Never serialized. */ + bool compression_level_set; + bool checksum_choice_set; // Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are // never serialized to the wire (the receiver must not learn them). diff --git a/src/shared/utils.c b/src/shared/utils.c index fb01f5f..aec553b 100644 --- a/src/shared/utils.c +++ b/src/shared/utils.c @@ -2,6 +2,7 @@ #include "array_list.h" #include "log.h" #include +#include #include #include #include @@ -142,6 +143,47 @@ char* str_dup(const char* string) { return new_string; } +int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified) { + if (specified) + *specified = false; + if (!env_name || !resolve) + return -1; + const char* env = getenv(env_name); + if (!env) + return -1; + + bool saw_nonblank = false; + const char* p = env; + while (*p) { + if (*p == '&') + break; + if (isspace((unsigned char)*p)) { + p++; + continue; + } + saw_nonblank = true; + char token[64]; + size_t len = 0; + while (*p && *p != '&' && !isspace((unsigned char)*p)) { + if (len < sizeof(token) - 1) + token[len++] = *p; + p++; + } + token[len] = '\0'; + if (len > 0) { + int id = resolve(token); + if (id >= 0) { + if (specified) + *specified = true; + return id; + } + } + } + if (specified) + *specified = saw_nonblank; + return -1; +} + #define STR_HASH_SET_MIN_CAPACITY 16 static size_t str_hash_set_hash(const char* key, size_t len) { diff --git a/src/shared/utils.h b/src/shared/utils.h index c01548b..b70f44c 100644 --- a/src/shared/utils.h +++ b/src/shared/utils.h @@ -81,6 +81,17 @@ bool path_index_has_descendant(const PathIndex* index, const char* path); char* str_dup(const char* string); char* output_escape(const char* string, bool eight_bit_output); + +/* Resolve the first supported name from a rsync algorithm-preference + * environment variable (RSYNC_COMPRESS_LIST / RSYNC_CHECKSUM_LIST). `resolve` + * maps a case-insensitive name to an algorithm id (>= 0) or -1 for an unknown + * name. rsync's syntax is a whitespace-separated list (comma/colon are NOT + * separators); the client-side half ends at '&'. Unknown entries are skipped + * and the first resolvable one wins. *specified is set true when the variable + * holds at least one non-blank character. Returns the first resolvable id, or + * -1 when the variable is unset/blank or names no supported algorithm. */ +int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified); + /* Upper bound on one line/token read from a local list file (--files-from, * --exclude-from/--include-from, .rsync-filter). Mirrors MAX_STRING_SIZE and * stops a hostile multi-gigabyte line from forcing unbounded allocation. */ diff --git a/tests/integration/test_client_cli.py b/tests/integration/test_client_cli.py new file mode 100644 index 0000000..91d582f --- /dev/null +++ b/tests/integration/test_client_cli.py @@ -0,0 +1,213 @@ +"""Differential tests for the client CLI's codec defaults and env lists. + +Track 3a of the rsync-parity plan pins two rsync 3.4.1 behaviors that are +resolved entirely on the client: + +* the per-codec default ``--compress-level`` (zstd 3, zlib/zlibx 6, lz4 + ignored) applied when the user omits ``--compress-level``/``--zl``, with an + explicit level clamped to the codec's range; and +* the ``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` preference lists that + rsync's ``auto`` consults before its compiled-in order (whitespace-separated, + unknown names skipped, first supported wins, all-unknown is exit 4). + +The rsync side is observed through ``--debug=NSTR1``; FastSync publishes its +resolved codec/level through ``--debug=util``. The checksum side is confirmed +byte-for-byte through ``--out-format %C``. The rsync-based tests skip cleanly +when rsync is not installed. +""" +import os +import re +import shutil +import subprocess +import sys + +import pytest + +sys.path.insert(0, os.path.dirname(__file__)) +from common import ( + TEST_DATA_DIR, + run_client, + clean_dir, + get_dest_received_dir, +) + +RSYNC = shutil.which("rsync") +requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed") + +CODEC_ROOT = os.path.join(TEST_DATA_DIR, "cli_differential") + +_COMPRESS_RE = re.compile(r"compress(?:ion)?: (\w+) \(level (-?\d+)\)") + + +def _rsync(args): + env = dict(os.environ, LC_ALL="C") + return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120) + + +def _scratch(tag): + path = os.path.join(CODEC_ROOT, tag) + clean_dir(path) + os.makedirs(path, exist_ok=True) + return path + + +def _make_corpus(root): + clean_dir(root) + os.makedirs(root, exist_ok=True) + with open(os.path.join(root, "big.bin"), "wb") as fh: + fh.write(b"FastSync codec payload " * 4096) + with open(os.path.join(root, "small.txt"), "wb") as fh: + fh.write(b"hello codec world\n" * 32) + return root + + +def _rsync_compress_level(choice, level): + src = _make_corpus(_scratch(f"lvl_src_{choice}_{level}")) + dst = _scratch(f"lvl_rsync_{choice}_{level}") + args = ["-a", "-z", f"--zc={choice}"] + if level is not None: + args.append(f"--zl={level}") + args += ["--debug=NSTR1", src + "/", dst + "/"] + result = _rsync(args) + assert result.returncode == 0, result.stderr + match = _COMPRESS_RE.search(result.stdout + result.stderr) + assert match, (result.stdout, result.stderr) + return match.group(1), int(match.group(2)) + + +def _fastsync_compress_level(choice, level, shared_server): + src = _make_corpus(_scratch(f"lvl_src_fs_{choice}_{level}")) + dst = _scratch(f"lvl_fs_{choice}_{level}") + args = ["-a", "-z", f"--zc={choice}"] + if level is not None: + args.append(f"--zl={level}") + args += ["-v", "--debug=util"] + result, _ = run_client(src, dst, flags=args, port=shared_server.port) + assert result.returncode == 0, (result.stderr or result.stdout)[:300] + match = _COMPRESS_RE.search(result.stdout) + assert match, result.stdout[:500] + return match.group(1), int(match.group(2)) + + +class TestPerCodecCompressionLevelDefaults: + """``--compress-level`` defaults and clamping match rsync per codec.""" + + # FastSync uses a positive lz4 placeholder because its "level > 0" gate + # enables compression; lz4_compress ignores the value, so rsync's level 0 + # and FastSync's level 1 produce the same bytes. + CASES = [ + ("zstd", None, 3), + ("zlib", None, 6), + ("zlibx", None, 6), + ("lz4", None, 1), + ("zstd", 10, 10), + ("zlib", 15, 9), + ("zlib", 3, 3), + ("lz4", 15, 1), + ] + + @requires_rsync + @pytest.mark.ci + @pytest.mark.parametrize("choice,level,fs_level", CASES) + def test_level_matches_rsync(self, choice, level, fs_level, shared_server): + rsync_algo, rsync_level = _rsync_compress_level(choice, level) + fs_algo, fs_level_actual = _fastsync_compress_level(choice, level, shared_server) + assert rsync_algo == choice + assert fs_algo == choice + if choice == "lz4": + assert rsync_level == 0 and fs_level_actual > 0 + else: + assert rsync_level == fs_level + assert fs_level_actual == fs_level + + +class TestEnvPreferenceLists: + """``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` drive auto like rsync.""" + + # (env value, expected codec, rsync level, FastSync level) + COMPRESS_CASES = [ + ("zlib lz4", "zlib", 6, 6), + ("lz4 zstd", "lz4", 0, 1), + ("bogus zstd zlib", "zstd", 3, 3), + (" ", "zstd", 3, 3), + ] + + @requires_rsync + @pytest.mark.ci + @pytest.mark.parametrize("env,algo,rsync_level,fs_level", COMPRESS_CASES) + def test_compress_list_matches_rsync(self, env, algo, rsync_level, fs_level, shared_server, + monkeypatch): + monkeypatch.setenv("RSYNC_COMPRESS_LIST", env) + src = _make_corpus(_scratch(f"envc_src_{algo}")) + rdst = _scratch(f"envc_rsync_{algo}") + rs = _rsync(["-a", "-z", "--debug=NSTR1", src + "/", rdst + "/"]) + assert rs.returncode == 0, rs.stderr + rm = _COMPRESS_RE.search(rs.stdout + rs.stderr) + assert rm, (rs.stdout, rs.stderr) + assert rm.group(1) == algo + assert int(rm.group(2)) == rsync_level + + fdst = _scratch(f"envc_fs_{algo}") + result, _ = run_client(src, fdst, flags=["-a", "-z", "-v", "--debug=util"], + port=shared_server.port) + assert result.returncode == 0, (result.stderr or result.stdout)[:300] + fm = _COMPRESS_RE.search(result.stdout) + assert fm, result.stdout[:500] + assert fm.group(1) == algo + assert int(fm.group(2)) == fs_level + received = get_dest_received_dir(fdst, src) + assert _tree_bytes(received) == _tree_bytes(src) + + @requires_rsync + @pytest.mark.ci + @pytest.mark.parametrize("env,algo", [("md5", "md5"), ("sha1", "sha1"), ("xxh3 md5", "xxh3")]) + def test_checksum_list_matches_rsync(self, env, algo, shared_server, monkeypatch): + monkeypatch.setenv("RSYNC_CHECKSUM_LIST", env) + src = _make_corpus(_scratch(f"envcc_src_{algo}")) + rdst = _scratch(f"envcc_rsync_{algo}") + rs = _rsync(["-a", "--checksum", "--out-format=%C %n", src + "/", rdst + "/"]) + assert rs.returncode == 0, rs.stderr + rs_digests = _digests(rs.stdout) + + fdst = _scratch(f"envcc_fs_{algo}") + result, _ = run_client(src, fdst, flags=["-a", "--checksum", "--out-format=%C %n"], + port=shared_server.port) + assert result.returncode == 0, (result.stderr or result.stdout)[:300] + assert _digests(result.stdout) == rs_digests + + @requires_rsync + @pytest.mark.ci + def test_all_unknown_lists_fail_like_rsync(self, shared_server, monkeypatch): + src = _make_corpus(_scratch("envbad_src")) + monkeypatch.setenv("RSYNC_COMPRESS_LIST", "bogus") + rs = _rsync(["-a", "-z", src + "/", _scratch("envbad_rsync_c") + "/"]) + assert rs.returncode == 4, rs.stderr + result, _ = run_client(src, _scratch("envbad_fs_c"), flags=["-a", "-z"], + port=shared_server.port) + assert result.returncode == 4, (result.stderr or result.stdout)[:200] + + monkeypatch.setenv("RSYNC_CHECKSUM_LIST", "bogus") + rs = _rsync(["-a", src + "/", _scratch("envbad_rsync_s") + "/"]) + assert rs.returncode == 4, rs.stderr + result, _ = run_client(src, _scratch("envbad_fs_s"), flags=["-a"], + port=shared_server.port) + assert result.returncode == 4, (result.stderr or result.stdout)[:200] + + +def _tree_bytes(root): + out = {} + for dirpath, _dirs, files in os.walk(root): + for name in files: + path = os.path.join(dirpath, name) + with open(path, "rb") as fh: + out[os.path.relpath(path, root)] = fh.read() + return out + + +def _digests(output): + out = {} + for line in output.splitlines(): + parts = line.split() + if len(parts) == 2 and parts[0]: + out[parts[1]] = parts[0] + return out diff --git a/tests/runner.c b/tests/runner.c index 5a56c36..de1b4b5 100644 --- a/tests/runner.c +++ b/tests/runner.c @@ -43,6 +43,7 @@ #include "test_utils.h" #include "test_xattr.h" #include +#include #include // Define global test state variables @@ -52,6 +53,11 @@ bool current_test_failed = false; int main() { signal(SIGPIPE, SIG_IGN); + /* The codec/checksum resolvers consult rsync's preference-list environment + * variables; clear them so a developer's shell cannot change test outcomes. + * The env-specific tests set and restore their own values. */ + unsetenv("RSYNC_COMPRESS_LIST"); + unsetenv("RSYNC_CHECKSUM_LIST"); printf("\033[1;36m=== RUNNING UNIT TESTS ===\033[0m\n\n"); RUN_TEST(test_queue); diff --git a/tests/test_checksum.c b/tests/test_checksum.c index 26eea3c..81107ea 100644 --- a/tests/test_checksum.c +++ b/tests/test_checksum.c @@ -265,6 +265,28 @@ static void test_checksum_digest_file_matches_oneshot(void) { free(data); } +/* RSYNC_CHECKSUM_LIST precedence, syntax and fallback. */ +static void test_checksum_choice_env_list() { + unsetenv("RSYNC_CHECKSUM_LIST"); + EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128); + EXPECT_EQ_INT((int)checksum_negotiate_default(), (int)CHECKSUM_ALGO_XXH128); + + setenv("RSYNC_CHECKSUM_LIST", "bogus md5 xxh3", 1); + EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_MD5); + + setenv("RSYNC_CHECKSUM_LIST", "SHA1", 1); + EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_SHA1); + + /* Whitespace-separated only: comma is not a separator in rsync's syntax. */ + setenv("RSYNC_CHECKSUM_LIST", "md5,xxh3", 1); + EXPECT_EQ_INT(checksum_choice_resolve(), -1); + + setenv("RSYNC_CHECKSUM_LIST", " ", 1); + EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128); + + unsetenv("RSYNC_CHECKSUM_LIST"); +} + void test_checksum(void) { test_checksum_xxh64_seed0(); test_checksum_xxh64_empty(); @@ -281,4 +303,5 @@ void test_checksum(void) { test_checksum_truncated_buffer_rejected(); test_checksum_null_empty_digest(); test_checksum_digest_file_matches_oneshot(); + test_checksum_choice_env_list(); } diff --git a/tests/test_client_cli.c b/tests/test_client_cli.c index b0f1694..d00df46 100644 --- a/tests/test_client_cli.c +++ b/tests/test_client_cli.c @@ -2431,6 +2431,119 @@ static void test_parse_args_rejects_invalid_compression_choice() { config_delete(cfg); } +/* rsync gives each codec its own default --compress-level; an omitted level + * resolves to that default and an explicit one is clamped to the codec range. */ +static void test_parse_args_per_codec_compression_level_defaults() { + unsetenv("RSYNC_COMPRESS_LIST"); + struct { + const char* choice; + int level; + } cases[] = { + {"zstd", 3}, + {"zlib", 6}, + {"zlibx", 6}, + {"lz4", 1}, + }; + for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) { + Config* cfg = config_create(); + char* argv[] = {"fastsync", "-z", "--compress-choice", (char*)cases[i].choice, "/src", "/dst"}; + int positional_args[2]; + int positional_count = 0; + EXPECT_EQ_INT(parse_args(cfg, 6, argv, positional_args, &positional_count), 0); + EXPECT_TRUE(cfg->use_compression); + EXPECT_EQ_INT(cfg->compression_level, cases[i].level); + config_delete(cfg); + } + + /* Bare -z resolves to the zstd default. */ + Config* cfg = config_create(); + char* bare[] = {"fastsync", "-z", "/src", "/dst"}; + int positional_args[2]; + int positional_count = 0; + EXPECT_EQ_INT(parse_args(cfg, 4, bare, positional_args, &positional_count), 0); + EXPECT_EQ_INT(cfg->compression_level, 3); + config_delete(cfg); + + /* An explicit level wins unchanged for zstd... */ + cfg = config_create(); + char* zv[] = {"fastsync", "-z", "--compress-level", "10", "/src", "/dst"}; + positional_count = 0; + EXPECT_EQ_INT(parse_args(cfg, 5, zv, positional_args, &positional_count), 0); + EXPECT_EQ_INT(cfg->compression_level, 10); + config_delete(cfg); + + /* ...but zlib clamps an over-range level to 9 like rsync. */ + cfg = config_create(); + char* zc[] = {"fastsync", "-z", "--compress-choice", "zlib", "--compress-level", "15", + "/src", "/dst"}; + positional_count = 0; + EXPECT_EQ_INT(parse_args(cfg, 8, zc, positional_args, &positional_count), 0); + EXPECT_EQ_INT(cfg->compression_level, 9); + config_delete(cfg); +} + +/* RSYNC_COMPRESS_LIST drives the bare -z ("auto") resolution. */ +static void test_parse_args_compression_env_list() { + Config* cfg = config_create(); + char* argv[] = {"fastsync", "-z", "/src", "/dst"}; + int positional_args[2]; + int positional_count = 0; + + setenv("RSYNC_COMPRESS_LIST", "zlib lz4", 1); + EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0); + EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZLIB); + EXPECT_EQ_INT(cfg->compression_level, 6); + config_delete(cfg); + + /* An explicit --compress-choice beats the env list. */ + cfg = config_create(); + char* explicit_argv[] = {"fastsync", "-z", "--compress-choice", "zstd", "/src", "/dst"}; + positional_count = 0; + EXPECT_EQ_INT(parse_args(cfg, 6, explicit_argv, positional_args, &positional_count), 0); + EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZSTD); + config_delete(cfg); + + /* A list with no supported name is rsync's failed negotiation (exit 4). */ + setenv("RSYNC_COMPRESS_LIST", "bogus", 1); + cfg = config_create(); + positional_count = 0; + EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1); + EXPECT_EQ_INT(cfg->cli_exit_code, 4); + config_delete(cfg); + unsetenv("RSYNC_COMPRESS_LIST"); +} + +/* RSYNC_CHECKSUM_LIST drives the default checksum choice. */ +static void test_parse_args_checksum_env_list() { + Config* cfg = config_create(); + char* argv[] = {"fastsync", "--checksum", "/src", "/dst"}; + int positional_args[2]; + int positional_count = 0; + + setenv("RSYNC_CHECKSUM_LIST", "md5", 1); + EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0); + EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD5); + EXPECT_EQ_INT(cfg->checksum_transfer_algo, (int)CHECKSUM_ALGO_MD5); + config_delete(cfg); + + /* An explicit --cc wins. */ + cfg = config_create(); + char* cc_argv[] = {"fastsync", "--checksum", "--cc=sha1", "/src", "/dst"}; + positional_count = 0; + EXPECT_EQ_INT(parse_args(cfg, 5, cc_argv, positional_args, &positional_count), 0); + EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_SHA1); + config_delete(cfg); + + /* A list with no supported name is rsync's failed negotiation (exit 4). */ + setenv("RSYNC_CHECKSUM_LIST", "bogus", 1); + cfg = config_create(); + positional_count = 0; + EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1); + EXPECT_EQ_INT(cfg->cli_exit_code, 4); + config_delete(cfg); + unsetenv("RSYNC_CHECKSUM_LIST"); +} + /* Every value-taking table option accepts an inline "--opt=value" form. */ static void test_parse_args_table_equals_size_options() { Config* cfg = config_create(); @@ -4682,6 +4795,9 @@ void test_client_cli() { test_parse_args_compression_alias_equals(); test_parse_args_rejects_invalid_compression_level_equals(); test_parse_args_rejects_invalid_compression_choice(); + test_parse_args_per_codec_compression_level_defaults(); + test_parse_args_compression_env_list(); + test_parse_args_checksum_env_list(); test_parse_args_table_equals_size_options(); test_parse_args_table_equals_string_and_int_options(); test_parse_args_missing_argument_diagnostic(); diff --git a/tests/test_compression.c b/tests/test_compression.c index 833835e..1e35293 100644 --- a/tests/test_compression.c +++ b/tests/test_compression.c @@ -4,6 +4,7 @@ #include "data.h" #include "file.h" #include "utils.h" +#include #include #include #include @@ -390,6 +391,56 @@ static void test_codec_name_mapping() { EXPECT_TRUE(compression_algo_enabled(COMPRESSION_ALGO_ZSTD)); } +/* rsync 3.4.1's per-codec default levels and its clamping ranges. */ +static void test_codec_level_defaults_and_clamp() { + EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), ZSTD_CLEVEL_DEFAULT); + EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), 3); + EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIB), 6); + EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIBX), 6); + /* lz4 has no tunable level; a positive placeholder keeps the codec engaged. */ + EXPECT_TRUE(compression_default_level(COMPRESSION_ALGO_LZ4) > 0); + EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_NONE), 0); + + EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 1), 1); + EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 22), 22); + EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 23), 22); + EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 0), 1); + EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 15), 9); + EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIBX, 15), 9); + EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 1), 1); + EXPECT_TRUE(compression_clamp_level(COMPRESSION_ALGO_LZ4, 20) > 0); + EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_NONE, 20), 0); +} + +/* RSYNC_COMPRESS_LIST precedence, syntax and fallback. */ +static void test_codec_choice_env_list() { + unsetenv("RSYNC_COMPRESS_LIST"); + EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD); + EXPECT_EQ_INT((int)compression_negotiate_default(), (int)COMPRESSION_ALGO_ZSTD); + + /* Unknown entries are skipped; the first supported wins. */ + setenv("RSYNC_COMPRESS_LIST", "bogus zlib lz4", 1); + EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZLIB); + + /* Case-insensitive. */ + setenv("RSYNC_COMPRESS_LIST", "ZSTD", 1); + EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD); + + /* Whitespace-separated; the client half ends at '&'. */ + setenv("RSYNC_COMPRESS_LIST", "lz4 zlib & zstd", 1); + EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_LZ4); + + /* Blank falls back to the compiled-in order. */ + setenv("RSYNC_COMPRESS_LIST", " ", 1); + EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD); + + /* rsync's syntax has no comma/colon separator: this is one unknown name. */ + setenv("RSYNC_COMPRESS_LIST", "bogus,lz4", 1); + EXPECT_EQ_INT(compression_choice_resolve(), -1); + + unsetenv("RSYNC_COMPRESS_LIST"); +} + /* The process-global codec selects what the legacy wrappers produce. */ static void test_codec_global_selection() { Data* original = data_create_empty(64); @@ -423,5 +474,7 @@ void test_compression() { test_chunk_compress_decompress_roundtrip(); test_codec_roundtrips(); test_codec_name_mapping(); + test_codec_level_defaults_and_clamp(); + test_codec_choice_env_list(); test_codec_global_selection(); } diff --git a/tests/test_shared_utils.c b/tests/test_shared_utils.c index af4ab15..ff0e3c0 100644 --- a/tests/test_shared_utils.c +++ b/tests/test_shared_utils.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -578,7 +579,43 @@ static void test_getdelim_bounded() { fclose(fp); } +static int test_env_resolver(const char* name) { + if (strcasecmp(name, "alpha") == 0) + return 10; + if (strcasecmp(name, "beta") == 0) + return 20; + return -1; +} + +static void test_env_choice_first_parsing() { + const char* var = "FASTSYNC_TEST_CHOICE_LIST"; + bool specified = true; + unsetenv(var); + EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1); + EXPECT_FALSE(specified); + + /* Unknown entries are skipped, case-insensitive, first supported wins. */ + setenv(var, "bogus BETA alpha", 1); + EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 20); + EXPECT_TRUE(specified); + + /* The client half ends at '&'. */ + setenv(var, "alpha & beta", 1); + EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 10); + + /* Blank means "unspecified"; all-unknown means "specified but no match". */ + setenv(var, " ", 1); + EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1); + EXPECT_FALSE(specified); + setenv(var, "nope,alpha", 1); + EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1); + EXPECT_TRUE(specified); + + unsetenv(var); +} + void test_shared_utils() { + test_env_choice_first_parsing(); test_path_index_bounded(); test_path_index_semantics(); test_getdelim_bounded();