feat(codecs): per-codec level defaults, RSYNC_*_LIST auto, zlibx reclassify
- rsync 3.4.1 per-codec defaults (zstd 3, zlib/zlibx 6, lz4 level ignored) and per-codec clamping; explicit --zl still wins - auto resolves via whitespace-separated RSYNC_COMPRESS_LIST / RSYNC_CHECKSUM_LIST (first supported wins; all-unknown exits 4) - zlibx reclassified: FastSync's zlib stream already excludes matched data, so its tree/stdout/exit match zlib - --compress/-z and --compress-choice flip to parity (113/12/32)
This commit is contained in:
@@ -117,9 +117,12 @@ matrix is classified as parity, caveat, or divergent in
|
|||||||
per-type `Number of files` breakdown is not reproduced. `--progress` prints
|
per-type `Number of files` breakdown is not reproduced. `--progress` prints
|
||||||
rsync-style per-file blocks (without rsync's leading `./` line).
|
rsync-style per-file blocks (without rsync's leading `./` line).
|
||||||
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
|
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
|
||||||
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums, negotiated with
|
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums. `auto` honors
|
||||||
`auto`; `zlibx` behaves as `zlib`, and the transfer checksum is not separately
|
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` and otherwise follows rsync's
|
||||||
selectable.
|
compiled-in order. An omitted `--compress-level` uses the codec's rsync
|
||||||
|
default (zstd 3, zlib/zlibx 6, lz4 ignored); `zlib`/`zlibx` share the
|
||||||
|
literal-only zlib path (rsync's zlibx semantics), and the transfer checksum is
|
||||||
|
not separately selectable.
|
||||||
|
|
||||||
The detailed flag matrix is maintained in
|
The detailed flag matrix is maintained in
|
||||||
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It reports each row as **parity**,
|
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It reports each row as **parity**,
|
||||||
@@ -505,8 +508,8 @@ features without changing the meaning of ordinary compatibility options.
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. |
|
| `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. |
|
||||||
| `-z [level]`, `--compress [level]` | Enable streaming compression (default `zstd`), levels 1-22. |
|
| `-z [level]`, `--compress [level]` | Enable streaming compression (default `zstd`), levels 1-22. |
|
||||||
| `--compress-level <n>` | Set the compression level. |
|
| `--compress-level <n>` | Set the compression level (1-22). Omitted, each codec uses its rsync default: zstd 3, zlib/zlibx 6, lz4 ignored. |
|
||||||
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlibx` behaves as `zlib`. |
|
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlib`/`zlibx` share the same literal-only zlib path. |
|
||||||
| `--zl <n>` | Alias for `--compress-level`. |
|
| `--zl <n>` | Alias for `--compress-level`. |
|
||||||
| `--skip-compress <list>` | Skip compression for `/`- or `,`-separated suffixes; defaults to rsync 3.4.1's built-in list. Incompatible with `--chunk-serialization`. |
|
| `--skip-compress <list>` | Skip compression for `/`- or `,`-separated suffixes; defaults to rsync 3.4.1's built-in list. Incompatible with `--chunk-serialization`. |
|
||||||
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
|
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
|
||||||
|
|||||||
+13
-10
@@ -6,8 +6,8 @@ This document maps rsync's full feature set to FastSync's current implementation
|
|||||||
|
|
||||||
| Status | Count | Description |
|
| Status | Count | Description |
|
||||||
|--------|-------|-------------|
|
|--------|-------|-------------|
|
||||||
| ✅ Parity | 111 | Reproduces rsync's semantics for this option's scope |
|
| ✅ Parity | 113 | Reproduces rsync's semantics for this option's scope |
|
||||||
| ⚠️ Caveat | 14 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
|
| ⚠️ Caveat | 12 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
|
||||||
| ❌ Divergent | 32 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
|
| ❌ Divergent | 32 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
|
||||||
| **Total** | **157** | One row per rsync option/feature group; a row may name several spellings |
|
| **Total** | **157** | One row per rsync option/feature group; a row may name several spellings |
|
||||||
|
|
||||||
@@ -41,6 +41,8 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
|
|||||||
|
|
||||||
**Parity track 2b (no-wire, on `feat/parity-2.28`).** An opt-in, paths-only metadata pre-count for `--progress`/`-P`/`--info=progress` (only when not `--quiet`) now gives the `to-chk` denominator rsync's full file-list total (every regular file, directory, symlink and special plus the transfer root) and emits the per-directory/symlink/special name lines, in both the sequential and `--threads` paths; `--delete-during`/`--delete-delay` reuse their keep-set pre-scan so no second walk happens, and non-progress runs are unaffected. A fresh multi-directory differential against rsync 3.4.1 matches the name set and the `to-chk` denominator, while a single-file transfer stays byte-identical; the remaining caveats are emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream, so the `to-chk` numerator and interleaving differ) and re-run/receiver-state-driven over-naming (unconditional `./`, ancestor dirs emitted with a transferred child, and no quick-check for symlinks/empty dirs), keeping the row ⚠️. The matrix is unchanged at **111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
|
**Parity track 2b (no-wire, on `feat/parity-2.28`).** An opt-in, paths-only metadata pre-count for `--progress`/`-P`/`--info=progress` (only when not `--quiet`) now gives the `to-chk` denominator rsync's full file-list total (every regular file, directory, symlink and special plus the transfer root) and emits the per-directory/symlink/special name lines, in both the sequential and `--threads` paths; `--delete-during`/`--delete-delay` reuse their keep-set pre-scan so no second walk happens, and non-progress runs are unaffected. A fresh multi-directory differential against rsync 3.4.1 matches the name set and the `to-chk` denominator, while a single-file transfer stays byte-identical; the remaining caveats are emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream, so the `to-chk` numerator and interleaving differ) and re-run/receiver-state-driven over-naming (unconditional `./`, ancestor dirs emitted with a transferred child, and no quick-check for symlinks/empty dirs), keeping the row ⚠️. The matrix is unchanged at **111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
|
||||||
|
|
||||||
|
**Parity track 3a (no-wire, on `feat/parity-2.28`).** Three codec refinements, none of which change the wire layout (`PROTOCOL_VERSION` stays 2.28.0): (1) an omitted `--compress-level` now resolves to rsync 3.4.1's per-codec default (zstd 3, zlib/zlibx 6, lz4 ignored) and an explicit level is clamped per codec (zstd 1-22, zlib/zlibx 1-9), verified against `rsync --debug=NSTR1`; (2) `auto` now honors `RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` (rsync's whitespace-separated preference syntax, unknown names skipped, first supported wins, all-unknown is exit 4) before the compiled-in order, with an explicit `--zc`/`--cc` still winning; and (3) `--compress-choice=zlibx` is reclassified out of the caveat list — FastSync's zlib stream already carries only the literal/delta bytes (rsync's zlibx semantics) and is observably identical to `--zc=zlib`, so the zlib/zlibx aliasing is only an implementation detail. The matrix is now **113 ✅ / 12 ⚠️ / 32 ❌ = 157**.
|
||||||
|
|
||||||
**Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the
|
**Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the
|
||||||
remaining gaps the rsync-parity wave left open (delete timing, wire counters and
|
remaining gaps the rsync-parity wave left open (delete timing, wire counters and
|
||||||
output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side
|
output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side
|
||||||
@@ -649,7 +651,7 @@ targets verbatim, matching rsync.
|
|||||||
| Flag | Rsync Description | FastSync Status | Notes |
|
| Flag | Rsync Description | FastSync Status | Notes |
|
||||||
|------|-------------------|-----------------|-------|
|
|------|-------------------|-----------------|-------|
|
||||||
| `--checksum` | Skip based on checksum | ✅ Parity | `-c`/`--checksum` compares per-file whole-file content digests to skip unchanged files. **As of protocol 2.23.0 the short `-c` implies the checksum quick-check**, so a plain `-c` run verifies content rather than only affecting the `--incremental` handshake. The digest algorithm is `xxh128` by default (protocol 2.26.0's negotiated default) and is selectable via `--checksum-choice`/`--cc` (`xxh128`/`xxh3`/`xxh64`/`xxhash`/`md5`/`md4`/`sha1`/`none`/`auto`, plus rsync's two-name form) and `--checksum-seed=NUM` (see those rows) |
|
| `--checksum` | Skip based on checksum | ✅ Parity | `-c`/`--checksum` compares per-file whole-file content digests to skip unchanged files. **As of protocol 2.23.0 the short `-c` implies the checksum quick-check**, so a plain `-c` run verifies content rather than only affecting the `--incremental` handshake. The digest algorithm is `xxh128` by default (protocol 2.26.0's negotiated default) and is selectable via `--checksum-choice`/`--cc` (`xxh128`/`xxh3`/`xxh64`/`xxhash`/`md5`/`md4`/`sha1`/`none`/`auto`, plus rsync's two-name form) and `--checksum-seed=NUM` (see those rows) |
|
||||||
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ⚠️ Caveat | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and basis-dir verification. **Protocol 2.26.0 accepts rsync 3.4.1's full set** — `xxh128` (the negotiated default), `xxh3`, `xxh64`, `xxhash`, `md5`, `md4`, `sha1`, `none`, `auto`, and the two-name `transfer,pre-transfer` form — with rsync's exit-4 rejection of an unknown name and of `none` on the transfer side when `--checksum` is on. `--cc=ALG` and space forms both parse. The algorithm id and seed cross the wire; the receiver hashes its old file with the same algorithm+seed and the per-file `STATUS_CHECK` handshake carries a bounded digest pinned to the negotiated length. `checksum_digest_file` now streams **every** supported algorithm (md4 via the self-contained RFC 1320 code, sha1/md5 via EVP, none as an empty digest), so the streaming path matches its contract, and `--out-format %C` uses the selected **transfer** half of a two-name choice and renders each algorithm byte-for-byte like rsync (xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, none a blank 2-char column) — differential-tested across all algorithms. **Remaining divergences:** rsync uses this choice for the block checksum on the wire too, while FastSync selects only the whole-file comparison digest and keeps the delta BLOCK strong checksum at xxHash32; the `RSYNC_CHECKSUM_LIST` environment variable is not consulted; and `auto` always resolves deterministically to the first supported entry in rsync's preference order rather than probing the peer |
|
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ⚠️ Caveat | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and basis-dir verification. **Protocol 2.26.0 accepts rsync 3.4.1's full set** — `xxh128` (the negotiated default), `xxh3`, `xxh64`, `xxhash`, `md5`, `md4`, `sha1`, `none`, `auto`, and the two-name `transfer,pre-transfer` form — with rsync's exit-4 rejection of an unknown name and of `none` on the transfer side when `--checksum` is on. `--cc=ALG` and space forms both parse. The algorithm id and seed cross the wire; the receiver hashes its old file with the same algorithm+seed and the per-file `STATUS_CHECK` handshake carries a bounded digest pinned to the negotiated length. `checksum_digest_file` now streams **every** supported algorithm (md4 via the self-contained RFC 1320 code, sha1/md5 via EVP, none as an empty digest), so the streaming path matches its contract, and `--out-format %C` uses the selected **transfer** half of a two-name choice and renders each algorithm byte-for-byte like rsync (xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, none a blank 2-char column) — differential-tested across all algorithms. **Remaining divergence:** rsync uses this choice for the block checksum on the wire too, while FastSync selects only the whole-file comparison digest and keeps the delta BLOCK strong checksum at xxHash32. `auto` now consults `RSYNC_CHECKSUM_LIST` (rsync's whitespace-separated preference list; unknown names skipped, first supported wins, all-unknown is exit 4) before the compiled-in order; because both peers run the identical build this deterministic resolution needs no rsync peer probe, and an explicit `--cc` still wins. The list is differential-tested through `--out-format %C` (byte-identical digests to rsync for md5/sha1/xxh3) |
|
||||||
| `--compare-dest=DIR` | Compare dest files relative to DIR | ❌ Divergent | DIR is a receiver-side basis; protocol 2.26.0 uses an absolute path verbatim (rsync semantics) and resolves a relative path below the destination root (`..` components are rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. **Reclassified Divergent (differential evidence):** FastSync verifies a basis hit's content with xxHash64 while rsync's `--size-only` quick check trusts size (and mtime) alone, so with a same-size/different-content basis rsync skips/links the *wrong* basis content while FastSync transfers the source — a deliberate safety-stricter behavior that cannot match rsync (see `test_basis_dir_size_only_content_residual` in `tests/integration/test_parity_quickwins.py`). Attribute-only differences on a match are also not re-applied (data is skipped so the sender never sends metadata). Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) |
|
| `--compare-dest=DIR` | Compare dest files relative to DIR | ❌ Divergent | DIR is a receiver-side basis; protocol 2.26.0 uses an absolute path verbatim (rsync semantics) and resolves a relative path below the destination root (`..` components are rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. **Reclassified Divergent (differential evidence):** FastSync verifies a basis hit's content with xxHash64 while rsync's `--size-only` quick check trusts size (and mtime) alone, so with a same-size/different-content basis rsync skips/links the *wrong* basis content while FastSync transfers the source — a deliberate safety-stricter behavior that cannot match rsync (see `test_basis_dir_size_only_content_residual` in `tests/integration/test_parity_quickwins.py`). Attribute-only differences on a match are also not re-applied (data is skipped so the sender never sends metadata). Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) |
|
||||||
| `--copy-dest=DIR` | Include copies of unchanged files | ❌ Divergent | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. **Reclassified Divergent** for the same basis-hit verification divergence as `--compare-dest`: rsync's `--size-only` size/quick-check match rings a same-size/different-content file as unchanged and copies the wrong basis bytes, while FastSync's xxHash verification transfers the source (differential test `test_basis_dir_size_only_content_residual`); a basis-hit also keeps whatever metadata the copy derived from the basis rather than rsync's "copy + fix attributes" in attribute-only cases. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
| `--copy-dest=DIR` | Include copies of unchanged files | ❌ Divergent | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. **Reclassified Divergent** for the same basis-hit verification divergence as `--compare-dest`: rsync's `--size-only` size/quick-check match rings a same-size/different-content file as unchanged and copies the wrong basis bytes, while FastSync's xxHash verification transfers the source (differential test `test_basis_dir_size_only_content_residual`); a basis-hit also keeps whatever metadata the copy derived from the basis rather than rsync's "copy + fix attributes" in attribute-only cases. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
||||||
| `--link-dest=DIR` | Hardlink to files when unchanged | ❌ Divergent | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. **Reclassified Divergent** for the shared basis-hit divergence: with `--size-only` a same-size/different-content basis is linked by rsync (installing wrong content) but FastSync detects the xxHash mismatch and transfers the source (differential test `test_basis_dir_size_only_content_residual`). Other inherent caveats: protocol 2.26.0 re-links an already up-to-date destination file to the basis; a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
| `--link-dest=DIR` | Hardlink to files when unchanged | ❌ Divergent | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. **Reclassified Divergent** for the shared basis-hit divergence: with `--size-only` a same-size/different-content basis is linked by rsync (installing wrong content) but FastSync detects the xxHash mismatch and transfers the source (differential test `test_basis_dir_size_only_content_residual`). Other inherent caveats: protocol 2.26.0 re-links an already up-to-date destination file to the basis; a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
||||||
@@ -659,9 +661,9 @@ targets verbatim, matching rsync.
|
|||||||
|
|
||||||
| Flag | Rsync Description | FastSync Status | Notes |
|
| Flag | Rsync Description | FastSync Status | Notes |
|
||||||
|------|-------------------|-----------------|-------|
|
|------|-------------------|-----------------|-------|
|
||||||
| `-z`, `--compress` | Compress file data | ⚠️ Caveat | Streaming compression. **Protocol 2.26.0 implements rsync 3.4.1's codec set** (`zstd` default, `lz4`, `zlib`, `zlibx`, `none`), selectable via `--compress-choice`/`--zc` and negotiated with `auto`. `-z` is the compression short form; `-c` is rsync's `--checksum`. `--skip-compress` applies rsync 3.4.1's default suffix list when no list is given. **Remaining codec divergence:** `zlibx` is treated as `zlib`, per-codec level defaults are not mirrored, and `auto` does not probe the peer |
|
| `-z`, `--compress` | Compress file data | ✅ Parity | Streaming compression. **Protocol 2.26.0 implements rsync 3.4.1's codec set** (`zstd` default, `lz4`, `zlib`, `zlibx`, `none`), selectable via `--compress-choice`/`--zc` and negotiated with `auto`. `-z` is the compression short form; `-c` is rsync's `--checksum`. `--skip-compress` applies rsync 3.4.1's default suffix list when no list is given. **Track 3a closes the codec caveats:** `zlibx` is no longer a divergence — FastSync's zlib stream already carries only the delta/token (literal) bytes, which is exactly rsync's zlibx semantics, so `--zc=zlib` and `--zc=zlibx` land the same tree/stdout/exit (differential `test_compress_codec_matches_rsync_bytes`) and the zlib/zlibx aliasing is only an implementation detail. Each codec now uses rsync's own default `--compress-level` (zstd 3, zlib/zlibx 6, lz4 ignored) and `auto` consults `RSYNC_COMPRESS_LIST` before the compiled-in order; the deterministic same-build resolution needs no peer probe |
|
||||||
| `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ⚠️ Caveat | Protocol 2.26.0 accepts rsync 3.4.1's compiled-in choices — `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, `auto` — and rejects an unknown name with exit 4 like rsync. The negotiated codec id crosses the wire (`compression_algo`), so the receiver decodes with the sender's codec. `--zc` is the alias. **Remaining divergences:** `zlibx` behaves as `zlib` (there is no separate zlibx path), the per-codec compression-level defaults differ from rsync's, and `auto` always resolves to the first supported entry in rsync's preference order rather than probing the peer (no `RSYNC_COMPRESS_LIST` handling) |
|
| `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ✅ Parity | Protocol 2.26.0 accepts rsync 3.4.1's compiled-in choices — `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, `auto` — and rejects an unknown name with exit 4 like rsync. The negotiated codec id crosses the wire (`compression_algo`), so the receiver decodes with the sender's codec. `--zc` is the alias. `auto` now resolves through `RSYNC_COMPRESS_LIST` (whitespace-separated; unknown names skipped, first supported wins, all-unknown is exit 4) and then the compiled-in order, and an explicit `--zc` wins; the deterministic same-build resolution needs no peer probe. `zlib`/`zlibx` share FastSync's literal-only zlib path, which is rsync's zlibx behavior and is observably identical for both, so `zlibx` is not a divergence (the aliasing is an implementation detail) |
|
||||||
| `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Parity | 1-22, default 5 |
|
| `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Parity | Accepted range 1-22. When omitted, rsync 3.4.1's **per-codec default** applies: zstd 3 (`ZSTD_CLEVEL_DEFAULT`), zlib/zlibx 6 (`Z_DEFAULT_COMPRESSION` resolved), lz4 ignored (no tunable level; FastSync keeps a positive gate value and `lz4_compress` ignores it, so the bytes match rsync). An explicit level is clamped per codec like rsync's `init_compression_level()`: zstd 1-22, zlib/zlibx 1-9, lz4 ignored. Verified against `rsync --debug=NSTR1`, which reports the same effective level per codec |
|
||||||
| `--compress-threads=NUM` | Set compression threads | ✅ Parity | `compression_threads` config field (client-only; does not cross the wire). Sets the number of worker threads used by the zstd compression pool to NUM (1..64; 0/garbage/oversized rejected up front). Accepted in both `--compress-threads=NUM` and two-argument `--compress-threads NUM` forms. Composes with `-z`/compression; under the `-j`/`--threads` multithreaded pipeline it parallelizes compressed chunk encoding. See test_tcp.py `-z --compress-threads=2` and test_client_cli.c |
|
| `--compress-threads=NUM` | Set compression threads | ✅ Parity | `compression_threads` config field (client-only; does not cross the wire). Sets the number of worker threads used by the zstd compression pool to NUM (1..64; 0/garbage/oversized rejected up front). Accepted in both `--compress-threads=NUM` and two-argument `--compress-threads NUM` forms. Composes with `-z`/compression; under the `-j`/`--threads` multithreaded pipeline it parallelizes compressed chunk encoding. See test_tcp.py `-z --compress-threads=2` and test_client_cli.c |
|
||||||
| `--skip-compress=LIST` | Skip compress for suffixes | ✅ Parity | Comma-separated (or `/`-separated, as in rsync) case-insensitive suffix list; a leading dot is optional; an empty list skips none. **When the option is omitted, rsync 3.4.1's built-in default suffix list applies** (`3g2 3gp 7z aac … zip zst`); an explicit list replaces that default entirely, matching rsync. A user-supplied list is a client-side compression choice; incompatible with FastSync chunk serialization (`-s`) |
|
| `--skip-compress=LIST` | Skip compress for suffixes | ✅ Parity | Comma-separated (or `/`-separated, as in rsync) case-insensitive suffix list; a leading dot is optional; an empty list skips none. **When the option is omitted, rsync 3.4.1's built-in default suffix list applies** (`3g2 3gp 7z aac … zip zst`); an explicit list replaces that default entirely, matching rsync. A user-supplied list is a client-side compression choice; incompatible with FastSync chunk serialization (`-s`) |
|
||||||
|
|
||||||
@@ -1111,10 +1113,11 @@ wire protocol three times (full rationale in `src/shared/config.h`):
|
|||||||
`sha1`, `none`, `auto`, plus the two-name `transfer,pre-transfer` form.
|
`sha1`, `none`, `auto`, plus the two-name `transfer,pre-transfer` form.
|
||||||
Unknown names and `none` on the transfer side under `--checksum` exit 4 like
|
Unknown names and `none` on the transfer side under `--checksum` exit 4 like
|
||||||
rsync.
|
rsync.
|
||||||
- **Remaining codec residuals:** `zlibx` behaves as `zlib`; the transfer
|
- **Codec residuals:** the transfer checksum is not independently selectable
|
||||||
checksum is not independently selectable (only the whole-file comparison
|
(only the whole-file comparison digest is); `zlib`/`zlibx` share FastSync's
|
||||||
digest is); per-codec level defaults differ; and `RSYNC_CHECKSUM_LIST`/
|
literal-only zlib path (rsync's zlibx behavior, observably identical for both,
|
||||||
`RSYNC_COMPRESS_LIST` are not consulted.
|
so `zlibx` is not a divergence). Per-codec compression-level defaults and
|
||||||
|
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` are implemented (track 3a).
|
||||||
|
|
||||||
### Selection, paths, and filters
|
### Selection, paths, and filters
|
||||||
|
|
||||||
|
|||||||
+61
-8
@@ -161,10 +161,16 @@ static int set_compression_choice(Config* config, const char* value) {
|
|||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
int algo;
|
int algo;
|
||||||
if (strcasecmp(value, "auto") == 0)
|
if (strcasecmp(value, "auto") == 0) {
|
||||||
algo = (int)compression_negotiate_default();
|
algo = compression_choice_resolve();
|
||||||
else
|
if (algo < 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
|
||||||
|
config->cli_exit_code = 4;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
algo = compression_algo_from_name(value);
|
algo = compression_algo_from_name(value);
|
||||||
|
}
|
||||||
if (algo < 0) {
|
if (algo < 0) {
|
||||||
log_message(LOG_LEVEL_ERROR,
|
log_message(LOG_LEVEL_ERROR,
|
||||||
"--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, "
|
"--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, "
|
||||||
@@ -228,16 +234,25 @@ static int set_checksum_choice(Config* config, const char* value) {
|
|||||||
config->cli_exit_code = 4;
|
config->cli_exit_code = 4;
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
ChecksumAlgo negotiated = checksum_negotiate_default();
|
int negotiated = -1;
|
||||||
|
if (rc1 == 1 || rc2 == 1) {
|
||||||
|
negotiated = checksum_choice_resolve();
|
||||||
|
if (negotiated < 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
|
||||||
|
config->cli_exit_code = 4;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
if (rc1 == 1)
|
if (rc1 == 1)
|
||||||
transfer = (int)negotiated;
|
transfer = negotiated;
|
||||||
if (!name2)
|
if (!name2)
|
||||||
pre = transfer;
|
pre = transfer;
|
||||||
else if (rc2 == 1)
|
else if (rc2 == 1)
|
||||||
pre = (int)negotiated;
|
pre = negotiated;
|
||||||
|
|
||||||
config->checksum_algo = pre;
|
config->checksum_algo = pre;
|
||||||
config->checksum_transfer_algo = transfer;
|
config->checksum_transfer_algo = transfer;
|
||||||
|
config->checksum_choice_set = true;
|
||||||
/* rsync: "none" for the transfer checksum forces --whole-file. */
|
/* rsync: "none" for the transfer checksum forces --whole-file. */
|
||||||
if (transfer == (int)CHECKSUM_ALGO_NONE)
|
if (transfer == (int)CHECKSUM_ALGO_NONE)
|
||||||
config->whole_file = true;
|
config->whole_file = true;
|
||||||
@@ -1464,6 +1479,8 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
|
|||||||
ctx->exit_code = -1;
|
ctx->exit_code = -1;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
if (entry->offset == offsetof(Config, compression_level))
|
||||||
|
config->compression_level_set = true;
|
||||||
if (entry->offset == offsetof(Config, chmod_spec)) {
|
if (entry->offset == offsetof(Config, chmod_spec)) {
|
||||||
mode_t ignored;
|
mode_t ignored;
|
||||||
if (!chmod_apply(0, config->chmod_spec, &ignored)) {
|
if (!chmod_apply(0, config->chmod_spec, &ignored)) {
|
||||||
@@ -1748,6 +1765,7 @@ static bool cli_handle_transfer_flags(CliParseCtx* ctx) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
config->compression_level = (int)level;
|
config->compression_level = (int)level;
|
||||||
|
config->compression_level_set = true;
|
||||||
log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level);
|
log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level);
|
||||||
ctx->i++;
|
ctx->i++;
|
||||||
}
|
}
|
||||||
@@ -2550,8 +2568,43 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
|
|||||||
config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE);
|
config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (config->use_compression && config->compression_algo == (int)COMPRESSION_ALGO_NONE)
|
/* A bare -z (no --compress-choice) resolves like rsync's "auto": the
|
||||||
config->compression_algo = (int)compression_negotiate_default();
|
* RSYNC_COMPRESS_LIST preference list first, then the compiled-in order. A
|
||||||
|
* list that names no supported codec is rsync's failed negotiation (exit 4). */
|
||||||
|
if (config->use_compression && !config->compress_choice) {
|
||||||
|
int resolved = compression_choice_resolve();
|
||||||
|
if (resolved < 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
|
||||||
|
config->cli_exit_code = 4;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
config->compression_algo = resolved;
|
||||||
|
if (resolved == (int)COMPRESSION_ALGO_NONE)
|
||||||
|
config->use_compression = false;
|
||||||
|
}
|
||||||
|
/* Apply rsync's per-codec compression level: an explicit --compress-level is
|
||||||
|
* clamped to the codec's range, otherwise the codec's own default is used. */
|
||||||
|
if (config->use_compression) {
|
||||||
|
CompressionAlgo algo = (CompressionAlgo)config->compression_algo;
|
||||||
|
config->compression_level = config->compression_level_set
|
||||||
|
? compression_clamp_level(algo, config->compression_level)
|
||||||
|
: compression_default_level(algo);
|
||||||
|
log_debug_message(LOG_DEBUG_UTIL, "Client compression: %s (level %d)",
|
||||||
|
compression_algo_name(algo), config->compression_level);
|
||||||
|
}
|
||||||
|
/* The negotiated checksum is always resolved (rsync negotiates one for the
|
||||||
|
* delta strong sum even without --checksum): RSYNC_CHECKSUM_LIST first, then
|
||||||
|
* the compiled-in order. An explicit --checksum-choice already set it. */
|
||||||
|
if (!config->checksum_choice_set) {
|
||||||
|
int resolved = checksum_choice_resolve();
|
||||||
|
if (resolved < 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
|
||||||
|
config->cli_exit_code = 4;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
config->checksum_algo = resolved;
|
||||||
|
config->checksum_transfer_algo = resolved;
|
||||||
|
}
|
||||||
/* rsync parity: "none" as the pre-transfer checksum cannot be combined with
|
/* rsync parity: "none" as the pre-transfer checksum cannot be combined with
|
||||||
* --checksum (exit 4). The check runs here because --checksum may appear on
|
* --checksum (exit 4). The check runs here because --checksum may appear on
|
||||||
* either side of --checksum-choice. */
|
* either side of --checksum-choice. */
|
||||||
|
|||||||
+15
-1
@@ -1,4 +1,5 @@
|
|||||||
#include "checksum.h"
|
#include "checksum.h"
|
||||||
|
#include "utils.h"
|
||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
#include <openssl/evp.h>
|
#include <openssl/evp.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
@@ -396,7 +397,7 @@ uint8_t checksum_digest_len(ChecksumAlgo algo) {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
ChecksumAlgo checksum_negotiate_default(void) {
|
static ChecksumAlgo compiled_checksum_preference_first(void) {
|
||||||
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
||||||
* resolves to xxh128. */
|
* resolves to xxh128. */
|
||||||
static const ChecksumAlgo preference[] = {
|
static const ChecksumAlgo preference[] = {
|
||||||
@@ -409,3 +410,16 @@ ChecksumAlgo checksum_negotiate_default(void) {
|
|||||||
}
|
}
|
||||||
return CHECKSUM_ALGO_XXH64;
|
return CHECKSUM_ALGO_XXH64;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int checksum_choice_resolve(void) {
|
||||||
|
bool specified = false;
|
||||||
|
int env = env_choice_first("RSYNC_CHECKSUM_LIST", checksum_algo_from_name, &specified);
|
||||||
|
if (specified)
|
||||||
|
return env; /* -1 = the list named no supported checksum */
|
||||||
|
return (int)compiled_checksum_preference_first();
|
||||||
|
}
|
||||||
|
|
||||||
|
ChecksumAlgo checksum_negotiate_default(void) {
|
||||||
|
int resolved = checksum_choice_resolve();
|
||||||
|
return resolved >= 0 ? (ChecksumAlgo)resolved : compiled_checksum_preference_first();
|
||||||
|
}
|
||||||
|
|||||||
@@ -72,4 +72,11 @@ uint8_t checksum_digest_len(ChecksumAlgo algo);
|
|||||||
* xxh128 xxh3 xxh64 md5 md4 sha1 none). Used to resolve "auto". */
|
* xxh128 xxh3 xxh64 md5 md4 sha1 none). Used to resolve "auto". */
|
||||||
ChecksumAlgo checksum_negotiate_default(void);
|
ChecksumAlgo checksum_negotiate_default(void);
|
||||||
|
|
||||||
|
/* Resolve "auto" the way rsync does: the first supported name in
|
||||||
|
* RSYNC_CHECKSUM_LIST (whitespace-separated, client half ends at '&'), then the
|
||||||
|
* compiled-in preference order when the variable is unset/blank. Returns -1
|
||||||
|
* when the variable is set but names no supported checksum (rsync's failed
|
||||||
|
* negotiation), otherwise a valid ChecksumAlgo id. */
|
||||||
|
int checksum_choice_resolve(void);
|
||||||
|
|
||||||
#endif /* CHECKSUM_H */
|
#endif /* CHECKSUM_H */
|
||||||
|
|||||||
@@ -2,6 +2,7 @@
|
|||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
|
#include "utils.h"
|
||||||
#include <limits.h>
|
#include <limits.h>
|
||||||
#include <lz4.h>
|
#include <lz4.h>
|
||||||
#include <stdatomic.h>
|
#include <stdatomic.h>
|
||||||
@@ -119,7 +120,7 @@ bool compression_algo_enabled(CompressionAlgo algo) {
|
|||||||
return algo != COMPRESSION_ALGO_NONE;
|
return algo != COMPRESSION_ALGO_NONE;
|
||||||
}
|
}
|
||||||
|
|
||||||
CompressionAlgo compression_negotiate_default(void) {
|
static CompressionAlgo compiled_preference_first(void) {
|
||||||
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
||||||
* resolves to zstd. */
|
* resolves to zstd. */
|
||||||
static const CompressionAlgo preference[] = {
|
static const CompressionAlgo preference[] = {
|
||||||
@@ -133,6 +134,57 @@ CompressionAlgo compression_negotiate_default(void) {
|
|||||||
return COMPRESSION_ALGO_ZSTD;
|
return COMPRESSION_ALGO_ZSTD;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int compression_choice_resolve(void) {
|
||||||
|
bool specified = false;
|
||||||
|
int env = env_choice_first("RSYNC_COMPRESS_LIST", compression_algo_from_name, &specified);
|
||||||
|
if (specified)
|
||||||
|
return env; /* -1 = the list named no supported codec */
|
||||||
|
return (int)compiled_preference_first();
|
||||||
|
}
|
||||||
|
|
||||||
|
CompressionAlgo compression_negotiate_default(void) {
|
||||||
|
int resolved = compression_choice_resolve();
|
||||||
|
return resolved >= 0 ? (CompressionAlgo)resolved : compiled_preference_first();
|
||||||
|
}
|
||||||
|
|
||||||
|
int compression_default_level(CompressionAlgo algo) {
|
||||||
|
switch (algo) {
|
||||||
|
case COMPRESSION_ALGO_ZSTD:
|
||||||
|
return ZSTD_CLEVEL_DEFAULT;
|
||||||
|
case COMPRESSION_ALGO_ZLIB:
|
||||||
|
case COMPRESSION_ALGO_ZLIBX:
|
||||||
|
return 6; /* rsync resolves zlib's Z_DEFAULT_COMPRESSION (-1) to 6 */
|
||||||
|
case COMPRESSION_ALGO_LZ4:
|
||||||
|
return 1; /* rsync lz4 level is 0/ignored; positive keeps the gate on */
|
||||||
|
case COMPRESSION_ALGO_NONE:
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
int compression_clamp_level(CompressionAlgo algo, int level) {
|
||||||
|
switch (algo) {
|
||||||
|
case COMPRESSION_ALGO_ZSTD:
|
||||||
|
if (level < 1)
|
||||||
|
return 1;
|
||||||
|
if (level > 22)
|
||||||
|
return 22;
|
||||||
|
return level;
|
||||||
|
case COMPRESSION_ALGO_ZLIB:
|
||||||
|
case COMPRESSION_ALGO_ZLIBX:
|
||||||
|
if (level < 1)
|
||||||
|
return 1;
|
||||||
|
if (level > 9)
|
||||||
|
return 9;
|
||||||
|
return level;
|
||||||
|
case COMPRESSION_ALGO_LZ4:
|
||||||
|
return 1; /* ignored by lz4_compress; keeps the "compress" gate on */
|
||||||
|
case COMPRESSION_ALGO_NONE:
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
return level;
|
||||||
|
}
|
||||||
|
|
||||||
void compression_set_algo(CompressionAlgo algo) {
|
void compression_set_algo(CompressionAlgo algo) {
|
||||||
if (compression_algo_valid((int)algo))
|
if (compression_algo_valid((int)algo))
|
||||||
atomic_store(&g_compression_algo, (int)algo);
|
atomic_store(&g_compression_algo, (int)algo);
|
||||||
|
|||||||
@@ -35,6 +35,26 @@ bool compression_algo_valid(int algo);
|
|||||||
* "auto". */
|
* "auto". */
|
||||||
CompressionAlgo compression_negotiate_default(void);
|
CompressionAlgo compression_negotiate_default(void);
|
||||||
|
|
||||||
|
/* Resolve "auto" the way rsync does: the first supported name in
|
||||||
|
* RSYNC_COMPRESS_LIST (whitespace-separated, client half ends at '&'), then the
|
||||||
|
* compiled-in preference order when the variable is unset/blank. Returns -1
|
||||||
|
* when the variable is set but names no supported codec (rsync's failed
|
||||||
|
* negotiation), otherwise a valid CompressionAlgo id. */
|
||||||
|
int compression_choice_resolve(void);
|
||||||
|
|
||||||
|
/* rsync 3.4.1's per-codec default level, applied when the user did not pass
|
||||||
|
* --compress-level/--zl. zstd uses ZSTD_CLEVEL_DEFAULT (3) and zlib/zlibx the
|
||||||
|
* resolved Z_DEFAULT_COMPRESSION (6). lz4 has no tunable level in rsync
|
||||||
|
* (always the default acceleration); FastSync returns a positive placeholder so
|
||||||
|
* its "level > 0" compression gate stays engaged, and lz4_compress ignores the
|
||||||
|
* value, so the output is identical to rsync's. none is 0. */
|
||||||
|
int compression_default_level(CompressionAlgo algo);
|
||||||
|
|
||||||
|
/* Clamp an explicit --compress-level to the codec's accepted range the way
|
||||||
|
* rsync's init_compression_level() does: zstd 1..22, zlib/zlibx 1..9, lz4
|
||||||
|
* ignored (fixed positive placeholder), none 0. */
|
||||||
|
int compression_clamp_level(CompressionAlgo algo, int level);
|
||||||
|
|
||||||
/* True when the algorithm actually compresses (i.e. is not NONE). */
|
/* True when the algorithm actually compresses (i.e. is not NONE). */
|
||||||
bool compression_algo_enabled(CompressionAlgo algo);
|
bool compression_algo_enabled(CompressionAlgo algo);
|
||||||
|
|
||||||
|
|||||||
@@ -70,6 +70,8 @@ static void config_set_defaults(Config* config) {
|
|||||||
config->ignore_missing_args = false;
|
config->ignore_missing_args = false;
|
||||||
config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
|
config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
|
||||||
config->cli_exit_code = 0;
|
config->cli_exit_code = 0;
|
||||||
|
config->compression_level_set = false;
|
||||||
|
config->checksum_choice_set = false;
|
||||||
config->filters = NULL;
|
config->filters = NULL;
|
||||||
config->files_from = NULL;
|
config->files_from = NULL;
|
||||||
config->files_from_set = NULL;
|
config->files_from_set = NULL;
|
||||||
|
|||||||
@@ -419,6 +419,12 @@ typedef struct Config {
|
|||||||
* for an unsupported checksum/compress algorithm) so main() can mirror it. */
|
* for an unsupported checksum/compress algorithm) so main() can mirror it. */
|
||||||
int checksum_transfer_algo;
|
int checksum_transfer_algo;
|
||||||
int cli_exit_code;
|
int cli_exit_code;
|
||||||
|
/* Client-only "the user explicitly chose" bits. They let the per-codec
|
||||||
|
* default level / checksum list be applied only when the corresponding
|
||||||
|
* rsync option was omitted (an explicit --compress-level / --checksum-choice
|
||||||
|
* always wins). Never serialized. */
|
||||||
|
bool compression_level_set;
|
||||||
|
bool checksum_choice_set;
|
||||||
|
|
||||||
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
|
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
|
||||||
// never serialized to the wire (the receiver must not learn them).
|
// never serialized to the wire (the receiver must not learn them).
|
||||||
|
|||||||
@@ -2,6 +2,7 @@
|
|||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include <arpa/inet.h>
|
#include <arpa/inet.h>
|
||||||
|
#include <ctype.h>
|
||||||
#include <dirent.h>
|
#include <dirent.h>
|
||||||
#include <errno.h>
|
#include <errno.h>
|
||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
@@ -142,6 +143,47 @@ char* str_dup(const char* string) {
|
|||||||
return new_string;
|
return new_string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified) {
|
||||||
|
if (specified)
|
||||||
|
*specified = false;
|
||||||
|
if (!env_name || !resolve)
|
||||||
|
return -1;
|
||||||
|
const char* env = getenv(env_name);
|
||||||
|
if (!env)
|
||||||
|
return -1;
|
||||||
|
|
||||||
|
bool saw_nonblank = false;
|
||||||
|
const char* p = env;
|
||||||
|
while (*p) {
|
||||||
|
if (*p == '&')
|
||||||
|
break;
|
||||||
|
if (isspace((unsigned char)*p)) {
|
||||||
|
p++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
saw_nonblank = true;
|
||||||
|
char token[64];
|
||||||
|
size_t len = 0;
|
||||||
|
while (*p && *p != '&' && !isspace((unsigned char)*p)) {
|
||||||
|
if (len < sizeof(token) - 1)
|
||||||
|
token[len++] = *p;
|
||||||
|
p++;
|
||||||
|
}
|
||||||
|
token[len] = '\0';
|
||||||
|
if (len > 0) {
|
||||||
|
int id = resolve(token);
|
||||||
|
if (id >= 0) {
|
||||||
|
if (specified)
|
||||||
|
*specified = true;
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (specified)
|
||||||
|
*specified = saw_nonblank;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
#define STR_HASH_SET_MIN_CAPACITY 16
|
#define STR_HASH_SET_MIN_CAPACITY 16
|
||||||
|
|
||||||
static size_t str_hash_set_hash(const char* key, size_t len) {
|
static size_t str_hash_set_hash(const char* key, size_t len) {
|
||||||
|
|||||||
@@ -81,6 +81,17 @@ bool path_index_has_descendant(const PathIndex* index, const char* path);
|
|||||||
|
|
||||||
char* str_dup(const char* string);
|
char* str_dup(const char* string);
|
||||||
char* output_escape(const char* string, bool eight_bit_output);
|
char* output_escape(const char* string, bool eight_bit_output);
|
||||||
|
|
||||||
|
/* Resolve the first supported name from a rsync algorithm-preference
|
||||||
|
* environment variable (RSYNC_COMPRESS_LIST / RSYNC_CHECKSUM_LIST). `resolve`
|
||||||
|
* maps a case-insensitive name to an algorithm id (>= 0) or -1 for an unknown
|
||||||
|
* name. rsync's syntax is a whitespace-separated list (comma/colon are NOT
|
||||||
|
* separators); the client-side half ends at '&'. Unknown entries are skipped
|
||||||
|
* and the first resolvable one wins. *specified is set true when the variable
|
||||||
|
* holds at least one non-blank character. Returns the first resolvable id, or
|
||||||
|
* -1 when the variable is unset/blank or names no supported algorithm. */
|
||||||
|
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified);
|
||||||
|
|
||||||
/* Upper bound on one line/token read from a local list file (--files-from,
|
/* Upper bound on one line/token read from a local list file (--files-from,
|
||||||
* --exclude-from/--include-from, .rsync-filter). Mirrors MAX_STRING_SIZE and
|
* --exclude-from/--include-from, .rsync-filter). Mirrors MAX_STRING_SIZE and
|
||||||
* stops a hostile multi-gigabyte line from forcing unbounded allocation. */
|
* stops a hostile multi-gigabyte line from forcing unbounded allocation. */
|
||||||
|
|||||||
@@ -0,0 +1,213 @@
|
|||||||
|
"""Differential tests for the client CLI's codec defaults and env lists.
|
||||||
|
|
||||||
|
Track 3a of the rsync-parity plan pins two rsync 3.4.1 behaviors that are
|
||||||
|
resolved entirely on the client:
|
||||||
|
|
||||||
|
* the per-codec default ``--compress-level`` (zstd 3, zlib/zlibx 6, lz4
|
||||||
|
ignored) applied when the user omits ``--compress-level``/``--zl``, with an
|
||||||
|
explicit level clamped to the codec's range; and
|
||||||
|
* the ``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` preference lists that
|
||||||
|
rsync's ``auto`` consults before its compiled-in order (whitespace-separated,
|
||||||
|
unknown names skipped, first supported wins, all-unknown is exit 4).
|
||||||
|
|
||||||
|
The rsync side is observed through ``--debug=NSTR1``; FastSync publishes its
|
||||||
|
resolved codec/level through ``--debug=util``. The checksum side is confirmed
|
||||||
|
byte-for-byte through ``--out-format %C``. The rsync-based tests skip cleanly
|
||||||
|
when rsync is not installed.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(__file__))
|
||||||
|
from common import (
|
||||||
|
TEST_DATA_DIR,
|
||||||
|
run_client,
|
||||||
|
clean_dir,
|
||||||
|
get_dest_received_dir,
|
||||||
|
)
|
||||||
|
|
||||||
|
RSYNC = shutil.which("rsync")
|
||||||
|
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
|
||||||
|
|
||||||
|
CODEC_ROOT = os.path.join(TEST_DATA_DIR, "cli_differential")
|
||||||
|
|
||||||
|
_COMPRESS_RE = re.compile(r"compress(?:ion)?: (\w+) \(level (-?\d+)\)")
|
||||||
|
|
||||||
|
|
||||||
|
def _rsync(args):
|
||||||
|
env = dict(os.environ, LC_ALL="C")
|
||||||
|
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120)
|
||||||
|
|
||||||
|
|
||||||
|
def _scratch(tag):
|
||||||
|
path = os.path.join(CODEC_ROOT, tag)
|
||||||
|
clean_dir(path)
|
||||||
|
os.makedirs(path, exist_ok=True)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _make_corpus(root):
|
||||||
|
clean_dir(root)
|
||||||
|
os.makedirs(root, exist_ok=True)
|
||||||
|
with open(os.path.join(root, "big.bin"), "wb") as fh:
|
||||||
|
fh.write(b"FastSync codec payload " * 4096)
|
||||||
|
with open(os.path.join(root, "small.txt"), "wb") as fh:
|
||||||
|
fh.write(b"hello codec world\n" * 32)
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
def _rsync_compress_level(choice, level):
|
||||||
|
src = _make_corpus(_scratch(f"lvl_src_{choice}_{level}"))
|
||||||
|
dst = _scratch(f"lvl_rsync_{choice}_{level}")
|
||||||
|
args = ["-a", "-z", f"--zc={choice}"]
|
||||||
|
if level is not None:
|
||||||
|
args.append(f"--zl={level}")
|
||||||
|
args += ["--debug=NSTR1", src + "/", dst + "/"]
|
||||||
|
result = _rsync(args)
|
||||||
|
assert result.returncode == 0, result.stderr
|
||||||
|
match = _COMPRESS_RE.search(result.stdout + result.stderr)
|
||||||
|
assert match, (result.stdout, result.stderr)
|
||||||
|
return match.group(1), int(match.group(2))
|
||||||
|
|
||||||
|
|
||||||
|
def _fastsync_compress_level(choice, level, shared_server):
|
||||||
|
src = _make_corpus(_scratch(f"lvl_src_fs_{choice}_{level}"))
|
||||||
|
dst = _scratch(f"lvl_fs_{choice}_{level}")
|
||||||
|
args = ["-a", "-z", f"--zc={choice}"]
|
||||||
|
if level is not None:
|
||||||
|
args.append(f"--zl={level}")
|
||||||
|
args += ["-v", "--debug=util"]
|
||||||
|
result, _ = run_client(src, dst, flags=args, port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
match = _COMPRESS_RE.search(result.stdout)
|
||||||
|
assert match, result.stdout[:500]
|
||||||
|
return match.group(1), int(match.group(2))
|
||||||
|
|
||||||
|
|
||||||
|
class TestPerCodecCompressionLevelDefaults:
|
||||||
|
"""``--compress-level`` defaults and clamping match rsync per codec."""
|
||||||
|
|
||||||
|
# FastSync uses a positive lz4 placeholder because its "level > 0" gate
|
||||||
|
# enables compression; lz4_compress ignores the value, so rsync's level 0
|
||||||
|
# and FastSync's level 1 produce the same bytes.
|
||||||
|
CASES = [
|
||||||
|
("zstd", None, 3),
|
||||||
|
("zlib", None, 6),
|
||||||
|
("zlibx", None, 6),
|
||||||
|
("lz4", None, 1),
|
||||||
|
("zstd", 10, 10),
|
||||||
|
("zlib", 15, 9),
|
||||||
|
("zlib", 3, 3),
|
||||||
|
("lz4", 15, 1),
|
||||||
|
]
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
@pytest.mark.parametrize("choice,level,fs_level", CASES)
|
||||||
|
def test_level_matches_rsync(self, choice, level, fs_level, shared_server):
|
||||||
|
rsync_algo, rsync_level = _rsync_compress_level(choice, level)
|
||||||
|
fs_algo, fs_level_actual = _fastsync_compress_level(choice, level, shared_server)
|
||||||
|
assert rsync_algo == choice
|
||||||
|
assert fs_algo == choice
|
||||||
|
if choice == "lz4":
|
||||||
|
assert rsync_level == 0 and fs_level_actual > 0
|
||||||
|
else:
|
||||||
|
assert rsync_level == fs_level
|
||||||
|
assert fs_level_actual == fs_level
|
||||||
|
|
||||||
|
|
||||||
|
class TestEnvPreferenceLists:
|
||||||
|
"""``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` drive auto like rsync."""
|
||||||
|
|
||||||
|
# (env value, expected codec, rsync level, FastSync level)
|
||||||
|
COMPRESS_CASES = [
|
||||||
|
("zlib lz4", "zlib", 6, 6),
|
||||||
|
("lz4 zstd", "lz4", 0, 1),
|
||||||
|
("bogus zstd zlib", "zstd", 3, 3),
|
||||||
|
(" ", "zstd", 3, 3),
|
||||||
|
]
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
@pytest.mark.parametrize("env,algo,rsync_level,fs_level", COMPRESS_CASES)
|
||||||
|
def test_compress_list_matches_rsync(self, env, algo, rsync_level, fs_level, shared_server,
|
||||||
|
monkeypatch):
|
||||||
|
monkeypatch.setenv("RSYNC_COMPRESS_LIST", env)
|
||||||
|
src = _make_corpus(_scratch(f"envc_src_{algo}"))
|
||||||
|
rdst = _scratch(f"envc_rsync_{algo}")
|
||||||
|
rs = _rsync(["-a", "-z", "--debug=NSTR1", src + "/", rdst + "/"])
|
||||||
|
assert rs.returncode == 0, rs.stderr
|
||||||
|
rm = _COMPRESS_RE.search(rs.stdout + rs.stderr)
|
||||||
|
assert rm, (rs.stdout, rs.stderr)
|
||||||
|
assert rm.group(1) == algo
|
||||||
|
assert int(rm.group(2)) == rsync_level
|
||||||
|
|
||||||
|
fdst = _scratch(f"envc_fs_{algo}")
|
||||||
|
result, _ = run_client(src, fdst, flags=["-a", "-z", "-v", "--debug=util"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
fm = _COMPRESS_RE.search(result.stdout)
|
||||||
|
assert fm, result.stdout[:500]
|
||||||
|
assert fm.group(1) == algo
|
||||||
|
assert int(fm.group(2)) == fs_level
|
||||||
|
received = get_dest_received_dir(fdst, src)
|
||||||
|
assert _tree_bytes(received) == _tree_bytes(src)
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
@pytest.mark.parametrize("env,algo", [("md5", "md5"), ("sha1", "sha1"), ("xxh3 md5", "xxh3")])
|
||||||
|
def test_checksum_list_matches_rsync(self, env, algo, shared_server, monkeypatch):
|
||||||
|
monkeypatch.setenv("RSYNC_CHECKSUM_LIST", env)
|
||||||
|
src = _make_corpus(_scratch(f"envcc_src_{algo}"))
|
||||||
|
rdst = _scratch(f"envcc_rsync_{algo}")
|
||||||
|
rs = _rsync(["-a", "--checksum", "--out-format=%C %n", src + "/", rdst + "/"])
|
||||||
|
assert rs.returncode == 0, rs.stderr
|
||||||
|
rs_digests = _digests(rs.stdout)
|
||||||
|
|
||||||
|
fdst = _scratch(f"envcc_fs_{algo}")
|
||||||
|
result, _ = run_client(src, fdst, flags=["-a", "--checksum", "--out-format=%C %n"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
assert _digests(result.stdout) == rs_digests
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_all_unknown_lists_fail_like_rsync(self, shared_server, monkeypatch):
|
||||||
|
src = _make_corpus(_scratch("envbad_src"))
|
||||||
|
monkeypatch.setenv("RSYNC_COMPRESS_LIST", "bogus")
|
||||||
|
rs = _rsync(["-a", "-z", src + "/", _scratch("envbad_rsync_c") + "/"])
|
||||||
|
assert rs.returncode == 4, rs.stderr
|
||||||
|
result, _ = run_client(src, _scratch("envbad_fs_c"), flags=["-a", "-z"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
|
||||||
|
|
||||||
|
monkeypatch.setenv("RSYNC_CHECKSUM_LIST", "bogus")
|
||||||
|
rs = _rsync(["-a", src + "/", _scratch("envbad_rsync_s") + "/"])
|
||||||
|
assert rs.returncode == 4, rs.stderr
|
||||||
|
result, _ = run_client(src, _scratch("envbad_fs_s"), flags=["-a"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
|
||||||
|
|
||||||
|
|
||||||
|
def _tree_bytes(root):
|
||||||
|
out = {}
|
||||||
|
for dirpath, _dirs, files in os.walk(root):
|
||||||
|
for name in files:
|
||||||
|
path = os.path.join(dirpath, name)
|
||||||
|
with open(path, "rb") as fh:
|
||||||
|
out[os.path.relpath(path, root)] = fh.read()
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _digests(output):
|
||||||
|
out = {}
|
||||||
|
for line in output.splitlines():
|
||||||
|
parts = line.split()
|
||||||
|
if len(parts) == 2 and parts[0]:
|
||||||
|
out[parts[1]] = parts[0]
|
||||||
|
return out
|
||||||
@@ -43,6 +43,7 @@
|
|||||||
#include "test_utils.h"
|
#include "test_utils.h"
|
||||||
#include "test_xattr.h"
|
#include "test_xattr.h"
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
#include <signal.h>
|
#include <signal.h>
|
||||||
|
|
||||||
// Define global test state variables
|
// Define global test state variables
|
||||||
@@ -52,6 +53,11 @@ bool current_test_failed = false;
|
|||||||
|
|
||||||
int main() {
|
int main() {
|
||||||
signal(SIGPIPE, SIG_IGN);
|
signal(SIGPIPE, SIG_IGN);
|
||||||
|
/* The codec/checksum resolvers consult rsync's preference-list environment
|
||||||
|
* variables; clear them so a developer's shell cannot change test outcomes.
|
||||||
|
* The env-specific tests set and restore their own values. */
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
unsetenv("RSYNC_CHECKSUM_LIST");
|
||||||
printf("\033[1;36m=== RUNNING UNIT TESTS ===\033[0m\n\n");
|
printf("\033[1;36m=== RUNNING UNIT TESTS ===\033[0m\n\n");
|
||||||
|
|
||||||
RUN_TEST(test_queue);
|
RUN_TEST(test_queue);
|
||||||
|
|||||||
@@ -265,6 +265,28 @@ static void test_checksum_digest_file_matches_oneshot(void) {
|
|||||||
free(data);
|
free(data);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* RSYNC_CHECKSUM_LIST precedence, syntax and fallback. */
|
||||||
|
static void test_checksum_choice_env_list() {
|
||||||
|
unsetenv("RSYNC_CHECKSUM_LIST");
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128);
|
||||||
|
EXPECT_EQ_INT((int)checksum_negotiate_default(), (int)CHECKSUM_ALGO_XXH128);
|
||||||
|
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "bogus md5 xxh3", 1);
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_MD5);
|
||||||
|
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "SHA1", 1);
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_SHA1);
|
||||||
|
|
||||||
|
/* Whitespace-separated only: comma is not a separator in rsync's syntax. */
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "md5,xxh3", 1);
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), -1);
|
||||||
|
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", " ", 1);
|
||||||
|
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128);
|
||||||
|
|
||||||
|
unsetenv("RSYNC_CHECKSUM_LIST");
|
||||||
|
}
|
||||||
|
|
||||||
void test_checksum(void) {
|
void test_checksum(void) {
|
||||||
test_checksum_xxh64_seed0();
|
test_checksum_xxh64_seed0();
|
||||||
test_checksum_xxh64_empty();
|
test_checksum_xxh64_empty();
|
||||||
@@ -281,4 +303,5 @@ void test_checksum(void) {
|
|||||||
test_checksum_truncated_buffer_rejected();
|
test_checksum_truncated_buffer_rejected();
|
||||||
test_checksum_null_empty_digest();
|
test_checksum_null_empty_digest();
|
||||||
test_checksum_digest_file_matches_oneshot();
|
test_checksum_digest_file_matches_oneshot();
|
||||||
|
test_checksum_choice_env_list();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2431,6 +2431,119 @@ static void test_parse_args_rejects_invalid_compression_choice() {
|
|||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* rsync gives each codec its own default --compress-level; an omitted level
|
||||||
|
* resolves to that default and an explicit one is clamped to the codec range. */
|
||||||
|
static void test_parse_args_per_codec_compression_level_defaults() {
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
struct {
|
||||||
|
const char* choice;
|
||||||
|
int level;
|
||||||
|
} cases[] = {
|
||||||
|
{"zstd", 3},
|
||||||
|
{"zlib", 6},
|
||||||
|
{"zlibx", 6},
|
||||||
|
{"lz4", 1},
|
||||||
|
};
|
||||||
|
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* argv[] = {"fastsync", "-z", "--compress-choice", (char*)cases[i].choice, "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 6, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->use_compression);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, cases[i].level);
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Bare -z resolves to the zstd default. */
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* bare[] = {"fastsync", "-z", "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, bare, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, 3);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* An explicit level wins unchanged for zstd... */
|
||||||
|
cfg = config_create();
|
||||||
|
char* zv[] = {"fastsync", "-z", "--compress-level", "10", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, zv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, 10);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* ...but zlib clamps an over-range level to 9 like rsync. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* zc[] = {"fastsync", "-z", "--compress-choice", "zlib", "--compress-level", "15",
|
||||||
|
"/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 8, zc, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, 9);
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* RSYNC_COMPRESS_LIST drives the bare -z ("auto") resolution. */
|
||||||
|
static void test_parse_args_compression_env_list() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* argv[] = {"fastsync", "-z", "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "zlib lz4", 1);
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZLIB);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_level, 6);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* An explicit --compress-choice beats the env list. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* explicit_argv[] = {"fastsync", "-z", "--compress-choice", "zstd", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 6, explicit_argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* A list with no supported name is rsync's failed negotiation (exit 4). */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "bogus", 1);
|
||||||
|
cfg = config_create();
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
|
||||||
|
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
|
||||||
|
config_delete(cfg);
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
}
|
||||||
|
|
||||||
|
/* RSYNC_CHECKSUM_LIST drives the default checksum choice. */
|
||||||
|
static void test_parse_args_checksum_env_list() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* argv[] = {"fastsync", "--checksum", "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "md5", 1);
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD5);
|
||||||
|
EXPECT_EQ_INT(cfg->checksum_transfer_algo, (int)CHECKSUM_ALGO_MD5);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* An explicit --cc wins. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* cc_argv[] = {"fastsync", "--checksum", "--cc=sha1", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, cc_argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_SHA1);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* A list with no supported name is rsync's failed negotiation (exit 4). */
|
||||||
|
setenv("RSYNC_CHECKSUM_LIST", "bogus", 1);
|
||||||
|
cfg = config_create();
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
|
||||||
|
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
|
||||||
|
config_delete(cfg);
|
||||||
|
unsetenv("RSYNC_CHECKSUM_LIST");
|
||||||
|
}
|
||||||
|
|
||||||
/* Every value-taking table option accepts an inline "--opt=value" form. */
|
/* Every value-taking table option accepts an inline "--opt=value" form. */
|
||||||
static void test_parse_args_table_equals_size_options() {
|
static void test_parse_args_table_equals_size_options() {
|
||||||
Config* cfg = config_create();
|
Config* cfg = config_create();
|
||||||
@@ -4682,6 +4795,9 @@ void test_client_cli() {
|
|||||||
test_parse_args_compression_alias_equals();
|
test_parse_args_compression_alias_equals();
|
||||||
test_parse_args_rejects_invalid_compression_level_equals();
|
test_parse_args_rejects_invalid_compression_level_equals();
|
||||||
test_parse_args_rejects_invalid_compression_choice();
|
test_parse_args_rejects_invalid_compression_choice();
|
||||||
|
test_parse_args_per_codec_compression_level_defaults();
|
||||||
|
test_parse_args_compression_env_list();
|
||||||
|
test_parse_args_checksum_env_list();
|
||||||
test_parse_args_table_equals_size_options();
|
test_parse_args_table_equals_size_options();
|
||||||
test_parse_args_table_equals_string_and_int_options();
|
test_parse_args_table_equals_string_and_int_options();
|
||||||
test_parse_args_missing_argument_diagnostic();
|
test_parse_args_missing_argument_diagnostic();
|
||||||
|
|||||||
@@ -4,6 +4,7 @@
|
|||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
#include <sys/wait.h>
|
#include <sys/wait.h>
|
||||||
@@ -390,6 +391,56 @@ static void test_codec_name_mapping() {
|
|||||||
EXPECT_TRUE(compression_algo_enabled(COMPRESSION_ALGO_ZSTD));
|
EXPECT_TRUE(compression_algo_enabled(COMPRESSION_ALGO_ZSTD));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* rsync 3.4.1's per-codec default levels and its clamping ranges. */
|
||||||
|
static void test_codec_level_defaults_and_clamp() {
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), ZSTD_CLEVEL_DEFAULT);
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), 3);
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIB), 6);
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIBX), 6);
|
||||||
|
/* lz4 has no tunable level; a positive placeholder keeps the codec engaged. */
|
||||||
|
EXPECT_TRUE(compression_default_level(COMPRESSION_ALGO_LZ4) > 0);
|
||||||
|
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_NONE), 0);
|
||||||
|
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 1), 1);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 22), 22);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 23), 22);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 0), 1);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 15), 9);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIBX, 15), 9);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 1), 1);
|
||||||
|
EXPECT_TRUE(compression_clamp_level(COMPRESSION_ALGO_LZ4, 20) > 0);
|
||||||
|
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_NONE, 20), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* RSYNC_COMPRESS_LIST precedence, syntax and fallback. */
|
||||||
|
static void test_codec_choice_env_list() {
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
EXPECT_EQ_INT((int)compression_negotiate_default(), (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
|
||||||
|
/* Unknown entries are skipped; the first supported wins. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "bogus zlib lz4", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZLIB);
|
||||||
|
|
||||||
|
/* Case-insensitive. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "ZSTD", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
|
||||||
|
/* Whitespace-separated; the client half ends at '&'. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "lz4 zlib & zstd", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_LZ4);
|
||||||
|
|
||||||
|
/* Blank falls back to the compiled-in order. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", " ", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
|
||||||
|
|
||||||
|
/* rsync's syntax has no comma/colon separator: this is one unknown name. */
|
||||||
|
setenv("RSYNC_COMPRESS_LIST", "bogus,lz4", 1);
|
||||||
|
EXPECT_EQ_INT(compression_choice_resolve(), -1);
|
||||||
|
|
||||||
|
unsetenv("RSYNC_COMPRESS_LIST");
|
||||||
|
}
|
||||||
|
|
||||||
/* The process-global codec selects what the legacy wrappers produce. */
|
/* The process-global codec selects what the legacy wrappers produce. */
|
||||||
static void test_codec_global_selection() {
|
static void test_codec_global_selection() {
|
||||||
Data* original = data_create_empty(64);
|
Data* original = data_create_empty(64);
|
||||||
@@ -423,5 +474,7 @@ void test_compression() {
|
|||||||
test_chunk_compress_decompress_roundtrip();
|
test_chunk_compress_decompress_roundtrip();
|
||||||
test_codec_roundtrips();
|
test_codec_roundtrips();
|
||||||
test_codec_name_mapping();
|
test_codec_name_mapping();
|
||||||
|
test_codec_level_defaults_and_clamp();
|
||||||
|
test_codec_choice_env_list();
|
||||||
test_codec_global_selection();
|
test_codec_global_selection();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,6 +10,7 @@
|
|||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
|
#include <strings.h>
|
||||||
#include <sys/socket.h>
|
#include <sys/socket.h>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
#include <threads.h>
|
#include <threads.h>
|
||||||
@@ -578,7 +579,43 @@ static void test_getdelim_bounded() {
|
|||||||
fclose(fp);
|
fclose(fp);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static int test_env_resolver(const char* name) {
|
||||||
|
if (strcasecmp(name, "alpha") == 0)
|
||||||
|
return 10;
|
||||||
|
if (strcasecmp(name, "beta") == 0)
|
||||||
|
return 20;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_env_choice_first_parsing() {
|
||||||
|
const char* var = "FASTSYNC_TEST_CHOICE_LIST";
|
||||||
|
bool specified = true;
|
||||||
|
unsetenv(var);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
|
||||||
|
EXPECT_FALSE(specified);
|
||||||
|
|
||||||
|
/* Unknown entries are skipped, case-insensitive, first supported wins. */
|
||||||
|
setenv(var, "bogus BETA alpha", 1);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 20);
|
||||||
|
EXPECT_TRUE(specified);
|
||||||
|
|
||||||
|
/* The client half ends at '&'. */
|
||||||
|
setenv(var, "alpha & beta", 1);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 10);
|
||||||
|
|
||||||
|
/* Blank means "unspecified"; all-unknown means "specified but no match". */
|
||||||
|
setenv(var, " ", 1);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
|
||||||
|
EXPECT_FALSE(specified);
|
||||||
|
setenv(var, "nope,alpha", 1);
|
||||||
|
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
|
||||||
|
EXPECT_TRUE(specified);
|
||||||
|
|
||||||
|
unsetenv(var);
|
||||||
|
}
|
||||||
|
|
||||||
void test_shared_utils() {
|
void test_shared_utils() {
|
||||||
|
test_env_choice_first_parsing();
|
||||||
test_path_index_bounded();
|
test_path_index_bounded();
|
||||||
test_path_index_semantics();
|
test_path_index_semantics();
|
||||||
test_getdelim_bounded();
|
test_getdelim_bounded();
|
||||||
|
|||||||
Reference in New Issue
Block a user