Merge PR #308: close valid residuals of issues #286-#297
CI / lint (push) Successful in 1m45s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 21s
CI / sanitizers (address) (push) Successful in 52s
CI / sanitizers (undefined) (push) Successful in 42s
CI / build-and-test (push) Successful in 1m16s
CI / fuzz-build (push) Successful in 47s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Successful in 2m18s

This commit was merged in pull request #308.
This commit is contained in:
2026-09-22 17:30:36 +02:00
16 changed files with 1523 additions and 119 deletions
+29 -1
View File
@@ -21,7 +21,9 @@ reclassification is `--filter=RULE` moving ✅ → ⚠️, because its merge-onl
`e`/`n`/`w`/`-` modifiers are now accepted and consumed but their semantics
remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ /
11 ⚠️ / 27 ❌** of 157 rows. The affected rows' notes and the summary tally in
`RSYNC_COMPAT.md` were updated.
`RSYNC_COMPAT.md` were updated. A following triage-fix cycle (see **Triage
fixes** below) moves `-F` and `-i` to ⚠️, for a final **117 ✅ / 13 ⚠️ / 27 ❌**
of 157 rows.
### Changed
@@ -135,6 +137,32 @@ remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ /
`CHANGELOG.md` and `HANDOFF.md` were updated for the audit cycle; the
`RSYNC_COMPAT.md` summary tally was corrected to match the rows.
### Triage fixes
- **`--dirs` directory xattrs applied inline.** A `-d/--dirs` transfer now
applies captured directory `-X`/`-A` xattrs fd-relative on the directory entry
instead of dropping them, so directory xattrs survive the non-recursive path
(`src/shared/file_save.c`, `tests/test_xattr.c`).
- **Directory/root itemize and `--out-format` lines.** `-i`/`--itemize-changes`
and `--out-format` now emit the transfer-root `./` line and per-directory
`cd...`/`.d..t...` lines, rendered by the shared itemize code. This matches
rsync's fresh-transfer output; because the root line is unconditional and an
incremental re-run may itemize directories/symlinks that rsync's quick-check
leaves silent, `-i` is now a ⚠️ Caveat row.
- **FROM name globs for identity maps.** `--usermap`/`--groupmap` `FROM` tokens
now accept `*`/`?`/`[...]` globs, expanded sender-side against the passwd/group
database and collapsed into bounded numeric ranges (`MAX_IDENTITY_MAP`),
matching rsync.
- **Transport fallback unit tests.** Added unit coverage for the TCP/TLS
transport fallback paths (`tests/test_transport_tcp.c`,
`tests/test_transport_tls.c`).
- **Docs corrections.** `RSYNC_COMPAT.md`/`README.md` corrected stale parity
claims for issues #286–#297: the `-F` and `-i` reclassifications, the
`--munge-links` direction, the accepted checksum/compression name sets,
`--bwlimit` parsing, `--stop-at` grammar, `--trust-sender`, symlink xattrs, and
the native/non-interoperable batch and credential notes. The summary tally is
now **117 ✅ / 13 ⚠️ / 27 ❌** of 157 rows.
## [2.28.0] - 2026-09-20
The rsync-parity cycle. `PROTOCOL_VERSION` moves `2.26.0 → 2.27.0 → 2.28.0`;
+20 -18
View File
@@ -233,9 +233,9 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `-h, --human-readable` | Format transfer byte/rate counts with rsync's decimal (base-1000) units |
| `--max-depth <n>` | Maximum directory depth to recurse (0 = unlimited, default: 0) |
| `--log-file <path>` | Write log messages to file instead of stderr |
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree |
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server) |
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server) |
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree (FastSync-native format, not rsync-interoperable) |
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server); FastSync-native format, not rsync-interoperable |
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server); FastSync-native format, not rsync-interoperable |
| `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
| `--dest-dir <path>` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) |
| `--save-to-disk` | Write received files to disk |
@@ -248,12 +248,12 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `-4, --ipv4` | Force IPv4 for destination resolution |
| `-6, --ipv6` | Force IPv6 for destination resolution |
| `--sockopts=OPTS` | Comma-separated OPT=VAL socket options applied before connect (`TCP_NODELAY`, `SO_KEEPALIVE`, `SO_RCVBUF`, `SO_SNDBUF`, `SO_REUSEADDR`) |
| `--bwlimit <KB/s>` | Bandwidth limit in kilobytes per second; also paces `--sendfile` transfers |
| `--bwlimit <RATE>` | Bandwidth limit, using rsync's exact `parse_size_arg` grammar: a bare value is KiB/s; `K`/`M`/`G`/`T`/`P` are binary suffixes; `KB`/`MB` are decimal and `KiB`/`MiB` binary; decimals are accepted and quantized to whole KiB; `0` (or empty) means no limit. Also paces `--sendfile` transfers |
| `--chunk-size <n>` | Chunk size in bytes (default: 10485760) |
| `--timeout <sec>` | I/O timeout in seconds, applied to both the socket (`SO_RCVTIMEO`/`SO_SNDTIMEO`) and the per-message protocol poll deadline. Default `0` = disabled (matching rsync); `0` disables it. `--no-timeout` is the negation. The value is not sent on the wire; the server side keeps its own safe floor. |
| `--contimeout <sec>` | Connection timeout in seconds (default: 60, matching rsync); `0` disables it (`--no-contimeout` is the negation) |
| `--stop-after=MINS` | Stop the transfer after MINS minutes (a positive integer); whatever was already transferred is kept |
| `--stop-at=TIME` | Stop at an absolute time (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`); an early stop skips the late `--delete` keep-set |
| `--stop-at=TIME` | Stop at an absolute time. Accepts rsync's `parse_time` forms (`Y-M-DTh:m`, `Y/M/DTh:m`, `Y-M-D`, `M-D`, `D`, `h:m`, `:m`, `T h:m`; omitted fields resolve to the next matching point in the local timezone), plus `now+N[smhd]` and FastSync's `HH:MM`/`HH:MM:SS` clock-time spelling. An early stop skips the late `--delete` keep-set |
| `-b, --backup` | Backup existing destination files before overwriting |
| `--backup-dir <dir>` | Target directory for backups (requires `--backup`) |
| `--tls` | Enable TLS encryption |
@@ -535,7 +535,7 @@ features without changing the meaning of ordinary compatibility options.
| `--server-host <host>` | Select the TCP server host. |
| `--server-port <port>` | Select the TCP server port (`--port <port>` and `--port=<port>` are rsync-friendly aliases). |
| `--tls` | Enable TLS for TCP transport. |
| `--bwlimit <KB/s>` | Apply token-bucket bandwidth limiting (also paces `--sendfile` transfers). |
| `--bwlimit <RATE>` | Apply token-bucket bandwidth limiting with rsync's exact `parse_size_arg` grammar (bare = KiB/s, `K`/`M`/`G`/`T`/`P` binary, `KB`/`MB` decimal, `KiB`/`MiB` binary, decimals quantized to whole KiB, `0`/empty = no limit; also paces `--sendfile` transfers). |
| `--progress` | Show rsync-style per-file progress blocks from the receiver's wire counters; the root `./` line is printed whenever progress is active (rsync prints it only when the transfer root is created). |
| `--stats` | Print transfer statistics, including the receiver-only counters reported over the wire; `Number of files`/`Number of created files` carry rsync's per-type breakdown (deleted files are a single total). |
| `--timeout <seconds>` | Set the socket **and** per-message protocol I/O timeout. Default `0` = disabled (matching rsync); `0` disables it. |
@@ -613,11 +613,11 @@ remote SSH argv is already built injection-safe.
| `--partial-dir <dir>` | Set a relative partial-transfer directory below the server destination root. Implies `--partial`. Rejected together with `--inplace` (`--inplace cannot be used with --partial-dir`, matching rsync), because the inplace path bypasses partial/temp staging. |
| `--inplace` | Write directly to the destination instead of using a temporary file. Cannot be combined with `--partial-dir`. |
| `--fsync` | Fsync every written file before publication. |
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree. |
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server). |
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server). |
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree (FastSync-native format, not rsync-interoperable). |
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server); FastSync-native format, not rsync-interoperable. |
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server); FastSync-native format, not rsync-interoperable. |
| `--stop-after=MINS` | Stop the transfer after MINS minutes; whatever was already transferred is kept. |
| `--stop-at=TIME` | Stop at an absolute time (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`). An early stop skips the late `--delete` keep-set. |
| `--stop-at=TIME` | Stop at an absolute time. Accepts rsync's `parse_time` forms (`Y-M-DTh:m`, `Y/M/DTh:m`, `Y-M-D`, `M-D`, `D`, `h:m`, `:m`, `T h:m`; omitted fields resolve to the next matching point in the local timezone), plus `now+N[smhd]` and FastSync's `HH:MM`/`HH:MM:SS` clock-time spelling. An early stop skips the late `--delete` keep-set. |
### Metadata and links
@@ -689,7 +689,7 @@ remote SSH argv is already built injection-safe.
| `--fastsync-server-path <path>` | Remote FastSync server path for SSH mode (client-only; never crosses the wire). |
| `--rsync-path <path>` | Alias for `--fastsync-server-path`. |
| `-M`, `--remote-option=OPT` | Append OPT to the remote server invocation over SSH (repeatable; rejected for daemon/TCP destinations). |
| `--trust-sender` | Receiver-local: trust the remote sender's file list and skip path re-validation (does not affect symlink targets). |
| `--trust-sender` | Receiver-local: trust the remote sender's file list and skip path re-validation (does not affect symlink targets). **On the client this flag alone is inert** — it is never sent on the wire; the server must be started with its own `--trust-sender`, or the client must forward it with `-M--trust-sender` (SSH only). |
| `--timeout <sec>` | Socket + per-message I/O timeout; default `0` = disabled. |
| `--contimeout <sec>` | Connection timeout; default 60; `0` disables. |
| `--source-dir <path>` | Set the source directory explicitly. |
@@ -732,14 +732,14 @@ remote SSH argv is already built injection-safe.
| `-6`, `--ipv6` | Bind an IPv6 socket. |
| `--allow-delete` | Permit client delete manifests. Deletion is refused by default. This also gates `--force` (which can recursively replace/remove a destination directory tree). |
| `--allow-super` | Standalone TCP listener only: keep super-user activities enabled for a **root** receiver. Without it a root standalone server forces `SUPER_MODE_OFF`, so client `--devices`/`--write-devices`/`--super` and client-chosen ownership requests are skipped/refused. Rejected with `--stdio` (the SSH remote argv is client-composed; use a forced command if the default must hold). No effect when not root. Daemon modules opt in per module with `client owner = yes`. |
| `--trust-sender` | Trust the remote sender's file list: skip the receiver's up-front path-traversal re-validation (fewer checks, faster, potentially unsafe; off by default). It does not affect symlink targets, which are stored verbatim either way. |
| `--trust-sender` | Trust the remote sender's file list: skip the receiver's up-front path-traversal re-validation (fewer checks, faster, potentially unsafe; off by default). It does not affect symlink targets, which are stored verbatim either way. A client `--trust-sender` is never sent over the wire — the server must set this flag itself, or the client must forward it via `-M--trust-sender`. |
| `--no-super` | Operator veto: never attempt super-user activities (ownership, device nodes) even as root, and refuse any client `--copy-as`/`--super` request. |
| `--allow-unauthenticated` | Permit plaintext/anonymous network clients; an auth-required module still accepts only opted-in loopback plaintext. |
| `--iconv=LOCAL[,REMOTE]` | Declare this server's LOCAL charset for file-name conversion. |
| `--password-file=FILE` | Credential store for modules that declare `auth users`. Requires `--daemon`. |
| `--early-input=FILE` | Second credential store layered over `--password-file`. Requires `--daemon`. |
| `--hash-credentials <file>` | Read `<file>`'s `user:password` lines and print PBKDF2 credential-store lines to stdout, then exit. Cannot be combined with `--daemon` or `--stdio`. |
| `--iterations N` | PBKDF2 iteration count for `--hash-credentials` (default 600000, range 100000–10000000). Requires `--hash-credentials`. |
| `--password-file=FILE` | Credential store for modules that declare `auth users`. Requires `--daemon`. FastSync-native SCRAM/PBKDF2 format, not rsync-interoperable. |
| `--early-input=FILE` | Second credential store layered over `--password-file`. Requires `--daemon`. FastSync-native format, not rsync-interoperable. |
| `--hash-credentials <file>` | Read `<file>`'s `user:password` lines and print PBKDF2 credential-store lines to stdout, then exit. Cannot be combined with `--daemon` or `--stdio`. FastSync-native, not rsync-interoperable. |
| `--iterations N` | PBKDF2 iteration count for `--hash-credentials` (default 600000, range 100000–10000000). Requires `--hash-credentials`. FastSync-native, not rsync-interoperable. |
| `-v`, `--verbose` | Enable debug logging. |
| `--help` | Print server usage. |
@@ -897,8 +897,10 @@ The project will reach the drop-in replacement goal in stages:
completion wave's scope; the tests live in `tests/integration/` and skip
cleanly when rsync is unavailable.
3. `-a` implements full rsync `-rlptgoD`; under `-p` the source mode is copied
exactly (no masking). Ownership application stays privilege-gated, as in
rsync.
exactly, including group/other-write bits, with setuid/setgid/sticky copied
only when super-user activities are permitted (masked under
`SUPER_MODE_OFF`/`--no-super`). Ownership application stays privilege-gated,
as in rsync.
4. Symlink (verbatim storage), sparse-file, metadata, delete-policy (including
`--max-delete` partial + exit 25, per-directory `--delete-during`/
`--delete-delay`), codecs, and resumable-write semantics are implemented;
+54 -39
View File
@@ -6,8 +6,8 @@ This document maps rsync's full feature set to FastSync's current implementation
| Status | Count | Description |
|--------|-------|-------------|
| ✅ Parity | 119 | Reproduces rsync's semantics for this option's scope |
| ⚠️ Caveat | 11 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
| ✅ Parity | 117 | Reproduces rsync's semantics for this option's scope |
| ⚠️ Caveat | 13 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
| ❌ Divergent | 27 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
| **Total** | **157** | One row per rsync option/feature group; a row may name several spellings |
@@ -62,7 +62,7 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
- **Fuzzy eligibility.** The `-y/--fuzzy` candidate search no longer inherits the ordinary delta engine's 16 KiB minimum or 10× ratio bound, so an oversized or sub-16-KiB sibling is reused as rsync reuses it (`test_parity_basis_fuzzy.py`).
- **Output partials.** `--info=mount`/`--info=stats`, the `--stats` `dir:` breakdown under `-r`, and real `--debug` output for `flist`/`del`/`hash`/`deltasum`/`recv`/`filter`/`send` were added (`test_parity_info_mount_stats.py`, `test_output_parity.py`, `test_parity_debug.py`); those rows stay ⚠️ for their remaining documented residuals. `--delete-before`'s phase-0 late-file divergence and the `--progress` root/ancestor/symlink feedback remain open (they need a receiver→sender event channel), and the >256 MiB single-file streaming limit (B4) was not addressed. The matrix is now **120 ✅ / 10 ⚠️ / 27 ❌ = 157**.
**Audit cycle (no wire change; `PROTOCOL_VERSION` stays 2.28.0).** A security-and-correctness audit pass ran against the parity-2.29 baseline, followed by a set of audit follow-ups (filter merge modifiers, the `--inplace`/`--partial-dir` conflict, credential-file hardening, and small leak/log/test fixes). The only classification change is `--filter=RULE` moving ✅ → ⚠️, because its merge-only `e`/`n`/`w`/`-` modifiers are now accepted and consumed but their semantics remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ / 11 ⚠️ / 27 ❌ = 157**. The affected rows (`-z`/`--compress`, `--bwlimit`, `-T`/`--temp-dir`, `-p`/`--chmod`, `--partial-dir`, `--filter`) had their notes updated in place:
**Audit cycle (no wire change; `PROTOCOL_VERSION` stays 2.28.0).** A security-and-correctness audit pass ran against the parity-2.29 baseline, followed by a set of audit follow-ups (filter merge modifiers, the `--inplace`/`--partial-dir` conflict, credential-file hardening, and small leak/log/test fixes). The only classification change is `--filter=RULE` moving ✅ → ⚠️, because its merge-only `e`/`n`/`w`/`-` modifiers are now accepted and consumed but their semantics remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ / 11 ⚠️ / 27 ❌ = 157**. The affected rows (`-z`/`--compress`, `--bwlimit`, `-T`/`--temp-dir`, `-p`/`--chmod`, `--partial-dir`, `--filter`) had their notes updated in place. A later triage cycle moved `-F` and `-i` ✅ → ⚠️ (see the triage-cycle note below), giving **117 ✅ / 13 ⚠️ / 27 ❌ = 157**:
- **Decompression ceiling.** `MAX_DECOMPRESSED_SIZE` was 100 MiB while the receiver advertises and the sender compresses whole files up to `MAX_RECEIVE_WHOLE_FILE_SIZE` (256 MiB), so `-z` on a 100–256 MiB regular file failed with `Declared decompressed size exceeds 104857600 bytes`. The ceiling is now defined in terms of the protocol whole-file bound (still a real allocation-clamped bomb guard), so the two cannot drift; `-z` on 100–256 MiB files now works.
- **`--bwlimit` with `--sendfile`.** The plaintext-TCP `--sendfile` fast path wrote through `sendfile(2)` without passing through the protocol's token bucket, so `--bwlimit` was ignored on that path. It is now paced through the same per-session leaky bucket, so TLS and plaintext transports share identical `--bwlimit` semantics.
@@ -73,6 +73,8 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
- **Bounds and wire validation.** `--filter` rule count is now checked client-side against `MAX_FILTER_RULES` (with an actionable message before any network I/O) rather than surfacing as an opaque receiver protocol error; `send_protect_entries()` still re-checks the expanded count. Unknown wire `Status` values are rejected as protocol errors (`status_is_valid()`), and the audit also fixed a mutex leak on an init-failure path, an `errno`-after-`free()` in deferred delete application, `log_perror` misuse for non-`errno` conditions, `SSL_read` length clamping, `sendfile` `poll` `EINTR` retry, and printf-format/attribute issues.
- **Credential-file hardening follow-up.** `secret_file_open()` now opens `--password-file`/`--early-input`/`--hash-credentials` inputs with `O_NOFOLLOW`, so a symlinked credential path fails closed (`ELOOP`) instead of being followed before the owner/mode gate; literal fd-backed paths (`/dev/fd/<digits>`, `/proc/self/fd/<digits>`, which is what a bash process substitution passes) are exempt, so process substitution still works. A FIFO/process-substitution read now waits under a bounded ~3 s deadline for its writer, so a slow producer works while a connected-but-silent FIFO fails instead of hanging. The follow-up also fixed a `config_create` allocation leak on its `server_host` failure path (`config_delete` now releases it), corrected the decompression-limit log message to print the effective bound rather than the compile-time ceiling, and hardened the daemon umask/root test fixtures.
**Triage cycle (no wire change; `PROTOCOL_VERSION` stays 2.28.0).** A documentation-and-correctness triage pass over the parity baseline corrected stale prose and reclassified two rows that carried a real behavioral residual: `-F` moves ✅ → ⚠️ (its own note already documented that per-directory merge rules are not carried to the receiver filter engine, so a destination-only entry matching ONLY a `.rsync-filter` rule is not shielded from `--delete`), and `-i`/`--itemize-changes` moves ✅ → ⚠️ (directory and transfer-root lines are now emitted, but the root `./` line is emitted unconditionally and an incremental re-run itemizes directories/symlinks that rsync's quick-check leaves silent). The matrix is **117 ✅ / 13 ⚠️ / 27 ❌ = 157**.
**Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the
remaining gaps the rsync-parity wave left open (delete timing, wire counters and
output, codec breadth, general `-R`/`-d`, the filter grammar (the unsupported
@@ -121,10 +123,10 @@ Every one of those has an entry below with its remaining caveats.
|------|-------------------|-----------------|-------|
| `--stats` | Give transfer stats | ⚠️ Caveat | Prints transfer statistics. Protocol 2.25.0 populates the receiver-only counters the sender cannot observe (`Matched data`, `Number of deleted files`) from the receiver's `STATUS_STATS` report; the sender tracks the scanned file list per type so `Number of files` carries rsync's `(reg: X, dir: Y, link: Z, special: W)` breakdown (directories come from the scanner's captured directory list for `-a`/`-t`/`-p`, or from a lightweight traversed-directory counter on a plain `-r` run so the `dir:` category is present there too), `Number of regular files transferred` excludes symlinks/specials and up-to-date files, `Total file size` includes symlink target lengths, and `Total transferred file size` counts only transferred files. **Protocol 2.28.0 extends `STATUS_STATS`** with receiver-observed `literal_bytes` and the four `created_*` counters: `Number of created files` now carries rsync's `(reg/dir/link/special)` breakdown (the receiver reports which destination entries it newly created, including implicitly-created parent directories below the transfer root) and `Literal data` is exact for a delta transfer (the receiver counts the literal fragments it stored, not the whole source size) — all differential-tested in the sequential and `--threads` paths against rsync 3.4.1 for fresh-create, update and delta shapes. **Remaining divergences:** rsync's per-type breakdown on `Number of deleted files` is not reproduced; and `Total bytes sent`/`received` are FastSync wire bytes framed differently from rsync's, so they are not numerically comparable |
| `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units. **A lone `-h` with no transfer arguments prints help instead** (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer remains human-readable |
| `-i`, `--itemize-changes` | Per-file change summary | ✅ Parity | Prints rsync-style `>f+++++++++` lines to stdout only for files actually sent (also under `-j`/`--threads`); unchanged files print nothing, matching single-`-i` behavior |
| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision; **every** non-root directory is rendered as created (`cd+++++++++`) because the sender never probes a directory's destination state, so a pre-existing destination directory that rsync reports as unchanged (`.d..t......`) is still itemized as created — this is not limited to re-runs; an incremental re-run additionally itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`); and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
| `--progress` | Show progress | ⚠️ Caveat | Protocol 2.25.0 prints rsync-style per-file progress blocks (percent, transferred/total bytes, rate, elapsed, `(xfr#N, to-chk=M/T)`) fed by the receiver's `STATUS_STATS`, in both the sequential and `--threads` send paths. FastSync also prints rsync's leading `./` transfer-root line and, when progress is requested (`--progress`/`-P`/`--info=progress`) and not `--quiet`, runs a **paths-only metadata pre-scan** (no file reads, no hashing) that supplies rsync's file-list total `T` for the `to-chk` denominator and the directory names; `--delete-during`/`--delete-delay` reuse their existing keep-set pre-scan instead of walking twice, and non-progress runs are untouched. Per-directory name lines are emitted (trailing `/`), and symlink (` -> target`) and special entries are named too, so a **fresh multi-directory tree's name set and `to-chk` denominator match rsync 3.4.1** (differential test, sequential and `--threads`) and a **single-file transfer's name lines and deterministic frames remain byte-identical** to rsync. **Order parity (parity-2.29):** the sequential scanner now emits entries in rsync's sorted depth-first flist order (non-directories ascending, then directories ascending), so the interleaving and the `to-chk` numerator match rsync for the default single-threaded transfer (differential `test_parity_order.py`; `--threads` has no rsync analogue and stays unordered). **Remaining divergences:** the leading `./` root line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an ancestor directory line is emitted whenever a child transfers (rsync suppresses it when the directory itself is unchanged); on a re-run, entries without a quick-check (symlinks, empty directories) are still named where rsync stays silent; and the rate/ETA are wall-clock dependent |
| `-P` | Same as --partial --progress | ✅ Parity | Parses to `--partial` + `--progress`. The independent `--partial` retention semantics are rsync parity: an interrupted write retains the already-written temp at the destination (best-effort) so a later `--append`/`--append-verify` can resume. Progress presentation is owned by the `--progress` row; there is no separate `-P` divergence |
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible |
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally, every non-root directory renders as created because the sender does not probe directory destination state (so a pre-existing unchanged directory still shows `cd+++++++++`), and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
| `--log-file=FILE` | Log to file | ✅ Parity | `log_file` config field |
| `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` as the wire byte count) |
| `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Parity | Applies to displayed paths and protocol debug output |
@@ -148,7 +150,7 @@ Every one of those has an entry below with its remaining caveats.
| `--ignore-existing` | Skip updating existing files | ✅ Parity | `ignore_existing` config field (crosses the wire; receiver-side policy). Protocol 2.26.0 short-circuits in the per-file check **before any payload**: when the destination entry already exists, the receiver answers the skip during the incremental handshake instead of letting the sender stream data that would be discarded, so an existing 4 MiB destination costs only the config/check frames (verified with a counting proxy, matching rsync). The write-time paths (regular, delay-updates-staged, hardlink-sibling, special/device) still return `FILE_SAVE_SKIPPED` without overwriting, and `--backup` is disabled for skipped files. Like rsync, it does not apply to directories/symlinks. Combines with `-j`/`--threads` and `--delay-updates` |
| `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Parity | |
| `-x`, `--one-file-system` | Do not cross filesystem boundaries | ✅ Parity | Sender scanner captures the root device and does not descend into mount-point crossings (`st_dev` differs). **Protocol 2.23.0 matches rsync's entry emission:** the mount-point directory itself is emitted as a payload-less directory entry (so the destination gets an empty directory) while its contents are skipped; previously the crossing subdirectory was dropped entirely |
| `-F` | Add the default `.rsync-filter` rules | ✅ Parity | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (first match wins). **A single `-F` transfers the `.rsync-filter` files themselves, matching rsync; a repeated `-FF` additionally excludes them** (rsync 3.4.1's `-F`/`-FF` are exactly these two rules, with no `.cvsignore` branch). Unsupported/unparseable rules inside a per-directory file fail the scan with a clear error. **Residual (track 4a):** per-directory rules are still enforced receiver-side only through the sender-derived source-mirror protected prefixes; the base-rule receiver filter engine does not carry per-directory rules, so a destination-only entry matching ONLY a `.rsync-filter` rule is not yet shielded from `--delete` |
| `-F` | Add the default `.rsync-filter` rules | ⚠️ Caveat | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (first match wins). **A single `-F` transfers the `.rsync-filter` files themselves, matching rsync; a repeated `-FF` additionally excludes them** (rsync 3.4.1's `-F`/`-FF` are exactly these two rules, with no `.cvsignore` branch). Unsupported/unparseable rules inside a per-directory file fail the scan with a clear error. **Residual (track 4a):** per-directory rules are still enforced receiver-side only through the sender-derived source-mirror protected prefixes; the base-rule receiver filter engine does not carry per-directory rules, so a destination-only entry matching ONLY a `.rsync-filter` rule is not yet shielded from `--delete` |
## 4. Directory Options
@@ -337,7 +339,7 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
| `-E`, `--executability` | Preserve executability | ✅ Parity | Preserves executable permission bits (implies metadata preservation) |
| `--chmod=CHMOD` | Affect file permissions | ✅ Parity | Faithful port of rsync 3.4.1's `parse_chmod`/`tweak_mode`: numeric octal and symbolic `ugo`/`rwx` changes, `D`/`F` directory/file selectors, `X` (execute only on directories or already-executable files), `s`/`t` setuid/setgid/sticky, and append semantics — repeated clauses and repeated `--chmod` options accumulate in order (joined with commas). The changes are applied to the new mode **without sanitization** (matching rsync), except that setuid/setgid/sticky are masked when the connection forbids super-user activities (audit-cycle fix, see `-p`), and `--chmod` does **not** imply `-p` (rsync parity). Applied to files and directories on the receiver |
| `-A`, `--acls` | Preserve ACLs | ✅ Parity | Implemented on Linux via the POSIX-ACL xattr representation: the sender captures the `system.posix_acl_access` / `system.posix_acl_default` xattrs and the receiver re-applies them fd-relative. A differential test with `setfacl` confirms the complete access and default ACL sets (including `mask`) are identical to rsync's on a directory. libacl is not required; a `fsetxattr` an unprivileged receiver may not perform is logged and skipped, never fatal. Only the `system.posix_acl_*` namespaces plus `user.*` are ever applied; privileged namespaces are never applied. Implies metadata transmission |
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Divergent | Deliberately restricted to unprivileged `user.*` extended attributes plus the two POSIX ACL xattrs; `security.*` (SELinux, capabilities, ...) and `trusted.*` are **never** captured or applied — a client can never force a privileged attribute onto the destination, and the receiver independently re-validates every incoming name against the whitelist. This is a security-policy divergence from rsync, which can preserve the privileged namespaces with the needed privilege; implementing them would defeat FastSync's privilege-escalation guard. `user.*` capture/apply matches rsync in a differential test. Payloads are bounded on both ends. Incompatible with `-s` |
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Divergent | Deliberately restricted to unprivileged `user.*` extended attributes plus the two POSIX ACL xattrs; `security.*` (SELinux, capabilities, ...) and `trusted.*` are **never** captured or applied — a client can never force a privileged attribute onto the destination, and the receiver independently re-validates every incoming name against the whitelist. This is a security-policy divergence from rsync, which can preserve the privileged namespaces with the needed privilege; implementing them would defeat FastSync's privilege-escalation guard. `user.*` capture/apply matches rsync in a differential test. Payloads are bounded on both ends. Incompatible with `-s`. **Also divergent: symlink xattrs/ACLs are not captured or applied** — `-X`/`-A` with `-l` carries only the link's owner/times/mode, not its xattrs (the capture uses path-following `listxattr`/`getxattr`, so the link's own xattrs are never read, and the receiver's symlink write path applies no xattr block). Closing this needs a dedicated symlink-xattr wire block and a `PROTOCOL_VERSION` bump |
| `-H`, `--hard-links` | Preserve hard links | ✅ Parity | Files on the source that share an inode (`st_dev`+`st_ino`, e.g. a `cp -al` tree) are re-created as hard links to one another on the destination, so duplicate links stay deduplicated and only the first member's data is sent (later members are transmitted as payload-less `STATUS_HARDLINK` frames). The receiver links each sibling to the first member's installed file with an atomic link + rename; on `link()` failure it falls back to a byte-identical local copy of the first member, never a partial/corrupt file. Requires the sequential scan for ordering (the first member is always emitted and installed before any sibling is linked). Works single-threaded and under `-j`/`--threads`, `--inplace`, `--delay-updates` (links staged and published by rename) and `--partial`. Crosses the wire (`preserve_hard_links` bool; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0**, peers must match). Incompatible with `-s` (chunk serialization) and `--append`/`--append-verify`, rejected up front with a distinct error. See the Phase-4 hard-links notes below |
| `-D` | Same as --devices --specials | ✅ Parity | Implies `--devices --specials`. `-D` was unassigned in FastSync (verified: no collision), so it is free to imply both device-node and special-file preservation. As of protocol 2.23.0 `--specials` genuinely covers **both FIFOs and unix sockets**, so `-D` covers the full rsync set. See the `--devices`/`--specials` rows and the Phase-4 devices notes below |
| `--devices` | Preserve device files | ❌ Divergent | Recreates char/block device nodes with `mknodat` (type + rdev strictly validated, confined fd-relative below the receive root), but only when the receiver has `CAP_MKNOD`: a non-root receiver logs a warning and skips the entry instead of erroring, so a transfer with devices never aborts. Deliberate privilege-model divergence from rsync, which errors when it cannot create the node. `--specials` (FIFOs and unix sockets) is unprivileged and remains parity |
@@ -352,7 +354,7 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
| `--fake-super` | Store/recover privileged attrs via xattrs | ❌ Divergent | Records the resolved `uid:gid:mode:mtime_sec:mtime_nsec` in a reserved `user.fastsync.stat` xattr and immediately replays mode/times fd-relative, but **never performs a real `chown`** (the owner is recorded for a later privileged restore). The on-disk key and format are FastSync-native, not rsync's `user.rsync.%stat%`, so recordings are not interoperable with rsync — the same class as the native auth and batch formats. Implies metadata transmission; incompatible with `-s` |
| `--open-noatime` | Avoid changing access time when opening files | ✅ Parity | Sender-side policy: the sender opens source files with `O_NOATIME` (Linux) when reading them for transfer, so the open/read does NOT bump the source's on-disk access time. Degrades safely when `O_NOATIME` is unavailable (not defined) or refused (`EPERM`, since it needs `CAP_FOWNER` or file ownership): the code falls back to a normal open, so the data always transfers — only the atime-bump is skipped. It does not itself capture/preserve atime; it only avoids modifying it. **Client-only, never crosses the wire.** Exposed as `file_open_for_read()` and applied to both the buffered data path and the sendfile path |
| `--numeric-ids` | Do not map uid/gid by name | ✅ Parity | **A mapping modifier only:** when ownership is being applied it uses the transmitted numeric uid/gid directly, skipping the name lookup. It does **not** request ownership application on its own — combine it with `-o`/`-g`, `-a`, or an explicit map (`--chown`/`--usermap`/`--groupmap`) — and it does not need any metadata flag merely to parse. Ownership is only applied when metadata (hence the source uid/gid) is actually transmitted (see the Phase-4 identity notes) |
| `--usermap=STRING` | Map usernames | ✅ Parity | Opt-in ownership application. Comma-separated `FROM:TO` rules evaluated in order, first match wins. `FROM` accepts a source-resolved user name, an `@N`/bare `N` numeric id, an inclusive `LOW-HIGH` id range, `*`, or an empty field (ids with no source name). `TO` accepts a receiver-resolved **name** (protocol 2.26.0 resolves it on the receiving side against the receiver's account database, matching rsync), an `@N`/bare `N` id, or `*` (the receiving process's euid). Rules travel as resolved numeric pairs plus an optional TO name; the receiver applies a matching rule, else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup, via fd-relative `fchown`. Malformed specs are clear errors. Implies metadata; only effective where the receiver can chown (otherwise a warning) |
| `--usermap=STRING` | Map usernames | ✅ Parity | Opt-in ownership application. Comma-separated `FROM:TO` rules evaluated in order, first match wins. `FROM` accepts a source-resolved user name, a name **glob** (`*`/`?`/`[...]`, expanded sender-side at CLI-parse time against the sender's passwd/group database and collapsed into numeric `LOW-HIGH` ranges, bounded by `MAX_IDENTITY_MAP`), an `@N`/bare `N` numeric id, an inclusive `LOW-HIGH` id range, `*`, or an empty field (ids with no source name). `TO` accepts a receiver-resolved **name** (protocol 2.26.0 resolves it on the receiving side against the receiver's account database, matching rsync), an `@N`/bare `N` id, or `*` (the receiving process's euid). Rules travel as resolved numeric pairs plus an optional TO name; the receiver applies a matching rule, else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup, via fd-relative `fchown`. Malformed specs are clear errors. Implies metadata; only effective where the receiver can chown (otherwise a warning) |
| `--groupmap=STRING` | Map group names | ✅ Parity | Same rules and receiver-side `TO`-name resolution as `--usermap`, applied to the group (gid) side |
| `--chown=USER:GROUP` | Map owner and group | ✅ Parity | Opt-in ownership override. Forms `USER:GROUP`, `USER`, `:GROUP`; `*` means the current user/group as appropriate; `@N`/bare `N` ids; a name may escape `:` as `\:`. A name that resolves on the sender is sent as an id; an unresolvable name is carried as a receiver-resolved `TO` name (protocol 2.26.0), matching rsync's receiver-side resolution. Equivalent to a trailing `*:*` usermap+groupmap rule (an explicit map match wins). Conflicts with `--usermap`/`--groupmap` on the same side are a clear configuration error. Implies metadata; a non-root receiver warns and continues (rsync parity) |
| `--copy-as=USER[:GROUP]` | Perform the copy as another user/group | ❌ Divergent | Close-refusal safe subset. FastSync never switches process credentials (its receiver is multithreaded, so a real `setuid`/`setgid` would be unsafe); instead the receiver forces the ownership of every entry it writes to the client-resolved ids through the confined fd-relative identity path. A privileged (root) receiver is required: an unprivileged receiver refuses the whole transfer at the config handshake, before any data, rather than produce wrong ownership. Deliberate divergence from rsync's real identity switching; a daemon refuses it unless the module sets `client owner = yes` |
@@ -497,8 +499,10 @@ names, `--chown` names) are resolved to numbers at CLI parse time against the
**client (sender) machine's** account databases; this reproduces rsync's
semantics on a shared-account source/destination and is documented for a
genuinely different destination. The interesting named-value subset is
supported (`*` FROM wildcard, `*` TO = current user, `@N`/bare-`N` numerics); a
lone-`@` "use the FROM value unchanged" rsync form is not implemented. Also
supported (FROM name globs `*`/`?`/`[...]` expanded sender-side against the
passwd/group database and bounded by `MAX_IDENTITY_MAP`, `*` FROM wildcard,
`*` TO = current user, `@N`/bare-`N` numerics); a lone-`@` "use the FROM value
unchanged" rsync form is not implemented. Also
unlike rsync, plain `-M` never applies ownership and `--usermap`/`--groupmap`/
`--chown` each imply metadata preservation so the source uid/gid actually travel
(the flags only take effect where ownership is being preserved/applied).
@@ -599,7 +603,7 @@ warning + skip, never a system-clobbering write or an abort.
| `-L`, `--copy-links` | Transform symlink to referent | ✅ Parity | Sender-side: every symlink is replaced by its referent's content. A referent that cannot be read, including a broken symlink, makes the run exit 23 (`RERR_PARTIAL`) like rsync while the rest of the tree still transfers, in both the sequential and `--threads` paths (differential test). The transferred tree matches rsync |
| `--copy-unsafe-links` | Transform unsafe symlinks | ✅ Parity | Sender-side: only symlinks whose target is unsafe (absolute or escaping via `..`, matching rsync's `unsafe_symlink()` semantics) are dereferenced into their referent; safe links stay symlinks. A broken unsafe referent makes the run exit 23 like rsync (differential test), while a safe broken symlink is not dereferenced and exits 0 |
| `--safe-links` | Ignore symlinks outside tree | ✅ Parity | Sender-side: a symlink whose target is unsafe is not transmitted at all (skipped), matching rsync's `--safe-links`. Because FastSync applies this while scanning the source, the receiver does not need to repeat it (`safe_links` config field) |
| `--munge-links` | Munge symlinks for safety | ✅ Parity | Sender rewrites each transmitted symlink target with rsync's `/rsyncd-munged/` prefix; the receiver strips the marker (only when the negotiated `munge_links` policy is on, so a source link that genuinely begins with the marker round-trips verbatim) and restores the exact real target. Unlike rsync, FastSync prefixes on the *sender* and un-munges on the receiver, but the wire result and the stored marker match rsync. See the Phase-4 symlink-trust notes |
| `--munge-links` | Munge symlinks for safety | ✅ Parity | The **receiver** munges: it prefixes each stored symlink target with rsync's `/rsyncd-munged/` marker (only when the negotiated `munge_links` policy is on, so a source link that genuinely begins with the marker round-trips verbatim). The **sender** un-munges a source target that already begins with the marker before transmitting, so a munged tree round-trips through the receiver's re-munging exactly like rsync. Matching rsync, the prefix is applied on the receiver and stripped on the sender; the wire result and the stored marker match rsync. See the Phase-4 symlink-trust notes |
| `-k`, `--copy-dirlinks` | Transform symlink to dir | ✅ Parity | A symlink whose referent is a directory is dereferenced and recursed as a real directory; a symlink to a regular file stays a symlink. Sender-side only. See the Phase-4 symlink-trust notes |
| `-K`, `--keep-dirlinks` | Treat symlinked dir as dir | ✅ Parity | On the receiver, an existing destination symlink-to-a-directory is used as that directory (followed) instead of being replaced; it is followed only when it resolves to a directory that stays beneath the receive root. See the Phase-4 symlink-trust notes |
@@ -649,14 +653,15 @@ was bumped **2.12.0 → 2.13.0** (peers must match, exactly as prior phases did)
divergence for `--delete` over an existing symlinked dir). Without `-K` the
destination symlink is not followed (the O_NOFOLLOW walk fails the write),
which is the safe default.
- **`--munge-links`** (sender rewrite; crosses the wire so the receiver
unmunges): every transmitted symlink target is prefixed with rsync's marker
`SYMLINK_MUNGE_PREFIX` = `/rsyncd-munged/`; the receiver strips the marker
(only when the negotiated `munge_links` policy is on — a plain `-l` run never
strips the prefix, so a source symlink that genuinely begins with
`/rsyncd-munged/` round-trips verbatim) and restores the exact real target.
This matches rsync's stored marker and its both-ends-negotiated model, with the
prefix applied on the sender rather than the receiver. The link *value* is
- **`--munge-links`** (receiver rewrite; crosses the wire so the receiver
munges): every stored symlink target is prefixed with rsync's marker
`SYMLINK_MUNGE_PREFIX` = `/rsyncd-munged/` by the **receiver**; the sender
un-munges a source target that already begins with the marker before
transmitting, so a munged tree round-trips verbatim. The marker is applied only
when the negotiated `munge_links` policy is on — a plain `-l` run never
prefixes, so a source symlink that genuinely begins with
`/rsyncd-munged/` round-trips verbatim. This matches rsync's stored marker and
its both-ends-negotiated model, with the prefix applied on the receiver. The link *value* is
otherwise stored verbatim; the *placement* path still goes through
`file_symlink_at_secure`'s confined fd walk (`has_path_traversal` on the
destination path, no symlink follow). When no symlink is being transmitted
@@ -843,7 +848,7 @@ These features are moderate because they affect traversal, temporary files, mani
| `--relative`, `-R`; `--no-implied-dirs`; `--dirs`, `-d`; `--mkpath` | M | Extend path-list construction and destination directory creation while preserving traversal safety. |
| `--temp-dir`, `-T` | M | Separate temporary-file placement from FastSync's timeout alias and define collision, permissions, and cleanup rules. |
| `--delay-updates` | L | Stage all successful updates and publish them at completion, including crash and cancellation cleanup. |
| `--files-from=FILE`; `--from0`, `-0`; `--filter=RULE`, `-f`; `-F`; `--cvs-exclude`, `-C` | L | Build a complete filter/parser layer and integrate it with scanner pruning, manifests, and delete behavior. `-f` conflicts with FastSync sendfile mode. |
| `--files-from=FILE`; `--from0`, `-0`; `--filter=RULE`, `-f`; `-F`; `--cvs-exclude`, `-C` | L | Build a complete filter/parser layer and integrate it with scanner pruning, manifests, and delete behavior. (Done: `-f` is bound to `--filter`; the old sendfile conflict is gone, since sendfile is long-only `--sendfile`.) |
| `--list-only`; `--itemize-changes`, `-i`; `--out-format=FORMAT`; `--log-file-format=FMT` | M | Add a structured change-event model so output modes share one source of truth. |
### Phase 3: Deletion, Comparison, and Delta Compatibility
@@ -951,7 +956,7 @@ These are the last compatibility items and the closing phase toward rsync flag p
**Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity.
**Honest status after the parity 2.29 cycle (protocol 2.28.0, no wire change), updated by the parity cycle 2.29 pass and the audit-cycle follow-ups.** ✅ Parity 119 / ⚠️ Caveat 11 / ❌ Divergent 27 = 157 rows. The 2.29 cycle closed the scanner-order, delete-timing, relative-basis, and fuzzy-eligibility residuals (moving `-n`/`--delete`/`--del`/`--delete-delay` to ✅) and improved the `--info`/`--stats`/`--debug` partial rows; the remaining ⚠️ rows are `--info`, `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `--delete-before`, `--filter`, the three basis-dir options, and `-y/--fuzzy`. Earlier: **Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP
**Honest status after the parity 2.29 cycle (protocol 2.28.0, no wire change), updated by the parity cycle 2.29 pass, the audit-cycle follow-ups, and the triage cycle.** ✅ Parity 117 / ⚠️ Caveat 13 / ❌ Divergent 27 = 157 rows. The 2.29 cycle closed the scanner-order, delete-timing, relative-basis, and fuzzy-eligibility residuals (moving `-n`/`--delete`/`--del`/`--delete-delay` to ✅) and improved the `--info`/`--stats`/`--debug` partial rows; the triage cycle moved `-F` and `-i`/`--itemize-changes` ✅ → ⚠️ for their documented residuals. The remaining ⚠️ rows are `--info`, `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `-i`, `--delete-before`, `--filter`, `-F`, the three basis-dir options, and `-y/--fuzzy`. Earlier: **Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP
and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel /
receiver filter engine); the wire parity-track-4a pass later added that
receiver filter engine, flipping `--filter=RULE` back to ✅ (see above; the
@@ -990,13 +995,17 @@ integration tests unless it is explicitly listed as a limitation.
### Checksums and compression
- **`--checksum-choice`/`--cc`** accepts `xxh64` (default), `xxhash`, `xxh3`,
`xxh128`, `md5`, and `auto`; `md4`, `sha1`, `none`, and the two-name
`transfer,pre-transfer` form are **rejected by name**.
- **`--checksum-choice`/`--cc`** accepts the full rsync 3.4.1 set: `xxh64`
(default), `xxhash`, `xxh3`, `xxh128`, `md5`, `md4`, `sha1`, `none`, the
two-name `transfer,pre-transfer` form, and `auto` (which honors
`RSYNC_CHECKSUM_LIST` before the compiled-in order). A genuinely unknown name
is still rejected by name, matching rsync.
- **`--checksum-seed=0` is randomized per transfer** (the chosen seed is sent to
the receiver), matching rsync; an explicit non-zero seed is used verbatim.
- **`--compress-choice`/`--zc`** accepts `zstd` (default), `none`, and `auto`;
rsync's `lz4`/`zlib`/`zlibx` are **rejected by name**.
- **`--compress-choice`/`--zc`** accepts the full rsync 3.4.1 set: `zstd`
(default), `lz4`, `zlib`, `zlibx`, `none`, and `auto` (which honors
`RSYNC_COMPRESS_LIST` before the compiled-in order). A genuinely unknown name
is still rejected by name, matching rsync.
- **`--skip-compress`** uses rsync 3.4.1's built-in default suffix list when no
list is supplied; an explicit list replaces it.
- **`--no-whole-file`** is accepted as the rsync spelling that clears
@@ -1040,9 +1049,10 @@ integration tests unless it is explicitly listed as a limitation.
- **`--numeric-ids` is a mapping modifier only** — it changes *how* ids map, not
*whether* ownership is applied; combine it with `-o`/`-g`, `-a`, or an
explicit map.
- **`--usermap`/`--groupmap`** support names, `@N`/bare `N` ids, inclusive
`LOW-HIGH` ranges, `*`, empty-`FROM` (unnamed ids), and receiver-resolved `TO`
names.
- **`--usermap`/`--groupmap`** support names, FROM name **globs**
(`*`/`?`/`[...]`, expanded sender-side against the passwd/group database and
bounded by `MAX_IDENTITY_MAP`), `@N`/bare `N` ids, inclusive `LOW-HIGH` ranges,
`*`, empty-`FROM` (unnamed ids), and receiver-resolved `TO` names.
- **`--chown` conflicts with `--usermap`/`--groupmap` on the same side** and is a
clear configuration error (matching rsync) instead of an order-dependent
winner.
@@ -1056,19 +1066,23 @@ integration tests unless it is explicitly listed as a limitation.
### Symlinks and special files
- **`-l`/`--links` stores symlink targets verbatim** (absolute and `..`-bearing
targets included), matching rsync. `--safe-links`, `--copy-unsafe-links`, and
`--munge-links` (which now uses rsync's `/rsyncd-munged/` marker) match rsync
and are applied sender-side.
targets included), matching rsync. `--safe-links` and `--copy-unsafe-links`
match rsync and are applied sender-side; `--munge-links` (which now uses
rsync's `/rsyncd-munged/` marker) matches rsync too but is applied
**receiver-side** (the sender un-munges an already-marked source target).
- **`--specials` recreates unix sockets** with `mknodat(..., S_IFSOCK)`, so
`-D`/`--devices --specials` now covers the full rsync node set.
- **`--copy-devices`** is implemented (see its caveat below).
### Output
- **`-i`/`--out-format`** print rsync-style change lines; **`--list-only`**
- **`-i`/`--out-format`** print rsync-style change lines, including the
transfer-root `./` and per-directory `cd...`/`.d..t...` lines; **`--list-only`**
scans the source only and contacts no server; **`-h`** uses rsync's decimal
units; **`--progress`** is an aggregate line; **`--stats`** prints the counters
FastSync can observe locally (receiver-only counters are 0).
units; **`--progress`** prints rsync-style per-file progress blocks;
**`--stats`** prints the transfer-statistics block, whose receiver-only
counters (`Matched data`, `Number of deleted files`) are populated from the
receiver's `STATUS_STATS` report.
- **Server `--port`** is an alias of the `-p <port>` TCP listen port
(`--dparam port=` overrides the daemon config).
@@ -1090,8 +1104,10 @@ These remain after the wave; they are the reasons a row above is ⚠️.
- **New directories without `-p` still use FastSync's `0755` creation default**
rather than `source & ~umask`; directory metadata is only applied when a
directory attribute is requested.
- **`--stats` receiver-only counters** (matched data, file-list bytes, deleted
count) are reported as 0; `--progress` is an aggregate line, not per-file.
- **`--stats` byte totals** (`Total bytes sent`/`received`) are FastSync wire
bytes framed differently from rsync's, so they are not numerically comparable;
the remaining `--stats`/`--progress` divergences are the ones named in their
rows (per-type deleted-file breakdown, root-line/ancestor suppression).
- **`--password-file`/`--early-input`/`--hash-credentials`/`--iterations` are
FastSync-native** (SCRAM/PBKDF2), not rsync semantics; the batch format is not
rsync-interoperable. Credential files are opened with `O_NOFOLLOW` (a symlinked
@@ -1230,8 +1246,7 @@ These remain after the wave; the individual rows carry the precise wording.
name heuristic, but its candidate eligibility is bounded by the delta
engine (both files ≥ 16 KiB, size ratio ≤ 10×), a narrower window than
rsync's, so the selected basis — and the `--stats` bandwidth counters —
can differ while the tree stays byte-exact; and **`--bwlimit`** rejects rsync's
`0`/decimal/suffixed rates.
can differ while the tree stays byte-exact.
- **`--inc-recursive`/`--no-inc-recursive`** are not implemented (rejected).
### Intentional divergences (explicit ❌ rows)
+11 -4
View File
@@ -141,6 +141,10 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
update = 'h';
else if (created)
update = (event->is_directory || event->is_symlink || event->is_special) ? 'c' : '>';
else if (event->is_directory)
/* rsync: an existing directory that only has attribute changes carries no
transfer, so the update column is `.` rather than `>`. */
update = '.';
else
update = '>';
code[0] = update;
@@ -168,12 +172,15 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
code[11] = '\0';
}
/* rsync %n: the transfer-relative name, with a trailing slash for directories. */
/* rsync %n: the transfer-relative name, with a trailing slash for directories.
* The transfer root is `.` (so `%n` renders `./`), matching rsync's root entry. */
static bool append_name(StrBuf* buf, const ChangeEvent* event) {
if (!strbuf_append(buf, event->name != NULL ? event->name : ""))
const char* name = event->name != NULL ? event->name : "";
if (event->is_directory && name[0] == '\0')
return strbuf_append(buf, "./");
if (!strbuf_append(buf, name))
return false;
if (event->is_directory && (event->name == NULL || event->name[0] == '\0' ||
event->name[strlen(event->name) - 1] != '/'))
if (event->is_directory && name[strlen(name) - 1] != '/')
return strbuf_append_char(buf, '/');
return true;
}
+369 -20
View File
@@ -243,17 +243,31 @@ void transfer_stats_note_transferred(TransferStats* stats, const File* file) {
#define RSYNC_PROGRESS_IO_WINDOW (32ULL * 1024ULL)
/* Paths-only pre-count of the source file list, built once at transfer start
* when progress output is requested. rsync's `to-chk` denominator is the whole
* file list -- every regular file, directory, symlink and special plus the
* transfer root -- while the streaming scan never emits directories. A
* metadata-only walk (no file reads, no hashing) supplies that total and the
* directory names, so the opt-in pass leaves non-progress runs untouched. */
* when progress output or -i/--out-format needs it. rsync's `to-chk`
* denominator is the whole file list -- every regular file, directory, symlink
* and special plus the transfer root -- while the streaming scan only emits
* empty directories. A metadata-only walk (no file reads, no hashing) supplies
* that total and a metadata-bearing File for every directory, so --progress can
* name them and -i/--out-format can itemize them without a second full scan. */
/* One directory in the pre-count, keyed by its transfer-relative display name
* ("" is the transfer root). `file` is owned by ProgressPrecount.dir_files and
* carries the source metadata needed by -i/--out-format (%M/%B/%U/%G). */
typedef struct {
char* name; /* owned */
File* file;
} DirRef;
typedef struct {
unsigned long long total;
ArrayList* dir_paths; /* owned char* in transfer-relative display form */
ArrayList* dir_files; /* owned File* captured during the metadata walk */
ArrayList* dir_refs; /* owned DirRef*, sorted by name for prefix lookup */
} ProgressPrecount;
static bool g_progress_active;
/* True when -i/--out-format need the pre-counted directory entries fed into the
* change-event stream (independent of --progress). */
static bool g_change_dirs_active;
static unsigned long long g_progress_xferred;
static unsigned long long g_progress_index;
static unsigned long long g_progress_total;
@@ -270,14 +284,101 @@ bool progress_requested(const Config* config) {
(config->show_progress || (config->info_level & LOG_INFO_PROGRESS) != 0);
}
static void dir_ref_destroy(void* item) {
DirRef* ref = (DirRef*)item;
if (ref == NULL)
return;
free(ref->name);
free(ref);
}
/* Sort DirRef pointers by their transfer-relative name for binary search. */
static int dir_ref_compare(const void* left, const void* right) {
const DirRef* a = *(const DirRef* const*)left;
const DirRef* b = *(const DirRef* const*)right;
return strcmp(a->name, b->name);
}
/* Look up the pre-counted directory File for a transfer-relative name ("" is
* the transfer root). Returns NULL when no pre-count was built or the name is
* not a known directory. */
static File* progress_dir_lookup(const char* name) {
if (name == NULL || g_progress_precount.dir_refs == NULL)
return NULL;
ArrayList* refs = g_progress_precount.dir_refs;
size_t lo = 0;
size_t hi = (size_t)refs->size;
while (lo < hi) {
size_t mid = lo + (hi - lo) / 2;
DirRef* ref = (DirRef*)refs->items[mid];
int cmp = strcmp(ref->name, name);
if (cmp < 0)
lo = mid + 1;
else if (cmp > 0)
hi = mid;
else
return ref->file;
}
return NULL;
}
static void progress_precount_dispose(ProgressPrecount* p) {
if (p->dir_paths != NULL) {
array_list_delete(p->dir_paths);
p->dir_paths = NULL;
}
if (p->dir_files != NULL) {
array_list_delete(p->dir_files);
p->dir_files = NULL;
}
if (p->dir_refs != NULL) {
array_list_delete(p->dir_refs);
p->dir_refs = NULL;
}
p->total = 0;
}
/* Record the transfer root's pre-transfer state for -i/--out-format. The
* receive root always exists, so rsync never marks it `cd`; its only observable
* change is its timestamp, which FastSync cannot observe remotely. Force a time
* mismatch so the root renders rsync's `.d..t...... ./` rather than the `cd`
* a zeroed destination state would produce. */
static void progress_precount_mark_root(File* root) {
if (root == NULL)
return;
root->dest_state.known = true;
root->dest_state.existed = true;
root->dest_state.mode = root->metadata != NULL ? root->metadata->mode : 0;
root->dest_state.uid = root->metadata != NULL ? root->metadata->uid : 0;
root->dest_state.gid = root->metadata != NULL ? root->metadata->gid : 0;
root->dest_state.size = 0;
root->dest_state.mtime_sec = (root->metadata != NULL ? root->metadata->mtime_sec : 0) - 3600;
root->dest_state.mtime_nsec = root->metadata != NULL ? root->metadata->mtime_nsec : 0;
}
/* Append one DirRef (name -> file) to the pre-count, marking the transfer
* root's destination state. Returns false on allocation failure. */
static bool progress_precount_add_ref(ProgressPrecount* p, const Config* config, File* file) {
const char* rel = delete_display_path(config, file_wire_path(file));
char* name = rel != NULL ? str_dup(rel) : NULL;
if (name == NULL)
return false;
DirRef* ref = malloc(sizeof(*ref));
if (ref == NULL) {
free(name);
return false;
}
ref->name = name;
ref->file = file;
if (name[0] == '\0')
progress_precount_mark_root(file);
if (!array_list_add(p->dir_refs, ref)) {
dir_ref_destroy(ref);
return false;
}
return true;
}
void client_progress_cleanup(void) {
if (g_progress_dir_index_valid) {
path_index_free(&g_progress_dir_index);
@@ -293,6 +394,7 @@ void client_progress_cleanup(void) {
}
progress_precount_dispose(&g_progress_precount);
g_progress_active = false;
g_change_dirs_active = false;
g_progress_total = 0;
g_progress_index = 0;
g_progress_xferred = 0;
@@ -395,6 +497,11 @@ void print_delete_reports(const Config* config, const ArrayList* paths) {
fflush(stdout);
}
/* Emit every not-yet-seen ancestor directory of `rel`, outermost first, in the
* order rsync's depth-first flist walk visits them. With -i/--out-format each
* ancestor becomes a real change line (`cd+++++++++ sub/`, `.d..t...... ./`)
* rendered by the shared itemize code; otherwise it is the `--info=name` /
* --progress directory name line. */
static void client_progress_emit_ancestors(const Config* config, const char* rel) {
if (!g_progress_dir_index_valid || !g_progress_emitted_valid || g_progress_emitted_keys == NULL ||
rel == NULL)
@@ -413,9 +520,15 @@ static void client_progress_emit_ancestors(const Config* config, const char* rel
char* key = str_dup(prefix);
if (key != NULL && array_list_add(g_progress_emitted_keys, key)) {
str_hash_set_insert_ref(&g_progress_emitted, key);
if (g_change_dirs_active) {
const File* dir = progress_dir_lookup(prefix);
if (dir != NULL)
change_emit_dir_sent(config, dir);
} else {
char* escaped = output_escape(prefix, config->eight_bit_output);
printf("%s/\n", escaped ? escaped : prefix);
free(escaped);
}
g_progress_index++;
} else {
free(key);
@@ -425,6 +538,20 @@ static void client_progress_emit_ancestors(const Config* config, const char* rel
}
}
/* Feed a transferred entry's ancestor directories into the change-event stream
* before the entry's own line, so -i/--out-format and --progress report
* directories in rsync's depth-first order. Every directory is an ancestor of
* some emitted entry (a file, symlink, special, hard link or the empty-directory
* entry the scanner emits for a leaf), so this covers the whole tree. */
void client_change_emit_ancestors(const Config* config, const File* file) {
if (config == NULL || file == NULL)
return;
if (!g_progress_active && !g_change_dirs_active)
return;
const char* rel = delete_display_path(config, file_wire_path(file));
client_progress_emit_ancestors(config, rel);
}
/* rsync's --info=name/progress line for one entry: transfer-relative name (a
* trailing slash for directories) plus the ` -> target` symlink suffix. */
static char* progress_entry_line(const File* file, const char* rel) {
@@ -462,7 +589,7 @@ void client_progress_begin(const Config* config) {
g_progress_active = progress_requested(config);
g_progress_xferred = 0;
g_progress_index = 1; /* the transfer root is file-list entry #0 */
if (!g_progress_active) {
if (!g_progress_active && !g_change_dirs_active) {
/* `--info=flist` prints rsync's file-list header even without progress. */
if (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST)) {
printf("sending incremental file list\n");
@@ -470,11 +597,21 @@ void client_progress_begin(const Config* config) {
}
return;
}
/* -i/--out-format alone do not print the header, but --progress always does
and --info=flist does under any output mode (rsync prints it for
`-i --info=flist` and `--out-format=... --info=flist` too). */
if (g_progress_active || (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST)))
printf("sending incremental file list\n");
/* rsync prints the transfer-root directory's name before the first file when
that directory is created; FastSync mirrors the source root below the
receive root and creates it on a fresh destination, so emit it here. */
/* rsync prints the transfer-root directory before the first entry. Under
-i/--out-format it is the root change line (`.d..t...... ./`); otherwise it
is the plain --info=name / --progress name line. */
if (g_change_dirs_active) {
const File* root = progress_dir_lookup("");
if (root != NULL)
change_emit_dir_sent(config, root);
} else {
printf("./\n");
}
fflush(stdout);
}
@@ -487,7 +624,6 @@ void client_progress_file(const Config* config, const File* file) {
unsigned long long size = file->data->size;
if (!config->itemize_changes && config->out_format == NULL) {
const char* rel = delete_display_path(config, file_wire_path(file));
client_progress_emit_ancestors(config, rel);
char* escaped = output_escape(rel, config->eight_bit_output);
printf("%s\n", escaped ? escaped : (rel ? rel : ""));
free(escaped);
@@ -510,7 +646,6 @@ void client_progress_name(const Config* config, const File* file) {
return;
const char* rel = delete_display_path(config, file_wire_path(file));
if (!config->itemize_changes && config->out_format == NULL) {
client_progress_emit_ancestors(config, rel);
char* line = progress_entry_line(file, rel ? rel : "");
if (line != NULL) {
char* escaped = output_escape(line, config->eight_bit_output);
@@ -545,17 +680,152 @@ static bool progress_precount_add_dir(ProgressPrecount* p, const char* path) {
return false;
}
static int progress_path_compare(const void* left, const void* right) {
const char* const* a = (const char* const*)left;
const char* const* b = (const char* const*)right;
return strcmp(*a, *b);
}
/* Sort the collected directory paths and drop duplicates so a large
* --files-from list (many entries sharing an implied ancestor) cannot grow the
* list without bound. */
static void progress_precount_dedup_dirs(ArrayList* dir_paths) {
if (dir_paths == NULL || dir_paths->size < 2)
return;
qsort(dir_paths->items, (size_t)dir_paths->size, sizeof(char*), progress_path_compare);
int write = 0;
for (int read = 0; read < dir_paths->size; read++) {
char* current = (char*)dir_paths->items[read];
if (write > 0 && strcmp((char*)dir_paths->items[write - 1], current) == 0) {
free(current);
continue;
}
dir_paths->items[write++] = current;
}
dir_paths->size = write;
}
/* Create a metadata-bearing directory File for the transfer-relative directory
* `rel` ("" is the transfer root), stat'ing it below config->send_directory.
* The -d/--files-from dirs generator never traverses directories, so this
* synthesizes the metadata the recursive scanner captures through
* scanner_capture_dir_time, letting -i/--out-format render %M/%B/%U/%G and the
* transfer-root/ancestor lines identically on both paths. Returns NULL when
* the path cannot be stat'd as a directory or on allocation failure (the line
* is then simply omitted, exactly as before). */
static File* progress_precount_make_dir(const Config* config, const char* rel) {
if (config == NULL || config->send_directory == NULL)
return NULL;
char* fs_path = (rel == NULL || rel[0] == '\0') ? str_dup(config->send_directory)
: path_cat(config->send_directory, rel);
if (fs_path == NULL)
return NULL;
struct stat st;
if (stat(fs_path, &st) != 0 || !S_ISDIR(st.st_mode)) {
free(fs_path);
return NULL;
}
File* file = file_create(fs_path);
if (file == NULL) {
free(fs_path);
return NULL;
}
file->is_dir = true;
file->metadata =
file_metadata_create(fs_path, &st, config->preserve_atimes, config->preserve_crtimes);
file->send_path = str_dup(rel != NULL ? rel : "");
free(fs_path);
if (file->metadata == NULL || file->send_path == NULL) {
file_destroy(file);
return NULL;
}
return file;
}
/* The -d/--files-from dirs generator neither traverses nor records directories,
* so its metadata walk captures no Files. Synthesize the transfer root and
* every listed/implied directory from `entry_rels` so -i/--out-format emits the
* same root and ancestor lines the recursive scan does. `root_emitted` is true
* when the generator itself emits the root entry (bare `-d <dir>`), whose
* data-pass line must not be duplicated. Returns false only on allocation
* failure. */
static bool progress_precount_synthesize_dirs(const Config* config, ProgressPrecount* out,
const ArrayList* entry_rels, bool root_emitted) {
for (int i = 0; i < entry_rels->size; i++) {
const char* rel = (const char*)entry_rels->items[i];
if (rel == NULL)
continue;
size_t len = strlen(rel);
for (size_t j = 1; j < len; j++) {
if (rel[j] != '/')
continue;
/* --no-implied-dirs: rsync neither creates nor itemizes an implied parent,
so only explicitly listed directories get a line. */
if (config->no_implied_dirs)
break;
char* prefix = malloc(j + 1);
if (prefix == NULL)
return false;
memcpy(prefix, rel, j);
prefix[j] = '\0';
if (!progress_precount_add_dir(out, prefix)) {
free(prefix);
return false;
}
free(prefix);
}
}
progress_precount_dedup_dirs(out->dir_paths);
for (int i = 0; i < out->dir_paths->size; i++) {
const char* rel = (const char*)out->dir_paths->items[i];
File* dir = progress_precount_make_dir(config, rel);
if (dir == NULL)
continue;
if (!array_list_add(out->dir_files, dir)) {
file_destroy(dir);
return false;
}
}
/* Emit the transfer root only when the generator actually emitted an entry:
--prune-empty-dirs (or an empty --files-from list) transfers nothing, and
rsync prints no root line then either. */
if (!root_emitted && entry_rels->size > 0) {
File* root = progress_precount_make_dir(config, "");
if (root != NULL && !array_list_add(out->dir_files, root)) {
file_destroy(root);
return false;
}
}
return true;
}
/* Metadata-only walk collecting the full file-list total and every directory
* name. It uses its own scanner (fresh filter compilation and hard-link table)
* so the data pass's link-group state is never perturbed. */
static bool progress_precount_scan(const Config* config, ProgressPrecount* out) {
out->dir_paths = array_list_create(free);
if (out->dir_paths == NULL)
out->dir_files = array_list_create(file_destroy);
out->dir_refs = array_list_create(dir_ref_destroy);
if (out->dir_paths == NULL || out->dir_files == NULL || out->dir_refs == NULL) {
progress_precount_dispose(out);
return false;
}
out->total = 0;
/* The -d/--files-from dirs generator never calls scanner_capture_dir_time, so
the walk below captures no directory Files. Record every emitted entry's
transfer-relative name so the implied ancestors can be synthesized once the
walk is done. */
bool synthesize = g_change_dirs_active && config->dirs;
ArrayList* entry_rels = synthesize ? array_list_create(free) : NULL;
if (synthesize && entry_rels == NULL) {
progress_precount_dispose(out);
return false;
}
bool root_emitted = false;
PreparedScanner prepared;
memset(&prepared, 0, sizeof(prepared));
if (!prepare_scanner(config, 0, &prepared)) {
array_list_delete(entry_rels);
progress_precount_dispose(out);
return false;
}
@@ -568,12 +838,14 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
local.preserve_xattrs = false;
local.preserve_acls = false;
local.checksum = false;
local.capture_dir_times = false;
/* Capture one metadata-bearing File per traversed directory (including the
transfer root) so -i/--out-format can render %M/%B/%U/%G for directories. */
local.capture_dir_times = true;
local.excluded_paths = NULL;
local.size_skipped_paths = NULL;
local.synced_dirs = NULL;
local.plan_dirs = NULL;
local.dir_entries = NULL;
local.dir_entries = out->dir_files;
local.dir_entries_mutex = NULL;
local.hardlinks = NULL;
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
@@ -584,8 +856,23 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
out->total += (unsigned long long)chunk->element_count;
for (int i = 0; i < chunk->element_count && ok; i++) {
const File* f = chunk->items[i];
if (f != NULL && f->is_dir)
ok = progress_precount_add_dir(out, delete_display_path(config, file_wire_path(f)));
if (f == NULL)
continue;
const char* rel = delete_display_path(config, file_wire_path(f));
if (f->is_dir) {
if (rel != NULL && rel[0] == '\0')
root_emitted = true;
ok = progress_precount_add_dir(out, rel);
if (!ok)
break;
}
if (synthesize && rel != NULL) {
char* dup = str_dup(rel);
if (dup == NULL || !array_list_add(entry_rels, dup)) {
free(dup);
ok = false;
}
}
}
chunk_destroy(chunk);
}
@@ -595,9 +882,30 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
}
prepared_scanner_destroy(&prepared);
if (!ok) {
array_list_delete(entry_rels);
progress_precount_dispose(out);
return false;
}
if (synthesize) {
bool synth_ok = progress_precount_synthesize_dirs(config, out, entry_rels, root_emitted);
array_list_delete(entry_rels);
if (!synth_ok) {
progress_precount_dispose(out);
return false;
}
}
/* Build the name -> File lookup from the captured directory Files. */
for (int i = 0; i < out->dir_files->size; i++) {
File* f = (File*)out->dir_files->items[i];
if (f == NULL)
continue;
if (!progress_precount_add_ref(out, config, f)) {
progress_precount_dispose(out);
return false;
}
}
if (out->dir_refs->size > 1)
qsort(out->dir_refs->items, (size_t)out->dir_refs->size, sizeof(DirRef*), dir_ref_compare);
out->total += 1; /* the transfer root "." */
return true;
}
@@ -609,9 +917,26 @@ static bool progress_precount_from_plan_dirs(const Config* config, const ArrayLi
unsigned long long non_dir_count,
ProgressPrecount* out) {
out->dir_paths = array_list_create(free);
if (out->dir_paths == NULL)
out->dir_files = array_list_create(file_destroy);
out->dir_refs = array_list_create(dir_ref_destroy);
if (out->dir_paths == NULL || out->dir_files == NULL || out->dir_refs == NULL) {
progress_precount_dispose(out);
return false;
}
out->total = non_dir_count + 1;
/* The delete pre-scan's plan list omits the transfer root, so synthesize its
entry here; it is only used for the root change line. */
File* root = file_create("");
if (root == NULL || !array_list_add(out->dir_files, root)) {
file_destroy(root);
progress_precount_dispose(out);
return false;
}
root->is_dir = true;
if (!progress_precount_add_ref(out, config, root)) {
progress_precount_dispose(out);
return false;
}
for (int i = 0; i < plan_dirs->size; i++) {
const char* path = (const char*)plan_dirs->items[i];
const char* rel = config->send_directory != NULL
@@ -621,7 +946,25 @@ static bool progress_precount_from_plan_dirs(const Config* config, const ArrayLi
progress_precount_dispose(out);
return false;
}
File* dir = file_create("");
if (dir == NULL) {
progress_precount_dispose(out);
return false;
}
dir->is_dir = true;
dir->send_path = str_dup(rel != NULL ? rel : "");
if (dir->send_path == NULL || !array_list_add(out->dir_files, dir)) {
file_destroy(dir);
progress_precount_dispose(out);
return false;
}
if (!progress_precount_add_ref(out, config, dir)) {
progress_precount_dispose(out);
return false;
}
}
if (out->dir_refs->size > 1)
qsort(out->dir_refs->items, (size_t)out->dir_refs->size, sizeof(DirRef*), dir_ref_compare);
out->total += (unsigned long long)out->dir_paths->size;
return true;
}
@@ -633,10 +976,16 @@ void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
unsigned long long plan_non_dir_count) {
client_progress_cleanup();
g_progress_active = progress_requested(config);
if (!g_progress_active)
g_change_dirs_active = config->itemize_changes || config->out_format != NULL;
if (!g_progress_active && !g_change_dirs_active)
return;
bool ok = plan_dirs != NULL ? progress_precount_from_plan_dirs(
config, plan_dirs, plan_non_dir_count, &g_progress_precount)
/* -i/--out-format render directory metadata (%M/%B/%U/%G) that only the
metadata walk captures; the --delete-during/--delete-delay plan list has no
metadata, so prefer the walk whenever a change line is rendered. Pure
--progress keeps reusing the plan list and its cheaper path-only pass. */
bool ok = (plan_dirs != NULL && !g_change_dirs_active)
? progress_precount_from_plan_dirs(config, plan_dirs, plan_non_dir_count,
&g_progress_precount)
: progress_precount_scan(config, &g_progress_precount);
if (!ok) {
g_progress_total = 0;
+16 -3
View File
@@ -838,6 +838,10 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (chunk->items[i] == NULL)
continue;
transfer_stats_note_entry(stats, chunk->items[i]);
/* The chunk-serialization path emits no --progress name lines, so only
feed -i/--out-format its ancestor directory lines here. */
if (config->itemize_changes || config->out_format != NULL)
client_change_emit_ancestors(config, chunk->items[i]);
if (chunk->items[i]->is_dir)
change_emit_dir_sent(config, chunk->items[i]);
else
@@ -859,6 +863,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
source to remove and no incremental check. */
if (!send_directory_entry(client, f, config))
return -1;
client_change_emit_ancestors(config, f);
change_emit_dir_sent(config, f);
client_progress_name(config, f);
continue;
@@ -873,6 +878,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
!send_int(client->file_descriptor, f->link_group) ||
!send_wire_str(client->file_descriptor, f->hardlink_target))
return -1;
client_change_emit_ancestors(config, f);
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
@@ -881,6 +887,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (f->is_symlink) {
if (!send_symlink_entry(client, f, config))
return -1;
client_change_emit_ancestors(config, f);
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
@@ -890,6 +897,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (f->is_special) {
if (!file_send_special(f, client->file_descriptor, config->use_metadata))
return -1;
client_change_emit_ancestors(config, f);
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
@@ -913,6 +921,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
return -1;
}
transfer_stats_note_transferred(stats, f);
client_change_emit_ancestors(config, f);
change_emit_file_sent_bytes(config, f, protocol_bytes_written() - bytes_before,
protocol_bytes_read() - read_before);
client_progress_file(config, f);
@@ -1567,7 +1576,10 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
* runs the shared cleanup). */
static bool send_files_run(Config* config, SendFilesState* state) {
Client* client = state->client;
if (progress_requested(config))
/* --progress needs the file-list total; -i/--out-format needs the directory
entries. Either way one paths-only pre-count supplies both, and a
--delete-during/--delete-delay pre-scan is reused when present. */
if (progress_requested(config) || config->itemize_changes || config->out_format != NULL)
client_progress_prepare(config, state->plan_dirs, state->per_dir_non_dir_count);
/* Phase 6: compute the client-only stop deadline once at transfer start. The
early-delete pre-scan above deliberately ignores it so the keep-set (and
@@ -2045,9 +2057,10 @@ int send_files_multithreaded(Config* config) {
return 1;
}
/* --progress/--info=progress: pre-count the file list for rsync's to-chk
denominator, reusing a --delete-during/--delete-delay pre-scan when one
denominator; -i/--out-format: pre-count the directory entries. One pass
supplies both, reusing a --delete-during/--delete-delay pre-scan when one
already ran. */
if (progress_requested(config))
if (progress_requested(config) || config->itemize_changes || config->out_format != NULL)
client_progress_prepare(config, context->plan_dirs, pre_scan_non_dir);
thrd_t scanner, loader, sender;
+3
View File
@@ -64,6 +64,9 @@ void client_progress_cleanup(void);
void client_progress_begin(const Config* config);
void client_progress_file(const Config* config, const File* file);
void client_progress_name(const Config* config, const File* file);
/* Emit a transferred entry's ancestor directories (as -i/--out-format change
* lines or --progress name lines) before the entry's own line. */
void client_change_emit_ancestors(const Config* config, const File* file);
void client_progress_uptodate(const Config* config, const File* file);
void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
unsigned long long plan_non_dir_count);
+64 -6
View File
@@ -714,27 +714,85 @@ static FileSaveResult file_save_directory_to_disk(const FileSavePlan* plan, bool
return FILE_SAVE_ERROR;
bool dir_existed = file_path_exists_secure(dir_path);
bool ok = file_ensure_directory_secure(dir_path);
/* One confined, no-follow descriptor drives ownership/mode/xattr/timestamp
application so none of them can follow a same-named symlink planted after
the mkdir. This mirrors the O_DIRECTORY|O_NOFOLLOW fd that
dir_metadata_list_apply() opens for the recursive path; the fd is reached
through the already-confined parent. */
char* leaf = NULL;
int parent_fd = -1;
int dir_fd = -1;
if (ok) {
parent_fd = file_open_secure_parent(dir_path, &leaf, false);
if (parent_fd >= 0)
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
/* P7 Wave E: apply the negotiated ownership to the directory ITSELF (not
just the files inside it). --copy-as and every explicit identity policy
own every entry, so a directory must not keep the receiver's owner while
its children get the policy owner. Applied no-follow on the confined
parent fd after the mkdir; identity_apply_ownership_link() is itself a
no-op unless an identity policy is active. */
parent fd; identity_apply_ownership_link() is itself a no-op unless an
identity policy is active. Ownership runs before the mode because a chown
clears setuid/setgid. A failed REQUIRED --copy-as ownership fails the
entry; every other policy stays best-effort. */
if (ok && file->metadata && identity_active_enabled()) {
char* leaf = NULL;
int parent_fd = file_open_secure_parent(dir_path, &leaf, false);
if (parent_fd >= 0) {
if (!identity_apply_ownership_link(parent_fd, leaf, (int32_t)file->metadata->uid,
(int32_t)file->metadata->gid))
ok = false;
close(parent_fd);
} else if (identity_copy_as_active()) {
/* The directory exists (ok) but its required --copy-as ownership could
not be applied because the confined parent could not be opened. */
ok = false;
}
free(leaf);
} else if (ok && identity_copy_as_active()) {
ok = false;
}
/* The final source MODE is deliberately NOT applied inline. A restrictive
source mode (for example 0555) would make the directory unwritable before
its children are created, so a non-root receiver fails each child with
EACCES. The receiver feeds every is_dir entry -- including this explicit
--dirs/STATUS_MKDIR one -- into the deferred DirTimeList, and
dir_metadata_list_apply() stamps the exact mode once the whole transfer has
finished, exactly as it does for the recursive path. Leaving the directory
at its creation mode keeps it writable for the children until then.
The xattrs below are still applied inline so a direct
file_save_to_disk_full() caller (which has no deferred pass) also gets
--dirs directory xattrs. Because the inline mode is absent, the inline
order here is ownership, then xattrs, then timestamps; the recursive path
(which DOES apply a mode) orders them times, mode, xattrs -- the difference
is intentional, and the deferred pass re-stamps mode and xattrs last.
Best-effort: a per-attribute failure is logged and skipped by
xattr_apply_fd(), never fatal. */
if (ok && plan->config && plan->config->use_xattrs && dir_fd >= 0 && file->xattrs)
xattr_apply_fd(dir_fd, file->xattrs);
/* Timestamps last so no later inline ownership/xattr change is mistaken for a
content update; the deferred pass re-stamps them after every child write.
-J/--omit-dir-times suppresses the directory mtime; --atimes/-U applies
only when the source atime is valid, exactly as the recursive path. */
if (ok && file->metadata && plan->config && plan->config->preserve_times &&
!plan->config->omit_dir_times) {
struct timespec times[2] = {
{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
{.tv_sec = file->metadata->mtime_sec, .tv_nsec = file->metadata->mtime_nsec}};
if (plan->config->preserve_atimes && file->metadata->atime_valid) {
times[0].tv_sec = file->metadata->atime_sec;
times[0].tv_nsec = file->metadata->atime_nsec;
}
if (parent_fd >= 0 && utimensat(parent_fd, leaf, times, AT_SYMLINK_NOFOLLOW) != 0) {
int saved_errno = errno;
char* escaped_path = output_escape(dir_path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Failed to set directory timestamps on %s: %s",
escaped_path ? escaped_path : "<allocation failed>", strerror(saved_errno));
free(escaped_path);
}
}
if (dir_fd >= 0)
close(dir_fd);
if (parent_fd >= 0)
close(parent_fd);
free(leaf);
free(dir_path);
if (ok && created && !dir_existed)
*created = true;
+163 -11
View File
@@ -3,6 +3,7 @@
#include "utils.h"
#include <errno.h>
#include <fcntl.h>
#include <fnmatch.h>
#include <grp.h>
#include <limits.h>
#include <pwd.h>
@@ -12,6 +13,8 @@
#include <sys/stat.h>
#include <unistd.h>
static bool identity_id_fits_int32(unsigned long id);
/* The active identity snapshot lives in a per-process global. The TCP server
* forks one child process per connection, so a connection never shares this
* with another; within a connection the multithreaded receiver reads it without
@@ -419,17 +422,10 @@ static int identity_parse_from(const char* token, bool is_group, int32_t* out_fr
/* Not a numeric LOW-HIGH range: fall through and treat as a name (a
* hyphenated account name like "wayne-smith" must still resolve). */
}
/* A sender-side name. A wildcard other than the bare '*' is matched by rsync
* against the sender's names; because FastSync transmits numeric ids only, the
* receiver cannot evaluate it, so reject rather than silently mis-match. */
if (identity_token_has_glob(token)) {
log_message(LOG_LEVEL_ERROR,
"%smap FROM '%s': name wildcards other than '*' are not supported "
"(FastSync transmits numeric ids, so sender names are unavailable on the "
"receiver)",
is_group ? "--group" : "--user", token);
return -1;
}
/* A sender-side name. A FROM name wildcard other than the bare '*' is handled
* by identity_expand_from_glob() in the caller (it expands against the
* sender's account database at CLI-parse time), so this function only sees the
* bare '*' or a literal name here. */
int32_t id;
if (identity_resolve_token(token, is_group, &id) != 0)
return -1;
@@ -486,6 +482,138 @@ static int identity_append_rule(IdentityMap** map, int* count, const IdentityMap
return 0;
}
/* True when `lo` and `hi` are adjacent ids (no overflow at INT32_MAX). */
static bool identity_ids_adjacent(int32_t lo, int32_t hi) {
return lo < INT32_MAX && hi == lo + 1;
}
static int identity_id_cmp(const void* a, const void* b) {
int32_t x = *(const int32_t*)a;
int32_t y = *(const int32_t*)b;
return (x > y) - (x < y);
}
static bool identity_ids_push(int32_t** ids, size_t* count, size_t* cap, int32_t id) {
if (*count == *cap) {
size_t grown_cap = *cap ? *cap * 2 : 16;
int32_t* grown = realloc(*ids, grown_cap * sizeof(int32_t));
if (!grown)
return false;
*ids = grown;
*cap = grown_cap;
}
(*ids)[(*count)++] = id;
return true;
}
/* Expand a FROM name wildcard (rsync's match against sender-side account names)
* into one rule per contiguous run of matching numeric ids, all sharing the same
* TO side. FastSync transmits numeric ids only, so the wildcard must be
* resolved here -- at CLI-parse time -- against the SENDER's passwd/group
* database; the receiver has no sender names to match. Contiguous matched ids
* are collapsed into a single LOW-HIGH range (a range of adjacent ids contains
* exactly the ids it spans, so this is semantically exact). Returns 0 on
* success, -1 on an allocation failure, a wildcard that matches no sender
* account, or an expansion that would push the map past MAX_IDENTITY_MAP. */
static int identity_expand_from_glob(Config* config, const char* glob, bool is_group,
const IdentityMap* to_rule) {
const char* optname = is_group ? "--groupmap" : "--usermap";
size_t cap = 0;
size_t n = 0;
int32_t* ids = NULL;
bool alloc_failed = false;
if (is_group) {
setgrent();
struct group* gr;
while ((gr = getgrent()) != NULL) {
if (fnmatch(glob, gr->gr_name, 0) != 0)
continue;
if (!identity_id_fits_int32((unsigned long)gr->gr_gid))
continue;
if (!identity_ids_push(&ids, &n, &cap, (int32_t)gr->gr_gid)) {
alloc_failed = true;
break;
}
}
endgrent();
} else {
setpwent();
struct passwd* pw;
while ((pw = getpwent()) != NULL) {
if (fnmatch(glob, pw->pw_name, 0) != 0)
continue;
if (!identity_id_fits_int32((unsigned long)pw->pw_uid))
continue;
if (!identity_ids_push(&ids, &n, &cap, (int32_t)pw->pw_uid)) {
alloc_failed = true;
break;
}
}
endpwent();
}
if (alloc_failed) {
free(ids);
log_message(LOG_LEVEL_ERROR, "%s: memory allocation failed expanding FROM '%s'", optname, glob);
return -1;
}
if (n == 0) {
free(ids);
log_message(LOG_LEVEL_ERROR, "%s FROM '%s': no source account name matches the wildcard",
optname, glob);
return -1;
}
qsort(ids, n, sizeof(int32_t), identity_id_cmp);
size_t unique = 0;
for (size_t i = 0; i < n; i++) {
if (unique == 0 || ids[unique - 1] != ids[i])
ids[unique++] = ids[i];
}
n = unique;
int runs = 0;
for (size_t i = 0; i < n; i++) {
if (i == 0 || !identity_ids_adjacent(ids[i - 1], ids[i]))
runs++;
}
IdentityMap** map = is_group ? &config->groupmap : &config->usermap;
int* count = is_group ? &config->groupmap_count : &config->usermap_count;
if (*count > MAX_IDENTITY_MAP - runs) {
log_message(LOG_LEVEL_ERROR,
"%s FROM '%s': the name wildcard expands to %d rule(s), which would exceed "
"the maximum of %d map rules",
optname, glob, runs, MAX_IDENTITY_MAP);
free(ids);
return -1;
}
for (size_t i = 0; i < n;) {
size_t j = i;
while (j + 1 < n && identity_ids_adjacent(ids[j], ids[j + 1]))
j++;
IdentityMap rule;
rule.from = ids[i];
rule.from_hi = ids[j];
rule.to = to_rule->to;
rule.to_name = to_rule->to_name ? str_dup(to_rule->to_name) : NULL;
if (to_rule->to_name && !rule.to_name) {
free(ids);
return -1;
}
if (identity_append_rule(map, count, &rule) != 0) {
free(rule.to_name);
free(ids);
return -1;
}
i = j + 1;
}
free(ids);
return 0;
}
int identity_parse_map(Config* config, const char* value, bool is_group) {
if (!config || !value || *value == '\0') {
log_message(LOG_LEVEL_ERROR, "%smap requires a value", is_group ? "--group" : "--user");
@@ -509,6 +637,30 @@ int identity_parse_map(Config* config, const char* value, bool is_group) {
char* to_token = colon + 1;
IdentityMap parsed;
memset(&parsed, 0, sizeof(parsed));
/* A FROM name wildcard (anything with a glob metacharacter other than the
* bare '*') is expanded against the sender's account database here, while
* the sender's passwd/group DB is still available; the resulting numeric
* rules travel on the wire like an explicit list. The TO side is parsed
* first so every expanded rule shares it. */
if (strcmp(from_token, "*") != 0 && identity_token_has_glob(from_token)) {
if (identity_parse_to(to_token, is_group, &parsed.to, &parsed.to_name) != 0) {
log_message(LOG_LEVEL_ERROR, "%s could not parse TO '%s' in '%s'", optname, to_token,
value);
free(list);
return -1;
}
if (identity_expand_from_glob(config, from_token, is_group, &parsed) != 0) {
free(parsed.to_name);
free(list);
return -1;
}
/* Every rule emitted by the expansion took its own str_dup of the name,
* so the parse-time copy is unreachable on success: release it here (the
* failure path above already does). `parsed.to_name` is NULL for a
* numeric TO. */
free(parsed.to_name);
continue;
}
if (identity_parse_from(from_token, is_group, &parsed.from, &parsed.from_hi) != 0) {
log_message(LOG_LEVEL_ERROR,
"%s could not resolve FROM '%s' in '%s' (a name must exist on the "
+8 -2
View File
@@ -25,8 +25,14 @@
/* Parse one --usermap= / --groupmap= value (comma-separated FROM:TO rules,
* first match wins) into config->usermap / config->groupmap. is_group selects
* the group tables and name databases. Returns 0 on success, -1 on a
* malformed spec or an unresolvable name (never a silent no-op). */
* the group tables and name databases. A FROM name wildcard (containing `*`,
* `?` or `[...]`, but not the bare `*`) is expanded against the SENDER's
* account database at parse time into one or more numeric id/range rules
* (contiguous ids collapse to a range) sharing the same TO, because only
* numeric ids cross the wire; the expansion is capped at MAX_IDENTITY_MAP and a
* wildcard matching no account is an error. Returns 0 on success, -1 on a
* malformed spec, an unresolvable name, an unmatched wildcard, or a map that
* would exceed MAX_IDENTITY_MAP (never a silent no-op). */
int identity_parse_map(Config* config, const char* value, bool is_group);
/* Parse --chown=USER:GROUP. Supports USER:GROUP, USER (owner only), :GROUP
+31 -1
View File
@@ -2786,7 +2786,12 @@ class TestItemizeChanges:
flags=["--preserve", "-i", "--incremental"],
port=shared_server.port)
assert result.returncode == 0, f"incremental itemize failed: {result.stderr[:200]}"
itemized = [line for line in result.stdout.splitlines() if line and line[0] in ">.<c"]
# -i also emits the transfer-root and directory lines; only FILE entries
# matter here, so drop any line whose name has a trailing '/'.
itemized = [
line for line in result.stdout.splitlines()
if line and line[0] in ">.<c" and not line.rsplit(" ", 1)[-1].endswith("/")
]
assert itemized == [], f"unchanged files were itemized: {itemized[:5]}"
def test_multithreaded_emits_same_itemize_lines(self, shared_server):
@@ -6750,6 +6755,31 @@ class TestExtendedAttributes:
assert os.getxattr(received, "user.rootdir") == b"r"
assert os.getxattr(os.path.join(received, "sub"), "user.subdir") == b"s"
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_dirs_directory_xattr_applied(self, shared_server, mt):
"""#286.3: -d/-X must apply a transferred directory's user.* xattr at the
destination through the --dirs STATUS_MKDIR path (both the
single-threaded and -m/--threads receiver paths)."""
source, dest = self._source_and_dest("dirsxattr")
sub = os.path.join(source, "sub")
os.makedirs(sub)
if not _xattr_supported(sub):
pytest.skip("filesystem does not support user xattrs")
os.setxattr(sub, "user.dirsdir", b"dirs-value")
lst = os.path.join(TEST_DATA_DIR, "dirs_xattr_list.txt")
with open(lst, "wb") as fh:
fh.write(b"sub\n")
flags = ["--files-from", lst, "--dirs", "-R", "-X"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, \
f"--dirs -X sync failed: {(result.stderr or result.stdout)[:300]}"
received = os.path.join(dest, "sub")
assert os.path.isdir(received), "--dirs directory entry was not created"
assert os.getxattr(received, "user.dirsdir") == b"dirs-value", \
"the --dirs directory's user.* xattr was not applied at the destination"
@pytest.mark.ci
def test_directory_default_acl_preserved(self, shared_server):
"""#286.3: -aA must preserve a directory's default POSIX ACL (the
+173 -5
View File
@@ -26,6 +26,17 @@ def _rsync(args):
)
def _file_entry_line(text):
"""The file entry line for a single-file transfer.
-i/--out-format emit the transfer-root (and directory) lines too, so the
file entry is not necessarily the first line; for the one-file corpora used
by the wire-counter tests it is the last non-empty line.
"""
lines = [line for line in text.splitlines() if line.strip()]
return lines[-1] if lines else ""
def _make_selection_tree(root):
clean_dir(root)
os.makedirs(os.path.join(root, "sub"))
@@ -184,6 +195,120 @@ class TestItemizeParity:
)
assert fast_lines == rsync_lines, f"rsync={rsync_lines} fastsync={fast_lines}"
@requires_rsync
@pytest.mark.ci
def test_itemize_directory_lines_match_rsync(self, shared_server):
"""#292: -i/--out-format emit rsync's directory lines (including the
transfer root) in rsync's depth-first order."""
source = os.path.join(TEST_DATA_DIR, "out_itemdir_src")
dest = os.path.join(TEST_DATA_DIR, "out_itemdir_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_itemdir_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub", "deep"))
os.makedirs(os.path.join(source, "emptydir"))
with open(os.path.join(source, "a.txt"), "wb") as fh:
fh.write(b"hello\n")
with open(os.path.join(source, "sub", "b.txt"), "wb") as fh:
fh.write(b"world\n")
with open(os.path.join(source, "sub", "deep", "d.txt"), "wb") as fh:
fh.write(b"deep\n")
clean_dir(dest)
clean_dir(rdst)
def dir_lines(text):
# Any line whose name ends with '/' is a directory entry.
return sorted(
line for line in text.splitlines()
if line.rsplit(" ", 1)[-1].endswith("/")
)
for fmt in (None, "%i %n%L"):
rsync_flags = ["-a", "-i"] if fmt is None else ["-a", "--out-format=" + fmt]
fast_flags = rsync_flags
clean_dir(rdst)
clean_dir(dest)
rsync_result = _rsync(rsync_flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=fast_flags,
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
expected = [l for l in dir_lines(rsync_result.stdout)
if not l.rsplit(" ", 1)[-1] == "./"]
fast = dir_lines(result.stdout)
assert [l for l in fast if not l.rsplit(" ", 1)[-1] == "./"] == expected, (
f"fmt={fmt} rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
assert ".d..t...... ./" in fast, f"missing root line: {result.stdout!r}"
@requires_rsync
@pytest.mark.ci
def test_itemize_info_flist_header_matches_rsync(self, shared_server):
"""`-i --info=flist` prints rsync's file-list header: the -i change
lines alone do not enable the flist category, but an explicit --info=flist
must not be suppressed when itemizing."""
source = os.path.join(TEST_DATA_DIR, "out_itemfl_src")
dest = os.path.join(TEST_DATA_DIR, "out_itemfl_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_itemfl_rdst")
_make_output_tree(source)
clean_dir(dest)
clean_dir(rdst)
flags = ["-a", "-i", "--info=flist"]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert "sending incremental file list" in rsync_result.stdout
assert "sending incremental file list" in result.stdout, result.stdout
# -i alone (no explicit --info=flist) must stay silent like rsync.
clean_dir(dest)
clean_dir(rdst)
rsync_plain = _rsync(["-a", "-i", source + "/", rdst + "/"])
plain, _ = run_client(source, dest, flags=["-a", "-i"],
port=shared_server.port)
assert "sending incremental file list" not in rsync_plain.stdout
assert "sending incremental file list" not in plain.stdout, plain.stdout
@requires_rsync
@pytest.mark.ci
def test_itemize_files_from_dirs_root_and_ancestors(self, shared_server):
"""The -d/--files-from dirs generator emits the transfer-root line and
rsync's implied ancestor directory lines. The generator traverses no
directories, so those must be synthesized from the listed entries."""
source = os.path.join(TEST_DATA_DIR, "out_itemff_src")
dest = os.path.join(TEST_DATA_DIR, "out_itemff_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_itemff_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub", "deep"))
with open(os.path.join(source, "sub", "deep", "d.txt"), "wb") as fh:
fh.write(b"deep\n")
clean_dir(dest)
clean_dir(rdst)
listing = os.path.join(TEST_DATA_DIR, "out_itemff.list")
with open(listing, "w") as fh:
fh.write("sub/deep/d.txt\n")
flags = ["-d", "-i", "--files-from=" + listing]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
def dir_lines(text):
return sorted(line for line in text.splitlines()
if line.rsplit(" ", 1)[-1].endswith("/"))
# rsync emits the implied parents (sub/, sub/deep/) but never the root
# here; FastSync emits the same set plus its unconditional root line.
expected = [line for line in dir_lines(rsync_result.stdout)
if not line.rsplit(" ", 1)[-1] == "./"]
fast = dir_lines(result.stdout)
assert [line for line in fast if not line.rsplit(" ", 1)[-1] == "./"] == expected, (
f"rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
assert "cd+++++++++ sub/" in fast, result.stdout
assert "cd+++++++++ sub/deep/" in fast, result.stdout
assert any(line.rsplit(" ", 1)[-1] == "./" for line in fast), result.stdout
@requires_rsync
@pytest.mark.ci
def test_itemize_modified_file_matches_rsync(self, shared_server):
@@ -262,6 +387,47 @@ class TestOutFormatParity:
if line:
assert pattern.match(line), f"bad %M format: {line!r}"
@requires_rsync
@pytest.mark.ci
def test_out_format_directory_metadata_with_delete_during(self):
"""--delete-during/--delete-delay reuse the per-directory plan pre-scan,
whose list carries no metadata. Directory %M/%B/%U/%G must still come
from the source, exactly as the plain recursive scan renders them."""
source = os.path.join(TEST_DATA_DIR, "out_fmtmeta_src")
dest = os.path.join(TEST_DATA_DIR, "out_fmtmeta_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_fmtmeta_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub", "deep"))
with open(os.path.join(source, "a.txt"), "wb") as fh:
fh.write(b"hello\n")
with open(os.path.join(source, "sub", "b.txt"), "wb") as fh:
fh.write(b"world\n")
clean_dir(dest)
clean_dir(rdst)
def dir_lines(text):
# Directory names are the last whitespace-separated token.
return sorted(line for line in text.splitlines()
if line.rsplit(" ", 1)[-1].endswith("/")
and line.rsplit(" ", 1)[-1] != "./")
for timing in ("--delete-during", "--delete-delay"):
for fmt in ("%M %n", "%B %n", "%U %G %n"):
clean_dir(dest)
clean_dir(rdst)
flags = ["-a", "--out-format=" + fmt, timing]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, result.stderr[:300]
assert dir_lines(result.stdout) == dir_lines(rsync_result.stdout), (
f"{timing} {fmt}: rsync={rsync_result.stdout!r} "
f"fastsync={result.stdout!r}"
)
assert "1970/" not in result.stdout, result.stdout
class TestListOnlyParity:
@requires_rsync
@@ -387,8 +553,8 @@ class TestWireStatsParity:
result, _ = run_client(source, dest, flags=["-a", "--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
rb, rl = (int(x) for x in rsync_result.stdout.split()[:2])
fb, fl = (int(x) for x in result.stdout.split()[:2])
rb, rl = (int(x) for x in _file_entry_line(rsync_result.stdout).split()[:2])
fb, fl = (int(x) for x in _file_entry_line(result.stdout).split()[:2])
assert rl == fl == 5000, (rsync_result.stdout, result.stdout)
assert rb > rl, f"rsync %b must include framing: {rsync_result.stdout!r}"
assert fb > fl, f"fastsync %b must include framing: {result.stdout!r}"
@@ -421,7 +587,9 @@ class TestWireStatsParity:
assert file_lines(result.stdout) == file_lines(rsync_result.stdout), (
f"rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
assert result.stdout.split()[0] == rsync_result.stdout.split()[0] == "16", (
fs_c = _file_entry_line(result.stdout).split()[0]
rs_c = _file_entry_line(rsync_result.stdout).split()[0]
assert fs_c == rs_c == "16", (
f"%c must be rsync's 16-byte sum header: {result.stdout!r}"
)
@@ -450,8 +618,8 @@ class TestWireStatsParity:
"--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
rs_c = int(rsync_result.stdout.split()[0])
fs_c = int(result.stdout.split()[0])
rs_c = int(_file_entry_line(rsync_result.stdout).split()[0])
fs_c = int(_file_entry_line(result.stdout).split()[0])
# No basis exists, so rsync still reports only its sum header.
assert rs_c == 16, rsync_result.stdout
# FastSync reports its own handshake bytes and is not aligned.
+246 -1
View File
@@ -10,6 +10,8 @@
#include "protocol.h"
#include "test_utils.h"
#include "utils.h"
#include <fnmatch.h>
#include <grp.h>
#include <pwd.h>
#include <stddef.h>
#include <stdint.h>
@@ -3537,6 +3539,243 @@ static void test_parse_args_usermap_rsync_forms() {
config_delete(cfg);
}
/* Independent oracle for the FROM name-glob tests: enumerate the sender's
* account database and fill `ids` with the DISTINCT ids whose name matches
* `glob`, sorted ascending. Returns the count, or -1 if the matching set
* exceeds `max` distinct ids -- production has no such bound on the number of
* candidates it scans, so a truncated set would under-count runs and flake on
* hosts with very large account databases. Callers must skip (not fail) on
* -1. */
static int cli_collect_glob_ids(const char* glob, bool is_group, int32_t* ids, int max) {
int n = 0;
bool overflow = false;
if (is_group) {
setgrent();
struct group* gr;
while ((gr = getgrent()) != NULL) {
if (fnmatch(glob, gr->gr_name, 0) != 0)
continue;
if ((unsigned long)gr->gr_gid > (unsigned long)INT32_MAX)
continue;
int32_t id = (int32_t)gr->gr_gid;
bool dup = false;
for (int i = 0; i < n; i++)
if (ids[i] == id)
dup = true;
if (dup)
continue;
if (n >= max) {
overflow = true;
break;
}
ids[n++] = id;
}
endgrent();
} else {
setpwent();
struct passwd* pw;
while ((pw = getpwent()) != NULL) {
if (fnmatch(glob, pw->pw_name, 0) != 0)
continue;
if ((unsigned long)pw->pw_uid > (unsigned long)INT32_MAX)
continue;
int32_t id = (int32_t)pw->pw_uid;
bool dup = false;
for (int i = 0; i < n; i++)
if (ids[i] == id)
dup = true;
if (dup)
continue;
if (n >= max) {
overflow = true;
break;
}
ids[n++] = id;
}
endpwent();
}
for (int i = 1; i < n; i++) {
int32_t key = ids[i];
int j = i - 1;
while (j >= 0 && ids[j] > key) {
ids[j + 1] = ids[j];
j--;
}
ids[j + 1] = key;
}
return overflow ? -1 : n;
}
static int cli_count_runs(const int32_t* ids, int n) {
int runs = 0;
for (int i = 0; i < n; i++) {
if (i == 0 || ids[i - 1] == INT32_MAX || ids[i] != ids[i - 1] + 1)
runs++;
}
return runs;
}
/* #294: a FROM name wildcard must expand, at CLI-parse time, against the
* sender's account database into numeric id/range rules. Prefer a prefix that
* matches >=2 DISTINCT NON-contiguous ids (exercising multi-rule expansion); if
* no such prefix exists on this host, fall back to one whose ids are contiguous
* (exercising range collapse). The expected rules are derived independently by
* enumerating the same database. */
static void test_parse_args_identity_map_from_name_glob(bool is_group) {
int32_t ids[512];
int chosen_n = 0;
int chosen_runs = 0;
char chosen_c = 0;
for (char c = 'a'; c <= 'z'; c++) {
const char glob[3] = {c, '*', '\0'};
int n = cli_collect_glob_ids(glob, is_group, ids, (int)(sizeof(ids) / sizeof(ids[0])));
if (n < 2)
continue; /* no matches, or the set overflowed the oracle's buffer */
int runs = cli_count_runs(ids, n);
if (runs > MAX_IDENTITY_MAP)
continue; /* production would reject this expansion; try another prefix */
if (runs >= 2 || chosen_c == 0) {
chosen_c = c;
chosen_n = n;
chosen_runs = runs;
}
if (runs >= 2)
break;
}
if (chosen_c == 0)
return; /* no multi-match prefix on this host (skipped, not failed) */
const char glob[3] = {chosen_c, '*', '\0'};
chosen_n = cli_collect_glob_ids(glob, is_group, ids, (int)(sizeof(ids) / sizeof(ids[0])));
if (chosen_n < 2)
return; /* account DB changed under us: skip, don't flake */
chosen_runs = cli_count_runs(ids, chosen_n);
EXPECT_TRUE(chosen_n >= 2);
char map_value[16];
snprintf(map_value, sizeof(map_value), "%s:@0", glob);
Config* cfg = config_create();
char* argv[] = {"fastsync", is_group ? "--groupmap" : "--usermap", map_value, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
int got = is_group ? cfg->groupmap_count : cfg->usermap_count;
EXPECT_EQ_INT(got, chosen_runs);
const IdentityMap* map = is_group ? cfg->groupmap : cfg->usermap;
/* Every matched id is covered by some expanded rule. */
for (int i = 0; i < chosen_n; i++) {
bool covered = false;
for (int r = 0; r < got; r++)
if (ids[i] >= map[r].from && ids[i] <= map[r].from_hi)
covered = true;
EXPECT_TRUE(covered);
}
/* Every id inside every expanded range is one the glob actually matched, so
* the range collapse cannot over-match a name that does not fit the glob. */
for (int r = 0; r < got; r++) {
EXPECT_EQ_INT(map[r].to, 0);
for (int32_t v = map[r].from; v <= map[r].from_hi; v++) {
bool expected = false;
for (int i = 0; i < chosen_n; i++)
if (ids[i] == v)
expected = true;
EXPECT_TRUE(expected);
if (v == INT32_MAX)
break;
}
}
config_delete(cfg);
}
static void test_parse_args_usermap_from_name_glob() {
test_parse_args_identity_map_from_name_glob(false);
}
static void test_parse_args_groupmap_from_name_glob() {
test_parse_args_identity_map_from_name_glob(true);
}
/* Regression for a leak in the FROM name-glob success path: the TO side is
* parsed into `parsed.to_name` before the glob is expanded, and every emitted
* rule takes its own str_dup of that name -- so the parse-time copy must be
* released before the branch continues. A numeric TO has to_name == NULL and
* cannot expose the leak, hence this uses a NAME TO. The name is resolved on
* the receiver (not here), so any well-formed non-glob name works. Run this
* under ASan/valgrind to catch the leak. */
static void test_parse_args_identity_map_from_name_glob_name_to(bool is_group) {
int32_t ids[512];
char chosen_c = 0;
for (char c = 'a'; c <= 'z'; c++) {
const char glob[3] = {c, '*', '\0'};
int n = cli_collect_glob_ids(glob, is_group, ids, (int)(sizeof(ids) / sizeof(ids[0])));
if (n < 2)
continue;
if (cli_count_runs(ids, n) > MAX_IDENTITY_MAP)
continue;
chosen_c = c;
break;
}
if (chosen_c == 0)
return; /* no multi-match prefix on this host (skipped, not failed) */
const char glob[3] = {chosen_c, '*', '\0'};
char map_value[32];
snprintf(map_value, sizeof(map_value), "%s:nobody", glob);
Config* cfg = config_create();
char* argv[] = {"fastsync", is_group ? "--groupmap" : "--usermap", map_value, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
int got = is_group ? cfg->groupmap_count : cfg->usermap_count;
EXPECT_TRUE(got >= 1);
const IdentityMap* map = is_group ? cfg->groupmap : cfg->usermap;
for (int r = 0; r < got; r++) {
EXPECT_EQ_INT(map[r].to, 0);
EXPECT_NOT_NULL(map[r].to_name);
if (map[r].to_name)
EXPECT_EQ_STR(map[r].to_name, "nobody");
}
config_delete(cfg);
}
static void test_parse_args_usermap_from_name_glob_name_to() {
test_parse_args_identity_map_from_name_glob_name_to(false);
}
static void test_parse_args_groupmap_from_name_glob_name_to() {
test_parse_args_identity_map_from_name_glob_name_to(true);
}
/* #294: an expansion that would push the map past MAX_IDENTITY_MAP must fail
* with a clear error rather than silently truncating. Prefill the map to the
* cap and then add a wildcard guaranteed to match at least the current user. */
static void test_parse_args_identity_map_from_name_glob_over_cap() {
const struct passwd* self = getpwuid(geteuid());
if (!self || self->pw_name[0] == '\0')
return;
char glob[8];
snprintf(glob, sizeof(glob), "%c*", self->pw_name[0]);
size_t need = (size_t)MAX_IDENTITY_MAP * 6 + strlen(glob) + 4 + 1;
char* value = malloc(need);
if (!value)
return;
size_t off = 0;
for (int i = 0; i < MAX_IDENTITY_MAP; i++)
off += (size_t)snprintf(value + off, need - off, "@0:@0,");
snprintf(value + off, need - off, "%s:@0", glob);
Config* cfg = config_create();
char* argv[] = {"fastsync", "--usermap", value, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
config_delete(cfg);
free(value);
}
/* #294: rsync refuses to mix --chown with --usermap/--groupmap on the same
* side (either order). --chown=USER conflicts with a prior --usermap;
* --chown=:GROUP conflicts with a prior --groupmap; the opposite side is fine. */
@@ -3716,7 +3955,8 @@ static void test_parse_args_rejects_malformed_identity() {
{"--usermap", "definitely_not_a_real_user_zzz:@1"},
{"--usermap", "0-"},
{"--usermap", "5-2:@1"},
{"--usermap", "roo*:@1"},
{"--usermap", "zzz_definitely_no_such_user_glob_zzz*:@1"},
{"--groupmap", "zzz_definitely_no_such_group_glob_zzz*:@1"},
{"--groupmap", "@1"},
{"--groupmap", "no_such_group_qqq:x"},
{"--chown", "a:b:c"},
@@ -4882,6 +5122,11 @@ void test_client_cli() {
test_parse_args_groupmap();
test_parse_args_usermap_name_resolution();
test_parse_args_usermap_rsync_forms();
test_parse_args_usermap_from_name_glob();
test_parse_args_groupmap_from_name_glob();
test_parse_args_usermap_from_name_glob_name_to();
test_parse_args_groupmap_from_name_glob_name_to();
test_parse_args_identity_map_from_name_glob_over_cap();
test_parse_args_identity_map_chown_conflict();
test_parse_args_chown();
test_parse_args_copy_as();
+228 -1
View File
@@ -3,11 +3,13 @@
#include "test_utils.h"
#include "transport_tcp.h"
#include <arpa/inet.h>
#include <dirent.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <unistd.h>
/* -4/-6 map to a getaddrinfo ai_family hint: -4 -> AF_INET, -6 -> AF_INET6,
* and neither -> AF_UNSPEC. Both flags together are rejected earlier (in
@@ -246,6 +248,227 @@ static void test_tcp_nodelay_default_and_override() {
server_delete(&s);
}
/* Count the process's open descriptors via /proc/self/fd. The opendir
* descriptor is itself counted and closed before returning, so repeated calls
* are consistent and a before/after delta reflects only the code under test. */
static int count_open_fds(void) {
DIR* dir = opendir("/proc/self/fd");
if (!dir)
return -1;
int count = 0;
const struct dirent* ent;
while ((ent = readdir(dir)) != NULL) {
if (strcmp(ent->d_name, ".") == 0 || strcmp(ent->d_name, "..") == 0)
continue;
count++;
}
closedir(dir);
return count;
}
/* True when two sockaddrs name the same endpoint (family, address, and port).
* Comparing only the IP would let a connection to a different port on the same
* host pass, so the port is part of the identity. */
static bool sockaddr_same_endpoint(const struct sockaddr_storage* a,
const struct sockaddr_storage* b) {
if (a->ss_family != b->ss_family)
return false;
if (a->ss_family == AF_INET) {
const struct sockaddr_in* ia = (const struct sockaddr_in*)a;
const struct sockaddr_in* ib = (const struct sockaddr_in*)b;
return ia->sin_port == ib->sin_port && ia->sin_addr.s_addr == ib->sin_addr.s_addr;
}
if (a->ss_family == AF_INET6) {
const struct sockaddr_in6* ia = (const struct sockaddr_in6*)a;
const struct sockaddr_in6* ib = (const struct sockaddr_in6*)b;
return ia->sin6_port == ib->sin6_port &&
memcmp(&ia->sin6_addr, &ib->sin6_addr, sizeof(ia->sin6_addr)) == 0;
}
return false;
}
/* Bind + listen on the SECOND address getaddrinfo returns for "localhost", so
* the first candidate is connection-refused and the shared connect loop must
* fall back to a later one. On success the actual bound endpoint is written to
* out_bound/out_bound_len (the caller asserts the winning connect landed on it).
* Returns the listener fd and its port, or -1 when this host does not resolve
* localhost to at least two addresses (the test then skips rather than claiming
* coverage it does not have). */
static int bind_second_localhost_address(int* out_port, struct sockaddr_storage* out_bound,
socklen_t* out_bound_len) {
struct addrinfo hints;
memset(&hints, 0, sizeof(hints));
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
struct addrinfo* res = NULL;
if (getaddrinfo("localhost", "0", &hints, &res) != 0 || !res)
return -1;
const struct addrinfo* chosen = res->ai_next;
if (!chosen) {
freeaddrinfo(res);
return -1;
}
int fd = socket(chosen->ai_family, chosen->ai_socktype, chosen->ai_protocol);
if (fd < 0) {
freeaddrinfo(res);
return -1;
}
int opt = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
if (bind(fd, chosen->ai_addr, chosen->ai_addrlen) != 0 || listen(fd, 1) != 0) {
close(fd);
freeaddrinfo(res);
return -1;
}
struct sockaddr_storage bound;
socklen_t bound_len = sizeof(bound);
if (getsockname(fd, (struct sockaddr*)&bound, &bound_len) != 0) {
close(fd);
freeaddrinfo(res);
return -1;
}
if (out_bound)
*out_bound = bound;
if (out_bound_len)
*out_bound_len = bound_len;
if (bound.ss_family == AF_INET6)
*out_port = ntohs(((struct sockaddr_in6*)&bound)->sin6_port);
else
*out_port = ntohs(((struct sockaddr_in*)&bound)->sin_port);
freeaddrinfo(res);
return fd;
}
/* #219 AC3: when the first getaddrinfo candidate is refused, the connect loop
* must fall back to the next address and end with exactly ONE open descriptor
* (proving the failed attempt's fd was closed before the retry). */
static void test_tcp_connect_falls_back_to_next_address() {
int port = 0;
struct sockaddr_storage bound;
int listener = bind_second_localhost_address(&port, &bound, NULL);
if (listener < 0)
return; /* localhost is single-address on this host: cannot exercise fallback */
/* The /proc/self/fd delta is unreliable under valgrind (its own lazy fd
* activity perturbs the baseline), so only the functional assertions run
* there; the fd-count checks are skipped. */
bool check_fds = !is_running_under_valgrind();
int before = check_fds ? count_open_fds() : -1;
Client* c = client_create();
EXPECT_NOT_NULL(c);
EXPECT_TRUE(client_connect(c, "localhost", port));
EXPECT_TRUE(c->file_descriptor >= 0);
/* The winning candidate must be the endpoint we bound (the second
* getaddrinfo entry). Without this, a re-resolution that dropped the second
* address would make the test pass without ever exercising fallback. */
EXPECT_TRUE(sockaddr_same_endpoint(&c->address, &bound));
if (check_fds && before >= 0)
EXPECT_EQ_INT(count_open_fds(), before + 1);
client_disconnect(c);
if (check_fds && before >= 0)
EXPECT_EQ_INT(count_open_fds(), before);
client_delete(c);
close(listener);
}
/* #219 AC3: a connect that fails on every candidate leaves at most one
* descriptor (the last failed attempt) and none after client_disconnect. */
static void test_tcp_connect_failed_attempts_do_not_leak_fds() {
if (is_running_under_valgrind())
return; /* /proc/self/fd delta is perturbed by valgrind's own lazy fds */
/* Keep an ephemeral loopback port bound (but NOT listening) for the whole
* assertion: the port stays occupied by our own socket, so the kernel
* deterministically refuses a connect() to it. This closes the bind/close/
* connect TOCTOU window in which a parallel test could claim the port. */
int probe = socket(AF_INET, SOCK_STREAM, 0);
EXPECT_TRUE(probe >= 0);
struct sockaddr_in addr;
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
addr.sin_port = 0;
EXPECT_EQ_INT(bind(probe, (struct sockaddr*)&addr, sizeof(addr)), 0);
socklen_t addr_len = sizeof(addr);
EXPECT_EQ_INT(getsockname(probe, (struct sockaddr*)&addr, &addr_len), 0);
int port = ntohs(addr.sin_port);
int before = count_open_fds();
Client* c = client_create();
EXPECT_NOT_NULL(c);
/* The literal loopback address has a single getaddrinfo candidate -- the one
* our bound socket owns -- so the connect is deterministically refused. */
EXPECT_FALSE(client_connect(c, "127.0.0.1", port));
if (before >= 0)
EXPECT_TRUE(count_open_fds() <= before + 1);
client_disconnect(c);
if (before >= 0)
EXPECT_EQ_INT(count_open_fds(), before);
client_delete(c);
close(probe);
}
/* #219 AC3: the shared tcp_connect_socket_ex() (used by both the plain and TLS
* entry points) must install the --contimeout as SO_RCVTIMEO/SO_SNDTIMEO before
* connecting. Calling it directly lets us observe the pre-connect state (the
* plain wrapper later overrides the receive timeout with the IO --timeout). */
static void test_tcp_connect_socket_ex_applies_contimeout() {
Server* s = server_create(0);
EXPECT_NOT_NULL(s);
EXPECT_EQ_INT(listen(s->file_descriptor, 1), 0);
struct sockaddr_in bound;
socklen_t bound_len = sizeof(bound);
EXPECT_EQ_INT(getsockname(s->file_descriptor, (struct sockaddr*)&bound, &bound_len), 0);
int port = ntohs(bound.sin_port);
tcp_set_timeouts(30, 7);
Client* c = client_create();
EXPECT_NOT_NULL(c);
TcpConnectOptions opts;
memset(&opts, 0, sizeof(opts));
EXPECT_TRUE(tcp_connect_socket_ex(c, "127.0.0.1", port, &opts));
struct timeval tv;
socklen_t tv_len = sizeof(tv);
EXPECT_EQ_INT(getsockopt(c->file_descriptor, SOL_SOCKET, SO_RCVTIMEO, &tv, &tv_len), 0);
EXPECT_EQ_INT((int)tv.tv_sec, 7);
tv_len = sizeof(tv);
EXPECT_EQ_INT(getsockopt(c->file_descriptor, SOL_SOCKET, SO_SNDTIMEO, &tv, &tv_len), 0);
EXPECT_EQ_INT((int)tv.tv_sec, 7);
client_disconnect(c);
client_delete(c);
tcp_set_timeouts(30, 10);
server_delete(&s);
}
/* The plain wrapper applies the post-connect IO --timeout, which supersedes the
* contimeout installed during connect. */
static void test_tcp_connect_post_timeout_applied() {
Server* s = server_create(0);
EXPECT_NOT_NULL(s);
EXPECT_EQ_INT(listen(s->file_descriptor, 1), 0);
struct sockaddr_in bound;
socklen_t bound_len = sizeof(bound);
EXPECT_EQ_INT(getsockname(s->file_descriptor, (struct sockaddr*)&bound, &bound_len), 0);
int port = ntohs(bound.sin_port);
tcp_set_timeouts(5, 7);
Client* c = client_create();
EXPECT_NOT_NULL(c);
EXPECT_TRUE(client_connect(c, "127.0.0.1", port));
struct timeval tv;
socklen_t tv_len = sizeof(tv);
EXPECT_EQ_INT(getsockopt(c->file_descriptor, SOL_SOCKET, SO_RCVTIMEO, &tv, &tv_len), 0);
EXPECT_EQ_INT((int)tv.tv_sec, 5);
tv_len = sizeof(tv);
EXPECT_EQ_INT(getsockopt(c->file_descriptor, SOL_SOCKET, SO_SNDTIMEO, &tv, &tv_len), 0);
EXPECT_EQ_INT((int)tv.tv_sec, 5);
client_disconnect(c);
client_delete(c);
tcp_set_timeouts(30, 10);
server_delete(&s);
}
void test_transport_tcp() {
test_server_create_ephemeral();
test_server_delete_null();
@@ -263,4 +486,8 @@ void test_transport_tcp() {
test_server_create_bind_address();
test_server_create_bind_ipv6();
test_tcp_nodelay_default_and_override();
test_tcp_connect_falls_back_to_next_address();
test_tcp_connect_failed_attempts_do_not_leak_fds();
test_tcp_connect_socket_ex_applies_contimeout();
test_tcp_connect_post_timeout_applied();
}
+51
View File
@@ -3,8 +3,11 @@
#include "test_utils.h"
#include "transport_tcp.h"
#include "transport_tls.h"
#include <dirent.h>
#include <netinet/in.h>
#include <openssl/ssl.h>
#include <string.h>
#include <sys/socket.h>
#include <unistd.h>
static void test_tls_global_init() {
@@ -71,9 +74,57 @@ static void test_server_create_tls_empty_certs() {
EXPECT_NULL(s);
}
/* Count the process's open descriptors via /proc/self/fd (see the TCP tests). */
static int tls_count_open_fds(void) {
DIR* dir = opendir("/proc/self/fd");
if (!dir)
return -1;
int count = 0;
const struct dirent* ent;
while ((ent = readdir(dir)) != NULL) {
if (strcmp(ent->d_name, ".") == 0 || strcmp(ent->d_name, "..") == 0)
continue;
count++;
}
closedir(dir);
return count;
}
/* #219 AC3: client_connect_tls_ex() reuses the shared tcp_connect_socket_ex()
* for the TCP connect, and a later TLS-setup failure must release that
* descriptor. Passing no CA path makes create_ssl_ctx() fail deterministically
* AFTER a successful TCP connect, so the cleanup path is exercised without a
* TLS handshake or a certificate. (The multi-address fallback itself is covered
* by the shared tcp_connect_socket_ex() tests in test_transport_tcp.c, which the
* TLS entry point calls.) */
static void test_client_connect_tls_releases_fd_on_setup_failure() {
Server* s = server_create(0);
EXPECT_NOT_NULL(s);
EXPECT_EQ_INT(listen(s->file_descriptor, 1), 0);
struct sockaddr_in bound;
socklen_t bound_len = sizeof(bound);
EXPECT_EQ_INT(getsockname(s->file_descriptor, (struct sockaddr*)&bound, &bound_len), 0);
int port = ntohs(bound.sin_port);
/* The /proc/self/fd delta is unreliable under valgrind (its own lazy fd
* activity perturbs the baseline); keep the functional assertions and skip
* only the count checks there. */
bool check_fds = !is_running_under_valgrind();
int before = check_fds ? tls_count_open_fds() : -1;
Client* c = client_create();
EXPECT_NOT_NULL(c);
EXPECT_FALSE(client_connect_tls(c, "127.0.0.1", port, NULL, NULL, NULL));
EXPECT_TRUE(c->file_descriptor == -1);
if (check_fds && before >= 0)
EXPECT_EQ_INT(tls_count_open_fds(), before);
client_delete(c);
server_delete(&s);
}
void test_transport_tls() {
test_tls_global_init();
test_server_create_tls_without_certs();
test_client_connect_tls_fail();
test_client_connect_tls_releases_fd_on_setup_failure();
test_server_create_tls_empty_certs();
}
+50
View File
@@ -2,6 +2,7 @@
#include "xattr.h"
#include "config.h"
#include "file.h"
#include "file_save.h"
#include "identity.h"
#include "protocol.h"
#include "test_utils.h"
@@ -540,6 +541,54 @@ static void test_xattr_list_clone() {
xattr_list_free(clone);
}
/* #286.3: an explicit directory entry (--dirs, STATUS_MKDIR) that carries a
* captured user.* xattr must have it applied fd-relative by the directory
* install path itself -- not only by the receiver's deferred DirTimeList, which
* a direct file_save_to_disk_full() caller does not use. */
static void test_file_save_directory_applies_xattrs() {
const char* root = "test_save_dir_xattr_tmp";
const char* leaf = "subdir";
const char* path = "test_save_dir_xattr_tmp/subdir";
rmdir(path);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
/* The working directory may be a filesystem without user xattrs (e.g. some
tmpfs mounts): skip cleanly rather than fail the suite. */
if (setxattr(root, "user.fastsync-dirprobe", "p", 1, 0) != 0) {
rmdir(root);
return;
}
removexattr(root, "user.fastsync-dirprobe");
File* dir = file_create(leaf);
EXPECT_NOT_NULL(dir);
dir->is_dir = true;
FileXattrList* xattrs = xattr_list_new();
EXPECT_NOT_NULL(xattrs);
EXPECT_TRUE(xattr_list_append(xattrs, "user.dirxattr", "dirvalue", 8));
dir->xattrs = xattrs;
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->use_metadata = true;
config->use_xattrs = true;
config->preserve_xattrs = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, dir, config), FILE_SAVE_WRITTEN);
EXPECT_EQ_INT(access(path, F_OK), 0);
char value[32];
ssize_t got = getxattr(path, "user.dirxattr", value, sizeof(value));
EXPECT_EQ_INT((int)got, 8);
EXPECT_TRUE(got == 8 && memcmp(value, "dirvalue", 8) == 0);
file_destroy(dir);
config_delete(config);
removexattr(path, "user.dirxattr");
rmdir(path);
rmdir(root);
}
void test_xattr() {
test_xattr_list_clone();
test_xattr_wire_roundtrip();
@@ -553,4 +602,5 @@ void test_xattr() {
test_fake_super_restore();
test_fake_super_no_real_chown();
test_fake_super_storage_resolution();
test_file_save_directory_applies_xattrs();
}