Compare commits
59
Commits
d780a4625e
..
dev
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b7e95b8d96 | ||
|
|
13627e82fc | ||
|
|
09ab81120e | ||
|
|
2a7afbda0a | ||
|
|
bab6fcc706 | ||
|
|
6a426cffa8 | ||
|
|
b74182c7c1 | ||
|
|
f2a8eeaea7 | ||
|
|
18d7d495cf | ||
|
|
59d20e867a | ||
|
|
cc931790e9 | ||
|
|
60776b78fc | ||
|
|
52ad0aa512 | ||
|
|
759ffea117 | ||
|
|
ef37c2b988 | ||
|
|
a14fbb2c10 | ||
|
|
7408686370 | ||
|
|
90f2fb613f | ||
|
|
8843f1e3cf | ||
|
|
c387beb182 | ||
|
|
96e02f52c0 | ||
|
|
f7d5dda93b | ||
|
|
6d9cdb83ba | ||
|
|
7c748f1f7a | ||
|
|
884530c9a2 | ||
|
|
729f3ef8e1 | ||
|
|
b414f197af | ||
|
|
29f8be161c | ||
|
|
cb2979fdf1 | ||
|
|
00197102bf | ||
|
|
13b257d1dd | ||
|
|
f3d7672694 | ||
|
|
18b821d32c | ||
|
|
6ad065925f | ||
|
|
46dcefe218 | ||
|
|
b88acdbd3c | ||
|
|
c29bce54fc | ||
|
|
d98e971fcc | ||
|
|
492ce0ce89 | ||
|
|
ec6692ac41 | ||
|
|
1f8d60e30d | ||
|
|
6db9827b87 | ||
|
|
a07cc00bb0 | ||
|
|
3ae7685655 | ||
|
|
4f945a8e39 | ||
|
|
15f38f5b76 | ||
|
|
787967d3ce | ||
|
|
06e7aef5c9 | ||
|
|
ee265d78ba | ||
|
|
47b1b9b915 | ||
|
|
bb6c788cf9 | ||
|
|
0b40c47d6a | ||
|
|
6f81005094 | ||
|
|
193d358c64 | ||
|
|
8792e3265a | ||
|
|
3ec0ffb644 | ||
|
|
80e8d7f450 | ||
|
|
86741725fd | ||
|
|
f96f1764af |
No files matched your search
@@ -16,7 +16,7 @@ Ask the user or determine from context:
|
|||||||
- **Minor** (x.Y.0) — new features, backward compatible
|
- **Minor** (x.Y.0) — new features, backward compatible
|
||||||
- **Patch** (x.y.Z) — bug fixes, no protocol changes
|
- **Patch** (x.y.Z) — bug fixes, no protocol changes
|
||||||
|
|
||||||
Current version: `PROTOCOL_VERSION "2.26.0"` in `src/shared/config.h`
|
Current version: `PROTOCOL_VERSION "2.29.0"` in `src/shared/config.h`
|
||||||
|
|
||||||
### Step 2: Check Protocol Version
|
### Step 2: Check Protocol Version
|
||||||
|
|
||||||
|
|||||||
+75
-2
@@ -6,6 +6,38 @@ run the same version because the handshake is strict.
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
Wire backlog cycle (protocol 2.29.0 → 2.30.0; config-frame layout unchanged).
|
||||||
|
|
||||||
|
- **`--stderr=client` client-message channel (#313):** the client now accepts
|
||||||
|
`--stderr=client` (and maps the deprecated `--no-msgs2stderr` to it), routing
|
||||||
|
its own diagnostics over the new bounded `STATUS_CLIENT_MSG` client->server
|
||||||
|
frame instead of writing them locally; the server writes each received
|
||||||
|
message to its stderr (respecting the server log destination). `errors`/`all`
|
||||||
|
behavior is unchanged.
|
||||||
|
- **Receiver partial failures exit 23 (#320):** a per-entry receiver failure
|
||||||
|
that does not abort the stream (e.g. an unprivileged `--devices` mknod) now
|
||||||
|
sends the terminal `STATUS_PARTIAL`; the client exits 23 like rsync and, under
|
||||||
|
`--remove-source-files`, still removes the sources it successfully
|
||||||
|
transferred. A clean run stays 0 and a fatal/connection error stays non-23.
|
||||||
|
- **Directory/symlink destination-state itemize (#314):** when `report_dest_info`
|
||||||
|
is negotiated (now also for `--progress`), the receiver answers `STATUS_MKDIR`
|
||||||
|
and `STATUS_SYMLINK` with the entry's pre-transfer destination snapshot
|
||||||
|
(existence, type, perms/owner/group/time, and whether an existing symlink's
|
||||||
|
target already matches), and the sender probes every ancestor directory before
|
||||||
|
the receiver creates it implicitly. A re-run over an unchanged tree no longer
|
||||||
|
emits per-directory `cd+++++++++` or unchanged-symlink lines, a changed
|
||||||
|
directory renders rsync's `.d..t......`, and a changed symlink renders
|
||||||
|
`cLc........` / `.L..t......`. Directory/symlink time comparison uses whole
|
||||||
|
seconds (rsync's `cmp_time`). The `STATUS_MKDIR` body gains a probe flag and
|
||||||
|
the `STATUS_DEST_INFO` record gains a symlink-target-match field; the
|
||||||
|
config-frame layout is unchanged. Differential-tested against rsync 3.4.1.
|
||||||
|
- **`--stats` deleted per-type breakdown (#316):** `STATUS_STATS` gains
|
||||||
|
`deleted_reg/dir/link/special`, tallied by the delete observers and rendered
|
||||||
|
as rsync's `Number of deleted files: X (reg: A, dir: B, link: C, special: D)`.
|
||||||
|
Differential-tested against rsync 3.4.1 for a mixed-type `--delete` tree.
|
||||||
|
|
||||||
|
## [2.29.0] - 2026-09-23
|
||||||
|
|
||||||
The rsync-parity cycle 2.29 (no wire change; `PROTOCOL_VERSION` stays 2.28.0).
|
The rsync-parity cycle 2.29 (no wire change; `PROTOCOL_VERSION` stays 2.28.0).
|
||||||
`RSYNC_COMPAT.md` moves from **116 ✅ / 14 ⚠️ / 27 ❌** to
|
`RSYNC_COMPAT.md` moves from **116 ✅ / 14 ⚠️ / 27 ❌** to
|
||||||
**120 ✅ / 10 ⚠️ / 27 ❌** of 157 rows.
|
**120 ✅ / 10 ⚠️ / 27 ❌** of 157 rows.
|
||||||
@@ -25,6 +57,28 @@ remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ /
|
|||||||
fixes** below) moves `-F` and `-i` to ⚠️, for a final **117 ✅ / 13 ⚠️ / 27 ❌**
|
fixes** below) moves `-F` and `-i` to ⚠️, for a final **117 ✅ / 13 ⚠️ / 27 ❌**
|
||||||
of 157 rows.
|
of 157 rows.
|
||||||
|
|
||||||
|
A no-wire parity burn-down cycle follows on 2.28.0: it accepts
|
||||||
|
`--inc-recursive`/`--no-inc-recursive` as inert no-ops, accepts an absolute
|
||||||
|
`--temp-dir` that canonicalizes inside the receive root, closes the
|
||||||
|
`--delete-before` phase-0 divergence (both the single-threaded and `--threads`
|
||||||
|
data passes replay the pre-scan list), makes `--fake-super` interoperable with
|
||||||
|
rsync's `user.rsync.%stat` key/grammar (regular files and char/block devices
|
||||||
|
faked as regular files), turns a failed device `mknod` into a continuing
|
||||||
|
per-entry failure, and accepts a practical subset of rsync's `rsyncd.conf`
|
||||||
|
grammar (modules are read-only by default, and accepted-but-unenforced
|
||||||
|
access-control keys emit a startup warning). The matrix moves to **119 ✅ /
|
||||||
|
14 ⚠️ / 24 ❌** of 157 rows.
|
||||||
|
|
||||||
|
A structural cycle then lands a transport I/O vtable over TCP/TLS (fixing the
|
||||||
|
TLS-multithreaded sendfile path and making the per-thread SSL resolution
|
||||||
|
explicit) and bumps the wire to **2.29.0**: the `STATUS_SYMLINK` frame grows an
|
||||||
|
optional symlink-xattr block (captured no-follow with `llistxattr`/`lgetxattr`,
|
||||||
|
applied no-follow with `lsetxattr`). Because the handshake is strict, 2.28.0 and
|
||||||
|
2.29.0 peers are incompatible. Note: Linux refuses to associate xattrs with a
|
||||||
|
symlink at all, so the symlink-xattr block is a no-op on Linux and is carried
|
||||||
|
for correctness on platforms/filesystems that do support it; the config-frame
|
||||||
|
layout is unchanged (golden length still 886).
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
- **rsync-exact traversal order.** The sequential scanner now walks each
|
- **rsync-exact traversal order.** The sequential scanner now walks each
|
||||||
@@ -58,8 +112,9 @@ of 157 rows.
|
|||||||
wording, and symlink/empty-directory quick-checks.
|
wording, and symlink/empty-directory quick-checks.
|
||||||
- `--delete-before`'s phase-0 late-file divergence remains (rsync's pre-scan
|
- `--delete-before`'s phase-0 late-file divergence remains (rsync's pre-scan
|
||||||
fixes the file list before the data pass).
|
fixes the file list before the data pass).
|
||||||
- A single file larger than 256 MiB cannot be streamed in the default path
|
- A whole-file sender that cannot stream its codec (lz4's one-shot block
|
||||||
(a general whole-file limit, not basis-specific).
|
format) or a `--append`/delta source above the bound still buffers; the
|
||||||
|
default zstd/zlib and the uncompressed paths stream (see #318).
|
||||||
- `--stats` byte totals and `--msgs2stderr` stay documented divergences.
|
- `--stats` byte totals and `--msgs2stderr` stay documented divergences.
|
||||||
|
|
||||||
### Security
|
### Security
|
||||||
@@ -79,6 +134,24 @@ of 157 rows.
|
|||||||
- **Daemon umask no longer forced to `0`.** `daemonize()` now sets the
|
- **Daemon umask no longer forced to `0`.** `daemonize()` now sets the
|
||||||
conventional `022`, so implied parent directories created without `-p` are no
|
conventional `022`, so implied parent directories created without `-p` are no
|
||||||
longer world-writable `0777`.
|
longer world-writable `0777`.
|
||||||
|
- **Daemon modules are read-only by default.** A `--daemon` module is now
|
||||||
|
served read-only unless it sets `read only = no` (or rsync's `write only =
|
||||||
|
yes`), matching rsync: a real `rsyncd.conf` that omits `read only` is no
|
||||||
|
longer silently writable. A global `read only` still sets the default for
|
||||||
|
later modules, and an explicit module value wins. This is a behavior change
|
||||||
|
for existing FastSync-native configs that relied on the old writable default;
|
||||||
|
add `read only = no` to keep them writable. An rsync `write only = yes` is
|
||||||
|
mapped to writability (FastSync is push-only, so a module can never be read
|
||||||
|
from the network).
|
||||||
|
- **Accepted-but-unenforced rsync security keys now warn at startup.** The
|
||||||
|
rsync keys FastSync recognizes but does not implement — `secrets file`,
|
||||||
|
`refuse options`, `exclude`/`include`/`filter`, `max size`/`min size`,
|
||||||
|
`pre-xfer exec`/`post-xfer exec`, `incoming chmod`/`outgoing chmod`,
|
||||||
|
`name converter`, `use chroot`, `uid`/`gid`, and the rest of the
|
||||||
|
access-control set — load for migration compatibility but now emit a
|
||||||
|
`WARN` naming the key (and module) so an operator does not believe the
|
||||||
|
restriction is enforced. `auth users`/`secrets file` stay fail-closed: a
|
||||||
|
module declaring `auth users` still requires a FastSync credential store.
|
||||||
- **Credentials and signal handling hardened.** Secret files are opened with
|
- **Credentials and signal handling hardened.** Secret files are opened with
|
||||||
`O_NOFOLLOW|O_NONBLOCK` (while allowing fd-backed store paths and bound-waiting
|
`O_NOFOLLOW|O_NONBLOCK` (while allowing fd-backed store paths and bound-waiting
|
||||||
a FIFO read for ~3 s so a slow process substitution works but a connected-but-
|
a FIFO read for ~3 s so a slow process substitution works but a connected-but-
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
cmake_minimum_required(VERSION 3.22)
|
cmake_minimum_required(VERSION 3.22)
|
||||||
|
|
||||||
project(FastFileTransfer VERSION 2.28.0)
|
project(FastFileTransfer VERSION 2.30.0)
|
||||||
|
|
||||||
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
|
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
|
||||||
set(CMAKE_C_STANDARD 11)
|
set(CMAKE_C_STANDARD 11)
|
||||||
|
|||||||
+3
-1
@@ -235,7 +235,9 @@
|
|||||||
and symlink/empty-dir quick-check feedback.
|
and symlink/empty-dir quick-check feedback.
|
||||||
- **`--delete-before` phase-0 keep-set** (rsync fixes the file list before
|
- **`--delete-before` phase-0 keep-set** (rsync fixes the file list before
|
||||||
the data pass; FastSync keeps its pre-scan snapshot race).
|
the data pass; FastSync keeps its pre-scan snapshot race).
|
||||||
- **>256 MiB single-file streaming** (B4, the general whole-file limit).
|
- ~~**>256 MiB single-file streaming** (B4, the general whole-file limit).~~
|
||||||
|
Closed by #318: the whole-file payload, the basis read/verify and the fuzzy
|
||||||
|
basis are streamed through bounded buffers (lz4/append remain buffered).
|
||||||
- **Wire native-size framing:** lengths are native `size_t` and the protocol
|
- **Wire native-size framing:** lengths are native `size_t` and the protocol
|
||||||
assumes homogeneous word size/endianness — document or move to fixed-width
|
assumes homogeneous word size/endianness — document or move to fixed-width
|
||||||
framing.
|
framing.
|
||||||
|
|||||||
@@ -187,7 +187,7 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
|||||||
| `--groupmap=MAP` | Map group names when applying ownership |
|
| `--groupmap=MAP` | Map group names when applying ownership |
|
||||||
| `--numeric-ids` | Apply source numeric uid/gid directly instead of mapping by name |
|
| `--numeric-ids` | Apply source numeric uid/gid directly instead of mapping by name |
|
||||||
| `--copy-as=USER[:GROUP]` | Force every written entry to USER[:GROUP] (requires a privileged receiver) |
|
| `--copy-as=USER[:GROUP]` | Force every written entry to USER[:GROUP] (requires a privileged receiver) |
|
||||||
| `--fake-super` | Record the resolved owner plus mode/time in a reserved `user.fastsync.stat` xattr and replay mode/time; never performs a real chown |
|
| `--fake-super` | Record the resolved owner plus full mode/rdev in rsync's reserved `user.rsync.%stat` xattr (rsync 3.4.1 grammar) and replay the permission bits; never performs a real chown |
|
||||||
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes) |
|
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes) |
|
||||||
| `-D` | Preserve device and special files (implies `--devices --specials`) |
|
| `-D` | Preserve device and special files (implies `--devices --specials`) |
|
||||||
| `--devices` | Recreate device nodes on the destination (privileged; skipped without `CAP_MKNOD`) |
|
| `--devices` | Recreate device nodes on the destination (privileged; skipped without `CAP_MKNOD`) |
|
||||||
@@ -206,9 +206,9 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
|||||||
| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `--chunk-serialization`. |
|
| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `--chunk-serialization`. |
|
||||||
| `--existing` | Skip files not already present at the destination; update existing files normally. |
|
| `--existing` | Skip files not already present at the destination; update existing files normally. |
|
||||||
| `--ignore-existing` | Skip files that already exist on the receiver; like rsync it does not apply to directories or symlinks. |
|
| `--ignore-existing` | Skip files that already exist on the receiver; like rsync it does not apply to directories or symlinks. |
|
||||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`; a basis MISS above the 256 MiB whole-file payload bound is refused — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`; a basis of any size is supported, streamed in bounded chunks — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
||||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination (same basis-size caveat; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination (the copy is streamed, so a basis of any size works; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
||||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win; same basis-size caveat; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win; a basis of any size works; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
||||||
| `--verify-basis` | FastSync-only: require a basis hit (`--compare-dest`/`--copy-dest`/`--link-dest`) to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync) |
|
| `--verify-basis` | FastSync-only: require a basis hit (`--compare-dest`/`--copy-dest`/`--link-dest`) to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync) |
|
||||||
| `--delete` | Delete files on receiver not present in source (default timing: delete-during, matching rsync, so destination space is freed progressively). Scoped to the synchronized directories, so `--files-from` subsets are safe |
|
| `--delete` | Delete files on receiver not present in source (default timing: delete-during, matching rsync, so destination space is freed progressively). Scoped to the synchronized directories, so `--files-from` subsets are safe |
|
||||||
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
|
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
|
||||||
@@ -219,7 +219,7 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
|||||||
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
|
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
|
||||||
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
|
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
|
||||||
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication; the fixed `.fastsync-stage` staging name diverges from rsync — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication; the fixed `.fastsync-stage` staging name diverges from rsync — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
||||||
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install; confined to the receive root (relative only), with an `EXDEV` non-atomic copy fallback |
|
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install; confined to the receive root (a relative path resolves below it; an absolute path is accepted only when it canonicalizes inside it), with an `EXDEV` non-atomic copy fallback |
|
||||||
| `-n, --dry-run` | Report what would be transferred without mutating the destination. Since protocol 2.21.0 a server-routed target contacts the receiver and reports would-transfer based on receiver state; a plain local destination keeps the client-side scan. Never mutates or deletes. |
|
| `-n, --dry-run` | Report what would be transferred without mutating the destination. Since protocol 2.21.0 a server-routed target contacts the receiver and reports would-transfer based on receiver state; a plain local destination keeps the client-side scan. Never mutates or deletes. |
|
||||||
| `-v, --verbose` | Enable debug logging |
|
| `-v, --verbose` | Enable debug logging |
|
||||||
| `-q, --quiet` | Suppress non-error output |
|
| `-q, --quiet` | Suppress non-error output |
|
||||||
@@ -300,6 +300,7 @@ transfer is never aborted.
|
|||||||
| `FASTSYNC_SOURCE_DIR` | — | Source directory fallback |
|
| `FASTSYNC_SOURCE_DIR` | — | Source directory fallback |
|
||||||
| `FASTSYNC_DEST_DIR` | — | Destination directory fallback |
|
| `FASTSYNC_DEST_DIR` | — | Destination directory fallback |
|
||||||
| `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback |
|
| `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback |
|
||||||
|
| `FASTSYNC_MAX_WHOLE_FILE_SIZE` | `268435456` | Receiver-only test hook: a byte count that lowers the whole-file streaming bound. Payloads above it are streamed through a bounded buffer. Values are clamped to the 256 MiB protocol ceiling, so it can only lower, never raise, the bound. |
|
||||||
|
|
||||||
## Implementation Details
|
## Implementation Details
|
||||||
|
|
||||||
@@ -580,9 +581,9 @@ remote SSH argv is already built injection-safe.
|
|||||||
| `--files-from <file>` | Read the source file list from FILE (paths relative to the source root). |
|
| `--files-from <file>` | Read the source file list from FILE (paths relative to the source root). |
|
||||||
| `-0, --from0` | Treat entries in `--files-from` files as NUL-delimited instead of newline-delimited. |
|
| `-0, --from0` | Treat entries in `--files-from` files as NUL-delimited instead of newline-delimited. |
|
||||||
| `--delay-updates` | Put updated files into place only at the end of the transfer (the fixed `.fastsync-stage` staging name diverges from rsync; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
| `--delay-updates` | Put updated files into place only at the end of the transfer (the fixed `.fastsync-stage` staging name diverges from rsync; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
||||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`; a basis MISS above the 256 MiB whole-file payload bound is refused — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`; a basis of any size is supported, streamed in bounded chunks — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
||||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination (same basis-size caveat; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination (the copy is streamed, so a basis of any size works; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
||||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win; same basis-size caveat; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win; a basis of any size works; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
||||||
| `--verify-basis` | FastSync-only: require a basis hit to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync). |
|
| `--verify-basis` | FastSync-only: require a basis hit to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync). |
|
||||||
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
|
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
|
||||||
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
|
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
|
||||||
@@ -606,7 +607,7 @@ remote SSH argv is already built injection-safe.
|
|||||||
| `--max-alloc <SIZE>` | Maximum single allocation (binary units; default 1G; `0` = no local limit). |
|
| `--max-alloc <SIZE>` | Maximum single allocation (binary units; default 1G; `0` = no local limit). |
|
||||||
| `--max-depth <n>` | Limit recursive scanning depth; zero means unlimited. |
|
| `--max-depth <n>` | Limit recursive scanning depth; zero means unlimited. |
|
||||||
| `-b, --backup` | Back up overwritten files. |
|
| `-b, --backup` | Back up overwritten files. |
|
||||||
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install (confined to the receive root; `EXDEV` falls back to a non-atomic copy). |
|
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install (confined to the receive root: relative resolves below it, absolute must canonicalize inside it; `EXDEV` falls back to a non-atomic copy). |
|
||||||
| `--backup-dir <dir>` | Store backups under a separate directory (requires `--backup`). |
|
| `--backup-dir <dir>` | Store backups under a separate directory (requires `--backup`). |
|
||||||
| `--suffix <suffix>` | Set the backup filename suffix (default: `~`). |
|
| `--suffix <suffix>` | Set the backup filename suffix (default: `~`). |
|
||||||
| `--partial` | Select partial-transfer handling. On failed/interrupted writes the already-written temp file is retained (best-effort) for resumption. With `--partial --partial-dir <dir>`, completed files are written under the partial directory and installed atomically. |
|
| `--partial` | Select partial-transfer handling. On failed/interrupted writes the already-written temp file is retained (best-effort) for resumption. With `--partial --partial-dir <dir>`, completed files are written under the partial directory and installed atomically. |
|
||||||
@@ -643,7 +644,7 @@ remote SSH argv is already built injection-safe.
|
|||||||
| `--groupmap=MAP` | Map group names when applying ownership (same syntax as `--usermap`). |
|
| `--groupmap=MAP` | Map group names when applying ownership (same syntax as `--usermap`). |
|
||||||
| `--numeric-ids` | Mapping modifier: apply the source numeric uid/gid directly instead of mapping by name (combine with `-o`/`-g`, `-a`, or a map). |
|
| `--numeric-ids` | Mapping modifier: apply the source numeric uid/gid directly instead of mapping by name (combine with `-o`/`-g`, `-a`, or a map). |
|
||||||
| `--copy-as=USER[:GROUP]` | Force every written entry to USER[:GROUP]; requires a privileged receiver. |
|
| `--copy-as=USER[:GROUP]` | Force every written entry to USER[:GROUP]; requires a privileged receiver. |
|
||||||
| `--fake-super` | Record the resolved owner plus mode/time in a reserved `user.fastsync.stat` xattr and replay mode/time; never performs a real chown. |
|
| `--fake-super` | Record the resolved owner plus full mode/rdev in rsync's reserved `user.rsync.%stat` xattr (rsync 3.4.1 grammar) and replay the permission bits; never performs a real chown. |
|
||||||
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes). |
|
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes). |
|
||||||
| `--no-super` | Forbid those super-user activities even when the receiver is root. |
|
| `--no-super` | Forbid those super-user activities even when the receiver is root. |
|
||||||
| `-l`, `--links` | Copy symlinks as symlinks; the target is stored verbatim (absolute and `..`-bearing targets included), matching rsync. |
|
| `-l`, `--links` | Copy symlinks as symlinks; the target is stored verbatim (absolute and `..`-bearing targets included), matching rsync. |
|
||||||
@@ -674,9 +675,9 @@ remote SSH argv is already built injection-safe.
|
|||||||
| `--outbuf=MODE` | stdout/stderr buffering: `N` (none/unbuffered), `L` (line-buffered), or `B` (block-buffered, default). |
|
| `--outbuf=MODE` | stdout/stderr buffering: `N` (none/unbuffered), `L` (line-buffered), or `B` (block-buffered, default). |
|
||||||
| `--log-file <path>` | Write log output to a file. |
|
| `--log-file <path>` | Write log output to a file. |
|
||||||
| `--log-file-format=FORMAT` | Per-file log-line format (requires `--log-file`). |
|
| `--log-file-format=FORMAT` | Per-file log-line format (requires `--log-file`). |
|
||||||
| `--stderr=MODE` | Route logging to stderr: `errors` or `all`. |
|
| `--stderr=MODE` | Route logging: `errors` (default), `all`, or `client` (forward the client's diagnostics to the server's stderr over the client-message channel). |
|
||||||
| `--msgs2stderr` | Route all messages to stderr (deprecated spelling of `--stderr=all`). |
|
| `--msgs2stderr` | Route all messages to stderr (deprecated spelling of `--stderr=all`). |
|
||||||
| `--no-msgs2stderr` | Select errors-only stderr (deprecated spelling; the default). |
|
| `--no-msgs2stderr` | Forward the client's diagnostics to the server (deprecated spelling of `--stderr=client`). |
|
||||||
| `-V`, `--version` | Print the FastSync protocol version. |
|
| `-V`, `--version` | Print the FastSync protocol version. |
|
||||||
| `--help` | Print command usage. |
|
| `--help` | Print command usage. |
|
||||||
|
|
||||||
@@ -768,9 +769,17 @@ and `address`, the global section accepts:
|
|||||||
- `hosts allow` / `hosts deny` — comma- and/or whitespace-separated host access
|
- `hosts allow` / `hosts deny` — comma- and/or whitespace-separated host access
|
||||||
patterns.
|
patterns.
|
||||||
|
|
||||||
A `[module]` requires `path`, and may also set `read only`, `client owner`,
|
A `[module]` requires `path`, and may also set `read only`, `write only`,
|
||||||
`auth users`, `max connections` (0 = unlimited; enforced per module across all
|
`client owner`, `auth users`, `max connections` (0 = unlimited; enforced per
|
||||||
connection children), and its own `hosts allow`/`hosts deny`.
|
module across all connection children), and its own `hosts allow`/`hosts deny`.
|
||||||
|
|
||||||
|
Like rsync, a module is **read-only by default**: a bare `[module]` with only a
|
||||||
|
`path` refuses a write transfer. Opt a module into writability explicitly with
|
||||||
|
`read only = no` or `write only = yes`; a global `read only` value in the
|
||||||
|
section before the first `[module]` sets the default for later modules, and a
|
||||||
|
module's own `read only`/`write only = yes` always wins over it. An
|
||||||
|
rsync-style `write only = yes` is mapped to writability because FastSync is
|
||||||
|
push-only (a module can never be read from the network).
|
||||||
|
|
||||||
The per-host cap and the shared auth lockout identify a source by its numeric
|
The per-host cap and the shared auth lockout identify a source by its numeric
|
||||||
peer IP. **Loopback peers (127.0.0.0/8, IPv6 `::1`) are exempt**: every local
|
peer IP. **Loopback peers (127.0.0.0/8, IPv6 `::1`) are exempt**: every local
|
||||||
@@ -824,7 +833,7 @@ before the module list, before authentication, and the connecting peer address
|
|||||||
|
|
||||||
## Protocol and Security
|
## Protocol and Security
|
||||||
|
|
||||||
FastSync protocol version `2.28.0` is shared by the client and server. The
|
FastSync protocol version `2.30.0` is shared by the client and server. The
|
||||||
current protocol is sender-driven and includes configuration negotiation,
|
current protocol is sender-driven and includes configuration negotiation,
|
||||||
including the maximum allocation limit, incremental checks, checksums,
|
including the maximum allocation limit, incremental checks, checksums,
|
||||||
manifests, keep-alives, abort handling, per-file remove-source results, and
|
manifests, keep-alives, abort handling, per-file remove-source results, and
|
||||||
|
|||||||
+103
-74
@@ -6,18 +6,18 @@ This document maps rsync's full feature set to FastSync's current implementation
|
|||||||
|
|
||||||
| Status | Count | Description |
|
| Status | Count | Description |
|
||||||
|--------|-------|-------------|
|
|--------|-------|-------------|
|
||||||
| ✅ Parity | 117 | Reproduces rsync's semantics for this option's scope |
|
| ✅ Parity | 119 | Reproduces rsync's semantics for this option's scope |
|
||||||
| ⚠️ Caveat | 13 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
|
| ⚠️ Caveat | 16 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
|
||||||
| ❌ Divergent | 27 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
|
| ❌ Divergent | 22 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
|
||||||
| **Total** | **157** | One row per rsync option/feature group; a row may name several spellings |
|
| **Total** | **157** | One row per rsync option/feature group; a row may name several spellings |
|
||||||
|
|
||||||
This matrix reports honest rsync parity, not "implemented" as a synonym for
|
This matrix reports honest rsync parity, not "implemented" as a synonym for
|
||||||
"parsed". A ✅ row matches rsync for the option's scope. A ⚠️ row is real and
|
"parsed". A ✅ row matches rsync for the option's scope. A ⚠️ row is real and
|
||||||
tested but diverges in at least one documented way. An ❌ row is either
|
tested but diverges in at least one documented way. An ❌ row is either
|
||||||
rejected (`--protocol` with any value but the current one, `--inc-recursive`),
|
rejected (`--protocol` with any value but the current one),
|
||||||
an accepted no-op (`-s`/`--secluded-args`, `--protect-args`, `--old-args`),
|
an accepted no-op (`-s`/`--secluded-args`, `--protect-args`, `--old-args`),
|
||||||
deliberately non-rsync and non-interoperable (the FastSync daemon config/auth,
|
deliberately non-rsync and non-interoperable (the FastSync daemon config/auth,
|
||||||
the batch container, `--fake-super`'s xattr format, `--copy-as` credential
|
the batch container, `--copy-as` credential
|
||||||
switching), or impossible (`-N`/`--crtimes`). The counts are derived from the
|
switching), or impossible (`-N`/`--crtimes`). The counts are derived from the
|
||||||
rows below; update them together with the table.
|
rows below; update them together with the table.
|
||||||
|
|
||||||
@@ -55,10 +55,12 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
|
|||||||
|
|
||||||
**Lockstep track 6 (delete default; `PROTOCOL_VERSION` stays 2.28.0).** Plain `--delete` with no explicit timing flag now defaults to rsync's delete-during (`--del`) timing: the client normalizes it onto the existing `delete_during` wire bool in `cli_finalize_config`, so no config-frame field was added, and a tight destination no longer has to hold the whole old+new tree at once (the old atomic commit could hit `ENOSPC`). The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`, which selects the identical `delete_after` timing (documented equivalence). Precedence is unchanged and order-independent: each timing flag implies `--delete`, at most one timing flag may be given, and a timing flag with `--no-delete` is rejected. `-d/--dirs` still falls back to the end commit; `--delay-updates` still deletes genuine extras before publication (the per-directory skip list protects the staging dir); `--files-from`/`-R` scope is unchanged. The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag (still 2.28.0): the one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) is now always sent first on a config-only carrier with `apply=false`, fixing a latent bug where a `--delete-missing-args` run whose `--files-from` list synchronized no directory never transmitted its exact deletions. Differential evidence: `delete` (plain, vs rsync's default), `delete_commit` (FastSync `--delete-commit` vs rsync `--delete-after`), and `filter_protect_after` (whole-tree protect) cases; `TestDeleteTimingFinalStateParity` compares plain `--delete`/`--delete-commit` against rsync on completed runs, and `TestDeleteTimingFailure` proves plain `--delete` removes reached extras on a mid-transfer abort while `--delete-commit` removes nothing. The matrix is unchanged at **116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay ⚠️ for the abort boundary; `--delete-after` stays ✅).
|
**Lockstep track 6 (delete default; `PROTOCOL_VERSION` stays 2.28.0).** Plain `--delete` with no explicit timing flag now defaults to rsync's delete-during (`--del`) timing: the client normalizes it onto the existing `delete_during` wire bool in `cli_finalize_config`, so no config-frame field was added, and a tight destination no longer has to hold the whole old+new tree at once (the old atomic commit could hit `ENOSPC`). The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`, which selects the identical `delete_after` timing (documented equivalence). Precedence is unchanged and order-independent: each timing flag implies `--delete`, at most one timing flag may be given, and a timing flag with `--no-delete` is rejected. `-d/--dirs` still falls back to the end commit; `--delay-updates` still deletes genuine extras before publication (the per-directory skip list protects the staging dir); `--files-from`/`-R` scope is unchanged. The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag (still 2.28.0): the one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) is now always sent first on a config-only carrier with `apply=false`, fixing a latent bug where a `--delete-missing-args` run whose `--files-from` list synchronized no directory never transmitted its exact deletions. Differential evidence: `delete` (plain, vs rsync's default), `delete_commit` (FastSync `--delete-commit` vs rsync `--delete-after`), and `filter_protect_after` (whole-tree protect) cases; `TestDeleteTimingFinalStateParity` compares plain `--delete`/`--delete-commit` against rsync on completed runs, and `TestDeleteTimingFailure` proves plain `--delete` removes reached extras on a mid-transfer abort while `--delete-commit` removes nothing. The matrix is unchanged at **116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay ⚠️ for the abort boundary; `--delete-after` stays ✅).
|
||||||
|
|
||||||
|
**Parity2 #317 (no wire change; `PROTOCOL_VERSION` stays 2.30.0).** Two `--delay-updates` residuals closed. (1) The receiver now stages under a per-run unique `.fastsync-stage.<pid>.<entropy>` directory instead of a fixed name, and `delay_updates_prepare` creates it with O_EXCL semantics: if that exact path already exists it refuses rather than wiping it, so a genuine destination entry named like the staging prefix is never destroyed (rsync likewise leaves a real entry named like its `.~tmp~` temp untouched; differential `test_delay_updates_staging_name_collision_preserved`). The delete walker now skips this transfer's runtime staging name. (2) `--delay-updates` implies `--delete-after`: the client normalizes it onto the existing `delete_after` wire bool in `cli_finalize_config` (no new wire field), and both the single-threaded and `--threads` receivers publish every staged file before committing the deferred delete. `--backup` files are now shielded from the delete-after pass (rsync never treats a backup as an extra), and a destination directory blocking a staged file is cleared at publication when `--delete`/`--force` is active (rsync's make-way). New differential cases `delay_updates` and `delay_updates_delete` plus `TestDelayUpdates` coverage assert final-state parity with rsync 3.4.1. Residuals: FastSync still does not create a backup of a *deleted* extra (rsync's `--backup --delete` does), and a crash-leftover staging directory is never reused (the next run picks a fresh name). The `--delay-updates` row moves ❌ → ⚠️. The matrix is now **119 ✅ / 16 ⚠️ / 22 ❌ = 157**.
|
||||||
|
|
||||||
**Parity cycle 2.29 (on `feat/parity-2.29`; `PROTOCOL_VERSION` stays 2.28.0 — no wire change was needed).** Five independent residuals were closed and four rows moved to ✅:
|
**Parity cycle 2.29 (on `feat/parity-2.29`; `PROTOCOL_VERSION` stays 2.28.0 — no wire change was needed).** Five independent residuals were closed and four rows moved to ✅:
|
||||||
- **Scanner order.** The sequential scanner now buffers and sorts each directory's inspected entries (non-directories ascending, then directories ascending) and walks them depth-first, reproducing rsync 3.4.1's flist order. This makes the `--info=name` transfer order, the `--delete-during`/`--delete-delay`/`-n` would-delete order, and the partial-`--max-delete` survivor set byte-identical to rsync (`test_parity_order.py`). `--threads` has no rsync analogue and stays unordered.
|
- **Scanner order.** The sequential scanner now buffers and sorts each directory's inspected entries (non-directories ascending, then directories ascending) and walks them depth-first, reproducing rsync 3.4.1's flist order. This makes the `--info=name` transfer order, the `--delete-during`/`--delete-delay`/`-n` would-delete order, and the partial-`--max-delete` survivor set byte-identical to rsync (`test_parity_order.py`). `--threads` has no rsync analogue and stays unordered.
|
||||||
- **Delete timing.** The complete per-directory plan set is transmitted before the first data frame, so a mid-transfer abort has already removed every planned extra like rsync's generator; `-d/--dirs` uses the same per-directory plans (shielded untraversed subdirectories) instead of the end-of-transfer commit (`test_delete_boundary_parity.py`). `-n`/`--delete`/`--del`/`--delete-delay` move ⚠️ → ✅.
|
- **Delete timing.** The complete per-directory plan set is transmitted before the first data frame, so a mid-transfer abort has already removed every planned extra like rsync's generator; `-d/--dirs` uses the same per-directory plans (shielded untraversed subdirectories) instead of the end-of-transfer commit (`test_delete_boundary_parity.py`). `-n`/`--delete`/`--del`/`--delete-delay` move ⚠️ → ✅.
|
||||||
- **Basis relative-DIR.** A relative `--compare-dest`/`--copy-dest`/`--link-dest` DIR resolves against the destination directory with the transfer-relative name appended, exactly like rsync (`test_parity_basis_fuzzy.py`); the >256 MiB basis-MISS limit remains (a general whole-file limit, not basis-specific).
|
- **Basis relative-DIR.** A relative `--compare-dest`/`--copy-dest`/`--link-dest` DIR resolves against the destination directory with the transfer-relative name appended, exactly like rsync (`test_parity_basis_fuzzy.py`); the >256 MiB basis-MISS limit is closed (#318 streams a MISS above the bound, `test_large_stream.py`).
|
||||||
- **Fuzzy eligibility.** The `-y/--fuzzy` candidate search no longer inherits the ordinary delta engine's 16 KiB minimum or 10× ratio bound, so an oversized or sub-16-KiB sibling is reused as rsync reuses it (`test_parity_basis_fuzzy.py`).
|
- **Fuzzy eligibility.** The `-y/--fuzzy` candidate search no longer inherits the ordinary delta engine's 16 KiB minimum or 10× ratio bound, so an oversized or sub-16-KiB sibling is reused as rsync reuses it (`test_parity_basis_fuzzy.py`).
|
||||||
- **Output partials.** `--info=mount`/`--info=stats`, the `--stats` `dir:` breakdown under `-r`, and real `--debug` output for `flist`/`del`/`hash`/`deltasum`/`recv`/`filter`/`send` were added (`test_parity_info_mount_stats.py`, `test_output_parity.py`, `test_parity_debug.py`); those rows stay ⚠️ for their remaining documented residuals. `--delete-before`'s phase-0 late-file divergence and the `--progress` root/ancestor/symlink feedback remain open (they need a receiver→sender event channel), and the >256 MiB single-file streaming limit (B4) was not addressed. The matrix is now **120 ✅ / 10 ⚠️ / 27 ❌ = 157**.
|
- **Output partials.** `--info=mount`/`--info=stats`, the `--stats` `dir:` breakdown under `-r`, and real `--debug` output for `flist`/`del`/`hash`/`deltasum`/`recv`/`filter`/`send` were added (`test_parity_info_mount_stats.py`, `test_output_parity.py`, `test_parity_debug.py`); those rows stay ⚠️ for their remaining documented residuals. `--delete-before`'s phase-0 late-file divergence and the `--progress` root/ancestor/symlink feedback remain open (they need a receiver→sender event channel), and the >256 MiB single-file streaming limit (B4) was not addressed. The matrix is now **120 ✅ / 10 ⚠️ / 27 ❌ = 157**.
|
||||||
|
|
||||||
@@ -83,8 +85,8 @@ output, codec breadth, general `-R`/`-d`, the filter grammar (the unsupported
|
|||||||
rejected elsewhere — see the audit-cycle follow-up note above), receiver-side
|
rejected elsewhere — see the audit-cycle follow-up note above), receiver-side
|
||||||
name resolution, absolute basis dirs, and the remaining client quick wins) and
|
name resolution, absolute basis dirs, and the remaining client quick wins) and
|
||||||
reclassified the inherently non-rsync rows as **divergent** (native daemon
|
reclassified the inherently non-rsync rows as **divergent** (native daemon
|
||||||
config/auth, the non-interoperable batch container, `--fake-super`'s xattr
|
config/auth, the non-interoperable batch container,
|
||||||
format, `-X`'s privileged namespaces, and the safe-subset device/privilege
|
`-X`'s privileged namespaces, and the safe-subset device/privilege
|
||||||
flags). It moved `PROTOCOL_VERSION` three times (`2.23.0 → 2.24.0` delete
|
flags). It moved `PROTOCOL_VERSION` three times (`2.23.0 → 2.24.0` delete
|
||||||
timing, `2.24.0 → 2.25.0` wire stats, `2.25.0 → 2.26.0` codecs). See the
|
timing, `2.24.0 → 2.25.0` wire stats, `2.25.0 → 2.26.0` codecs). See the
|
||||||
**Parity Completion Wave (protocol 2.26.0)** section near the end for the full
|
**Parity Completion Wave (protocol 2.26.0)** section near the end for the full
|
||||||
@@ -110,8 +112,8 @@ Every one of those has an entry below with its remaining caveats.
|
|||||||
| `-V`, `--version` | Print version | ✅ Parity | |
|
| `-V`, `--version` | Print version | ✅ Parity | |
|
||||||
| `--info=FLAGS` | Fine-grained info verbosity | ⚠️ Caveat | Accepts rsync 3.4.1's full `--info` vocabulary — `backup`, `copy`, `del`, `flist`, `misc`, `mount`, `name`, `nonreg`, `progress`, `remove`, `skip`, `stats`, `symsafe`, `all`, `none` — with optional level suffixes (`--info=stats2`), so a valid rsync invocation is never rejected up front. Protocol 2.27.0 wires the categories that map to a real FastSync event, matching rsync's line format: `name` prints the updated entry names (with the ` -> target` link suffix), `flist` prints `sending incremental file list`, `del` prints `deleting PATH` (or `*deleting PATH` under `-i`/`--out-format`) for both dry-run would-delete and real deletions (real runs carry the removed paths over the new `report_deletes` wire bool), `remove` prints `sender removed PATH`, `nonreg` prints `skipping non-regular file "NAME"`, `progress` drives the per-file progress output, `copy`/`misc`/`skip` keep their existing channels, `stats` enables the same transfer-statistics block as `--stats`, and `mount` prints rsync's `[sender] skipping mount-point dir NAME` when `-xx` drops a mount-point directory (plain `-x` keeps the empty directory entry and stays silent, matching rsync; both differential-tested). `none` suppresses info output, explicit flags override `--verbose`, and a genuinely unknown name is still rejected by name (matching rsync). **Fixed (no-wire):** `--info=name2` (and higher) also prints rsync's `NAME is uptodate` lines for entries the receiver already has, and `--info=name` emits the leading transfer-root `./` name line before the first transferred entry (the marker rides in the existing `info_level` bitset; differential tests vs rsync 3.4.1). **Caveat:** the root `./` line is emitted before the first transferred name rather than keyed off rsync's root-attribute-change decision, so a pre-existing root that rsync leaves untouched can differ; the categories with no client-observable event stay accepted-but-silent — `symsafe` and `backup` (the backup happens on the receiver, which FastSync's protocol does not echo back); and `skip` maps to FastSync's sender-side skip logging rather than rsync's receiver-side "not creating new file" lines |
|
| `--info=FLAGS` | Fine-grained info verbosity | ⚠️ Caveat | Accepts rsync 3.4.1's full `--info` vocabulary — `backup`, `copy`, `del`, `flist`, `misc`, `mount`, `name`, `nonreg`, `progress`, `remove`, `skip`, `stats`, `symsafe`, `all`, `none` — with optional level suffixes (`--info=stats2`), so a valid rsync invocation is never rejected up front. Protocol 2.27.0 wires the categories that map to a real FastSync event, matching rsync's line format: `name` prints the updated entry names (with the ` -> target` link suffix), `flist` prints `sending incremental file list`, `del` prints `deleting PATH` (or `*deleting PATH` under `-i`/`--out-format`) for both dry-run would-delete and real deletions (real runs carry the removed paths over the new `report_deletes` wire bool), `remove` prints `sender removed PATH`, `nonreg` prints `skipping non-regular file "NAME"`, `progress` drives the per-file progress output, `copy`/`misc`/`skip` keep their existing channels, `stats` enables the same transfer-statistics block as `--stats`, and `mount` prints rsync's `[sender] skipping mount-point dir NAME` when `-xx` drops a mount-point directory (plain `-x` keeps the empty directory entry and stays silent, matching rsync; both differential-tested). `none` suppresses info output, explicit flags override `--verbose`, and a genuinely unknown name is still rejected by name (matching rsync). **Fixed (no-wire):** `--info=name2` (and higher) also prints rsync's `NAME is uptodate` lines for entries the receiver already has, and `--info=name` emits the leading transfer-root `./` name line before the first transferred entry (the marker rides in the existing `info_level` bitset; differential tests vs rsync 3.4.1). **Caveat:** the root `./` line is emitted before the first transferred name rather than keyed off rsync's root-attribute-change decision, so a pre-existing root that rsync leaves untouched can differ; the categories with no client-observable event stay accepted-but-silent — `symsafe` and `backup` (the backup happens on the receiver, which FastSync's protocol does not echo back); and `skip` maps to FastSync's sender-side skip logging rather than rsync's receiver-side "not creating new file" lines |
|
||||||
| `--debug=FLAGS` | Fine-grained debug verbosity | ⚠️ Caveat | Protocol 2.26.0 accepts rsync 3.4.1's full `--debug` vocabulary with optional level suffixes. FastSync emits for its own channels (`io`, `proto`, `pack`, `util`, plus the aliases `hl`/`owner`) and maps the remaining categories that have a natural FastSync event onto real debug output: `flist` (per-directory scan progress), `del` (receiver-removed paths, riding the existing `report_deletes` wire bool), `hash`/`deltasum` (whole-file hashing and delta-sum generation), `recv` (receiver verdicts/signatures), `filter` (selection/exclusion decisions) and `send` (files handed to the sender). A normal run prints none of it; `--debug=help` lists the flags and a genuinely unknown name is rejected by name. **Caveat:** the output is FastSync's own timestamped debug format (it does not reproduce rsync's exact per-category lines), and the synthetic/rsync-internal categories (`acl`, `backup`, `bind`, `time`, ...) stay accepted-but-silent, so the row remains ⚠️ |
|
| `--debug=FLAGS` | Fine-grained debug verbosity | ⚠️ Caveat | Protocol 2.26.0 accepts rsync 3.4.1's full `--debug` vocabulary with optional level suffixes. FastSync emits for its own channels (`io`, `proto`, `pack`, `util`, plus the aliases `hl`/`owner`) and maps the remaining categories that have a natural FastSync event onto real debug output: `flist` (per-directory scan progress), `del` (receiver-removed paths, riding the existing `report_deletes` wire bool), `hash`/`deltasum` (whole-file hashing and delta-sum generation), `recv` (receiver verdicts/signatures), `filter` (selection/exclusion decisions) and `send` (files handed to the sender). A normal run prints none of it; `--debug=help` lists the flags and a genuinely unknown name is rejected by name. **Caveat:** the output is FastSync's own timestamped debug format (it does not reproduce rsync's exact per-category lines), and the synthetic/rsync-internal categories (`acl`, `backup`, `bind`, `time`, ...) stay accepted-but-silent, so the row remains ⚠️ |
|
||||||
| `--stderr=MODE` | Change stderr output mode | ❌ Divergent | `errors` (default) and `all` are supported; `client` is rejected with a clear error (`--stderr=client is not supported`) because FastSync has no rsync client-message channel — the rejection itself is the documented behavior (Phase 7 Wave B decision). The modes that exist work; the missing rsync channel cannot be emulated without a wire change |
|
| `--stderr=MODE` | Change stderr output mode | ⚠️ Caveat | `errors` (default) and `all` are supported and match rsync. As of protocol 2.30.0 `client` is accepted, and the client's own diagnostics are forwarded to the peer's stderr over the new bounded `STATUS_CLIENT_MSG` client-message channel (the server writes each received message to its stderr respecting the server log destination) instead of writing locally. Caveat: rsync's `client` mode is its historical single-client-process multiplexing of every process's messages (the client's errors on its own stderr, info on stdout), whereas FastSync's push-only protocol has no server->client message stream, so only the client->server direction is reproduced |
|
||||||
| `--msgs2stderr`, `--no-msgs2stderr` | Deprecated `--stderr` aliases | ⚠️ Caveat | `--msgs2stderr` maps to `--stderr=all` (supported, matching rsync). `--no-msgs2stderr` is rsync's spelling of `--stderr=client`, which FastSync has no client-message channel for, so it maps to the errors-only default instead of reproducing rsync's client mode. See `--stderr=MODE` |
|
| `--msgs2stderr`, `--no-msgs2stderr` | Deprecated `--stderr` aliases | ⚠️ Caveat | `--msgs2stderr` maps to `--stderr=all` (supported, matching rsync). `--no-msgs2stderr` is rsync's spelling of `--stderr=client`; as of protocol 2.30.0 it maps to the `client` mode and forwards the client's diagnostics to the server's stderr over the `STATUS_CLIENT_MSG` channel instead of the old errors-only approximation. See `--stderr=MODE` for the one-direction caveat |
|
||||||
| `--no-motd` | Suppress daemon MOTD | ✅ Parity | Client-only display switch (Wave C): the daemon still sends the configured `motd file` on a `host::module/path` connection; the client reads and discards the frame without showing it. Without the flag the MOTD is printed to stdout after the config/auth handshake and escaped so control bytes cannot inject terminal sequences |
|
| `--no-motd` | Suppress daemon MOTD | ✅ Parity | Client-only display switch (Wave C): the daemon still sends the configured `motd file` on a `host::module/path` connection; the client reads and discards the frame without showing it. Without the flag the MOTD is printed to stdout after the config/auth handshake and escaped so control bytes cannot inject terminal sequences |
|
||||||
| `--exclude=PATTERN` | Exclude files matching pattern | ✅ Parity | Glob matching in scanner |
|
| `--exclude=PATTERN` | Exclude files matching pattern | ✅ Parity | Glob matching in scanner |
|
||||||
| `--include=PATTERN` | Include files matching pattern | ✅ Parity | Glob matching in scanner |
|
| `--include=PATTERN` | Include files matching pattern | ✅ Parity | Glob matching in scanner |
|
||||||
@@ -121,12 +123,12 @@ Every one of those has an entry below with its remaining caveats.
|
|||||||
|
|
||||||
| Flag | Rsync Description | FastSync Status | Notes |
|
| Flag | Rsync Description | FastSync Status | Notes |
|
||||||
|------|-------------------|-----------------|-------|
|
|------|-------------------|-----------------|-------|
|
||||||
| `--stats` | Give transfer stats | ⚠️ Caveat | Prints transfer statistics. Protocol 2.25.0 populates the receiver-only counters the sender cannot observe (`Matched data`, `Number of deleted files`) from the receiver's `STATUS_STATS` report; the sender tracks the scanned file list per type so `Number of files` carries rsync's `(reg: X, dir: Y, link: Z, special: W)` breakdown (directories come from the scanner's captured directory list for `-a`/`-t`/`-p`, or from a lightweight traversed-directory counter on a plain `-r` run so the `dir:` category is present there too), `Number of regular files transferred` excludes symlinks/specials and up-to-date files, `Total file size` includes symlink target lengths, and `Total transferred file size` counts only transferred files. **Protocol 2.28.0 extends `STATUS_STATS`** with receiver-observed `literal_bytes` and the four `created_*` counters: `Number of created files` now carries rsync's `(reg/dir/link/special)` breakdown (the receiver reports which destination entries it newly created, including implicitly-created parent directories below the transfer root) and `Literal data` is exact for a delta transfer (the receiver counts the literal fragments it stored, not the whole source size) — all differential-tested in the sequential and `--threads` paths against rsync 3.4.1 for fresh-create, update and delta shapes. **Remaining divergences:** rsync's per-type breakdown on `Number of deleted files` is not reproduced; and `Total bytes sent`/`received` are FastSync wire bytes framed differently from rsync's, so they are not numerically comparable |
|
| `--stats` | Give transfer stats | ⚠️ Caveat | Prints transfer statistics. Protocol 2.25.0 populates the receiver-only counters the sender cannot observe (`Matched data`, `Number of deleted files`) from the receiver's `STATUS_STATS` report; the sender tracks the scanned file list per type so `Number of files` carries rsync's `(reg: X, dir: Y, link: Z, special: W)` breakdown (directories come from the scanner's captured directory list for `-a`/`-t`/`-p`, or from a lightweight traversed-directory counter on a plain `-r` run so the `dir:` category is present there too), `Number of regular files transferred` excludes symlinks/specials and up-to-date files, `Total file size` includes symlink target lengths, and `Total transferred file size` counts only transferred files. **Protocol 2.28.0 extends `STATUS_STATS`** with receiver-observed `literal_bytes` and the four `created_*` counters: `Number of created files` now carries rsync's `(reg/dir/link/special)` breakdown (the receiver reports which destination entries it newly created, including implicitly-created parent directories below the transfer root) and `Literal data` is exact for a delta transfer (the receiver counts the literal fragments it stored, not the whole source size) — all differential-tested in the sequential and `--threads` paths against rsync 3.4.1 for fresh-create, update and delta shapes. **Protocol 2.30.0 appends the four `deleted_*` counters**: the delete observers classify every entry the receiver ACTUALLY removed (regular/dir/symlink/special, a strict partition of the existing scalar), so `Number of deleted files: X (reg: A, dir: B, link: C, special: D)` matches rsync exactly — differential-tested for a mixed-type `--delete` tree (`test_differential_parity.py::test_stats_deleted_breakdown_matches_rsync`). **Remaining divergence:** `Total bytes sent`/`received` are FastSync wire bytes framed differently from rsync's, so they are not numerically comparable |
|
||||||
| `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units. **A lone `-h` with no transfer arguments prints help instead** (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer remains human-readable |
|
| `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units. **A lone `-h` with no transfer arguments prints help instead** (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer remains human-readable |
|
||||||
| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision; **every** non-root directory is rendered as created (`cd+++++++++`) because the sender never probes a directory's destination state, so a pre-existing destination directory that rsync reports as unchanged (`.d..t......`) is still itemized as created — this is not limited to re-runs; an incremental re-run additionally itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`); and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
|
| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Protocol 2.30.0 closes the directory/symlink destination-state residual (#314):** when `report_dest_info` is negotiated (now also for `--progress`) the receiver answers `STATUS_MKDIR`/`STATUS_SYMLINK` with the entry's pre-transfer snapshot — including whether an existing symlink's target already matches — and the sender probes every ancestor directory before the receiver creates it implicitly. A re-run over an unchanged tree therefore emits no per-directory `cd+++++++++` and no unchanged-symlink line (matching rsync), a changed directory renders `.d..t......` (not `cd`), and a changed symlink renders `cLc........` / `.L..t......`. The time column compares whole seconds for directories/symlinks (rsync's `cmp_time`), so a sub-second-only difference is not a spurious `t`. Differential: `test_differential_parity.py::test_itemize_rerun_dirs_symlinks_matches_rsync`. **Residual:** the root `./` line is still emitted unconditionally rather than keyed off rsync's root-attribute-change decision; a directory whose ONLY change is an attribute and which has no transferred child is itemized at the end of the run (the pending-directory flush) rather than in rsync's depth-first position; and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
|
||||||
| `--progress` | Show progress | ⚠️ Caveat | Protocol 2.25.0 prints rsync-style per-file progress blocks (percent, transferred/total bytes, rate, elapsed, `(xfr#N, to-chk=M/T)`) fed by the receiver's `STATUS_STATS`, in both the sequential and `--threads` send paths. FastSync also prints rsync's leading `./` transfer-root line and, when progress is requested (`--progress`/`-P`/`--info=progress`) and not `--quiet`, runs a **paths-only metadata pre-scan** (no file reads, no hashing) that supplies rsync's file-list total `T` for the `to-chk` denominator and the directory names; `--delete-during`/`--delete-delay` reuse their existing keep-set pre-scan instead of walking twice, and non-progress runs are untouched. Per-directory name lines are emitted (trailing `/`), and symlink (` -> target`) and special entries are named too, so a **fresh multi-directory tree's name set and `to-chk` denominator match rsync 3.4.1** (differential test, sequential and `--threads`) and a **single-file transfer's name lines and deterministic frames remain byte-identical** to rsync. **Order parity (parity-2.29):** the sequential scanner now emits entries in rsync's sorted depth-first flist order (non-directories ascending, then directories ascending), so the interleaving and the `to-chk` numerator match rsync for the default single-threaded transfer (differential `test_parity_order.py`; `--threads` has no rsync analogue and stays unordered). **Remaining divergences:** the leading `./` root line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an ancestor directory line is emitted whenever a child transfers (rsync suppresses it when the directory itself is unchanged); on a re-run, entries without a quick-check (symlinks, empty directories) are still named where rsync stays silent; and the rate/ETA are wall-clock dependent |
|
| `--progress` | Show progress | ⚠️ Caveat | Protocol 2.25.0 prints rsync-style per-file progress blocks (percent, transferred/total bytes, rate, elapsed, `(xfr#N, to-chk=M/T)`) fed by the receiver's `STATUS_STATS`, in both the sequential and `--threads` send paths. FastSync also prints rsync's leading `./` transfer-root line and, when progress is requested (`--progress`/`-P`/`--info=progress`) and not `--quiet`, runs a **paths-only metadata pre-scan** (no file reads, no hashing) that supplies rsync's file-list total `T` for the `to-chk` denominator and the directory names; `--delete-during`/`--delete-delay` reuse their existing keep-set pre-scan instead of walking twice, and non-progress runs are untouched. Per-directory name lines are emitted (trailing `/`), and symlink (` -> target`) and special entries are named too, so a **fresh multi-directory tree's name set and `to-chk` denominator match rsync 3.4.1** (differential test, sequential and `--threads`) and a **single-file transfer's name lines and deterministic frames remain byte-identical** to rsync. **Order parity (parity-2.29):** the sequential scanner now emits entries in rsync's sorted depth-first flist order (non-directories ascending, then directories ascending), so the interleaving and the `to-chk` numerator match rsync for the default single-threaded transfer (differential `test_parity_order.py`; `--threads` has no rsync analogue and stays unordered). **Destination-state suppression (#314, protocol 2.30.0):** a pre-existing directory is no longer named (rsync names a directory only when it creates it) and an unchanged symlink is no longer named, so a re-run over an unchanged tree prints no directory/symlink name lines where rsync stays silent. **Remaining divergences:** the leading `./` root line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision; and the rate/ETA are wall-clock dependent |
|
||||||
| `-P` | Same as --partial --progress | ✅ Parity | Parses to `--partial` + `--progress`. The independent `--partial` retention semantics are rsync parity: an interrupted write retains the already-written temp at the destination (best-effort) so a later `--append`/`--append-verify` can resume. Progress presentation is owned by the `--progress` row; there is no separate `-P` divergence |
|
| `-P` | Same as --partial --progress | ✅ Parity | Parses to `--partial` + `--progress`. The independent `--partial` retention semantics are rsync parity: an interrupted write retains the already-written temp at the destination (best-effort) so a later `--append`/`--append-verify` can resume. Progress presentation is owned by the `--progress` row; there is no separate `-P` divergence |
|
||||||
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally, every non-root directory renders as created because the sender does not probe directory destination state (so a pre-existing unchanged directory still shows `cd+++++++++`), and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
|
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines); protocol 2.30.0 also gives them destination state (#314), so a pre-existing unchanged directory renders `.d..t......` (or nothing) instead of `cd+++++++++` and an unchanged symlink is suppressed, with the same `-i` residuals: the root line is emitted unconditionally and a directory whose only change is an attribute (no transferred child) is itemized at the end of the run; directory attribute columns (`%M`/`%U`/`%G`) come from the source |
|
||||||
| `--log-file=FILE` | Log to file | ✅ Parity | `log_file` config field |
|
| `--log-file=FILE` | Log to file | ✅ Parity | `log_file` config field |
|
||||||
| `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` as the wire byte count) |
|
| `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` as the wire byte count) |
|
||||||
| `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Parity | Applies to displayed paths and protocol debug output |
|
| `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Parity | Applies to displayed paths and protocol debug output |
|
||||||
@@ -138,7 +140,7 @@ Every one of those has an entry below with its remaining caveats.
|
|||||||
|------|-------------------|-----------------|-------|
|
|------|-------------------|-----------------|-------|
|
||||||
| `--exclude-from=FILE` | Read exclude patterns from file | ✅ Parity | Reads patterns from file |
|
| `--exclude-from=FILE` | Read exclude patterns from file | ✅ Parity | Reads patterns from file |
|
||||||
| `--include-from=FILE` | Read include patterns from file | ✅ Parity | Reads patterns from file |
|
| `--include-from=FILE` | Read include patterns from file | ✅ Parity | Reads patterns from file |
|
||||||
| `--filter=RULE` | Add file-filtering rule | ⚠️ Caveat | The short `-f` **is** bound to `--filter` (the old FastSync sendfile conflict is gone; sendfile is long-only `--sendfile`), and `-f RULE`, `-f=RULE`, `--filter=RULE` and the two-argument form all parse. Protocol 2.26.0 implements rsync's filter grammar: `+`/`-`, `include`/`exclude`, a leading `/` anchor (to the transfer root or a `.rsync-filter` file's directory), a trailing `/` dir-only rule, and the `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R` and `clear`/`!` words, including the `:`/`.` modifiers. The xattr-name `x` modifier is **explicitly rejected everywhere with a clear error**. The merge-only `e`/`n`/`w` and `-` modifiers are **accepted and consumed on `merge`/`dir-merge` rules** (so they no longer leak into the merge filename) while still being **rejected on non-merge rules**, matching rsync; their semantics remain unimplemented, so they are accepted-but-ignored (the reason this row is a caveat rather than parity). A token made up solely of modifier characters that names an unsupported modifier is rejected on non-merge rules, while glued patterns (`-newfile`, `-e2e`) and mixed tokens (`H,!secret`) keep their historical parsing. First match wins; the filter layer is independent of `--exclude`/`--include`. **Track 4a (protocol 2.28.0) adds the receiver filter engine:** the sender compiles its root-level rules exactly as the scanner does (`filter_base_build`) and streams them as one bounded, self-describing config-frame block; the receiver reconstructs them and re-applies first-match-wins to every extraneous destination path during deletion, so a `P *.log` rule protects a destination-only `extra.log` (differential `filter_protect`/`filter_protect_during`/`filter_protect_delay` vs rsync 3.4.1, plus the `-n` would-delete enumeration) — matching rsync's dual-sided engine for the command-line rule set. **Remaining residual:** per-directory merge (`:`/`.`, and therefore `-F`) is not yet re-derived on the receiver; a destination-only entry that matches ONLY a per-directory merge rule is still protected only through the sender-derived source-mirror prefixes, not by the received base rule list |
|
| `--filter=RULE` | Add file-filtering rule | ⚠️ Caveat | The short `-f` **is** bound to `--filter` (the old FastSync sendfile conflict is gone; sendfile is long-only `--sendfile`), and `-f RULE`, `-f=RULE`, `--filter=RULE` and the two-argument form all parse. Protocol 2.26.0 implements rsync's filter grammar: `+`/`-`, `include`/`exclude`, a leading `/` anchor (to the transfer root or a `.rsync-filter` file's directory), a trailing `/` dir-only rule, and the `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R` and `clear`/`!` words, including the `:`/`.` modifiers. The xattr-name `x` modifier is **explicitly rejected everywhere with a clear error**. The merge-only `e` (exclude the merge file itself), `n` (do not inherit into subdirectories), `w` (word-split the file on whitespace) and `-` (read the file as bare exclude/include patterns) modifiers are **implemented** on `merge`/`dir-merge` rules (they are still **rejected on non-merge rules**, matching rsync 3.4.1), verified by the `dir_merge_e`/`dir_merge_n`/`dir_merge_dash`/`dir_merge_w` differential cases. A token made up solely of modifier characters that names an unsupported modifier is rejected on non-merge rules, while glued patterns (`-newfile`, `-e2e`) and mixed tokens (`H,!secret`) keep their historical parsing. First match wins; the filter layer is independent of `--exclude`/`--include`. **Track 4a (protocol 2.28.0) adds the receiver filter engine:** the sender compiles its root-level rules exactly as the scanner does (`filter_base_build`) and streams them as one bounded, self-describing config-frame block; the receiver reconstructs them and re-applies first-match-wins to every extraneous destination path during deletion, so a `P *.log` rule protects a destination-only `extra.log` (differential `filter_protect`/`filter_protect_during`/`filter_protect_delay` vs rsync 3.4.1, plus the `-n` would-delete enumeration) — matching rsync's dual-sided engine for the command-line rule set. **Per-directory rules (protocol 2.30.0, issue #315) now reach the receiver:** the sender streams each traversed directory's compiled rules (owner directory + no-inherit flag, self-describing and bounded/validated) on the delete carrier, and the receiver evaluates the containing directory's rules before each ancestor's and then the command-line base (rsync's per-directory-before-ancestors order), so a destination-only entry that matches ONLY a per-directory `.rsync-filter`/dir-merge rule is shielded under every delete timing — the whole-tree commit, the `--delete-during`/`--delete-delay` per-directory plans, and the `-n` would-delete enumeration (differential `filter_perdir_protect{,_during,_delay,_after}`, `filter_perdir_exclude_{protect,deleted}`, plus the `TestFilterProtect::test_perdir_protect_*` regressions). **Narrowed residual:** rsync's receiver reads the merge file from its OWN side (the destination), whereas FastSync carries the sender's compiled source-side rules, so the two differ when the merge files differ — most visibly a destination-only `.rsync-filter` with no source counterpart is honored by rsync but not by FastSync; rsync's merge `C` (CVS-compatible) modifier is also not implemented |
|
||||||
| `--files-from=FILE` | Read source file list from file | ✅ Parity | Entries are paths relative to the source root (leading `./` stripped, `..`/absolute rejected at parse time, blank lines ignored; NUL-delimited with `-0`). A listed regular file is transferred; a listed directory transfers its whole subtree (FastSync recursion is always on). Non-listed paths are pruned by the scanner; the delete manifest is scoped to the listed directory subtrees. A listed entry that does not exist is a hard error unless `--ignore-missing-args`/`--delete-missing-args` is given. **An empty list is a zero-transfer success (exit 0), matching rsync 3.4.1** — the earlier claim that rsync reports "no source files specified" was wrong. Scalability note: `file_list_affects` is O(list size) per scanned entry, so a very large list against a huge tree is quadratic (the documented bound) |
|
| `--files-from=FILE` | Read source file list from file | ✅ Parity | Entries are paths relative to the source root (leading `./` stripped, `..`/absolute rejected at parse time, blank lines ignored; NUL-delimited with `-0`). A listed regular file is transferred; a listed directory transfers its whole subtree (FastSync recursion is always on). Non-listed paths are pruned by the scanner; the delete manifest is scoped to the listed directory subtrees. A listed entry that does not exist is a hard error unless `--ignore-missing-args`/`--delete-missing-args` is given. **An empty list is a zero-transfer success (exit 0), matching rsync 3.4.1** — the earlier claim that rsync reports "no source files specified" was wrong. Scalability note: `file_list_affects` is O(list size) per scanned entry, so a very large list against a huge tree is quadratic (the documented bound) |
|
||||||
| `-0`, `--from0` | Delimit *-from files with NULs | ✅ Parity | `--files-from` entries become NUL-delimited; the flag may appear before or after `--files-from` on the command line. NUL mode preserves entry bytes exactly (trailing CR/LF are part of the name; only newline mode trims them) |
|
| `-0`, `--from0` | Delimit *-from files with NULs | ✅ Parity | `--files-from` entries become NUL-delimited; the flag may appear before or after `--files-from` on the command line. NUL mode preserves entry bytes exactly (trailing CR/LF are part of the name; only newline mode trims them) |
|
||||||
| `--max-size=SIZE` | Skip files larger than SIZE | ✅ Parity | `max_size` in scanner |
|
| `--max-size=SIZE` | Skip files larger than SIZE | ✅ Parity | `max_size` in scanner |
|
||||||
@@ -150,7 +152,7 @@ Every one of those has an entry below with its remaining caveats.
|
|||||||
| `--ignore-existing` | Skip updating existing files | ✅ Parity | `ignore_existing` config field (crosses the wire; receiver-side policy). Protocol 2.26.0 short-circuits in the per-file check **before any payload**: when the destination entry already exists, the receiver answers the skip during the incremental handshake instead of letting the sender stream data that would be discarded, so an existing 4 MiB destination costs only the config/check frames (verified with a counting proxy, matching rsync). The write-time paths (regular, delay-updates-staged, hardlink-sibling, special/device) still return `FILE_SAVE_SKIPPED` without overwriting, and `--backup` is disabled for skipped files. Like rsync, it does not apply to directories/symlinks. Combines with `-j`/`--threads` and `--delay-updates` |
|
| `--ignore-existing` | Skip updating existing files | ✅ Parity | `ignore_existing` config field (crosses the wire; receiver-side policy). Protocol 2.26.0 short-circuits in the per-file check **before any payload**: when the destination entry already exists, the receiver answers the skip during the incremental handshake instead of letting the sender stream data that would be discarded, so an existing 4 MiB destination costs only the config/check frames (verified with a counting proxy, matching rsync). The write-time paths (regular, delay-updates-staged, hardlink-sibling, special/device) still return `FILE_SAVE_SKIPPED` without overwriting, and `--backup` is disabled for skipped files. Like rsync, it does not apply to directories/symlinks. Combines with `-j`/`--threads` and `--delay-updates` |
|
||||||
| `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Parity | |
|
| `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Parity | |
|
||||||
| `-x`, `--one-file-system` | Do not cross filesystem boundaries | ✅ Parity | Sender scanner captures the root device and does not descend into mount-point crossings (`st_dev` differs). **Protocol 2.23.0 matches rsync's entry emission:** the mount-point directory itself is emitted as a payload-less directory entry (so the destination gets an empty directory) while its contents are skipped; previously the crossing subdirectory was dropped entirely |
|
| `-x`, `--one-file-system` | Do not cross filesystem boundaries | ✅ Parity | Sender scanner captures the root device and does not descend into mount-point crossings (`st_dev` differs). **Protocol 2.23.0 matches rsync's entry emission:** the mount-point directory itself is emitted as a payload-less directory entry (so the destination gets an empty directory) while its contents are skipped; previously the crossing subdirectory was dropped entirely |
|
||||||
| `-F` | Add the default `.rsync-filter` rules | ⚠️ Caveat | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (first match wins). **A single `-F` transfers the `.rsync-filter` files themselves, matching rsync; a repeated `-FF` additionally excludes them** (rsync 3.4.1's `-F`/`-FF` are exactly these two rules, with no `.cvsignore` branch). Unsupported/unparseable rules inside a per-directory file fail the scan with a clear error. **Residual (track 4a):** per-directory rules are still enforced receiver-side only through the sender-derived source-mirror protected prefixes; the base-rule receiver filter engine does not carry per-directory rules, so a destination-only entry matching ONLY a `.rsync-filter` rule is not yet shielded from `--delete` |
|
| `-F` | Add the default `.rsync-filter` rules | ⚠️ Caveat | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (first match wins). **A single `-F` transfers the `.rsync-filter` files themselves, matching rsync; a repeated `-FF` additionally excludes them** (rsync 3.4.1's `-F`/`-FF` are exactly these two rules, with no `.cvsignore` branch). Unsupported/unparseable rules inside a per-directory file fail the scan with a clear error. The merge-only `e`/`n`/`w`/`-` modifiers on a `dir-merge` registration are implemented (`e` excludes the merge file, `n` stops inheritance into subdirectories, `w` word-splits, `-` reads bare patterns). **Per-directory rules (protocol 2.30.0, issue #315) are now carried to the receiver:** the sender streams each traversed directory's compiled rules (owner + no-inherit flag, bounded and validated) on the delete carrier and the receiver applies them deepest-directory-first before the command-line base, so a destination-only entry matching ONLY a `.rsync-filter` rule is shielded from `--delete` under every timing, including the `-n` would-delete enumeration (`filter_perdir_protect*` differential and `TestFilterProtect::test_perdir_protect_*`). **Residual:** rsync's receiver scans the DESTINATION's `.rsync-filter` files while FastSync carries the source's compiled rules, so a rule present only in a destination-side `.rsync-filter` is honored by rsync but not by FastSync |
|
||||||
|
|
||||||
## 4. Directory Options
|
## 4. Directory Options
|
||||||
|
|
||||||
@@ -161,7 +163,7 @@ Every one of those has an entry below with its remaining caveats.
|
|||||||
| `--no-implied-dirs` | Don't send implied dirs with -R | ✅ Parity | With `-R`, rsync creates the ancestor directories implied by a listed path and, with `--no-implied-dirs`, omits their attributes from the transfer so they keep the destination's own state (or are created with default attributes when absent). Protocol 2.26.0 matches this: without `--files-from` the implied-dir walk applies per-attribute metadata only to explicitly transferred directories, and with `-R --files-from` a listed file whose parent is not itself listed is placed normally — the missing implied parent is created with default attributes (not the source's) and the file transfers with `rc 0`, exactly like rsync 3.4.1 (a differential test verifies the modes and mtimes with and without the flag). Works single-threaded and under `-j`/`--threads` |
|
| `--no-implied-dirs` | Don't send implied dirs with -R | ✅ Parity | With `-R`, rsync creates the ancestor directories implied by a listed path and, with `--no-implied-dirs`, omits their attributes from the transfer so they keep the destination's own state (or are created with default attributes when absent). Protocol 2.26.0 matches this: without `--files-from` the implied-dir walk applies per-attribute metadata only to explicitly transferred directories, and with `-R --files-from` a listed file whose parent is not itself listed is placed normally — the missing implied parent is created with default attributes (not the source's) and the file transfers with `rc 0`, exactly like rsync 3.4.1 (a differential test verifies the modes and mtimes with and without the flag). Works single-threaded and under `-j`/`--threads` |
|
||||||
| `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ✅ Parity | Protocol 2.26.0 implements rsync's one-level `-d` listing for `dir`, `dir/` and `.`: the source's immediate contents are transferred (files with content, directories as explicit entries), matching rsync's destination tree in a differential test. `--dirs --files-from` transfers exactly the listed items — a listed directory is created empty and a listed file with content — under the same `-R` layout rules. A plain recursive scan also recreates empty source directories now: the scanner emits a payload-less directory entry (with metadata) for every traversed directory that produced no transferred or descended child, unless `-m/--prune-empty-dirs` suppresses it or the run is `--files-from`/`--list-only` (a directory emptied by filtering is recreated too, matching rsync). Directory entries cross as `STATUS_MKDIR` and appear in the delete manifest, so `--delete` prunes correctly and an empty listed directory survives; an incoming directory replaces a destination regular file (rsync removes the non-directory and creates the directory), verified differentially. Directory times are applied at the end of the transfer; modes/ownership follow the per-attribute policy. Under `--delay-updates` directories are created immediately while only regular files are staged, exactly as rsync does |
|
| `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ✅ Parity | Protocol 2.26.0 implements rsync's one-level `-d` listing for `dir`, `dir/` and `.`: the source's immediate contents are transferred (files with content, directories as explicit entries), matching rsync's destination tree in a differential test. `--dirs --files-from` transfers exactly the listed items — a listed directory is created empty and a listed file with content — under the same `-R` layout rules. A plain recursive scan also recreates empty source directories now: the scanner emits a payload-less directory entry (with metadata) for every traversed directory that produced no transferred or descended child, unless `-m/--prune-empty-dirs` suppresses it or the run is `--files-from`/`--list-only` (a directory emptied by filtering is recreated too, matching rsync). Directory entries cross as `STATUS_MKDIR` and appear in the delete manifest, so `--delete` prunes correctly and an empty listed directory survives; an incoming directory replaces a destination regular file (rsync removes the non-directory and creates the directory), verified differentially. Directory times are applied at the end of the transfer; modes/ownership follow the per-attribute policy. Under `--delay-updates` directories are created immediately while only regular files are staged, exactly as rsync does |
|
||||||
| `--mkpath` | Create missing path components | ✅ Parity | Wire option (client → server). At connection start the server creates the client's destination root directory (and any missing leading components below its own authorized root) when `--mkpath` is set, failing the connection cleanly if it cannot. Without `--mkpath` a destination root that does not exist yet is rejected up front (rsync semantics), so the flag is the only way to transfer into a not-yet-created destination directory. Creation is confined by the same secure mkdir walk as file writes (`O_NOFOLLOW`, no `..`) |
|
| `--mkpath` | Create missing path components | ✅ Parity | Wire option (client → server). At connection start the server creates the client's destination root directory (and any missing leading components below its own authorized root) when `--mkpath` is set, failing the connection cleanly if it cannot. Without `--mkpath` a destination root that does not exist yet is rejected up front (rsync semantics), so the flag is the only way to transfer into a not-yet-created destination directory. Creation is confined by the same secure mkdir walk as file writes (`O_NOFOLLOW`, no `..`) |
|
||||||
| `--inc-recursive`, `--no-inc-recursive` | Incremental recursion mode | ❌ Divergent | rsync's man-page-only scanning-mode switch (and its short aliases). FastSync always performs a single full recursive scan, so both spellings are rejected as unknown options rather than accepted as a no-op; there is no incremental-recursion engine to toggle. A genuine implementation would be a scan-architecture change with no benefit for FastSync's push model |
|
| `--inc-recursive`, `--no-inc-recursive` | Incremental recursion mode | ✅ Parity | rsync's man-page-only scanning-mode switch. FastSync always performs a single full recursive scan, so both spellings are accepted as inert no-ops and the destination is identical whichever mode the caller requests — the same treatment as `-r`/`--recursive`, which is likewise a no-op. The switch is a scan-implementation detail with no observable effect on the final tree (rsync's own `--no-inc-recursive` selects a full scan, which is exactly FastSync's behavior) |
|
||||||
|
|
||||||
## 5. Transfer Modifications
|
## 5. Transfer Modifications
|
||||||
|
|
||||||
@@ -183,8 +185,8 @@ Every one of those has an entry below with its remaining caveats.
|
|||||||
| `-b`, `--backup` | Make backups of overwritten files | ✅ Parity | Backup before overwrite |
|
| `-b`, `--backup` | Make backups of overwritten files | ✅ Parity | Backup before overwrite |
|
||||||
| `--backup-dir=DIR` | Backup directory hierarchy | ✅ Parity | `backup_dir` config field |
|
| `--backup-dir=DIR` | Backup directory hierarchy | ✅ Parity | `backup_dir` config field |
|
||||||
| `--suffix=SUFFIX` | Backup suffix (default ~) | ✅ Parity | `suffix` config field |
|
| `--suffix=SUFFIX` | Backup suffix (default ~) | ✅ Parity | `suffix` config field |
|
||||||
| `--delay-updates` | Put updated files in place at end | ❌ Divergent | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication, and **`--force` is honored at publication** (protocol 2.23.0): a staged regular file or symlink may replace a destination directory that blocks it. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). **Reclassified Divergent (differential evidence):** the staging name is fixed and a delayed run wipes a pre-existing destination tree of that name at start even without `--delete`, whereas rsync uses its own internal temp name and leaves a genuine destination entry named `.fastsync-stage` untouched (`test_delay_updates_staging_name_collision_residual`); deletion also runs before publication while rsync's `--delay-updates` implies `--delete-after`. Works in single-threaded and `-j`/`--threads` modes |
|
| `--delay-updates` | Put updated files in place at end | ⚠️ Caveat | Successfully received files are staged under a private 0700 `.fastsync-stage.<pid>.<entropy>` directory inside the receive root and atomically renamed into their final destinations only after the whole transfer succeeds. The name is **per-run unique**, and prepare creates it with O_EXCL semantics: a genuine destination entry that happens to share the exact name is never wiped (the run refuses), matching rsync's own internal temp name (a pre-existing entry named like the staging prefix survives; differential `test_delay_updates_staging_name_collision_preserved`). The delete walker skips this transfer's runtime staging name at the receive root. **`--delay-updates` implies `--delete-after`** (client-normalized onto the existing `delete_after` wire bool; no new field): both the single-threaded and `--threads` receivers publish every staged file first and commit the deferred deletion afterwards, so extras are removed only after all updates land. A `--backup` file is shielded from that delete pass (rsync never treats a backup as an extra), and a destination directory blocking a staged regular file/symlink is cleared at publication when `--delete` or `--force` is active (rsync's generator make-way); without either, a non-empty blocker fails the run. `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging prefix is reserved; both are rejected). Aborting or failing before publication installs nothing and removes the per-run staging tree; a stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). **Reclassified ⚠️ Caveat (#317):** the fixed-name wipe and the delete-before-publication ordering are both gone; the remaining documented differences are that FastSync does not create a backup of a *deleted* extra the way rsync's `--backup --delete` does, and that a crash-leftover staging directory is never reused (each run picks a fresh name, so leftovers are not auto-cleaned). Works in single-threaded and `-j`/`--threads` modes |
|
||||||
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ❌ Divergent | `--temp-dir` with the rsync short `-T` (the timeout alias moved to long-only `--timeout`). A **relative** dir matches rsync exactly: it is resolved below the receive/destination root and must already exist (differentially verified: `rsync -a --temp-dir=scratch src/ dst/` and FastSync produce identical trees and an empty scratch dir). **Reclassified as a deliberate divergence because an absolute `--temp-dir` is rejected by the receiver** — it is resolved verbatim by rsync standalone (which will use `/tmp` or any other absolute directory, including one outside the destination), but FastSync's security-reviewed receiver confines the scratch dir to the authorized receive root and rejects any absolute path or one containing `..`. **Audit-cycle hardening:** the opened dir is additionally judged by the real path of its fd (`/proc/self/fd`), so a client-planted symlink under the receive root cannot redirect receiver scratch files outside the authorized root (an escaping target is refused with `EACCES`), while an in-root symlink to another filesystem — the `EXDEV` fallback case — still works. A differential test confirms rsync exits 0 using an absolute scratch dir while FastSync refuses before writing anything into it (the scratch dir stays empty). Its daemon mode also confines relative to the module, but standalone rsync's absolute-temp-dir behavior is not reproduced because it would let a client place receiver scratch files outside the sandbox. Temp copies use a unique name in the scratch dir and are atomically renamed into place; **on `EXDEV` (scratch dir and destination on different filesystems, reachable via a confined relative symlink) the receiver falls back to a non-atomic copy instead of aborting**, matching rsync. `--inplace` and `--partial-dir` writes bypass the scratch dir |
|
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ❌ Divergent | `--temp-dir` with the rsync short `-T` (the timeout alias moved to long-only `--timeout`). A **relative** dir matches rsync exactly: it is resolved below the receive/destination root and must already exist (differentially verified: `rsync -a --temp-dir=scratch src/ dst/` and FastSync produce identical trees and an empty scratch dir). An **absolute** dir is accepted when it canonicalizes (`realpath(3)`) inside the receive root, so an in-root absolute scratch path is usable and used (unit- and integration-tested for both the local batch apply and a live TCP transfer). **Remaining divergence:** an absolute `--temp-dir` that escapes the receive root is rejected, and so is a relative one containing `..` — rsync standalone resolves an absolute `--temp-dir` verbatim (it will use `/tmp` or any other directory, including one outside the destination), but FastSync's security-reviewed receiver confines the scratch dir to the authorized receive root and refuses an out-of-root path before writing anything. **Audit-cycle hardening:** the opened dir is additionally judged by the real path of its fd (`/proc/self/fd`), so a client-planted symlink under the receive root cannot redirect receiver scratch files outside the authorized root (an escaping target is refused with `EACCES`), while an in-root symlink to another filesystem — the `EXDEV` fallback case — still works. A differential test confirms rsync exits 0 using an out-of-root absolute scratch dir while FastSync refuses before writing anything into it (the scratch dir stays empty). Its daemon mode also confines relative to the module. Temp copies use a unique name in the scratch dir and are atomically renamed into place; **on `EXDEV` (scratch dir and destination on different filesystems, reachable via a confined relative symlink) the receiver falls back to a non-atomic copy instead of aborting**, matching rsync. `--inplace` and `--partial-dir` writes bypass the scratch dir |
|
||||||
| `--partial` | Keep partially transferred files | ✅ Parity | On a failed/interrupted write the already-written temp file is retained at the destination path (best-effort rename instead of unlink) so a later `--append`/`--append-verify` run can resume it. Retention never runs when no data was actually written or under `--ignore-existing`/`--existing` (the destination is not ours to overwrite), and it only ever renames the already-written temp. A failed rename falls back to the normal unlink |
|
| `--partial` | Keep partially transferred files | ✅ Parity | On a failed/interrupted write the already-written temp file is retained at the destination path (best-effort rename instead of unlink) so a later `--append`/`--append-verify` run can resume it. Retention never runs when no data was actually written or under `--ignore-existing`/`--existing` (the destination is not ours to overwrite), and it only ever renames the already-written temp. A failed rename falls back to the normal unlink |
|
||||||
| `--partial-dir=DIR` | Keep partial files in DIR | ✅ Parity | The working file is written under the confined partial directory (a relative dir below the receive root) and atomically renamed into place once complete, so an interrupted transfer leaves a resumable copy there and completed transfers do not linger under it. `--inplace` bypasses the partial dir (rsync parity), and combining `--inplace` with `--partial-dir` is now **rejected up front** with rsync's message (`--inplace cannot be used with --partial-dir`) instead of silently ignoring the partial dir. **Implies `--partial`** (audit-cycle fix, matching rsync 3.4.1, which sets `keep_partial` after option parsing): `--partial-dir=DIR` alone retains an interrupted transfer's partial, and the implication wins over an explicit `--no-partial` regardless of order |
|
| `--partial-dir=DIR` | Keep partial files in DIR | ✅ Parity | The working file is written under the confined partial directory (a relative dir below the receive root) and atomically renamed into place once complete, so an interrupted transfer leaves a resumable copy there and completed transfers do not linger under it. `--inplace` bypasses the partial dir (rsync parity), and combining `--inplace` with `--partial-dir` is now **rejected up front** with rsync's message (`--inplace cannot be used with --partial-dir`) instead of silently ignoring the partial dir. **Implies `--partial`** (audit-cycle fix, matching rsync 3.4.1, which sets `keep_partial` after option parsing): `--partial-dir=DIR` alone retains an interrupted transfer's partial, and the implication wins over an explicit `--no-partial` regardless of order |
|
||||||
|
|
||||||
@@ -192,8 +194,8 @@ Every one of those has an entry below with its remaining caveats.
|
|||||||
|
|
||||||
| Flag | Rsync Description | FastSync Status | Notes |
|
| Flag | Rsync Description | FastSync Status | Notes |
|
||||||
|------|-------------------|-----------------|-------|
|
|------|-------------------|-----------------|-------|
|
||||||
| `--delete` | Delete extraneous files from dest | ✅ Parity | `use_delete` config field. Deletion is always derived from the keep-set the sender actually transmitted (the per-directory `STATUS_DELETE_PLAN` set by default, or the whole-tree manifest for the late timings — never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. **Lockstep track 6 (protocol 2.28.0): plain `--delete` with no explicit timing flag now defaults to `--delete-during`**, exactly like rsync's `--del` (the client normalizes it to the existing `delete_during` wire bool; no new wire field). This frees destination space progressively during the transfer and avoids the whole-old+new-tree peak that could `ENOSPC` a tight destination. The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`. **Abort/ordering parity (parity-2.29):** the complete per-directory plan set is transmitted before the first data frame, so a mid-transfer abort has already applied every planned removal exactly like rsync's generator (which runs ahead of its throttled sender); `-d/--dirs` uses the same per-directory plans (the generator records only the directories whose direct children it enumerated, so an untraversed subdirectory's mirror is shielded); and the sorted depth-first traversal makes the removal order — and therefore the survivor set under a partial `--max-delete` — match rsync exactly (`test_delete_boundary_parity.py`, `test_parity_order.py`). By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent on the wire (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing |
|
| `--delete` | Delete extraneous files from dest | ✅ Parity | `use_delete` config field. Deletion is always derived from the keep-set the sender actually transmitted (the per-directory `STATUS_DELETE_PLAN` set by default, or the whole-tree manifest for the late timings — never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the runtime `--delay-updates` staging directory. **Lockstep track 6 (protocol 2.28.0): plain `--delete` with no explicit timing flag now defaults to `--delete-during`**, exactly like rsync's `--del` (the client normalizes it to the existing `delete_during` wire bool; no new wire field). This frees destination space progressively during the transfer and avoids the whole-old+new-tree peak that could `ENOSPC` a tight destination. The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`. **Abort/ordering parity (parity-2.29):** the complete per-directory plan set is transmitted before the first data frame, so a mid-transfer abort has already applied every planned removal exactly like rsync's generator (which runs ahead of its throttled sender); `-d/--dirs` uses the same per-directory plans (the generator records only the directories whose direct children it enumerated, so an untraversed subdirectory's mirror is shielded); and the sorted depth-first traversal makes the removal order — and therefore the survivor set under a partial `--max-delete` — match rsync exactly (`test_delete_boundary_parity.py`, `test_parity_order.py`). By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent on the wire (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing |
|
||||||
| `--delete-before` | Delete before transfer | ⚠️ Caveat | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. **Phase-0 divergence (sharpened):** rsync builds the full file list first, so a source file created after that scan is NOT transferred and its destination extra is deleted; FastSync's single-threaded data pass re-scans the source, so the late file IS transferred (a safe superset), while FastSync `--threads` pipelines the scan and matches rsync |
|
| `--delete-before` | Delete before transfer | ✅ Parity | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. **Phase-0 divergence closed (no-wire):** both data passes now replay the exact file list the pre-scan built for the keep-set instead of re-reading the source — the single-threaded send loop and the `--threads` pipeline (whose scanner thread feeds the retained pre-scan chunks into the pipeline rather than re-scanning) — so a source file created after that scan is NOT transferred and its destination extra is deleted, exactly like rsync's single file list. The pre-scan captures the deferred directory times and the `--stats` directory count because no later scan runs (`test_delete_timing_parity.py::TestDeleteBeforeLateFileParity`, parametrized single-threaded vs `--threads=4`, differential vs rsync 3.4.1) |
|
||||||
| `--del`, `--delete-during` | Delete during transfer | ✅ Parity | Both spellings accepted; imply `--delete`, and since lockstep track 6 this is also the default timing of a plain `--delete`. **Protocol 2.24.0 implements per-directory delete plans:** as the sender reaches each source directory it streams a `STATUS_DELETE_PLAN` for that directory and the receiver removes that directory's extras (verified with a byte-slicing proxy). The one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) rides a dedicated config-only carrier frame with an `apply=false` flag, so it reaches the receiver even when the scope allows no directory plan at all (a `--files-from` list of bare files synchronizes no directory). **Abort/ordering parity (parity-2.29):** the complete plan set is transmitted before the first data frame, so on a mid-transfer abort every planned extra has already been removed exactly like rsync's generator (which runs ahead of its throttled sender); `-d/--dirs` no longer falls back to the end-of-transfer commit but records only the directories whose direct children it enumerated; and the sorted depth-first traversal makes the removal order — and the partial-`--max-delete` survivor set — identical to rsync (`test_delete_boundary_parity.py`, `test_parity_order.py`). `-R` plans are scoped to the transferred prefix subtree |
|
| `--del`, `--delete-during` | Delete during transfer | ✅ Parity | Both spellings accepted; imply `--delete`, and since lockstep track 6 this is also the default timing of a plain `--delete`. **Protocol 2.24.0 implements per-directory delete plans:** as the sender reaches each source directory it streams a `STATUS_DELETE_PLAN` for that directory and the receiver removes that directory's extras (verified with a byte-slicing proxy). The one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) rides a dedicated config-only carrier frame with an `apply=false` flag, so it reaches the receiver even when the scope allows no directory plan at all (a `--files-from` list of bare files synchronizes no directory). **Abort/ordering parity (parity-2.29):** the complete plan set is transmitted before the first data frame, so on a mid-transfer abort every planned extra has already been removed exactly like rsync's generator (which runs ahead of its throttled sender); `-d/--dirs` no longer falls back to the end-of-transfer commit but records only the directories whose direct children it enumerated; and the sorted depth-first traversal makes the removal order — and the partial-`--max-delete` survivor set — identical to rsync (`test_delete_boundary_parity.py`, `test_parity_order.py`). `-R` plans are scoped to the transferred prefix subtree |
|
||||||
| `--delete-delay` | Find deletions during, delete after | ✅ Parity | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The **reported** deleted count advances only on an actual removal. **Fixed (no-wire):** the `--max-delete` budget is now charged on ACTUAL removals (an unlink/rmdir that succeeded), not at plan/snapshot time, and a queued directory is re-scanned at commit and removed recursively (content created after the plan included), matching rsync: a snapshotted entry that fails or is skipped consumes no budget, so a later extra rsync would delete is still deleted. The deferred snapshot list keeps an independent hard cap (`DELETE_PLAN_SERVER_LIMIT`) so it cannot grow without bound now that the budget is no longer charged while scanning. A `--max-delete=2` partial delete reports exactly 2 and exits 25 in both tools, and the refilled-directory differential (late content removed, directory removed, budget shared) now matches rsync 3.4.1 on both sides (`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`). Unit tests cover recursive removal, actual-removal charging, and the bounded deferred list. **Ordering parity (parity-2.29):** the sorted depth-first traversal plus the up-front plan set make the order in which extras are removed — and therefore the survivor set under a partial `--max-delete` — match rsync exactly (differential `test_parity_order.py::test_delete_delay_deletion_order_matches_rsync` and `::test_partial_max_delete_survivor_order_matches_rsync`) |
|
| `--delete-delay` | Find deletions during, delete after | ✅ Parity | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The **reported** deleted count advances only on an actual removal. **Fixed (no-wire):** the `--max-delete` budget is now charged on ACTUAL removals (an unlink/rmdir that succeeded), not at plan/snapshot time, and a queued directory is re-scanned at commit and removed recursively (content created after the plan included), matching rsync: a snapshotted entry that fails or is skipped consumes no budget, so a later extra rsync would delete is still deleted. The deferred snapshot list keeps an independent hard cap (`DELETE_PLAN_SERVER_LIMIT`) so it cannot grow without bound now that the budget is no longer charged while scanning. A `--max-delete=2` partial delete reports exactly 2 and exits 25 in both tools, and the refilled-directory differential (late content removed, directory removed, budget shared) now matches rsync 3.4.1 on both sides (`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`). Unit tests cover recursive removal, actual-removal charging, and the bounded deferred list. **Ordering parity (parity-2.29):** the sorted depth-first traversal plus the up-front plan set make the order in which extras are removed — and therefore the survivor set under a partial `--max-delete` — match rsync exactly (differential `test_parity_order.py::test_delete_delay_deletion_order_matches_rsync` and `::test_partial_max_delete_survivor_order_matches_rsync`) |
|
||||||
| `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. Selects the late whole-tree commit: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing. Since lockstep track 6 a plain `--delete` defaults to delete-during (rsync's `--del`); `--delete-after` — or the FastSync-only `--delete-commit` spelling, which selects the identical timing — is the explicit way to keep the old commit-style behavior |
|
| `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. Selects the late whole-tree commit: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing. Since lockstep track 6 a plain `--delete` defaults to delete-during (rsync's `--del`); `--delete-after` — or the FastSync-only `--delete-commit` spelling, which selects the identical timing — is the explicit way to keep the old commit-style behavior |
|
||||||
@@ -339,10 +341,10 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
|
|||||||
| `-E`, `--executability` | Preserve executability | ✅ Parity | Preserves executable permission bits (implies metadata preservation) |
|
| `-E`, `--executability` | Preserve executability | ✅ Parity | Preserves executable permission bits (implies metadata preservation) |
|
||||||
| `--chmod=CHMOD` | Affect file permissions | ✅ Parity | Faithful port of rsync 3.4.1's `parse_chmod`/`tweak_mode`: numeric octal and symbolic `ugo`/`rwx` changes, `D`/`F` directory/file selectors, `X` (execute only on directories or already-executable files), `s`/`t` setuid/setgid/sticky, and append semantics — repeated clauses and repeated `--chmod` options accumulate in order (joined with commas). The changes are applied to the new mode **without sanitization** (matching rsync), except that setuid/setgid/sticky are masked when the connection forbids super-user activities (audit-cycle fix, see `-p`), and `--chmod` does **not** imply `-p` (rsync parity). Applied to files and directories on the receiver |
|
| `--chmod=CHMOD` | Affect file permissions | ✅ Parity | Faithful port of rsync 3.4.1's `parse_chmod`/`tweak_mode`: numeric octal and symbolic `ugo`/`rwx` changes, `D`/`F` directory/file selectors, `X` (execute only on directories or already-executable files), `s`/`t` setuid/setgid/sticky, and append semantics — repeated clauses and repeated `--chmod` options accumulate in order (joined with commas). The changes are applied to the new mode **without sanitization** (matching rsync), except that setuid/setgid/sticky are masked when the connection forbids super-user activities (audit-cycle fix, see `-p`), and `--chmod` does **not** imply `-p` (rsync parity). Applied to files and directories on the receiver |
|
||||||
| `-A`, `--acls` | Preserve ACLs | ✅ Parity | Implemented on Linux via the POSIX-ACL xattr representation: the sender captures the `system.posix_acl_access` / `system.posix_acl_default` xattrs and the receiver re-applies them fd-relative. A differential test with `setfacl` confirms the complete access and default ACL sets (including `mask`) are identical to rsync's on a directory. libacl is not required; a `fsetxattr` an unprivileged receiver may not perform is logged and skipped, never fatal. Only the `system.posix_acl_*` namespaces plus `user.*` are ever applied; privileged namespaces are never applied. Implies metadata transmission |
|
| `-A`, `--acls` | Preserve ACLs | ✅ Parity | Implemented on Linux via the POSIX-ACL xattr representation: the sender captures the `system.posix_acl_access` / `system.posix_acl_default` xattrs and the receiver re-applies them fd-relative. A differential test with `setfacl` confirms the complete access and default ACL sets (including `mask`) are identical to rsync's on a directory. libacl is not required; a `fsetxattr` an unprivileged receiver may not perform is logged and skipped, never fatal. Only the `system.posix_acl_*` namespaces plus `user.*` are ever applied; privileged namespaces are never applied. Implies metadata transmission |
|
||||||
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Divergent | Deliberately restricted to unprivileged `user.*` extended attributes plus the two POSIX ACL xattrs; `security.*` (SELinux, capabilities, ...) and `trusted.*` are **never** captured or applied — a client can never force a privileged attribute onto the destination, and the receiver independently re-validates every incoming name against the whitelist. This is a security-policy divergence from rsync, which can preserve the privileged namespaces with the needed privilege; implementing them would defeat FastSync's privilege-escalation guard. `user.*` capture/apply matches rsync in a differential test. Payloads are bounded on both ends. Incompatible with `-s`. **Also divergent: symlink xattrs/ACLs are not captured or applied** — `-X`/`-A` with `-l` carries only the link's owner/times/mode, not its xattrs (the capture uses path-following `listxattr`/`getxattr`, so the link's own xattrs are never read, and the receiver's symlink write path applies no xattr block). Closing this needs a dedicated symlink-xattr wire block and a `PROTOCOL_VERSION` bump |
|
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Divergent | Deliberately restricted to unprivileged `user.*` extended attributes plus the two POSIX ACL xattrs; `security.*` (SELinux, capabilities, ...) and `trusted.*` are **never** captured or applied — a client can never force a privileged attribute onto the destination, and the receiver independently re-validates every incoming name against the whitelist. This is a security-policy divergence from rsync, which can preserve the privileged namespaces with the needed privilege; implementing them would defeat FastSync's privilege-escalation guard. `user.*` capture/apply matches rsync in a differential test. Payloads are bounded on both ends. Incompatible with `-s`. **Symlink xattrs are now carried (protocol 2.29.0):** a symlink entry appends the same bounded trailing xattr block to its `STATUS_SYMLINK` frame as every other entry kind, captured with `llistxattr`/`lgetxattr` so the link's OWN attributes are read and never the referent's, and re-applied no-follow with `lsetxattr` through the already-confined parent directory (`fsetxattr` cannot target a symlink: there is no `*at` xattr syscall and an `O_PATH` fd is rejected). On Linux the VFS refuses to associate xattrs with a symlink at all — every `lsetxattr` on a link fails with `EPERM` for `user.*`, `trusted.*` and `security.*`, even as root, verified in the CI container — so on FastSync's supported platforms the captured block is always empty and the apply is a no-op; the wire block is present for correctness and for a filesystem/platform that does support symlink xattrs. rsync 3.4.1's `--fake-super` is not a counterexample: it stores a symlink as a regular file whose `user.rsync.%stat` records the `S_IFLNK` mode bits, not an xattr on a real symlink. The row stays divergent only for the never-preserved privileged namespaces above |
|
||||||
| `-H`, `--hard-links` | Preserve hard links | ✅ Parity | Files on the source that share an inode (`st_dev`+`st_ino`, e.g. a `cp -al` tree) are re-created as hard links to one another on the destination, so duplicate links stay deduplicated and only the first member's data is sent (later members are transmitted as payload-less `STATUS_HARDLINK` frames). The receiver links each sibling to the first member's installed file with an atomic link + rename; on `link()` failure it falls back to a byte-identical local copy of the first member, never a partial/corrupt file. Requires the sequential scan for ordering (the first member is always emitted and installed before any sibling is linked). Works single-threaded and under `-j`/`--threads`, `--inplace`, `--delay-updates` (links staged and published by rename) and `--partial`. Crosses the wire (`preserve_hard_links` bool; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0**, peers must match). Incompatible with `-s` (chunk serialization) and `--append`/`--append-verify`, rejected up front with a distinct error. See the Phase-4 hard-links notes below |
|
| `-H`, `--hard-links` | Preserve hard links | ✅ Parity | Files on the source that share an inode (`st_dev`+`st_ino`, e.g. a `cp -al` tree) are re-created as hard links to one another on the destination, so duplicate links stay deduplicated and only the first member's data is sent (later members are transmitted as payload-less `STATUS_HARDLINK` frames). The receiver links each sibling to the first member's installed file with an atomic link + rename; on `link()` failure it falls back to a byte-identical local copy of the first member, never a partial/corrupt file. Requires the sequential scan for ordering (the first member is always emitted and installed before any sibling is linked). Works single-threaded and under `-j`/`--threads`, `--inplace`, `--delay-updates` (links staged and published by rename) and `--partial`. Crosses the wire (`preserve_hard_links` bool; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0**, peers must match). Incompatible with `-s` (chunk serialization) and `--append`/`--append-verify`, rejected up front with a distinct error. See the Phase-4 hard-links notes below |
|
||||||
| `-D` | Same as --devices --specials | ✅ Parity | Implies `--devices --specials`. `-D` was unassigned in FastSync (verified: no collision), so it is free to imply both device-node and special-file preservation. As of protocol 2.23.0 `--specials` genuinely covers **both FIFOs and unix sockets**, so `-D` covers the full rsync set. See the `--devices`/`--specials` rows and the Phase-4 devices notes below |
|
| `-D` | Same as --devices --specials | ✅ Parity | Implies `--devices --specials`. `-D` was unassigned in FastSync (verified: no collision), so it is free to imply both device-node and special-file preservation. As of protocol 2.23.0 `--specials` genuinely covers **both FIFOs and unix sockets**, so `-D` covers the full rsync set. See the `--devices`/`--specials` rows and the Phase-4 devices notes below |
|
||||||
| `--devices` | Preserve device files | ❌ Divergent | Recreates char/block device nodes with `mknodat` (type + rdev strictly validated, confined fd-relative below the receive root), but only when the receiver has `CAP_MKNOD`: a non-root receiver logs a warning and skips the entry instead of erroring, so a transfer with devices never aborts. Deliberate privilege-model divergence from rsync, which errors when it cannot create the node. `--specials` (FIFOs and unix sockets) is unprivileged and remains parity |
|
| `--devices` | Preserve device files | ⚠️ Caveat | Recreates char/block device nodes with `mknodat` (type + rdev strictly validated, confined fd-relative below the receive root). A device whose `mknodat` fails with `EPERM`/`EACCES` (no `CAP_MKNOD`, or super-user activity forbidden) is a **per-entry failure**: FastSync logs `cannot create device ...` (rsync logs `mknod ... failed`), counts it, **continues with the remaining files**, and ends the run with a partial terminal status (`STATUS_PARTIAL`, protocol 2.30.0) so the client exits 23 like rsync, and under `--remove-source-files` the successfully transferred sources are still removed. rsync parity: rsync likewise continues and exits partial (23). Residual: FastSync's default AUTO still *attempts* the node on a non-root receiver and therefore reports the per-entry failure, whereas rsync without `--super` silently ignores `--devices` and skips the non-regular entry with exit 0 — use `--no-super` for rsync's silent-skip behavior. `--specials` (FIFOs and unix sockets) keeps the unprivileged skip path and remains parity |
|
||||||
| `--specials` | Preserve special files | ✅ Parity | **FIFO and unix-socket recreation work** (protocol 2.23.0): FIFOs are recreated with `mkfifoat`, and sockets with `mknodat(..., S_IFSOCK)` — the latter is unprivileged on Linux because it materializes the socket *node*, not a live bound socket, so it is a real, assertable behavior under CI (it matches rsync, which also recreates a socket by `mknod`). Node creation is confined below the receive root (fd-relative parent; no `..`, no symlink follow) and type/rdev are validated strictly; a matching existing node is left in place and an unrelated entry is never replaced. Crosses the wire like `--devices` (the `STATUS_SPECIAL` frame). See the Phase-4 devices notes |
|
| `--specials` | Preserve special files | ✅ Parity | **FIFO and unix-socket recreation work** (protocol 2.23.0): FIFOs are recreated with `mkfifoat`, and sockets with `mknodat(..., S_IFSOCK)` — the latter is unprivileged on Linux because it materializes the socket *node*, not a live bound socket, so it is a real, assertable behavior under CI (it matches rsync, which also recreates a socket by `mknod`). Node creation is confined below the receive root (fd-relative parent; no `..`, no symlink follow) and type/rdev are validated strictly; a matching existing node is left in place and an unrelated entry is never replaced. Crosses the wire like `--devices` (the `STATUS_SPECIAL` frame). See the Phase-4 devices notes |
|
||||||
| `--copy-devices` | Copy device contents as file | ❌ Divergent | Copies a device/FIFO's reported `st_size` into an ordinary regular file and never reads an unbounded pseudo-device, so `--sendfile` cannot hang and the run always succeeds. Deliberate safe divergence from rsync's dd-like unbounded device read, which can block; the dangerous behavior will not be implemented |
|
| `--copy-devices` | Copy device contents as file | ❌ Divergent | Copies a device/FIFO's reported `st_size` into an ordinary regular file and never reads an unbounded pseudo-device, so `--sendfile` cannot hang and the run always succeeds. Deliberate safe divergence from rsync's dd-like unbounded device read, which can block; the dangerous behavior will not be implemented |
|
||||||
| `--write-devices` | Write to devices as files | ❌ Divergent | Writes only into an existing char/block node under the confined receive root (`O_NOFOLLOW` + `O_NONBLOCK`); a missing, symlinked, FIFO-with-no-reader, non-device, or otherwise unusable destination is skipped with a warning rather than allowed or aborted. Deliberate confinement divergence from rsync's more permissive behavior |
|
| `--write-devices` | Write to devices as files | ❌ Divergent | Writes only into an existing char/block node under the confined receive root (`O_NOFOLLOW` + `O_NONBLOCK`); a missing, symlinked, FIFO-with-no-reader, non-device, or otherwise unusable destination is skipped with a warning rather than allowed or aborted. Deliberate confinement divergence from rsync's more permissive behavior |
|
||||||
@@ -351,7 +353,7 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
|
|||||||
| `-O`, `--omit-dir-times` | Omit dirs from --times | ✅ Parity | Real modifier now that FastSync preserves directory times. With metadata on, the scanner captures every traversed source directory's mtime (and atime under `-U`) and the sender transmits them in trailing `STATUS_DIR_TIMES` frame(s) **after all file data and the optional delete manifest** (chunked at the receiver's `MAX_MANIFEST_ENTRIES` per-frame cap); a dir-time entry only RECORDS metadata and never creates the directory (an empty source directory is created by the separate `STATUS_MKDIR` entry the scanner now emits, and `-m/--prune-empty-dirs` suppresses that; the trailing dir-time simply re-applies the metadata). The receiver defers applying them until its delete / `--delay-updates` publication phases have committed, so writing or removing a child never clobbers a parent directory's mtime (rsync applies directory times at the end for exactly this reason). When `-O` is set (the boolean crosses the wire) the receiver does not apply any of them; without `-O` an `-a`/`--preserve` transfer now restores directory times (reversing the old "never preserves dir times" divergence). Wire change: the terminal `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
|
| `-O`, `--omit-dir-times` | Omit dirs from --times | ✅ Parity | Real modifier now that FastSync preserves directory times. With metadata on, the scanner captures every traversed source directory's mtime (and atime under `-U`) and the sender transmits them in trailing `STATUS_DIR_TIMES` frame(s) **after all file data and the optional delete manifest** (chunked at the receiver's `MAX_MANIFEST_ENTRIES` per-frame cap); a dir-time entry only RECORDS metadata and never creates the directory (an empty source directory is created by the separate `STATUS_MKDIR` entry the scanner now emits, and `-m/--prune-empty-dirs` suppresses that; the trailing dir-time simply re-applies the metadata). The receiver defers applying them until its delete / `--delay-updates` publication phases have committed, so writing or removing a child never clobbers a parent directory's mtime (rsync applies directory times at the end for exactly this reason). When `-O` is set (the boolean crosses the wire) the receiver does not apply any of them; without `-O` an `-a`/`--preserve` transfer now restores directory times (reversing the old "never preserves dir times" divergence). Wire change: the terminal `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
|
||||||
| `-J`, `--omit-link-times` | Omit symlinks from --times | ✅ Parity | Real modifier now that FastSync preserves symlink times. Symlink entries already carried their metadata on `STATUS_SYMLINK`; the receiver now applies it with **no-follow primitives only** (`utimensat(..., AT_SYMLINK_NOFOLLOW)`, plus best-effort `fchmodat(..., AT_SYMLINK_NOFOLLOW)` and policy-gated `fchownat(..., AT_SYMLINK_NOFOLLOW)`), so the link itself is stamped without ever dereferencing it, confined fd-relative below the authorized receive root. A symlink has no children, so the times are applied immediately at creation. When `-J` is set (the boolean crosses the wire) the receiver skips the timestamps (mode/ownership are unaffected); without `-J` an `-a`/`-l` transfer restores symlink mtimes. Wire change alongside `-O`: the shared `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
|
| `-J`, `--omit-link-times` | Omit symlinks from --times | ✅ Parity | Real modifier now that FastSync preserves symlink times. Symlink entries already carried their metadata on `STATUS_SYMLINK`; the receiver now applies it with **no-follow primitives only** (`utimensat(..., AT_SYMLINK_NOFOLLOW)`, plus best-effort `fchmodat(..., AT_SYMLINK_NOFOLLOW)` and policy-gated `fchownat(..., AT_SYMLINK_NOFOLLOW)`), so the link itself is stamped without ever dereferencing it, confined fd-relative below the authorized receive root. A symlink has no children, so the times are applied immediately at creation. When `-J` is set (the boolean crosses the wire) the receiver skips the timestamps (mode/ownership are unaffected); without `-J` an `-a`/`-l` transfer restores symlink mtimes. Wire change alongside `-O`: the shared `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
|
||||||
| `--super` | Receiver attempts super-user activities | ❌ Divergent | Safe-subset privilege model. `--super` permits the receiver to attempt already-confined super-user activities (ownership application, char/block device-node creation, `--write-devices`); `--no-super` forbids them even for root; `auto` keeps the historical best-effort attempt. **FastSync never elevates** — no `setuid`/`seteuid`/`setgid` — and `--super` never bypasses the confinement floor, so it diverges from rsync's real elevation. A server `--no-super` veto forces it off for every connection; a privileged standalone listener defaults off without `--allow-super`; daemon modules opt in with `client owner = yes` |
|
| `--super` | Receiver attempts super-user activities | ❌ Divergent | Safe-subset privilege model. `--super` permits the receiver to attempt already-confined super-user activities (ownership application, char/block device-node creation, `--write-devices`); `--no-super` forbids them even for root; `auto` keeps the historical best-effort attempt. **FastSync never elevates** — no `setuid`/`seteuid`/`setgid` — and `--super` never bypasses the confinement floor, so it diverges from rsync's real elevation. A server `--no-super` veto forces it off for every connection; a privileged standalone listener defaults off without `--allow-super`; daemon modules opt in with `client owner = yes` |
|
||||||
| `--fake-super` | Store/recover privileged attrs via xattrs | ❌ Divergent | Records the resolved `uid:gid:mode:mtime_sec:mtime_nsec` in a reserved `user.fastsync.stat` xattr and immediately replays mode/times fd-relative, but **never performs a real `chown`** (the owner is recorded for a later privileged restore). The on-disk key and format are FastSync-native, not rsync's `user.rsync.%stat%`, so recordings are not interoperable with rsync — the same class as the native auth and batch formats. Implies metadata transmission; incompatible with `-s` |
|
| `--fake-super` | Store/recover privileged attrs via xattrs | ⚠️ Caveat | Writes rsync 3.4.1's reserved `user.rsync.%stat` xattr with rsync's exact value grammar `<octal st_mode with S_IFMT> <rdev_major>,<rdev_minor> <uid>:<gid>` (e.g. `104711 0,0 1234:5678`), recording the RESOLVED owner (the `--chown`/`--usermap`/`--groupmap`/`--copy-as` mapping when active, else the source's own id) plus the full mode and rdev; it **never performs a real `chown`**. mtime is carried by the file's own timestamp, exactly as rsync does it (there is no mtime field). The receiver parses the same grammar and replays the permission bits fd-relative, stripping the recorded special bits on disk exactly like rsync's fake-super receiver. Regular files are interoperable with real rsync 3.4.1 in both directions (the differential test has rsync read a FastSync fake-super tree and re-emit the identical record). Char/block devices **are** faked: a device is written as a regular empty file and its `user.rsync.%stat` records the real `rdev` (e.g. `20644 1,3 0:0`), never `mknod`'d, on both privileged and unprivileged receivers, exactly as rsync does. Directories **are** faked too: the directory's full stat (with `S_IFDIR` and any special bits) is parked on the directory ITSELF when it is created (explicit `--dirs`/`STATUS_MKDIR`) and again in the deferred directory-metadata pass that runs after every child is written, and the receiver replays only the permission bits on disk (the setgid/sticky bits stay in the record). Real rsync 3.4.1 reads a FastSync directory record and re-emits it verbatim (differential-tested). The record parser range-checks every field (mode/rdev/uid/gid) and rejects malformed records cleanly. Residual: symlinks are not faked — FastSync creates real symlinks, whereas rsync writes a regular file carrying an `S_IFLNK` (`120777`) `%stat` record; and FastSync writes a directory record unconditionally, where rsync omits it when the on-disk mode already fully represents the source (a benign extra xattr, still read correctly by rsync). Implies metadata transmission; incompatible with `-s` |
|
||||||
| `--open-noatime` | Avoid changing access time when opening files | ✅ Parity | Sender-side policy: the sender opens source files with `O_NOATIME` (Linux) when reading them for transfer, so the open/read does NOT bump the source's on-disk access time. Degrades safely when `O_NOATIME` is unavailable (not defined) or refused (`EPERM`, since it needs `CAP_FOWNER` or file ownership): the code falls back to a normal open, so the data always transfers — only the atime-bump is skipped. It does not itself capture/preserve atime; it only avoids modifying it. **Client-only, never crosses the wire.** Exposed as `file_open_for_read()` and applied to both the buffered data path and the sendfile path |
|
| `--open-noatime` | Avoid changing access time when opening files | ✅ Parity | Sender-side policy: the sender opens source files with `O_NOATIME` (Linux) when reading them for transfer, so the open/read does NOT bump the source's on-disk access time. Degrades safely when `O_NOATIME` is unavailable (not defined) or refused (`EPERM`, since it needs `CAP_FOWNER` or file ownership): the code falls back to a normal open, so the data always transfers — only the atime-bump is skipped. It does not itself capture/preserve atime; it only avoids modifying it. **Client-only, never crosses the wire.** Exposed as `file_open_for_read()` and applied to both the buffered data path and the sendfile path |
|
||||||
| `--numeric-ids` | Do not map uid/gid by name | ✅ Parity | **A mapping modifier only:** when ownership is being applied it uses the transmitted numeric uid/gid directly, skipping the name lookup. It does **not** request ownership application on its own — combine it with `-o`/`-g`, `-a`, or an explicit map (`--chown`/`--usermap`/`--groupmap`) — and it does not need any metadata flag merely to parse. Ownership is only applied when metadata (hence the source uid/gid) is actually transmitted (see the Phase-4 identity notes) |
|
| `--numeric-ids` | Do not map uid/gid by name | ✅ Parity | **A mapping modifier only:** when ownership is being applied it uses the transmitted numeric uid/gid directly, skipping the name lookup. It does **not** request ownership application on its own — combine it with `-o`/`-g`, `-a`, or an explicit map (`--chown`/`--usermap`/`--groupmap`) — and it does not need any metadata flag merely to parse. Ownership is only applied when metadata (hence the source uid/gid) is actually transmitted (see the Phase-4 identity notes) |
|
||||||
| `--usermap=STRING` | Map usernames | ✅ Parity | Opt-in ownership application. Comma-separated `FROM:TO` rules evaluated in order, first match wins. `FROM` accepts a source-resolved user name, a name **glob** (`*`/`?`/`[...]`, expanded sender-side at CLI-parse time against the sender's passwd/group database and collapsed into numeric `LOW-HIGH` ranges, bounded by `MAX_IDENTITY_MAP`), an `@N`/bare `N` numeric id, an inclusive `LOW-HIGH` id range, `*`, or an empty field (ids with no source name). `TO` accepts a receiver-resolved **name** (protocol 2.26.0 resolves it on the receiving side against the receiver's account database, matching rsync), an `@N`/bare `N` id, or `*` (the receiving process's euid). Rules travel as resolved numeric pairs plus an optional TO name; the receiver applies a matching rule, else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup, via fd-relative `fchown`. Malformed specs are clear errors. Implies metadata; only effective where the receiver can chown (otherwise a warning) |
|
| `--usermap=STRING` | Map usernames | ✅ Parity | Opt-in ownership application. Comma-separated `FROM:TO` rules evaluated in order, first match wins. `FROM` accepts a source-resolved user name, a name **glob** (`*`/`?`/`[...]`, expanded sender-side at CLI-parse time against the sender's passwd/group database and collapsed into numeric `LOW-HIGH` ranges, bounded by `MAX_IDENTITY_MAP`), an `@N`/bare `N` numeric id, an inclusive `LOW-HIGH` id range, `*`, or an empty field (ids with no source name). `TO` accepts a receiver-resolved **name** (protocol 2.26.0 resolves it on the receiving side against the receiver's account database, matching rsync), an `@N`/bare `N` id, or `*` (the receiving process's euid). Rules travel as resolved numeric pairs plus an optional TO name; the receiver applies a matching rule, else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup, via fd-relative `fchown`. Malformed specs are clear errors. Implies metadata; only effective where the receiver can chown (otherwise a warning) |
|
||||||
@@ -408,7 +410,7 @@ match, exactly as prior phases did).
|
|||||||
is refused) also re-applies the incoming (or, for `-H`, the first member's)
|
is refused) also re-applies the incoming (or, for `-H`, the first member's)
|
||||||
xattrs and the `--fake-super` stat, so attributes are preserved rather than
|
xattrs and the `--fake-super` stat, so attributes are preserved rather than
|
||||||
silently dropped when the link fails.
|
silently dropped when the link fails.
|
||||||
- **Reserved fake-super key is receiver-only:** the `user.fastsync.stat` key is
|
- **Reserved fake-super key is receiver-only:** the `user.rsync.%stat` key is
|
||||||
excluded from sender capture AND from receiver application, so it can only be
|
excluded from sender capture AND from receiver application, so it can only be
|
||||||
written by the receiver's own `--fake-super` handling. A source file that
|
written by the receiver's own `--fake-super` handling. A source file that
|
||||||
already carries such a record is never forwarded on a plain `-X` run, so it
|
already carries such a record is never forwarded on a plain `-X` run, so it
|
||||||
@@ -417,15 +419,22 @@ match, exactly as prior phases did).
|
|||||||
Applying an ACL is owner-privileged: `fsetxattr` failure (e.g. non-root,
|
Applying an ACL is owner-privileged: `fsetxattr` failure (e.g. non-root,
|
||||||
unsupported filesystem) is logged (collapsed to one line per file) and never
|
unsupported filesystem) is logged (collapsed to one line per file) and never
|
||||||
fatal.
|
fatal.
|
||||||
- **`--fake-super`**: see the row above; the reserved key is `user.fastsync.stat`
|
- **`--fake-super`**: see the row above; the reserved key is rsync's own
|
||||||
with the documented `uid:gid:mode:mtime_sec:mtime_nsec` (mode octal) format.
|
`user.rsync.%stat` with rsync 3.4.1's exact `<octal st_mode> <rdev_major>,
|
||||||
**Replay exists**: after each stored record the receiver immediately re-applies
|
<rdev_minor> <uid>:<gid>` value (mtime is not stored — the file's own
|
||||||
the recorded mode and times fd-relative (`fake_super_restore_fd`), but it
|
timestamp carries it, exactly as rsync does). **Replay exists**: after each
|
||||||
deliberately never performs a real `chown` — `--fake-super` only *records*
|
stored record the receiver immediately re-applies the recorded permission bits
|
||||||
the resolved owner (the active `--chown`/`--usermap`/`--groupmap`/`--copy-as`
|
fd-relative (`fake_super_restore_fd`, with the recorded special bits stripped
|
||||||
mapping when one is in effect, otherwise the source's own id) for a later
|
on disk exactly like rsync), but it deliberately never performs a real
|
||||||
privileged restore. The recording format diverges from rsync's
|
`chown` — `--fake-super` only *records* the resolved owner (the active
|
||||||
`user.rsync.%stat%`; no cross-tool conversion is attempted.
|
`--chown`/`--usermap`/`--groupmap`/`--copy-as` mapping when one is in effect,
|
||||||
|
otherwise the source's own id) for a later privileged restore. Because the
|
||||||
|
key and grammar are rsync's, a regular-file, device and directory fake-super
|
||||||
|
tree is interoperable with rsync 3.4.1 in both directions; a directory's
|
||||||
|
record is written on the directory itself at creation and re-stamped by the
|
||||||
|
deferred directory-metadata pass. Symlinks are the remaining residual:
|
||||||
|
FastSync creates a real symlink where rsync writes a regular file carrying an
|
||||||
|
`S_IFLNK` (`120777`) record.
|
||||||
- **Chunk serialization (`-s`) incompatibility:** the per-file xattr block rides
|
- **Chunk serialization (`-s`) incompatibility:** the per-file xattr block rides
|
||||||
the streaming per-file frame, which `-s` replaces with a fixed buffer format,
|
the streaming per-file frame, which `-s` replaces with a fixed buffer format,
|
||||||
so `-X` / `-A` combined with `-s` is rejected up front on both ends (mirroring
|
so `-X` / `-A` combined with `-s` is rejected up front on both ends (mirroring
|
||||||
@@ -554,18 +563,22 @@ marker + rdev so `--devices/--specials` also work under `-s`. `PROTOCOL_VERSION`
|
|||||||
was bumped **2.12.0 → 2.13.0** (peers must match, exactly as prior phases did).
|
was bumped **2.12.0 → 2.13.0** (peers must match, exactly as prior phases did).
|
||||||
|
|
||||||
**Privilege gating (the crux):** making a device node requires `CAP_MKNOD` (root).
|
**Privilege gating (the crux):** making a device node requires `CAP_MKNOD` (root).
|
||||||
CI runs the integration suite as a NON-ROOT user (via setpriv), so `mknod` fails
|
When the receiver attempts a device `mknod` and the kernel refuses with
|
||||||
with `EPERM`. The receiver treats this as a graceful, logged *skip of the entry*
|
`EPERM`/`EACCES`, FastSync now reports a genuine transfer error (the receiver's
|
||||||
returned as a success/skip outcome — the whole transfer NEVER aborts just because
|
outcome aggregation fails the entry), matching rsync, which logs
|
||||||
the environment cannot create the node. `mkfifo` (FIFOs) is unprivileged, so
|
`mknod ... failed` and exits partial (23) whenever it attempts the node (as root
|
||||||
`--specials` FIFO creation is a real, assertable behavior under CI. **Sockets are
|
or with `--super`); with `--no-super` the device entry is pre-skipped instead.
|
||||||
recreated too** (protocol 2.23.0) with `mknodat(..., S_IFSOCK)`: Linux allows an
|
Only `mkfifo` (FIFOs) is unprivileged, so `--specials` FIFO creation is a real,
|
||||||
unprivileged `mknod` of a socket node because no live bound socket is created,
|
assertable behavior under CI. **Sockets are recreated too** (protocol 2.23.0)
|
||||||
so a source socket materializes as a socket-type filesystem entry exactly as
|
with `mknodat(..., S_IFSOCK)`: Linux allows an unprivileged `mknod` of a socket
|
||||||
rsync does. The "device actually created" integration assertions are guarded to
|
node because no live bound socket is created, so a source socket materializes as
|
||||||
run only as root. User-facing expectation: point `--devices` at devices and a
|
a socket-type filesystem entry exactly as rsync does. The "device actually
|
||||||
non-root receiver will faithfully skip them while transferring everything else;
|
created" integration assertions are guarded to run only as root; a root runner
|
||||||
`--specials` recreates FIFOs and socket nodes for any receiver.
|
additionally drops the receiver to an unprivileged user (setpriv) to assert the
|
||||||
|
`CAP_MKNOD` failure surfaces as a failed transfer rather than a silent skip.
|
||||||
|
User-facing expectation: point `--devices` at devices and a receiver without
|
||||||
|
`CAP_MKNOD` reports the failure, while `--specials` recreates FIFOs and socket
|
||||||
|
nodes for any receiver.
|
||||||
|
|
||||||
**Confinement & validation:** a special/device node is created with
|
**Confinement & validation:** a special/device node is created with
|
||||||
`mknodat`/`mkfifoat` on the parent directory opened fd-relative below the receive
|
`mknodat`/`mkfifoat` on the parent directory opened fd-relative below the receive
|
||||||
@@ -693,10 +706,10 @@ targets verbatim, matching rsync.
|
|||||||
|------|-------------------|-----------------|-------|
|
|------|-------------------|-----------------|-------|
|
||||||
| `--checksum` | Skip based on checksum | ✅ Parity | `-c`/`--checksum` compares per-file whole-file content digests to skip unchanged files. **As of protocol 2.23.0 the short `-c` implies the checksum quick-check**, so a plain `-c` run verifies content rather than only affecting the `--incremental` handshake. The digest algorithm is `xxh128` by default (protocol 2.26.0's negotiated default) and is selectable via `--checksum-choice`/`--cc` (`xxh128`/`xxh3`/`xxh64`/`xxhash`/`md5`/`md4`/`sha1`/`none`/`auto`, plus rsync's two-name form) and `--checksum-seed=NUM` (see those rows) |
|
| `--checksum` | Skip based on checksum | ✅ Parity | `-c`/`--checksum` compares per-file whole-file content digests to skip unchanged files. **As of protocol 2.23.0 the short `-c` implies the checksum quick-check**, so a plain `-c` run verifies content rather than only affecting the `--incremental` handshake. The digest algorithm is `xxh128` by default (protocol 2.26.0's negotiated default) and is selectable via `--checksum-choice`/`--cc` (`xxh128`/`xxh3`/`xxh64`/`xxhash`/`md5`/`md4`/`sha1`/`none`/`auto`, plus rsync's two-name form) and `--checksum-seed=NUM` (see those rows) |
|
||||||
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ✅ Parity | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and basis-dir verification. **Protocol 2.26.0 accepts rsync 3.4.1's full set** — `xxh128` (the negotiated default), `xxh3`, `xxh64`, `xxhash`, `md5`, `md4`, `sha1`, `none`, `auto`, and the two-name `transfer,pre-transfer` form — with rsync's exit-4 rejection of an unknown name and of `none` on the transfer side when `--checksum` is on. `--cc=ALG` and space forms both parse. The algorithm id and seed cross the wire; the receiver hashes its old file with the same algorithm+seed and the per-file `STATUS_CHECK` handshake carries a bounded digest pinned to the negotiated length. `checksum_digest_file` now streams **every** supported algorithm (md4 via the self-contained RFC 1320 code, sha1/md5 via EVP, none as an empty digest), so the streaming path matches its contract, and `--out-format %C` uses the selected **transfer** half of a two-name choice and renders each algorithm byte-for-byte like rsync (xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, none a blank 2-char column) — differential-tested across all algorithms. **Track-3b finding — the block-checksum residual is not observable.** rsync applies the choice to the block checksum on its wire too, while FastSync selects only the whole-file comparison digest and keeps the delta BLOCK strong checksum fixed at xxHash32 (`DeltaBlockSig`, `delta_signature_create_seeded`). Because FastSync does not interoperate with rsync on the wire, only the compared surface matters, and a pre-seeded delta differential against rsync 3.4.1 (`--no-whole-file -B8192 --stats --out-format=%c|%C %n` vs `--incremental --delta`) shows the choice does not move it: across `xxh64`, `xxh128`, `xxh3`, `md5`, `md4`, `sha1` and both two-name orders the destination tree is byte-identical, `Matched data`/`Literal data`/`Total transferred file size` are unchanged (and equal to rsync's with the block size pinned), the `%c` block-checksum token is invariant (rsync `16 + 6·ceil(size/block)`, already documented under `--out-format`; FastSync its own basis-read counter), and the exit code is 0. The negotiated algorithm is visible only in `%C`, which applies it to the whole-file transfer digest and matches rsync byte-for-byte. A false block match would require the 4-byte adler32 AND the 4-byte xxHash32 to collide; at the 256 MiB maximum with the 1 KiB minimum block size the expected false matches are ≤2⁻¹⁸, and the choice cannot change this because FastSync's block strong sum is fixed. `auto` now consults `RSYNC_CHECKSUM_LIST` (rsync's whitespace-separated preference list; unknown names skipped, first supported wins, all-unknown is exit 4) before the compiled-in order; because both peers run the identical build this deterministic resolution needs no rsync peer probe, and an explicit `--cc` still wins. The list is differential-tested through `--out-format %C` (byte-identical digests to rsync for md5/sha1/xxh3) |
|
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ✅ Parity | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and basis-dir verification. **Protocol 2.26.0 accepts rsync 3.4.1's full set** — `xxh128` (the negotiated default), `xxh3`, `xxh64`, `xxhash`, `md5`, `md4`, `sha1`, `none`, `auto`, and the two-name `transfer,pre-transfer` form — with rsync's exit-4 rejection of an unknown name and of `none` on the transfer side when `--checksum` is on. `--cc=ALG` and space forms both parse. The algorithm id and seed cross the wire; the receiver hashes its old file with the same algorithm+seed and the per-file `STATUS_CHECK` handshake carries a bounded digest pinned to the negotiated length. `checksum_digest_file` now streams **every** supported algorithm (md4 via the self-contained RFC 1320 code, sha1/md5 via EVP, none as an empty digest), so the streaming path matches its contract, and `--out-format %C` uses the selected **transfer** half of a two-name choice and renders each algorithm byte-for-byte like rsync (xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, none a blank 2-char column) — differential-tested across all algorithms. **Track-3b finding — the block-checksum residual is not observable.** rsync applies the choice to the block checksum on its wire too, while FastSync selects only the whole-file comparison digest and keeps the delta BLOCK strong checksum fixed at xxHash32 (`DeltaBlockSig`, `delta_signature_create_seeded`). Because FastSync does not interoperate with rsync on the wire, only the compared surface matters, and a pre-seeded delta differential against rsync 3.4.1 (`--no-whole-file -B8192 --stats --out-format=%c|%C %n` vs `--incremental --delta`) shows the choice does not move it: across `xxh64`, `xxh128`, `xxh3`, `md5`, `md4`, `sha1` and both two-name orders the destination tree is byte-identical, `Matched data`/`Literal data`/`Total transferred file size` are unchanged (and equal to rsync's with the block size pinned), the `%c` block-checksum token is invariant (rsync `16 + 6·ceil(size/block)`, already documented under `--out-format`; FastSync its own basis-read counter), and the exit code is 0. The negotiated algorithm is visible only in `%C`, which applies it to the whole-file transfer digest and matches rsync byte-for-byte. A false block match would require the 4-byte adler32 AND the 4-byte xxHash32 to collide; at the 256 MiB maximum with the 1 KiB minimum block size the expected false matches are ≤2⁻¹⁸, and the choice cannot change this because FastSync's block strong sum is fixed. `auto` now consults `RSYNC_CHECKSUM_LIST` (rsync's whitespace-separated preference list; unknown names skipped, first supported wins, all-unknown is exit 4) before the compiled-in order; because both peers run the identical build this deterministic resolution needs no rsync peer probe, and an explicit `--cc` still wins. The list is differential-tested through `--out-format %C` (byte-identical digests to rsync for md5/sha1/xxh3) |
|
||||||
| `--compare-dest=DIR` | Compare dest files relative to DIR | ⚠️ Caveat | DIR is a receiver-side basis; protocol 2.26.0 uses an absolute path verbatim (rsync semantics) and resolves a relative path below the destination root (`..` components are rejected, `//` collapsed and trailing `/` dropped) — note rsync resolves a relative DIR against the destination directory while FastSync resolves it below the receive root and appends the mirrored source path, so the same relative spelling addresses a different tree (use an absolute DIR for exact parity). On the receiver's per-file check (implies `--incremental`) an exact match is rsync's metadata quick-check: same size and mtime (unless `--size-only`; `-I` disables matching), with NO content digest required by default (track 5a). A match suppresses the data transfer. The FastSync-only `--verify-basis` restores the stricter whole-file content equality. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Differential-tested against rsync 3.4.1 (`compare_dest`, and `test_verify_basis_restores_strict_content`). **Relative-DIR parity (parity-2.29):** a relative DIR now resolves against the destination directory with the file's transfer-relative name appended, exactly like rsync 3.4.1, instead of FastSync's source-mirrored wire path (the historical spelling stays as a fallback; differential `test_parity_basis_fuzzy.py`). Residual: a basis MISS above the 256 MiB whole-file payload bound is refused up front (FastSync's general whole-file limit, not basis-specific); rsync applies basis dirs to arbitrary sizes. Wire: a basis-count field plus the `verify_basis` bool are present on the config frame (protocol 2.9.0/2.28.0) |
|
| `--compare-dest=DIR` | Compare dest files relative to DIR | ⚠️ Caveat | DIR is a receiver-side basis; protocol 2.26.0 uses an absolute path verbatim (rsync semantics) and resolves a relative path below the destination root (`..` components are rejected, `//` collapsed and trailing `/` dropped) — note rsync resolves a relative DIR against the destination directory while FastSync resolves it below the receive root and appends the mirrored source path, so the same relative spelling addresses a different tree (use an absolute DIR for exact parity). On the receiver's per-file check (implies `--incremental`) an exact match is rsync's metadata quick-check: same size and mtime (unless `--size-only`; `-I` disables matching), with NO content digest required by default (track 5a). A match suppresses the data transfer. The FastSync-only `--verify-basis` restores the stricter whole-file content equality. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Differential-tested against rsync 3.4.1 (`compare_dest`, and `test_verify_basis_restores_strict_content`). **Relative-DIR parity (parity-2.29):** a relative DIR now resolves against the destination directory with the file's transfer-relative name appended, exactly like rsync 3.4.1, instead of FastSync's source-mirrored wire path (the historical spelling stays as a fallback; differential `test_parity_basis_fuzzy.py`). **Streaming (parity-2.29/#318):** the over-256 MiB basis-MISS residual is closed — a basis MISS above the whole-file bound is no longer refused; it falls through to the streaming whole-file transfer, so basis dirs now apply to files of arbitrary size (`test_large_stream.py`). Residual: a relative DIR additionally probes the historical mirror-appended spelling as a fallback, where rsync probes only the destination-relative name. Wire: a basis-count field plus the `verify_basis` bool are present on the config frame (protocol 2.9.0/2.28.0) |
|
||||||
| `--copy-dest=DIR` | Include copies of unchanged files | ⚠️ Caveat | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Track 5a re-applies the SOURCE attributes on the copy (rsync's "copy then fix attributes"): the sender transmits the source metadata with the basis check frame, so the copy's mode/uid/gid/mtime match the source rather than the basis inode (differential `copy_dest` compares modes). The copy streams the basis file through a bounded buffer, so a basis larger than the whole-file payload bound still materializes. Repeatable; command-line order = priority. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
| `--copy-dest=DIR` | Include copies of unchanged files | ⚠️ Caveat | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Track 5a re-applies the SOURCE attributes on the copy (rsync's "copy then fix attributes"): the sender transmits the source metadata with the basis check frame, so the copy's mode/uid/gid/mtime match the source rather than the basis inode (differential `copy_dest` compares modes). The copy streams the basis file through a bounded buffer, so a basis larger than the whole-file payload bound still materializes, and a basis MISS above the bound now falls through to the streaming whole-file transfer (the historical over-256 MiB MISS refusal is closed; `test_large_stream.py`). Repeatable; command-line order = priority. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
||||||
| `--link-dest=DIR` | Hardlink to files when unchanged | ⚠️ Caveat | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy (streamed from the basis, so an over-limit basis still works), never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Differential-tested against rsync 3.4.1 (`link_dest`). Inherent shared-inode semantics (identical to rsync): a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); protocol 2.26.0 re-links an already up-to-date destination file to the basis; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. **Relative-DIR parity (parity-2.29):** a relative DIR resolves against the destination directory with the transfer-relative name appended, exactly like rsync 3.4.1 (differential `test_parity_basis_fuzzy.py`). Residual: a basis MISS above the 256 MiB whole-file payload bound is refused (FastSync's general whole-file limit). Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
| `--link-dest=DIR` | Hardlink to files when unchanged | ⚠️ Caveat | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy (streamed from the basis, so an over-limit basis still works), never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Differential-tested against rsync 3.4.1 (`link_dest`). Inherent shared-inode semantics (identical to rsync): a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); protocol 2.26.0 re-links an already up-to-date destination file to the basis; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. **Relative-DIR parity (parity-2.29):** a relative DIR resolves against the destination directory with the transfer-relative name appended, exactly like rsync 3.4.1 (differential `test_parity_basis_fuzzy.py`). **Streaming (parity-2.29/#318):** the over-256 MiB basis-MISS residual is closed — a MISS above the bound now falls through to the streaming whole-file transfer (`test_large_stream.py`). Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
|
||||||
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ⚠️ Caveat | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (a deterministic port of rsync 3.4.1's matcher — `util1.c` `fuzzy_distance`/`find_filename_suffix` plus `generator.c find_fuzzy`'s exact size+mtime pass — documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×); rsync's fuzzy matcher is not tied to a delta size bound and empirically reuses a basis well outside FastSync's window (a 64 KiB source against a repeated-content sibling from 0.25× to 10000×, and files as small as 300 B), so candidate ELIGIBILITY — and hence the chosen basis — can differ even though the name heuristic is the same; protocol 2.26.0 uses rsync's weighted-Levenshtein name/suffix distance plus an exact size+mtime pass and reads a single best candidate; the tie-break (smallest size gap, then lexical name) is deterministic where rsync leaves equal distances to its file-list order; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: the name matching is rsync's own rule; the residual is eligibility bounded by FastSync's delta engine, so no name-matcher port can widen it. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file). **Reclassified Caveat (track 5b):** the output is always byte-exact regardless of the basis, and the name rule is rsync's, **Eligibility parity (parity-2.29):** the fuzzy candidate search no longer inherits the ordinary delta engine's 16 KiB minimum or 10× size-ratio bound, so an oversized or sub-16-KiB sibling is now reused exactly as rsync reuses it — the chosen basis (and therefore `Matched data`/`Literal data`/`Total transferred file size`) matches rsync where the block size is pinned (differential `test_parity_basis_fuzzy.py`; the old boundary tests were flipped to assert both tools reuse the basis). Residual: the whole-basis buffer cap (`MAX_RECEIVE_WHOLE_FILE_SIZE`, 256 MiB) and the deterministic tie-break where Line truncated
|
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ⚠️ Caveat | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (a deterministic port of rsync 3.4.1's matcher — `util1.c` `fuzzy_distance`/`find_filename_suffix` plus `generator.c find_fuzzy`'s exact size+mtime pass — documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×); rsync's fuzzy matcher is not tied to a delta size bound and empirically reuses a basis well outside FastSync's window (a 64 KiB source against a repeated-content sibling from 0.25× to 10000×, and files as small as 300 B), so candidate ELIGIBILITY — and hence the chosen basis — can differ even though the name heuristic is the same; protocol 2.26.0 uses rsync's weighted-Levenshtein name/suffix distance plus an exact size+mtime pass and reads a single best candidate; the tie-break (smallest size gap, then lexical name) is deterministic where rsync leaves equal distances to its file-list order; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: the name matching is rsync's own rule; the residual is eligibility bounded by FastSync's delta engine, so no name-matcher port can widen it. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file). **Reclassified Caveat (track 5b):** the output is always byte-exact regardless of the basis, and the name rule is rsync's, **Eligibility parity (parity-2.29):** the fuzzy candidate search no longer inherits the ordinary delta engine's 16 KiB minimum or 10× size-ratio bound, so an oversized or sub-16-KiB sibling is now reused exactly as rsync reuses it — the chosen basis (and therefore `Matched data`/`Literal data`/`Total transferred file size`) matches rsync where the block size is pinned (differential `test_parity_basis_fuzzy.py`; the old boundary tests were flipped to assert both tools reuse the basis). Residual: the deterministic tie-break where rsync leaves equal distances to its file-list order. (The whole-basis buLine truncated
|
||||||
|
|
||||||
## 12. Compression
|
## 12. Compression
|
||||||
|
|
||||||
@@ -730,8 +743,8 @@ targets verbatim, matching rsync.
|
|||||||
| Flag | Rsync Description | FastSync Status | Notes |
|
| Flag | Rsync Description | FastSync Status | Notes |
|
||||||
|------|-------------------|-----------------|-------|
|
|------|-------------------|-----------------|-------|
|
||||||
| `--daemon` | Run as rsync daemon | ❌ Divergent | Wave A: a real persistent listener. `fastsync-server --daemon --config FILE` (plus `--no-detach` to stay foreground; without it the listener detaches to the background after binding) reads a FastSync-native module config file and serves each connection confined to the requested module's `path` root (never a client-chosen root; every client-chosen-ownership/super-user request (`--numeric-ids`/`--chown`/`--usermap`/`--groupmap`/`--fake-super`/`--copy-as`/explicit `--super`) is refused unless the module opts in with `client owner = yes`, and the operator `--no-super` veto is honored). TCP/TLS via the existing `--tls` stack; plaintext still requires `--allow-unauthenticated` (same secure default as the standalone server). Client destinations use rsync's `host::module/path` form. Wire/protocol: the config frame gained a trailing daemon-module string and `PROTOCOL_VERSION` was bumped **2.14.0 → 2.15.0** (see the Daemon Mode notes below). Daemon mode is built in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding |
|
| `--daemon` | Run as rsync daemon | ❌ Divergent | Wave A: a real persistent listener. `fastsync-server --daemon --config FILE` (plus `--no-detach` to stay foreground; without it the listener detaches to the background after binding) reads a FastSync-native module config file and serves each connection confined to the requested module's `path` root (never a client-chosen root; every client-chosen-ownership/super-user request (`--numeric-ids`/`--chown`/`--usermap`/`--groupmap`/`--fake-super`/`--copy-as`/explicit `--super`) is refused unless the module opts in with `client owner = yes`, and the operator `--no-super` veto is honored). TCP/TLS via the existing `--tls` stack; plaintext still requires `--allow-unauthenticated` (same secure default as the standalone server). Client destinations use rsync's `host::module/path` form. Wire/protocol: the config frame gained a trailing daemon-module string and `PROTOCOL_VERSION` was bumped **2.14.0 → 2.15.0** (see the Daemon Mode notes below). Daemon mode is built in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding |
|
||||||
| `--config=FILE` | Alternate rsyncd.conf file | ❌ Divergent | Wave A: selects the daemon config file. Default when omitted (in `--daemon` mode): `~/.config/fastsync/fastsyncd.conf` if it exists, else `/etc/fastsyncd.conf`. The grammar is FastSync-native (documented in the Daemon Mode notes below) and strictly rejects unknown keys so a typo can never silently change what a module serves; requires `--daemon` |
|
| `--config=FILE` | Alternate rsyncd.conf file | ❌ Divergent | Wave A: selects the daemon config file. Default when omitted (in `--daemon` mode): `~/.config/fastsync/fastsyncd.conf` if it exists, else `/etc/fastsyncd.conf`. The grammar is FastSync-native (documented in the Daemon Mode notes below) and still strictly rejects a genuinely unknown key so a typo can never silently change what a module serves; requires `--daemon`. **rsync 3.4.1 key subset accepted:** the common rsyncd.conf GLOBAL keys (`port`, `address`, `motd file`, `max connections`, `hosts allow`/`hosts deny`, plus the inert `pid file`, `log file`, `socket options`/`sockopts`, `listen backlog`, `syslog facility`, `syslog tag`, `log format`, `use chroot`, `uid`, `gid`, `timeout`, `max verbosity`/`min verbosity`, `lock file`, `transfer logging`, `strict modes`, `reverse lookup`/`forward lookup`, `ignore errors`, `ignore nonreadable`, `dont compress`) and MODULE keys (`path`, `read only`, `max connections`, `auth users`, `hosts allow`/`hosts deny`, plus the inert `comment`, `use chroot`, `uid`/`gid`/`daemon uid`/`daemon gid`, `exclude`, `include`, `exclude from`/`include from`, `filter`, `secrets file`, `auth digest`, `max verbosity`/`min verbosity`, `lock file`, `transfer logging`, `log file`/`log format`/`syslog facility`/`syslog tag`, `timeout`, `strict modes`, `numeric ids`, `fake super`, `munge symlinks`, `write only`, `list`, `dont compress`, `charset`, `refuse options`, `incoming chmod`/`outgoing chmod`, `open noatime`, `max size`/`min size`, `temp dir`, `pre-xfer exec`/`post-xfer exec`, `name converter`, `proxy protocol`/`proxy protocol hosts`, `reverse lookup`/`forward lookup`, `ignore errors`, `ignore nonreadable`) are recognized. Keys with a FastSync equivalent map onto it. Modules are **read-only by default**, exactly like rsync: `read only = no` (or `write only = yes`, which FastSync maps to writability because it is push-only) opts a module in; a global `read only` sets the default for later modules, and an explicit module value always wins. Keys with no FastSync equivalent load **inert** (no effect) rather than failing the whole config, and every inert key whose intent is access control (`secrets file`, `refuse options`, `exclude`/`include`/`filter`, `max size`/`min size`, `pre-xfer exec`/`post-xfer exec`, `incoming chmod`/`outgoing chmod`, `name converter`, `use chroot`, `uid`/`gid`, ...) emits a startup **WARN** naming the key (and module), so an operator cannot mistake an unenforced restriction for an enforced one. Residual: the native grammar still differs from rsync's (no `\` line continuation, `%VAR%` expansion, `[global]` re-entry, or inline `#` comments), and the inert keys are genuinely not enforced — in particular a daemon-side `exclude`/`filter` is NOT applied and `secrets file` is NOT read (use `path`, `--password-file`, and client-side filters instead) |
|
||||||
| `--dparam=OVERRIDE` | Override global daemon config | ❌ Divergent | Wave A: overrides one global scalar from the command line (`--dparam port=8734` and `--dparam=KEY=VALUE` both work). Limited to the global keys the grammar defines (`port`, `motd file`, `address`, `max connections`, `max connections per host`, `auth failure delay`, `auth lockout threshold`, `auth lockout duration`, `hosts allow`, `hosts deny`); keys are case-insensitive and unknown keys/invalid values are rejected. Requires `--daemon` |
|
| `--dparam=OVERRIDE` | Override global daemon config | ❌ Divergent | Wave A: overrides one global scalar from the command line (`--dparam port=8734` and `--dparam=KEY=VALUE` both work). Reuses the exact same global-key dispatch as `--config`, so it accepts the native global keys (`port`, `motd file`, `address`, `read only`, `max connections`, `max connections per host`, `auth failure delay`, `auth lockout threshold`, `auth lockout duration`, `hosts allow`, `hosts deny`), the recognized inert rsync global keys, and rsync's compact spellings (`motdfile`, `pidfile`, `logfile`); keys are case-insensitive. `read only` sets the global default and re-applies it to every module that did not set its own value; the default is `yes` (rsync modules are read-only unless `read only = no` / `write only = yes`), so `--dparam read only=no` is required to make modules without their own value writable, and an inert security global key (`use chroot`, `uid`, `gid`, `strict modes`) emits the same startup **WARN** as `--config`. Genuinely unknown keys and invalid values are rejected. Requires `--daemon` |
|
||||||
| `--no-detach` | Don't detach from parent | ✅ Parity | Wave A: with `--daemon`, keeps the listener in the foreground (what integration tests use). Without it the daemonizes (fork/setsid, stdio redirected to /dev/null) after the listening socket is bound. Requires `--daemon` |
|
| `--no-detach` | Don't detach from parent | ✅ Parity | Wave A: with `--daemon`, keeps the listener in the foreground (what integration tests use). Without it the daemonizes (fork/setsid, stdio redirected to /dev/null) after the listening socket is bound. Requires `--daemon` |
|
||||||
| `--password-file=FILE` | Read daemon password from file | ❌ Divergent | A7 daemon auth. Client: `--password-file` supplies `user:password` for a `host::module/path` destination (the username is taken from this file, so `user@host::module` stays rejected); the literal password is held client-side only for the SCRAM handshake and wiped at teardown. Server (`fastsync-server --daemon --password-file FILE`): the salted-PBKDF2 verifier store that modules with `auth users` are verified against. **Neither the password nor any replayable bearer value crosses the wire or is stored server-side** — the store holds a per-user salt plus derived keys, and the daemon proves the secret with a per-connection nonce challenge. The file must be private to its owner: both the client and server verify the exact inode they read (open-then-`fstat`, so the check cannot be raced) and refuse a `--password-file`/`--early-input` that is not owned by the current user or grants any group/other permission bit (mode 0600), mirroring the TLS private-key check. A process-substitution pipe (`--early-input <(vault ...)`) is still accepted when it satisfies those checks. **Hardening follow-up:** the file is opened with `O_NOFOLLOW`, so a symlinked credential path fails closed (`ELOOP`) instead of being followed before the owner/mode gate; literal fd-backed paths (`/dev/fd/<digits>`, `/proc/self/fd/<digits>`, which is what a bash process substitution passes) are exempt, so process substitution still works. A FIFO/process-substitution read now waits under a bounded ~3 s deadline for its writer, so a slow producer works while a connected-but-silent FIFO fails instead of hanging. See the Daemon Mode notes below for the file formats and the plaintext/TLS caveat |
|
| `--password-file=FILE` | Read daemon password from file | ❌ Divergent | A7 daemon auth. Client: `--password-file` supplies `user:password` for a `host::module/path` destination (the username is taken from this file, so `user@host::module` stays rejected); the literal password is held client-side only for the SCRAM handshake and wiped at teardown. Server (`fastsync-server --daemon --password-file FILE`): the salted-PBKDF2 verifier store that modules with `auth users` are verified against. **Neither the password nor any replayable bearer value crosses the wire or is stored server-side** — the store holds a per-user salt plus derived keys, and the daemon proves the secret with a per-connection nonce challenge. The file must be private to its owner: both the client and server verify the exact inode they read (open-then-`fstat`, so the check cannot be raced) and refuse a `--password-file`/`--early-input` that is not owned by the current user or grants any group/other permission bit (mode 0600), mirroring the TLS private-key check. A process-substitution pipe (`--early-input <(vault ...)`) is still accepted when it satisfies those checks. **Hardening follow-up:** the file is opened with `O_NOFOLLOW`, so a symlinked credential path fails closed (`ELOOP`) instead of being followed before the owner/mode gate; literal fd-backed paths (`/dev/fd/<digits>`, `/proc/self/fd/<digits>`, which is what a bash process substitution passes) are exempt, so process substitution still works. A FIFO/process-substitution read now waits under a bounded ~3 s deadline for its writer, so a slow producer works while a connected-but-silent FIFO fails instead of hanging. See the Daemon Mode notes below for the file formats and the plaintext/TLS caveat |
|
||||||
| `--early-input=FILE` | Use FILE for daemon early exec | ❌ Divergent | Server-only (requires `--daemon`): a second credential-store file, same new-format grammar as `--password-file`, read before the listener accepts connections (a secrets-manager / process-substitution source). Its entries layer over `--password-file`: byte-identical verifiers dedupe, a conflicting verifier for the same user is a startup error. Opened with the same `O_NOFOLLOW` hardening as `--password-file` (a symlinked path fails closed with `ELOOP`; fd-backed `/dev/fd/N`/`/proc/self/fd/N` process-substitution paths are exempt) and a FIFO read is bound-waited (~3 s) so a slow producer works while a writer-less FIFO cannot hang. A daemon whose modules declare `auth users` must be given at least one of the two, or it refuses to start (fail closed) |
|
| `--early-input=FILE` | Use FILE for daemon early exec | ❌ Divergent | Server-only (requires `--daemon`): a second credential-store file, same new-format grammar as `--password-file`, read before the listener accepts connections (a secrets-manager / process-substitution source). Its entries layer over `--password-file`: byte-identical verifiers dedupe, a conflicting verifier for the same user is a startup error. Opened with the same `O_NOFOLLOW` hardening as `--password-file` (a symlinked path fails closed with `ELOOP`; fd-backed `/dev/fd/N`/`/proc/self/fd/N` process-substitution paths are exempt) and a FIFO read is bound-waited (~3 s) so a slow producer works while a writer-less FIFO cannot hang. A daemon whose modules declare `auth users` must be given at least one of the two, or it refuses to start (fail closed) |
|
||||||
@@ -739,7 +752,7 @@ targets verbatim, matching rsync.
|
|||||||
|
|
||||||
**Daemon Mode notes (Wave A protocol 2.15.0; A7 auth protocol 2.19.0; MOTD no bump):** FastSync daemon mode is supported in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding.
|
**Daemon Mode notes (Wave A protocol 2.15.0; A7 auth protocol 2.19.0; MOTD no bump):** FastSync daemon mode is supported in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding.
|
||||||
|
|
||||||
- **Config grammar** (`fastsyncd.conf`): line-based; an implicit global section first, then `[module]` sections. Keys are case-insensitive, values are trimmed and may be wrapped in one layer of double quotes (`path = "/srv/my dir"`). `#` and `;` at the start of a line (after leading whitespace) are full-line comments; inline comments and `\` continuations are not supported. Lines are bounded (4096 chars), and at most 256 `[module]` sections are accepted. Global keys: `port` (default 873), `motd file` (the daemon sends its bounded, escaped content to a client after the module gate/auth accepts, unless the client passes `--no-motd`), `address` (optional bind address), `max connections` (positive integer cap on concurrent connections, default 100; 0/negative/garbage is a parse error), `max connections per host` (concurrent-connection cap per source IP, default 0 = unlimited), `auth failure delay` (milliseconds to sleep after a failed authentication, default 500; 0 disables, capped at 5000), `auth lockout threshold` (failed authentications from one source before lockout, default 10; 0 disables), `auth lockout duration` (seconds a locked-out source is refused, default 300), `hosts allow` and `hosts deny` (comma- and/or whitespace-separated host access patterns — see the host access control note below). Module keys: `path` (required; the daemon-side authorized root for that module), `read only` (yes/no/true/false/1/0, default no), `client owner` (yes/no/true/false/1/0, default no; opts the module into client-chosen ownership — see below), `auth users` (comma list), `max connections` (optional per-module cap, 0 = unlimited; enforced across all connection children), `hosts allow`/`hosts deny` (per-module host access lists). **Unknown keys and malformed lines are parse-and-reject errors** (never silently ignored), so a typo cannot change what a module serves.
|
- **Config grammar** (`fastsyncd.conf`): line-based; an implicit global section first, then `[module]` sections. Keys are case-insensitive, values are trimmed and may be wrapped in one layer of double quotes (`path = "/srv/my dir"`). `#` and `;` at the start of a line (after leading whitespace) are full-line comments; inline comments and `\` continuations are not supported. Lines are bounded (4096 chars), and at most 256 `[module]` sections are accepted. Global keys: `port` (default 873), `motd file` (the daemon sends its bounded, escaped content to a client after the module gate/auth accepts, unless the client passes `--no-motd`), `address` (optional bind address), `max connections` (positive integer cap on concurrent connections, default 100; 0/negative/garbage is a parse error), `max connections per host` (concurrent-connection cap per source IP, default 0 = unlimited), `auth failure delay` (milliseconds to sleep after a failed authentication, default 500; 0 disables, capped at 5000), `auth lockout threshold` (failed authentications from one source before lockout, default 10; 0 disables), `auth lockout duration` (seconds a locked-out source is refused, default 300), `hosts allow` and `hosts deny` (comma- and/or whitespace-separated host access patterns — see the host access control note below). Module keys: `path` (required; the daemon-side authorized root for that module), `read only` (yes/no/true/false/1/0, default yes — rsync modules are read-only unless `read only = no`/`write only = yes`), `client owner` (yes/no/true/false/1/0, default no; opts the module into client-chosen ownership — see below), `auth users` (comma list), `max connections` (optional per-module cap, 0 = unlimited; enforced across all connection children), `hosts allow`/`hosts deny` (per-module host access lists). **Unknown keys and malformed lines are parse-and-reject errors** (never silently ignored), so a typo cannot change what a module serves. To reduce the rsync divergence, the parser additionally accepts the common rsync 3.4.1 GLOBAL and MODULE keys: the keys with a FastSync equivalent (`path`, `read only`, `max connections`, `auth users`, `hosts allow`/`hosts deny`, and the global `port`/`address`/`motd file`) map onto it, a global `read only` becomes the default for modules defined after it, and the keys with no FastSync equivalent (e.g. `pid file`, `log file`, `use chroot`, `uid`/`gid`, `comment`, `exclude`/`include`, `max verbosity`, `lock file`, `transfer logging`, `timeout`, `secrets file`) are recognized and loaded **inert** (accepted-but-ignored) instead of failing the whole file. `--dparam` reuses the same dispatch, so it also accepts the inert rsync global keys and the compact spellings `motdfile`/`pidfile`/`logfile`. A key outside both sets is still rejected. The inert keys are genuinely not enforced: a daemon-side `exclude`/`include`/`filter` is not applied and a `secrets file` is not read (use `--password-file`/`--early-input`), so an rsync config that relies on those must be edited rather than trusted.
|
||||||
- **Host access control (`hosts allow`/`hosts deny`):** both keys accept a comma- and/or whitespace-separated list of patterns and may appear globally and/or per module (multiple config-file lines append; a `--dparam` override replaces). Supported patterns are `*` (match all), an IPv4 or IPv6 literal (`10.0.0.1`, `2001:db8::1`), and an IPv4/IPv6 CIDR (`10.0.0.0/8`, `2001:db8::/32`). Hostname patterns are **not** supported: because the peer is always a numeric address and no reverse DNS is performed, a hostname/glob pattern would silently never match, so it is rejected at load time (fail-closed) instead of being accepted as a dead rule. An IPv4 peer on a dual-stack IPv6 listener is normalized from its `::ffff:a.b.c.d` form so IPv4 patterns match it. rsync-like semantics: a matching `hosts deny` rejects; if any `hosts allow` entries exist, a peer matching none of them is rejected; deny takes precedence over allow. The daemon enforces the global list first, then the selected module's list, **before authentication** in `server_module_gate`, with an audit log line naming the peer, the module and the outcome. The numeric peer address is obtained with `getpeername`+`inet_ntop` (`utils_fd_peer_ip`, handling both address families); when it cannot be obtained a module with any ACL fails closed (refused), while an ACL-free module continues and logs at debug. A malformed pattern (e.g. an out-of-range CIDR prefix) is a parse error at load time.
|
- **Host access control (`hosts allow`/`hosts deny`):** both keys accept a comma- and/or whitespace-separated list of patterns and may appear globally and/or per module (multiple config-file lines append; a `--dparam` override replaces). Supported patterns are `*` (match all), an IPv4 or IPv6 literal (`10.0.0.1`, `2001:db8::1`), and an IPv4/IPv6 CIDR (`10.0.0.0/8`, `2001:db8::/32`). Hostname patterns are **not** supported: because the peer is always a numeric address and no reverse DNS is performed, a hostname/glob pattern would silently never match, so it is rejected at load time (fail-closed) instead of being accepted as a dead rule. An IPv4 peer on a dual-stack IPv6 listener is normalized from its `::ffff:a.b.c.d` form so IPv4 patterns match it. rsync-like semantics: a matching `hosts deny` rejects; if any `hosts allow` entries exist, a peer matching none of them is rejected; deny takes precedence over allow. The daemon enforces the global list first, then the selected module's list, **before authentication** in `server_module_gate`, with an audit log line naming the peer, the module and the outcome. The numeric peer address is obtained with `getpeername`+`inet_ntop` (`utils_fd_peer_ip`, handling both address families); when it cannot be obtained a module with any ACL fails closed (refused), while an ACL-free module continues and logs at debug. A malformed pattern (e.g. an out-of-range CIDR prefix) is a parse error at load time.
|
||||||
- **Connection caps, shared registry and auth lockout:** the global `max connections` key (default 100) is plumbed into the listener (`transport_tcp.c`), which rejects a connection once the accept-loop parent's active-child count reaches it; the IPv4/IPv6 peer is logged for every accepted connection. Because the listener forks one child per connection, the per-module `max connections` cap, the global `max connections per host` cap, and the auth-failure counter live in a fixed-size registry carved from an anonymous shared mapping (`daemon_limits.c`, `mmap(MAP_SHARED|MAP_ANONYMOUS)`) created by the parent before the accept loop, so every forked child shares the same counters (C11 atomics only — never a pthread lock, which can deadlock in a forked child). The parent reserves a registry slot per accepted connection and the child records the selected module and source IP once known; the parent's `SIGCHLD` handler reclaims the slot when the child dies (including `SIGKILL`) and re-derives the per-module and per-source occupancy counts from the surviving REGISTERED slots, so a child killed mid-registration cannot leak a count. The per-source table has a bounded lifetime: an entry with no live connection is reclaimed after its lockout expires or it has been idle (300 s); if the table is genuinely full the per-source cap/lockout fails open for new sources (per-module cap and ACLs still apply) with a rate-limited warning. The per-module cap (0 = unlimited) is enforced after the module lookup and before auth; per-source identity reuses the normalized numeric peer address (`utils_fd_peer_ip`, IPv4-mapped IPv6 collapsed to IPv4), and a trusted loopback peer (127.0.0.0/8 / `::1`, `utils_fd_peer_is_local`) is exempt from the per-source cap and the auth lockout because all local clients share one address (the per-module/global caps still apply). Clients behind a shared NAT/proxy address likewise share one per-source budget and lockout counter. A failed authentication increments the shared per-source failure count and, once `auth lockout threshold` (default 10; 0 disables) is reached, the source is refused for `auth lockout duration` seconds (default 300) before any challenge is sent, even when the next attempt is handled by a different forked child; a successful authentication clears the counter. On a failed authentication the per-connection child still sleeps the global `auth failure delay` (default 500 ms, 0 disables, capped at 5000) via `nanosleep`, rate-limiting online guessing without delaying a success. A missing registry (allocation failure) degrades to the global cap and host ACLs rather than refusing to start.
|
- **Connection caps, shared registry and auth lockout:** the global `max connections` key (default 100) is plumbed into the listener (`transport_tcp.c`), which rejects a connection once the accept-loop parent's active-child count reaches it; the IPv4/IPv6 peer is logged for every accepted connection. Because the listener forks one child per connection, the per-module `max connections` cap, the global `max connections per host` cap, and the auth-failure counter live in a fixed-size registry carved from an anonymous shared mapping (`daemon_limits.c`, `mmap(MAP_SHARED|MAP_ANONYMOUS)`) created by the parent before the accept loop, so every forked child shares the same counters (C11 atomics only — never a pthread lock, which can deadlock in a forked child). The parent reserves a registry slot per accepted connection and the child records the selected module and source IP once known; the parent's `SIGCHLD` handler reclaims the slot when the child dies (including `SIGKILL`) and re-derives the per-module and per-source occupancy counts from the surviving REGISTERED slots, so a child killed mid-registration cannot leak a count. The per-source table has a bounded lifetime: an entry with no live connection is reclaimed after its lockout expires or it has been idle (300 s); if the table is genuinely full the per-source cap/lockout fails open for new sources (per-module cap and ACLs still apply) with a rate-limited warning. The per-module cap (0 = unlimited) is enforced after the module lookup and before auth; per-source identity reuses the normalized numeric peer address (`utils_fd_peer_ip`, IPv4-mapped IPv6 collapsed to IPv4), and a trusted loopback peer (127.0.0.0/8 / `::1`, `utils_fd_peer_is_local`) is exempt from the per-source cap and the auth lockout because all local clients share one address (the per-module/global caps still apply). Clients behind a shared NAT/proxy address likewise share one per-source budget and lockout counter. A failed authentication increments the shared per-source failure count and, once `auth lockout threshold` (default 10; 0 disables) is reached, the source is refused for `auth lockout duration` seconds (default 300) before any challenge is sent, even when the next attempt is handled by a different forked child; a successful authentication clears the counter. On a failed authentication the per-connection child still sleeps the global `auth failure delay` (default 500 ms, 0 disables, capped at 5000) via `nanosleep`, rate-limiting online guessing without delaying a success. A missing registry (allocation failure) degrades to the global cap and host ACLs rather than refusing to start.
|
||||||
- **Module selection & confinement:** the client requests a module with an rsync-style `host::module[/path]` destination. The module name crosses the wire as a trailing string on the config frame (bumping `PROTOCOL_VERSION` 2.14.0 → 2.15.0; the bump is required because the config-frame layout changed and the strict same-version handshake is what prevents a peer from desynchronizing on the new trailing field). The daemon looks the module up in ITS OWN config and uses the module's `path` as the authorized root through the exact same `configure_authorization` confinement the standalone server applies to `--destination-root` (`file_open_secure_parent`, `has_path_traversal`, `path_is_within`); the client never supplies the root, every client-chosen-ownership/super-user request is refused unless the module declares `client owner = yes` (the daemon's per-module opt-in, see below), and the operator `--no-super` veto forces super-user activities off for every daemon connection. The client's `/path` part is relative inside the module and is rejected if absolute or if it contains `..`. Unknown modules are refused before any data moves (the run fails cleanly at the config handshake). An absolute destination and a module request against a non-daemon server are also refused.
|
- **Module selection & confinement:** the client requests a module with an rsync-style `host::module[/path]` destination. The module name crosses the wire as a trailing string on the config frame (bumping `PROTOCOL_VERSION` 2.14.0 → 2.15.0; the bump is required because the config-frame layout changed and the strict same-version handshake is what prevents a peer from desynchronizing on the new trailing field). The daemon looks the module up in ITS OWN config and uses the module's `path` as the authorized root through the exact same `configure_authorization` confinement the standalone server applies to `--destination-root` (`file_open_secure_parent`, `has_path_traversal`, `path_is_within`); the client never supplies the root, every client-chosen-ownership/super-user request is refused unless the module declares `client owner = yes` (the daemon's per-module opt-in, see below), and the operator `--no-super` veto forces super-user activities off for every daemon connection. The client's `/path` part is relative inside the module and is rejected if absolute or if it contains `..`. Unknown modules are refused before any data moves (the run fails cleanly at the config handshake). An absolute destination and a module request against a non-daemon server are also refused.
|
||||||
@@ -796,7 +809,7 @@ modes or links.
|
|||||||
| `--stop-after=MINS` | Stop after N minutes | ✅ Parity | Client-only sender stop deadline (Phase 6): computing `--stop-after=MINS` (a positive minute count; 0/negative/garbage rejected) and `--stop-at=TIME` (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`; a past time stops immediately). The transfer stops ELEGANTLY at the next chunk boundary: everything already fully sent is kept and applied, the run returns 0, and --delete (late/delete-after timing) does NOT wipe the destination — when the scan is cut short the partial keep-set manifest is suppressed with a warning (the delete walk is skipped rather than acting on an incomplete keep-set, so unscanned source mirrors survive). `--delete-before`/`--delete-during` still run their complete pre-scan (which ignores the deadline). Local client-only fields: never serialized into the wire config frame, so no PROTOCOL_VERSION bump. `--stop-after` uses CLOCK_MONOTONIC; `--stop-at` uses the wall clock. Works single-threaded and under `-j`/`--threads` (multithreaded). Divergence: rsync computes `--stop-after` from the run start; FastSync likewise. When both are given, the earlier of the two deadlines wins (checked per iteration). See the Phase-6 stop notes below |
|
| `--stop-after=MINS` | Stop after N minutes | ✅ Parity | Client-only sender stop deadline (Phase 6): computing `--stop-after=MINS` (a positive minute count; 0/negative/garbage rejected) and `--stop-at=TIME` (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`; a past time stops immediately). The transfer stops ELEGANTLY at the next chunk boundary: everything already fully sent is kept and applied, the run returns 0, and --delete (late/delete-after timing) does NOT wipe the destination — when the scan is cut short the partial keep-set manifest is suppressed with a warning (the delete walk is skipped rather than acting on an incomplete keep-set, so unscanned source mirrors survive). `--delete-before`/`--delete-during` still run their complete pre-scan (which ignores the deadline). Local client-only fields: never serialized into the wire config frame, so no PROTOCOL_VERSION bump. `--stop-after` uses CLOCK_MONOTONIC; `--stop-at` uses the wall clock. Works single-threaded and under `-j`/`--threads` (multithreaded). Divergence: rsync computes `--stop-after` from the run start; FastSync likewise. When both are given, the earlier of the two deadlines wins (checked per iteration). See the Phase-6 stop notes below |
|
||||||
| `--stop-at=TIME` | Stop at specified time | ✅ Parity | Deadline transfer stop (client-only, never serialized). Protocol 2.26.0 accepts rsync's full date/time grammar (`2030-12-31T23:59`, `2030/12/31T23:59`, `2030-12-31`, `12-31`, `14:00`, `:59`, `1`) in addition to FastSync's `HH:MM[:SS]` and `now+N[smhd]`; a past time stops immediately. Everything already transferred is kept and an early stop suppresses the late `--delete` keep-set so unscanned source mirrors survive. Works single-threaded and under `-j`/`--threads` |
|
| `--stop-at=TIME` | Stop at specified time | ✅ Parity | Deadline transfer stop (client-only, never serialized). Protocol 2.26.0 accepts rsync's full date/time grammar (`2030-12-31T23:59`, `2030/12/31T23:59`, `2030-12-31`, `12-31`, `14:00`, `:59`, `1`) in addition to FastSync's `HH:MM[:SS]` and `now+N[smhd]`; a past time stops immediately. Everything already transferred is kept and an early stop suppresses the late `--delete` keep-set so unscanned source mirrors survive. Works single-threaded and under `-j`/`--threads` |
|
||||||
| `--fsync` | Fsync every written file before publication | ✅ Parity | |
|
| `--fsync` | Fsync every written file before publication | ✅ Parity | |
|
||||||
| `--protocol=NUM` | Force older protocol version | ❌ Divergent | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.28.0) with no downgrade/backward-compat code paths, so `--protocol=2.28.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.27.0`/`2.26.0`/`2.25.0`/`2.24.0`/`2.23.0`/`2.22.0`/`2.21.0`/`2.20.0`/`2.19.0`/`2.18.0`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
|
| `--protocol=NUM` | Force older protocol version | ❌ Divergent | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.30.0) with no downgrade/backward-compat code paths, so `--protocol=2.30.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.29.0`/`2.28.0`/`2.27.0`/`2.26.0`/`2.25.0`/`2.24.0`/`2.23.0`/`2.22.0`/`2.21.0`/`2.20.0`/`2.19.0`/`2.18.0`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
|
||||||
| `--iconv=CONVERT_SPEC` | Charset conversion | ✅ Parity | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, matching rsync's rule that the spec "stays the same whether you're pushing or pulling": on a PUSH the destination end's charset is the spec's REMOTE half, so the default receiver writes the wire bytes verbatim, and only a server started with its own `--iconv` (the daemon `charset` analog) declares a different destination charset and converts REMOTE→that LOCAL (rsync push parity, differential-tested with and without a server `--iconv`). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front; protocol 2.26.0 additionally accepts `--iconv=.` (the locale's default charset for both directions), `--iconv=-` and `--no-iconv` (disable conversion). Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below |
|
| `--iconv=CONVERT_SPEC` | Charset conversion | ✅ Parity | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, matching rsync's rule that the spec "stays the same whether you're pushing or pulling": on a PUSH the destination end's charset is the spec's REMOTE half, so the default receiver writes the wire bytes verbatim, and only a server started with its own `--iconv` (the daemon `charset` analog) declares a different destination charset and converts REMOTE→that LOCAL (rsync push parity, differential-tested with and without a server `--iconv`). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front; protocol 2.26.0 additionally accepts `--iconv=.` (the locale's default charset for both directions), `--iconv=-` and `--no-iconv` (disable conversion). Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below |
|
||||||
| `--checksum-seed=NUM` | Set checksum seed | ✅ Parity | Sets the seed for FastSync's whole-file xxHash digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). **As of protocol 2.23.0 a seed of `0` — the default when the flag is unset — is randomized per transfer and the chosen seed is sent to the receiver**, exactly like rsync, so two runs against different content do not share a predictable seed; an explicit non-zero seed is used verbatim, so an explicit seed deterministically reproduces every computed digest on BOTH endpoints (the seed crosses in the config frame). `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta` |
|
| `--checksum-seed=NUM` | Set checksum seed | ✅ Parity | Sets the seed for FastSync's whole-file xxHash digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). **As of protocol 2.23.0 a seed of `0` — the default when the flag is unset — is randomized per transfer and the chosen seed is sent to the receiver**, exactly like rsync, so two runs against different content do not share a predictable seed; an explicit non-zero seed is used verbatim, so an explicit seed deterministically reproduces every computed digest on BOTH endpoints (the seed crosses in the config frame). `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta` |
|
||||||
| `--secluded-args`, `-s` | Use protocol to send args | ❌ Divergent | Accepted for CLI compatibility (including the rsync short `-s`, Phase 7 Wave A) but a documented **no-op / divergence**. rsync's `-s` protects arguments from shell expansion by shipping them over the protocol; FastSync never passes remote arguments through a shell expansion boundary in the first place — its SSH transport builds the remote argv as **single-quote-escaped shell words** (`ssh_build_remote_command`), so the injection/leak that `-s` guards against does not exist and there is nothing to "seclude". Implementing a true arg-send protocol would mean replacing the argv-based SSH launch with an in-band argument channel, a large redesign of the transport that buys no security here. Chunk serialization remains the long-only `--chunk-serialization`. |
|
| `--secluded-args`, `-s` | Use protocol to send args | ❌ Divergent | Accepted for CLI compatibility (including the rsync short `-s`, Phase 7 Wave A) but a documented **no-op / divergence**. rsync's `-s` protects arguments from shell expansion by shipping them over the protocol; FastSync never passes remote arguments through a shell expansion boundary in the first place — its SSH transport builds the remote argv as **single-quote-escaped shell words** (`ssh_build_remote_command`), so the injection/leak that `-s` guards against does not exist and there is nothing to "seclude". Implementing a true arg-send protocol would mean replacing the argv-based SSH launch with an in-band argument channel, a large redesign of the transport that buys no security here. Chunk serialization remains the long-only `--chunk-serialization`. |
|
||||||
@@ -936,7 +949,7 @@ These are the last compatibility items and the closing phase toward rsync flag p
|
|||||||
| `-T` / `--timeout` | `-T` = `--temp-dir` | → `--timeout` (long-only) |
|
| `-T` / `--timeout` | `-T` = `--temp-dir` | → `--timeout` (long-only) |
|
||||||
| `-a` / `--archive` (= `-c -m -M`) | `-a` = `-rlptD` | → becomes **real rsync `-a`** after the renames |
|
| `-a` / `--archive` (= `-c -m -M`) | `-a` = `-rlptD` | → becomes **real rsync `-a`** after the renames |
|
||||||
|
|
||||||
**Wave B — Output & filesystem completion (✅ implemented).** `-S`/`--sparse` (`⚠️→✅`): real hole preservation — a sparse-aware writer (`write_all_sparse`) skips all-zero runs ≥ 4096 bytes with `lseek(SEEK_CUR)` and `ftruncate`s the final size, wired into both the atomic temp+rename store and `--inplace` receiver-side with **no wire change** (the full file image is already in memory; the ftruncate presize is kept). `-P` (`⚠️→✅`): interrupted-write retention — on a save failure after data reached the temp fd, `--partial` now renames the already-written temp to the destination path (best-effort; falls through to the normal unlink on failure, never retains when `--partial` is off) so a later `--append`/`--append-verify` run can resume. `--block-size=SIZE` (`⚠️→✅`): promoted after verification — `--block-size` is now an alias for `--delta-block`, both set `config->delta_block_size`, which the delta engine already honored end-to-end (`delta_signature_create_seeded` + `delta_apply`); out-of-range values keep the default. `--fake-super` (`⚠️→✅`): added `fake_super_restore_fd` to parse and re-apply the recorded `user.fastsync.stat` record fd-relative (mode/time only — protocol 2.23.0: **never a real chown**; the resolved owner is recorded for a later privileged restore); a save under `--fake-super` now re-applies the recorded attrs instead of only recording them, with the recording format unchanged. `--stderr=client` (`⚠️→❌ Divergent`): FastSync has no rsync client-message channel, and `client` is rejected at CLI parse — the rejection is the documented behavior (unit-tested). `-N`/`--crtimes` (`⚠️→❌ Divergent`): birth-times cannot be set by any portable fs call (`utimensat` sets only atime/mtime); capture/transmit stays, setting is impossible, the flag is accepted and safely inert. Review-hardening (post-eval): fake-super replay applies the mode through the shared `metadata_mode_for_policy` helper (protocol 2.23.0: exactly the source mode under `-p`, with no masking); `--sparse` takes precedence over `--preallocate` (posix_fallocate skipped so holes survive) — **reversed by the parity-completion wave: `--preallocate` now wins, matching rsync**; `--partial` retention is disabled for `--no_replace` (ignore/existing) and only marks a write-attempt after the actual write begins; `--block-size=SIZE`/`--delta-block=SIZE` inline forms are accepted.
|
**Wave B — Output & filesystem completion (✅ implemented).** `-S`/`--sparse` (`⚠️→✅`): real hole preservation — a sparse-aware writer (`write_all_sparse`) skips all-zero runs ≥ 4096 bytes with `lseek(SEEK_CUR)` and `ftruncate`s the final size, wired into both the atomic temp+rename store and `--inplace` receiver-side with **no wire change** (the full file image is already in memory; the ftruncate presize is kept). `-P` (`⚠️→✅`): interrupted-write retention — on a save failure after data reached the temp fd, `--partial` now renames the already-written temp to the destination path (best-effort; falls through to the normal unlink on failure, never retains when `--partial` is off) so a later `--append`/`--append-verify` run can resume. `--block-size=SIZE` (`⚠️→✅`): promoted after verification — `--block-size` is now an alias for `--delta-block`, both set `config->delta_block_size`, which the delta engine already honored end-to-end (`delta_signature_create_seeded` + `delta_apply`); out-of-range values keep the default. `--fake-super` (`⚠️→✅`): added `fake_super_restore_fd` to parse and re-apply the recorded `user.fastsync.stat` record fd-relative (mode/time only — protocol 2.23.0: **never a real chown**; the resolved owner is recorded for a later privileged restore); a save under `--fake-super` now re-applies the recorded attrs instead of only recording them. (The later fake-super xattr-interop pass replaced that native `user.fastsync.stat` format with rsync's `user.rsync.%stat` grammar — see the row and Phase-4 notes.) `--stderr=client` (`⚠️→❌ Divergent` then, in the wire backlog cycle, `❌→⚠️ Caveat`): the Phase-7 Wave B decision rejected `client` at CLI parse because no client-message channel existed; protocol 2.30.0 adds one (`STATUS_CLIENT_MSG`), so `client` is now accepted and forwards the client's diagnostics to the server's stderr (see the row for the remaining one-direction caveat). `-N`/`--crtimes` (`⚠️→❌ Divergent`): birth-times cannot be set by any portable fs call (`utimensat` sets only atime/mtime); capture/transmit stays, setting is impossible, the flag is accepted and safely inert. Review-hardening (post-eval): fake-super replay applies the mode through the shared `metadata_mode_for_policy` helper (protocol 2.23.0: exactly the source mode under `-p`, with no masking); `--sparse` takes precedence over `--preallocate` (posix_fallocate skipped so holes survive) — **reversed by the parity-completion wave: `--preallocate` now wins, matching rsync**; `--partial` retention is disabled for `--no_replace` (ignore/existing) and only marks a write-attempt after the actual write begins; `--block-size=SIZE`/`--delta-block=SIZE` inline forms are accepted.
|
||||||
|
|
||||||
**Wave C — Devices & special files (finalize statuses + tests) (✅ implemented).** The four special-file rows are finalized with coverage tests. `--devices`, `--copy-devices`, and `--write-devices` are **✅ Implemented**, each with a documented, safety-driven divergence: device-node creation is privilege-gated, so a receiver without `CAP_MKNOD` skips that entry with a warning (a per-entry skip, never a transfer failure); `--copy-devices` copies a device/FIFO's reported size into an ordinary regular file (a size-bounded safe divergence from rsync's unbounded dd-like read); `--write-devices` writes only into an existing char/block node under the confined receive root and skips every unusable target rather than clobbering or aborting. `--specials` reclassified from **⛔ Impossible/Divergence** to **✅ Parity** in protocol 2.23.0: **FIFO recreation works** (unprivileged `mkfifo`) **and unix sockets are recreated** with `mknod(S_IFSOCK)`, which Linux permits unprivileged (the flag previously assumed sockets were impossible — see the `--specials` row). Tests assert FIFO recreation, socket recreation, the regular-file result of `--copy-devices`, the skipped/missing and non-device `--write-devices` targets, and (root-gated) real device-node creation; a root runner additionally drops the receiver to an unprivileged user to assert the `CAP_MKNOD` skip is graceful. (The parity-completion wave later reclassified `--devices`, `--copy-devices`, and `--write-devices` as explicit **❌ Divergent** rows, because their safe subsets are deliberately not rsync's behavior; the implementation itself is unchanged.)
|
**Wave C — Devices & special files (finalize statuses + tests) (✅ implemented).** The four special-file rows are finalized with coverage tests. `--devices`, `--copy-devices`, and `--write-devices` are **✅ Implemented**, each with a documented, safety-driven divergence: device-node creation is privilege-gated, so a receiver without `CAP_MKNOD` skips that entry with a warning (a per-entry skip, never a transfer failure); `--copy-devices` copies a device/FIFO's reported size into an ordinary regular file (a size-bounded safe divergence from rsync's unbounded dd-like read); `--write-devices` writes only into an existing char/block node under the confined receive root and skips every unusable target rather than clobbering or aborting. `--specials` reclassified from **⛔ Impossible/Divergence** to **✅ Parity** in protocol 2.23.0: **FIFO recreation works** (unprivileged `mkfifo`) **and unix sockets are recreated** with `mknod(S_IFSOCK)`, which Linux permits unprivileged (the flag previously assumed sockets were impossible — see the `--specials` row). Tests assert FIFO recreation, socket recreation, the regular-file result of `--copy-devices`, the skipped/missing and non-device `--write-devices` targets, and (root-gated) real device-node creation; a root runner additionally drops the receiver to an unprivileged user to assert the `CAP_MKNOD` skip is graceful. (The parity-completion wave later reclassified `--devices`, `--copy-devices`, and `--write-devices` as explicit **❌ Divergent** rows, because their safe subsets are deliberately not rsync's behavior; the implementation itself is unchanged.)
|
||||||
|
|
||||||
@@ -956,7 +969,7 @@ These are the last compatibility items and the closing phase toward rsync flag p
|
|||||||
|
|
||||||
**Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity.
|
**Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity.
|
||||||
|
|
||||||
**Honest status after the parity 2.29 cycle (protocol 2.28.0, no wire change), updated by the parity cycle 2.29 pass, the audit-cycle follow-ups, and the triage cycle.** ✅ Parity 117 / ⚠️ Caveat 13 / ❌ Divergent 27 = 157 rows. The 2.29 cycle closed the scanner-order, delete-timing, relative-basis, and fuzzy-eligibility residuals (moving `-n`/`--delete`/`--del`/`--delete-delay` to ✅) and improved the `--info`/`--stats`/`--debug` partial rows; the triage cycle moved `-F` and `-i`/`--itemize-changes` ✅ → ⚠️ for their documented residuals. The remaining ⚠️ rows are `--info`, `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `-i`, `--delete-before`, `--filter`, `-F`, the three basis-dir options, and `-y/--fuzzy`. Earlier: **Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP
|
**Honest status after the parity 2.29 cycle (protocol 2.29.0 since the symlink-xattr wire wave, which adds no config-frame field and leaves this matrix unchanged), updated by the parity cycle 2.29 pass, the audit-cycle follow-ups, the triage cycle, and a later no-wire parity pass.** The wire backlog cycle (protocol 2.30.0) then moved `--stderr=MODE` ❌ → ⚠️ (the `client` mode is now accepted over the new `STATUS_CLIENT_MSG` channel; only the client->server direction is reproduced) and closed the `--devices` exit-code and `--remove-source-files` residuals via `STATUS_PARTIAL` (exit 23 with successful sources removed), leaving ✅ Parity 119 / ⚠️ Caveat 15 / ❌ Divergent 23 = 157 rows. The same cycle then extended the destination-state report to directories and symlinks (#314: the receiver answers `STATUS_MKDIR`/`STATUS_SYMLINK` and an ancestor probe, so `-i`/`--progress`/`--out-format` render `.d..t......`/`cLc........` instead of `cd`/`cL` and suppress unchanged entries) and appended the per-type `deleted_*` counters to `STATUS_STATS` (#316: `--stats` now reproduces rsync's `Number of deleted files (reg/dir/link/special)` breakdown) — both differential-tested against rsync 3.4.1; the affected rows' caveats narrow but their classifications are unchanged, so the matrix stays **119 ✅ / 15 ⚠️ / 23 ❌ = 157**. The no-wire parity pass accepted `--inc-recursive`/`--no-inc-recursive` as inert no-ops (❌ → ✅, since FastSync's full scan is rsync's `--no-inc-recursive` and the destination is identical), narrowed the `--temp-dir` divergence by accepting an absolute path that canonicalizes inside the receive root (the row stays ❌ for out-of-root absolute paths), closed the `--delete-before` phase-0 divergence (⚠️ → ✅: both the single-threaded and the `--threads` data passes now replay the pre-scan file list, so a source file created after the scan is neither transferred nor kept, matching rsync), and moved `--fake-super` and `--devices` ❌ → ⚠️ (`--fake-super` now writes/reads rsync's exact `user.rsync.%stat` key and `<octal-mode> <rdev_major>,<rdev_minor> <uid>:<gid>` grammar, interoperating with real rsync 3.4.1 for regular files and faking char/block devices as regular files carrying the real rdev; `--devices` now logs a failed device `mknod` as a per-entry failure that continues the transfer instead of a silent non-root skip — see those rows for the remaining symlink-faking and exit-code residuals). A review pass then hardened the fake-super stat parser (strict range-checked parsing), made rsync-style daemon modules read-only by default with a startup warning for accepted-but-unenforced access-control keys, and extended the `--delete-before` replay to the `--threads` path. The 2.29 cycle closed the scanner-order, delete-timing, relative-basis, and fuzzy-eligibility residuals (moving `-n`/`--delete`/`--del`/`--delete-delay` to ✅) and improved the `--info`/`--stats`/`--debug` partial rows; the triage cycle moved `-F` and `-i`/`--itemize-changes` ✅ → ⚠️ for their documented residuals. The remaining ⚠️ rows are `--info`, `--debug`, `--stderr=MODE`, `--msgs2stderr`, `--stats`, `--progress`, `-i`, `--filter`, `-F`, the three basis-dir options, `-y/--fuzzy`, `--fake-super`, `--devices`, and `--delay-updates` (16). The wire cycle for issue #315 then closed the `--filter`/`-F` merge-modifier and per-directory-receiver residuals (protocol 2.30.0 carries each directory's compiled rules and implements `e`/`n`/`w`/`-`), narrowing both rows to the merge-file-side residual (rsync reads the destination's merge file, FastSync carries the source's) and leaving the tally at **119 ✅ / 15 ⚠️ / 23 ❌ = 157**; the #317 pass then moved `--delay-updates` ❌ → ⚠️, for **119 ✅ / 16 ⚠️ / 22 ❌ = 157**. Earlier: **Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot tLine truncated
|
||||||
and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel /
|
and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel /
|
||||||
receiver filter engine); the wire parity-track-4a pass later added that
|
receiver filter engine); the wire parity-track-4a pass later added that
|
||||||
receiver filter engine, flipping `--filter=RULE` back to ✅ (see above; the
|
receiver filter engine, flipping `--filter=RULE` back to ✅ (see above; the
|
||||||
@@ -1021,7 +1034,9 @@ integration tests unless it is explicitly listed as a limitation.
|
|||||||
### Filesystem and deletion semantics
|
### Filesystem and deletion semantics
|
||||||
|
|
||||||
- **`--temp-dir` is confined to the receive root on the receiver:** a relative
|
- **`--temp-dir` is confined to the receive root on the receiver:** a relative
|
||||||
dir resolves below it; an absolute path or one containing `..` is rejected.
|
dir resolves below it, and an absolute path is accepted only when
|
||||||
|
`realpath(3)` confirms it is inside the canonical receive root; an
|
||||||
|
out-of-root absolute path or one containing `..` is rejected.
|
||||||
An `EXDEV` install falls back to a non-atomic copy instead of aborting. (The
|
An `EXDEV` install falls back to a non-atomic copy instead of aborting. (The
|
||||||
confined receiver path cannot be mount-tested in the CI container — no
|
confined receiver path cannot be mount-tested in the CI container — no
|
||||||
`CAP_SYS_ADMIN` and unprivileged user namespaces are disabled — so the
|
`CAP_SYS_ADMIN` and unprivileged user namespaces are disabled — so the
|
||||||
@@ -1057,9 +1072,12 @@ integration tests unless it is explicitly listed as a limitation.
|
|||||||
clear configuration error (matching rsync) instead of an order-dependent
|
clear configuration error (matching rsync) instead of an order-dependent
|
||||||
winner.
|
winner.
|
||||||
- **`--fake-super` never real-chowns.** It records the *resolved* owner (the
|
- **`--fake-super` never real-chowns.** It records the *resolved* owner (the
|
||||||
active mapping, else the source id) in `user.fastsync.stat` for a later
|
active mapping, else the source id) in rsync's `user.rsync.%stat` for a later
|
||||||
privileged restore and replays only mode/times. Directory ownership and
|
privileged restore and replays only the permission bits (mtime travels through
|
||||||
directory xattrs/ACLs are preserved alongside file entries.
|
the normal metadata path). Directory ownership, xattrs/ACLs and the
|
||||||
|
directory's own `%stat` record are preserved alongside file entries: the
|
||||||
|
record is written on the directory at creation and re-stamped by the deferred
|
||||||
|
directory-metadata pass.
|
||||||
- **`--chmod`** implements rsync's `D`/`F`/`X` selectors, `s`/`t`, append
|
- **`--chmod`** implements rsync's `D`/`F`/`X` selectors, `s`/`t`, append
|
||||||
semantics, does not imply `-p`, and applies its changes without sanitization.
|
semantics, does not imply `-p`, and applies its changes without sanitization.
|
||||||
|
|
||||||
@@ -1095,8 +1113,9 @@ These remain after the wave; they are the reasons a row above is ⚠️.
|
|||||||
link-following tool can follow a link outside the receive root. Use
|
link-following tool can follow a link outside the receive root. Use
|
||||||
`--safe-links` when the source is untrusted. `--trust-sender` does **not**
|
`--safe-links` when the source is untrusted. `--trust-sender` does **not**
|
||||||
affect symlink targets.
|
affect symlink targets.
|
||||||
- **`--temp-dir` absolute/foreign-filesystem paths are rejected by the
|
- **`--temp-dir` out-of-root absolute and foreign-filesystem paths are rejected
|
||||||
receiver** (rsync's daemon also confines; standalone rsync differs).
|
by the receiver** (an absolute path that canonicalizes inside the receive root
|
||||||
|
is accepted; rsync's daemon also confines; standalone rsync differs).
|
||||||
- **`--copy-devices` reads a bounded `st_size`** rather than rsync's unbounded
|
- **`--copy-devices` reads a bounded `st_size`** rather than rsync's unbounded
|
||||||
device read.
|
device read.
|
||||||
- **A broken symlink referent under `--copy-links`/`--copy-unsafe-links` exits 0**
|
- **A broken symlink referent under `--copy-links`/`--copy-unsafe-links` exits 0**
|
||||||
@@ -1194,9 +1213,12 @@ wire protocol three times (full rationale in `src/shared/config.h`):
|
|||||||
modifiers; `-f` is bound to `--filter`; a single `-F` transfers
|
modifiers; `-f` is bound to `--filter`; a single `-F` transfers
|
||||||
`.rsync-filter` and `-FF` excludes it. The xattr-name `x` modifier is **not
|
`.rsync-filter` and `-FF` excludes it. The xattr-name `x` modifier is **not
|
||||||
implemented** and is rejected with a clear error everywhere. The merge-only
|
implemented** and is rejected with a clear error everywhere. The merge-only
|
||||||
`e`/`n`/`w` and `-` modifiers are accepted and consumed on `merge`/`dir-merge`
|
`e`/`n`/`w` and `-` modifiers are implemented on `merge`/`dir-merge` rules
|
||||||
rules (rejected elsewhere, matching rsync), but their semantics are **not
|
(rejected elsewhere, matching rsync). Per-directory rules are carried to the
|
||||||
implemented** (accepted-but-ignored).
|
receiver (protocol 2.30.0) and re-applied deepest-first before the
|
||||||
|
command-line base, so a destination-only entry matching only a per-directory
|
||||||
|
rule is shielded; the residual is that rsync reads the destination's merge
|
||||||
|
file while FastSync carries the source's.
|
||||||
- **Absolute basis directories** are used verbatim (rsync semantics) and
|
- **Absolute basis directories** are used verbatim (rsync semantics) and
|
||||||
**`--link-dest`** relinks an already up-to-date destination.
|
**`--link-dest`** relinks an already up-to-date destination.
|
||||||
|
|
||||||
@@ -1229,14 +1251,19 @@ These remain after the wave; the individual rows carry the precise wording.
|
|||||||
rsync's generator removes all extras ahead of its throttled sender while
|
rsync's generator removes all extras ahead of its throttled sender while
|
||||||
FastSync removes only the reached directories (completed runs agree).
|
FastSync removes only the reached directories (completed runs agree).
|
||||||
`--delete-before` keeps its pre-scan snapshot race; and while
|
`--delete-before` keeps its pre-scan snapshot race; and while
|
||||||
base-rule `protect`/`risk` rules are now re-applied on the receiver (track 4a),
|
base-rule `protect`/`risk` rules and the per-directory merge rules are now
|
||||||
per-directory merge (`.rsync-filter`) protection is still sender-derived, so a
|
re-applied on the receiver (protocol 2.30.0), so a destination-only entry
|
||||||
destination-only entry matching only a per-directory rule is not re-derived.
|
matching only a per-directory rule is re-derived; the residual is the side
|
||||||
|
the merge file is read from (rsync reads the destination's, FastSync carries
|
||||||
|
the source's).
|
||||||
`--ignore-errors` exits 23 but its EACCES differential is not
|
`--ignore-errors` exits 23 but its EACCES differential is not
|
||||||
exercised in CI.
|
exercised in CI.
|
||||||
- **`--delay-updates`** uses a fixed staging name with an advisory lock and
|
- **`--delay-updates`** stages under a per-run unique name (refusing to wipe a
|
||||||
deletes before publication; **`--temp-dir`** rejects absolute/foreign paths
|
same-named destination entry) and publishes before its implied
|
||||||
(deliberately confined, see the row); **`--remote-option`** is SSH-only.
|
`--delete-after` commit; it does not create a backup of a deleted extra.
|
||||||
|
**`--temp-dir`** rejects out-of-root absolute and
|
||||||
|
foreign paths (in-root absolute paths are accepted; deliberately confined, see
|
||||||
|
the row); **`--remote-option`** is SSH-only.
|
||||||
**`--iconv`** now matches rsync's push direction (destination charset = the
|
**`--iconv`** now matches rsync's push direction (destination charset = the
|
||||||
spec's REMOTE half; a server `--iconv` overrides it).
|
spec's REMOTE half; a server `--iconv` overrides it).
|
||||||
- **Basis dirs** now use rsync's metadata quick-check by default (track 5a) and
|
- **Basis dirs** now use rsync's metadata quick-check by default (track 5a) and
|
||||||
@@ -1247,15 +1274,17 @@ These remain after the wave; the individual rows carry the precise wording.
|
|||||||
engine (both files ≥ 16 KiB, size ratio ≤ 10×), a narrower window than
|
engine (both files ≥ 16 KiB, size ratio ≤ 10×), a narrower window than
|
||||||
rsync's, so the selected basis — and the `--stats` bandwidth counters —
|
rsync's, so the selected basis — and the `--stats` bandwidth counters —
|
||||||
can differ while the tree stays byte-exact.
|
can differ while the tree stays byte-exact.
|
||||||
- **`--inc-recursive`/`--no-inc-recursive`** are not implemented (rejected).
|
- **`--inc-recursive`/`--no-inc-recursive`** are accepted as inert no-ops:
|
||||||
|
FastSync always performs a single full recursive scan (equivalent to
|
||||||
|
rsync's `--no-inc-recursive`), so the destination is identical either way.
|
||||||
|
|
||||||
### Intentional divergences (explicit ❌ rows)
|
### Intentional divergences (explicit ❌ rows)
|
||||||
|
|
||||||
Native daemon config/auth (`--daemon`, `--config`, `--dparam`,
|
Native daemon config/auth (`--daemon`, `--config`, `--dparam`,
|
||||||
`--password-file`, `--early-input`, `--hash-credentials`/`--iterations`), the
|
`--password-file`, `--early-input`, `--hash-credentials`/`--iterations`), the
|
||||||
non-interoperable batch container (`--write-batch`/`--only-write-batch`/
|
non-interoperable batch container (`--write-batch`/`--only-write-batch`/
|
||||||
`--read-batch`), `--fake-super`'s native xattr format, `-X`'s privileged
|
`--read-batch`), `-X`'s privileged
|
||||||
namespaces, `--devices`/`--copy-devices`/`--write-devices`'s safe subsets,
|
namespaces, `--copy-devices`/`--write-devices`'s safe subsets,
|
||||||
`--super`/`--copy-as`'s refusal to elevate or switch credentials, and the
|
`--super`/`--copy-as`'s refusal to elevate or switch credentials, and the
|
||||||
`-s`/`--secluded-args`/`--protect-args`/`--old-args` accepted no-ops.
|
`-s`/`--secluded-args`/`--protect-args`/`--old-args` accepted no-ops.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
nested
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
nested
|
||||||
@@ -123,8 +123,15 @@ static char itemize_type_char(const ChangeEvent* event) {
|
|||||||
static bool times_match(const Config* config, const ChangeEvent* event) {
|
static bool times_match(const Config* config, const ChangeEvent* event) {
|
||||||
if (!event->dest.known || !event->dest.existed)
|
if (!event->dest.known || !event->dest.existed)
|
||||||
return false;
|
return false;
|
||||||
if (event->mtime_sec == event->dest.mtime_sec)
|
if (event->mtime_sec == event->dest.mtime_sec) {
|
||||||
|
/* A regular file's sub-second mtime IS preserved by the receiver, so an nsec
|
||||||
|
difference is a real change. A directory or symlink has no preserved
|
||||||
|
sub-second mtime (rsync's quick-check compares whole seconds there), so a
|
||||||
|
nanosecond-only difference must not render a spurious `.d..t` / `.L..t`. */
|
||||||
|
if (event->is_directory || event->is_symlink || event->is_special)
|
||||||
|
return true;
|
||||||
return event->mtime_nsec == event->dest.mtime_nsec;
|
return event->mtime_nsec == event->dest.mtime_nsec;
|
||||||
|
}
|
||||||
long long delta = (long long)event->mtime_sec - (long long)event->dest.mtime_sec;
|
long long delta = (long long)event->mtime_sec - (long long)event->dest.mtime_sec;
|
||||||
if (delta < 0)
|
if (delta < 0)
|
||||||
delta = -delta;
|
delta = -delta;
|
||||||
@@ -145,6 +152,10 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
|
|||||||
/* rsync: an existing directory that only has attribute changes carries no
|
/* rsync: an existing directory that only has attribute changes carries no
|
||||||
transfer, so the update column is `.` rather than `>`. */
|
transfer, so the update column is `.` rather than `>`. */
|
||||||
update = '.';
|
update = '.';
|
||||||
|
else if (event->is_symlink)
|
||||||
|
/* rsync: an existing symlink whose target is unchanged is a `.` update
|
||||||
|
(attributes only); a changed target is `c` (the link value changed). */
|
||||||
|
update = event->dest.target_matches ? '.' : 'c';
|
||||||
else
|
else
|
||||||
update = '>';
|
update = '>';
|
||||||
code[0] = update;
|
code[0] = update;
|
||||||
@@ -155,12 +166,20 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
|
|||||||
code[11] = '\0';
|
code[11] = '\0';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
bool size_diff = event->size != event->dest.size;
|
/* rsync's value/checksum column: `c` for a symlink whose target changed (the
|
||||||
bool time_diff = !times_match(config, event);
|
link value is the compared content); no destination digest is available for
|
||||||
|
a regular file. */
|
||||||
|
bool value_diff = event->is_symlink && !event->dest.target_matches;
|
||||||
|
/* rsync itemizes size only for regular files: a directory's st_size and a
|
||||||
|
symlink's target length are not compared. */
|
||||||
|
bool size_diff = !event->is_directory && !event->is_symlink && !event->is_special &&
|
||||||
|
event->size != event->dest.size;
|
||||||
|
/* rsync itemizes the time column only when -t/--times is in effect. */
|
||||||
|
bool time_diff = config->preserve_times && !times_match(config, event);
|
||||||
bool perms_diff = (event->mode & 07777) != (event->dest.mode & 07777);
|
bool perms_diff = (event->mode & 07777) != (event->dest.mode & 07777);
|
||||||
bool owner_diff = event->uid != (uid_t)event->dest.uid;
|
bool owner_diff = event->uid != (uid_t)event->dest.uid;
|
||||||
bool group_diff = event->gid != (gid_t)event->dest.gid;
|
bool group_diff = event->gid != (gid_t)event->dest.gid;
|
||||||
code[2] = '.'; /* checksum: no destination digest available */
|
code[2] = value_diff ? 'c' : '.';
|
||||||
code[3] = size_diff ? 's' : '.';
|
code[3] = size_diff ? 's' : '.';
|
||||||
code[4] = time_diff ? 't' : '.';
|
code[4] = time_diff ? 't' : '.';
|
||||||
code[5] = (config->preserve_perms && perms_diff) ? 'p' : '.';
|
code[5] = (config->preserve_perms && perms_diff) ? 'p' : '.';
|
||||||
@@ -172,6 +191,24 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
|
|||||||
code[11] = '\0';
|
code[11] = '\0';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* True when the itemized destination entry is unchanged, i.e. rsync would print
|
||||||
|
* no line at all. Reuses itemize_code so suppression is exactly consistent
|
||||||
|
* with what would have been rendered: the update column must be `.` and every
|
||||||
|
* attribute column must be `.`. */
|
||||||
|
static bool itemize_is_unchanged(const Config* config, const ChangeEvent* event) {
|
||||||
|
if (!event->dest.known || !event->dest.existed)
|
||||||
|
return false;
|
||||||
|
char code[12];
|
||||||
|
itemize_code(config, event, code);
|
||||||
|
if (code[0] != '.')
|
||||||
|
return false;
|
||||||
|
for (int i = 2; i < 11; i++) {
|
||||||
|
if (code[i] != '.')
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
/* rsync %n: the transfer-relative name, with a trailing slash for directories.
|
/* rsync %n: the transfer-relative name, with a trailing slash for directories.
|
||||||
* The transfer root is `.` (so `%n` renders `./`), matching rsync's root entry. */
|
* The transfer root is `.` (so `%n` renders `./`), matching rsync's root entry. */
|
||||||
static bool append_name(StrBuf* buf, const ChangeEvent* event) {
|
static bool append_name(StrBuf* buf, const ChangeEvent* event) {
|
||||||
@@ -678,6 +715,19 @@ void change_emit_file_sent_bytes(const Config* config, const File* file,
|
|||||||
char* name = NULL;
|
char* name = NULL;
|
||||||
char* path = NULL;
|
char* path = NULL;
|
||||||
fill_event_from_file(config, file, &event, &name, &path);
|
fill_event_from_file(config, file, &event, &name, &path);
|
||||||
|
if (file->is_symlink) {
|
||||||
|
/* Output parity (protocol 2.30.0): an unchanged symlink is silent, like
|
||||||
|
rsync's quick check. The itemize/log stream suppresses it only when every
|
||||||
|
attribute matches; the name stream suppresses it whenever the link target
|
||||||
|
is unchanged (rsync names a symlink only when it relinks or creates it). */
|
||||||
|
bool itemize_output = config->itemize_changes || config->out_format != NULL ||
|
||||||
|
(config->log_file != NULL && config->log_file_format != NULL);
|
||||||
|
bool suppress = itemize_output
|
||||||
|
? itemize_is_unchanged(config, &event)
|
||||||
|
: (event.dest.known && event.dest.existed && event.dest.target_matches);
|
||||||
|
if (suppress)
|
||||||
|
event.decision = CHANGE_UP_TO_DATE;
|
||||||
|
}
|
||||||
if (name != NULL && path != NULL) {
|
if (name != NULL && path != NULL) {
|
||||||
fill_event_checksum(config, file, &event);
|
fill_event_checksum(config, file, &event);
|
||||||
change_emit(config, &event);
|
change_emit(config, &event);
|
||||||
@@ -729,6 +779,19 @@ void change_emit_dir_sent(const Config* config, const File* file) {
|
|||||||
char* name = NULL;
|
char* name = NULL;
|
||||||
char* path = NULL;
|
char* path = NULL;
|
||||||
fill_event_from_file(config, file, &event, &name, &path);
|
fill_event_from_file(config, file, &event, &name, &path);
|
||||||
|
/* Output parity (protocol 2.30.0): suppress a directory rsync would leave
|
||||||
|
silent. The itemize/log stream suppresses it only when every attribute
|
||||||
|
matches (`.d.........`); the name stream suppresses any pre-existing
|
||||||
|
directory (rsync names a directory only when it is created). */
|
||||||
|
bool itemize_output = config->itemize_changes || config->out_format != NULL ||
|
||||||
|
(config->log_file != NULL && config->log_file_format != NULL);
|
||||||
|
bool suppress = itemize_output ? itemize_is_unchanged(config, &event)
|
||||||
|
: (event.dest.known && event.dest.existed);
|
||||||
|
/* The transfer root's line is an unconditional FastSync residual (rsync keys
|
||||||
|
it off the root's own attribute change); keep emitting it. */
|
||||||
|
bool is_root = event.name != NULL && event.name[0] == '\0';
|
||||||
|
if (suppress && !is_root)
|
||||||
|
event.decision = CHANGE_UP_TO_DATE;
|
||||||
if (name != NULL && path != NULL)
|
if (name != NULL && path != NULL)
|
||||||
change_emit(config, &event);
|
change_emit(config, &event);
|
||||||
free(name);
|
free(name);
|
||||||
|
|||||||
+34
-10
@@ -433,12 +433,10 @@ static int set_stderr_mode(const char* value) {
|
|||||||
log_set_stderr_mode(LOG_STDERR_ERRORS);
|
log_set_stderr_mode(LOG_STDERR_ERRORS);
|
||||||
else if (strcmp(value, "all") == 0 || strcmp(value, "a") == 0)
|
else if (strcmp(value, "all") == 0 || strcmp(value, "a") == 0)
|
||||||
log_set_stderr_mode(LOG_STDERR_ALL);
|
log_set_stderr_mode(LOG_STDERR_ALL);
|
||||||
else if (strcmp(value, "client") == 0 || strcmp(value, "c") == 0) {
|
else if (strcmp(value, "client") == 0 || strcmp(value, "c") == 0)
|
||||||
log_message(LOG_LEVEL_ERROR,
|
log_set_stderr_mode(LOG_STDERR_CLIENT);
|
||||||
"--stderr=client is not supported: FastSync has no client message channel");
|
else {
|
||||||
return -1;
|
log_message(LOG_LEVEL_ERROR, "--stderr must be errors, all, or client");
|
||||||
} else {
|
|
||||||
log_message(LOG_LEVEL_ERROR, "--stderr must be errors or all");
|
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
return 0;
|
return 0;
|
||||||
@@ -962,6 +960,13 @@ static const OptionEntry OPTION_TABLE[] = {
|
|||||||
/* rsync -r/--recursive: FastSync is always recursive, so this is a
|
/* rsync -r/--recursive: FastSync is always recursive, so this is a
|
||||||
* faithful no-op (accepted silently, never consumes an argument). */
|
* faithful no-op (accepted silently, never consumes an argument). */
|
||||||
{"--recursive", "-r", OPT_NOOP, 0},
|
{"--recursive", "-r", OPT_NOOP, 0},
|
||||||
|
/* rsync's incremental-recursion scan-mode switch. FastSync always performs
|
||||||
|
* a single full recursive scan, so both spellings are accepted as no-ops:
|
||||||
|
* the destination is identical whichever mode the caller requests.
|
||||||
|
* --no-inc-recursive is handled before the generic --no-* negation branch
|
||||||
|
* (see cli_handle_pre_negation) but is registered here for discoverability. */
|
||||||
|
{"--inc-recursive", NULL, OPT_NOOP, 0},
|
||||||
|
{"--no-inc-recursive", NULL, OPT_NOOP, 0},
|
||||||
{"--update", "-u", OPT_FLAG, offsetof(Config, update)},
|
{"--update", "-u", OPT_FLAG, offsetof(Config, update)},
|
||||||
/* rsync's --old-args: accepted for CLI compatibility as a documented no-op
|
/* rsync's --old-args: accepted for CLI compatibility as a documented no-op
|
||||||
* (the remote server path is always safely quoted; see usage.c). It is
|
* (the remote server path is always safely quoted; see usage.c). It is
|
||||||
@@ -1346,10 +1351,9 @@ static bool cli_handle_pre_negation(CliParseCtx* ctx) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
/* "--no-msgs2stderr" is the deprecated spelling of --stderr=client (rsync
|
/* "--no-msgs2stderr" is the deprecated spelling of --stderr=client (rsync
|
||||||
* 3.4.1). FastSync has no separate client message channel, so the closest
|
* 3.4.1); the client-message channel now exists, so it maps to `client`. */
|
||||||
* supported mode is the errors-only default. */
|
|
||||||
if (strcmp(arg, "--no-msgs2stderr") == 0)
|
if (strcmp(arg, "--no-msgs2stderr") == 0)
|
||||||
return set_stderr_mode("errors") == 0;
|
return set_stderr_mode("client") == 0;
|
||||||
/* "--no-motd" is a real rsync option name (client-side daemon MOTD display
|
/* "--no-motd" is a real rsync option name (client-side daemon MOTD display
|
||||||
* suppression), not a negation of a "--motd" flag, so it is handled before
|
* suppression), not a negation of a "--motd" flag, so it is handled before
|
||||||
* the generic --no-* negation branch. */
|
* the generic --no-* negation branch. */
|
||||||
@@ -1387,6 +1391,12 @@ static bool cli_handle_pre_negation(CliParseCtx* ctx) {
|
|||||||
ctx->no_delta = true;
|
ctx->no_delta = true;
|
||||||
else if (strcmp(arg, "--no-incremental") == 0)
|
else if (strcmp(arg, "--no-incremental") == 0)
|
||||||
ctx->no_incremental = true;
|
ctx->no_incremental = true;
|
||||||
|
/* Real rsync option names that merely start with "--no-" and are inert
|
||||||
|
* no-ops (e.g. --no-inc-recursive) are registered as OPT_NOOP entries;
|
||||||
|
* accept them before the generic negation table would reject the name. */
|
||||||
|
const OptionEntry* noop = find_table_option(arg);
|
||||||
|
if (noop && noop->kind == OPT_NOOP)
|
||||||
|
return true;
|
||||||
if (apply_negation(config, arg) != 0) {
|
if (apply_negation(config, arg) != 0) {
|
||||||
ctx->exit_code = -1;
|
ctx->exit_code = -1;
|
||||||
return true;
|
return true;
|
||||||
@@ -2616,6 +2626,17 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
|
|||||||
bool debug_enabled = verbose || config->debug_level != 0;
|
bool debug_enabled = verbose || config->debug_level != 0;
|
||||||
set_log_level(config->quiet ? LOG_LEVEL_ERROR
|
set_log_level(config->quiet ? LOG_LEVEL_ERROR
|
||||||
: (debug_enabled ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
|
: (debug_enabled ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
|
||||||
|
/* rsync parity: --delay-updates implies --delete-after. Every staged file is
|
||||||
|
published first and only then are extras removed. Normalize onto the
|
||||||
|
existing delete_after wire bool (no new wire field), overriding any other
|
||||||
|
explicit timing exactly as rsync does; without --delete there is no
|
||||||
|
deletion, so no timing is set (and the wire config stays valid). */
|
||||||
|
if (config->delay_updates && config->use_delete) {
|
||||||
|
config->delete_before = false;
|
||||||
|
config->delete_during = false;
|
||||||
|
config->delete_delay = false;
|
||||||
|
config->delete_after = true;
|
||||||
|
}
|
||||||
/* rsync's plain --delete defaults to delete-during (--del): each directory's
|
/* rsync's plain --delete defaults to delete-during (--del): each directory's
|
||||||
extras are removed as that directory is processed, so space is freed
|
extras are removed as that directory is processed, so space is freed
|
||||||
progressively and a tight destination never has to hold the whole old+new
|
progressively and a tight destination never has to hold the whole old+new
|
||||||
@@ -2797,8 +2818,11 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
|
|||||||
* need the pre-transfer destination snapshot (new vs modified and which
|
* need the pre-transfer destination snapshot (new vs modified and which
|
||||||
* attributes differ), so ask the receiver to report it on every per-file
|
* attributes differ), so ask the receiver to report it on every per-file
|
||||||
* check. This is a wire field. */
|
* check. This is a wire field. */
|
||||||
|
bool progress_active =
|
||||||
|
!config->quiet && (config->show_progress || (config->info_level & LOG_INFO_PROGRESS) != 0);
|
||||||
config->report_dest_info = config->itemize_changes || config->out_format != NULL ||
|
config->report_dest_info = config->itemize_changes || config->out_format != NULL ||
|
||||||
(config->log_file != NULL && config->log_file_format != NULL);
|
(config->log_file != NULL && config->log_file_format != NULL) ||
|
||||||
|
progress_active;
|
||||||
/* Wire-stats parity: --stats, --progress/-P, an --out-format token that needs
|
/* Wire-stats parity: --stats, --progress/-P, an --out-format token that needs
|
||||||
* a wire counter (%b/%c), or a dry-run --delete need the receiver's
|
* a wire counter (%b/%c), or a dry-run --delete need the receiver's
|
||||||
* end-of-transfer STATUS_STATS report. This is a wire field (protocol
|
* end-of-transfer STATUS_STATS report. This is a wire field (protocol
|
||||||
|
|||||||
@@ -64,16 +64,8 @@ bool add_chunk_to_manifest(ArrayList* manifest, const Chunk* chunk) {
|
|||||||
int send_dry_run_manifest(const Config* config) {
|
int send_dry_run_manifest(const Config* config) {
|
||||||
int skipped = 0;
|
int skipped = 0;
|
||||||
ArrayList* missing_dest = NULL;
|
ArrayList* missing_dest = NULL;
|
||||||
if (config->delete_missing_args) {
|
if (!client_prepare_files_from(config, &missing_dest, &skipped))
|
||||||
missing_dest = array_list_create(free);
|
|
||||||
if (!missing_dest)
|
|
||||||
return -1;
|
return -1;
|
||||||
}
|
|
||||||
if (!files_from_list_check(config, missing_dest, &skipped)) {
|
|
||||||
if (missing_dest)
|
|
||||||
array_list_delete(missing_dest);
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
PreparedScanner prepared;
|
PreparedScanner prepared;
|
||||||
if (!prepare_scanner(config, 0, &prepared)) {
|
if (!prepare_scanner(config, 0, &prepared)) {
|
||||||
if (missing_dest)
|
if (missing_dest)
|
||||||
@@ -330,9 +322,11 @@ int send_list_only(const Config* config) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/* Send the delete manifest to the server. Returns 0 on success, -1 on
|
/* Send the delete manifest to the server. Returns 0 on success, -1 on
|
||||||
failure. It carries FOUR sections: the keep-set paths, the protected
|
failure. It carries FOUR path sections (keep-set paths, protected excluded
|
||||||
excluded prefixes, the --delete-missing-args exact-delete paths, and the
|
prefixes, --delete-missing-args exact-delete paths, and the destination-
|
||||||
destination-relative directories the sender synchronized this run.
|
relative directories the sender synchronized this run) followed by the
|
||||||
|
protocol-2.30.0 per-directory filter-rule block (`per_dir_rules`, the rules
|
||||||
|
the scan compiled from each directory's merge files).
|
||||||
When --delete-excluded is given `protected` is empty: excluded destination
|
When --delete-excluded is given `protected` is empty: excluded destination
|
||||||
mirrors are then ordinary extras and are removed. When
|
mirrors are then ordinary extras and are removed. When
|
||||||
--delete-missing-args is active `missing_args` holds the destination mirrors
|
--delete-missing-args is active `missing_args` holds the destination mirrors
|
||||||
@@ -346,7 +340,8 @@ int send_list_only(const Config* config) {
|
|||||||
frame. A heavily filtered source whose exclusion list is large therefore
|
frame. A heavily filtered source whose exclusion list is large therefore
|
||||||
fails the run cleanly on the receiver rather than being truncated. */
|
fails the run cleanly on the receiver rather than being truncated. */
|
||||||
int send_delete_manifest(int fd, ArrayList* manifest, ArrayList* protected_prefixes,
|
int send_delete_manifest(int fd, ArrayList* manifest, ArrayList* protected_prefixes,
|
||||||
ArrayList* size_skipped, ArrayList* missing_args, ArrayList* synced_dirs) {
|
ArrayList* size_skipped, ArrayList* missing_args, ArrayList* synced_dirs,
|
||||||
|
const FilterRuleList* per_dir_rules) {
|
||||||
if (!send_status(fd, STATUS_MANIFEST))
|
if (!send_status(fd, STATUS_MANIFEST))
|
||||||
return -1;
|
return -1;
|
||||||
int keep_count = manifest ? manifest->size : 0;
|
int keep_count = manifest ? manifest->size : 0;
|
||||||
@@ -389,6 +384,11 @@ int send_delete_manifest(int fd, ArrayList* manifest, ArrayList* protected_prefi
|
|||||||
if (!send_wire_str(fd, (char*)synced_dirs->items[i]))
|
if (!send_wire_str(fd, (char*)synced_dirs->items[i]))
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
|
/* Protocol 2.30.0: the receiver-side per-directory filter rules discovered by
|
||||||
|
the sender's scan, so the whole-tree commit walker can shield a
|
||||||
|
destination-only entry that matches only a per-directory merge rule. */
|
||||||
|
if (!delete_filter_dir_rules_send(fd, per_dir_rules))
|
||||||
|
return -1;
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -409,11 +409,11 @@ int send_delete_manifest(int fd, ArrayList* manifest, ArrayList* protected_prefi
|
|||||||
|
|
||||||
bool send_delete_manifest_early(Client* client, ArrayList* manifest, ArrayList* protected_prefixes,
|
bool send_delete_manifest_early(Client* client, ArrayList* manifest, ArrayList* protected_prefixes,
|
||||||
ArrayList* size_skipped, ArrayList* missing_args,
|
ArrayList* size_skipped, ArrayList* missing_args,
|
||||||
ArrayList* synced_dirs) {
|
ArrayList* synced_dirs, const FilterRuleList* per_dir_rules) {
|
||||||
if (!client || !manifest)
|
if (!client || !manifest)
|
||||||
return false;
|
return false;
|
||||||
if (send_delete_manifest(client->file_descriptor, manifest, protected_prefixes, size_skipped,
|
if (send_delete_manifest(client->file_descriptor, manifest, protected_prefixes, size_skipped,
|
||||||
missing_args, synced_dirs) != 0)
|
missing_args, synced_dirs, per_dir_rules) != 0)
|
||||||
return false;
|
return false;
|
||||||
Status ack;
|
Status ack;
|
||||||
/* The wait is long (up to an hour) and runs inline on this thread: a helper
|
/* The wait is long (up to an hour) and runs inline on this thread: a helper
|
||||||
@@ -458,35 +458,21 @@ bool send_delete_manifest_early(Client* client, ArrayList* manifest, ArrayList*
|
|||||||
int send_dry_run_remote(Config* config) {
|
int send_dry_run_remote(Config* config) {
|
||||||
int from_skipped = 0;
|
int from_skipped = 0;
|
||||||
ArrayList* missing_args = NULL;
|
ArrayList* missing_args = NULL;
|
||||||
if (config->delete_missing_args) {
|
if (!client_prepare_files_from(config, &missing_args, &from_skipped))
|
||||||
missing_args = array_list_create(free);
|
|
||||||
if (!missing_args)
|
|
||||||
return 1;
|
return 1;
|
||||||
}
|
|
||||||
if (!files_from_list_check(config, missing_args, &from_skipped)) {
|
|
||||||
if (missing_args)
|
|
||||||
array_list_delete(missing_args);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
if (missing_args)
|
if (missing_args)
|
||||||
array_list_delete(missing_args);
|
array_list_delete(missing_args);
|
||||||
/* A live session may follow, so arm graceful abort handling. */
|
/* A live session may follow, so arm graceful abort handling. */
|
||||||
client_set_abort_armed(true);
|
client_set_abort_armed(true);
|
||||||
Client* client = connect_transfer_client(config);
|
ProtocolSession session;
|
||||||
|
Client* client = client_connect_and_bind_session(config, &session);
|
||||||
if (!client) {
|
if (!client) {
|
||||||
if (config->transport == TRANSPORT_TCP)
|
|
||||||
log_message(LOG_LEVEL_ERROR, "could not connect to server%s",
|
|
||||||
config->use_tls ? " via TLS" : "");
|
|
||||||
client_set_abort_armed(false);
|
client_set_abort_armed(false);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
ProtocolSession session;
|
|
||||||
protocol_session_init(&session, client->file_descriptor, client->file_descriptor);
|
|
||||||
protocol_session_set_io_timeout(&session, config->timeout);
|
|
||||||
protocol_session_set_ssl(&session, (SSL*)client->ssl);
|
|
||||||
protocol_session_bind(&session);
|
|
||||||
|
|
||||||
int ret = 1;
|
int ret = 1;
|
||||||
|
bool partial = false;
|
||||||
time_t dry_start = time(NULL);
|
time_t dry_start = time(NULL);
|
||||||
ReceiverStats dry_stats;
|
ReceiverStats dry_stats;
|
||||||
memset(&dry_stats, 0, sizeof(dry_stats));
|
memset(&dry_stats, 0, sizeof(dry_stats));
|
||||||
@@ -497,6 +483,7 @@ int send_dry_run_remote(Config* config) {
|
|||||||
ArrayList* dry_dirs = NULL;
|
ArrayList* dry_dirs = NULL;
|
||||||
ArrayList* dry_excluded = NULL;
|
ArrayList* dry_excluded = NULL;
|
||||||
ArrayList* dry_size_skipped = NULL;
|
ArrayList* dry_size_skipped = NULL;
|
||||||
|
FilterRuleList* dry_per_dir = NULL;
|
||||||
if (!config_send(client->file_descriptor, config))
|
if (!config_send(client->file_descriptor, config))
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
receive_daemon_motd(client, config);
|
receive_daemon_motd(client, config);
|
||||||
@@ -509,8 +496,10 @@ int send_dry_run_remote(Config* config) {
|
|||||||
dry_manifest = array_list_create(free);
|
dry_manifest = array_list_create(free);
|
||||||
dry_dirs = array_list_create(free);
|
dry_dirs = array_list_create(free);
|
||||||
dry_size_skipped = array_list_create(free);
|
dry_size_skipped = array_list_create(free);
|
||||||
if (!dry_manifest || !dry_dirs || !dry_size_skipped)
|
dry_per_dir = filter_rule_list_create();
|
||||||
|
if (!dry_manifest || !dry_dirs || !dry_size_skipped || !dry_per_dir)
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
|
prepared.options.per_dir_rules = dry_per_dir;
|
||||||
if (!config->delete_excluded) {
|
if (!config->delete_excluded) {
|
||||||
dry_excluded = array_list_create(free);
|
dry_excluded = array_list_create(free);
|
||||||
if (!dry_excluded)
|
if (!dry_excluded)
|
||||||
@@ -612,7 +601,7 @@ int send_dry_run_remote(Config* config) {
|
|||||||
bool early_delete = config->use_delete && config_delete_timing_early(config);
|
bool early_delete = config->use_delete && config_delete_timing_early(config);
|
||||||
if (dry_manifest) {
|
if (dry_manifest) {
|
||||||
if (send_delete_manifest(client->file_descriptor, dry_manifest, dry_excluded, dry_size_skipped,
|
if (send_delete_manifest(client->file_descriptor, dry_manifest, dry_excluded, dry_size_skipped,
|
||||||
NULL, dry_dirs) != 0)
|
NULL, dry_dirs, dry_per_dir) != 0)
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
if (early_delete) {
|
if (early_delete) {
|
||||||
Status ack;
|
Status ack;
|
||||||
@@ -642,8 +631,14 @@ int send_dry_run_remote(Config* config) {
|
|||||||
if (!receive_status(client->file_descriptor, &status))
|
if (!receive_status(client->file_descriptor, &status))
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
}
|
}
|
||||||
if (status != STATUS_OK)
|
/* A per-entry receiver failure is rsync's PARTIAL transfer (exit 23), not a
|
||||||
|
hard failure: a dry run transfers nothing, but keep the verdict consistent
|
||||||
|
with the normal path instead of treating it as a protocol error. */
|
||||||
|
if (status == STATUS_PARTIAL) {
|
||||||
|
partial = true;
|
||||||
|
} else if (status != STATUS_OK) {
|
||||||
goto dry_fail;
|
goto dry_fail;
|
||||||
|
}
|
||||||
if (!config->quiet) {
|
if (!config->quiet) {
|
||||||
if (config->human_readable)
|
if (config->human_readable)
|
||||||
printf("Total: %d files, %s\n", file_count,
|
printf("Total: %d files, %s\n", file_count,
|
||||||
@@ -661,7 +656,7 @@ int send_dry_run_remote(Config* config) {
|
|||||||
dry_transfer.literal_data = total_bytes;
|
dry_transfer.literal_data = total_bytes;
|
||||||
report_transfer_stats(config, &dry_transfer, dry_start, &dry_stats);
|
report_transfer_stats(config, &dry_transfer, dry_start, &dry_stats);
|
||||||
}
|
}
|
||||||
ret = io_error ? 1 : 0;
|
ret = io_error ? 1 : (partial ? 23 : 0);
|
||||||
|
|
||||||
dry_fail:
|
dry_fail:
|
||||||
if (dry_manifest)
|
if (dry_manifest)
|
||||||
@@ -672,6 +667,8 @@ dry_fail:
|
|||||||
array_list_delete(dry_excluded);
|
array_list_delete(dry_excluded);
|
||||||
if (dry_size_skipped)
|
if (dry_size_skipped)
|
||||||
array_list_delete(dry_size_skipped);
|
array_list_delete(dry_size_skipped);
|
||||||
|
if (dry_per_dir)
|
||||||
|
filter_rule_list_free(dry_per_dir);
|
||||||
if (scanner)
|
if (scanner)
|
||||||
directory_scanner_destroy(scanner);
|
directory_scanner_destroy(scanner);
|
||||||
prepared_scanner_destroy(&prepared);
|
prepared_scanner_destroy(&prepared);
|
||||||
|
|||||||
@@ -12,6 +12,7 @@
|
|||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
|
#include <threads.h>
|
||||||
#include <time.h>
|
#include <time.h>
|
||||||
|
|
||||||
/* Surface a server rejection to the user. When the last status exchange
|
/* Surface a server rejection to the user. When the last status exchange
|
||||||
@@ -161,6 +162,9 @@ void report_transfer_stats(const Config* config, const TransferStats* stats, tim
|
|||||||
created_breakdown, sizeof(created_breakdown));
|
created_breakdown, sizeof(created_breakdown));
|
||||||
unsigned long long created_total =
|
unsigned long long created_total =
|
||||||
recv->created_reg + recv->created_dir + recv->created_link + recv->created_special;
|
recv->created_reg + recv->created_dir + recv->created_link + recv->created_special;
|
||||||
|
char deleted_breakdown[128];
|
||||||
|
type_breakdown(recv->deleted_reg, recv->deleted_dir, recv->deleted_link, recv->deleted_special,
|
||||||
|
deleted_breakdown, sizeof(deleted_breakdown));
|
||||||
printf("\n");
|
printf("\n");
|
||||||
if (breakdown[0] != '\0')
|
if (breakdown[0] != '\0')
|
||||||
printf("Number of files: %llu %s\n", flist_total, breakdown);
|
printf("Number of files: %llu %s\n", flist_total, breakdown);
|
||||||
@@ -172,6 +176,12 @@ void report_transfer_stats(const Config* config, const TransferStats* stats, tim
|
|||||||
printf("Number of created files: %llu %s\n", created_total, created_breakdown);
|
printf("Number of created files: %llu %s\n", created_total, created_breakdown);
|
||||||
else
|
else
|
||||||
printf("Number of created files: %llu\n", created_total);
|
printf("Number of created files: %llu\n", created_total);
|
||||||
|
/* Protocol 2.30.0: the receiver reports the removed entries split by type, so
|
||||||
|
this line matches rsync's `Number of deleted files: X (reg: A, dir: B,
|
||||||
|
link: C, special: D)` (only the non-zero categories are listed). */
|
||||||
|
if (deleted_breakdown[0] != '\0')
|
||||||
|
printf("Number of deleted files: %llu %s\n", recv->deleted_files, deleted_breakdown);
|
||||||
|
else
|
||||||
printf("Number of deleted files: %llu\n", recv->deleted_files);
|
printf("Number of deleted files: %llu\n", recv->deleted_files);
|
||||||
printf("Number of regular files transferred: %llu\n", stats->transferred_regular);
|
printf("Number of regular files transferred: %llu\n", stats->transferred_regular);
|
||||||
printf("Total file size: %s bytes\n", total);
|
printf("Total file size: %s bytes\n", total);
|
||||||
@@ -525,10 +535,17 @@ static void client_progress_emit_ancestors(const Config* config, const char* rel
|
|||||||
if (dir != NULL)
|
if (dir != NULL)
|
||||||
change_emit_dir_sent(config, dir);
|
change_emit_dir_sent(config, dir);
|
||||||
} else {
|
} else {
|
||||||
|
/* --progress/-P names a directory only when it is newly created;
|
||||||
|
rsync stays silent for a pre-existing directory even when one of
|
||||||
|
its children changed (protocol 2.30.0 dest-state report). */
|
||||||
|
const File* dir = progress_dir_lookup(prefix);
|
||||||
|
bool existed = dir != NULL && dir->dest_state.known && dir->dest_state.existed;
|
||||||
|
if (!existed) {
|
||||||
char* escaped = output_escape(prefix, config->eight_bit_output);
|
char* escaped = output_escape(prefix, config->eight_bit_output);
|
||||||
printf("%s/\n", escaped ? escaped : prefix);
|
printf("%s/\n", escaped ? escaped : prefix);
|
||||||
free(escaped);
|
free(escaped);
|
||||||
}
|
}
|
||||||
|
}
|
||||||
g_progress_index++;
|
g_progress_index++;
|
||||||
} else {
|
} else {
|
||||||
free(key);
|
free(key);
|
||||||
@@ -552,6 +569,122 @@ void client_change_emit_ancestors(const Config* config, const File* file) {
|
|||||||
client_progress_emit_ancestors(config, rel);
|
client_progress_emit_ancestors(config, rel);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Send one STATUS_MKDIR probe (probe=1) for a pre-count directory and cache the
|
||||||
|
* receiver's pre-transfer destination snapshot in `dir->dest_state`. Returns
|
||||||
|
* false on a protocol/transport error. */
|
||||||
|
static bool client_probe_dir_state(int fd, File* dir) {
|
||||||
|
if (dir == NULL || file_wire_path(dir) == NULL)
|
||||||
|
return true;
|
||||||
|
if (!send_status(fd, STATUS_MKDIR) || !send_int(fd, 1) || !send_wire_str(fd, file_wire_path(dir)))
|
||||||
|
return false;
|
||||||
|
Status status = STATUS_ERROR;
|
||||||
|
if (!receive_status(fd, &status) || status != STATUS_DEST_INFO ||
|
||||||
|
!format_dest_state_receive(fd, &dir->dest_state)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Directory destination-state probe failed");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Output parity (protocol 2.30.0): ask the receiver for each not-yet-probed
|
||||||
|
* ancestor directory's pre-transfer state BEFORE the entry that first triggers
|
||||||
|
* it is sent, so the ancestor's -i/--out-format line renders rsync's
|
||||||
|
* `.d..t......` (existing, attributes changed) versus `cd+++++++++` (created)
|
||||||
|
* and an unchanged directory is suppressed. The probe is a STATUS_MKDIR frame
|
||||||
|
* with probe=1 (the receiver reports and creates nothing), so it must run before
|
||||||
|
* the receiver implicitly creates the parent for the child. Each directory is
|
||||||
|
* probed at most once; the result is cached in the pre-count File's dest_state.
|
||||||
|
* Returns false on a protocol/transport error (the caller aborts the transfer). */
|
||||||
|
bool client_change_probe_ancestors(const Config* config, const File* file, int fd) {
|
||||||
|
if (config == NULL || file == NULL || fd < 0 || !config->report_dest_info)
|
||||||
|
return true;
|
||||||
|
if (!g_progress_dir_index_valid || !g_progress_precount.dir_refs)
|
||||||
|
return true;
|
||||||
|
const char* rel = delete_display_path(config, file_wire_path(file));
|
||||||
|
if (rel == NULL)
|
||||||
|
return true;
|
||||||
|
size_t rel_len = strlen(rel);
|
||||||
|
for (size_t i = 1; i < rel_len; i++) {
|
||||||
|
if (rel[i] != '/')
|
||||||
|
continue;
|
||||||
|
char* prefix = malloc(i + 1);
|
||||||
|
if (prefix == NULL)
|
||||||
|
return false;
|
||||||
|
memcpy(prefix, rel, i);
|
||||||
|
prefix[i] = '\0';
|
||||||
|
if (path_index_contains(&g_progress_dir_index, prefix)) {
|
||||||
|
File* dir = progress_dir_lookup(prefix);
|
||||||
|
/* The probe path is the same wire path the real STATUS_MKDIR would carry
|
||||||
|
(the pre-count File's send_path, or its absolute source path for a
|
||||||
|
plain recursive scan), not the display-relative prefix. */
|
||||||
|
if (dir != NULL && !dir->dest_state.known && !client_probe_dir_state(fd, dir)) {
|
||||||
|
free(prefix);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
free(prefix);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Mark a directory the data pass already itemized/named so the end-of-transfer
|
||||||
|
* pending-directory flush does not report it a second time. `file` is a
|
||||||
|
* transferred directory entry (an empty-directory STATUS_MKDIR). */
|
||||||
|
void client_change_mark_dir(const Config* config, const File* file) {
|
||||||
|
if (config == NULL || file == NULL || !g_progress_emitted_valid ||
|
||||||
|
g_progress_emitted_keys == NULL)
|
||||||
|
return;
|
||||||
|
const char* rel = delete_display_path(config, file_wire_path(file));
|
||||||
|
if (rel == NULL || rel[0] == '\0' || str_hash_set_lookup(&g_progress_emitted, rel))
|
||||||
|
return;
|
||||||
|
char* key = str_dup(rel);
|
||||||
|
if (key == NULL)
|
||||||
|
return;
|
||||||
|
if (!array_list_add(g_progress_emitted_keys, key)) {
|
||||||
|
free(key);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
str_hash_set_insert_ref(&g_progress_emitted, key);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Emit the itemize lines for source directories that CHANGED but had no
|
||||||
|
* transferred child, so no ancestor emission reached them (rsync reports a
|
||||||
|
* directory whose attributes changed even when its contents did not). Runs at
|
||||||
|
* the end of the data pass, BEFORE the deferred STATUS_DIR_TIMES apply, so the
|
||||||
|
* probe still observes each untouched directory's pre-transfer state. Only the
|
||||||
|
* itemize/out-format/log streams report attribute-only directory changes;
|
||||||
|
* --progress/-P stays silent for them, matching rsync. Best-effort: a probe
|
||||||
|
* failure simply stops the flush (the transfer's verdict is unaffected). */
|
||||||
|
void client_change_emit_pending_dirs(const Config* config, int fd) {
|
||||||
|
if (config == NULL || fd < 0 || !config->report_dest_info)
|
||||||
|
return;
|
||||||
|
bool itemize_output = config->itemize_changes || config->out_format != NULL ||
|
||||||
|
(config->log_file != NULL && config->log_file_format != NULL);
|
||||||
|
if (!itemize_output)
|
||||||
|
return;
|
||||||
|
if (!g_progress_dir_index_valid || g_progress_precount.dir_refs == NULL ||
|
||||||
|
!g_progress_emitted_valid || g_progress_emitted_keys == NULL)
|
||||||
|
return;
|
||||||
|
for (int i = 0; i < g_progress_precount.dir_refs->size; i++) {
|
||||||
|
DirRef* ref = (DirRef*)g_progress_precount.dir_refs->items[i];
|
||||||
|
if (ref == NULL || ref->name == NULL || ref->name[0] == '\0')
|
||||||
|
continue;
|
||||||
|
if (str_hash_set_lookup(&g_progress_emitted, ref->name))
|
||||||
|
continue;
|
||||||
|
File* dir = ref->file;
|
||||||
|
if (dir == NULL)
|
||||||
|
continue;
|
||||||
|
if (!dir->dest_state.known && !client_probe_dir_state(fd, dir))
|
||||||
|
return;
|
||||||
|
change_emit_dir_sent(config, dir);
|
||||||
|
char* key = str_dup(ref->name);
|
||||||
|
if (key != NULL && array_list_add(g_progress_emitted_keys, key))
|
||||||
|
str_hash_set_insert_ref(&g_progress_emitted, key);
|
||||||
|
else
|
||||||
|
free(key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/* rsync's --info=name/progress line for one entry: transfer-relative name (a
|
/* rsync's --info=name/progress line for one entry: transfer-relative name (a
|
||||||
* trailing slash for directories) plus the ` -> target` symlink suffix. */
|
* trailing slash for directories) plus the ` -> target` symlink suffix. */
|
||||||
static char* progress_entry_line(const File* file, const char* rel) {
|
static char* progress_entry_line(const File* file, const char* rel) {
|
||||||
@@ -644,6 +777,15 @@ void client_progress_file(const Config* config, const File* file) {
|
|||||||
void client_progress_name(const Config* config, const File* file) {
|
void client_progress_name(const Config* config, const File* file) {
|
||||||
if (!g_progress_active || file == NULL)
|
if (!g_progress_active || file == NULL)
|
||||||
return;
|
return;
|
||||||
|
/* rsync's --progress/-P name stream reports an entry only when it is created
|
||||||
|
or (for a symlink) actually relinked: a pre-existing directory or an
|
||||||
|
unchanged symlink is silent (protocol 2.30.0 dest-state report). */
|
||||||
|
if (file->dest_state.known && file->dest_state.existed) {
|
||||||
|
if (file->is_dir || (file->is_symlink && file->dest_state.target_matches)) {
|
||||||
|
g_progress_index++;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
const char* rel = delete_display_path(config, file_wire_path(file));
|
const char* rel = delete_display_path(config, file_wire_path(file));
|
||||||
if (!config->itemize_changes && config->out_format == NULL) {
|
if (!config->itemize_changes && config->out_format == NULL) {
|
||||||
char* line = progress_entry_line(file, rel ? rel : "");
|
char* line = progress_entry_line(file, rel ? rel : "");
|
||||||
@@ -1051,3 +1193,129 @@ const char* delete_display_path(const Config* config, const char* path) {
|
|||||||
return path;
|
return path;
|
||||||
return utils_strip_transfer_root(path, config->send_directory);
|
return utils_strip_transfer_root(path, config->send_directory);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ---- --stderr=client diagnostic channel (protocol 2.30.0) ----
|
||||||
|
*
|
||||||
|
* When the client's --stderr mode is `client`, log_message() hands each of the
|
||||||
|
* client's own diagnostics to the sink installed here instead of writing them
|
||||||
|
* locally. The sink QUEUES the text (it may be called from scanner worker
|
||||||
|
* threads while the sender is streaming) and the sender thread -- the sole
|
||||||
|
* writer of the protocol stream -- drains the queue over the wire at frame
|
||||||
|
* boundaries via client_flush_client_messages(). A bounded queue caps the
|
||||||
|
* memory a chatty run can pin; overflow falls back to local output so a
|
||||||
|
* diagnostic is never silently dropped. */
|
||||||
|
#define CLIENT_MSG_MAX_QUEUED 256
|
||||||
|
#define CLIENT_MSG_MAX_BYTES (256 * 1024)
|
||||||
|
|
||||||
|
static mtx_t client_msg_mutex;
|
||||||
|
static once_flag client_msg_mutex_once = ONCE_FLAG_INIT;
|
||||||
|
static ArrayList* client_msg_queue = NULL; /* owns char* */
|
||||||
|
static size_t client_msg_bytes = 0;
|
||||||
|
/* True only while a live transfer session exists: before the connection is up
|
||||||
|
(or after it drops) the sink declines so log_message falls back to local
|
||||||
|
output, matching rsync's documented fallback. Written by the sender thread
|
||||||
|
(client_messages_activate) and read by scanner worker threads in
|
||||||
|
client_msg_enqueue, so it must be atomic: the queue itself stays guarded by
|
||||||
|
client_msg_mutex, but the flag is polled before taking that lock. */
|
||||||
|
static _Atomic bool client_msg_active = false;
|
||||||
|
|
||||||
|
static void client_msg_mutex_init(void) {
|
||||||
|
mtx_init(&client_msg_mutex, mtx_plain);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool client_msg_enqueue(const char* message);
|
||||||
|
|
||||||
|
/* Install the global log sink for the duration of one transfer. Safe to call
|
||||||
|
* more than once; the queue is created lazily. */
|
||||||
|
void client_messages_install(void) {
|
||||||
|
call_once(&client_msg_mutex_once, client_msg_mutex_init);
|
||||||
|
mtx_lock(&client_msg_mutex);
|
||||||
|
if (!client_msg_queue)
|
||||||
|
client_msg_queue = array_list_create(free);
|
||||||
|
bool ready = client_msg_queue != NULL;
|
||||||
|
mtx_unlock(&client_msg_mutex);
|
||||||
|
/* Only arm the sink once the queue exists; on allocation failure leave the
|
||||||
|
sink uninstalled so log_message keeps writing locally instead of handing
|
||||||
|
messages to a sink that would silently drop them. */
|
||||||
|
if (ready)
|
||||||
|
log_set_client_msg_sink(client_msg_enqueue);
|
||||||
|
}
|
||||||
|
|
||||||
|
void client_messages_activate(bool active) {
|
||||||
|
atomic_store(&client_msg_active, active);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* log_message sink: takes ownership (queues) the message when a session is
|
||||||
|
* live; returns false otherwise so the caller writes it locally. */
|
||||||
|
static bool client_msg_enqueue(const char* message) {
|
||||||
|
bool active = atomic_load(&client_msg_active);
|
||||||
|
if (!message || message[0] == '\0')
|
||||||
|
return active;
|
||||||
|
if (!active)
|
||||||
|
return false;
|
||||||
|
size_t len = strlen(message);
|
||||||
|
call_once(&client_msg_mutex_once, client_msg_mutex_init);
|
||||||
|
mtx_lock(&client_msg_mutex);
|
||||||
|
bool queued = false;
|
||||||
|
if (client_msg_queue && (size_t)client_msg_queue->size < CLIENT_MSG_MAX_QUEUED &&
|
||||||
|
client_msg_bytes + len <= CLIENT_MSG_MAX_BYTES) {
|
||||||
|
char* copy = str_dup(message);
|
||||||
|
if (copy) {
|
||||||
|
if (array_list_add(client_msg_queue, copy)) {
|
||||||
|
client_msg_bytes += len;
|
||||||
|
queued = true;
|
||||||
|
} else {
|
||||||
|
free(copy);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mtx_unlock(&client_msg_mutex);
|
||||||
|
return queued;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Drain the queued diagnostics as STATUS_CLIENT_MSG frames on the sender
|
||||||
|
* thread. Swaps the queue out under the mutex so a concurrent worker logging
|
||||||
|
* never blocks on the wire. Must be called at a protocol frame boundary. */
|
||||||
|
void client_flush_client_messages(int fd) {
|
||||||
|
if (fd < 0)
|
||||||
|
return;
|
||||||
|
call_once(&client_msg_mutex_once, client_msg_mutex_init);
|
||||||
|
mtx_lock(&client_msg_mutex);
|
||||||
|
ArrayList* pending = client_msg_queue;
|
||||||
|
if (pending) {
|
||||||
|
ArrayList* fresh = array_list_create(free);
|
||||||
|
if (fresh) {
|
||||||
|
client_msg_queue = fresh;
|
||||||
|
} else {
|
||||||
|
/* No memory for a replacement queue: stop queuing new diagnostics (they
|
||||||
|
fall back to local output) and drain this batch below so nothing is
|
||||||
|
silently dropped. */
|
||||||
|
client_msg_queue = NULL;
|
||||||
|
log_set_client_msg_sink(NULL);
|
||||||
|
}
|
||||||
|
client_msg_bytes = 0;
|
||||||
|
}
|
||||||
|
mtx_unlock(&client_msg_mutex);
|
||||||
|
if (!pending)
|
||||||
|
return;
|
||||||
|
for (int i = 0; i < pending->size; i++) {
|
||||||
|
const char* message = pending->items[i];
|
||||||
|
if (message && message[0] != '\0' && !send_client_message(fd, message))
|
||||||
|
break; /* peer is gone; the rest would fail too */
|
||||||
|
}
|
||||||
|
array_list_delete(pending);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Tear down the sink after a transfer and free anything still queued. */
|
||||||
|
void client_messages_end(void) {
|
||||||
|
log_set_client_msg_sink(NULL);
|
||||||
|
atomic_store(&client_msg_active, false);
|
||||||
|
call_once(&client_msg_mutex_once, client_msg_mutex_init);
|
||||||
|
mtx_lock(&client_msg_mutex);
|
||||||
|
ArrayList* pending = client_msg_queue;
|
||||||
|
client_msg_queue = NULL;
|
||||||
|
client_msg_bytes = 0;
|
||||||
|
mtx_unlock(&client_msg_mutex);
|
||||||
|
if (pending)
|
||||||
|
array_list_delete(pending);
|
||||||
|
}
|
||||||
@@ -381,21 +381,28 @@ bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* s
|
|||||||
paths, loading and sending nothing. --delete-before/--delete-during need the
|
paths, loading and sending nothing. --delete-before/--delete-during need the
|
||||||
complete keep-set manifest before the first data byte, so it is built by a
|
complete keep-set manifest before the first data byte, so it is built by a
|
||||||
dedicated pre-scan pass and transmitted early; the data pass then re-scans
|
dedicated pre-scan pass and transmitted early; the data pass then re-scans
|
||||||
with a fresh scanner. A source I/O error is fatal unless the options carry
|
with a fresh scanner. --delete-before additionally replays this very scan as
|
||||||
--ignore-errors, in which case the scan continues past the unreadable
|
its data pass (rsync's single file list), so `chunks_out` (optional) retains
|
||||||
directory and *io_error_out reports it (the caller still performs the
|
the scanned Chunk objects for the caller to send instead of destroying them;
|
||||||
deletion but reports the run as errored). */
|
the caller owns the list and must give it a chunk_destroy destructor. A
|
||||||
|
source I/O error is fatal unless the options carry --ignore-errors, in which
|
||||||
|
case the scan continues past the unreadable directory and *io_error_out
|
||||||
|
reports it (the caller still performs the deletion but reports the run as
|
||||||
|
errored). */
|
||||||
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
|
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
|
||||||
DeletePlanSender* plans, bool* io_error_out,
|
DeletePlanSender* plans, bool* io_error_out,
|
||||||
unsigned long long* non_dir_count_out) {
|
unsigned long long* non_dir_count_out, ArrayList* chunks_out,
|
||||||
|
bool emit_nonreg) {
|
||||||
if (io_error_out)
|
if (io_error_out)
|
||||||
*io_error_out = false;
|
*io_error_out = false;
|
||||||
if (non_dir_count_out)
|
if (non_dir_count_out)
|
||||||
*non_dir_count_out = 0;
|
*non_dir_count_out = 0;
|
||||||
ScannerOptions local = *options;
|
ScannerOptions local = *options;
|
||||||
/* The pre-scan is a paths-only pass with no client output; it must not emit
|
/* The pre-scan is normally a paths-only pass with no client output: it must
|
||||||
--info=nonreg lines (the data pass does that once). */
|
not emit --info=nonreg lines because the data pass re-scans and emits them
|
||||||
local.note_nonreg = false;
|
once. When the caller replays this scan as the data pass (--delete-before)
|
||||||
|
there is no later scan, so it opts in and the lines are emitted here. */
|
||||||
|
local.note_nonreg = emit_nonreg && options->note_nonreg;
|
||||||
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
|
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
|
||||||
if (!scanner)
|
if (!scanner)
|
||||||
return false;
|
return false;
|
||||||
@@ -430,7 +437,16 @@ bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayL
|
|||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (chunks_out) {
|
||||||
|
/* Retain the chunk for the caller's data pass; ownership moves with it. */
|
||||||
|
if (!array_list_add(chunks_out, chunk)) {
|
||||||
|
ok = false;
|
||||||
chunk_destroy(chunk);
|
chunk_destroy(chunk);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
chunk_destroy(chunk);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if (ok) {
|
if (ok) {
|
||||||
/* Keep every traversed source directory, including empty ones, so a plan
|
/* Keep every traversed source directory, including empty ones, so a plan
|
||||||
|
|||||||
+342
-78
@@ -127,10 +127,59 @@ Client* connect_transfer_client(const Config* config) {
|
|||||||
void disconnect_transfer_client(Client* client) {
|
void disconnect_transfer_client(Client* client) {
|
||||||
if (!client)
|
if (!client)
|
||||||
return;
|
return;
|
||||||
|
/* --stderr=client: push any diagnostics logged during the transfer to the
|
||||||
|
peer before the socket closes; once deactivated, later messages fall back
|
||||||
|
to local output instead of being lost. */
|
||||||
|
client_flush_client_messages(client->file_descriptor);
|
||||||
|
client_messages_activate(false);
|
||||||
client_disconnect(client);
|
client_disconnect(client);
|
||||||
client_delete(client);
|
client_delete(client);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Connect the configured transport and install the per-thread protocol session
|
||||||
|
* on it: init with the socket fd pair, apply the I/O timeout and (when
|
||||||
|
* negotiated) the TLS object, then bind it to this thread. Returns the
|
||||||
|
* connected client, or NULL (after logging the connect failure) when the
|
||||||
|
* transport could not connect. */
|
||||||
|
Client* client_connect_and_bind_session(const Config* config, ProtocolSession* session) {
|
||||||
|
Client* client = connect_transfer_client(config);
|
||||||
|
if (!client) {
|
||||||
|
if (config->transport == TRANSPORT_TCP)
|
||||||
|
log_message(LOG_LEVEL_ERROR, "could not connect to server%s",
|
||||||
|
config->use_tls ? " via TLS" : "");
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
protocol_session_init(session, client->file_descriptor, client->file_descriptor);
|
||||||
|
protocol_session_set_io_timeout(session, config->timeout);
|
||||||
|
protocol_session_set_ssl(session, (SSL*)client->ssl);
|
||||||
|
protocol_session_bind(session);
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Shared --files-from/--delete-missing-args preamble: allocate the missing-args
|
||||||
|
* destination list when the option is set, then validate the --files-from list
|
||||||
|
* (collecting the destination mirrors of missing entries for the receiver's
|
||||||
|
* exact-deletion request). On success the caller owns *missing_args_out (NULL
|
||||||
|
* when the option is off); on failure the list is freed and false is returned. */
|
||||||
|
bool client_prepare_files_from(const Config* config, ArrayList** missing_args_out,
|
||||||
|
int* skipped_out) {
|
||||||
|
ArrayList* missing_args = NULL;
|
||||||
|
if (config->delete_missing_args) {
|
||||||
|
missing_args = array_list_create(free);
|
||||||
|
if (!missing_args)
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
int skipped = 0;
|
||||||
|
if (!files_from_list_check(config, missing_args, &skipped)) {
|
||||||
|
if (missing_args)
|
||||||
|
array_list_delete(missing_args);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
*missing_args_out = missing_args;
|
||||||
|
*skipped_out = skipped;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
/* (finalize_transfer is defined after the SourceFile helpers below.) */
|
/* (finalize_transfer is defined after the SourceFile helpers below.) */
|
||||||
|
|
||||||
typedef struct SourceFile {
|
typedef struct SourceFile {
|
||||||
@@ -240,13 +289,28 @@ static void mark_sender_done(PipelineContextSender* context) {
|
|||||||
When --remove-source-files is active the receiver acknowledges each data
|
When --remove-source-files is active the receiver acknowledges each data
|
||||||
file it processed, in send order: STATUS_NEXT means the file was written,
|
file it processed, in send order: STATUS_NEXT means the file was written,
|
||||||
STATUS_OK means the file was skipped/unchanged. Skipped sources are marked
|
STATUS_OK means the file was skipped/unchanged. Skipped sources are marked
|
||||||
so the later removal pass keeps them. */
|
so the later removal pass keeps them. `partial_out` is set when the receiver
|
||||||
|
reported STATUS_PARTIAL (a per-entry receiver failure): the transfer is
|
||||||
|
otherwise complete, so successfully stored sources are still removed and the
|
||||||
|
caller exits 23 (rsync's partial transfer) instead of a fatal non-zero. */
|
||||||
static bool finalize_transfer(Client* client, const Config* config, ArrayList* remove_sources,
|
static bool finalize_transfer(Client* client, const Config* config, ArrayList* remove_sources,
|
||||||
bool* delete_limit_out, ReceiverStats* stats_out) {
|
bool* delete_limit_out, bool* partial_out, ReceiverStats* stats_out) {
|
||||||
if (delete_limit_out)
|
if (delete_limit_out)
|
||||||
*delete_limit_out = false;
|
*delete_limit_out = false;
|
||||||
|
if (partial_out)
|
||||||
|
*partial_out = false;
|
||||||
|
/* --stderr=client: the receiver consumes frames until it reads
|
||||||
|
STATUS_FINISHED, after which it no longer reads. Flush every diagnostic
|
||||||
|
queued during the transfer here -- the last frame boundary at which the
|
||||||
|
peer is still reading -- so nothing is stranded in the queue. */
|
||||||
|
client_flush_client_messages(client->file_descriptor);
|
||||||
if (!send_status(client->file_descriptor, STATUS_FINISHED))
|
if (!send_status(client->file_descriptor, STATUS_FINISHED))
|
||||||
return false;
|
return false;
|
||||||
|
/* Past STATUS_FINISHED the receiver has stopped reading, so any diagnostic
|
||||||
|
logged from here on (notably the STATUS_PARTIAL warning below) can no
|
||||||
|
longer be forwarded. Deactivate the channel so those messages fall back
|
||||||
|
to local output instead of being queued for a closed peer and lost. */
|
||||||
|
client_messages_activate(false);
|
||||||
/* The receiver emits its optional wire-stats frame (protocol 2.25.0) FIRST,
|
/* The receiver emits its optional wire-stats frame (protocol 2.25.0) FIRST,
|
||||||
then any per-file --remove-source-files acks, then the terminal status. */
|
then any per-file --remove-source-files acks, then the terminal status. */
|
||||||
Status status;
|
Status status;
|
||||||
@@ -298,6 +362,16 @@ static bool finalize_transfer(Client* client, const Config* config, ArrayList* r
|
|||||||
*delete_limit_out = true;
|
*delete_limit_out = true;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
/* A per-entry receiver failure the receiver chose to continue past is a
|
||||||
|
rsync PARTIAL transfer: everything else succeeded and the stored sources
|
||||||
|
may be removed, but the client must exit 23. */
|
||||||
|
if (status == STATUS_PARTIAL) {
|
||||||
|
log_message(LOG_LEVEL_WARNING,
|
||||||
|
"some files could not be transferred (see the server log for details)");
|
||||||
|
if (partial_out)
|
||||||
|
*partial_out = true;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
if (status != STATUS_OK) {
|
if (status != STATUS_OK) {
|
||||||
log_server_rejection("Receiver reported transfer failure");
|
log_server_rejection("Receiver reported transfer failure");
|
||||||
return false;
|
return false;
|
||||||
@@ -595,14 +669,34 @@ static bool send_file_direct(File* file, int fd, bool use_metadata, int compress
|
|||||||
static bool send_directory_entry(const Client* client, File* file, const Config* config) {
|
static bool send_directory_entry(const Client* client, File* file, const Config* config) {
|
||||||
if (!file || !file_wire_path(file))
|
if (!file || !file_wire_path(file))
|
||||||
return false;
|
return false;
|
||||||
if (!send_status(client->file_descriptor, STATUS_MKDIR) ||
|
int fd = client->file_descriptor;
|
||||||
!send_wire_str(client->file_descriptor, file_wire_path(file)))
|
if (!send_status(fd, STATUS_MKDIR))
|
||||||
return false;
|
return false;
|
||||||
if (config->use_metadata && !metadata_send(client->file_descriptor, file->metadata))
|
/* Output parity (protocol 2.30.0): when report_dest_info is negotiated every
|
||||||
|
STATUS_MKDIR body is prefixed with a probe flag (1 = probe only, 0 = a real
|
||||||
|
create), so the receiver knows whether to expect the metadata/xattr block. */
|
||||||
|
if (config->report_dest_info && !send_int(fd, 0))
|
||||||
|
return false;
|
||||||
|
if (!send_wire_str(fd, file_wire_path(file)))
|
||||||
|
return false;
|
||||||
|
if (config->use_metadata && !metadata_send(fd, file->metadata))
|
||||||
return false;
|
return false;
|
||||||
/* Directory xattrs/ACLs (-X/-A) ride the same trailing block as regular files
|
/* Directory xattrs/ACLs (-X/-A) ride the same trailing block as regular files
|
||||||
when the xattr transport was negotiated. */
|
when the xattr transport was negotiated. */
|
||||||
return !config->use_xattrs || xattr_send(client->file_descriptor, file->xattrs);
|
if (config->use_xattrs && !xattr_send(fd, file->xattrs))
|
||||||
|
return false;
|
||||||
|
/* The receiver answers with the directory's pre-transfer destination state
|
||||||
|
BEFORE creating it, so the sender can render rsync's `.d..t......` versus
|
||||||
|
`cd+++++++++` and suppress an unchanged directory. */
|
||||||
|
if (config->report_dest_info) {
|
||||||
|
Status status;
|
||||||
|
if (!receive_status(fd, &status) || status != STATUS_DEST_INFO ||
|
||||||
|
!format_dest_state_receive(fd, &file->dest_state)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Unexpected reply to the directory destination-state report");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* P7 Wave D: transmit every captured source directory's metadata in terminal
|
/* P7 Wave D: transmit every captured source directory's metadata in terminal
|
||||||
@@ -654,7 +748,25 @@ static bool send_symlink_entry(const Client* client, File* file, const Config* c
|
|||||||
if (!send_status(fd, STATUS_SYMLINK) || !send_wire_str(fd, file_wire_path(file)) ||
|
if (!send_status(fd, STATUS_SYMLINK) || !send_wire_str(fd, file_wire_path(file)) ||
|
||||||
!send_wire_str(fd, file->symlink_target))
|
!send_wire_str(fd, file->symlink_target))
|
||||||
return false;
|
return false;
|
||||||
return !config->use_metadata || metadata_send(fd, file->metadata);
|
if (config->use_metadata && !metadata_send(fd, file->metadata))
|
||||||
|
return false;
|
||||||
|
/* Symlink xattrs/ACLs (-X/-A) ride the same trailing block as regular files
|
||||||
|
and directories when the xattr transport was negotiated. */
|
||||||
|
if (config->use_xattrs && !xattr_send(fd, file->xattrs))
|
||||||
|
return false;
|
||||||
|
/* The receiver answers with the symlink's pre-transfer destination state
|
||||||
|
(including whether the on-disk link target already matches) BEFORE creating
|
||||||
|
it, so the sender can render rsync's `cLc........` / `.L..t......` and
|
||||||
|
suppress an unchanged symlink. */
|
||||||
|
if (config->report_dest_info) {
|
||||||
|
Status status;
|
||||||
|
if (!receive_status(fd, &status) || status != STATUS_DEST_INFO ||
|
||||||
|
!format_dest_state_receive(fd, &file->dest_state)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Unexpected reply to the symlink destination-state report");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Send a single file directly via sendfile (non-incremental path).
|
// Send a single file directly via sendfile (non-incremental path).
|
||||||
@@ -809,8 +921,28 @@ static bool source_is_regular_file(const File* file) {
|
|||||||
return stat(file->path, &st) == 0 && S_ISREG(st.st_mode);
|
return stat(file->path, &st) == 0 && S_ISREG(st.st_mode);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* True when an over-threshold source will be sent by STREAMING from disk rather
|
||||||
|
* than loaded into memory: either the zero-copy sendfile path (no compression)
|
||||||
|
* or the sender-side streaming compressor (zstd/zlib, when this file is not on
|
||||||
|
* the --skip-compress list). Otherwise the loader must materialize it. */
|
||||||
|
static bool loader_can_stream(const Config* config, const File* file) {
|
||||||
|
if (!config || !file || !file->data || file->data->size <= STREAM_THRESHOLD)
|
||||||
|
return false;
|
||||||
|
if (!config->use_compression)
|
||||||
|
return true;
|
||||||
|
if (!compression_stream_compress_supported(compression_get_algo()) ||
|
||||||
|
config->compression_level <= 0)
|
||||||
|
return false;
|
||||||
|
int skip_count = config->skip_compress_set ? config->skip_compress_count : -1;
|
||||||
|
return !compression_should_skip_with_suffixes(file->path, config->skip_compress_suffixes,
|
||||||
|
skip_count);
|
||||||
|
}
|
||||||
|
|
||||||
static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
||||||
ArrayList* remove_sources, TransferStats* stats) {
|
ArrayList* remove_sources, TransferStats* stats) {
|
||||||
|
/* --stderr=client: this is a frame boundary, so forward any diagnostics the
|
||||||
|
scanner/log emitted since the previous chunk before the next frame. */
|
||||||
|
client_flush_client_messages(client->file_descriptor);
|
||||||
if (config->use_chunk_serialization) {
|
if (config->use_chunk_serialization) {
|
||||||
if (remove_sources) {
|
if (remove_sources) {
|
||||||
for (int i = 0; i < chunk->element_count; i++) {
|
for (int i = 0; i < chunk->element_count; i++) {
|
||||||
@@ -838,14 +970,23 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
if (chunk->items[i] == NULL)
|
if (chunk->items[i] == NULL)
|
||||||
continue;
|
continue;
|
||||||
transfer_stats_note_entry(stats, chunk->items[i]);
|
transfer_stats_note_entry(stats, chunk->items[i]);
|
||||||
|
/* Output parity: probe each entry's ancestor directories' destination
|
||||||
|
state before emitting its itemize line, exactly as the non-serialized
|
||||||
|
loop does. Without this, dest_state.known stays false and -i/-P
|
||||||
|
renders an existing dir/symlink as created instead of `.d..t...` (or
|
||||||
|
suppressing it). */
|
||||||
|
if (!client_change_probe_ancestors(config, chunk->items[i], client->file_descriptor))
|
||||||
|
return -1;
|
||||||
/* The chunk-serialization path emits no --progress name lines, so only
|
/* The chunk-serialization path emits no --progress name lines, so only
|
||||||
feed -i/--out-format its ancestor directory lines here. */
|
feed -i/--out-format its ancestor directory lines here. */
|
||||||
if (config->itemize_changes || config->out_format != NULL)
|
if (config->itemize_changes || config->out_format != NULL)
|
||||||
client_change_emit_ancestors(config, chunk->items[i]);
|
client_change_emit_ancestors(config, chunk->items[i]);
|
||||||
if (chunk->items[i]->is_dir)
|
if (chunk->items[i]->is_dir) {
|
||||||
change_emit_dir_sent(config, chunk->items[i]);
|
change_emit_dir_sent(config, chunk->items[i]);
|
||||||
else
|
client_change_mark_dir(config, chunk->items[i]);
|
||||||
|
} else {
|
||||||
change_emit_file_sent(config, chunk->items[i]);
|
change_emit_file_sent(config, chunk->items[i]);
|
||||||
|
}
|
||||||
if (!chunk->items[i]->is_dir)
|
if (!chunk->items[i]->is_dir)
|
||||||
transfer_stats_note_transferred(stats, chunk->items[i]);
|
transfer_stats_note_transferred(stats, chunk->items[i]);
|
||||||
}
|
}
|
||||||
@@ -857,6 +998,11 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
if (f == NULL)
|
if (f == NULL)
|
||||||
continue;
|
continue;
|
||||||
transfer_stats_note_entry(stats, f);
|
transfer_stats_note_entry(stats, f);
|
||||||
|
/* Output parity: probe this entry's ancestor directories' destination state
|
||||||
|
before the entry (or the first child below them) is sent, while the
|
||||||
|
receiver has not yet created them implicitly. */
|
||||||
|
if (!client_change_probe_ancestors(config, f, client->file_descriptor))
|
||||||
|
return -1;
|
||||||
if (f->is_dir) {
|
if (f->is_dir) {
|
||||||
/* Explicit directory entry (--dirs): a MKDIR frame carrying the
|
/* Explicit directory entry (--dirs): a MKDIR frame carrying the
|
||||||
destination path (and metadata when negotiated). Directories have no
|
destination path (and metadata when negotiated). Directories have no
|
||||||
@@ -865,6 +1011,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
return -1;
|
return -1;
|
||||||
client_change_emit_ancestors(config, f);
|
client_change_emit_ancestors(config, f);
|
||||||
change_emit_dir_sent(config, f);
|
change_emit_dir_sent(config, f);
|
||||||
|
client_change_mark_dir(config, f);
|
||||||
client_progress_name(config, f);
|
client_progress_name(config, f);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -936,20 +1083,14 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
|
|||||||
static int send_chunks_multithreaded(void* pipeline_context) {
|
static int send_chunks_multithreaded(void* pipeline_context) {
|
||||||
PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
|
PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
|
||||||
time_t start = time(NULL);
|
time_t start = time(NULL);
|
||||||
Client* client = connect_transfer_client(context->config);
|
ProtocolSession session;
|
||||||
|
Client* client = client_connect_and_bind_session(context->config, &session);
|
||||||
if (!client) {
|
if (!client) {
|
||||||
if (context->config->transport == TRANSPORT_TCP)
|
|
||||||
log_message(LOG_LEVEL_ERROR, "could not connect to server%s",
|
|
||||||
context->config->use_tls ? " via TLS" : "");
|
|
||||||
pipeline_cancel(context);
|
pipeline_cancel(context);
|
||||||
mark_sender_done(context);
|
mark_sender_done(context);
|
||||||
return thrd_error;
|
return thrd_error;
|
||||||
}
|
}
|
||||||
ProtocolSession session;
|
client_messages_activate(true);
|
||||||
protocol_session_init(&session, client->file_descriptor, client->file_descriptor);
|
|
||||||
protocol_session_set_io_timeout(&session, context->config->timeout);
|
|
||||||
protocol_session_set_ssl(&session, (SSL*)client->ssl);
|
|
||||||
protocol_session_bind(&session);
|
|
||||||
if (!config_send(client->file_descriptor, context->config)) {
|
if (!config_send(client->file_descriptor, context->config)) {
|
||||||
pipeline_cancel(context);
|
pipeline_cancel(context);
|
||||||
disconnect_transfer_client(client);
|
disconnect_transfer_client(client);
|
||||||
@@ -963,7 +1104,7 @@ static int send_chunks_multithreaded(void* pipeline_context) {
|
|||||||
and wait for the receiver to delete extras before streaming any data. */
|
and wait for the receiver to delete extras before streaming any data. */
|
||||||
if (!send_delete_manifest_early(client, context->manifest, context->excluded_paths,
|
if (!send_delete_manifest_early(client, context->manifest, context->excluded_paths,
|
||||||
context->size_skipped_paths, context->missing_args,
|
context->size_skipped_paths, context->missing_args,
|
||||||
context->synced_dirs)) {
|
context->synced_dirs, context->per_dir_rules)) {
|
||||||
pipeline_cancel(context);
|
pipeline_cancel(context);
|
||||||
disconnect_transfer_client(client);
|
disconnect_transfer_client(client);
|
||||||
mark_sender_done(context);
|
mark_sender_done(context);
|
||||||
@@ -1093,7 +1234,8 @@ static int send_chunks_multithreaded(void* pipeline_context) {
|
|||||||
log_message(LOG_LEVEL_WARNING, "IO error encountered -- skipping file deletion");
|
log_message(LOG_LEVEL_WARNING, "IO error encountered -- skipping file deletion");
|
||||||
} else if (send_delete_manifest(client->file_descriptor, context->manifest,
|
} else if (send_delete_manifest(client->file_descriptor, context->manifest,
|
||||||
context->excluded_paths, context->size_skipped_paths,
|
context->excluded_paths, context->size_skipped_paths,
|
||||||
context->missing_args, context->synced_dirs) != 0) {
|
context->missing_args, context->synced_dirs,
|
||||||
|
context->per_dir_rules) != 0) {
|
||||||
goto send_fail;
|
goto send_fail;
|
||||||
}
|
}
|
||||||
} else if (context->config->delete_missing_args && !context->early_delete &&
|
} else if (context->config->delete_missing_args && !context->early_delete &&
|
||||||
@@ -1101,7 +1243,7 @@ static int send_chunks_multithreaded(void* pipeline_context) {
|
|||||||
/* --delete-missing-args without --delete: no keep-set is built, but the
|
/* --delete-missing-args without --delete: no keep-set is built, but the
|
||||||
exact-delete paths still ride the same manifest frame (commit once the
|
exact-delete paths still ride the same manifest frame (commit once the
|
||||||
transfer succeeded). */
|
transfer succeeded). */
|
||||||
if (send_delete_manifest(client->file_descriptor, NULL, NULL, NULL, context->missing_args,
|
if (send_delete_manifest(client->file_descriptor, NULL, NULL, NULL, context->missing_args, NULL,
|
||||||
NULL) != 0)
|
NULL) != 0)
|
||||||
goto send_fail;
|
goto send_fail;
|
||||||
}
|
}
|
||||||
@@ -1109,15 +1251,19 @@ static int send_chunks_multithreaded(void* pipeline_context) {
|
|||||||
(and all parallel workers) has been joined before scanner_done was set, so
|
(and all parallel workers) has been joined before scanner_done was set, so
|
||||||
the list is complete and race-free; on an early stop the list may be
|
the list is complete and race-free; on an early stop the list may be
|
||||||
incomplete and is deliberately not sent. */
|
incomplete and is deliberately not sent. */
|
||||||
|
client_change_emit_pending_dirs(context->config, client->file_descriptor);
|
||||||
if (!context->scan_stopped_early &&
|
if (!context->scan_stopped_early &&
|
||||||
!send_dir_times(client, context->config, context->dir_entries))
|
!send_dir_times(client, context->config, context->dir_entries))
|
||||||
goto send_fail;
|
goto send_fail;
|
||||||
bool delete_limit = false;
|
bool delete_limit = false;
|
||||||
|
bool partial = false;
|
||||||
ReceiverStats recv_stats;
|
ReceiverStats recv_stats;
|
||||||
memset(&recv_stats, 0, sizeof(recv_stats));
|
memset(&recv_stats, 0, sizeof(recv_stats));
|
||||||
|
client_flush_client_messages(client->file_descriptor);
|
||||||
bool ok = finalize_transfer(client, context->config, context->remove_source_files, &delete_limit,
|
bool ok = finalize_transfer(client, context->config, context->remove_source_files, &delete_limit,
|
||||||
&recv_stats);
|
&partial, &recv_stats);
|
||||||
context->delete_limit = delete_limit;
|
context->delete_limit = delete_limit;
|
||||||
|
context->partial = partial;
|
||||||
if (!ok && context->config->use_delete)
|
if (!ok && context->config->use_delete)
|
||||||
log_message(LOG_LEVEL_ERROR,
|
log_message(LOG_LEVEL_ERROR,
|
||||||
"server reported a deletion failure (--delete); see the server log for the reason");
|
"server reported a deletion failure (--delete); see the server log for the reason");
|
||||||
@@ -1148,6 +1294,43 @@ send_fail:
|
|||||||
static int scan_directory_multithreaded(void* pipeline_context) {
|
static int scan_directory_multithreaded(void* pipeline_context) {
|
||||||
PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
|
PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
|
||||||
protocol_session_bind(&context->allocation_session);
|
protocol_session_bind(&context->allocation_session);
|
||||||
|
if (context->prescan_chunks != NULL) {
|
||||||
|
/* --delete-before replays the pre-scan that built the early keep-set as the
|
||||||
|
data pass (rsync builds one file list). Feed the retained chunks straight
|
||||||
|
into the pipeline instead of re-reading the source, so a file created
|
||||||
|
after the pre-scan is neither transferred nor kept. The chunk also
|
||||||
|
carries the directory times captured by that scan (there is no later
|
||||||
|
scan), so no scanner is created here. */
|
||||||
|
bool failed = false;
|
||||||
|
for (int i = 0; i < context->prescan_chunks->size; i++) {
|
||||||
|
Chunk* chunk = (Chunk*)context->prescan_chunks->items[i];
|
||||||
|
/* Move ownership out of the retained list so a cleanup here never
|
||||||
|
double-frees a chunk the queue now owns. */
|
||||||
|
context->prescan_chunks->items[i] = NULL;
|
||||||
|
if (chunk == NULL)
|
||||||
|
continue;
|
||||||
|
if (!queue_enqueue_multithreaded_cancel(
|
||||||
|
context->queue_scanner, chunk, &context->mutex_scanner,
|
||||||
|
&context->condition_not_empty_scanner, &context->condition_not_full_scanner,
|
||||||
|
&context->cancelled)) {
|
||||||
|
chunk_destroy(chunk);
|
||||||
|
failed = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mtx_lock(&context->mutex_scanner);
|
||||||
|
context->scanner_done = true;
|
||||||
|
cnd_broadcast(&context->condition_not_empty_scanner);
|
||||||
|
cnd_broadcast(&context->condition_not_full_scanner);
|
||||||
|
mtx_unlock(&context->mutex_scanner);
|
||||||
|
if (failed) {
|
||||||
|
pipeline_cancel(context);
|
||||||
|
protocol_session_unbind();
|
||||||
|
return thrd_error;
|
||||||
|
}
|
||||||
|
protocol_session_unbind();
|
||||||
|
return thrd_success;
|
||||||
|
}
|
||||||
PreparedScanner prepared;
|
PreparedScanner prepared;
|
||||||
/* -j/--threads=N sizes the parallel scanner's worker pool; 0 (bare -j) lets
|
/* -j/--threads=N sizes the parallel scanner's worker pool; 0 (bare -j) lets
|
||||||
* the scanner apply its built-in default. */
|
* the scanner apply its built-in default. */
|
||||||
@@ -1174,6 +1357,7 @@ static int scan_directory_multithreaded(void* pipeline_context) {
|
|||||||
its protected lists, so the data pass must not append to them again. */
|
its protected lists, so the data pass must not append to them again. */
|
||||||
if (!context->early_delete && !context->delete_plans) {
|
if (!context->early_delete && !context->delete_plans) {
|
||||||
prepared.options.excluded_paths = context->excluded_paths;
|
prepared.options.excluded_paths = context->excluded_paths;
|
||||||
|
prepared.options.per_dir_rules = context->per_dir_rules;
|
||||||
/* The root marker for a full recursive transfer is already in the list; do
|
/* The root marker for a full recursive transfer is already in the list; do
|
||||||
not let the scanner append every directory to it. */
|
not let the scanner append every directory to it. */
|
||||||
if (context->config->files_from_set != NULL)
|
if (context->config->files_from_set != NULL)
|
||||||
@@ -1293,7 +1477,7 @@ static int load_files_multithreaded(void* pipeline_context) {
|
|||||||
if (!context->config->use_sendfile) {
|
if (!context->config->use_sendfile) {
|
||||||
for (int i = 0; i < chunk->element_count; i++) {
|
for (int i = 0; i < chunk->element_count; i++) {
|
||||||
File* f = chunk->items[i];
|
File* f = chunk->items[i];
|
||||||
if (f->data->size > STREAM_THRESHOLD && !context->config->use_compression)
|
if (loader_can_stream(context->config, f))
|
||||||
continue;
|
continue;
|
||||||
if (!file_load_data(f)) {
|
if (!file_load_data(f)) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to load file data");
|
log_message(LOG_LEVEL_ERROR, "Failed to load file data");
|
||||||
@@ -1394,6 +1578,10 @@ typedef struct {
|
|||||||
Client* client;
|
Client* client;
|
||||||
DirectoryScanner* scanner;
|
DirectoryScanner* scanner;
|
||||||
ArrayList* manifest;
|
ArrayList* manifest;
|
||||||
|
/* --delete-before: the pre-scan that built the keep-set, retained as the data
|
||||||
|
pass's file list (owning Chunk*; consumed chunks are NULLed as they are
|
||||||
|
sent). NULL in every other mode, where the data pass scans normally. */
|
||||||
|
ArrayList* prescan_chunks;
|
||||||
DeletePlanSender* plan_sender;
|
DeletePlanSender* plan_sender;
|
||||||
ArrayList* remove_sources;
|
ArrayList* remove_sources;
|
||||||
ArrayList* dir_entries;
|
ArrayList* dir_entries;
|
||||||
@@ -1403,6 +1591,8 @@ typedef struct {
|
|||||||
ArrayList* synced_dirs;
|
ArrayList* synced_dirs;
|
||||||
ArrayList* plan_dirs;
|
ArrayList* plan_dirs;
|
||||||
ArrayList* missing_args;
|
ArrayList* missing_args;
|
||||||
|
/* Per-directory filter rules compiled by the scan (protocol 2.30.0). */
|
||||||
|
FilterRuleList* per_dir_rules;
|
||||||
PreparedScanner prepared;
|
PreparedScanner prepared;
|
||||||
StopCondition stop;
|
StopCondition stop;
|
||||||
TransferStats transfer_stats;
|
TransferStats transfer_stats;
|
||||||
@@ -1447,9 +1637,11 @@ static bool send_files_prepare(Config* config, SendFilesState* state) {
|
|||||||
}
|
}
|
||||||
state->size_skipped = array_list_create(free);
|
state->size_skipped = array_list_create(free);
|
||||||
state->synced_dirs = array_list_create(free);
|
state->synced_dirs = array_list_create(free);
|
||||||
if (!state->size_skipped || !state->synced_dirs)
|
state->per_dir_rules = filter_rule_list_create();
|
||||||
|
if (!state->size_skipped || !state->synced_dirs || !state->per_dir_rules)
|
||||||
return false;
|
return false;
|
||||||
state->prepared.options.size_skipped_paths = state->size_skipped;
|
state->prepared.options.size_skipped_paths = state->size_skipped;
|
||||||
|
state->prepared.options.per_dir_rules = state->per_dir_rules;
|
||||||
/* Only a --files-from subset confines the extras walk to the directories
|
/* Only a --files-from subset confines the extras walk to the directories
|
||||||
the scan synchronized; a full recursive transfer deletes throughout the
|
the scan synchronized; a full recursive transfer deletes throughout the
|
||||||
receive root, so mark the root itself (the "." sentinel) and let the
|
receive root, so mark the root itself (the "." sentinel) and let the
|
||||||
@@ -1478,12 +1670,23 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
|||||||
if (state->delete_early) {
|
if (state->delete_early) {
|
||||||
/* Pass 1: collect the complete keep-set (paths only, no data loaded) and
|
/* Pass 1: collect the complete keep-set (paths only, no data loaded) and
|
||||||
transmit it now, before any file data. The receiver removes extras and
|
transmit it now, before any file data. The receiver removes extras and
|
||||||
acks; the transfer aborts here if the deletion could not commit. */
|
acks; the transfer aborts here if the deletion could not commit. The
|
||||||
|
scanned chunks are retained so the data pass can replay this exact list
|
||||||
|
instead of re-reading the source (rsync builds one file list and never
|
||||||
|
transfers a file created after it). */
|
||||||
ArrayList* early_manifest = array_list_create(free);
|
ArrayList* early_manifest = array_list_create(free);
|
||||||
if (!early_manifest)
|
ArrayList* prescan_chunks = array_list_create(chunk_destroy);
|
||||||
|
if (!early_manifest || !prescan_chunks) {
|
||||||
|
array_list_delete(early_manifest);
|
||||||
|
array_list_delete(prescan_chunks);
|
||||||
return false;
|
return false;
|
||||||
|
}
|
||||||
|
/* No later scan runs for --delete-before, so this pass must also capture the
|
||||||
|
deferred directory times and the --stats directory count. */
|
||||||
|
state->prepared.options.dir_entries = state->dir_entries;
|
||||||
|
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
|
||||||
bool prescan_ok = scan_paths_only(config, &state->prepared.options, early_manifest, NULL,
|
bool prescan_ok = scan_paths_only(config, &state->prepared.options, early_manifest, NULL,
|
||||||
&state->had_scan_io, NULL);
|
&state->had_scan_io, NULL, prescan_chunks, true);
|
||||||
bool early_ok = false;
|
bool early_ok = false;
|
||||||
bool skip_delete = false;
|
bool skip_delete = false;
|
||||||
if (prescan_ok) {
|
if (prescan_ok) {
|
||||||
@@ -1503,9 +1706,9 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
|||||||
log_message(LOG_LEVEL_WARNING, "IO error encountered -- skipping file deletion");
|
log_message(LOG_LEVEL_WARNING, "IO error encountered -- skipping file deletion");
|
||||||
skip_delete = true;
|
skip_delete = true;
|
||||||
} else {
|
} else {
|
||||||
early_ok =
|
early_ok = send_delete_manifest_early(client, early_manifest, state->excluded,
|
||||||
send_delete_manifest_early(client, early_manifest, state->excluded, state->size_skipped,
|
state->size_skipped, state->missing_args,
|
||||||
state->missing_args, state->synced_dirs);
|
state->synced_dirs, state->per_dir_rules);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
array_list_delete(early_manifest);
|
array_list_delete(early_manifest);
|
||||||
@@ -1514,8 +1717,14 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
|||||||
state->prepared.options.excluded_paths = NULL;
|
state->prepared.options.excluded_paths = NULL;
|
||||||
state->prepared.options.size_skipped_paths = NULL;
|
state->prepared.options.size_skipped_paths = NULL;
|
||||||
state->prepared.options.synced_dirs = NULL;
|
state->prepared.options.synced_dirs = NULL;
|
||||||
if (!prescan_ok || (!early_ok && !skip_delete))
|
state->prepared.options.per_dir_rules = NULL;
|
||||||
|
if (!prescan_ok || (!early_ok && !skip_delete)) {
|
||||||
|
array_list_delete(prescan_chunks);
|
||||||
return false;
|
return false;
|
||||||
|
}
|
||||||
|
/* Adopt the captured scan as the data pass's file list (including when an
|
||||||
|
I/O error suppressed only the deletion: the list is still complete). */
|
||||||
|
state->prescan_chunks = prescan_chunks;
|
||||||
} else if (state->delete_per_dir) {
|
} else if (state->delete_per_dir) {
|
||||||
/* --delete-during/--delete-delay: build one plan per source directory from a
|
/* --delete-during/--delete-delay: build one plan per source directory from a
|
||||||
path-only pre-scan and transmit the COMPLETE plan set now, before any data,
|
path-only pre-scan and transmit the COMPLETE plan set now, before any data,
|
||||||
@@ -1527,8 +1736,9 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
|||||||
if (!state->plan_sender || !state->plan_dirs)
|
if (!state->plan_sender || !state->plan_dirs)
|
||||||
return false;
|
return false;
|
||||||
state->prepared.options.plan_dirs = state->plan_dirs;
|
state->prepared.options.plan_dirs = state->plan_dirs;
|
||||||
bool prescan_ok = scan_paths_only(config, &state->prepared.options, NULL, state->plan_sender,
|
bool prescan_ok =
|
||||||
&state->had_scan_io, &state->per_dir_non_dir_count);
|
scan_paths_only(config, &state->prepared.options, NULL, state->plan_sender,
|
||||||
|
&state->had_scan_io, &state->per_dir_non_dir_count, NULL, false);
|
||||||
bool plans_ok = false;
|
bool plans_ok = false;
|
||||||
bool skip_delete = false;
|
bool skip_delete = false;
|
||||||
if (prescan_ok) {
|
if (prescan_ok) {
|
||||||
@@ -1537,7 +1747,7 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
|||||||
config->files_from_set ? state->synced_dirs : (walk_root ? state->synced_dirs : NULL);
|
config->files_from_set ? state->synced_dirs : (walk_root ? state->synced_dirs : NULL);
|
||||||
delete_plan_sender_finalize(state->plan_sender, scope, walk_root);
|
delete_plan_sender_finalize(state->plan_sender, scope, walk_root);
|
||||||
delete_plan_sender_set_config(state->plan_sender, state->excluded, state->size_skipped,
|
delete_plan_sender_set_config(state->plan_sender, state->excluded, state->size_skipped,
|
||||||
state->missing_args);
|
state->missing_args, state->per_dir_rules);
|
||||||
if (state->had_scan_io && delete_plan_sender_empty(state->plan_sender)) {
|
if (state->had_scan_io && delete_plan_sender_empty(state->plan_sender)) {
|
||||||
log_message(LOG_LEVEL_ERROR,
|
log_message(LOG_LEVEL_ERROR,
|
||||||
"source scan hit an I/O error before finding any file; refusing to delete "
|
"source scan hit an I/O error before finding any file; refusing to delete "
|
||||||
@@ -1561,6 +1771,7 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
|
|||||||
state->prepared.options.size_skipped_paths = NULL;
|
state->prepared.options.size_skipped_paths = NULL;
|
||||||
state->prepared.options.synced_dirs = NULL;
|
state->prepared.options.synced_dirs = NULL;
|
||||||
state->prepared.options.plan_dirs = NULL;
|
state->prepared.options.plan_dirs = NULL;
|
||||||
|
state->prepared.options.per_dir_rules = NULL;
|
||||||
if (!prescan_ok || (!plans_ok && !skip_delete))
|
if (!prescan_ok || (!plans_ok && !skip_delete))
|
||||||
return false;
|
return false;
|
||||||
} else if (config->use_delete) {
|
} else if (config->use_delete) {
|
||||||
@@ -1592,24 +1803,42 @@ static bool send_files_run(Config* config, SendFilesState* state) {
|
|||||||
state->stop = stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins,
|
state->stop = stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins,
|
||||||
config->cli.stop_at_set, config->stop_at, now_mono);
|
config->cli.stop_at_set, config->stop_at, now_mono);
|
||||||
state->prepared.options.stop_condition = &state->stop;
|
state->prepared.options.stop_condition = &state->stop;
|
||||||
/* The early-delete pre-scan above already ran; only the data pass should feed
|
/* --delete-before reuses the pre-scan that built the keep-set as the data
|
||||||
the directory-time list (otherwise every directory would be captured
|
pass's file list, so a source file created after that scan is neither
|
||||||
twice). */
|
transferred nor kept (rsync builds one file list). That pre-scan captured
|
||||||
|
the deferred directory times and the --stats directory count because no
|
||||||
|
later scan runs; every other mode opens a fresh data scanner here. */
|
||||||
|
if (state->prescan_chunks == NULL) {
|
||||||
state->prepared.options.dir_entries = state->dir_entries;
|
state->prepared.options.dir_entries = state->dir_entries;
|
||||||
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
|
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
|
||||||
state->scanner =
|
state->scanner =
|
||||||
directory_scanner_create_with_options(config->send_directory, &state->prepared.options);
|
directory_scanner_create_with_options(config->send_directory, &state->prepared.options);
|
||||||
if (!state->scanner)
|
if (!state->scanner)
|
||||||
return false;
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
Chunk* current_chunk;
|
Chunk* current_chunk;
|
||||||
|
int prescan_index = 0;
|
||||||
memset(&state->transfer_stats, 0, sizeof(state->transfer_stats));
|
memset(&state->transfer_stats, 0, sizeof(state->transfer_stats));
|
||||||
state->start = time(NULL);
|
state->start = time(NULL);
|
||||||
client_progress_begin(config);
|
client_progress_begin(config);
|
||||||
/* True when the stop deadline cut the scan short so the keep-set manifest is
|
/* True when the stop deadline cut the scan short so the keep-set manifest is
|
||||||
only a prefix of the source. */
|
only a prefix of the source. */
|
||||||
bool send_failed = false;
|
bool send_failed = false;
|
||||||
while ((current_chunk = directory_scanner_next(state->scanner)) != NULL) {
|
while (true) {
|
||||||
|
if (state->prescan_chunks != NULL) {
|
||||||
|
if (prescan_index >= state->prescan_chunks->size)
|
||||||
|
break;
|
||||||
|
/* Move ownership out of the retained list so chunk_destroy below (and the
|
||||||
|
cleanup tail for an early exit) never double-frees it. */
|
||||||
|
current_chunk = (Chunk*)state->prescan_chunks->items[prescan_index];
|
||||||
|
state->prescan_chunks->items[prescan_index] = NULL;
|
||||||
|
prescan_index++;
|
||||||
|
} else {
|
||||||
|
current_chunk = directory_scanner_next(state->scanner);
|
||||||
|
if (current_chunk == NULL)
|
||||||
|
break;
|
||||||
|
}
|
||||||
/* Graceful abort (Ctrl-C/SIGTERM): notify the receiver and clean up. The
|
/* Graceful abort (Ctrl-C/SIGTERM): notify the receiver and clean up. The
|
||||||
session is active (config_send already succeeded); a send failure here is
|
session is active (config_send already succeeded); a send failure here is
|
||||||
fine because the client is exiting anyway. */
|
fine because the client is exiting anyway. */
|
||||||
@@ -1637,7 +1866,7 @@ static bool send_files_run(Config* config, SendFilesState* state) {
|
|||||||
bool load_ok = true;
|
bool load_ok = true;
|
||||||
for (int i = 0; i < current_chunk->element_count; i++) {
|
for (int i = 0; i < current_chunk->element_count; i++) {
|
||||||
File* f = current_chunk->items[i];
|
File* f = current_chunk->items[i];
|
||||||
if (f->data->size > STREAM_THRESHOLD && !config->use_compression)
|
if (loader_can_stream(config, f))
|
||||||
continue;
|
continue;
|
||||||
if (!file_load_data(f)) {
|
if (!file_load_data(f)) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to load file data");
|
log_message(LOG_LEVEL_ERROR, "Failed to load file data");
|
||||||
@@ -1667,10 +1896,14 @@ static bool send_files_run(Config* config, SendFilesState* state) {
|
|||||||
* Returns the rsync-compatible exit code. */
|
* Returns the rsync-compatible exit code. */
|
||||||
static int send_files_finalize(const Config* config, SendFilesState* state) {
|
static int send_files_finalize(const Config* config, SendFilesState* state) {
|
||||||
Client* client = state->client;
|
Client* client = state->client;
|
||||||
|
/* A --delete-before run replays the pre-scan and owns no data scanner; its
|
||||||
|
I/O-error verdict was already recorded by that pre-scan. */
|
||||||
|
if (state->scanner != NULL) {
|
||||||
if (directory_scanner_failed(state->scanner))
|
if (directory_scanner_failed(state->scanner))
|
||||||
return 1;
|
return 1;
|
||||||
if (directory_scanner_had_io_error(state->scanner))
|
if (directory_scanner_had_io_error(state->scanner))
|
||||||
state->had_scan_io = true;
|
state->had_scan_io = true;
|
||||||
|
}
|
||||||
/* An abort that arrived after the last chunk must still stop the completion
|
/* An abort that arrived after the last chunk must still stop the completion
|
||||||
tail (manifest/finalize) rather than let it run to success. */
|
tail (manifest/finalize) rather than let it run to success. */
|
||||||
if (client_abort_pending()) {
|
if (client_abort_pending()) {
|
||||||
@@ -1720,7 +1953,8 @@ static int send_files_finalize(const Config* config, SendFilesState* state) {
|
|||||||
modes the deletion already went out with the data, so nothing is
|
modes the deletion already went out with the data, so nothing is
|
||||||
re-sent here. */
|
re-sent here. */
|
||||||
if (send_delete_manifest(client->file_descriptor, state->manifest, state->excluded,
|
if (send_delete_manifest(client->file_descriptor, state->manifest, state->excluded,
|
||||||
state->size_skipped, state->missing_args, state->synced_dirs) != 0) {
|
state->size_skipped, state->missing_args, state->synced_dirs,
|
||||||
|
state->per_dir_rules) != 0) {
|
||||||
if (state->manifest) {
|
if (state->manifest) {
|
||||||
array_list_delete(state->manifest);
|
array_list_delete(state->manifest);
|
||||||
state->manifest = NULL;
|
state->manifest = NULL;
|
||||||
@@ -1733,15 +1967,22 @@ static int send_files_finalize(const Config* config, SendFilesState* state) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/* Output parity: report changed directories that had no transferred child
|
||||||
|
before the deferred directory times are applied (so the probe still sees
|
||||||
|
their pre-transfer state). */
|
||||||
|
client_change_emit_pending_dirs(config, client->file_descriptor);
|
||||||
/* P7 Wave D: every directory has now been traversed (or the scan stopped
|
/* P7 Wave D: every directory has now been traversed (or the scan stopped
|
||||||
early), so transmit the captured directory times last. The receiver defers
|
early), so transmit the captured directory times last. The receiver defers
|
||||||
applying them until after its own deletion/publication phase. */
|
applying them until after its own deletion/publication phase. */
|
||||||
if (!send_dir_times(client, config, state->dir_entries))
|
if (!send_dir_times(client, config, state->dir_entries))
|
||||||
return 1;
|
return 1;
|
||||||
bool delete_limit = false;
|
bool delete_limit = false;
|
||||||
|
bool partial = false;
|
||||||
ReceiverStats recv_stats;
|
ReceiverStats recv_stats;
|
||||||
memset(&recv_stats, 0, sizeof(recv_stats));
|
memset(&recv_stats, 0, sizeof(recv_stats));
|
||||||
bool ok = finalize_transfer(client, config, state->remove_sources, &delete_limit, &recv_stats);
|
client_flush_client_messages(client->file_descriptor);
|
||||||
|
bool ok = finalize_transfer(client, config, state->remove_sources, &delete_limit, &partial,
|
||||||
|
&recv_stats);
|
||||||
if (!ok && config->use_delete)
|
if (!ok && config->use_delete)
|
||||||
log_message(LOG_LEVEL_ERROR,
|
log_message(LOG_LEVEL_ERROR,
|
||||||
"server reported a deletion failure (--delete); see the server log for the reason");
|
"server reported a deletion failure (--delete); see the server log for the reason");
|
||||||
@@ -1759,12 +2000,12 @@ static int send_files_finalize(const Config* config, SendFilesState* state) {
|
|||||||
(double)state->transfer_stats.transferred_file_size / (double)BYTES_PER_MIB);
|
(double)state->transfer_stats.transferred_file_size / (double)BYTES_PER_MIB);
|
||||||
/* A skipped source entry (--ignore-errors past an unreadable directory, or a
|
/* A skipped source entry (--ignore-errors past an unreadable directory, or a
|
||||||
dereferenced symlink with no referent) makes rsync report a partial
|
dereferenced symlink with no referent) makes rsync report a partial
|
||||||
transfer (exit 23) even though the rest of the run succeeded. A
|
transfer (exit 23), as does a receiver per-entry failure (STATUS_PARTIAL).
|
||||||
--max-delete-capped commit is a successful transfer that rsync reports
|
A --max-delete-capped commit is a successful transfer that rsync reports
|
||||||
with exit code 25. */
|
with exit code 25. */
|
||||||
if (!ok)
|
if (!ok)
|
||||||
return 1;
|
return 1;
|
||||||
if (state->had_scan_io)
|
if (state->had_scan_io || partial)
|
||||||
return 23;
|
return 23;
|
||||||
return delete_limit ? 25 : 0;
|
return delete_limit ? 25 : 0;
|
||||||
}
|
}
|
||||||
@@ -1774,6 +2015,8 @@ static int send_files_finalize(const Config* config, SendFilesState* state) {
|
|||||||
static void send_files_cleanup(SendFilesState* state) {
|
static void send_files_cleanup(SendFilesState* state) {
|
||||||
if (state->manifest)
|
if (state->manifest)
|
||||||
array_list_delete(state->manifest);
|
array_list_delete(state->manifest);
|
||||||
|
if (state->prescan_chunks)
|
||||||
|
array_list_delete(state->prescan_chunks);
|
||||||
if (state->plan_sender)
|
if (state->plan_sender)
|
||||||
delete_plan_sender_destroy(state->plan_sender);
|
delete_plan_sender_destroy(state->plan_sender);
|
||||||
if (state->excluded)
|
if (state->excluded)
|
||||||
@@ -1786,6 +2029,8 @@ static void send_files_cleanup(SendFilesState* state) {
|
|||||||
array_list_delete(state->plan_dirs);
|
array_list_delete(state->plan_dirs);
|
||||||
if (state->missing_args)
|
if (state->missing_args)
|
||||||
array_list_delete(state->missing_args);
|
array_list_delete(state->missing_args);
|
||||||
|
if (state->per_dir_rules)
|
||||||
|
filter_rule_list_free(state->per_dir_rules);
|
||||||
if (state->remove_sources)
|
if (state->remove_sources)
|
||||||
array_list_delete(state->remove_sources);
|
array_list_delete(state->remove_sources);
|
||||||
if (state->dir_entries)
|
if (state->dir_entries)
|
||||||
@@ -1799,7 +2044,18 @@ static void send_files_cleanup(SendFilesState* state) {
|
|||||||
client_set_abort_armed(false);
|
client_set_abort_armed(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static int send_files_impl(Config* config);
|
||||||
|
|
||||||
int send_files(Config* config) {
|
int send_files(Config* config) {
|
||||||
|
/* Install the --stderr=client sink for the whole run (it only queues while a
|
||||||
|
session is live) and release its queue on every return path. */
|
||||||
|
client_messages_install();
|
||||||
|
int rc = send_files_impl(config);
|
||||||
|
client_messages_end();
|
||||||
|
return rc;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int send_files_impl(Config* config) {
|
||||||
if (config->list_only)
|
if (config->list_only)
|
||||||
return send_list_only(config);
|
return send_list_only(config);
|
||||||
if (config->dry_run)
|
if (config->dry_run)
|
||||||
@@ -1816,35 +2072,21 @@ int send_files(Config* config) {
|
|||||||
shielded -- rsync's `-d DIR/ --delete`. */
|
shielded -- rsync's `-d DIR/ --delete`. */
|
||||||
state.delete_per_dir = config->use_delete && config_delete_timing_per_dir(config);
|
state.delete_per_dir = config->use_delete && config_delete_timing_per_dir(config);
|
||||||
int skipped = 0;
|
int skipped = 0;
|
||||||
if (config->delete_missing_args) {
|
if (!client_prepare_files_from(config, &state.missing_args, &skipped))
|
||||||
state.missing_args = array_list_create(free);
|
|
||||||
if (!state.missing_args)
|
|
||||||
return 1;
|
return 1;
|
||||||
}
|
|
||||||
if (!files_from_list_check(config, state.missing_args, &skipped)) {
|
|
||||||
if (state.missing_args)
|
|
||||||
array_list_delete(state.missing_args);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* From here on a server session may be live, so Ctrl-C/SIGTERM should set the
|
/* From here on a server session may be live, so Ctrl-C/SIGTERM should set the
|
||||||
abort flag (and be forwarded as STATUS_ABORT) instead of terminating. */
|
abort flag (and be forwarded as STATUS_ABORT) instead of terminating. */
|
||||||
client_set_abort_armed(true);
|
client_set_abort_armed(true);
|
||||||
Client* client = connect_transfer_client(config);
|
ProtocolSession session;
|
||||||
|
Client* client = client_connect_and_bind_session(config, &session);
|
||||||
if (!client) {
|
if (!client) {
|
||||||
if (config->transport == TRANSPORT_TCP)
|
|
||||||
log_message(LOG_LEVEL_ERROR, "could not connect to server%s",
|
|
||||||
config->use_tls ? " via TLS" : "");
|
|
||||||
if (state.missing_args)
|
if (state.missing_args)
|
||||||
array_list_delete(state.missing_args);
|
array_list_delete(state.missing_args);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
state.client = client;
|
state.client = client;
|
||||||
ProtocolSession session;
|
client_messages_activate(true);
|
||||||
protocol_session_init(&session, client->file_descriptor, client->file_descriptor);
|
|
||||||
protocol_session_set_io_timeout(&session, config->timeout);
|
|
||||||
protocol_session_set_ssl(&session, (SSL*)client->ssl);
|
|
||||||
protocol_session_bind(&session);
|
|
||||||
|
|
||||||
int ret = 1;
|
int ret = 1;
|
||||||
if (!send_files_prepare(config, &state))
|
if (!send_files_prepare(config, &state))
|
||||||
@@ -1860,7 +2102,16 @@ send_fail:
|
|||||||
return ret;
|
return ret;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static int send_files_multithreaded_impl(Config* config);
|
||||||
|
|
||||||
int send_files_multithreaded(Config* config) {
|
int send_files_multithreaded(Config* config) {
|
||||||
|
client_messages_install();
|
||||||
|
int rc = send_files_multithreaded_impl(config);
|
||||||
|
client_messages_end();
|
||||||
|
return rc;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int send_files_multithreaded_impl(Config* config) {
|
||||||
if (!config)
|
if (!config)
|
||||||
return 1;
|
return 1;
|
||||||
if (config->list_only)
|
if (config->list_only)
|
||||||
@@ -1870,16 +2121,8 @@ int send_files_multithreaded(Config* config) {
|
|||||||
: send_dry_run_manifest(config);
|
: send_dry_run_manifest(config);
|
||||||
ArrayList* missing_args = NULL;
|
ArrayList* missing_args = NULL;
|
||||||
int skipped = 0;
|
int skipped = 0;
|
||||||
if (config->delete_missing_args) {
|
if (!client_prepare_files_from(config, &missing_args, &skipped))
|
||||||
missing_args = array_list_create(free);
|
|
||||||
if (!missing_args)
|
|
||||||
return 1;
|
return 1;
|
||||||
}
|
|
||||||
if (!files_from_list_check(config, missing_args, &skipped)) {
|
|
||||||
if (missing_args)
|
|
||||||
array_list_delete(missing_args);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Armed only once a session may go live (see send_files). */
|
/* Armed only once a session may go live (see send_files). */
|
||||||
client_set_abort_armed(true);
|
client_set_abort_armed(true);
|
||||||
@@ -1908,6 +2151,8 @@ int send_files_multithreaded(Config* config) {
|
|||||||
queue_destroy(q1);
|
queue_destroy(q1);
|
||||||
if (q2)
|
if (q2)
|
||||||
queue_destroy(q2);
|
queue_destroy(q2);
|
||||||
|
if (missing_args)
|
||||||
|
array_list_delete(missing_args);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
PipelineContextSender* context = pipeline_context_sender_create(config, q1, q2);
|
PipelineContextSender* context = pipeline_context_sender_create(config, q1, q2);
|
||||||
@@ -1946,7 +2191,8 @@ int send_files_multithreaded(Config* config) {
|
|||||||
confined; only a --files-from subset records concrete directories. */
|
confined; only a --files-from subset records concrete directories. */
|
||||||
context->size_skipped_paths = array_list_create(free);
|
context->size_skipped_paths = array_list_create(free);
|
||||||
context->synced_dirs = array_list_create(free);
|
context->synced_dirs = array_list_create(free);
|
||||||
if (!context->size_skipped_paths || !context->synced_dirs) {
|
context->per_dir_rules = filter_rule_list_create();
|
||||||
|
if (!context->size_skipped_paths || !context->synced_dirs || !context->per_dir_rules) {
|
||||||
pipeline_context_sender_destroy(context);
|
pipeline_context_sender_destroy(context);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
@@ -1975,6 +2221,7 @@ int send_files_multithreaded(Config* config) {
|
|||||||
if (context->excluded_paths)
|
if (context->excluded_paths)
|
||||||
prepared.options.excluded_paths = context->excluded_paths;
|
prepared.options.excluded_paths = context->excluded_paths;
|
||||||
prepared.options.size_skipped_paths = context->size_skipped_paths;
|
prepared.options.size_skipped_paths = context->size_skipped_paths;
|
||||||
|
prepared.options.per_dir_rules = context->per_dir_rules;
|
||||||
/* The root marker for a full recursive transfer is already in the list;
|
/* The root marker for a full recursive transfer is already in the list;
|
||||||
only a --files-from subset needs the scanner to record directories. */
|
only a --files-from subset needs the scanner to record directories. */
|
||||||
if (config->files_from_set != NULL)
|
if (config->files_from_set != NULL)
|
||||||
@@ -1987,13 +2234,27 @@ int send_files_multithreaded(Config* config) {
|
|||||||
if (prepared_ok)
|
if (prepared_ok)
|
||||||
prepared.options.plan_dirs = context->plan_dirs;
|
prepared.options.plan_dirs = context->plan_dirs;
|
||||||
} else {
|
} else {
|
||||||
|
/* --delete-before: retain the pre-scan chunks as the pipeline's data
|
||||||
|
pass (rsync's single file list) so a source file created after the
|
||||||
|
scan is not transferred. No later scan runs, so this pass must also
|
||||||
|
capture the deferred directory times and the --stats directory
|
||||||
|
count. */
|
||||||
context->manifest = array_list_create(free);
|
context->manifest = array_list_create(free);
|
||||||
prepared_ok = prepared_ok && context->manifest != NULL;
|
context->prescan_chunks = array_list_create(chunk_destroy);
|
||||||
|
prepared_ok = prepared_ok && context->manifest != NULL && context->prescan_chunks != NULL;
|
||||||
|
if (prepared_ok) {
|
||||||
|
prepared.options.dir_entries = context->dir_entries;
|
||||||
|
prepared.options.dir_entries_mutex = &context->dir_entries_mutex;
|
||||||
|
prepared.options.dir_count = config->stats ? &context->dir_count : NULL;
|
||||||
|
if (!append_implied_dir_times(config, context->dir_entries))
|
||||||
|
prepared_ok = false;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
bool prebuilt =
|
bool prebuilt =
|
||||||
prepared_ok &&
|
prepared_ok &&
|
||||||
scan_paths_only(config, &prepared.options, context->manifest, context->delete_plans,
|
scan_paths_only(config, &prepared.options, context->manifest, context->delete_plans,
|
||||||
&context->scan_had_io_error, &pre_scan_non_dir);
|
&context->scan_had_io_error, &pre_scan_non_dir, context->prescan_chunks,
|
||||||
|
context->prescan_chunks != NULL);
|
||||||
prepared_scanner_destroy(&prepared);
|
prepared_scanner_destroy(&prepared);
|
||||||
if (per_dir && prebuilt) {
|
if (per_dir && prebuilt) {
|
||||||
const char* walk_root = delete_plan_walk_root(config, context->synced_dirs);
|
const char* walk_root = delete_plan_walk_root(config, context->synced_dirs);
|
||||||
@@ -2001,7 +2262,8 @@ int send_files_multithreaded(Config* config) {
|
|||||||
: (walk_root ? context->synced_dirs : NULL);
|
: (walk_root ? context->synced_dirs : NULL);
|
||||||
delete_plan_sender_finalize(context->delete_plans, scope, walk_root);
|
delete_plan_sender_finalize(context->delete_plans, scope, walk_root);
|
||||||
delete_plan_sender_set_config(context->delete_plans, context->excluded_paths,
|
delete_plan_sender_set_config(context->delete_plans, context->excluded_paths,
|
||||||
context->size_skipped_paths, context->missing_args);
|
context->size_skipped_paths, context->missing_args,
|
||||||
|
context->per_dir_rules);
|
||||||
}
|
}
|
||||||
bool empty = per_dir
|
bool empty = per_dir
|
||||||
? (context->delete_plans && delete_plan_sender_empty(context->delete_plans))
|
? (context->delete_plans && delete_plan_sender_empty(context->delete_plans))
|
||||||
@@ -2102,16 +2364,18 @@ int send_files_multithreaded(Config* config) {
|
|||||||
mtx_unlock(&context->mutex_scanner);
|
mtx_unlock(&context->mutex_scanner);
|
||||||
bool sender_ok = sender_result == thrd_success;
|
bool sender_ok = sender_result == thrd_success;
|
||||||
bool delete_limit = context->delete_limit;
|
bool delete_limit = context->delete_limit;
|
||||||
|
bool partial = context->partial;
|
||||||
/* A skipped source entry (--ignore-errors past an unreadable directory, or a
|
/* A skipped source entry (--ignore-errors past an unreadable directory, or a
|
||||||
dereferenced symlink with no referent) makes rsync report a partial
|
dereferenced symlink with no referent) makes rsync report a partial
|
||||||
transfer (exit 23). A --max-delete-capped commit is a successful transfer
|
transfer (exit 23), as does a receiver per-entry failure (STATUS_PARTIAL).
|
||||||
that rsync reports with exit code 25. */
|
A --max-delete-capped commit is a successful transfer that rsync reports
|
||||||
|
with exit code 25. */
|
||||||
pipeline_context_sender_destroy(context);
|
pipeline_context_sender_destroy(context);
|
||||||
client_progress_cleanup();
|
client_progress_cleanup();
|
||||||
client_set_abort_armed(false);
|
client_set_abort_armed(false);
|
||||||
if (!sender_ok)
|
if (!sender_ok)
|
||||||
return 1;
|
return 1;
|
||||||
if (scan_io)
|
if (scan_io || partial)
|
||||||
return 23;
|
return 23;
|
||||||
return delete_limit ? 25 : 0;
|
return delete_limit ? 25 : 0;
|
||||||
}
|
}
|
||||||
@@ -13,6 +13,7 @@
|
|||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
#include "format.h"
|
#include "format.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
|
#include "protocol.h"
|
||||||
#include "scanner.h"
|
#include "scanner.h"
|
||||||
#include <stdatomic.h>
|
#include <stdatomic.h>
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
@@ -44,7 +45,8 @@ const char* delete_plan_walk_root(const Config* config, const ArrayList* synced_
|
|||||||
bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* skipped_out);
|
bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* skipped_out);
|
||||||
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
|
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
|
||||||
DeletePlanSender* plans, bool* io_error_out,
|
DeletePlanSender* plans, bool* io_error_out,
|
||||||
unsigned long long* non_dir_count_out);
|
unsigned long long* non_dir_count_out, ArrayList* chunks_out,
|
||||||
|
bool emit_nonreg);
|
||||||
|
|
||||||
/* client_report.c */
|
/* client_report.c */
|
||||||
void log_server_rejection(const char* context);
|
void log_server_rejection(const char* context);
|
||||||
@@ -67,14 +69,43 @@ void client_progress_name(const Config* config, const File* file);
|
|||||||
/* Emit a transferred entry's ancestor directories (as -i/--out-format change
|
/* Emit a transferred entry's ancestor directories (as -i/--out-format change
|
||||||
* lines or --progress name lines) before the entry's own line. */
|
* lines or --progress name lines) before the entry's own line. */
|
||||||
void client_change_emit_ancestors(const Config* config, const File* file);
|
void client_change_emit_ancestors(const Config* config, const File* file);
|
||||||
|
/* Output parity (protocol 2.30.0): probe each not-yet-known ancestor directory's
|
||||||
|
* pre-transfer destination state before the entry that first triggers it is
|
||||||
|
* sent. Returns false on a protocol/transport error. */
|
||||||
|
bool client_change_probe_ancestors(const Config* config, const File* file, int fd);
|
||||||
|
/* Mark a transferred directory entry as already reported, and flush the
|
||||||
|
* itemize lines for changed directories that had no transferred child. */
|
||||||
|
void client_change_mark_dir(const Config* config, const File* file);
|
||||||
|
void client_change_emit_pending_dirs(const Config* config, int fd);
|
||||||
void client_progress_uptodate(const Config* config, const File* file);
|
void client_progress_uptodate(const Config* config, const File* file);
|
||||||
void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
|
void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
|
||||||
unsigned long long plan_non_dir_count);
|
unsigned long long plan_non_dir_count);
|
||||||
bool receive_stats_record(int fd, ReceiverStats* stats, ArrayList* would_delete);
|
bool receive_stats_record(int fd, ReceiverStats* stats, ArrayList* would_delete);
|
||||||
|
/* --stderr=client diagnostic channel (client_report.c): install the queueing
|
||||||
|
* log sink for a transfer, mark the session live, flush queued diagnostics over
|
||||||
|
* the wire at a frame boundary, and tear the sink down. */
|
||||||
|
void client_messages_install(void);
|
||||||
|
void client_messages_activate(bool active);
|
||||||
|
void client_flush_client_messages(int fd);
|
||||||
|
void client_messages_end(void);
|
||||||
|
|
||||||
/* client_send.c */
|
/* client_send.c */
|
||||||
void receive_daemon_motd(Client* client, const Config* config);
|
void receive_daemon_motd(Client* client, const Config* config);
|
||||||
Client* connect_transfer_client(const Config* config);
|
Client* connect_transfer_client(const Config* config);
|
||||||
|
/* Connect the configured transport and install `session` on it: init with the
|
||||||
|
* socket fd pair, apply the I/O timeout and (when negotiated) the TLS object,
|
||||||
|
* then bind the session to this thread. Returns the connected client, or NULL
|
||||||
|
* after logging the connect failure. The caller owns the client and must keep
|
||||||
|
* `session` alive until it calls protocol_session_unbind(). */
|
||||||
|
Client* client_connect_and_bind_session(const Config* config, ProtocolSession* session);
|
||||||
|
/* Shared --files-from/--delete-missing-args preamble for the send entry points:
|
||||||
|
* when --delete-missing-args is set, allocate the list that
|
||||||
|
* files_from_list_check fills with the destination mirrors of missing entries;
|
||||||
|
* then validate the --files-from list. On success returns true and stores the
|
||||||
|
* (possibly NULL) owned list in *missing_args_out plus the skipped count; on
|
||||||
|
* failure returns false after freeing the list. */
|
||||||
|
bool client_prepare_files_from(const Config* config, ArrayList** missing_args_out,
|
||||||
|
int* skipped_out);
|
||||||
void disconnect_transfer_client(Client* client);
|
void disconnect_transfer_client(Client* client);
|
||||||
int incremental_check(Client* client, File* file, const Config* config, DeltaSignature** out_sig,
|
int incremental_check(Client* client, File* file, const Config* config, DeltaSignature** out_sig,
|
||||||
unsigned long long* resume_offset);
|
unsigned long long* resume_offset);
|
||||||
@@ -86,9 +117,10 @@ int send_dry_run_manifest(const Config* config);
|
|||||||
int send_list_only(const Config* config);
|
int send_list_only(const Config* config);
|
||||||
int send_dry_run_remote(Config* config);
|
int send_dry_run_remote(Config* config);
|
||||||
int send_delete_manifest(int fd, ArrayList* manifest, ArrayList* protected_prefixes,
|
int send_delete_manifest(int fd, ArrayList* manifest, ArrayList* protected_prefixes,
|
||||||
ArrayList* size_skipped, ArrayList* missing_args, ArrayList* synced_dirs);
|
ArrayList* size_skipped, ArrayList* missing_args, ArrayList* synced_dirs,
|
||||||
|
const FilterRuleList* per_dir_rules);
|
||||||
bool send_delete_manifest_early(Client* client, ArrayList* manifest, ArrayList* protected_prefixes,
|
bool send_delete_manifest_early(Client* client, ArrayList* manifest, ArrayList* protected_prefixes,
|
||||||
ArrayList* size_skipped, ArrayList* missing_args,
|
ArrayList* size_skipped, ArrayList* missing_args,
|
||||||
ArrayList* synced_dirs);
|
ArrayList* synced_dirs, const FilterRuleList* per_dir_rules);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
+12
-52
@@ -149,7 +149,7 @@ DirectoryScanner* directory_scanner_create_with_options(const char* root_directo
|
|||||||
scanner->options = *options;
|
scanner->options = *options;
|
||||||
if (scanner->options.chunk_size == 0)
|
if (scanner->options.chunk_size == 0)
|
||||||
scanner->options.chunk_size = DESIRED_CHUNK_SIZE;
|
scanner->options.chunk_size = DESIRED_CHUNK_SIZE;
|
||||||
scanner->directories = queue_create(100, dir_entry_destroy);
|
scanner->directories = queue_create(SCANNER_RESULT_QUEUE_CAP, dir_entry_destroy);
|
||||||
if (!scanner->directories) {
|
if (!scanner->directories) {
|
||||||
free(scanner);
|
free(scanner);
|
||||||
return NULL;
|
return NULL;
|
||||||
@@ -1026,7 +1026,7 @@ static ScannerAction scanner_process_entry(DirectoryScanner* scanner, ArrayList*
|
|||||||
Chunk** out_chunk) {
|
Chunk** out_chunk) {
|
||||||
const char* name = sorted->name;
|
const char* name = sorted->name;
|
||||||
ScannerEntry* inspected = &sorted->entry;
|
ScannerEntry* inspected = &sorted->entry;
|
||||||
char* cur_path = inspected->path;
|
const char* cur_path = inspected->path;
|
||||||
struct stat stats = inspected->stats;
|
struct stat stats = inspected->stats;
|
||||||
|
|
||||||
/* --files-from allow-set and the filter layer apply to files and to
|
/* --files-from allow-set and the filter layer apply to files and to
|
||||||
@@ -1101,61 +1101,23 @@ static ScannerAction scanner_process_entry(DirectoryScanner* scanner, ArrayList*
|
|||||||
free(rel_copy);
|
free(rel_copy);
|
||||||
return SCANNER_ACTION_CONTINUE;
|
return SCANNER_ACTION_CONTINUE;
|
||||||
}
|
}
|
||||||
File* file = file_create(cur_path);
|
/* Entry construction (data size, -R wire path, special/devices, hardlink
|
||||||
if (file == NULL) {
|
group, metadata, xattrs) is shared with the parallel scanner. */
|
||||||
free(rel_copy);
|
File* file = NULL;
|
||||||
free(inspected->link_target);
|
bool build_failed = false;
|
||||||
inspected->link_target = NULL;
|
ScannerBuildStatus status =
|
||||||
scanner->failed = true;
|
scanner_build_file_entry(&scanner->options, inspected, rel_copy, &file, &build_failed);
|
||||||
return SCANNER_ACTION_CONTINUE;
|
|
||||||
}
|
|
||||||
if (inspected->is_symlink) {
|
|
||||||
file->is_symlink = true;
|
|
||||||
file->symlink_target = inspected->link_target;
|
|
||||||
inspected->link_target = NULL;
|
|
||||||
} else {
|
|
||||||
file->data->size = stats.st_size;
|
|
||||||
}
|
|
||||||
if (scanner->relative_mode) {
|
|
||||||
file->send_path = rel_copy;
|
|
||||||
rel_copy = NULL;
|
|
||||||
} else if (scanner->options.relative_prefix) {
|
|
||||||
file->send_path = scanner_prefix_send_path(scanner->options.relative_prefix, rel_copy);
|
|
||||||
free(rel_copy);
|
free(rel_copy);
|
||||||
rel_copy = NULL;
|
rel_copy = NULL;
|
||||||
if (!file->send_path) {
|
if (build_failed)
|
||||||
file_destroy(file);
|
|
||||||
scanner->failed = true;
|
scanner->failed = true;
|
||||||
return SCANNER_ACTION_BREAK;
|
if (status == SCANNER_BUILD_SKIP)
|
||||||
}
|
|
||||||
}
|
|
||||||
/* --devices/--specials: a device/FIFO/socket entry marked for preservation
|
|
||||||
becomes a node to recreate (is_special, no data, rdev captured); an
|
|
||||||
unrequested non-regular entry is skipped (rsync default). */
|
|
||||||
ScannerSpecial special =
|
|
||||||
scanner_prepare_special(scanner->options.preserve_devices, scanner->options.preserve_specials,
|
|
||||||
scanner->options.copy_devices, file, &stats);
|
|
||||||
if (special == SCANNER_SPECIAL_SKIP) {
|
|
||||||
scanner_note_nonreg(&scanner->options, file->path);
|
|
||||||
free(rel_copy);
|
|
||||||
file_destroy(file);
|
|
||||||
return SCANNER_ACTION_CONTINUE;
|
return SCANNER_ACTION_CONTINUE;
|
||||||
}
|
if (status != SCANNER_BUILD_OK) {
|
||||||
if (scanner->options.hardlinks && S_ISREG(stats.st_mode))
|
|
||||||
scanner_assign_hardlink(scanner, scanner->options.hardlinks, file, &stats);
|
|
||||||
if (scanner->options.use_metadata)
|
|
||||||
file->metadata = file_metadata_create(file->path, &stats, scanner->options.preserve_atimes,
|
|
||||||
scanner->options.preserve_crtimes);
|
|
||||||
if (scanner->options.use_metadata && !file->metadata) {
|
|
||||||
free(rel_copy);
|
|
||||||
file_destroy(file);
|
|
||||||
scanner->failed = true;
|
scanner->failed = true;
|
||||||
return SCANNER_ACTION_BREAK;
|
return status == SCANNER_BUILD_FAIL_CONTINUE ? SCANNER_ACTION_CONTINUE : SCANNER_ACTION_BREAK;
|
||||||
}
|
}
|
||||||
if (!(file->link_group != 0 && !file->link_first))
|
|
||||||
scanner_capture_xattrs(scanner, file);
|
|
||||||
if (!array_list_add(chunk_data, file)) {
|
if (!array_list_add(chunk_data, file)) {
|
||||||
free(rel_copy);
|
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
scanner->failed = true;
|
scanner->failed = true;
|
||||||
return SCANNER_ACTION_BREAK;
|
return SCANNER_ACTION_BREAK;
|
||||||
@@ -1163,14 +1125,12 @@ static ScannerAction scanner_process_entry(DirectoryScanner* scanner, ArrayList*
|
|||||||
scanner->current_dir_produced = true;
|
scanner->current_dir_produced = true;
|
||||||
*chunk_data_size += file->data->size;
|
*chunk_data_size += file->data->size;
|
||||||
if (*chunk_data_size > scanner->options.chunk_size) {
|
if (*chunk_data_size > scanner->options.chunk_size) {
|
||||||
free(rel_copy);
|
|
||||||
Chunk* result = chunk_data_to_chunk(chunk_data);
|
Chunk* result = chunk_data_to_chunk(chunk_data);
|
||||||
if (!result)
|
if (!result)
|
||||||
scanner->failed = true;
|
scanner->failed = true;
|
||||||
*out_chunk = result;
|
*out_chunk = result;
|
||||||
return SCANNER_ACTION_CHUNK;
|
return SCANNER_ACTION_CHUNK;
|
||||||
}
|
}
|
||||||
free(rel_copy);
|
|
||||||
return SCANNER_ACTION_CONTINUE;
|
return SCANNER_ACTION_CONTINUE;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -19,6 +19,11 @@
|
|||||||
* keeps one transfer from spawning an unbounded pool on a very large machine. */
|
* keeps one transfer from spawning an unbounded pool on a very large machine. */
|
||||||
#define MAX_SCANNER_THREADS 256
|
#define MAX_SCANNER_THREADS 256
|
||||||
|
|
||||||
|
/* Depth of the scanner's work queues: the sequential scanner's pending-directory
|
||||||
|
* stack and the parallel scanner's result queue. Bounds memory for a very wide
|
||||||
|
* or very deep tree while leaving ample headroom for normal scans. */
|
||||||
|
#define SCANNER_RESULT_QUEUE_CAP 100
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
bool use_metadata;
|
bool use_metadata;
|
||||||
/* Phase 4 metadata capture: -U/--atimes and -N/--crtimes tell the scanner to
|
/* Phase 4 metadata capture: -U/--atimes and -N/--crtimes tell the scanner to
|
||||||
@@ -115,6 +120,13 @@ typedef struct {
|
|||||||
* directories, exactly like rsync; the receive root is the "." sentinel.
|
* directories, exactly like rsync; the receive root is the "." sentinel.
|
||||||
* Guarded by `excluded_mutex`. */
|
* Guarded by `excluded_mutex`. */
|
||||||
ArrayList* synced_dirs;
|
ArrayList* synced_dirs;
|
||||||
|
/* Per-directory filter-rule sink (optional): when non-NULL the scanner appends
|
||||||
|
* a deep copy of every rule it reads from a per-directory merge file, each
|
||||||
|
* carrying its owner directory and no-inherit flag (see filter.h). The delete
|
||||||
|
* carriers transmit them so the receiver re-derives the per-directory
|
||||||
|
* protect/risk set for destination-only entries. Guarded by `excluded_mutex`
|
||||||
|
* like the other sinks. */
|
||||||
|
FilterRuleList* per_dir_rules;
|
||||||
/* Delete-plan directory sink (optional): when non-NULL the scanner appends
|
/* Delete-plan directory sink (optional): when non-NULL the scanner appends
|
||||||
* the destination-relative path of every directory it traverses (except the
|
* the destination-relative path of every directory it traverses (except the
|
||||||
* receive root). The per-directory --delete-during/--delete-delay plan
|
* receive root). The per-directory --delete-during/--delete-delay plan
|
||||||
|
|||||||
+151
-29
@@ -282,31 +282,37 @@ bool entry_passes_selection(const FileListSet* file_list, const FilterRuleList*
|
|||||||
}
|
}
|
||||||
|
|
||||||
/* Best-effort capture of the file's whitelisted xattrs (-X/-A). A failure to
|
/* Best-effort capture of the file's whitelisted xattrs (-X/-A). A failure to
|
||||||
* read xattrs is non-fatal: the file is transferred without them. */
|
* read xattrs is non-fatal: the file is transferred without them. A symlink
|
||||||
void scanner_capture_xattrs(const DirectoryScanner* scanner, File* file) {
|
* entry reads the LINK's own xattrs (never the referent's) with the no-follow
|
||||||
if (!scanner || !file || !(scanner->options.preserve_xattrs || scanner->options.preserve_acls))
|
* variant; on Linux the VFS refuses xattrs on symlinks, so that yields NULL. */
|
||||||
|
void scanner_capture_xattrs_opts(const ScannerOptions* options, File* file) {
|
||||||
|
if (!options || !file || !(options->preserve_xattrs || options->preserve_acls))
|
||||||
return;
|
return;
|
||||||
file->xattrs = xattr_capture_path(file->path, scanner->options.preserve_acls);
|
file->xattrs = file->is_symlink ? xattr_capture_path_nofollow(file->path, options->preserve_acls)
|
||||||
|
: xattr_capture_path(file->path, options->preserve_acls);
|
||||||
|
}
|
||||||
|
|
||||||
|
void scanner_capture_xattrs(const DirectoryScanner* scanner, File* file) {
|
||||||
|
if (!scanner)
|
||||||
|
return;
|
||||||
|
scanner_capture_xattrs_opts(&scanner->options, file);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Apply --hard-links (-H) detection to one regular File. On a sibling (a
|
/* Apply --hard-links (-H) detection to one regular File. On a sibling (a
|
||||||
* later member of an already-seen source inode) the File keeps the group id
|
* later member of an already-seen source inode) the File keeps the group id
|
||||||
* and the first member's wire path but carries NO data payload (size 0); the
|
* and the first member's wire path but carries NO data payload (size 0); the
|
||||||
* first member is left untouched (data present, link_first). Allocation
|
* first member is left untouched (data present, link_first). Returns false on
|
||||||
* failure is fatal: the scanner is marked failed. */
|
* allocation failure (the caller marks the scan failed); the File stays usable
|
||||||
void scanner_assign_hardlink(DirectoryScanner* scanner, HardLinkTable* table, File* file,
|
* either way. */
|
||||||
const struct stat* stats) {
|
bool scanner_assign_hardlink(HardLinkTable* table, File* file, const struct stat* stats) {
|
||||||
if (!table || !file || !stats)
|
if (!table || !file || !stats)
|
||||||
return;
|
return true;
|
||||||
int gid;
|
int gid;
|
||||||
bool is_first;
|
bool is_first;
|
||||||
char* first_path = NULL;
|
char* first_path = NULL;
|
||||||
if (!hardlink_table_assign(table, file_wire_path(file), stats->st_dev, stats->st_ino, &gid,
|
if (!hardlink_table_assign(table, file_wire_path(file), stats->st_dev, stats->st_ino, &gid,
|
||||||
&is_first, &first_path)) {
|
&is_first, &first_path))
|
||||||
if (scanner)
|
return false;
|
||||||
scanner->failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
file->link_group = gid;
|
file->link_group = gid;
|
||||||
file->link_first = is_first;
|
file->link_first = is_first;
|
||||||
if (!is_first) {
|
if (!is_first) {
|
||||||
@@ -315,6 +321,7 @@ void scanner_assign_hardlink(DirectoryScanner* scanner, HardLinkTable* table, Fi
|
|||||||
} else {
|
} else {
|
||||||
free(first_path);
|
free(first_path);
|
||||||
}
|
}
|
||||||
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Phase 4 special/devices decision for one non-regular entry, matching rsync:
|
/* Phase 4 special/devices decision for one non-regular entry, matching rsync:
|
||||||
@@ -395,6 +402,76 @@ void scanner_note_nonreg(const ScannerOptions* options, const char* fs_path) {
|
|||||||
fflush(stdout);
|
fflush(stdout);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Construct one non-directory File from an inspected entry. Shared by the
|
||||||
|
* sequential and parallel scanners so entry construction has a single
|
||||||
|
* implementation: data size (or carried symlink), -R wire path, special/devices
|
||||||
|
* classification, hardlink group, metadata and xattr capture all happen here in
|
||||||
|
* the same order for both. See the declaration for the ownership contract. */
|
||||||
|
ScannerBuildStatus scanner_build_file_entry(const ScannerOptions* options, ScannerEntry* inspected,
|
||||||
|
const char* rel, File** out_file, bool* failed) {
|
||||||
|
*out_file = NULL;
|
||||||
|
if (failed)
|
||||||
|
*failed = false;
|
||||||
|
File* file = file_create(inspected->path);
|
||||||
|
if (!file) {
|
||||||
|
/* The File never existed, so drop the not-yet-transferred symlink target
|
||||||
|
here; the caller's entry teardown would otherwise double-free it. */
|
||||||
|
free(inspected->link_target);
|
||||||
|
inspected->link_target = NULL;
|
||||||
|
return SCANNER_BUILD_FAIL_CONTINUE;
|
||||||
|
}
|
||||||
|
if (inspected->is_symlink) {
|
||||||
|
file->is_symlink = true;
|
||||||
|
file->symlink_target = inspected->link_target;
|
||||||
|
inspected->link_target = NULL;
|
||||||
|
} else {
|
||||||
|
file->data->size = inspected->stats.st_size;
|
||||||
|
}
|
||||||
|
/* -R + --files-from uses the bare transfer-relative path; -R without
|
||||||
|
--files-from prefixes it. Plain scans keep the source path. */
|
||||||
|
bool relative_mode = options->relative && options->file_list != NULL;
|
||||||
|
if (relative_mode) {
|
||||||
|
file->send_path = str_dup(rel);
|
||||||
|
} else if (options->relative_prefix) {
|
||||||
|
file->send_path = scanner_prefix_send_path(options->relative_prefix, rel);
|
||||||
|
}
|
||||||
|
if ((relative_mode || options->relative_prefix) && !file->send_path) {
|
||||||
|
file_destroy(file);
|
||||||
|
return SCANNER_BUILD_FAIL_BREAK;
|
||||||
|
}
|
||||||
|
/* --devices/--specials: a device/FIFO/socket entry marked for preservation
|
||||||
|
becomes a node to recreate (is_special, no data, rdev captured); an
|
||||||
|
unrequested non-regular entry is skipped (rsync default). */
|
||||||
|
ScannerSpecial special =
|
||||||
|
scanner_prepare_special(options->preserve_devices, options->preserve_specials,
|
||||||
|
options->copy_devices, file, &inspected->stats);
|
||||||
|
if (special == SCANNER_SPECIAL_SKIP) {
|
||||||
|
scanner_note_nonreg(options, file->path);
|
||||||
|
file_destroy(file);
|
||||||
|
return SCANNER_BUILD_SKIP;
|
||||||
|
}
|
||||||
|
if (options->hardlinks && S_ISREG(inspected->stats.st_mode) &&
|
||||||
|
!scanner_assign_hardlink(options->hardlinks, file, &inspected->stats)) {
|
||||||
|
/* Allocation failure is non-fatal to this entry (it is still emitted) but
|
||||||
|
marks the scan failed, matching the historical inlined behaviour. */
|
||||||
|
if (failed)
|
||||||
|
*failed = true;
|
||||||
|
}
|
||||||
|
if (options->use_metadata) {
|
||||||
|
file->metadata = file_metadata_create(file->path, &inspected->stats, options->preserve_atimes,
|
||||||
|
options->preserve_crtimes);
|
||||||
|
if (!file->metadata) {
|
||||||
|
file_destroy(file);
|
||||||
|
return SCANNER_BUILD_FAIL_BREAK;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
/* A hardlink sibling carries no data, so it carries no xattrs. */
|
||||||
|
if (!(file->link_group != 0 && !file->link_first))
|
||||||
|
scanner_capture_xattrs_opts(options, file);
|
||||||
|
*out_file = file;
|
||||||
|
return SCANNER_BUILD_OK;
|
||||||
|
}
|
||||||
|
|
||||||
/* rsync 3.4.1's `--info=mount` line, emitted when `-xx` drops a mount-point
|
/* rsync 3.4.1's `--info=mount` line, emitted when `-xx` drops a mount-point
|
||||||
* directory: `[sender] skipping mount-point dir NAME` (the client is the
|
* directory: `[sender] skipping mount-point dir NAME` (the client is the
|
||||||
* sender). Plain `-x` keeps the empty directory and prints nothing, matching
|
* sender). Plain `-x` keeps the empty directory and prints nothing, matching
|
||||||
@@ -439,17 +516,16 @@ void scanner_record_size_skipped(DirectoryScanner* scanner, const char* fs_path)
|
|||||||
scanner_record_protected(scanner, fs_path, scanner->options.size_skipped_paths);
|
scanner_record_protected(scanner, fs_path, scanner->options.size_skipped_paths);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Record a directory the scan synchronized. `fs_path` is its absolute path and
|
/* The destination-relative coordinate the receiver's delete walkers match
|
||||||
`rel` its path relative to the transfer root ("" for the root); the stored
|
against for an entry at `fs_path` (with `rel` its path relative to the
|
||||||
form matches the wire layout (the bare relative path in -R+--files-from, else
|
transfer root, "" for the root): `relative_prefix + rel` under -R+--relative,
|
||||||
the source path with a leading '/' removed, with "." for the receive root).
|
the bare relative path under -R+--files-from, else the source path with a
|
||||||
Returns false on allocation failure. */
|
leading '/' removed, with "." for the receive root. Shared by the
|
||||||
bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_path, const char* rel,
|
synchronized-directory sink and the mirrored per-directory rule owners so
|
||||||
|
both live in the same coordinate system. Returns an owned string, or NULL on
|
||||||
|
allocation failure. */
|
||||||
|
char* scanner_dest_rel_path(const ScannerOptions* options, const char* fs_path, const char* rel,
|
||||||
bool relative_mode) {
|
bool relative_mode) {
|
||||||
if (!options->synced_dirs && !options->plan_dirs)
|
|
||||||
return true;
|
|
||||||
if (!file_list_dir_in_scope(options->file_list, rel))
|
|
||||||
return true;
|
|
||||||
char* prefixed = NULL;
|
char* prefixed = NULL;
|
||||||
const char* dest;
|
const char* dest;
|
||||||
if (relative_mode) {
|
if (relative_mode) {
|
||||||
@@ -457,7 +533,7 @@ bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_pat
|
|||||||
} else if (options->relative_prefix) {
|
} else if (options->relative_prefix) {
|
||||||
prefixed = scanner_prefix_send_path(options->relative_prefix, rel);
|
prefixed = scanner_prefix_send_path(options->relative_prefix, rel);
|
||||||
if (!prefixed)
|
if (!prefixed)
|
||||||
return false;
|
return NULL;
|
||||||
dest = prefixed;
|
dest = prefixed;
|
||||||
} else {
|
} else {
|
||||||
dest = fs_path;
|
dest = fs_path;
|
||||||
@@ -466,6 +542,24 @@ bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_pat
|
|||||||
dest++;
|
dest++;
|
||||||
if (dest[0] == '\0')
|
if (dest[0] == '\0')
|
||||||
dest = ".";
|
dest = ".";
|
||||||
|
char* out = str_dup(dest);
|
||||||
|
free(prefixed);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Record a directory the scan synchronized. `fs_path` is its absolute path and
|
||||||
|
`rel` its path relative to the transfer root ("" for the root); the stored
|
||||||
|
form matches the wire layout (see scanner_dest_rel_path). Returns false on
|
||||||
|
allocation failure. */
|
||||||
|
bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_path, const char* rel,
|
||||||
|
bool relative_mode) {
|
||||||
|
if (!options->synced_dirs && !options->plan_dirs)
|
||||||
|
return true;
|
||||||
|
if (!file_list_dir_in_scope(options->file_list, rel))
|
||||||
|
return true;
|
||||||
|
char* dest = scanner_dest_rel_path(options, fs_path, rel, relative_mode);
|
||||||
|
if (!dest)
|
||||||
|
return false;
|
||||||
bool ok = true;
|
bool ok = true;
|
||||||
if (options->synced_dirs)
|
if (options->synced_dirs)
|
||||||
ok = excluded_sink_append(options->synced_dirs, options->excluded_mutex, dest);
|
ok = excluded_sink_append(options->synced_dirs, options->excluded_mutex, dest);
|
||||||
@@ -474,7 +568,7 @@ bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_pat
|
|||||||
than deleted as an extra; the receive root (".") is implicit. */
|
than deleted as an extra; the receive root (".") is implicit. */
|
||||||
if (ok && options->plan_dirs && strcmp(dest, ".") != 0)
|
if (ok && options->plan_dirs && strcmp(dest, ".") != 0)
|
||||||
ok = excluded_sink_append(options->plan_dirs, options->excluded_mutex, dest);
|
ok = excluded_sink_append(options->plan_dirs, options->excluded_mutex, dest);
|
||||||
free(prefixed);
|
free(dest);
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -483,7 +577,8 @@ bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_pat
|
|||||||
* fresh list. Returns NULL on allocation/parse failure (message in `err`);
|
* fresh list. Returns NULL on allocation/parse failure (message in `err`);
|
||||||
* returns an empty list (and *any_exists=false) when no file exists. */
|
* returns an empty list (and *any_exists=false) when no file exists. */
|
||||||
FilterRuleList* read_dir_filters(const ScannerOptions* options, const char* dir_path,
|
FilterRuleList* read_dir_filters(const ScannerOptions* options, const char* dir_path,
|
||||||
const char* rel, bool* any_exists, char* err, size_t err_size) {
|
const char* rel, bool relative_mode, bool* any_exists, char* err,
|
||||||
|
size_t err_size) {
|
||||||
if (err && err_size > 0)
|
if (err && err_size > 0)
|
||||||
err[0] = '\0';
|
err[0] = '\0';
|
||||||
const FilterRuleList* base = options->base_filters;
|
const FilterRuleList* base = options->base_filters;
|
||||||
@@ -507,13 +602,40 @@ FilterRuleList* read_dir_filters(const ScannerOptions* options, const char* dir_
|
|||||||
}
|
}
|
||||||
if (base) {
|
if (base) {
|
||||||
for (int i = 0; i < base->dir_merge_count; i++) {
|
for (int i = 0; i < base->dir_merge_count; i++) {
|
||||||
if (!filter_file_append(own, dir_path, base->dir_merge_names[i], rel, &opts, &exists, err,
|
if (!filter_dir_merge_append(own, dir_path, &base->dir_merges[i], rel, &opts, &exists, err,
|
||||||
err_size))
|
err_size))
|
||||||
goto fail;
|
goto fail;
|
||||||
if (exists && any_exists)
|
if (exists && any_exists)
|
||||||
*any_exists = true;
|
*any_exists = true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/* Mirror the directory's rules into the delete-carrier sink so the receiver
|
||||||
|
* can reconstruct its per-directory protect/risk set. The mirrored rules
|
||||||
|
* carry the destination-relative owner coordinate (not the transfer-root-
|
||||||
|
* relative one the sender's own evaluation uses) so the receiver's delete
|
||||||
|
* walkers, which match against receive-root-relative paths, find them. */
|
||||||
|
if (options->per_dir_rules && own->count > 0) {
|
||||||
|
char* owner = scanner_dest_rel_path(options, dir_path, rel, relative_mode);
|
||||||
|
if (!owner)
|
||||||
|
goto fail;
|
||||||
|
mtx_t* mtx = options->excluded_mutex;
|
||||||
|
if (mtx)
|
||||||
|
mtx_lock(mtx);
|
||||||
|
for (int i = 0; i < own->count; i++) {
|
||||||
|
FilterRule* copy = filter_rule_clone(own->items[i]);
|
||||||
|
if (!copy || !filter_rule_set_owner(copy, owner) ||
|
||||||
|
!filter_rule_list_add(options->per_dir_rules, copy)) {
|
||||||
|
filter_rule_free(copy);
|
||||||
|
if (mtx)
|
||||||
|
mtx_unlock(mtx);
|
||||||
|
free(owner);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (mtx)
|
||||||
|
mtx_unlock(mtx);
|
||||||
|
free(owner);
|
||||||
|
}
|
||||||
return own;
|
return own;
|
||||||
fail:
|
fail:
|
||||||
filter_rule_list_free(own);
|
filter_rule_list_free(own);
|
||||||
@@ -530,7 +652,7 @@ int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* i
|
|||||||
bool any_exists = false;
|
bool any_exists = false;
|
||||||
FilterRuleList* own = read_dir_filters(&scanner->options, scanner->current_path,
|
FilterRuleList* own = read_dir_filters(&scanner->options, scanner->current_path,
|
||||||
scanner->current_rel ? scanner->current_rel : "",
|
scanner->current_rel ? scanner->current_rel : "",
|
||||||
&any_exists, err, sizeof(err));
|
scanner->relative_mode, &any_exists, err, sizeof(err));
|
||||||
if (!own) {
|
if (!own) {
|
||||||
/* read_dir_filters() leaves `err` set on a parse/allocation failure even
|
/* read_dir_filters() leaves `err` set on a parse/allocation failure even
|
||||||
when an earlier merge file in the same directory existed (any_exists true);
|
when an earlier merge file in the same directory existed (any_exists true);
|
||||||
|
|||||||
@@ -62,6 +62,17 @@ typedef enum {
|
|||||||
SCANNER_SPECIAL_SKIP, /* non-regular entry not requested: skip */
|
SCANNER_SPECIAL_SKIP, /* non-regular entry not requested: skip */
|
||||||
} ScannerSpecial;
|
} ScannerSpecial;
|
||||||
|
|
||||||
|
/* Result of scanner_build_file_entry(). The two failure variants preserve the
|
||||||
|
* sequential scanner's historical distinction between a failure before the
|
||||||
|
* File existed (which kept walking the directory) and one afterwards (which cut
|
||||||
|
* the chunk short); both mark the scan failed. */
|
||||||
|
typedef enum {
|
||||||
|
SCANNER_BUILD_OK, /* File built; caller owns it */
|
||||||
|
SCANNER_BUILD_SKIP, /* non-regular entry not preserved; no File */
|
||||||
|
SCANNER_BUILD_FAIL_CONTINUE, /* failed before the File existed */
|
||||||
|
SCANNER_BUILD_FAIL_BREAK, /* failed after the File existed */
|
||||||
|
} ScannerBuildStatus;
|
||||||
|
|
||||||
/* scanner_filter.c */
|
/* scanner_filter.c */
|
||||||
void filter_node_destroy(void* item);
|
void filter_node_destroy(void* item);
|
||||||
FilterNode* filter_node_alloc(FilterNode* parent, FilterRuleList* own);
|
FilterNode* filter_node_alloc(FilterNode* parent, FilterRuleList* own);
|
||||||
@@ -73,14 +84,27 @@ File* scanner_build_dir_file(const char* path, const struct stat* stats,
|
|||||||
const ScannerOptions* options);
|
const ScannerOptions* options);
|
||||||
char* child_rel_path(const char* parent_rel, const char* name);
|
char* child_rel_path(const char* parent_rel, const char* name);
|
||||||
char* scanner_prefix_send_path(const char* prefix, const char* rel);
|
char* scanner_prefix_send_path(const char* prefix, const char* rel);
|
||||||
|
char* scanner_dest_rel_path(const ScannerOptions* options, const char* fs_path, const char* rel,
|
||||||
|
bool relative_mode);
|
||||||
bool entry_passes_selection(const FileListSet* file_list, const FilterRuleList* base,
|
bool entry_passes_selection(const FileListSet* file_list, const FilterRuleList* base,
|
||||||
const FilterNode* node, const char* rel, const char* leaf, bool is_dir,
|
const FilterNode* node, const char* rel, const char* leaf, bool is_dir,
|
||||||
bool per_dir_filters, bool exclude_filter_files, bool* protect_out);
|
bool per_dir_filters, bool exclude_filter_files, bool* protect_out);
|
||||||
void scanner_capture_xattrs(const DirectoryScanner* scanner, File* file);
|
void scanner_capture_xattrs(const DirectoryScanner* scanner, File* file);
|
||||||
void scanner_assign_hardlink(DirectoryScanner* scanner, HardLinkTable* table, File* file,
|
void scanner_capture_xattrs_opts(const ScannerOptions* options, File* file);
|
||||||
const struct stat* stats);
|
bool scanner_assign_hardlink(HardLinkTable* table, File* file, const struct stat* stats);
|
||||||
ScannerSpecial scanner_prepare_special(bool preserve_devices, bool preserve_specials,
|
ScannerSpecial scanner_prepare_special(bool preserve_devices, bool preserve_specials,
|
||||||
bool copy_devices, File* file, const struct stat* stats);
|
bool copy_devices, File* file, const struct stat* stats);
|
||||||
|
/* Build one non-directory transfer File from an inspected entry. `rel` is the
|
||||||
|
* entry's transfer-root-relative path (used for the -R wire path); `inspected`
|
||||||
|
* supplies the on-disk path, stats and (for a carried symlink) the target whose
|
||||||
|
* ownership transfers to the File. Populates data size, send_path, special-node
|
||||||
|
* state, hardlink group, metadata and xattrs. On SCANNER_BUILD_OK the caller
|
||||||
|
* owns *out_file; on SCANNER_BUILD_SKIP it is NULL and the entry is dropped; on
|
||||||
|
* either failure it is NULL and the caller must mark the scan failed. `*failed`
|
||||||
|
* additionally reports a non-fatal hardlink-table allocation failure, in which
|
||||||
|
* case a usable File is still returned. */
|
||||||
|
ScannerBuildStatus scanner_build_file_entry(const ScannerOptions* options, ScannerEntry* inspected,
|
||||||
|
const char* rel, File** out_file, bool* failed);
|
||||||
bool excluded_sink_append(ArrayList* list, mtx_t* mtx, const char* rel);
|
bool excluded_sink_append(ArrayList* list, mtx_t* mtx, const char* rel);
|
||||||
void scanner_note_nonreg(const ScannerOptions* options, const char* fs_path);
|
void scanner_note_nonreg(const ScannerOptions* options, const char* fs_path);
|
||||||
void scanner_note_mount(const ScannerOptions* options, const char* fs_path);
|
void scanner_note_mount(const ScannerOptions* options, const char* fs_path);
|
||||||
@@ -92,7 +116,8 @@ void scanner_record_size_skipped(DirectoryScanner* scanner, const char* fs_path)
|
|||||||
bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_path, const char* rel,
|
bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_path, const char* rel,
|
||||||
bool relative_mode);
|
bool relative_mode);
|
||||||
FilterRuleList* read_dir_filters(const ScannerOptions* options, const char* dir_path,
|
FilterRuleList* read_dir_filters(const ScannerOptions* options, const char* dir_path,
|
||||||
const char* rel, bool* any_exists, char* err, size_t err_size);
|
const char* rel, bool relative_mode, bool* any_exists, char* err,
|
||||||
|
size_t err_size);
|
||||||
int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* inherited);
|
int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* inherited);
|
||||||
int scanner_inspect_entry(const ScannerOptions* options, const char* containing_dir,
|
int scanner_inspect_entry(const ScannerOptions* options, const char* containing_dir,
|
||||||
const char* link_rel, const char* name, ScannerEntry* entry);
|
const char* link_rel, const char* name, ScannerEntry* entry);
|
||||||
|
|||||||
+170
-192
@@ -109,7 +109,7 @@ static void parallel_scanner_creation_failed(ParallelScanner* ps) {
|
|||||||
|
|
||||||
/* Initialize result queue and synchronization primitives. Returns true on success. */
|
/* Initialize result queue and synchronization primitives. Returns true on success. */
|
||||||
static bool parallel_scanner_init(ParallelScanner* ps) {
|
static bool parallel_scanner_init(ParallelScanner* ps) {
|
||||||
ps->result_queue = queue_create(100, chunk_destroy);
|
ps->result_queue = queue_create(SCANNER_RESULT_QUEUE_CAP, chunk_destroy);
|
||||||
if (!ps->result_queue)
|
if (!ps->result_queue)
|
||||||
return false;
|
return false;
|
||||||
atomic_init(&ps->cancelled, false);
|
atomic_init(&ps->cancelled, false);
|
||||||
@@ -210,6 +210,157 @@ static Chunk* batch_files(ArrayList* files, unsigned long long chunk_size, Queue
|
|||||||
return first;
|
return first;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Record the delete-protection mirror of a root entry that
|
||||||
|
* scanner_inspect_entry() skipped (inspection == 0): a dereferenced symlink
|
||||||
|
* with no referent is a partial-transfer I/O error and a user-selection or size
|
||||||
|
* prune protects the entry's destination mirror. */
|
||||||
|
static void scan_root_record_skipped(const ScannerOptions* options, const char* root_directory,
|
||||||
|
const char* name, const ScannerEntry* inspected,
|
||||||
|
ParallelScanner* ps) {
|
||||||
|
if (inspected->referent_error)
|
||||||
|
ps->io_error = true;
|
||||||
|
ArrayList* sink = NULL;
|
||||||
|
if (inspected->excluded)
|
||||||
|
sink = inspected->size_excluded ? options->size_skipped_paths : options->excluded_paths;
|
||||||
|
if (!sink)
|
||||||
|
return;
|
||||||
|
/* A root-level prune protects the destination mirror of the entry's wire
|
||||||
|
path: under -R + --files-from that is the bare relative name, otherwise it
|
||||||
|
is the full source path with a leading '/' removed (matching the
|
||||||
|
send_path/file_wire_path the scanner hands the sender). */
|
||||||
|
if (options->relative && options->file_list != NULL) {
|
||||||
|
if (!excluded_sink_append(sink, options->excluded_mutex, name))
|
||||||
|
ps->failed = true;
|
||||||
|
} else if (options->relative_prefix) {
|
||||||
|
char* wrel = scanner_prefix_send_path(options->relative_prefix, name);
|
||||||
|
if (!wrel) {
|
||||||
|
ps->failed = true;
|
||||||
|
} else {
|
||||||
|
if (!excluded_sink_append(sink, options->excluded_mutex, wrel))
|
||||||
|
ps->failed = true;
|
||||||
|
free(wrel);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
char* abs_path = path_cat(root_directory, name);
|
||||||
|
if (!abs_path) {
|
||||||
|
ps->failed = true;
|
||||||
|
} else {
|
||||||
|
const char* rel = *abs_path == '/' ? abs_path + 1 : abs_path;
|
||||||
|
if (!excluded_sink_append(sink, options->excluded_mutex, rel))
|
||||||
|
ps->failed = true;
|
||||||
|
free(abs_path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Record the delete-protection mirror of a root entry dropped by the
|
||||||
|
* --files-from allow-set or a filter rule. Returns false only when the -R
|
||||||
|
* prefix could not be built (the caller must abandon the entry immediately);
|
||||||
|
* other allocation failures mark the scan failed but let the caller continue to
|
||||||
|
* the filter-notice step, matching the historical inlined flow. */
|
||||||
|
static bool scan_root_record_protection(const ScannerOptions* options, const char* rel,
|
||||||
|
const char* name, const char* cur_path, bool protect,
|
||||||
|
bool passes, bool use_rel, ParallelScanner* ps) {
|
||||||
|
if (passes && !protect)
|
||||||
|
return true;
|
||||||
|
/* --files-from subset pruning is not a filter exclusion; -R bare-wire-path
|
||||||
|
exclusions are never recorded (see ScannerOptions.excluded_paths). */
|
||||||
|
bool files_from_prune = options->file_list && !file_list_affects(options->file_list, rel);
|
||||||
|
if ((!files_from_prune && !use_rel) || protect) {
|
||||||
|
const char* rel_path;
|
||||||
|
char* prefixed = NULL;
|
||||||
|
if (use_rel) {
|
||||||
|
/* -R + --files-from: the destination/wire path is the bare relative
|
||||||
|
name, not the source path. */
|
||||||
|
rel_path = rel;
|
||||||
|
} else if (options->relative_prefix) {
|
||||||
|
prefixed = scanner_prefix_send_path(options->relative_prefix, name);
|
||||||
|
if (!prefixed)
|
||||||
|
return false;
|
||||||
|
rel_path = prefixed;
|
||||||
|
} else {
|
||||||
|
rel_path = *cur_path == '/' ? cur_path + 1 : cur_path;
|
||||||
|
}
|
||||||
|
if (options->excluded_paths &&
|
||||||
|
!excluded_sink_append(options->excluded_paths, options->excluded_mutex, rel_path))
|
||||||
|
ps->failed = true;
|
||||||
|
free(prefixed);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Root-level directory node: apply -x/--one-file-system and either emit the
|
||||||
|
* mount-point directory (plain -x) or queue the directory for a worker. */
|
||||||
|
static void scan_root_dir(const ScannerOptions* options, const char* cur_path, const char* rel,
|
||||||
|
const struct stat* st, ArrayList* root_files, ArrayList* subdirs,
|
||||||
|
dev_t root_dev, ParallelScanner* ps) {
|
||||||
|
if (!scanner_same_filesystem(options->one_file_system, root_dev, st->st_dev)) {
|
||||||
|
if (options->one_file_system > 1) {
|
||||||
|
/* -xx: drop the mount-point directory entirely (rsync) and print the
|
||||||
|
--info=mount line when enabled. */
|
||||||
|
scanner_note_mount(options, cur_path);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
/* -x/--one-file-system: emit the mount-point directory entry (empty) but do
|
||||||
|
not descend into it (see the sequential scanner for the same rule). */
|
||||||
|
File* mount = scanner_build_dir_file(cur_path, st, options);
|
||||||
|
if (!mount) {
|
||||||
|
ps->failed = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (options->relative_prefix) {
|
||||||
|
mount->send_path = scanner_prefix_send_path(options->relative_prefix, rel);
|
||||||
|
if (!mount->send_path) {
|
||||||
|
file_destroy(mount);
|
||||||
|
ps->failed = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!array_list_add(root_files, mount)) {
|
||||||
|
file_destroy(mount);
|
||||||
|
ps->failed = true;
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
char* dir = str_dup(cur_path);
|
||||||
|
if (!dir || !array_list_add(subdirs, dir)) {
|
||||||
|
free(dir);
|
||||||
|
ps->failed = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Build a non-directory root entry through the shared construction path and add
|
||||||
|
* it to `root_files`. A non-regular entry the options do not preserve is
|
||||||
|
* dropped by the builder (which prints rsync's nonreg line); an allocation
|
||||||
|
* failure marks the scan failed. */
|
||||||
|
static void scan_root_add_non_dir(const ScannerOptions* options, ScannerEntry* inspected,
|
||||||
|
const char* rel, ArrayList* root_files, ParallelScanner* ps) {
|
||||||
|
File* file = NULL;
|
||||||
|
bool failed = false;
|
||||||
|
ScannerBuildStatus status = scanner_build_file_entry(options, inspected, rel, &file, &failed);
|
||||||
|
if (failed || status == SCANNER_BUILD_FAIL_CONTINUE || status == SCANNER_BUILD_FAIL_BREAK)
|
||||||
|
ps->failed = true;
|
||||||
|
if (status != SCANNER_BUILD_OK)
|
||||||
|
return;
|
||||||
|
if (!array_list_add(root_files, file)) {
|
||||||
|
file_destroy(file);
|
||||||
|
ps->failed = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Regular file or carried symlink at the transfer root. */
|
||||||
|
static void scan_root_file(const ScannerOptions* options, ScannerEntry* inspected, const char* rel,
|
||||||
|
ArrayList* root_files, ParallelScanner* ps) {
|
||||||
|
scan_root_add_non_dir(options, inspected, rel, root_files, ps);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Device/FIFO/socket at the transfer root: recreated under --devices/--specials,
|
||||||
|
* otherwise dropped by the shared builder. */
|
||||||
|
static void scan_root_special(const ScannerOptions* options, ScannerEntry* inspected,
|
||||||
|
const char* rel, ArrayList* root_files, ParallelScanner* ps) {
|
||||||
|
scan_root_add_non_dir(options, inspected, rel, root_files, ps);
|
||||||
|
}
|
||||||
|
|
||||||
/* Scan one root-directory entry into either the subdirs or files list. */
|
/* Scan one root-directory entry into either the subdirs or files list. */
|
||||||
static void scan_root_entry(const ScannerOptions* options, const FilterNode* root_node,
|
static void scan_root_entry(const ScannerOptions* options, const FilterNode* root_node,
|
||||||
const char* root_directory, const struct dirent* entry,
|
const char* root_directory, const struct dirent* entry,
|
||||||
@@ -223,51 +374,16 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (inspection == 0) {
|
if (inspection == 0) {
|
||||||
if (inspected.referent_error)
|
scan_root_record_skipped(options, root_directory, entry->d_name, &inspected, ps);
|
||||||
ps->io_error = true;
|
|
||||||
ArrayList* sink = NULL;
|
|
||||||
if (inspected.excluded)
|
|
||||||
sink = inspected.size_excluded ? options->size_skipped_paths : options->excluded_paths;
|
|
||||||
if (sink) {
|
|
||||||
/* A root-level prune protects the destination mirror of the entry's wire
|
|
||||||
path: under -R + --files-from that is the bare relative name, otherwise
|
|
||||||
it is the full source path with a leading '/' removed (matching the
|
|
||||||
send_path/file_wire_path the scanner hands the sender). */
|
|
||||||
if (options->relative && options->file_list != NULL) {
|
|
||||||
if (!excluded_sink_append(sink, options->excluded_mutex, entry->d_name))
|
|
||||||
ps->failed = true;
|
|
||||||
} else if (options->relative_prefix) {
|
|
||||||
char* wrel = scanner_prefix_send_path(options->relative_prefix, entry->d_name);
|
|
||||||
if (!wrel) {
|
|
||||||
ps->failed = true;
|
|
||||||
} else {
|
|
||||||
if (!excluded_sink_append(sink, options->excluded_mutex, wrel))
|
|
||||||
ps->failed = true;
|
|
||||||
free(wrel);
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
char* abs_path = path_cat(root_directory, entry->d_name);
|
|
||||||
if (!abs_path) {
|
|
||||||
ps->failed = true;
|
|
||||||
} else {
|
|
||||||
const char* rel = *abs_path == '/' ? abs_path + 1 : abs_path;
|
|
||||||
if (!excluded_sink_append(sink, options->excluded_mutex, rel))
|
|
||||||
ps->failed = true;
|
|
||||||
free(abs_path);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
char* cur_path = inspected.path;
|
char* cur_path = inspected.path;
|
||||||
struct stat st = inspected.stats;
|
|
||||||
bool is_dir = inspected.is_directory;
|
|
||||||
char* rel = str_dup(entry->d_name);
|
char* rel = str_dup(entry->d_name);
|
||||||
if (!rel) {
|
if (!rel) {
|
||||||
free(cur_path);
|
|
||||||
ps->failed = true;
|
ps->failed = true;
|
||||||
return;
|
goto done;
|
||||||
}
|
}
|
||||||
|
bool is_dir = inspected.is_directory;
|
||||||
bool protect = false;
|
bool protect = false;
|
||||||
bool passes = entry_passes_selection(options->file_list, options->base_filters, root_node, rel,
|
bool passes = entry_passes_selection(options->file_list, options->base_filters, root_node, rel,
|
||||||
entry->d_name, is_dir, options->per_dir_filters,
|
entry->d_name, is_dir, options->per_dir_filters,
|
||||||
@@ -275,167 +391,28 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
|
|||||||
/* -R + --files-from: root-level files keep their bare relative send path. */
|
/* -R + --files-from: root-level files keep their bare relative send path. */
|
||||||
bool use_rel = options->relative && options->file_list != NULL;
|
bool use_rel = options->relative && options->file_list != NULL;
|
||||||
if (!passes || protect) {
|
if (!passes || protect) {
|
||||||
/* --files-from subset pruning is not a filter exclusion; -R bare-wire-path
|
if (!scan_root_record_protection(options, rel, entry->d_name, cur_path, protect, passes,
|
||||||
exclusions are never recorded (see ScannerOptions.excluded_paths). */
|
use_rel, ps)) {
|
||||||
bool files_from_prune = options->file_list && !file_list_affects(options->file_list, rel);
|
|
||||||
if ((!files_from_prune && !use_rel) || protect) {
|
|
||||||
const char* rel_path;
|
|
||||||
char* prefixed = NULL;
|
|
||||||
if (use_rel) {
|
|
||||||
/* -R + --files-from: the destination/wire path is the bare relative
|
|
||||||
name, not the source path. */
|
|
||||||
rel_path = rel;
|
|
||||||
} else if (options->relative_prefix) {
|
|
||||||
prefixed = scanner_prefix_send_path(options->relative_prefix, entry->d_name);
|
|
||||||
if (!prefixed) {
|
|
||||||
free(rel);
|
|
||||||
free(cur_path);
|
|
||||||
ps->failed = true;
|
ps->failed = true;
|
||||||
return;
|
goto done;
|
||||||
}
|
|
||||||
rel_path = prefixed;
|
|
||||||
} else {
|
|
||||||
rel_path = *cur_path == '/' ? cur_path + 1 : cur_path;
|
|
||||||
}
|
|
||||||
if (options->excluded_paths &&
|
|
||||||
!excluded_sink_append(options->excluded_paths, options->excluded_mutex, rel_path))
|
|
||||||
ps->failed = true;
|
|
||||||
free(prefixed);
|
|
||||||
}
|
}
|
||||||
if (!passes) {
|
if (!passes) {
|
||||||
scanner_note_filter(options, entry->d_name);
|
scanner_note_filter(options, entry->d_name);
|
||||||
free(rel);
|
goto done;
|
||||||
free(cur_path);
|
|
||||||
return;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (is_dir) {
|
if (is_dir) {
|
||||||
if (!scanner_same_filesystem(options->one_file_system, root_dev, st.st_dev)) {
|
scan_root_dir(options, cur_path, rel, &inspected.stats, root_files, subdirs, root_dev, ps);
|
||||||
if (options->one_file_system > 1) {
|
} else if (S_ISCHR(inspected.stats.st_mode) || S_ISBLK(inspected.stats.st_mode) ||
|
||||||
/* -xx: drop the mount-point directory entirely (rsync) and print the
|
S_ISFIFO(inspected.stats.st_mode) || S_ISSOCK(inspected.stats.st_mode)) {
|
||||||
--info=mount line when enabled. */
|
scan_root_special(options, &inspected, rel, root_files, ps);
|
||||||
scanner_note_mount(options, cur_path);
|
} else {
|
||||||
|
scan_root_file(options, &inspected, rel, root_files, ps);
|
||||||
|
}
|
||||||
|
done:
|
||||||
free(rel);
|
free(rel);
|
||||||
free(cur_path);
|
free(cur_path);
|
||||||
return;
|
|
||||||
}
|
|
||||||
/* -x/--one-file-system: emit the mount-point directory entry (empty) but
|
|
||||||
do not descend into it (see the sequential scanner for the same rule). */
|
|
||||||
File* mount = file_create(cur_path);
|
|
||||||
free(cur_path);
|
|
||||||
if (mount == NULL) {
|
|
||||||
free(rel);
|
|
||||||
ps->failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
mount->is_dir = true;
|
|
||||||
if (options->use_metadata) {
|
|
||||||
mount->metadata = file_metadata_create(mount->path, &st, options->preserve_atimes,
|
|
||||||
options->preserve_crtimes);
|
|
||||||
if (!mount->metadata) {
|
|
||||||
free(rel);
|
|
||||||
file_destroy(mount);
|
|
||||||
ps->failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (options->relative_prefix) {
|
|
||||||
mount->send_path = scanner_prefix_send_path(options->relative_prefix, rel);
|
|
||||||
if (!mount->send_path) {
|
|
||||||
free(rel);
|
|
||||||
file_destroy(mount);
|
|
||||||
ps->failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
free(rel);
|
|
||||||
if (!array_list_add(root_files, mount)) {
|
|
||||||
file_destroy(mount);
|
|
||||||
ps->failed = true;
|
|
||||||
}
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
free(rel);
|
|
||||||
if (!array_list_add(subdirs, cur_path)) {
|
|
||||||
free(cur_path);
|
|
||||||
ps->failed = true;
|
|
||||||
}
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
File* file = file_create(cur_path);
|
|
||||||
free(cur_path);
|
|
||||||
if (!file) {
|
|
||||||
free(rel);
|
|
||||||
free(inspected.link_target);
|
free(inspected.link_target);
|
||||||
inspected.link_target = NULL;
|
|
||||||
ps->failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (inspected.is_symlink) {
|
|
||||||
file->is_symlink = true;
|
|
||||||
file->symlink_target = inspected.link_target;
|
|
||||||
inspected.link_target = NULL;
|
|
||||||
} else {
|
|
||||||
file->data->size = st.st_size;
|
|
||||||
}
|
|
||||||
if (use_rel) {
|
|
||||||
file->send_path = rel;
|
|
||||||
rel = NULL;
|
|
||||||
} else if (options->relative_prefix) {
|
|
||||||
file->send_path = scanner_prefix_send_path(options->relative_prefix, rel);
|
|
||||||
free(rel);
|
|
||||||
rel = NULL;
|
|
||||||
if (!file->send_path) {
|
|
||||||
file_destroy(file);
|
|
||||||
ps->failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
ScannerSpecial special = scanner_prepare_special(
|
|
||||||
options->preserve_devices, options->preserve_specials, options->copy_devices, file, &st);
|
|
||||||
if (special == SCANNER_SPECIAL_SKIP) {
|
|
||||||
scanner_note_nonreg(ps->options, file->path);
|
|
||||||
free(rel);
|
|
||||||
file_destroy(file);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (options->hardlinks && S_ISREG(st.st_mode)) {
|
|
||||||
int gid;
|
|
||||||
bool is_first;
|
|
||||||
char* first_path = NULL;
|
|
||||||
if (!hardlink_table_assign((HardLinkTable*)options->hardlinks, file_wire_path(file), st.st_dev,
|
|
||||||
st.st_ino, &gid, &is_first, &first_path)) {
|
|
||||||
ps->failed = true;
|
|
||||||
} else {
|
|
||||||
file->link_group = gid;
|
|
||||||
file->link_first = is_first;
|
|
||||||
if (!is_first) {
|
|
||||||
file->hardlink_target = first_path;
|
|
||||||
file->data->size = 0;
|
|
||||||
} else {
|
|
||||||
free(first_path);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (options->use_metadata)
|
|
||||||
file->metadata =
|
|
||||||
file_metadata_create(file->path, &st, options->preserve_atimes, options->preserve_crtimes);
|
|
||||||
if (options->use_metadata && !file->metadata) {
|
|
||||||
free(rel);
|
|
||||||
file_destroy(file);
|
|
||||||
ps->failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if ((options->preserve_xattrs || options->preserve_acls) &&
|
|
||||||
!(file->link_group != 0 && !file->link_first))
|
|
||||||
file->xattrs = xattr_capture_path(file->path, options->preserve_acls);
|
|
||||||
if (!array_list_add(root_files, file)) {
|
|
||||||
free(rel);
|
|
||||||
file_destroy(file);
|
|
||||||
ps->failed = true;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
free(rel);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Scan the root directory itself, collecting root files and subdirectories.
|
/* Scan the root directory itself, collecting root files and subdirectories.
|
||||||
@@ -587,8 +564,9 @@ ParallelScanner* parallel_scanner_create_with_options(const char* root_directory
|
|||||||
{
|
{
|
||||||
char err[256];
|
char err[256];
|
||||||
bool any_exists = false;
|
bool any_exists = false;
|
||||||
FilterRuleList* own =
|
FilterRuleList* own = read_dir_filters(options, root_directory, "",
|
||||||
read_dir_filters(options, root_directory, "", &any_exists, err, sizeof(err));
|
options->relative && options->file_list != NULL,
|
||||||
|
&any_exists, err, sizeof(err));
|
||||||
if (!own) {
|
if (!own) {
|
||||||
/* A parse/allocation failure must fail the scan even when an earlier
|
/* A parse/allocation failure must fail the scan even when an earlier
|
||||||
merge file in the same directory existed (see the sequential scanner). */
|
merge file in the same directory existed (see the sequential scanner). */
|
||||||
|
|||||||
+13
-7
@@ -26,6 +26,9 @@ void print_usage(void) {
|
|||||||
printf(" owner, group, devices and specials; not\n");
|
printf(" owner, group, devices and specials; not\n");
|
||||||
printf(" compression/multithreading\n");
|
printf(" compression/multithreading\n");
|
||||||
printf(" -r, --recursive Recurse into directories (FastSync is always recursive)\n");
|
printf(" -r, --recursive Recurse into directories (FastSync is always recursive)\n");
|
||||||
|
printf(" --inc-recursive Accepted for rsync CLI compatibility; no effect (FastSync\n");
|
||||||
|
printf(" always performs a full scan, so the destination is identical)\n");
|
||||||
|
printf(" --no-inc-recursive Accepted for rsync CLI compatibility; no effect\n");
|
||||||
printf(" -n, --dry-run Show what would be transferred\n");
|
printf(" -n, --dry-run Show what would be transferred\n");
|
||||||
printf(" --remove-source-files Remove regular source files after successful transfer\n");
|
printf(" --remove-source-files Remove regular source files after successful transfer\n");
|
||||||
printf(" -p, --perms Preserve permission bits\n");
|
printf(" -p, --perms Preserve permission bits\n");
|
||||||
@@ -205,10 +208,11 @@ void print_usage(void) {
|
|||||||
printf(" -A, --acls Preserve POSIX ACLs (the system.posix_acl_* xattrs;\n");
|
printf(" -A, --acls Preserve POSIX ACLs (the system.posix_acl_* xattrs;\n");
|
||||||
printf(" setting an ACL the receiver is not permitted to\n");
|
printf(" setting an ACL the receiver is not permitted to\n");
|
||||||
printf(" set is warned and skipped, never fatal)\n");
|
printf(" set is warned and skipped, never fatal)\n");
|
||||||
printf(" --fake-super Store the source uid/gid/mode/mtime in a reserved\n");
|
printf(" --fake-super Store the source mode/rdev/uid/gid in rsync's\n");
|
||||||
printf(" user.fastsync.stat xattr on each written file and\n");
|
printf(" reserved user.rsync.%%stat xattr on each written\n");
|
||||||
printf(" re-apply it (fd-relative) on a privileged run; the\n");
|
printf(" file (interoperable with rsync); it never performs a\n");
|
||||||
printf(" recording format diverges from rsync's user.rsync.%%stat%%\n");
|
printf(" real chown, so an unprivileged receiver records the\n");
|
||||||
|
printf(" privileged stat for a later restore\n");
|
||||||
printf(" --super Permit the receiver to attempt super-user activities\n");
|
printf(" --super Permit the receiver to attempt super-user activities\n");
|
||||||
printf(" (char/block device-node creation, --write-devices)\n");
|
printf(" (char/block device-node creation, --write-devices)\n");
|
||||||
printf(" within the confined receive root. Never elevates\n");
|
printf(" within the confined receive root. Never elevates\n");
|
||||||
@@ -293,11 +297,13 @@ void print_usage(void) {
|
|||||||
printf(" --max-depth <n> Maximum directory depth (0=unlimited)\n");
|
printf(" --max-depth <n> Maximum directory depth (0=unlimited)\n");
|
||||||
printf(" -x, --one-file-system Do not cross filesystem boundaries\n");
|
printf(" -x, --one-file-system Do not cross filesystem boundaries\n");
|
||||||
printf(" --log-file <path>, --log-file=<path> Write log messages to file\n");
|
printf(" --log-file <path>, --log-file=<path> Write log messages to file\n");
|
||||||
printf(" --stderr=MODE Route logging to stderr: errors or all\n");
|
printf(" --stderr=MODE Route logging: errors (default), all, or client\n");
|
||||||
|
printf(" (forward the client's diagnostics to the server's\n");
|
||||||
|
printf(" stderr)\n");
|
||||||
printf(" --msgs2stderr Route all messages to stderr (deprecated spelling of\n");
|
printf(" --msgs2stderr Route all messages to stderr (deprecated spelling of\n");
|
||||||
printf(" --stderr=all)\n");
|
printf(" --stderr=all)\n");
|
||||||
printf(" --no-msgs2stderr Select errors-only stderr (deprecated spelling; the\n");
|
printf(" --no-msgs2stderr Forward the client's diagnostics to the server\n");
|
||||||
printf(" default)\n");
|
printf(" (deprecated spelling of --stderr=client)\n");
|
||||||
printf(" --partial Keep partial files on interrupted transfer\n");
|
printf(" --partial Keep partial files on interrupted transfer\n");
|
||||||
printf(" --partial-dir <dir> Directory for partial files (implies --partial)\n");
|
printf(" --partial-dir <dir> Directory for partial files (implies --partial)\n");
|
||||||
printf(" -T, --temp-dir <dir> Scratch dir for temp files before atomic install.\n");
|
printf(" -T, --temp-dir <dir> Scratch dir for temp files before atomic install.\n");
|
||||||
|
|||||||
+256
-51
@@ -11,10 +11,13 @@
|
|||||||
#include "metadata.h"
|
#include "metadata.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
|
#include <fcntl.h>
|
||||||
|
#include <limits.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
#include <time.h>
|
#include <time.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code) {
|
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code) {
|
||||||
if (!outcomes)
|
if (!outcomes)
|
||||||
@@ -53,7 +56,13 @@ bool receiver_send_final_success(int fd, const Config* config, const ReceiverOut
|
|||||||
return send_status(fd, final_status);
|
return send_status(fd, final_status);
|
||||||
size_t count = outcomes ? outcomes->count : 0;
|
size_t count = outcomes ? outcomes->count : 0;
|
||||||
for (size_t i = 0; i < count; i++) {
|
for (size_t i = 0; i < count; i++) {
|
||||||
Status per_file = outcomes->entries[i] == FILE_SAVE_WRITTEN ? STATUS_NEXT : STATUS_OK;
|
Status per_file;
|
||||||
|
if (outcomes->entries[i] == FILE_SAVE_WRITTEN)
|
||||||
|
per_file = STATUS_NEXT;
|
||||||
|
else if (outcomes->entries[i] == FILE_SAVE_FAILED)
|
||||||
|
per_file = STATUS_ERROR;
|
||||||
|
else
|
||||||
|
per_file = STATUS_OK;
|
||||||
if (!send_status(fd, per_file))
|
if (!send_status(fd, per_file))
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -95,14 +104,35 @@ static void receiver_tally_deleted(const ReceiverSink* sink, size_t deleted) {
|
|||||||
sink->stats->deleted_files += deleted;
|
sink->stats->deleted_files += deleted;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Observer for --info=del: record each truly-removed destination-relative path
|
/* Observer for --info=del/--stats: record each truly-removed destination-
|
||||||
in the ArrayList passed as the observer context, so the terminal STATUS_STATS
|
relative path (when the context carries a path list) and tally it by type
|
||||||
frame can list it. A failed append is best-effort (the deletion already
|
(when it carries a stats record), so the terminal STATUS_STATS frame can list
|
||||||
happened; output is cosmetic). Shared by the single-threaded receiver and
|
the paths and render rsync's per-type `Number of deleted files` breakdown. A
|
||||||
the -m pipeline's deferred commit. */
|
failed append is best-effort (the deletion already happened; output is
|
||||||
void receiver_record_deleted_path(void* context, const char* rel_path) {
|
cosmetic). Shared by the single-threaded receiver and the -m pipeline's
|
||||||
ArrayList* paths = context;
|
deferred commit. */
|
||||||
if (!paths || !rel_path)
|
void receiver_record_deleted_path(void* context, const char* rel_path, DeleteEntryType type) {
|
||||||
|
ReceiverDeleteContext* del = context;
|
||||||
|
if (!del || !rel_path)
|
||||||
|
return;
|
||||||
|
if (del->stats) {
|
||||||
|
switch (type) {
|
||||||
|
case DELETE_ENTRY_DIR:
|
||||||
|
del->stats->deleted_dir++;
|
||||||
|
break;
|
||||||
|
case DELETE_ENTRY_LINK:
|
||||||
|
del->stats->deleted_link++;
|
||||||
|
break;
|
||||||
|
case DELETE_ENTRY_SPECIAL:
|
||||||
|
del->stats->deleted_special++;
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
del->stats->deleted_reg++;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ArrayList* paths = del->deleted_paths;
|
||||||
|
if (!paths)
|
||||||
return;
|
return;
|
||||||
/* Bound the retained list like the keep-set manifest: only MAX_MANIFEST_ENTRIES
|
/* Bound the retained list like the keep-set manifest: only MAX_MANIFEST_ENTRIES
|
||||||
paths are ever transmitted in the terminal STATUS_STATS frame, so recording
|
paths are ever transmitted in the terminal STATUS_STATS frame, so recording
|
||||||
@@ -114,6 +144,16 @@ void receiver_record_deleted_path(void* context, const char* rel_path) {
|
|||||||
free(copy);
|
free(copy);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Install the delete observer (and its context) for one commit when the sink
|
||||||
|
carries a stats record or a path list. Returns NULL when neither is needed,
|
||||||
|
so the delete engines skip the observer entirely. */
|
||||||
|
static DeletePathObserver receiver_delete_observer(const ReceiverSink* sink,
|
||||||
|
ReceiverDeleteContext* del) {
|
||||||
|
del->stats = sink ? sink->stats : NULL;
|
||||||
|
del->deleted_paths = sink ? sink->deleted_paths : NULL;
|
||||||
|
return (del->stats || del->deleted_paths) ? receiver_record_deleted_path : NULL;
|
||||||
|
}
|
||||||
|
|
||||||
static bool receiver_process_chunk(Chunk* chunk, const ReceiverSink* sink) {
|
static bool receiver_process_chunk(Chunk* chunk, const ReceiverSink* sink) {
|
||||||
if (!chunk || !sink || !sink->store_file)
|
if (!chunk || !sink || !sink->store_file)
|
||||||
return false;
|
return false;
|
||||||
@@ -271,6 +311,7 @@ static bool status_counts_as_progress(Status status) {
|
|||||||
case STATUS_ABORT:
|
case STATUS_ABORT:
|
||||||
case STATUS_CHECK_BATCH:
|
case STATUS_CHECK_BATCH:
|
||||||
case STATUS_DIR_TIMES:
|
case STATUS_DIR_TIMES:
|
||||||
|
case STATUS_CLIENT_MSG:
|
||||||
return false;
|
return false;
|
||||||
default:
|
default:
|
||||||
return true;
|
return true;
|
||||||
@@ -300,7 +341,13 @@ static bool receiver_note_status(const struct timespec* session_start,
|
|||||||
}
|
}
|
||||||
|
|
||||||
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink) {
|
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink) {
|
||||||
return receiver_process_pending(config, file_descriptor, sink, NULL, NULL);
|
return receiver_process_pending_ctx(config, file_descriptor, sink, NULL, NULL, NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||||
|
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
|
||||||
|
return receiver_process_pending_ctx(config, file_descriptor, sink, pending_manifest,
|
||||||
|
pending_plans, NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Per-connection state threaded through the status handlers below. The parked
|
/* Per-connection state threaded through the status handlers below. The parked
|
||||||
@@ -321,6 +368,11 @@ typedef struct {
|
|||||||
a successful FINISHED it is either committed here or handed to
|
a successful FINISHED it is either committed here or handed to
|
||||||
*pending_plans so the -m caller commits after its disk writer drained. */
|
*pending_plans so the -m caller commits after its disk writer drained. */
|
||||||
DeletePlanSession* plan_session;
|
DeletePlanSession* plan_session;
|
||||||
|
/* Observer context for the per-directory delete session, which outlives the
|
||||||
|
frame handler; must stay alive until the session commits. For a session
|
||||||
|
handed to the caller (pending_plans) this points at a caller-owned
|
||||||
|
long-lived context; otherwise it points at an internal stack context. */
|
||||||
|
ReceiverDeleteContext* delete_ctx;
|
||||||
bool early_delete;
|
bool early_delete;
|
||||||
bool per_dir_delete;
|
bool per_dir_delete;
|
||||||
bool delete_limit_noted;
|
bool delete_limit_noted;
|
||||||
@@ -341,6 +393,23 @@ static ReceiverStep receiver_handle_keepalive(ReceiverPendingState* state) {
|
|||||||
return RECEIVER_STEP_NEXT;
|
return RECEIVER_STEP_NEXT;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* rsync --stderr=client: a client diagnostic forwarded over the wire. Read the
|
||||||
|
* bounded string and log it through the normal destination/level gate. The
|
||||||
|
* body is peer-controlled text: log_client_message() escapes every
|
||||||
|
* non-printable byte (newlines, CR, ANSI ESC, ...) before writing, so a hostile
|
||||||
|
* client cannot forge log lines or inject terminal control sequences. A
|
||||||
|
* malformed string (over-long or embedded NUL) is a framing error and tears the
|
||||||
|
* connection down. */
|
||||||
|
static ReceiverStep receiver_handle_client_msg(ReceiverPendingState* state) {
|
||||||
|
char* message = receive_str(state->fd);
|
||||||
|
if (!message)
|
||||||
|
return RECEIVER_STEP_FAIL;
|
||||||
|
if (message[0] != '\0')
|
||||||
|
log_client_message(message);
|
||||||
|
free(message);
|
||||||
|
return RECEIVER_STEP_NEXT;
|
||||||
|
}
|
||||||
|
|
||||||
static ReceiverStep receiver_handle_abort(ReceiverPendingState* state) {
|
static ReceiverStep receiver_handle_abort(ReceiverPendingState* state) {
|
||||||
(void)state;
|
(void)state;
|
||||||
log_message(LOG_LEVEL_INFO, "Received abort from client, cleaning up");
|
log_message(LOG_LEVEL_INFO, "Received abort from client, cleaning up");
|
||||||
@@ -377,9 +446,95 @@ static ReceiverStep receiver_handle_check_batch(ReceiverPendingState* state) {
|
|||||||
return RECEIVER_STEP_NEXT;
|
return RECEIVER_STEP_NEXT;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Probe a destination entry's pre-transfer state for the output-parity
|
||||||
|
dest-info report (protocol 2.30.0). `wire_path` is the destination-relative
|
||||||
|
path; `incoming_target` is non-NULL only for a symlink probe, in which case
|
||||||
|
the on-disk link target is compared with the target the receiver is about to
|
||||||
|
store (after --munge-links). The final component is never followed and the
|
||||||
|
parent walk is confined below the receive root. Returns false only on an
|
||||||
|
allocation/secure-walk failure; a missing entry is reported as existed=false. */
|
||||||
|
static bool receiver_probe_dest_state(const Config* config, const char* wire_path,
|
||||||
|
const char* incoming_target, OutputDestState* out) {
|
||||||
|
memset(out, 0, sizeof(*out));
|
||||||
|
out->known = true;
|
||||||
|
if (!config || !wire_path || wire_path[0] == '\0')
|
||||||
|
return false;
|
||||||
|
char* full = path_cat(config->receive_root_directory, wire_path);
|
||||||
|
if (!full)
|
||||||
|
return false;
|
||||||
|
char* leaf = NULL;
|
||||||
|
int parent_fd = file_open_secure_parent(full, &leaf, false);
|
||||||
|
free(full);
|
||||||
|
if (parent_fd < 0) {
|
||||||
|
/* A missing/unreachable parent means the entry cannot exist yet. */
|
||||||
|
free(leaf);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
struct stat st;
|
||||||
|
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) == 0) {
|
||||||
|
out->existed = true;
|
||||||
|
out->size = (unsigned long long)st.st_size;
|
||||||
|
out->mtime_sec = (long long)st.st_mtime;
|
||||||
|
#ifdef __linux__
|
||||||
|
out->mtime_nsec = st.st_mtim.tv_nsec;
|
||||||
|
#endif
|
||||||
|
out->mode = (uint32_t)st.st_mode;
|
||||||
|
out->uid = (int32_t)st.st_uid;
|
||||||
|
out->gid = (int32_t)st.st_gid;
|
||||||
|
if (incoming_target && S_ISLNK(st.st_mode)) {
|
||||||
|
char target_buf[PATH_MAX];
|
||||||
|
ssize_t n = readlinkat(parent_fd, leaf, target_buf, sizeof(target_buf) - 1);
|
||||||
|
if (n >= 0) {
|
||||||
|
target_buf[n] = '\0';
|
||||||
|
char* expected =
|
||||||
|
config->munge_links ? file_symlink_munge(incoming_target) : str_dup(incoming_target);
|
||||||
|
if (expected) {
|
||||||
|
out->target_matches = strcmp(target_buf, expected) == 0;
|
||||||
|
free(expected);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
static ReceiverStep receiver_handle_mkdir(ReceiverPendingState* state) {
|
static ReceiverStep receiver_handle_mkdir(ReceiverPendingState* state) {
|
||||||
File* dir = file_receive_directory(state->fd, state->config);
|
const Config* config = state->config;
|
||||||
if (!dir || !state->sink->store_file(dir, state->sink->context))
|
int fd = state->fd;
|
||||||
|
if (config->report_dest_info) {
|
||||||
|
int probe = 0;
|
||||||
|
if (!receive_int(fd, &probe) || (probe != 0 && probe != 1))
|
||||||
|
return RECEIVER_STEP_FAIL;
|
||||||
|
if (probe) {
|
||||||
|
/* Probe-only frame: report the destination state and create nothing. */
|
||||||
|
char* path = receive_wire_str(fd);
|
||||||
|
if (!path || path[0] == '\0' || (!file_get_trust_sender() && has_path_traversal(path))) {
|
||||||
|
free(path);
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
|
return RECEIVER_STEP_FAIL;
|
||||||
|
}
|
||||||
|
OutputDestState info;
|
||||||
|
bool ok = receiver_probe_dest_state(config, path, NULL, &info);
|
||||||
|
free(path);
|
||||||
|
if (!ok || !send_status(fd, STATUS_DEST_INFO) || !format_dest_state_send(fd, &info))
|
||||||
|
return RECEIVER_STEP_FAIL;
|
||||||
|
return RECEIVER_STEP_NEXT;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
File* dir = file_receive_directory(fd, config);
|
||||||
|
if (!dir)
|
||||||
|
return RECEIVER_STEP_ERROR;
|
||||||
|
if (config->report_dest_info) {
|
||||||
|
OutputDestState info;
|
||||||
|
bool ok = receiver_probe_dest_state(config, file_wire_path(dir), NULL, &info);
|
||||||
|
if (!ok || !send_status(fd, STATUS_DEST_INFO) || !format_dest_state_send(fd, &info)) {
|
||||||
|
file_destroy(dir);
|
||||||
|
return RECEIVER_STEP_FAIL;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!state->sink->store_file(dir, state->sink->context))
|
||||||
return RECEIVER_STEP_ERROR;
|
return RECEIVER_STEP_ERROR;
|
||||||
return RECEIVER_STEP_NEXT;
|
return RECEIVER_STEP_NEXT;
|
||||||
}
|
}
|
||||||
@@ -398,8 +553,20 @@ static ReceiverStep receiver_handle_hardlink(ReceiverPendingState* state) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
static ReceiverStep receiver_handle_symlink(ReceiverPendingState* state) {
|
static ReceiverStep receiver_handle_symlink(ReceiverPendingState* state) {
|
||||||
File* sym = file_receive_symlink(state->fd, state->config);
|
const Config* config = state->config;
|
||||||
if (!sym || !state->sink->store_file(sym, state->sink->context))
|
File* sym = file_receive_symlink(state->fd, config);
|
||||||
|
if (!sym)
|
||||||
|
return RECEIVER_STEP_ERROR;
|
||||||
|
if (config->report_dest_info) {
|
||||||
|
OutputDestState info;
|
||||||
|
bool ok = receiver_probe_dest_state(config, file_wire_path(sym), sym->symlink_target, &info);
|
||||||
|
if (!ok || !send_status(state->fd, STATUS_DEST_INFO) ||
|
||||||
|
!format_dest_state_send(state->fd, &info)) {
|
||||||
|
file_destroy(sym);
|
||||||
|
return RECEIVER_STEP_FAIL;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!state->sink->store_file(sym, state->sink->context))
|
||||||
return RECEIVER_STEP_ERROR;
|
return RECEIVER_STEP_ERROR;
|
||||||
return RECEIVER_STEP_NEXT;
|
return RECEIVER_STEP_NEXT;
|
||||||
}
|
}
|
||||||
@@ -442,12 +609,11 @@ static ReceiverStep receiver_handle_manifest(ReceiverPendingState* state) {
|
|||||||
--max-delete-capped commit still succeeds and the transfer proceeds;
|
--max-delete-capped commit still succeeds and the transfer proceeds;
|
||||||
the terminal success frame reports the cap. */
|
the terminal success frame reports the cap. */
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeletePathObserver observer =
|
ReceiverDeleteContext delctx;
|
||||||
(config->report_deletes && sink->deleted_paths) ? receiver_record_deleted_path : NULL;
|
DeletePathObserver observer = receiver_delete_observer(sink, &delctx);
|
||||||
DeleteCommitResult deletion =
|
DeleteCommitResult deletion =
|
||||||
(config->use_delete || config->delete_missing_args)
|
(config->use_delete || config->delete_missing_args)
|
||||||
? manifest_delete_all_observed(config, manifest, &deleted, observer,
|
? manifest_delete_all_observed(config, manifest, &deleted, observer, &delctx)
|
||||||
(void*)sink->deleted_paths)
|
|
||||||
: DELETE_COMMIT_OK;
|
: DELETE_COMMIT_OK;
|
||||||
receiver_tally_deleted(sink, deleted);
|
receiver_tally_deleted(sink, deleted);
|
||||||
delete_manifest_free(manifest);
|
delete_manifest_free(manifest);
|
||||||
@@ -490,9 +656,9 @@ static ReceiverStep receiver_handle_delete_plan(ReceiverPendingState* state) {
|
|||||||
}
|
}
|
||||||
if (!state->plan_session) {
|
if (!state->plan_session) {
|
||||||
state->plan_session = delete_plan_session_create(config);
|
state->plan_session = delete_plan_session_create(config);
|
||||||
if (state->plan_session && config->report_deletes && sink->deleted_paths)
|
if (state->plan_session && (sink->stats || sink->deleted_paths))
|
||||||
delete_plan_session_set_delete_observer(state->plan_session, receiver_record_deleted_path,
|
delete_plan_session_set_delete_observer(state->plan_session, receiver_record_deleted_path,
|
||||||
(void*)sink->deleted_paths);
|
state->delete_ctx);
|
||||||
}
|
}
|
||||||
if (!state->plan_session || delete_plan_session_receive(state->plan_session, config, fd) != 0)
|
if (!state->plan_session || delete_plan_session_receive(state->plan_session, config, fd) != 0)
|
||||||
return RECEIVER_STEP_FAIL;
|
return RECEIVER_STEP_FAIL;
|
||||||
@@ -521,6 +687,8 @@ static ReceiverStep receiver_dispatch_status(ReceiverPendingState* state, Status
|
|||||||
switch (status) {
|
switch (status) {
|
||||||
case STATUS_KEEPALIVE:
|
case STATUS_KEEPALIVE:
|
||||||
return receiver_handle_keepalive(state);
|
return receiver_handle_keepalive(state);
|
||||||
|
case STATUS_CLIENT_MSG:
|
||||||
|
return receiver_handle_client_msg(state);
|
||||||
case STATUS_ABORT:
|
case STATUS_ABORT:
|
||||||
return receiver_handle_abort(state);
|
return receiver_handle_abort(state);
|
||||||
case STATUS_CHECK:
|
case STATUS_CHECK:
|
||||||
@@ -573,8 +741,10 @@ static void receiver_drop_pending(ReceiverPendingState* state) {
|
|||||||
delete-during plan mode (no manifest at all). See the per-frame handlers
|
delete-during plan mode (no manifest at all). See the per-frame handlers
|
||||||
above for how the -m receiver defers that commit until its disk writer has
|
above for how the -m receiver defers that commit until its disk writer has
|
||||||
drained. */
|
drained. */
|
||||||
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
int receiver_process_pending_ctx(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||||
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
|
DeleteManifest** pending_manifest,
|
||||||
|
DeletePlanSession** pending_plans,
|
||||||
|
ReceiverDeleteContext* observer_ctx) {
|
||||||
Status status;
|
Status status;
|
||||||
if (!receive_status(file_descriptor, &status))
|
if (!receive_status(file_descriptor, &status))
|
||||||
return -1;
|
return -1;
|
||||||
@@ -587,6 +757,13 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
|
|||||||
last_progress = session_start;
|
last_progress = session_start;
|
||||||
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
|
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
|
||||||
return -1;
|
return -1;
|
||||||
|
/* A per-directory delete session handed to the caller outlives this stack
|
||||||
|
frame, so its observer context must be caller-owned (observer_ctx); only
|
||||||
|
the default inline-commit case may use the stack context. */
|
||||||
|
ReceiverDeleteContext local_ctx;
|
||||||
|
ReceiverDeleteContext* delete_ctx = observer_ctx ? observer_ctx : &local_ctx;
|
||||||
|
delete_ctx->stats = sink ? sink->stats : NULL;
|
||||||
|
delete_ctx->deleted_paths = sink ? sink->deleted_paths : NULL;
|
||||||
ReceiverPendingState state = {
|
ReceiverPendingState state = {
|
||||||
.config = config,
|
.config = config,
|
||||||
.fd = file_descriptor,
|
.fd = file_descriptor,
|
||||||
@@ -595,6 +772,7 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
|
|||||||
.pending_plans = pending_plans,
|
.pending_plans = pending_plans,
|
||||||
.deferred_manifest = NULL,
|
.deferred_manifest = NULL,
|
||||||
.plan_session = NULL,
|
.plan_session = NULL,
|
||||||
|
.delete_ctx = delete_ctx,
|
||||||
.early_delete = config_delete_timing_early(config),
|
.early_delete = config_delete_timing_early(config),
|
||||||
.per_dir_delete = config_delete_timing_per_dir(config),
|
.per_dir_delete = config_delete_timing_per_dir(config),
|
||||||
.delete_limit_noted = false,
|
.delete_limit_noted = false,
|
||||||
@@ -604,7 +782,7 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
|
|||||||
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH ||
|
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH ||
|
||||||
status == STATUS_MKDIR || status == STATUS_MANIFEST || status == STATUS_HARDLINK ||
|
status == STATUS_MKDIR || status == STATUS_MANIFEST || status == STATUS_HARDLINK ||
|
||||||
status == STATUS_SYMLINK || status == STATUS_SPECIAL || status == STATUS_DIR_TIMES ||
|
status == STATUS_SYMLINK || status == STATUS_SPECIAL || status == STATUS_DIR_TIMES ||
|
||||||
status == STATUS_DELETE_PLAN) {
|
status == STATUS_DELETE_PLAN || status == STATUS_CLIENT_MSG) {
|
||||||
ReceiverStep step = receiver_dispatch_status(&state, status);
|
ReceiverStep step = receiver_dispatch_status(&state, status);
|
||||||
if (step == RECEIVER_STEP_FAIL)
|
if (step == RECEIVER_STEP_FAIL)
|
||||||
goto fail;
|
goto fail;
|
||||||
@@ -619,25 +797,36 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
|
|||||||
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
|
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
|
||||||
goto receive_error;
|
goto receive_error;
|
||||||
}
|
}
|
||||||
|
/* --delay-updates: publish every staged file BEFORE the deferred delete
|
||||||
|
commit, matching rsync's --delete-after ordering (all updates land first,
|
||||||
|
then extras are removed). The single-threaded receiver stores files
|
||||||
|
synchronously, so every staged file is complete here. The -m receiver
|
||||||
|
hands both publication and deletion to its caller via
|
||||||
|
pending_manifest/pending_plans; that caller publishes first, after its disk
|
||||||
|
writer has drained. */
|
||||||
|
bool handoff = state.pending_manifest != NULL || state.pending_plans != NULL;
|
||||||
|
if (!handoff && !config->dry_run && config->delay_updates && config->delay_context) {
|
||||||
|
if (!delay_updates_publish(config->delay_context, config)) {
|
||||||
|
send_status(file_descriptor, STATUS_ERROR);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
}
|
||||||
/* Commit-style (late) deletion: every data frame has been received and the
|
/* Commit-style (late) deletion: every data frame has been received and the
|
||||||
sender proved the whole tree with STATUS_FINISHED. The single-threaded
|
sender proved the whole tree with STATUS_FINISHED. The single-threaded
|
||||||
receiver stores files synchronously, so everything is on disk here and the
|
receiver stores files synchronously, so everything is on disk here (and a
|
||||||
deletion can be committed before the --delay-updates publication in
|
--delay-updates run has already published above). The -m receiver passes
|
||||||
send_success (the walker skips the staging dir, so staged files are never
|
`pending_manifest` because its disk writer may still be draining; the
|
||||||
treated as extras). The -m receiver passes `pending_manifest` because its
|
caller commits after the writer has joined so no extra file is removed
|
||||||
disk writer may still be draining; the caller commits after the writer has
|
unless the transfer is known to have succeeded. */
|
||||||
joined so no extra file is removed unless the transfer is known to have
|
|
||||||
succeeded. */
|
|
||||||
if (state.deferred_manifest) {
|
if (state.deferred_manifest) {
|
||||||
if (state.pending_manifest) {
|
if (state.pending_manifest) {
|
||||||
*state.pending_manifest = state.deferred_manifest;
|
*state.pending_manifest = state.deferred_manifest;
|
||||||
state.deferred_manifest = NULL;
|
state.deferred_manifest = NULL;
|
||||||
} else {
|
} else {
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeletePathObserver observer =
|
DeletePathObserver observer = receiver_delete_observer(sink, state.delete_ctx);
|
||||||
(config->report_deletes && sink->deleted_paths) ? receiver_record_deleted_path : NULL;
|
|
||||||
DeleteCommitResult deletion = manifest_delete_all_observed(
|
DeleteCommitResult deletion = manifest_delete_all_observed(
|
||||||
config, state.deferred_manifest, &deleted, observer, (void*)sink->deleted_paths);
|
config, state.deferred_manifest, &deleted, observer, state.delete_ctx);
|
||||||
receiver_tally_deleted(sink, deleted);
|
receiver_tally_deleted(sink, deleted);
|
||||||
delete_manifest_free(state.deferred_manifest);
|
delete_manifest_free(state.deferred_manifest);
|
||||||
state.deferred_manifest = NULL;
|
state.deferred_manifest = NULL;
|
||||||
@@ -655,9 +844,9 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
|
|||||||
hands the session to its caller instead, which commits after the disk
|
hands the session to its caller instead, which commits after the disk
|
||||||
writer drained. */
|
writer drained. */
|
||||||
if (state.plan_session) {
|
if (state.plan_session) {
|
||||||
if (config->report_deletes && sink->deleted_paths)
|
if (sink->stats || sink->deleted_paths)
|
||||||
delete_plan_session_set_delete_observer(state.plan_session, receiver_record_deleted_path,
|
delete_plan_session_set_delete_observer(state.plan_session, receiver_record_deleted_path,
|
||||||
(void*)sink->deleted_paths);
|
state.delete_ctx);
|
||||||
if (state.pending_plans) {
|
if (state.pending_plans) {
|
||||||
*state.pending_plans = state.plan_session;
|
*state.pending_plans = state.plan_session;
|
||||||
state.plan_session = NULL;
|
state.plan_session = NULL;
|
||||||
@@ -719,6 +908,11 @@ typedef struct {
|
|||||||
ArrayList* would_delete;
|
ArrayList* would_delete;
|
||||||
/* --info=del: actually-removed paths collected during the delete commit. */
|
/* --info=del: actually-removed paths collected during the delete commit. */
|
||||||
ArrayList* deleted_paths;
|
ArrayList* deleted_paths;
|
||||||
|
/* Per-run count of entries that failed to materialize without aborting the
|
||||||
|
stream (currently ONLY a --devices mknod EPERM/EACCES). A nonzero count
|
||||||
|
makes the terminal frame carry a non-OK status so the client exits
|
||||||
|
non-zero, matching rsync's continue-and-exit-partial behavior. */
|
||||||
|
size_t failed_entries;
|
||||||
} ReceiverSaveContext;
|
} ReceiverSaveContext;
|
||||||
|
|
||||||
static bool receiver_save_file(File* file, void* context_pointer) {
|
static bool receiver_save_file(File* file, void* context_pointer) {
|
||||||
@@ -742,6 +936,11 @@ static bool receiver_save_file(File* file, void* context_pointer) {
|
|||||||
count as matched data in the end-of-transfer report. */
|
count as matched data in the end-of-transfer report. */
|
||||||
if (result != FILE_SAVE_ERROR && file->matched_bytes > 0)
|
if (result != FILE_SAVE_ERROR && file->matched_bytes > 0)
|
||||||
context->stats.matched_data += file->matched_bytes;
|
context->stats.matched_data += file->matched_bytes;
|
||||||
|
/* --devices parity: a device node the receiver could not mknod (EPERM/EACCES)
|
||||||
|
is counted per-run but does not abort the transfer. The terminal frame
|
||||||
|
turns a nonzero count into a non-OK status so the client exits non-zero. */
|
||||||
|
if (result == FILE_SAVE_FAILED)
|
||||||
|
context->failed_entries++;
|
||||||
/* Protocol 2.28.0: receiver-observed literal bytes and the created-entry
|
/* Protocol 2.28.0: receiver-observed literal bytes and the created-entry
|
||||||
breakdown (regular/dir/link/special) for the `--stats` report. */
|
breakdown (regular/dir/link/special) for the `--stats` report. */
|
||||||
if (result == FILE_SAVE_WRITTEN)
|
if (result == FILE_SAVE_WRITTEN)
|
||||||
@@ -775,9 +974,26 @@ static void receiver_note_delete_limit(void* context_pointer) {
|
|||||||
context->delete_limit_reached = true;
|
context->delete_limit_reached = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Terminal status for a run. A capped --delete limit wins (rsync exit 25);
|
||||||
|
otherwise any per-entry failure (for example an unprivileged --devices
|
||||||
|
mknod) makes the terminal frame STATUS_PARTIAL so the client exits 23
|
||||||
|
(rsync's "partial transfer due to error") while still removing the sources
|
||||||
|
it successfully transferred under --remove-source-files. A fatal stream
|
||||||
|
error keeps STATUS_ERROR (a non-23 exit). A clean run keeps STATUS_OK. */
|
||||||
|
static Status receiver_final_status(bool delete_limit_reached, size_t failed_entries) {
|
||||||
|
if (delete_limit_reached)
|
||||||
|
return STATUS_DELETE_LIMIT;
|
||||||
|
return failed_entries > 0 ? STATUS_PARTIAL : STATUS_OK;
|
||||||
|
}
|
||||||
|
|
||||||
static bool receiver_send_success_frame(int fd, void* context_pointer) {
|
static bool receiver_send_success_frame(int fd, void* context_pointer) {
|
||||||
ReceiverSaveContext* context = context_pointer;
|
ReceiverSaveContext* context = context_pointer;
|
||||||
Status final_status = context->delete_limit_reached ? STATUS_DELETE_LIMIT : STATUS_OK;
|
if (context->failed_entries > 0)
|
||||||
|
log_message(LOG_LEVEL_WARNING,
|
||||||
|
"%zu entr%s failed to materialize; continuing (partial transfer)",
|
||||||
|
context->failed_entries, context->failed_entries == 1 ? "y" : "ies");
|
||||||
|
Status final_status =
|
||||||
|
receiver_final_status(context->delete_limit_reached, context->failed_entries);
|
||||||
if (!receiver_send_stats_frame(fd, context->config, &context->stats, context->would_delete,
|
if (!receiver_send_stats_frame(fd, context->config, &context->stats, context->would_delete,
|
||||||
context->deleted_paths))
|
context->deleted_paths))
|
||||||
return false;
|
return false;
|
||||||
@@ -785,21 +1001,10 @@ static bool receiver_send_success_frame(int fd, void* context_pointer) {
|
|||||||
nothing to publish and no directory times to stamp. */
|
nothing to publish and no directory times to stamp. */
|
||||||
if (context->config->dry_run)
|
if (context->config->dry_run)
|
||||||
return receiver_send_final_success(fd, context->config, &context->outcomes, final_status);
|
return receiver_send_final_success(fd, context->config, &context->outcomes, final_status);
|
||||||
/* --delay-updates: the whole protocol stream (including manifest/delete
|
/* P7 Wave D: every child is now written and the --delay-updates publication
|
||||||
handling, which ran inside receiver_process) has succeeded and every
|
(done in receiver_process before the delete commit) plus the deferred
|
||||||
staged file was fully written. Publish them atomically now, before the
|
deletion have both committed, so it is finally safe to stamp directory
|
||||||
success/outcome frame tells a --remove-source-files sender it may delete
|
times. */
|
||||||
its sources. */
|
|
||||||
if (context->config->delay_updates && context->config->delay_context) {
|
|
||||||
if (!delay_updates_publish(context->config->delay_context, context->config)) {
|
|
||||||
send_status(fd, STATUS_ERROR);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
/* P7 Wave D: every child is now written and the delete / --delay-updates
|
|
||||||
phases have committed, so it is finally safe to stamp directory times.
|
|
||||||
This runs after the deferred deletion because receiver_process commits it
|
|
||||||
before calling this success frame. */
|
|
||||||
dir_metadata_list_apply(&context->dir_times, context->config->receive_root_directory,
|
dir_metadata_list_apply(&context->dir_times, context->config->receive_root_directory,
|
||||||
context->config);
|
context->config);
|
||||||
return receiver_send_final_success(fd, context->config, &context->outcomes, final_status);
|
return receiver_send_final_success(fd, context->config, &context->outcomes, final_status);
|
||||||
|
|||||||
+25
-4
@@ -55,10 +55,20 @@ typedef struct {
|
|||||||
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code);
|
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code);
|
||||||
void receiver_outcomes_destroy(ReceiverOutcomes* outcomes);
|
void receiver_outcomes_destroy(ReceiverOutcomes* outcomes);
|
||||||
|
|
||||||
/* DeletePathObserver implementation for --info=del: `context` is an ArrayList*
|
/* Delete observer context: `deleted_paths` (optional) receives owned copies of
|
||||||
that receives owned copies of every truly-removed destination-relative path.
|
every truly-removed destination-relative path for --info=del; `stats`
|
||||||
Shared by the single-threaded receiver and the -m pipeline's deferred commit. */
|
(optional) receives the per-type `Number of deleted files` tallies for
|
||||||
void receiver_record_deleted_path(void* context, const char* rel_path);
|
--stats. Both may be NULL, in which case the observer is a no-op. */
|
||||||
|
typedef struct {
|
||||||
|
ReceiverStats* stats;
|
||||||
|
struct ArrayList* deleted_paths;
|
||||||
|
} ReceiverDeleteContext;
|
||||||
|
|
||||||
|
/* DeletePathObserver implementation: records each truly-removed path (when the
|
||||||
|
context carries a path list) and tallies it by type (when it carries a stats
|
||||||
|
record). Shared by the single-threaded receiver and the -m pipeline's
|
||||||
|
deferred commit. */
|
||||||
|
void receiver_record_deleted_path(void* context, const char* rel_path, DeleteEntryType type);
|
||||||
|
|
||||||
/* Send the terminal success frame. `final_status` is usually STATUS_OK, or
|
/* Send the terminal success frame. `final_status` is usually STATUS_OK, or
|
||||||
STATUS_DELETE_LIMIT when a --max-delete commit was capped. */
|
STATUS_DELETE_LIMIT when a --max-delete commit was capped. */
|
||||||
@@ -83,6 +93,17 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
|
|||||||
for either to keep the default behaviour (delete before the success frame). */
|
for either to keep the default behaviour (delete before the success frame). */
|
||||||
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||||
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans);
|
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans);
|
||||||
|
/* receiver_process_pending() with an explicit observer context for a
|
||||||
|
per-directory delete session that is handed to the caller via
|
||||||
|
`pending_plans`. The session outlives this call (the -m pipeline commits it
|
||||||
|
after joining its disk writer), so its observer context must too: pass a
|
||||||
|
long-lived object such as PipelineContextReceiver.delete_ctx. When
|
||||||
|
`delete_ctx` is NULL an internal stack context is used, which is only safe
|
||||||
|
when the session is committed before returning (the default behaviour). */
|
||||||
|
int receiver_process_pending_ctx(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||||
|
DeleteManifest** pending_manifest,
|
||||||
|
DeletePlanSession** pending_plans,
|
||||||
|
ReceiverDeleteContext* delete_ctx);
|
||||||
int receiver_receive_files(Config* config, int file_descriptor);
|
int receiver_receive_files(Config* config, int file_descriptor);
|
||||||
|
|
||||||
/* ---- Connection time bounds (anti-slowloris) ----
|
/* ---- Connection time bounds (anti-slowloris) ----
|
||||||
|
|||||||
@@ -28,7 +28,10 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
|
|||||||
context->max_queue_bytes = 0;
|
context->max_queue_bytes = 0;
|
||||||
context->deferred_manifest = NULL;
|
context->deferred_manifest = NULL;
|
||||||
context->deferred_plans = NULL;
|
context->deferred_plans = NULL;
|
||||||
|
context->delete_ctx.stats = NULL;
|
||||||
|
context->delete_ctx.deleted_paths = NULL;
|
||||||
context->delete_limit_reached = false;
|
context->delete_limit_reached = false;
|
||||||
|
context->failed_entries = 0;
|
||||||
memset(&context->stats, 0, sizeof(context->stats));
|
memset(&context->stats, 0, sizeof(context->stats));
|
||||||
context->would_delete = NULL;
|
context->would_delete = NULL;
|
||||||
context->deleted_paths = NULL;
|
context->deleted_paths = NULL;
|
||||||
@@ -204,8 +207,9 @@ int receive_thread(void* pipeline_context) {
|
|||||||
&context->stats,
|
&context->stats,
|
||||||
context->would_delete,
|
context->would_delete,
|
||||||
context->deleted_paths};
|
context->deleted_paths};
|
||||||
if (receiver_process_pending((Config*)config, file_descriptor, &sink, &context->deferred_manifest,
|
if (receiver_process_pending_ctx((Config*)config, file_descriptor, &sink,
|
||||||
&context->deferred_plans) != 0) {
|
&context->deferred_manifest, &context->deferred_plans,
|
||||||
|
&context->delete_ctx) != 0) {
|
||||||
receiver_thread_fail(context);
|
receiver_thread_fail(context);
|
||||||
protocol_session_unbind();
|
protocol_session_unbind();
|
||||||
return thrd_error;
|
return thrd_error;
|
||||||
@@ -264,6 +268,13 @@ int write_thread(void* pipeline_context) {
|
|||||||
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
|
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
|
||||||
mtx_unlock(&context->mutex);
|
mtx_unlock(&context->mutex);
|
||||||
}
|
}
|
||||||
|
/* --devices parity: a device node that could not be mknod'ed is counted
|
||||||
|
per-run but does NOT abort the transfer. */
|
||||||
|
if (result == FILE_SAVE_FAILED) {
|
||||||
|
mtx_lock(&context->mutex);
|
||||||
|
context->failed_entries++;
|
||||||
|
mtx_unlock(&context->mutex);
|
||||||
|
}
|
||||||
if (result == FILE_SAVE_ERROR) {
|
if (result == FILE_SAVE_ERROR) {
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
pipeline_context_receiver_note_bytes_released(context, file_bytes);
|
pipeline_context_receiver_note_bytes_released(context, file_bytes);
|
||||||
|
|||||||
@@ -46,6 +46,11 @@ typedef struct PipelineContextReceiver {
|
|||||||
committing while the disk writer may still be draining; server.c commits it
|
committing while the disk writer may still be draining; server.c commits it
|
||||||
after both threads joined. NULL for every other timing. */
|
after both threads joined. NULL for every other timing. */
|
||||||
DeletePlanSession* deferred_plans;
|
DeletePlanSession* deferred_plans;
|
||||||
|
/* Observer context for `deferred_plans`. It must outlive the receive thread
|
||||||
|
(the session is committed by server.c after both threads join), so it lives
|
||||||
|
here rather than on receiver_process_pending()'s stack; receive_thread
|
||||||
|
installs it on the session. */
|
||||||
|
ReceiverDeleteContext delete_ctx;
|
||||||
/* Set by server.c when the deferred delete commit hit the --max-delete
|
/* Set by server.c when the deferred delete commit hit the --max-delete
|
||||||
budget; the terminal success frame then carries STATUS_DELETE_LIMIT
|
budget; the terminal success frame then carries STATUS_DELETE_LIMIT
|
||||||
(rsync exit 25) while the transfer itself still succeeds. */
|
(rsync exit 25) while the transfer itself still succeeds. */
|
||||||
@@ -64,6 +69,11 @@ typedef struct PipelineContextReceiver {
|
|||||||
/* --info=del actually-removed path list, collected by the deferred delete
|
/* --info=del actually-removed path list, collected by the deferred delete
|
||||||
commit in server.c and reported in the STATUS_STATS frame. */
|
commit in server.c and reported in the STATUS_STATS frame. */
|
||||||
struct ArrayList* deleted_paths;
|
struct ArrayList* deleted_paths;
|
||||||
|
/* Per-run count of entries that failed to materialize without aborting the
|
||||||
|
stream (currently ONLY a --devices mknod EPERM/EACCES). write_thread
|
||||||
|
increments it under `mutex`; server.c turns a nonzero count into a non-OK
|
||||||
|
terminal status so the client exits non-zero. */
|
||||||
|
size_t failed_entries;
|
||||||
} PipelineContextReceiver;
|
} PipelineContextReceiver;
|
||||||
|
|
||||||
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
|
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
|
||||||
|
|||||||
+202
-155
@@ -981,18 +981,31 @@ static void server_run_mt_receiver(ServerSession* state) {
|
|||||||
thrd_join(writer, &writer_result);
|
thrd_join(writer, &writer_result);
|
||||||
bool transfer_ok = receiver_result == thrd_success && writer_result == thrd_success;
|
bool transfer_ok = receiver_result == thrd_success && writer_result == thrd_success;
|
||||||
PipelineContextReceiver* context = state->context;
|
PipelineContextReceiver* context = state->context;
|
||||||
|
if (transfer_ok && !config->dry_run) {
|
||||||
|
/* --delay-updates: receive_thread has finished the whole protocol stream
|
||||||
|
and write_thread has drained its queue, so every staged file is complete.
|
||||||
|
Publish atomically BEFORE the deferred delete commit, matching rsync's
|
||||||
|
--delete-after ordering (all updates land first, then extras are
|
||||||
|
removed). */
|
||||||
|
if (config->delay_updates && config->delay_context &&
|
||||||
|
!delay_updates_publish(config->delay_context, config)) {
|
||||||
|
transfer_ok = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
if (transfer_ok && !config->dry_run) {
|
if (transfer_ok && !config->dry_run) {
|
||||||
/* Commit-style (late) deletion: receive_thread handed the keep-set
|
/* Commit-style (late) deletion: receive_thread handed the keep-set
|
||||||
manifest here instead of deleting while write_thread might still be
|
manifest here instead of deleting while write_thread might still be
|
||||||
draining, so by now every file is on disk and the whole transfer is
|
draining, so by now every file is on disk (and a --delay-updates run has
|
||||||
known to have succeeded. Remove the extras before publishing a
|
already published above) and the whole transfer is known to have
|
||||||
--delay-updates run; the walker skips the staging directory. A
|
succeeded. The walker skips the staging directory. A
|
||||||
server-contacting --dry-run deletes nothing (no manifest is sent). */
|
server-contacting --dry-run deletes nothing (no manifest is sent). */
|
||||||
if (context->deferred_manifest) {
|
if (context->deferred_manifest) {
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeletePathObserver observer = config->report_deletes ? receiver_record_deleted_path : NULL;
|
ReceiverDeleteContext delctx = {&context->stats, context->deleted_paths};
|
||||||
DeleteCommitResult deletion = manifest_delete_all_observed(
|
DeletePathObserver observer =
|
||||||
config, context->deferred_manifest, &deleted, observer, (void*)context->deleted_paths);
|
(delctx.stats || delctx.deleted_paths) ? receiver_record_deleted_path : NULL;
|
||||||
|
DeleteCommitResult deletion = manifest_delete_all_observed(config, context->deferred_manifest,
|
||||||
|
&deleted, observer, &delctx);
|
||||||
context->stats.deleted_files += deleted;
|
context->stats.deleted_files += deleted;
|
||||||
if (deletion == DELETE_COMMIT_ERROR) {
|
if (deletion == DELETE_COMMIT_ERROR) {
|
||||||
transfer_ok = false;
|
transfer_ok = false;
|
||||||
@@ -1009,10 +1022,9 @@ static void server_run_mt_receiver(ServerSession* state) {
|
|||||||
--delete-during already applied its plans on the receive thread. */
|
--delete-during already applied its plans on the receive thread. */
|
||||||
if (context->deferred_plans) {
|
if (context->deferred_plans) {
|
||||||
/* Defence in depth (the enclosing block already excludes dry-run): a
|
/* Defence in depth (the enclosing block already excludes dry-run): a
|
||||||
-n run never commits a deletion. */
|
-n run never commits a deletion. The session's observer context was
|
||||||
if (config->report_deletes)
|
installed by receive_thread from context->delete_ctx, which outlives
|
||||||
delete_plan_session_set_delete_observer(
|
both threads, so no stack context is needed here. */
|
||||||
context->deferred_plans, receiver_record_deleted_path, (void*)context->deleted_paths);
|
|
||||||
DeleteCommitResult deletion =
|
DeleteCommitResult deletion =
|
||||||
config->dry_run ? DELETE_COMMIT_OK
|
config->dry_run ? DELETE_COMMIT_OK
|
||||||
: delete_plan_session_commit(context->deferred_plans, config);
|
: delete_plan_session_commit(context->deferred_plans, config);
|
||||||
@@ -1025,26 +1037,21 @@ static void server_run_mt_receiver(ServerSession* state) {
|
|||||||
delete_plan_session_destroy(context->deferred_plans);
|
delete_plan_session_destroy(context->deferred_plans);
|
||||||
context->deferred_plans = NULL;
|
context->deferred_plans = NULL;
|
||||||
}
|
}
|
||||||
}
|
/* P7 Wave D: all writers have joined and the --delay-updates publication
|
||||||
if (transfer_ok && !config->dry_run) {
|
plus the late deletion have committed above, so it is finally safe to
|
||||||
/* --delay-updates: receive_thread has finished the whole protocol stream
|
stamp directory times; a directory's mtime must not be clobbered by its
|
||||||
(including manifest/delete handling) and write_thread has drained its
|
children or by an extra removal. */
|
||||||
queue, so every staged file is complete. Publish atomically before the
|
|
||||||
success/outcome frame so a --remove-source-files sender only learns of
|
|
||||||
files that were actually installed. */
|
|
||||||
if (config->delay_updates && config->delay_context &&
|
|
||||||
!delay_updates_publish(config->delay_context, config)) {
|
|
||||||
transfer_ok = false;
|
|
||||||
}
|
|
||||||
/* P7 Wave D: all writers have joined and the late deletion (and
|
|
||||||
--delay-updates publication) has committed above, so it is finally safe
|
|
||||||
to stamp directory times; a directory's mtime must not be clobbered by
|
|
||||||
its children or by an extra removal. */
|
|
||||||
if (transfer_ok)
|
if (transfer_ok)
|
||||||
dir_metadata_list_apply(&context->dir_times, config->receive_root_directory, config);
|
dir_metadata_list_apply(&context->dir_times, config->receive_root_directory, config);
|
||||||
}
|
}
|
||||||
if (transfer_ok) {
|
if (transfer_ok) {
|
||||||
Status final_status = context->delete_limit_reached ? STATUS_DELETE_LIMIT : STATUS_OK;
|
if (context->failed_entries > 0)
|
||||||
|
log_message(LOG_LEVEL_WARNING,
|
||||||
|
"%zu entr%s failed to materialize; continuing (partial transfer)",
|
||||||
|
context->failed_entries, context->failed_entries == 1 ? "y" : "ies");
|
||||||
|
Status final_status = context->delete_limit_reached
|
||||||
|
? STATUS_DELETE_LIMIT
|
||||||
|
: (context->failed_entries > 0 ? STATUS_PARTIAL : STATUS_OK);
|
||||||
/* Emit the optional wire-stats record first (protocol 2.25.0), then the
|
/* Emit the optional wire-stats record first (protocol 2.25.0), then the
|
||||||
success/outcome frame, exactly like the single-threaded receiver. */
|
success/outcome frame, exactly like the single-threaded receiver. */
|
||||||
if (!receiver_send_stats_frame(state->fd, config, &context->stats, context->would_delete,
|
if (!receiver_send_stats_frame(state->fd, config, &context->stats, context->would_delete,
|
||||||
@@ -1299,29 +1306,45 @@ static bool daemonize(void) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
int main(int argc, char* argv[]) {
|
/* Apply the process-wide policies shared by the stdio and listener
|
||||||
/* Capture the process umask now, while still single-threaded: the cached
|
* entrypoints: logging verbosity, signal handling, the parsed server
|
||||||
* value is what file_mode_base() uses, and reading it later would race with
|
* authorization policies, and the socket timeout floor. Runs after CLI
|
||||||
* receiver threads creating files. */
|
* parsing and after the standalone --hash-credentials tool has been ruled
|
||||||
file_umask_capture();
|
* out. */
|
||||||
ServerCliOptions opts;
|
static void configure_server_process(const ServerCliOptions* opts) {
|
||||||
char cli_err[512];
|
signal(SIGPIPE, SIG_IGN);
|
||||||
int parse_result = server_cli_parse(argc, argv, &opts, cli_err, sizeof(cli_err));
|
if (opts->verbose) {
|
||||||
if (parse_result == 1) {
|
set_log_level(LOG_LEVEL_DEBUG);
|
||||||
print_server_usage();
|
set_log_debug_flags(LOG_DEBUG_ALL);
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
if (parse_result < 0) {
|
|
||||||
server_cli_options_free(&opts);
|
|
||||||
fprintf(stderr, "Error: %s\n", cli_err);
|
|
||||||
print_server_usage();
|
|
||||||
return 1;
|
|
||||||
}
|
}
|
||||||
|
if (opts->tls_ca && !opts->use_tls)
|
||||||
|
log_message(LOG_LEVEL_WARNING, "--ca has no effect without --tls");
|
||||||
|
/* Persist the parsed server policies into the process-global policy state
|
||||||
|
* BEFORE the stdio branch: an SSH-launched `--stdio` server (whose argv came
|
||||||
|
* from the client via --remote-option and friends) must honor --allow-delete,
|
||||||
|
* --trust-sender and --client-cn exactly like the standalone listener. */
|
||||||
|
required_client_cn = opts->client_cn;
|
||||||
|
allow_delete = opts->allow_delete;
|
||||||
|
trust_sender = opts->trust_sender;
|
||||||
|
allow_unauthenticated = opts->allow_unauthenticated;
|
||||||
|
server_no_super = opts->no_super;
|
||||||
|
/* --stdio rejects --allow-super at parse time; force it off here as well so
|
||||||
|
* this process-global policy cannot be re-enabled by a future caller. */
|
||||||
|
server_allow_super = opts->allow_super && !opts->stdio_mode;
|
||||||
|
server_iconv_spec = opts->iconv_spec;
|
||||||
|
install_cleanup_handler(SIGINT);
|
||||||
|
install_cleanup_handler(SIGTERM);
|
||||||
|
/* Server-owned socket deadline floor: the client default --timeout=0 would
|
||||||
|
* otherwise leave accepted sockets without SO_RCVTIMEO/SO_SNDTIMEO and let a
|
||||||
|
* silent peer hold a connection (and its process slot) forever. */
|
||||||
|
tcp_set_timeouts(SERVER_IO_TIMEOUT_SEC, SERVER_IO_TIMEOUT_SEC);
|
||||||
|
}
|
||||||
|
|
||||||
/* --hash-credentials: standalone offline tool; read user:password lines and
|
/* --hash-credentials: standalone offline tool; read user:password lines and
|
||||||
* emit new-format credential-store lines, then exit. */
|
* emit new-format credential-store lines, then exit. Consumes and frees
|
||||||
if (opts.hash_credentials_file) {
|
* opts. */
|
||||||
uint32_t iters = opts.hash_iterations_set ? opts.hash_iterations : CREDENTIAL_DEFAULT_ITERS;
|
static int run_hash_credentials_tool(ServerCliOptions* opts) {
|
||||||
|
uint32_t iters = opts->hash_iterations_set ? opts->hash_iterations : CREDENTIAL_DEFAULT_ITERS;
|
||||||
/* The output is secret material: if it is redirected to a regular file,
|
/* The output is secret material: if it is redirected to a regular file,
|
||||||
* warn when that file is group/other-accessible (the store must be 0600). */
|
* warn when that file is group/other-accessible (the store must be 0600). */
|
||||||
struct stat out_st;
|
struct stat out_st;
|
||||||
@@ -1331,53 +1354,29 @@ int main(int argc, char* argv[]) {
|
|||||||
"Warning: credential-store output is a group/other-accessible file; restrict it to "
|
"Warning: credential-store output is a group/other-accessible file; restrict it to "
|
||||||
"mode 0600 (chmod 600)\n");
|
"mode 0600 (chmod 600)\n");
|
||||||
char hash_err[512];
|
char hash_err[512];
|
||||||
if (credentials_hash_file(opts.hash_credentials_file, iters, stdout, hash_err,
|
if (credentials_hash_file(opts->hash_credentials_file, iters, stdout, hash_err,
|
||||||
sizeof(hash_err)) != 0) {
|
sizeof(hash_err)) != 0) {
|
||||||
fprintf(stderr, "Error: %s\n", hash_err);
|
fprintf(stderr, "Error: %s\n", hash_err);
|
||||||
server_cli_options_free(&opts);
|
server_cli_options_free(opts);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
server_cli_options_free(&opts);
|
server_cli_options_free(opts);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
int exit_code = 0;
|
/* SSH --stdio session: the transport is authenticated by sshd outside of
|
||||||
signal(SIGPIPE, SIG_IGN);
|
* FastSync, so the single connection is served over STDIN/STDOUT and the
|
||||||
if (opts.verbose) {
|
* process exits. The destination root is authorized exactly like the listener
|
||||||
set_log_level(LOG_LEVEL_DEBUG);
|
* path. Consumes and frees opts. */
|
||||||
set_log_debug_flags(LOG_DEBUG_ALL);
|
static int run_stdio_server(ServerCliOptions* opts) {
|
||||||
}
|
|
||||||
if (opts.tls_ca && !opts.use_tls)
|
|
||||||
log_message(LOG_LEVEL_WARNING, "--ca has no effect without --tls");
|
|
||||||
/* Persist the parsed server policies into the process-global policy state
|
|
||||||
* BEFORE the stdio branch: an SSH-launched `--stdio` server (whose argv came
|
|
||||||
* from the client via --remote-option and friends) must honor --allow-delete,
|
|
||||||
* --trust-sender and --client-cn exactly like the standalone listener. */
|
|
||||||
required_client_cn = opts.client_cn;
|
|
||||||
allow_delete = opts.allow_delete;
|
|
||||||
trust_sender = opts.trust_sender;
|
|
||||||
allow_unauthenticated = opts.allow_unauthenticated;
|
|
||||||
server_no_super = opts.no_super;
|
|
||||||
/* --stdio rejects --allow-super at parse time; force it off here as well so
|
|
||||||
* this process-global policy cannot be re-enabled by a future caller. */
|
|
||||||
server_allow_super = opts.allow_super && !opts.stdio_mode;
|
|
||||||
server_iconv_spec = opts.iconv_spec;
|
|
||||||
install_cleanup_handler(SIGINT);
|
|
||||||
install_cleanup_handler(SIGTERM);
|
|
||||||
/* Server-owned socket deadline floor: the client default --timeout=0 would
|
|
||||||
* otherwise leave accepted sockets without SO_RCVTIMEO/SO_SNDTIMEO and let a
|
|
||||||
* silent peer hold a connection (and its process slot) forever. */
|
|
||||||
tcp_set_timeouts(SERVER_IO_TIMEOUT_SEC, SERVER_IO_TIMEOUT_SEC);
|
|
||||||
|
|
||||||
if (opts.stdio_mode) {
|
|
||||||
/* SSH authenticates the stdio transport outside of FastSync. */
|
/* SSH authenticates the stdio transport outside of FastSync. */
|
||||||
allow_unauthenticated = true;
|
allow_unauthenticated = true;
|
||||||
if (!configure_authorization(opts.destination_root)) {
|
if (!configure_authorization(opts->destination_root)) {
|
||||||
char* escaped = output_escape(opts.destination_root, false);
|
char* escaped = output_escape(opts->destination_root, false);
|
||||||
fprintf(stderr, "Error: invalid destination root '%s'\n",
|
fprintf(stderr, "Error: invalid destination root '%s'\n",
|
||||||
escaped ? escaped : "<allocation failed>");
|
escaped ? escaped : "<allocation failed>");
|
||||||
free(escaped);
|
free(escaped);
|
||||||
server_cli_options_free(&opts);
|
server_cli_options_free(opts);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
io_set_fds(STDIN_FILENO, STDOUT_FILENO);
|
io_set_fds(STDIN_FILENO, STDOUT_FILENO);
|
||||||
@@ -1386,34 +1385,32 @@ int main(int argc, char* argv[]) {
|
|||||||
* and are released by process exit. */
|
* and are released by process exit. */
|
||||||
handler(STDIN_FILENO);
|
handler(STDIN_FILENO);
|
||||||
release_authorization();
|
release_authorization();
|
||||||
server_cli_options_free(&opts);
|
server_cli_options_free(opts);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
int port = opts.port;
|
/* Load the daemon config, apply --dparam overrides, resolve the effective
|
||||||
int bind_family = opts.bind_family;
|
* port/address, and surface the operator-facing module warnings. On failure
|
||||||
const char* bind_address = opts.bind_address;
|
* the error is printed and false is returned. */
|
||||||
|
static bool load_daemon_policy(ServerCliOptions* opts, int* port, const char** bind_address,
|
||||||
if (opts.daemon_mode) {
|
char* err, size_t err_size) {
|
||||||
const char* config_path = opts.config_path ? opts.config_path : default_daemon_config_path();
|
const char* config_path = opts->config_path ? opts->config_path : default_daemon_config_path();
|
||||||
g_daemon_conf = daemon_conf_load(config_path, cli_err, sizeof(cli_err));
|
g_daemon_conf = daemon_conf_load(config_path, err, err_size);
|
||||||
if (!g_daemon_conf) {
|
if (!g_daemon_conf) {
|
||||||
server_cli_options_free(&opts);
|
fprintf(stderr, "Error: %s\n", err);
|
||||||
fprintf(stderr, "Error: %s\n", cli_err);
|
return false;
|
||||||
return 1;
|
|
||||||
}
|
}
|
||||||
for (int i = 0; i < opts.dparam_count; i++) {
|
for (int i = 0; i < opts->dparam_count; i++) {
|
||||||
if (daemon_conf_apply_dparam(g_daemon_conf, opts.dparams[i], cli_err, sizeof(cli_err)) != 0) {
|
if (daemon_conf_apply_dparam(g_daemon_conf, opts->dparams[i], err, err_size) != 0) {
|
||||||
fprintf(stderr, "Error: --dparam: %s\n", cli_err);
|
fprintf(stderr, "Error: --dparam: %s\n", err);
|
||||||
exit_code = 1;
|
return false;
|
||||||
goto out;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/* Effective port: -p (highest) > --dparam port > config port (default 873). */
|
/* Effective port: -p (highest) > --dparam port > config port (default 873). */
|
||||||
if (!opts.port_set)
|
if (!opts->port_set)
|
||||||
port = g_daemon_conf->global.port;
|
*port = g_daemon_conf->global.port;
|
||||||
if (!bind_address)
|
if (!*bind_address)
|
||||||
bind_address = g_daemon_conf->global.address;
|
*bind_address = g_daemon_conf->global.address;
|
||||||
if (g_daemon_conf->module_count == 0)
|
if (g_daemon_conf->module_count == 0)
|
||||||
log_message(LOG_LEVEL_WARNING,
|
log_message(LOG_LEVEL_WARNING,
|
||||||
"daemon config has no modules; every connection will be refused");
|
"daemon config has no modules; every connection will be refused");
|
||||||
@@ -1434,20 +1431,23 @@ int main(int argc, char* argv[]) {
|
|||||||
"across all connection children)",
|
"across all connection children)",
|
||||||
g_daemon_conf->modules[i].name, g_daemon_conf->modules[i].max_connections);
|
g_daemon_conf->modules[i].name, g_daemon_conf->modules[i].max_connections);
|
||||||
}
|
}
|
||||||
/* Daemon credential store (Wave B). --password-file and --early-input
|
return true;
|
||||||
* feed the same store, loaded BEFORE the listener forks so every
|
}
|
||||||
* connection child shares one read-only store. Fail closed at startup: a
|
|
||||||
* module that declares `auth users` without a store (or with an empty
|
/* Load the daemon credential store (Wave B) and enforce the fail-closed
|
||||||
* store) refuses to start rather than serving a module whose credentials
|
* startup check: a module that declares `auth users` without a store (or with
|
||||||
* can never be verified. */
|
* an empty store) refuses to start rather than serving a module whose
|
||||||
g_credentials =
|
* credentials can never be verified. On failure the error is printed and
|
||||||
credentials_load(opts.password_file, opts.early_input_file, cli_err, sizeof(cli_err));
|
* false is returned. */
|
||||||
|
static bool validate_daemon_credentials(const ServerCliOptions* opts, char* err, size_t err_size) {
|
||||||
|
/* --password-file and --early-input feed the same store, loaded BEFORE the
|
||||||
|
* listener forks so every connection child shares one read-only store. */
|
||||||
|
g_credentials = credentials_load(opts->password_file, opts->early_input_file, err, err_size);
|
||||||
if (!g_credentials) {
|
if (!g_credentials) {
|
||||||
server_cli_options_free(&opts);
|
fprintf(stderr, "Error: %s\n", err);
|
||||||
fprintf(stderr, "Error: %s\n", cli_err);
|
return false;
|
||||||
return 1;
|
|
||||||
}
|
}
|
||||||
bool credential_source_given = opts.password_file != NULL || opts.early_input_file != NULL;
|
bool credential_source_given = opts->password_file != NULL || opts->early_input_file != NULL;
|
||||||
for (int i = 0; i < g_daemon_conf->module_count; i++) {
|
for (int i = 0; i < g_daemon_conf->module_count; i++) {
|
||||||
const DaemonModule* module = &g_daemon_conf->modules[i];
|
const DaemonModule* module = &g_daemon_conf->modules[i];
|
||||||
if (module->auth_user_count == 0)
|
if (module->auth_user_count == 0)
|
||||||
@@ -1457,16 +1457,14 @@ int main(int argc, char* argv[]) {
|
|||||||
"Error: module '%s' declares 'auth users' but no credential store was given "
|
"Error: module '%s' declares 'auth users' but no credential store was given "
|
||||||
"(--password-file or --early-input); refusing to start (fail closed)\n",
|
"(--password-file or --early-input); refusing to start (fail closed)\n",
|
||||||
module->name);
|
module->name);
|
||||||
server_cli_options_free(&opts);
|
return false;
|
||||||
return 1;
|
|
||||||
}
|
}
|
||||||
if (credentials_store_size(g_credentials) == 0) {
|
if (credentials_store_size(g_credentials) == 0) {
|
||||||
fprintf(stderr,
|
fprintf(stderr,
|
||||||
"Error: module '%s' declares 'auth users' but the credential store is empty; "
|
"Error: module '%s' declares 'auth users' but the credential store is empty; "
|
||||||
"refusing to start (fail closed)\n",
|
"refusing to start (fail closed)\n",
|
||||||
module->name);
|
module->name);
|
||||||
server_cli_options_free(&opts);
|
return false;
|
||||||
return 1;
|
|
||||||
}
|
}
|
||||||
for (int j = 0; j < module->auth_user_count; j++) {
|
for (int j = 0; j < module->auth_user_count; j++) {
|
||||||
if (!credentials_store_has(g_credentials, module->auth_users[j]))
|
if (!credentials_store_has(g_credentials, module->auth_users[j]))
|
||||||
@@ -1476,32 +1474,30 @@ int main(int argc, char* argv[]) {
|
|||||||
module->name, module->auth_users[j]);
|
module->name, module->auth_users[j]);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/* Shared cross-process registry for the per-module / per-source caps and
|
return true;
|
||||||
* the auth lockout. Created HERE in the parent before any accept-loop
|
}
|
||||||
* fork; every connection child inherits the mapping. A failure degrades to
|
|
||||||
* "registry disabled" (the global cap and host ACLs still apply) rather
|
/* Create the shared cross-process registry for the per-module / per-source
|
||||||
* than refusing to start. */
|
* caps and the auth lockout. Called in the parent before any accept-loop fork;
|
||||||
g_daemon_limits = daemon_limits_create((int)g_daemon_conf->global.max_connections,
|
* every connection child inherits the mapping. A failure degrades to
|
||||||
g_daemon_conf->module_count,
|
* "registry disabled" (the global cap and host ACLs still apply) rather than
|
||||||
g_daemon_conf->global.max_connections_per_host,
|
* refusing to start. */
|
||||||
g_daemon_conf->global.auth_lockout_threshold,
|
static void create_daemon_limits(void) {
|
||||||
|
g_daemon_limits = daemon_limits_create(
|
||||||
|
(int)g_daemon_conf->global.max_connections, g_daemon_conf->module_count,
|
||||||
|
g_daemon_conf->global.max_connections_per_host, g_daemon_conf->global.auth_lockout_threshold,
|
||||||
g_daemon_conf->global.auth_lockout_duration_sec);
|
g_daemon_conf->global.auth_lockout_duration_sec);
|
||||||
if (!g_daemon_limits)
|
if (!g_daemon_limits)
|
||||||
log_message(LOG_LEVEL_WARNING,
|
log_message(LOG_LEVEL_WARNING,
|
||||||
"daemon: could not allocate the shared connection registry; per-module / "
|
"daemon: could not allocate the shared connection registry; per-module / "
|
||||||
"per-host caps and the cross-process auth lockout are disabled (the global "
|
"per-host caps and the cross-process auth lockout are disabled (the global "
|
||||||
"'max connections' cap and host ACLs still apply)");
|
"'max connections' cap and host ACLs still apply)");
|
||||||
} else {
|
}
|
||||||
if (!configure_authorization(opts.destination_root)) {
|
|
||||||
char* escaped = output_escape(opts.destination_root, false);
|
|
||||||
fprintf(stderr, "Error: invalid destination root '%s'\n",
|
|
||||||
escaped ? escaped : "<allocation failed>");
|
|
||||||
free(escaped);
|
|
||||||
server_cli_options_free(&opts);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
|
/* Bind the listener, apply the daemon caps, set up TLS when requested, detach
|
||||||
|
* when daemonizing, and run the accept loop. Returns the process exit code. */
|
||||||
|
static int start_listener(ServerCliOptions* opts, int port, int bind_family,
|
||||||
|
const char* bind_address) {
|
||||||
ServerBindOptions bind_opts;
|
ServerBindOptions bind_opts;
|
||||||
bind_opts.bind_address = bind_address;
|
bind_opts.bind_address = bind_address;
|
||||||
bind_opts.family = bind_family;
|
bind_opts.family = bind_family;
|
||||||
@@ -1509,50 +1505,73 @@ int main(int argc, char* argv[]) {
|
|||||||
if (!g_server) {
|
if (!g_server) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to create server");
|
log_message(LOG_LEVEL_ERROR, "Failed to create server");
|
||||||
release_authorization();
|
release_authorization();
|
||||||
exit_code = 1;
|
return 1;
|
||||||
goto out;
|
|
||||||
}
|
}
|
||||||
if (g_daemon_conf)
|
if (g_daemon_conf)
|
||||||
server_set_max_connections(g_server, (unsigned int)g_daemon_conf->global.max_connections);
|
server_set_max_connections(g_server, (unsigned int)g_daemon_conf->global.max_connections);
|
||||||
if (g_daemon_limits)
|
if (g_daemon_limits)
|
||||||
server_set_limit_registry(g_server, g_daemon_limits);
|
server_set_limit_registry(g_server, g_daemon_limits);
|
||||||
if (opts.use_tls) {
|
if (opts->use_tls) {
|
||||||
if (!opts.tls_cert || !opts.tls_key || !opts.tls_ca || !opts.client_cn) {
|
if (!opts->tls_cert || !opts->tls_key || !opts->tls_ca || !opts->client_cn) {
|
||||||
fprintf(stderr, "Error: --tls requires --cert, --key, --ca, and --client-cn\n");
|
fprintf(stderr, "Error: --tls requires --cert, --key, --ca, and --client-cn\n");
|
||||||
server_delete(&g_server);
|
server_delete(&g_server);
|
||||||
release_authorization();
|
release_authorization();
|
||||||
exit_code = 1;
|
return 1;
|
||||||
goto out;
|
|
||||||
}
|
}
|
||||||
tls_global_init();
|
tls_global_init();
|
||||||
if (!server_create_tls(g_server, opts.tls_cert, opts.tls_key, opts.tls_ca)) {
|
if (!server_create_tls(g_server, opts->tls_cert, opts->tls_key, opts->tls_ca)) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to set up TLS");
|
log_message(LOG_LEVEL_ERROR, "Failed to set up TLS");
|
||||||
server_delete(&g_server);
|
server_delete(&g_server);
|
||||||
release_authorization();
|
release_authorization();
|
||||||
exit_code = 1;
|
return 1;
|
||||||
goto out;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Detach after the listening socket (and TLS context) exist so the
|
/* Detach after the listening socket (and TLS context) exist so the
|
||||||
* background daemon inherits a fully-bound listener. --no-detach runs in
|
* background daemon inherits a fully-bound listener. --no-detach runs in
|
||||||
* the foreground, which is how tests drive the daemon. */
|
* the foreground, which is how tests drive the daemon. */
|
||||||
if (opts.daemon_mode && !opts.no_detach) {
|
if (opts->daemon_mode && !opts->no_detach) {
|
||||||
if (!daemonize()) {
|
if (!daemonize()) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to daemonize");
|
log_message(LOG_LEVEL_ERROR, "Failed to daemonize");
|
||||||
server_delete(&g_server);
|
server_delete(&g_server);
|
||||||
release_authorization();
|
release_authorization();
|
||||||
exit_code = 1;
|
return 1;
|
||||||
goto out;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (opts->use_tls)
|
||||||
if (opts.use_tls)
|
|
||||||
server_listen_tls(g_server, handler);
|
server_listen_tls(g_server, handler);
|
||||||
else
|
else
|
||||||
server_listen(g_server, handler);
|
server_listen(g_server, handler);
|
||||||
server_delete(&g_server);
|
server_delete(&g_server);
|
||||||
release_authorization();
|
release_authorization();
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Listener entrypoint: the daemon (config-driven, possibly detached) and the
|
||||||
|
* standalone TCP server share the same bind/TLS/listen path. Consumes and
|
||||||
|
* frees opts. */
|
||||||
|
static int run_daemon_server(ServerCliOptions* opts) {
|
||||||
|
int port = opts->port;
|
||||||
|
const char* bind_address = opts->bind_address;
|
||||||
|
char cli_err[512];
|
||||||
|
int exit_code = 0;
|
||||||
|
|
||||||
|
if (opts->daemon_mode) {
|
||||||
|
if (!load_daemon_policy(opts, &port, &bind_address, cli_err, sizeof(cli_err)) ||
|
||||||
|
!validate_daemon_credentials(opts, cli_err, sizeof(cli_err))) {
|
||||||
|
exit_code = 1;
|
||||||
|
goto out;
|
||||||
|
}
|
||||||
|
create_daemon_limits();
|
||||||
|
} else if (!configure_authorization(opts->destination_root)) {
|
||||||
|
char* escaped = output_escape(opts->destination_root, false);
|
||||||
|
fprintf(stderr, "Error: invalid destination root '%s'\n",
|
||||||
|
escaped ? escaped : "<allocation failed>");
|
||||||
|
free(escaped);
|
||||||
|
exit_code = 1;
|
||||||
|
goto out;
|
||||||
|
}
|
||||||
|
|
||||||
|
exit_code = start_listener(opts, port, opts->bind_family, bind_address);
|
||||||
|
|
||||||
out:
|
out:
|
||||||
daemon_limits_destroy(g_daemon_limits);
|
daemon_limits_destroy(g_daemon_limits);
|
||||||
@@ -1561,7 +1580,35 @@ out:
|
|||||||
g_daemon_conf = NULL;
|
g_daemon_conf = NULL;
|
||||||
credentials_free(g_credentials);
|
credentials_free(g_credentials);
|
||||||
g_credentials = NULL;
|
g_credentials = NULL;
|
||||||
server_cli_options_free(&opts);
|
server_cli_options_free(opts);
|
||||||
return exit_code;
|
return exit_code;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int main(int argc, char* argv[]) {
|
||||||
|
/* Capture the process umask now, while still single-threaded: the cached
|
||||||
|
* value is what file_mode_base() uses, and reading it later would race with
|
||||||
|
* receiver threads creating files. */
|
||||||
|
file_umask_capture();
|
||||||
|
ServerCliOptions opts;
|
||||||
|
char cli_err[512];
|
||||||
|
int parse_result = server_cli_parse(argc, argv, &opts, cli_err, sizeof(cli_err));
|
||||||
|
if (parse_result == 1) {
|
||||||
|
print_server_usage();
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
if (parse_result < 0) {
|
||||||
|
server_cli_options_free(&opts);
|
||||||
|
fprintf(stderr, "Error: %s\n", cli_err);
|
||||||
|
print_server_usage();
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (opts.hash_credentials_file)
|
||||||
|
return run_hash_credentials_tool(&opts);
|
||||||
|
|
||||||
|
configure_server_process(&opts);
|
||||||
|
if (opts.stdio_mode)
|
||||||
|
return run_stdio_server(&opts);
|
||||||
|
return run_daemon_server(&opts);
|
||||||
|
}
|
||||||
#endif
|
#endif
|
||||||
@@ -3,6 +3,7 @@
|
|||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
|
#include <errno.h>
|
||||||
#include <limits.h>
|
#include <limits.h>
|
||||||
#include <lz4.h>
|
#include <lz4.h>
|
||||||
#include <stdatomic.h>
|
#include <stdatomic.h>
|
||||||
@@ -716,3 +717,379 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
|
|||||||
Data* data_decompress(Data* compressed_data) {
|
Data* data_decompress(Data* compressed_data) {
|
||||||
return data_decompress_limited(compressed_data, MAX_DECOMPRESSED_SIZE);
|
return data_decompress_limited(compressed_data, MAX_DECOMPRESSED_SIZE);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ---- streaming decompression ---- */
|
||||||
|
|
||||||
|
#define STREAM_DECOMPRESS_OUT_CHUNK (256 * 1024)
|
||||||
|
|
||||||
|
struct CompressionStreamDecompressor {
|
||||||
|
CompressionAlgo algo;
|
||||||
|
unsigned long long expected_out;
|
||||||
|
unsigned long long total;
|
||||||
|
int out_fd;
|
||||||
|
unsigned char* out_buf;
|
||||||
|
ZSTD_DCtx* dctx;
|
||||||
|
z_stream zs;
|
||||||
|
bool zs_initialized;
|
||||||
|
bool failed;
|
||||||
|
};
|
||||||
|
|
||||||
|
static bool stream_write_all(int fd, const void* data, size_t size) {
|
||||||
|
const unsigned char* p = data;
|
||||||
|
size_t done = 0;
|
||||||
|
while (done < size) {
|
||||||
|
ssize_t n = write(fd, p + done, size - done);
|
||||||
|
if (n < 0 && errno == EINTR)
|
||||||
|
continue;
|
||||||
|
if (n <= 0)
|
||||||
|
return false;
|
||||||
|
done += (size_t)n;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
CompressionStreamDecompressor*
|
||||||
|
compression_stream_decompressor_create(CompressionAlgo algo, unsigned long long expected_out) {
|
||||||
|
CompressionStreamDecompressor* d = calloc(1, sizeof(*d));
|
||||||
|
if (!d)
|
||||||
|
return NULL;
|
||||||
|
d->algo = algo;
|
||||||
|
d->expected_out = expected_out;
|
||||||
|
d->out_fd = -1;
|
||||||
|
d->out_buf = malloc(STREAM_DECOMPRESS_OUT_CHUNK);
|
||||||
|
if (!d->out_buf) {
|
||||||
|
free(d);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
if (algo == COMPRESSION_ALGO_ZSTD) {
|
||||||
|
d->dctx = ZSTD_createDCtx();
|
||||||
|
if (!d->dctx) {
|
||||||
|
free(d->out_buf);
|
||||||
|
free(d);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
} else if (algo == COMPRESSION_ALGO_ZLIB || algo == COMPRESSION_ALGO_ZLIBX) {
|
||||||
|
if (inflateInit(&d->zs) != Z_OK) {
|
||||||
|
free(d->out_buf);
|
||||||
|
free(d);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
d->zs_initialized = true;
|
||||||
|
} else if (algo != COMPRESSION_ALGO_NONE) {
|
||||||
|
/* lz4's block format cannot be decompressed incrementally. */
|
||||||
|
free(d->out_buf);
|
||||||
|
free(d);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
return d;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool stream_emit(CompressionStreamDecompressor* d, const void* buf, size_t len) {
|
||||||
|
if (len == 0)
|
||||||
|
return true;
|
||||||
|
if (d->expected_out != 0 && (d->total > d->expected_out || len > d->expected_out - d->total)) {
|
||||||
|
d->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!stream_write_all(d->out_fd, buf, len)) {
|
||||||
|
d->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
d->total += len;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool stream_feed_none(CompressionStreamDecompressor* d, const void* in, size_t in_len,
|
||||||
|
bool* done) {
|
||||||
|
if (!stream_emit(d, in, in_len))
|
||||||
|
return false;
|
||||||
|
/* NONE has no end marker; the caller knows the frame length. */
|
||||||
|
*done = true;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool stream_feed_zstd(CompressionStreamDecompressor* d, const void* in, size_t in_len,
|
||||||
|
bool* done) {
|
||||||
|
ZSTD_inBuffer input = {in, in_len, 0};
|
||||||
|
while (input.pos < input.size) {
|
||||||
|
ZSTD_outBuffer output = {d->out_buf, STREAM_DECOMPRESS_OUT_CHUNK, 0};
|
||||||
|
size_t ret = ZSTD_decompressStream(d->dctx, &output, &input);
|
||||||
|
if (ZSTD_isError(ret)) {
|
||||||
|
d->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!stream_emit(d, d->out_buf, output.pos))
|
||||||
|
return false;
|
||||||
|
if (ret == 0) {
|
||||||
|
*done = true;
|
||||||
|
/* Trailing bytes after a complete frame are malformed; stop consuming. */
|
||||||
|
if (input.pos < input.size) {
|
||||||
|
d->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool stream_feed_zlib(CompressionStreamDecompressor* d, const void* in, size_t in_len,
|
||||||
|
bool* done) {
|
||||||
|
d->zs.next_in = (Bytef*)in;
|
||||||
|
d->zs.avail_in = (uInt)in_len;
|
||||||
|
while (d->zs.avail_in > 0) {
|
||||||
|
d->zs.next_out = d->out_buf;
|
||||||
|
d->zs.avail_out = STREAM_DECOMPRESS_OUT_CHUNK;
|
||||||
|
int rc = inflate(&d->zs, Z_NO_FLUSH);
|
||||||
|
if (rc != Z_OK && rc != Z_STREAM_END && rc != Z_BUF_ERROR) {
|
||||||
|
d->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
size_t produced = STREAM_DECOMPRESS_OUT_CHUNK - d->zs.avail_out;
|
||||||
|
if (!stream_emit(d, d->out_buf, produced))
|
||||||
|
return false;
|
||||||
|
if (rc == Z_STREAM_END) {
|
||||||
|
*done = true;
|
||||||
|
return d->zs.avail_in == 0;
|
||||||
|
}
|
||||||
|
if (rc == Z_BUF_ERROR && produced == 0) {
|
||||||
|
/* Need more input. */
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool compression_stream_decompressor_feed(CompressionStreamDecompressor* d, const void* in,
|
||||||
|
size_t in_len, int out_fd, bool* done) {
|
||||||
|
if (!d || d->failed)
|
||||||
|
return false;
|
||||||
|
d->out_fd = out_fd;
|
||||||
|
if (done)
|
||||||
|
*done = false;
|
||||||
|
switch (d->algo) {
|
||||||
|
case COMPRESSION_ALGO_NONE:
|
||||||
|
return stream_feed_none(d, in, in_len, done);
|
||||||
|
case COMPRESSION_ALGO_ZSTD:
|
||||||
|
return stream_feed_zstd(d, in, in_len, done);
|
||||||
|
case COMPRESSION_ALGO_ZLIB:
|
||||||
|
case COMPRESSION_ALGO_ZLIBX:
|
||||||
|
return stream_feed_zlib(d, in, in_len, done);
|
||||||
|
case COMPRESSION_ALGO_LZ4:
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
d->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
unsigned long long compression_stream_decompressor_total(const CompressionStreamDecompressor* d) {
|
||||||
|
return d ? d->total : 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
void compression_stream_decompressor_destroy(CompressionStreamDecompressor* d) {
|
||||||
|
if (!d)
|
||||||
|
return;
|
||||||
|
if (d->dctx)
|
||||||
|
ZSTD_freeDCtx(d->dctx);
|
||||||
|
if (d->zs_initialized)
|
||||||
|
inflateEnd(&d->zs);
|
||||||
|
free(d->out_buf);
|
||||||
|
free(d);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---- streaming compression ---- */
|
||||||
|
|
||||||
|
struct CompressionStreamCompressor {
|
||||||
|
CompressionAlgo algo;
|
||||||
|
int level;
|
||||||
|
ZSTD_CCtx* cctx;
|
||||||
|
z_stream zs;
|
||||||
|
bool zs_initialized;
|
||||||
|
unsigned char* out_buf;
|
||||||
|
bool failed;
|
||||||
|
};
|
||||||
|
|
||||||
|
bool compression_stream_compress_supported(CompressionAlgo algo) {
|
||||||
|
return algo == COMPRESSION_ALGO_ZSTD || algo == COMPRESSION_ALGO_ZLIB ||
|
||||||
|
algo == COMPRESSION_ALGO_ZLIBX;
|
||||||
|
}
|
||||||
|
|
||||||
|
CompressionStreamCompressor* compression_stream_compressor_create(CompressionAlgo algo, int level,
|
||||||
|
int threads) {
|
||||||
|
(void)threads;
|
||||||
|
if (!compression_algo_valid((int)algo) || algo == COMPRESSION_ALGO_LZ4)
|
||||||
|
return NULL;
|
||||||
|
CompressionStreamCompressor* c = calloc(1, sizeof(*c));
|
||||||
|
if (!c)
|
||||||
|
return NULL;
|
||||||
|
c->algo = algo;
|
||||||
|
c->level = level;
|
||||||
|
c->out_buf = malloc(STREAM_DECOMPRESS_OUT_CHUNK);
|
||||||
|
if (!c->out_buf) {
|
||||||
|
free(c);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
if (algo == COMPRESSION_ALGO_ZSTD) {
|
||||||
|
c->cctx = ZSTD_createCCtx();
|
||||||
|
if (!c->cctx) {
|
||||||
|
free(c->out_buf);
|
||||||
|
free(c);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
} else if (algo == COMPRESSION_ALGO_ZLIB || algo == COMPRESSION_ALGO_ZLIBX) {
|
||||||
|
if (deflateInit(&c->zs, level < 1 ? Z_DEFAULT_COMPRESSION : level) != Z_OK) {
|
||||||
|
free(c->out_buf);
|
||||||
|
free(c);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
c->zs_initialized = true;
|
||||||
|
}
|
||||||
|
return c;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool compression_stream_compressor_begin(CompressionStreamCompressor* c,
|
||||||
|
unsigned long long raw_size, int out_fd) {
|
||||||
|
if (!c || c->failed)
|
||||||
|
return false;
|
||||||
|
unsigned char hdr[1 + 4];
|
||||||
|
size_t hdr_len = 1;
|
||||||
|
hdr[0] = (unsigned char)c->algo;
|
||||||
|
if (c->algo == COMPRESSION_ALGO_ZLIB || c->algo == COMPRESSION_ALGO_ZLIBX) {
|
||||||
|
uint32_t size32 = raw_size > UINT32_MAX ? UINT32_MAX : (uint32_t)raw_size;
|
||||||
|
for (int i = 0; i < 4; i++)
|
||||||
|
hdr[1 + i] = (uint8_t)((size32 >> (8 * i)) & 0xff);
|
||||||
|
hdr_len = 5;
|
||||||
|
}
|
||||||
|
if (c->algo == COMPRESSION_ALGO_ZSTD) {
|
||||||
|
/* Pledge the source size and force the frame content-size field so the
|
||||||
|
receiver can decide whether to stream from the frame header alone. */
|
||||||
|
if (ZSTD_isError(ZSTD_CCtx_setPledgedSrcSize(c->cctx, raw_size)) ||
|
||||||
|
ZSTD_isError(ZSTD_CCtx_setParameter(c->cctx, ZSTD_c_compressionLevel, c->level)) ||
|
||||||
|
ZSTD_isError(ZSTD_CCtx_setParameter(c->cctx, ZSTD_c_contentSizeFlag, 1))) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (ZSTD_isError(ZSTD_CCtx_setParameter(c->cctx, ZSTD_c_checksumFlag, 0))) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!stream_write_all(out_fd, hdr, hdr_len)) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool stream_compress_zlib(CompressionStreamCompressor* c, const void* in, size_t in_len,
|
||||||
|
int out_fd, int flush) {
|
||||||
|
c->zs.next_in = (Bytef*)in;
|
||||||
|
c->zs.avail_in = (uInt)in_len;
|
||||||
|
do {
|
||||||
|
c->zs.next_out = c->out_buf;
|
||||||
|
c->zs.avail_out = STREAM_DECOMPRESS_OUT_CHUNK;
|
||||||
|
int rc = deflate(&c->zs, flush);
|
||||||
|
if (rc != Z_OK && rc != Z_STREAM_END && rc != Z_BUF_ERROR) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
size_t produced = STREAM_DECOMPRESS_OUT_CHUNK - c->zs.avail_out;
|
||||||
|
if (!stream_write_all(out_fd, c->out_buf, produced)) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (rc == Z_STREAM_END)
|
||||||
|
return true;
|
||||||
|
if (rc == Z_BUF_ERROR && produced == 0)
|
||||||
|
break;
|
||||||
|
} while (c->zs.avail_in > 0 || flush == Z_FINISH);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool compression_stream_compressor_feed(CompressionStreamCompressor* c, const void* in,
|
||||||
|
size_t in_len, int out_fd) {
|
||||||
|
if (!c || c->failed)
|
||||||
|
return false;
|
||||||
|
if (c->algo == COMPRESSION_ALGO_NONE)
|
||||||
|
return stream_write_all(out_fd, in, in_len);
|
||||||
|
if (c->algo == COMPRESSION_ALGO_ZSTD) {
|
||||||
|
ZSTD_inBuffer input = {in, in_len, 0};
|
||||||
|
while (input.pos < input.size) {
|
||||||
|
ZSTD_outBuffer output = {c->out_buf, STREAM_DECOMPRESS_OUT_CHUNK, 0};
|
||||||
|
size_t ret = ZSTD_compressStream2(c->cctx, &output, &input, ZSTD_e_continue);
|
||||||
|
if (ZSTD_isError(ret)) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!stream_write_all(out_fd, c->out_buf, output.pos)) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (output.pos == 0 && input.pos < input.size)
|
||||||
|
break; /* avoid spinning; zstd buffers the rest internally */
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return stream_compress_zlib(c, in, in_len, out_fd, Z_NO_FLUSH);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool compression_stream_compressor_finish(CompressionStreamCompressor* c, int out_fd) {
|
||||||
|
if (!c || c->failed)
|
||||||
|
return false;
|
||||||
|
if (c->algo == COMPRESSION_ALGO_NONE)
|
||||||
|
return true;
|
||||||
|
if (c->algo == COMPRESSION_ALGO_ZSTD) {
|
||||||
|
size_t ret;
|
||||||
|
do {
|
||||||
|
ZSTD_inBuffer input = {NULL, 0, 0};
|
||||||
|
ZSTD_outBuffer output = {c->out_buf, STREAM_DECOMPRESS_OUT_CHUNK, 0};
|
||||||
|
ret = ZSTD_compressStream2(c->cctx, &output, &input, ZSTD_e_end);
|
||||||
|
if (ZSTD_isError(ret)) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!stream_write_all(out_fd, c->out_buf, output.pos)) {
|
||||||
|
c->failed = true;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
} while (ret > 0);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return stream_compress_zlib(c, NULL, 0, out_fd, Z_FINISH);
|
||||||
|
}
|
||||||
|
|
||||||
|
void compression_stream_compressor_destroy(CompressionStreamCompressor* c) {
|
||||||
|
if (!c)
|
||||||
|
return;
|
||||||
|
if (c->cctx)
|
||||||
|
ZSTD_freeCCtx(c->cctx);
|
||||||
|
if (c->zs_initialized)
|
||||||
|
deflateEnd(&c->zs);
|
||||||
|
free(c->out_buf);
|
||||||
|
free(c);
|
||||||
|
}
|
||||||
|
|
||||||
|
unsigned long long compression_peek_frame_content_size(const void* buf, size_t len) {
|
||||||
|
if (!buf || len < 1)
|
||||||
|
return 0;
|
||||||
|
const uint8_t* p = buf;
|
||||||
|
uint8_t codec = p[0];
|
||||||
|
if (!compression_algo_valid(codec))
|
||||||
|
return 0;
|
||||||
|
if (codec == (uint8_t)COMPRESSION_ALGO_NONE)
|
||||||
|
return len - 1;
|
||||||
|
if (codec == (uint8_t)COMPRESSION_ALGO_ZSTD) {
|
||||||
|
if (len < 2)
|
||||||
|
return 0;
|
||||||
|
unsigned long long size = ZSTD_getFrameContentSize(p + 1, len - 1);
|
||||||
|
if (size == ZSTD_CONTENTSIZE_ERROR || size == ZSTD_CONTENTSIZE_UNKNOWN)
|
||||||
|
return 0;
|
||||||
|
return size;
|
||||||
|
}
|
||||||
|
if (len < 1 + LZ4_SIZE_PREFIX_LEN)
|
||||||
|
return 0;
|
||||||
|
uint32_t raw = 0;
|
||||||
|
for (int i = 0; i < LZ4_SIZE_PREFIX_LEN; i++)
|
||||||
|
raw |= (uint32_t)p[1 + i] << (8 * i);
|
||||||
|
return raw;
|
||||||
|
}
|
||||||
@@ -81,6 +81,54 @@ Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
|
|||||||
Data* data_decompress(Data* compressed_data);
|
Data* data_decompress(Data* compressed_data);
|
||||||
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count);
|
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count);
|
||||||
|
|
||||||
|
/* Streaming decompression for a payload too large to hold in memory. The
|
||||||
|
* caller consumes the frame's leading codec byte (and, for lz4/zlib/zlibx, the
|
||||||
|
* 4-byte little-endian raw-size prefix) and then feeds the remaining frame
|
||||||
|
* bytes in bounded chunks; decompressed output is written straight to `out_fd`
|
||||||
|
* so neither the compressed nor the decompressed image is ever materialized.
|
||||||
|
* Only zstd (the default), zlib/zlibx and none support streaming; lz4's block
|
||||||
|
* format is one-shot, so its stream decompressor reports failure and the caller
|
||||||
|
* falls back (the whole-buffer path keeps its existing bound). */
|
||||||
|
typedef struct CompressionStreamDecompressor CompressionStreamDecompressor;
|
||||||
|
|
||||||
|
CompressionStreamDecompressor*
|
||||||
|
compression_stream_decompressor_create(CompressionAlgo algo, unsigned long long expected_out);
|
||||||
|
/* Feed one chunk. Returns false on a malformed frame, an I/O error, or when the
|
||||||
|
* total output would exceed `expected_out` (when non-zero). *done is set once
|
||||||
|
* the frame end has been reached. */
|
||||||
|
bool compression_stream_decompressor_feed(CompressionStreamDecompressor* d, const void* in,
|
||||||
|
size_t in_len, int out_fd, bool* done);
|
||||||
|
unsigned long long compression_stream_decompressor_total(const CompressionStreamDecompressor* d);
|
||||||
|
void compression_stream_decompressor_destroy(CompressionStreamDecompressor* d);
|
||||||
|
|
||||||
|
/* Peek the logical (decompressed) size from the leading bytes of a compressed
|
||||||
|
* frame (codec byte + header), returning 0 when it cannot be determined from
|
||||||
|
* the supplied prefix. Used to decide whether a frame must take the streaming
|
||||||
|
* path before its body is read. */
|
||||||
|
unsigned long long compression_peek_frame_content_size(const void* buf, size_t len);
|
||||||
|
|
||||||
|
/* Streaming compression (sender side). Compresses a source in bounded chunks
|
||||||
|
* into `out_fd` as one self-describing frame (codec byte, the lz4/zlib raw-size
|
||||||
|
* prefix, then the codec stream), so a whole file can be compressed without
|
||||||
|
* materializing it in memory. zstd/zlib/zlibx/none are supported; lz4's block
|
||||||
|
* format is one-shot, so its create() returns NULL and the caller keeps the
|
||||||
|
* buffered path. `raw_size` is the known source length (used for the zlib
|
||||||
|
* prefix and, for zstd, the frame content-size field). */
|
||||||
|
typedef struct CompressionStreamCompressor CompressionStreamCompressor;
|
||||||
|
|
||||||
|
/* True when `algo` can be stream-compressed (zstd/zlib/zlibx; lz4's block format
|
||||||
|
* is one-shot). Used by the sender to decide whether an over-threshold source
|
||||||
|
* may stay unloaded. */
|
||||||
|
bool compression_stream_compress_supported(CompressionAlgo algo);
|
||||||
|
CompressionStreamCompressor* compression_stream_compressor_create(CompressionAlgo algo, int level,
|
||||||
|
int threads);
|
||||||
|
bool compression_stream_compressor_begin(CompressionStreamCompressor* c,
|
||||||
|
unsigned long long raw_size, int out_fd);
|
||||||
|
bool compression_stream_compressor_feed(CompressionStreamCompressor* c, const void* in,
|
||||||
|
size_t in_len, int out_fd);
|
||||||
|
bool compression_stream_compressor_finish(CompressionStreamCompressor* c, int out_fd);
|
||||||
|
void compression_stream_compressor_destroy(CompressionStreamCompressor* c);
|
||||||
|
|
||||||
/* Release the calling thread's cached zstd contexts (compressor, decompressor
|
/* Release the calling thread's cached zstd contexts (compressor, decompressor
|
||||||
* and scratch buffer). The cache is thread-local and is also released
|
* and scratch buffer). The cache is thread-local and is also released
|
||||||
* automatically when a worker thread exits (via a C11 tss destructor) and for
|
* automatically when a worker thread exits (via a C11 tss destructor) and for
|
||||||
|
|||||||
+34
-6
@@ -83,7 +83,7 @@ typedef struct {
|
|||||||
typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF = 2 } SuperMode;
|
typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF = 2 } SuperMode;
|
||||||
|
|
||||||
/* ===========================================================================
|
/* ===========================================================================
|
||||||
* Config wire-field table (single source of truth for protocol 2.28.0).
|
* Config wire-field table (single source of truth for protocol 2.30.0).
|
||||||
*
|
*
|
||||||
* Every field below crosses the wire. The table is the ONLY place a
|
* Every field below crosses the wire. The table is the ONLY place a
|
||||||
* serialized field is named: config.h expands CONFIG_WIRE_FIELDS() to declare
|
* serialized field is named: config.h expands CONFIG_WIRE_FIELDS() to declare
|
||||||
@@ -735,11 +735,13 @@ typedef struct Config {
|
|||||||
* --copy-as) imply it. */
|
* --copy-as) imply it. */
|
||||||
/* fake_super */
|
/* fake_super */
|
||||||
/* --fake-super: receiver-only. When set, each written file additionally gets
|
/* --fake-super: receiver-only. When set, each written file additionally gets
|
||||||
* a reserved user.fastsync.stat xattr recording the RESOLVED uid/gid (the
|
* rsync's reserved user.rsync.%stat xattr recording the RESOLVED uid/gid (the
|
||||||
* source's own when no ownership request is active, else the --chown/--usermap
|
* source's own when no ownership request is active, else the --chown/--usermap
|
||||||
* result) plus mode/mtime so a later privileged restore could re-apply them.
|
* result) plus the full mode and rdev, in rsync 3.4.1's grammar, so the tree is
|
||||||
* It NEVER real-chowns: the point is to record the source ownership on an
|
* interoperable and a later privileged restore could re-apply them. mtime is
|
||||||
* unprivileged receiver. Crosses the wire. */
|
* carried by the file's own timestamp, exactly as rsync does it. It NEVER
|
||||||
|
* real-chowns: the point is to record the source ownership on an unprivileged
|
||||||
|
* receiver. Crosses the wire. */
|
||||||
/* module */
|
/* module */
|
||||||
/* Daemon module selection (Wave A, protocol 2.15.0). Client-composed from a
|
/* Daemon module selection (Wave A, protocol 2.15.0). Client-composed from a
|
||||||
* host::module/path destination; NULL or "" means "no module" (the ordinary
|
* host::module/path destination; NULL or "" means "no module" (the ordinary
|
||||||
@@ -1069,7 +1071,33 @@ typedef struct Config {
|
|||||||
* filter rules so the receiver can protect DESTINATION-ONLY entries from
|
* filter rules so the receiver can protect DESTINATION-ONLY entries from
|
||||||
* --delete with `protect`/`risk` rules (rsync parity). The block appends after
|
* --delete with `protect`/`risk` rules (rsync parity). The block appends after
|
||||||
* compression_algo; see CONFIG_WIRE_PROTECT_FIELDS. */
|
* compression_algo; see CONFIG_WIRE_PROTECT_FIELDS. */
|
||||||
#define PROTOCOL_VERSION "2.28.0"
|
/* (10) Symlink xattrs/ACLs (protocol 2.29.0): the config-frame LAYOUT is
|
||||||
|
* unchanged (the derived use_xattrs bit already crosses the wire), but the
|
||||||
|
* STATUS_SYMLINK frame BODY grows a trailing bounded xattr block when -X/-A is
|
||||||
|
* negotiated -- exactly the block STATUS_MKDIR, STATUS_DIR_TIMES and regular
|
||||||
|
* files already carry. The sender captures the symlink's OWN xattrs with
|
||||||
|
* llistxattr/lgetxattr (so it can never attach the REFERENT's attributes to the
|
||||||
|
* link) and the receiver re-applies them to the link itself with lsetxattr on a
|
||||||
|
* confined /proc/self/fd/<parent>/<leaf> path (there is no *at xattr syscall and
|
||||||
|
* fsetxattr cannot target a symlink). A 2.28 peer that does not consume the new
|
||||||
|
* trailing block would desynchronize after every symlink, so the protocol
|
||||||
|
* version must bump; the strict same-version handshake (config_receive rejects a
|
||||||
|
* mismatched version before parsing anything else) keeps a 2.29 client and a
|
||||||
|
* 2.28 server from ever reaching that state. */
|
||||||
|
/* (11) Client-message channel + partial exit (protocol 2.30.0): the
|
||||||
|
* config-frame LAYOUT is unchanged (no new config field), but the frame stream
|
||||||
|
* gains two statuses. STATUS_CLIENT_MSG (client->server) carries a bounded,
|
||||||
|
* length-prefixed diagnostic string so a client running with --stderr=client
|
||||||
|
* (rsync's --no-msgs2stderr spelling) can forward its own diagnostics to the
|
||||||
|
* server's stderr. STATUS_PARTIAL (receiver->client) is the terminal status
|
||||||
|
* sent instead of STATUS_OK when a per-entry receiver failure (e.g. an
|
||||||
|
* unprivileged --devices mknod) did not abort the stream; the sender exits 23
|
||||||
|
* (rsync's partial transfer) and still removes successfully transferred
|
||||||
|
* --remove-source-files sources. A 2.29 peer that does not know these status
|
||||||
|
* values would reject them as an unknown status and tear the connection down,
|
||||||
|
* so the protocol version must bump; the strict same-version handshake keeps a
|
||||||
|
* 2.30 client and a 2.29 server from ever reaching that state. */
|
||||||
|
#define PROTOCOL_VERSION "2.30.0"
|
||||||
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
|
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
|
||||||
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
|
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
|
||||||
#define MAX_BASIS_DIRS 64
|
#define MAX_BASIS_DIRS 64
|
||||||
|
|||||||
+216
-1
@@ -1,5 +1,6 @@
|
|||||||
#include "daemon_conf.h"
|
#include "daemon_conf.h"
|
||||||
#include "credentials.h"
|
#include "credentials.h"
|
||||||
|
#include "log.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include <arpa/inet.h>
|
#include <arpa/inet.h>
|
||||||
#include <ctype.h>
|
#include <ctype.h>
|
||||||
@@ -33,6 +34,158 @@ static bool key_equals(const char* key, const char* canonical) {
|
|||||||
return strcasecmp(key, canonical) == 0;
|
return strcasecmp(key, canonical) == 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* True when `key` matches one of the NUL-terminated names in `list`. */
|
||||||
|
static bool key_in_list(const char* key, const char* const* list, size_t count) {
|
||||||
|
for (size_t i = 0; i < count; i++) {
|
||||||
|
if (strcasecmp(key, list[i]) == 0)
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* rsync 3.4.1 rsyncd.conf GLOBAL keys accepted in the pre-module section that
|
||||||
|
* have no FastSync equivalent. They are recognized and documented as inert:
|
||||||
|
* accepting a real rsync config must not fail on a logging/process key, but a
|
||||||
|
* silently-reinterpreted key is never invented. `pidfile`/`logfile` are the
|
||||||
|
* compact --dparam spellings rsync documents. The same list is used by the
|
||||||
|
* `--dparam` dispatch (apply_global_key), so there is a single impl. */
|
||||||
|
static const char* const kRsyncInertGlobalKeys[] = {
|
||||||
|
"pid file",
|
||||||
|
"pidfile",
|
||||||
|
"log file",
|
||||||
|
"logfile",
|
||||||
|
"socket options",
|
||||||
|
"sockopts",
|
||||||
|
"listen backlog",
|
||||||
|
"syslog facility",
|
||||||
|
"syslog tag",
|
||||||
|
"log format",
|
||||||
|
"use chroot",
|
||||||
|
"uid",
|
||||||
|
"gid",
|
||||||
|
"timeout",
|
||||||
|
"max verbosity",
|
||||||
|
"min verbosity",
|
||||||
|
"lock file",
|
||||||
|
"transfer logging",
|
||||||
|
"strict modes",
|
||||||
|
"reverse lookup",
|
||||||
|
"forward lookup",
|
||||||
|
"ignore errors",
|
||||||
|
"ignore nonreadable",
|
||||||
|
"dont compress",
|
||||||
|
};
|
||||||
|
|
||||||
|
/* rsync 3.4.1 rsyncd.conf MODULE keys accepted in a [module] section that have
|
||||||
|
* no FastSync equivalent (accepted-and-documented inert). Keys with a FastSync
|
||||||
|
* meaning (`path`, `read only`, `write only`, `auth users`, `max connections`,
|
||||||
|
* `hosts allow`/`hosts deny`, `client owner`) are handled by apply_module_key
|
||||||
|
* before this list is consulted. Security-relevant keys (`exclude`, `filter`,
|
||||||
|
* `secrets file`, `refuse options`, ...) are inert, so a daemon-side filter or
|
||||||
|
* rsync secrets file is NOT enforced: each is loudly warned about at load time
|
||||||
|
* (see kRsyncUnenforcedModuleSecurityKeys) and documented as a residual in
|
||||||
|
* RSYNC_COMPAT.md. */
|
||||||
|
static const char* const kRsyncInertModuleKeys[] = {
|
||||||
|
"comment",
|
||||||
|
"use chroot",
|
||||||
|
"daemon chroot",
|
||||||
|
"uid",
|
||||||
|
"gid",
|
||||||
|
"daemon uid",
|
||||||
|
"daemon gid",
|
||||||
|
"exclude",
|
||||||
|
"include",
|
||||||
|
"exclude from",
|
||||||
|
"include from",
|
||||||
|
"filter",
|
||||||
|
"max verbosity",
|
||||||
|
"min verbosity",
|
||||||
|
"lock file",
|
||||||
|
"transfer logging",
|
||||||
|
"log file",
|
||||||
|
"log format",
|
||||||
|
"syslog facility",
|
||||||
|
"syslog tag",
|
||||||
|
"timeout",
|
||||||
|
"secrets file",
|
||||||
|
"auth digest",
|
||||||
|
"strict modes",
|
||||||
|
"numeric ids",
|
||||||
|
"fake super",
|
||||||
|
"munge symlinks",
|
||||||
|
"list",
|
||||||
|
"dont compress",
|
||||||
|
"charset",
|
||||||
|
"refuse options",
|
||||||
|
"incoming chmod",
|
||||||
|
"outgoing chmod",
|
||||||
|
"open noatime",
|
||||||
|
"max size",
|
||||||
|
"min size",
|
||||||
|
"temp dir",
|
||||||
|
"pre-xfer exec",
|
||||||
|
"post-xfer exec",
|
||||||
|
"name converter",
|
||||||
|
"proxy protocol",
|
||||||
|
"proxy protocol hosts",
|
||||||
|
"reverse lookup",
|
||||||
|
"forward lookup",
|
||||||
|
"ignore errors",
|
||||||
|
"ignore nonreadable",
|
||||||
|
};
|
||||||
|
|
||||||
|
/* Subset of the inert rsync keys whose intent is access control (data
|
||||||
|
* visibility, credential source, transfer hooks, daemon privilege), plus the
|
||||||
|
* global keys that shape the daemon's privilege/identity. These load for
|
||||||
|
* rsync-config compatibility, but because FastSync ignores them an operator
|
||||||
|
* migrating a hardened rsyncd.conf must not believe the restriction applies.
|
||||||
|
* The loader emits one LOG_LEVEL_WARNING per occurrence naming the key (and the
|
||||||
|
* module, for a module key). `write only` is deliberately absent: it is mapped
|
||||||
|
* onto writability instead (FastSync is push-only, so a write-only module is
|
||||||
|
* simply writable). */
|
||||||
|
static const char* const kRsyncUnenforcedModuleSecurityKeys[] = {
|
||||||
|
"secrets file",
|
||||||
|
"auth digest",
|
||||||
|
"refuse options",
|
||||||
|
"exclude",
|
||||||
|
"include",
|
||||||
|
"exclude from",
|
||||||
|
"include from",
|
||||||
|
"filter",
|
||||||
|
"max size",
|
||||||
|
"min size",
|
||||||
|
"pre-xfer exec",
|
||||||
|
"post-xfer exec",
|
||||||
|
"incoming chmod",
|
||||||
|
"outgoing chmod",
|
||||||
|
"name converter",
|
||||||
|
"use chroot",
|
||||||
|
"daemon chroot",
|
||||||
|
"uid",
|
||||||
|
"gid",
|
||||||
|
"daemon uid",
|
||||||
|
"daemon gid",
|
||||||
|
"munge symlinks",
|
||||||
|
"fake super",
|
||||||
|
"strict modes",
|
||||||
|
"proxy protocol",
|
||||||
|
"proxy protocol hosts",
|
||||||
|
};
|
||||||
|
|
||||||
|
static const char* const kRsyncUnenforcedGlobalSecurityKeys[] = {
|
||||||
|
"use chroot",
|
||||||
|
"uid",
|
||||||
|
"gid",
|
||||||
|
"strict modes",
|
||||||
|
};
|
||||||
|
|
||||||
|
#define kRsyncInertGlobalCount (sizeof(kRsyncInertGlobalKeys) / sizeof(kRsyncInertGlobalKeys[0]))
|
||||||
|
#define kRsyncInertModuleCount (sizeof(kRsyncInertModuleKeys) / sizeof(kRsyncInertModuleKeys[0]))
|
||||||
|
#define kRsyncUnenforcedModuleSecurityCount \
|
||||||
|
(sizeof(kRsyncUnenforcedModuleSecurityKeys) / sizeof(kRsyncUnenforcedModuleSecurityKeys[0]))
|
||||||
|
#define kRsyncUnenforcedGlobalSecurityCount \
|
||||||
|
(sizeof(kRsyncUnenforcedGlobalSecurityKeys) / sizeof(kRsyncUnenforcedGlobalSecurityKeys[0]))
|
||||||
|
|
||||||
static bool parse_bool_value(const char* value, bool* out) {
|
static bool parse_bool_value(const char* value, bool* out) {
|
||||||
if (strcasecmp(value, "yes") == 0 || strcasecmp(value, "true") == 0 || strcmp(value, "1") == 0) {
|
if (strcasecmp(value, "yes") == 0 || strcasecmp(value, "true") == 0 || strcmp(value, "1") == 0) {
|
||||||
*out = true;
|
*out = true;
|
||||||
@@ -250,6 +403,10 @@ DaemonConf* daemon_conf_create(void) {
|
|||||||
if (!conf)
|
if (!conf)
|
||||||
return NULL;
|
return NULL;
|
||||||
conf->global.port = DAEMON_CONF_DEFAULT_PORT;
|
conf->global.port = DAEMON_CONF_DEFAULT_PORT;
|
||||||
|
/* rsync modules are READ-ONLY unless `read only = no` (or `write only = yes`)
|
||||||
|
* is set, so FastSync must default the same way: a migrated rsyncd.conf that
|
||||||
|
* omits `read only` is served read-only, never writable. */
|
||||||
|
conf->global.read_only_default = true;
|
||||||
conf->global.max_connections = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS;
|
conf->global.max_connections = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS;
|
||||||
conf->global.auth_failure_delay_ms = DAEMON_CONF_DEFAULT_AUTH_FAILURE_DELAY_MS;
|
conf->global.auth_failure_delay_ms = DAEMON_CONF_DEFAULT_AUTH_FAILURE_DELAY_MS;
|
||||||
conf->global.max_connections_per_host = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS_PER_HOST;
|
conf->global.max_connections_per_host = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS_PER_HOST;
|
||||||
@@ -324,7 +481,7 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
|
|||||||
char* err, size_t err_size) {
|
char* err, size_t err_size) {
|
||||||
if (key_equals(key, "port"))
|
if (key_equals(key, "port"))
|
||||||
return store_port(&conf->global.port, value, err, err_size);
|
return store_port(&conf->global.port, value, err, err_size);
|
||||||
if (key_equals(key, "motd file")) {
|
if (key_equals(key, "motd file") || key_equals(key, "motdfile")) {
|
||||||
if (!store_string(&conf->global.motd_file, value)) {
|
if (!store_string(&conf->global.motd_file, value)) {
|
||||||
set_error(err, err_size, "out of memory parsing 'motd file'");
|
set_error(err, err_size, "out of memory parsing 'motd file'");
|
||||||
return false;
|
return false;
|
||||||
@@ -338,6 +495,25 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
|
|||||||
}
|
}
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
/* rsync allows the `read only` module key in the global section as the
|
||||||
|
* default for modules defined after it. Map it to that default (a later
|
||||||
|
* --dparam re-applies it to modules that did not set their own value) so a
|
||||||
|
* global `read only = yes` cannot be silently dropped into a writable
|
||||||
|
* default. */
|
||||||
|
if (key_equals(key, "read only")) {
|
||||||
|
bool parsed;
|
||||||
|
if (!parse_bool_value(value, &parsed)) {
|
||||||
|
set_error(err, err_size, "global 'read only' must be yes/no (or true/false/1/0), got '%s'",
|
||||||
|
value);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
conf->global.read_only_default = parsed;
|
||||||
|
for (int i = 0; i < conf->module_count; i++) {
|
||||||
|
if (!conf->modules[i].read_only_explicit)
|
||||||
|
conf->modules[i].read_only = parsed;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
if (key_equals(key, "max connections"))
|
if (key_equals(key, "max connections"))
|
||||||
return store_max_connections(&conf->global.max_connections, value, NULL, err, err_size);
|
return store_max_connections(&conf->global.max_connections, value, NULL, err, err_size);
|
||||||
if (key_equals(key, "max connections per host"))
|
if (key_equals(key, "max connections per host"))
|
||||||
@@ -360,6 +536,15 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
|
|||||||
if (key_equals(key, "hosts deny"))
|
if (key_equals(key, "hosts deny"))
|
||||||
return store_host_list(&conf->global.hosts_deny, &conf->global.hosts_deny_count, value,
|
return store_host_list(&conf->global.hosts_deny, &conf->global.hosts_deny_count, value,
|
||||||
"hosts deny", NULL, replace_hosts, err, err_size);
|
"hosts deny", NULL, replace_hosts, err, err_size);
|
||||||
|
/* A recognized rsync global key with no FastSync equivalent loads inert. */
|
||||||
|
if (key_in_list(key, kRsyncInertGlobalKeys, kRsyncInertGlobalCount)) {
|
||||||
|
if (key_in_list(key, kRsyncUnenforcedGlobalSecurityKeys, kRsyncUnenforcedGlobalSecurityCount))
|
||||||
|
log_message(LOG_LEVEL_WARNING,
|
||||||
|
"daemon config: global key '%s' is accepted for rsync compatibility but is NOT "
|
||||||
|
"enforced by FastSync; the restriction it expresses will not be applied",
|
||||||
|
key);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
set_error(err, err_size, "unknown global key '%s'", key);
|
set_error(err, err_size, "unknown global key '%s'", key);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -388,6 +573,26 @@ static bool apply_module_key(DaemonModule* module, char* key, char* value, char*
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
module->read_only = parsed;
|
module->read_only = parsed;
|
||||||
|
module->read_only_explicit = true;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
/* rsync's `write only = yes` makes the module client-writable. FastSync has
|
||||||
|
* no read/pull path, so mapping it to writability is the exact
|
||||||
|
* security-relevant effect; set `read_only_explicit` so a global default
|
||||||
|
* cannot override the module's explicit choice. `write only = no` is the
|
||||||
|
* rsync default and leaves the module's read-only state untouched. */
|
||||||
|
if (key_equals(key, "write only")) {
|
||||||
|
bool parsed;
|
||||||
|
if (!parse_bool_value(value, &parsed)) {
|
||||||
|
set_error(err, err_size,
|
||||||
|
"module '%s': 'write only' must be yes/no (or true/false/1/0), got '%s'",
|
||||||
|
module->name, value);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (parsed) {
|
||||||
|
module->read_only = false;
|
||||||
|
module->read_only_explicit = true;
|
||||||
|
}
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
if (key_equals(key, "client owner")) {
|
if (key_equals(key, "client owner")) {
|
||||||
@@ -457,6 +662,15 @@ static bool apply_module_key(DaemonModule* module, char* key, char* value, char*
|
|||||||
if (key_equals(key, "hosts deny"))
|
if (key_equals(key, "hosts deny"))
|
||||||
return store_host_list(&module->hosts_deny, &module->hosts_deny_count, value, "hosts deny",
|
return store_host_list(&module->hosts_deny, &module->hosts_deny_count, value, "hosts deny",
|
||||||
module->name, false, err, err_size);
|
module->name, false, err, err_size);
|
||||||
|
/* A recognized rsync module key with no FastSync equivalent loads inert. */
|
||||||
|
if (key_in_list(key, kRsyncInertModuleKeys, kRsyncInertModuleCount)) {
|
||||||
|
if (key_in_list(key, kRsyncUnenforcedModuleSecurityKeys, kRsyncUnenforcedModuleSecurityCount))
|
||||||
|
log_message(LOG_LEVEL_WARNING,
|
||||||
|
"daemon config: module '%s' key '%s' is accepted for rsync compatibility but is "
|
||||||
|
"NOT enforced by FastSync; the restriction it expresses will not be applied",
|
||||||
|
module->name, key);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
set_error(err, err_size, "unknown key '%s' in module '%s'", key, module->name);
|
set_error(err, err_size, "unknown key '%s' in module '%s'", key, module->name);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -507,6 +721,7 @@ static int open_module(DaemonConf* conf, int* current_module, const char* name,
|
|||||||
}
|
}
|
||||||
conf->modules = grown;
|
conf->modules = grown;
|
||||||
memset(&conf->modules[conf->module_count], 0, sizeof(DaemonModule));
|
memset(&conf->modules[conf->module_count], 0, sizeof(DaemonModule));
|
||||||
|
conf->modules[conf->module_count].read_only = conf->global.read_only_default;
|
||||||
conf->modules[conf->module_count].name = str_dup(name);
|
conf->modules[conf->module_count].name = str_dup(name);
|
||||||
if (!conf->modules[conf->module_count].name) {
|
if (!conf->modules[conf->module_count].name) {
|
||||||
set_error(err, err_size, "out of memory adding module '%s'", name);
|
set_error(err, err_size, "out of memory adding module '%s'", name);
|
||||||
|
|||||||
@@ -17,7 +17,20 @@
|
|||||||
* DAEMON_CONF_MAX_LINE all fail the whole load with a clear, line-numbered
|
* DAEMON_CONF_MAX_LINE all fail the whole load with a clear, line-numbered
|
||||||
* error instead of being silently ignored. This keeps a typo from silently
|
* error instead of being silently ignored. This keeps a typo from silently
|
||||||
* changing what a module serves.
|
* changing what a module serves.
|
||||||
*/
|
*
|
||||||
|
* rsync compatibility: to reduce the divergence from rsync 3.4.1's rsyncd.conf
|
||||||
|
* grammar, the parser also ACCEPTS the common rsync GLOBAL and MODULE keys.
|
||||||
|
* Keys with a FastSync equivalent are mapped onto it (the native spellings are
|
||||||
|
* unchanged; `read only` defaults to yes like rsync, and `write only = yes`
|
||||||
|
* opts a module into writability). Keys with no FastSync equivalent are
|
||||||
|
* accepted and documented as inert (they load successfully but have no effect)
|
||||||
|
* rather than failing the whole config; the accepted inert set is listed in
|
||||||
|
* kRsyncInertGlobalKeys / kRsyncInertModuleKeys in daemon_conf.c and in
|
||||||
|
* RSYNC_COMPAT.md. Every inert key whose intent is access control is loudly
|
||||||
|
* warned about at load time (kRsyncUnenforced*SecurityKeys) so an operator
|
||||||
|
* migrating a hardened rsyncd.conf is never misled into believing the
|
||||||
|
* restriction is enforced. A key outside both the FastSync-native grammar and
|
||||||
|
* the recognized rsync subset is still rejected as unknown. */
|
||||||
|
|
||||||
/* A daemon module's configured root is used exactly like the standalone
|
/* A daemon module's configured root is used exactly like the standalone
|
||||||
* server's --destination-root: the daemon confines every connection that
|
* server's --destination-root: the daemon confines every connection that
|
||||||
@@ -44,7 +57,13 @@
|
|||||||
typedef struct DaemonModule {
|
typedef struct DaemonModule {
|
||||||
char* name; /* module name, as the client requests it */
|
char* name; /* module name, as the client requests it */
|
||||||
char* path; /* module root (daemon-side authorized root) */
|
char* path; /* module root (daemon-side authorized root) */
|
||||||
bool read_only; /* `read only = yes/no`; default no */
|
bool read_only; /* `read only = yes/no`; defaults to the global `read only`
|
||||||
|
default (rsync allows it in the global section), which is
|
||||||
|
itself default YES (rsync modules are read-only unless
|
||||||
|
`read only = no` / `write only = yes` opts in) */
|
||||||
|
bool read_only_explicit; /* set when this module set its own `read only` or
|
||||||
|
`write only = yes`, so a later global default (from a
|
||||||
|
`--dparam read only=`) does not override it */
|
||||||
bool client_owner; /* `client owner = yes/no`; default no. Per-module opt-in
|
bool client_owner; /* `client owner = yes/no`; default no. Per-module opt-in
|
||||||
that lets this module's clients choose ownership
|
that lets this module's clients choose ownership
|
||||||
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/
|
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/
|
||||||
@@ -69,6 +88,10 @@ typedef struct DaemonConfGlobals {
|
|||||||
int port; /* `port`, default DAEMON_CONF_DEFAULT_PORT (873) */
|
int port; /* `port`, default DAEMON_CONF_DEFAULT_PORT (873) */
|
||||||
char* motd_file; /* `motd file`, may be NULL */
|
char* motd_file; /* `motd file`, may be NULL */
|
||||||
char* address; /* `address` (optional bind address), may be NULL */
|
char* address; /* `address` (optional bind address), may be NULL */
|
||||||
|
bool read_only_default; /* global `read only` default for modules defined
|
||||||
|
after it (rsync allows the module key in the
|
||||||
|
global section); default YES to match rsync's
|
||||||
|
read-only modules */
|
||||||
int max_connections; /* `max connections`, default
|
int max_connections; /* `max connections`, default
|
||||||
DAEMON_CONF_DEFAULT_MAX_CONNECTIONS (100) */
|
DAEMON_CONF_DEFAULT_MAX_CONNECTIONS (100) */
|
||||||
int auth_failure_delay_ms; /* `auth failure delay`, milliseconds; default
|
int auth_failure_delay_ms; /* `auth failure delay`, milliseconds; default
|
||||||
@@ -152,10 +175,13 @@ const DaemonModule* daemon_conf_find_module(const DaemonConf* conf, const char*
|
|||||||
bool daemon_module_name_valid(const char* name);
|
bool daemon_module_name_valid(const char* name);
|
||||||
|
|
||||||
/* Parse one --dparam=KEY=VALUE (or "--dparam KEY=VALUE") override string and
|
/* Parse one --dparam=KEY=VALUE (or "--dparam KEY=VALUE") override string and
|
||||||
* apply it to the global keys only. Keys are case-insensitive and limited to
|
* apply it to the global keys only. Keys are case-insensitive and cover the
|
||||||
* the global keys defined by the grammar (port, motd file, address,
|
* global keys defined by the grammar (port, motd file, address, read only,
|
||||||
* max connections, max connections per host, auth failure delay,
|
* max connections, max connections per host, auth failure delay,
|
||||||
* auth lockout threshold, auth lockout duration, hosts allow, hosts deny).
|
* auth lockout threshold, auth lockout duration, hosts allow, hosts deny) plus
|
||||||
|
* the recognized inert rsync global keys and the compact rsync spellings
|
||||||
|
* (`motdfile`, `pidfile`, `logfile`). Applying `read only` sets the global
|
||||||
|
* default and re-applies it to every module that did not set its own value.
|
||||||
* Returns 0 on success, -1 on error (err filled). */
|
* Returns 0 on success, -1 on error (err filled). */
|
||||||
int daemon_conf_apply_dparam(DaemonConf* conf, const char* assignment, char* err, size_t err_size);
|
int daemon_conf_apply_dparam(DaemonConf* conf, const char* assignment, char* err, size_t err_size);
|
||||||
|
|
||||||
|
|||||||
+116
-34
@@ -8,13 +8,49 @@
|
|||||||
#include <errno.h>
|
#include <errno.h>
|
||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
#include <libgen.h>
|
#include <libgen.h>
|
||||||
|
#include <stdatomic.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
#include <sys/file.h>
|
#include <sys/file.h>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
|
#include <time.h>
|
||||||
#include <unistd.h>
|
#include <unistd.h>
|
||||||
|
|
||||||
|
/* Process-wide counter so two staging contexts created in the same process (or
|
||||||
|
within the same clock tick) can never pick the same name. */
|
||||||
|
static unsigned long long delay_updates_next_sequence(void) {
|
||||||
|
static atomic_ullong sequence;
|
||||||
|
return atomic_fetch_add_explicit(&sequence, 1, memory_order_relaxed);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Build the per-run staging directory basename: the reserved prefix plus the
|
||||||
|
pid and an entropy token. A fixed name could collide with a genuine
|
||||||
|
destination entry; the token makes such a collision vanishingly unlikely and,
|
||||||
|
if it ever happens, prepare() refuses to touch the existing directory. */
|
||||||
|
static char* delay_updates_make_staging_name(void) {
|
||||||
|
unsigned long long entropy = 0;
|
||||||
|
int fd = open("/dev/urandom", O_RDONLY | O_CLOEXEC);
|
||||||
|
if (fd >= 0) {
|
||||||
|
ssize_t got = read(fd, &entropy, sizeof(entropy));
|
||||||
|
close(fd);
|
||||||
|
if (got != (ssize_t)sizeof(entropy))
|
||||||
|
entropy = 0;
|
||||||
|
}
|
||||||
|
if (entropy == 0)
|
||||||
|
entropy = ((unsigned long long)time(NULL) << 20) ^ ((unsigned long long)getpid() << 8) ^
|
||||||
|
delay_updates_next_sequence();
|
||||||
|
int length = snprintf(NULL, 0, DELAY_UPDATES_STAGING_DIR ".%ld.%llx", (long)getpid(), entropy);
|
||||||
|
if (length < 0)
|
||||||
|
return NULL;
|
||||||
|
char* name = malloc((size_t)length + 1);
|
||||||
|
if (!name)
|
||||||
|
return NULL;
|
||||||
|
snprintf(name, (size_t)length + 1, DELAY_UPDATES_STAGING_DIR ".%ld.%llx", (long)getpid(),
|
||||||
|
entropy);
|
||||||
|
return name;
|
||||||
|
}
|
||||||
|
|
||||||
DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
|
DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
|
||||||
if (!root_directory)
|
if (!root_directory)
|
||||||
return NULL;
|
return NULL;
|
||||||
@@ -26,8 +62,15 @@ DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
|
|||||||
free(context);
|
free(context);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
context->staging_root = path_cat(root_directory, DELAY_UPDATES_STAGING_DIR);
|
context->staging_name = delay_updates_make_staging_name();
|
||||||
|
if (!context->staging_name) {
|
||||||
|
free(context->root_directory);
|
||||||
|
free(context);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
context->staging_root = path_cat(root_directory, context->staging_name);
|
||||||
if (!context->staging_root) {
|
if (!context->staging_root) {
|
||||||
|
free(context->staging_name);
|
||||||
free(context->root_directory);
|
free(context->root_directory);
|
||||||
free(context);
|
free(context);
|
||||||
return NULL;
|
return NULL;
|
||||||
@@ -39,6 +82,7 @@ DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
|
|||||||
context->lock_fd = -1;
|
context->lock_fd = -1;
|
||||||
if (mtx_init(&context->mutex, mtx_plain) != thrd_success) {
|
if (mtx_init(&context->mutex, mtx_plain) != thrd_success) {
|
||||||
free(context->staging_root);
|
free(context->staging_root);
|
||||||
|
free(context->staging_name);
|
||||||
free(context->root_directory);
|
free(context->root_directory);
|
||||||
free(context);
|
free(context);
|
||||||
return NULL;
|
return NULL;
|
||||||
@@ -54,6 +98,7 @@ void delay_updates_context_destroy(DelayUpdatesContext* context) {
|
|||||||
close(context->lock_fd);
|
close(context->lock_fd);
|
||||||
context->lock_fd = -1;
|
context->lock_fd = -1;
|
||||||
free(context->staging_root);
|
free(context->staging_root);
|
||||||
|
free(context->staging_name);
|
||||||
free(context->root_directory);
|
free(context->root_directory);
|
||||||
for (size_t i = 0; i < context->count; i++) {
|
for (size_t i = 0; i < context->count; i++) {
|
||||||
free(context->entries[i].staged_path);
|
free(context->entries[i].staged_path);
|
||||||
@@ -125,48 +170,81 @@ bool delay_updates_prepare(DelayUpdatesContext* context) {
|
|||||||
return false;
|
return false;
|
||||||
if (context->prepared)
|
if (context->prepared)
|
||||||
return true;
|
return true;
|
||||||
int fd = file_open_private_dir(context->staging_root);
|
/* Create the per-run staging directory with O_EXCL semantics. The name is
|
||||||
if (fd < 0) {
|
unique to this transfer, so if the path already exists it is NOT ours:
|
||||||
|
either a genuine destination entry that happens to share the name or a
|
||||||
|
leftover from another session. Refuse rather than wipe it -- the old
|
||||||
|
fixed-name design could destroy a real destination entry. A crash
|
||||||
|
leftover is never reused (the next run picks a fresh name). */
|
||||||
|
char* leaf = NULL;
|
||||||
|
int parent_fd = file_open_secure_parent(context->staging_root, &leaf, true);
|
||||||
|
if (parent_fd < 0) {
|
||||||
int saved_errno = errno;
|
int saved_errno = errno;
|
||||||
char* escaped = output_escape(context->staging_root, false);
|
char* escaped = output_escape(context->staging_root, false);
|
||||||
log_message(LOG_LEVEL_ERROR, "could not create --delay-updates staging directory '%s': %s",
|
log_message(LOG_LEVEL_ERROR, "could not create --delay-updates staging directory '%s': %s",
|
||||||
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||||
free(escaped);
|
free(escaped);
|
||||||
|
free(leaf);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
/* Hold an exclusive advisory lock on the staging directory for the whole
|
int fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
transfer. The staging directory name is fixed, so two simultaneous
|
if (fd >= 0) {
|
||||||
delayed transfers to the same destination root would otherwise share it
|
close(fd);
|
||||||
and destroy each other's staged files. The lock makes the second session
|
close(parent_fd);
|
||||||
fail cleanly instead of corrupting the first. The lock is released when
|
char* escaped = output_escape(context->staging_root, false);
|
||||||
the context (and its file descriptor) is destroyed. */
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"--delay-updates staging directory '%s' already exists and is not owned by this "
|
||||||
|
"transfer; refusing to overwrite it",
|
||||||
|
escaped ? escaped : "<allocation failed>");
|
||||||
|
free(escaped);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (errno != ENOENT) {
|
||||||
|
int saved_errno = errno;
|
||||||
|
close(parent_fd);
|
||||||
|
char* escaped = output_escape(context->staging_root, false);
|
||||||
|
log_message(LOG_LEVEL_ERROR, "could not open --delay-updates staging directory '%s': %s",
|
||||||
|
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||||
|
free(escaped);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (mkdirat(parent_fd, leaf, 0700) != 0) {
|
||||||
|
int saved_errno = errno;
|
||||||
|
close(parent_fd);
|
||||||
|
char* escaped = output_escape(context->staging_root, false);
|
||||||
|
log_message(LOG_LEVEL_ERROR, "could not create --delay-updates staging directory '%s': %s",
|
||||||
|
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||||
|
free(escaped);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
if (fd < 0) {
|
||||||
|
int saved_errno = errno;
|
||||||
|
char* escaped = output_escape(context->staging_root, false);
|
||||||
|
log_message(LOG_LEVEL_ERROR, "could not open --delay-updates staging directory '%s': %s",
|
||||||
|
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||||
|
free(escaped);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
/* Keep the exclusive advisory lock as defense in depth: the unique name
|
||||||
|
already prevents two sessions from sharing a staging directory, but the
|
||||||
|
lock also catches an improbable same-name collision that raced between the
|
||||||
|
existence check above and the open. */
|
||||||
if (flock(fd, LOCK_EX | LOCK_NB) != 0) {
|
if (flock(fd, LOCK_EX | LOCK_NB) != 0) {
|
||||||
int saved_errno = errno;
|
int saved_errno = errno;
|
||||||
close(fd);
|
close(fd);
|
||||||
if (saved_errno == EWOULDBLOCK || saved_errno == EAGAIN) {
|
|
||||||
char* escaped = output_escape(context->staging_root, false);
|
char* escaped = output_escape(context->staging_root, false);
|
||||||
log_message(LOG_LEVEL_ERROR,
|
|
||||||
"another --delay-updates transfer to '%s' is already in progress; refusing to "
|
|
||||||
"share the staging directory",
|
|
||||||
escaped ? escaped : "<allocation failed>");
|
|
||||||
free(escaped);
|
|
||||||
} else {
|
|
||||||
log_message(LOG_LEVEL_ERROR, "could not lock --delay-updates staging directory '%s': %s",
|
log_message(LOG_LEVEL_ERROR, "could not lock --delay-updates staging directory '%s': %s",
|
||||||
context->staging_root, strerror(saved_errno));
|
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||||
}
|
free(escaped);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
context->lock_fd = fd;
|
context->lock_fd = fd;
|
||||||
/* Only now, with exclusive ownership, wipe leftovers from an interrupted
|
|
||||||
earlier transfer; this can never race with a live session. */
|
|
||||||
bool ok = delay_wipe_dir_fd(fd);
|
|
||||||
if (!ok) {
|
|
||||||
log_message(LOG_LEVEL_ERROR, "could not clear stale --delay-updates staging files under '%s'",
|
|
||||||
context->staging_root);
|
|
||||||
close(context->lock_fd);
|
|
||||||
context->lock_fd = -1;
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
context->prepared = true;
|
context->prepared = true;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
@@ -264,12 +342,16 @@ static bool delay_publish_entry(DelayUpdatesContext* context, const Config* conf
|
|||||||
const StagedFileEntry* entry) {
|
const StagedFileEntry* entry) {
|
||||||
if (!delay_publish_backup(context, config, entry))
|
if (!delay_publish_backup(context, config, entry))
|
||||||
return false;
|
return false;
|
||||||
/* --force: an incoming regular file/symlink may replace a destination
|
/* An incoming regular file/symlink may replace a destination DIRECTORY that
|
||||||
DIRECTORY (possibly non-empty). The immediate-install path handles this in
|
blocks it. rsync removes the blocker recursively when --delete or --force
|
||||||
file_receive; a --delay-updates run stages elsewhere and only discovers the
|
is active (its generator's "make way" deletion), and a --delay-updates run
|
||||||
blocking directory here, so clear it before the rename (rsync's
|
stages elsewhere so it only discovers the blocker here. FastSync's
|
||||||
"could not make way for new regular file" without --force). */
|
immediate-install path clears it too; without --delete/--force a non-empty
|
||||||
if (config && config->force_delete && file_directory_exists_secure(entry->final_path)) {
|
blocker fails the run (rsync's "could not make way for new regular file").
|
||||||
|
use_delete is gated by the server --allow-delete policy, so a client can
|
||||||
|
never use this to bypass deletion authorization. */
|
||||||
|
if (config && (config->force_delete || config->use_delete) &&
|
||||||
|
file_directory_exists_secure(entry->final_path)) {
|
||||||
if (!file_remove_tree_secure(entry->final_path)) {
|
if (!file_remove_tree_secure(entry->final_path)) {
|
||||||
char* escaped = output_escape(entry->final_path, false);
|
char* escaped = output_escape(entry->final_path, false);
|
||||||
log_message(LOG_LEVEL_ERROR, "could not remove destination directory blocking '%s': %s",
|
log_message(LOG_LEVEL_ERROR, "could not remove destination directory blocking '%s': %s",
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ typedef struct {
|
|||||||
shared with the publish/cleanup phase that runs after the threads join. */
|
shared with the publish/cleanup phase that runs after the threads join. */
|
||||||
typedef struct DelayUpdatesContext {
|
typedef struct DelayUpdatesContext {
|
||||||
char* root_directory; /* receive root the staging dir lives under */
|
char* root_directory; /* receive root the staging dir lives under */
|
||||||
|
char* staging_name; /* per-run unique staging dir basename */
|
||||||
char* staging_root; /* root_directory/<staging dir name> */
|
char* staging_root; /* root_directory/<staging dir name> */
|
||||||
mtx_t mutex;
|
mtx_t mutex;
|
||||||
StagedFileEntry* entries;
|
StagedFileEntry* entries;
|
||||||
@@ -33,7 +34,11 @@ typedef struct DelayUpdatesContext {
|
|||||||
int lock_fd; /* advisory exclusive flock held on the staging dir, or -1 */
|
int lock_fd; /* advisory exclusive flock held on the staging dir, or -1 */
|
||||||
} DelayUpdatesContext;
|
} DelayUpdatesContext;
|
||||||
|
|
||||||
/* Name of the private staging subdirectory created under the receive root. */
|
/* Reserved prefix for the private staging subdirectory created under the
|
||||||
|
receive root. The actual directory name is per-run unique (the prefix plus a
|
||||||
|
pid/entropy token) so it can never clobber a genuine destination entry that
|
||||||
|
happens to share the name; the bare prefix is still what a --backup-dir must
|
||||||
|
not collide with. */
|
||||||
#define DELAY_UPDATES_STAGING_DIR ".fastsync-stage"
|
#define DELAY_UPDATES_STAGING_DIR ".fastsync-stage"
|
||||||
|
|
||||||
/* True when `dir` (ignoring a trailing "/") is the reserved staging directory
|
/* True when `dir` (ignoring a trailing "/") is the reserved staging directory
|
||||||
|
|||||||
+309
-188
@@ -32,6 +32,49 @@ static bool keep_is_file(const PathIndex* index, const char* rel_path) {
|
|||||||
return path_index_contains(index, rel_path);
|
return path_index_contains(index, rel_path);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* rsync's receiver-side verdict for one candidate extra: the per-directory
|
||||||
|
* chain first (deepest directory before ancestors), then the command-line base
|
||||||
|
* rules. Either rule set may be absent. */
|
||||||
|
FilterAction delete_protect_verdict(const DeleteProtectRules* protect, const char* rel_path,
|
||||||
|
const char* leaf, bool is_dir) {
|
||||||
|
if (!protect)
|
||||||
|
return FILTER_ACTION_NONE;
|
||||||
|
/* rsync protects its own --backup files from the delete pass: a name ending
|
||||||
|
in the backup suffix is never an extra. Checked before the filter rules so
|
||||||
|
an explicit exclude cannot be bypassed (the suffix is always a shield). */
|
||||||
|
if (protect->backup_suffix && protect->backup_suffix[0] != '\0') {
|
||||||
|
size_t name_len = strlen(leaf);
|
||||||
|
size_t suffix_len = strlen(protect->backup_suffix);
|
||||||
|
if (name_len > suffix_len &&
|
||||||
|
strcmp(leaf + (name_len - suffix_len), protect->backup_suffix) == 0)
|
||||||
|
return FILTER_ACTION_PROTECT;
|
||||||
|
}
|
||||||
|
FilterAction action = filter_dir_rules_apply_side(protect->dir_rules, rel_path, leaf, is_dir);
|
||||||
|
if (action != FILTER_ACTION_NONE)
|
||||||
|
return action;
|
||||||
|
return filter_rules_apply_side(protect->base_rules, rel_path, leaf, is_dir, FILTER_SIDE_RECEIVER);
|
||||||
|
}
|
||||||
|
|
||||||
|
const char* delete_backup_suffix(const Config* config) {
|
||||||
|
if (!config || !config->backup || config->ignore_existing)
|
||||||
|
return NULL;
|
||||||
|
const char* suffix = config->suffix ? config->suffix : "~";
|
||||||
|
if (!suffix[0] || strchr(suffix, '/'))
|
||||||
|
return NULL;
|
||||||
|
return suffix;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Classify a removed entry from its st_mode for the per-type delete counters. */
|
||||||
|
DeleteEntryType delete_entry_type_of_mode(mode_t mode) {
|
||||||
|
if (S_ISDIR(mode))
|
||||||
|
return DELETE_ENTRY_DIR;
|
||||||
|
if (S_ISLNK(mode))
|
||||||
|
return DELETE_ENTRY_LINK;
|
||||||
|
if (S_ISREG(mode))
|
||||||
|
return DELETE_ENTRY_REG;
|
||||||
|
return DELETE_ENTRY_SPECIAL;
|
||||||
|
}
|
||||||
|
|
||||||
/* True when child_rel is, or lies below, a protected entry. A prefix "a"
|
/* True when child_rel is, or lies below, a protected entry. A prefix "a"
|
||||||
therefore protects "a" and "a/b/c" but not "ab". Entries with top_level_only
|
therefore protects "a" and "a/b/c" but not "ab". Entries with top_level_only
|
||||||
set only protect DIRECT children of the receive root (at_root); nested
|
set only protect DIRECT children of the receive root (at_root); nested
|
||||||
@@ -126,6 +169,7 @@ bool delete_dir_entries_collect(int dirfd, DeleteDirEntry** out, size_t* count,
|
|||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
entries[used].is_dir = S_ISDIR(st.st_mode);
|
entries[used].is_dir = S_ISDIR(st.st_mode);
|
||||||
|
entries[used].mode = st.st_mode;
|
||||||
used++;
|
used++;
|
||||||
}
|
}
|
||||||
closedir(dir);
|
closedir(dir);
|
||||||
@@ -180,6 +224,235 @@ typedef struct {
|
|||||||
void* observer_context; /* DELETE mode */
|
void* observer_context; /* DELETE mode */
|
||||||
} DeleteWalkState;
|
} DeleteWalkState;
|
||||||
|
|
||||||
|
/* The per-walk invariants threaded unchanged through every recursive descent:
|
||||||
|
the keep/synchronized-dir indexes, the destination mode and the protection
|
||||||
|
rules. Bundling them keeps the recursive helpers below to a handful of
|
||||||
|
positional arguments. */
|
||||||
|
typedef struct {
|
||||||
|
const PathIndex* keep;
|
||||||
|
const PathIndex* dirs;
|
||||||
|
DeleteWalkState* state;
|
||||||
|
const DeleteSkipEntry* skips;
|
||||||
|
int skip_count;
|
||||||
|
const DeleteProtectRules* protect;
|
||||||
|
} DeleteWalkContext;
|
||||||
|
|
||||||
|
/* Duplicate `path` with rsync's trailing-slash convention, used to report a
|
||||||
|
removed (or would-be-removed) directory. Returns NULL on allocation
|
||||||
|
failure. */
|
||||||
|
static char* with_trailing_slash(const char* path) {
|
||||||
|
size_t len = strlen(path);
|
||||||
|
char* copy = malloc(len + 2);
|
||||||
|
if (!copy)
|
||||||
|
return NULL;
|
||||||
|
memcpy(copy, path, len);
|
||||||
|
copy[len] = '/';
|
||||||
|
copy[len + 1] = '\0';
|
||||||
|
return copy;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Forward declaration: the ordered passes below recurse through the driver. */
|
||||||
|
static bool delete_walk_fd(int dirfd, const char* rel_path, const DeleteWalkContext* ctx,
|
||||||
|
bool parent_deletable, bool* all_removed);
|
||||||
|
|
||||||
|
/* Descend into the child directory `name` of `dirfd`, walking it as part of the
|
||||||
|
current operation. Returns false on a genuine open/walk failure; on success
|
||||||
|
*child_all_removed reports whether the child removed everything it held (so
|
||||||
|
the caller may rmdir it). */
|
||||||
|
static bool delete_walk_child(int dirfd, const char* name, const char* child_rel,
|
||||||
|
const DeleteWalkContext* ctx, bool deletable,
|
||||||
|
bool* child_all_removed) {
|
||||||
|
*child_all_removed = false;
|
||||||
|
int childfd = openat(dirfd, name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
|
if (childfd < 0)
|
||||||
|
return errno == ENOENT;
|
||||||
|
bool ok = delete_walk_fd(childfd, child_rel, ctx, deletable, child_all_removed);
|
||||||
|
close(childfd);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Classify every entry up front (the verdict does not depend on processing
|
||||||
|
order) so the ordered passes below can act on it. Sets shielded[]/is_extra[]
|
||||||
|
and reports through *local_survives whether anything in this directory stays
|
||||||
|
in place. Returns false on a path-construction failure. */
|
||||||
|
static bool delete_walk_classify(const char* rel_path, const DeleteDirEntry* entries, size_t count,
|
||||||
|
const DeleteWalkContext* ctx, bool deletable, bool at_root,
|
||||||
|
bool* shielded, bool* is_extra, bool* local_survives) {
|
||||||
|
bool ok = true;
|
||||||
|
for (size_t i = 0; i < count; i++) {
|
||||||
|
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||||
|
if (!child_rel) {
|
||||||
|
ok = false;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
/* A --delay-updates run keeps its staging directory as a direct child of
|
||||||
|
the receive root, and basis-dir snapshots live below it too. Their
|
||||||
|
contents are not manifest entries, so descending into them would delete
|
||||||
|
every staged / basis file as an "extra". Only the staging name (a
|
||||||
|
top-level-only prefix) and the basis prefixes are protected: a nested
|
||||||
|
destination directory that happens to be called .fastsync-stage is
|
||||||
|
ordinary content. */
|
||||||
|
if (path_under_skip_prefix(child_rel, at_root, ctx->skips, ctx->skip_count)) {
|
||||||
|
shielded[i] = true;
|
||||||
|
*local_survives = true;
|
||||||
|
} else if (delete_protect_verdict(ctx->protect, child_rel, entries[i].name,
|
||||||
|
entries[i].is_dir) == FILTER_ACTION_PROTECT) {
|
||||||
|
/* A first-match protect rule shields the extra; for a directory the whole
|
||||||
|
subtree is shielded (rsync prunes an excluded directory), so do not
|
||||||
|
descend. */
|
||||||
|
shielded[i] = true;
|
||||||
|
*local_survives = true;
|
||||||
|
} else if (entries[i].is_dir) {
|
||||||
|
bool child_synced = ctx->dirs && path_index_contains(ctx->dirs, child_rel);
|
||||||
|
is_extra[i] = deletable && !child_synced && !keep_is_dir(ctx->keep, child_rel);
|
||||||
|
if (!is_extra[i])
|
||||||
|
*local_survives = true;
|
||||||
|
} else {
|
||||||
|
is_extra[i] = deletable && !keep_is_file(ctx->keep, child_rel);
|
||||||
|
if (!is_extra[i])
|
||||||
|
*local_survives = true;
|
||||||
|
}
|
||||||
|
free(child_rel);
|
||||||
|
}
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Pass 1: extraneous subdirectories, descending. Recurses into each and, when
|
||||||
|
the child removed everything it held, records or removes it and charges the
|
||||||
|
budget. */
|
||||||
|
static bool delete_walk_extra_dirs(int dirfd, const char* rel_path, const DeleteDirEntry* entries,
|
||||||
|
size_t dir_count, const DeleteWalkContext* ctx, bool deletable,
|
||||||
|
const bool* is_extra, bool* local_survives) {
|
||||||
|
bool ok = true;
|
||||||
|
for (size_t i = 0; i < dir_count; i++) {
|
||||||
|
if (!is_extra[i])
|
||||||
|
continue;
|
||||||
|
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||||
|
if (!child_rel) {
|
||||||
|
ok = false;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
bool child_all_removed = false;
|
||||||
|
if (!delete_walk_child(dirfd, entries[i].name, child_rel, ctx, deletable, &child_all_removed))
|
||||||
|
ok = false;
|
||||||
|
if (child_all_removed && deletable) {
|
||||||
|
if (ctx->state->mode == DELETE_WALK_MODE_LIST) {
|
||||||
|
/* Record the directory with rsync's trailing slash. */
|
||||||
|
char* copy = with_trailing_slash(child_rel);
|
||||||
|
if (!copy) {
|
||||||
|
ok = false;
|
||||||
|
} else if (!array_list_add(ctx->state->out, copy)) {
|
||||||
|
free(copy);
|
||||||
|
ok = false;
|
||||||
|
} else {
|
||||||
|
(*ctx->state->recorded)++;
|
||||||
|
}
|
||||||
|
} else if (ctx->state->budget->deleted >= ctx->state->budget->max_delete) {
|
||||||
|
ctx->state->budget->limit_hit = true;
|
||||||
|
ctx->state->budget->skipped++;
|
||||||
|
*local_survives = true;
|
||||||
|
} else if (unlinkat(dirfd, entries[i].name, AT_REMOVEDIR) != 0) {
|
||||||
|
/* ENOENT: already gone (fine). ENOTEMPTY/EEXIST: the directory still
|
||||||
|
holds entries the walker leaves in place (a protected excluded
|
||||||
|
prefix, a kept file the manifest protects, a symlink); rsync leaves
|
||||||
|
such a directory behind, so this is not an error. Only genuine I/O
|
||||||
|
failures abort the deletion. */
|
||||||
|
if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST)
|
||||||
|
ok = false;
|
||||||
|
*local_survives = true;
|
||||||
|
} else {
|
||||||
|
ctx->state->budget->deleted++;
|
||||||
|
/* rsync reports a removed directory with a trailing slash. */
|
||||||
|
if (ctx->state->observer) {
|
||||||
|
char* with_slash = with_trailing_slash(child_rel);
|
||||||
|
if (with_slash) {
|
||||||
|
ctx->state->observer(ctx->state->observer_context, with_slash, DELETE_ENTRY_DIR);
|
||||||
|
free(with_slash);
|
||||||
|
} else {
|
||||||
|
ctx->state->observer(ctx->state->observer_context, child_rel, DELETE_ENTRY_DIR);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
*local_survives = true;
|
||||||
|
}
|
||||||
|
free(child_rel);
|
||||||
|
}
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Pass 2: extraneous files, descending. */
|
||||||
|
static bool delete_walk_extra_files(int dirfd, const char* rel_path, const DeleteDirEntry* entries,
|
||||||
|
size_t dir_count, size_t count, const DeleteWalkContext* ctx,
|
||||||
|
const bool* is_extra, bool* local_survives) {
|
||||||
|
bool ok = true;
|
||||||
|
for (size_t i = dir_count; i < count; i++) {
|
||||||
|
if (!is_extra[i])
|
||||||
|
continue;
|
||||||
|
if (ctx->state->mode == DELETE_WALK_MODE_LIST) {
|
||||||
|
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||||
|
if (!child_rel) {
|
||||||
|
ok = false;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
char* copy = str_dup(child_rel);
|
||||||
|
if (!copy || !array_list_add(ctx->state->out, copy)) {
|
||||||
|
free(copy);
|
||||||
|
ok = false;
|
||||||
|
} else {
|
||||||
|
(*ctx->state->recorded)++;
|
||||||
|
}
|
||||||
|
free(child_rel);
|
||||||
|
} else if (ctx->state->budget->deleted >= ctx->state->budget->max_delete) {
|
||||||
|
ctx->state->budget->limit_hit = true;
|
||||||
|
ctx->state->budget->skipped++;
|
||||||
|
*local_survives = true;
|
||||||
|
} else if (unlinkat(dirfd, entries[i].name, 0) != 0) {
|
||||||
|
if (errno != ENOENT)
|
||||||
|
ok = false;
|
||||||
|
*local_survives = true;
|
||||||
|
} else {
|
||||||
|
ctx->state->budget->deleted++;
|
||||||
|
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||||
|
if (child_rel) {
|
||||||
|
if (ctx->state->observer)
|
||||||
|
ctx->state->observer(ctx->state->observer_context, child_rel,
|
||||||
|
delete_entry_type_of_mode(entries[i].mode));
|
||||||
|
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
|
||||||
|
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
|
||||||
|
free(escaped_path);
|
||||||
|
}
|
||||||
|
free(child_rel);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Pass 3: kept subdirectories, ascending (rsync descends into these only after
|
||||||
|
the parent's own extras have been handled). */
|
||||||
|
static bool delete_walk_kept_dirs(int dirfd, const char* rel_path, const DeleteDirEntry* entries,
|
||||||
|
size_t dir_count, const DeleteWalkContext* ctx, bool deletable,
|
||||||
|
const bool* is_extra, const bool* shielded,
|
||||||
|
bool* local_survives) {
|
||||||
|
bool ok = true;
|
||||||
|
for (size_t i = dir_count; i-- > 0;) {
|
||||||
|
if (is_extra[i] || shielded[i])
|
||||||
|
continue;
|
||||||
|
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||||
|
if (!child_rel) {
|
||||||
|
ok = false;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
bool child_all_removed = false;
|
||||||
|
if (!delete_walk_child(dirfd, entries[i].name, child_rel, ctx, deletable, &child_all_removed))
|
||||||
|
ok = false;
|
||||||
|
/* A kept/synchronized directory is never removed. */
|
||||||
|
*local_survives = true;
|
||||||
|
free(child_rel);
|
||||||
|
}
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
/* Remove the extras directly inside the directory open on `dirfd` (DELETE mode)
|
/* Remove the extras directly inside the directory open on `dirfd` (DELETE mode)
|
||||||
or record the paths that WOULD be removed (LIST mode), recursing into every
|
or record the paths that WOULD be removed (LIST mode), recursing into every
|
||||||
child directory so kept content below a synchronized prefix is reached.
|
child directory so kept content below a synchronized prefix is reached.
|
||||||
@@ -194,11 +467,8 @@ typedef struct {
|
|||||||
descending name order, then extraneous files, then kept subdirectories in
|
descending name order, then extraneous files, then kept subdirectories in
|
||||||
ascending order) rather than readdir() order, so `--max-delete` leaves the
|
ascending order) rather than readdir() order, so `--max-delete` leaves the
|
||||||
same survivors and the `--info=del`/dry-run line order matches rsync. */
|
same survivors and the `--info=del`/dry-run line order matches rsync. */
|
||||||
static bool delete_walk_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
static bool delete_walk_fd(int dirfd, const char* rel_path, const DeleteWalkContext* ctx,
|
||||||
const PathIndex* dirs, DeleteWalkState* state,
|
bool parent_deletable, bool* all_removed) {
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
|
||||||
const FilterRuleList* protect_rules, bool parent_deletable,
|
|
||||||
bool* all_removed) {
|
|
||||||
DeleteDirEntry* entries = NULL;
|
DeleteDirEntry* entries = NULL;
|
||||||
size_t count = 0;
|
size_t count = 0;
|
||||||
bool collect_ok = true;
|
bool collect_ok = true;
|
||||||
@@ -217,7 +487,7 @@ static bool delete_walk_fd(int dirfd, const char* rel_path, const PathIndex* kee
|
|||||||
/* A directory is deletable when it or ANY ancestor is synchronized; the
|
/* A directory is deletable when it or ANY ancestor is synchronized; the
|
||||||
`parent_deletable` flag carries that down the recursion so dest-only
|
`parent_deletable` flag carries that down the recursion so dest-only
|
||||||
directories below a synchronized root are removed wholesale. */
|
directories below a synchronized root are removed wholesale. */
|
||||||
bool deletable = parent_deletable || is_synced_dir(dirs, rel_path);
|
bool deletable = parent_deletable || is_synced_dir(ctx->dirs, rel_path);
|
||||||
bool at_root = rel_path[0] == '\0';
|
bool at_root = rel_path[0] == '\0';
|
||||||
|
|
||||||
/* Reproduce rsync's traversal order: extraneous subdirectories in descending
|
/* Reproduce rsync's traversal order: extraneous subdirectories in descending
|
||||||
@@ -230,182 +500,18 @@ static bool delete_walk_fd(int dirfd, const char* rel_path, const PathIndex* kee
|
|||||||
while (dir_count < count && entries[dir_count].is_dir)
|
while (dir_count < count && entries[dir_count].is_dir)
|
||||||
dir_count++;
|
dir_count++;
|
||||||
|
|
||||||
/* Classify every entry up front (the verdict does not depend on processing
|
if (!delete_walk_classify(rel_path, entries, count, ctx, deletable, at_root, shielded, is_extra,
|
||||||
order) so the ordered passes below can act on it. */
|
&local_survives))
|
||||||
for (size_t i = 0; i < count; i++) {
|
|
||||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
|
||||||
if (!child_rel) {
|
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
continue;
|
if (!delete_walk_extra_dirs(dirfd, rel_path, entries, dir_count, ctx, deletable, is_extra,
|
||||||
}
|
&local_survives))
|
||||||
/* A --delay-updates run keeps its staging directory as a direct child of
|
|
||||||
the receive root, and basis-dir snapshots live below it too. Their
|
|
||||||
contents are not manifest entries, so descending into them would delete
|
|
||||||
every staged / basis file as an "extra". Only the staging name (a
|
|
||||||
top-level-only prefix) and the basis prefixes are protected: a nested
|
|
||||||
destination directory that happens to be called .fastsync-stage is
|
|
||||||
ordinary content. */
|
|
||||||
if (path_under_skip_prefix(child_rel, at_root, skips, skip_count)) {
|
|
||||||
shielded[i] = true;
|
|
||||||
local_survives = true;
|
|
||||||
} else if (protect_rules &&
|
|
||||||
filter_rules_apply_side(protect_rules, child_rel, entries[i].name, entries[i].is_dir,
|
|
||||||
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
|
|
||||||
/* A first-match protect rule shields the extra; for a directory the whole
|
|
||||||
subtree is shielded (rsync prunes an excluded directory), so do not
|
|
||||||
descend. */
|
|
||||||
shielded[i] = true;
|
|
||||||
local_survives = true;
|
|
||||||
} else if (entries[i].is_dir) {
|
|
||||||
bool child_synced = dirs && path_index_contains(dirs, child_rel);
|
|
||||||
is_extra[i] = deletable && !child_synced && !keep_is_dir(keep, child_rel);
|
|
||||||
if (!is_extra[i])
|
|
||||||
local_survives = true;
|
|
||||||
} else {
|
|
||||||
is_extra[i] = deletable && !keep_is_file(keep, child_rel);
|
|
||||||
if (!is_extra[i])
|
|
||||||
local_survives = true;
|
|
||||||
}
|
|
||||||
free(child_rel);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Pass 1: extraneous subdirectories, descending. */
|
|
||||||
for (size_t i = 0; i < dir_count; i++) {
|
|
||||||
if (!is_extra[i])
|
|
||||||
continue;
|
|
||||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
|
||||||
if (!child_rel) {
|
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
continue;
|
if (!delete_walk_extra_files(dirfd, rel_path, entries, dir_count, count, ctx, is_extra,
|
||||||
}
|
&local_survives))
|
||||||
int childfd = openat(dirfd, entries[i].name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
|
||||||
bool child_all_removed = false;
|
|
||||||
if (childfd >= 0) {
|
|
||||||
if (!delete_walk_fd(childfd, child_rel, keep, dirs, state, skips, skip_count, protect_rules,
|
|
||||||
deletable, &child_all_removed))
|
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
close(childfd);
|
if (!delete_walk_kept_dirs(dirfd, rel_path, entries, dir_count, ctx, deletable, is_extra,
|
||||||
} else if (errno != ENOENT) {
|
shielded, &local_survives))
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
}
|
|
||||||
if (child_all_removed && deletable) {
|
|
||||||
if (state->mode == DELETE_WALK_MODE_LIST) {
|
|
||||||
/* Record the directory with rsync's trailing slash. */
|
|
||||||
size_t len = strlen(child_rel);
|
|
||||||
char* copy = malloc(len + 2);
|
|
||||||
if (!copy) {
|
|
||||||
operation_ok = false;
|
|
||||||
} else {
|
|
||||||
memcpy(copy, child_rel, len);
|
|
||||||
copy[len] = '/';
|
|
||||||
copy[len + 1] = '\0';
|
|
||||||
if (!array_list_add(state->out, copy)) {
|
|
||||||
free(copy);
|
|
||||||
operation_ok = false;
|
|
||||||
} else {
|
|
||||||
(*state->recorded)++;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else if (state->budget->deleted >= state->budget->max_delete) {
|
|
||||||
state->budget->limit_hit = true;
|
|
||||||
state->budget->skipped++;
|
|
||||||
local_survives = true;
|
|
||||||
} else if (unlinkat(dirfd, entries[i].name, AT_REMOVEDIR) != 0) {
|
|
||||||
/* ENOENT: already gone (fine). ENOTEMPTY/EEXIST: the directory still
|
|
||||||
holds entries the walker leaves in place (a protected excluded
|
|
||||||
prefix, a kept file the manifest protects, a symlink); rsync leaves
|
|
||||||
such a directory behind, so this is not an error. Only genuine I/O
|
|
||||||
failures abort the deletion. */
|
|
||||||
if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST)
|
|
||||||
operation_ok = false;
|
|
||||||
local_survives = true;
|
|
||||||
} else {
|
|
||||||
state->budget->deleted++;
|
|
||||||
/* rsync reports a removed directory with a trailing slash. */
|
|
||||||
if (state->observer) {
|
|
||||||
size_t len = strlen(child_rel);
|
|
||||||
char* with_slash = malloc(len + 2);
|
|
||||||
if (with_slash) {
|
|
||||||
memcpy(with_slash, child_rel, len);
|
|
||||||
with_slash[len] = '/';
|
|
||||||
with_slash[len + 1] = '\0';
|
|
||||||
state->observer(state->observer_context, with_slash);
|
|
||||||
free(with_slash);
|
|
||||||
} else {
|
|
||||||
state->observer(state->observer_context, child_rel);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
local_survives = true;
|
|
||||||
}
|
|
||||||
free(child_rel);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Pass 2: extraneous files, descending. */
|
|
||||||
for (size_t i = dir_count; i < count; i++) {
|
|
||||||
if (!is_extra[i])
|
|
||||||
continue;
|
|
||||||
if (state->mode == DELETE_WALK_MODE_LIST) {
|
|
||||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
|
||||||
if (!child_rel) {
|
|
||||||
operation_ok = false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
char* copy = str_dup(child_rel);
|
|
||||||
if (!copy || !array_list_add(state->out, copy)) {
|
|
||||||
free(copy);
|
|
||||||
operation_ok = false;
|
|
||||||
} else {
|
|
||||||
(*state->recorded)++;
|
|
||||||
}
|
|
||||||
free(child_rel);
|
|
||||||
} else if (state->budget->deleted >= state->budget->max_delete) {
|
|
||||||
state->budget->limit_hit = true;
|
|
||||||
state->budget->skipped++;
|
|
||||||
local_survives = true;
|
|
||||||
} else if (unlinkat(dirfd, entries[i].name, 0) != 0) {
|
|
||||||
if (errno != ENOENT)
|
|
||||||
operation_ok = false;
|
|
||||||
local_survives = true;
|
|
||||||
} else {
|
|
||||||
state->budget->deleted++;
|
|
||||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
|
||||||
if (child_rel) {
|
|
||||||
if (state->observer)
|
|
||||||
state->observer(state->observer_context, child_rel);
|
|
||||||
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
|
|
||||||
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
|
|
||||||
free(escaped_path);
|
|
||||||
}
|
|
||||||
free(child_rel);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Pass 3: kept subdirectories, ascending (rsync descends into these only
|
|
||||||
after the parent's own extras have been handled). */
|
|
||||||
for (size_t i = dir_count; i-- > 0;) {
|
|
||||||
if (is_extra[i] || shielded[i])
|
|
||||||
continue;
|
|
||||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
|
||||||
if (!child_rel) {
|
|
||||||
operation_ok = false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
int childfd = openat(dirfd, entries[i].name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
|
||||||
bool child_all_removed = false;
|
|
||||||
if (childfd >= 0) {
|
|
||||||
if (!delete_walk_fd(childfd, child_rel, keep, dirs, state, skips, skip_count, protect_rules,
|
|
||||||
deletable, &child_all_removed))
|
|
||||||
operation_ok = false;
|
|
||||||
close(childfd);
|
|
||||||
} else if (errno != ENOENT) {
|
|
||||||
operation_ok = false;
|
|
||||||
}
|
|
||||||
/* A kept/synchronized directory is never removed. */
|
|
||||||
local_survives = true;
|
|
||||||
free(child_rel);
|
|
||||||
}
|
|
||||||
|
|
||||||
free(shielded);
|
free(shielded);
|
||||||
free(is_extra);
|
free(is_extra);
|
||||||
@@ -430,7 +536,7 @@ static int open_destination_root(const char* dest_root) {
|
|||||||
|
|
||||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||||
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out) {
|
const DeleteProtectRules* protect, ArrayList* out, size_t* count_out) {
|
||||||
if (count_out)
|
if (count_out)
|
||||||
*count_out = 0;
|
*count_out = 0;
|
||||||
if (!manifest || !out)
|
if (!manifest || !out)
|
||||||
@@ -460,8 +566,13 @@ bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
|||||||
.recorded = &recorded,
|
.recorded = &recorded,
|
||||||
.observer = NULL,
|
.observer = NULL,
|
||||||
.observer_context = NULL};
|
.observer_context = NULL};
|
||||||
bool ok = delete_walk_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &state, skips, skip_count,
|
DeleteWalkContext ctx = {.keep = &keep,
|
||||||
protect_rules, false, &all_removed);
|
.dirs = have_dirs ? &dirs : NULL,
|
||||||
|
.state = &state,
|
||||||
|
.skips = skips,
|
||||||
|
.skip_count = skip_count,
|
||||||
|
.protect = protect};
|
||||||
|
bool ok = delete_walk_fd(rootfd, "", &ctx, false, &all_removed);
|
||||||
if (close(rootfd) != 0)
|
if (close(rootfd) != 0)
|
||||||
ok = false;
|
ok = false;
|
||||||
path_index_free(&keep);
|
path_index_free(&keep);
|
||||||
@@ -475,7 +586,7 @@ bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
|||||||
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
const FilterRuleList* protect_rules,
|
const DeleteProtectRules* protect,
|
||||||
size_t* deleted_out, size_t* skipped_out,
|
size_t* deleted_out, size_t* skipped_out,
|
||||||
DeletePathObserver observer,
|
DeletePathObserver observer,
|
||||||
void* observer_context) {
|
void* observer_context) {
|
||||||
@@ -513,8 +624,13 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
.recorded = NULL,
|
.recorded = NULL,
|
||||||
.observer = observer,
|
.observer = observer,
|
||||||
.observer_context = observer_context};
|
.observer_context = observer_context};
|
||||||
bool ok = delete_walk_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &state, skips, skip_count,
|
DeleteWalkContext ctx = {.keep = &keep,
|
||||||
protect_rules, false, &all_removed);
|
.dirs = have_dirs ? &dirs : NULL,
|
||||||
|
.state = &state,
|
||||||
|
.skips = skips,
|
||||||
|
.skip_count = skip_count,
|
||||||
|
.protect = protect};
|
||||||
|
bool ok = delete_walk_fd(rootfd, "", &ctx, false, &all_removed);
|
||||||
if (close(rootfd) != 0)
|
if (close(rootfd) != 0)
|
||||||
ok = false;
|
ok = false;
|
||||||
path_index_free(&keep);
|
path_index_free(&keep);
|
||||||
@@ -532,11 +648,10 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
const FilterRuleList* protect_rules, size_t* deleted_out,
|
const DeleteProtectRules* protect, size_t* deleted_out,
|
||||||
size_t* skipped_out) {
|
size_t* skipped_out) {
|
||||||
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
|
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
|
||||||
skip_count, protect_rules, deleted_out, skipped_out, NULL,
|
skip_count, protect, deleted_out, skipped_out, NULL, NULL);
|
||||||
NULL);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
|
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
|
||||||
@@ -607,7 +722,13 @@ bool delete_skips_build(const Config* config, const ArrayList* protected_paths,
|
|||||||
}
|
}
|
||||||
int idx = 0;
|
int idx = 0;
|
||||||
if (config->delay_updates) {
|
if (config->delay_updates) {
|
||||||
out->entries[idx].prefix = DELAY_UPDATES_STAGING_DIR;
|
/* Protect this transfer's actual (per-run unique) staging directory. The
|
||||||
|
runtime name is only known to the receiver-side context; fall back to the
|
||||||
|
reserved prefix for a context that was never created (e.g. a dry run). */
|
||||||
|
const char* staging_name = (config->delay_context && config->delay_context->staging_name)
|
||||||
|
? config->delay_context->staging_name
|
||||||
|
: DELAY_UPDATES_STAGING_DIR;
|
||||||
|
out->entries[idx].prefix = staging_name;
|
||||||
out->entries[idx].top_level_only = true;
|
out->entries[idx].top_level_only = true;
|
||||||
idx++;
|
idx++;
|
||||||
}
|
}
|
||||||
|
|||||||
+54
-7
@@ -3,8 +3,10 @@
|
|||||||
|
|
||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
|
#include "filter.h"
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
#include <stddef.h>
|
#include <stddef.h>
|
||||||
|
#include <sys/stat.h>
|
||||||
|
|
||||||
/* Delete engine.
|
/* Delete engine.
|
||||||
*
|
*
|
||||||
@@ -28,6 +30,34 @@ typedef enum {
|
|||||||
DELETE_WALK_ERROR
|
DELETE_WALK_ERROR
|
||||||
} DeleteWalkResult;
|
} DeleteWalkResult;
|
||||||
|
|
||||||
|
/* Receiver-side delete-protection rules for one walk. `base_rules` is the
|
||||||
|
* command-line rule set the config frame carried (owner "" rules); `dir_rules`
|
||||||
|
* is the received per-directory rule set (rules carrying their owner directory
|
||||||
|
* and no-inherit flag). Either may be NULL. */
|
||||||
|
typedef struct {
|
||||||
|
const FilterRuleList* base_rules;
|
||||||
|
const FilterRuleList* dir_rules;
|
||||||
|
/* When non-NULL and non-empty, a destination entry whose name ends with this
|
||||||
|
suffix is protected from deletion. rsync never treats a --backup file as
|
||||||
|
an extra, so a backup created at --delay-updates publication (or a
|
||||||
|
pre-existing one) survives the delete-after pass. */
|
||||||
|
const char* backup_suffix;
|
||||||
|
} DeleteProtectRules;
|
||||||
|
|
||||||
|
/* rsync's first-match-wins receiver verdict for one candidate extra: the
|
||||||
|
* per-directory chain is evaluated first (the containing directory's rules,
|
||||||
|
* then each ancestor's, then the receive root's), then the base rules. Returns
|
||||||
|
* FILTER_ACTION_PROTECT when the entry is shielded by a receiver-side exclude,
|
||||||
|
* FILTER_ACTION_RISK when an include explicitly leaves it at risk, or
|
||||||
|
* FILTER_ACTION_NONE when no rule matched. */
|
||||||
|
FilterAction delete_protect_verdict(const DeleteProtectRules* protect, const char* rel_path,
|
||||||
|
const char* leaf, bool is_dir);
|
||||||
|
|
||||||
|
/* The backup suffix the delete walker must shield from deletion, or NULL when
|
||||||
|
--backup is inactive or the configured suffix is unusable (empty, or holding
|
||||||
|
a path separator). Matches the suffix file_save uses for backups. */
|
||||||
|
const char* delete_backup_suffix(const Config* config);
|
||||||
|
|
||||||
/* One protected entry for the delete walker. When top_level_only is true the
|
/* One protected entry for the delete walker. When top_level_only is true the
|
||||||
prefix is skipped only as a DIRECT child of dest_root (the --delay-updates
|
prefix is skipped only as a DIRECT child of dest_root (the --delay-updates
|
||||||
staging directory, which must not hide genuine extras inside a nested
|
staging directory, which must not hide genuine extras inside a nested
|
||||||
@@ -66,6 +96,9 @@ bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSki
|
|||||||
typedef struct {
|
typedef struct {
|
||||||
char* name;
|
char* name;
|
||||||
bool is_dir;
|
bool is_dir;
|
||||||
|
/* The entry's full st_mode from the AT_SYMLINK_NOFOLLOW stat, so a delete
|
||||||
|
observer can classify a removed non-directory as reg/link/special. */
|
||||||
|
mode_t mode;
|
||||||
} DeleteDirEntry;
|
} DeleteDirEntry;
|
||||||
/* Collect the entries of the directory open on `dirfd` (excluding "." and ".."),
|
/* Collect the entries of the directory open on `dirfd` (excluding "." and ".."),
|
||||||
stat'ing each with AT_SYMLINK_NOFOLLOW. On success *out is a malloc'd array of
|
stat'ing each with AT_SYMLINK_NOFOLLOW. On success *out is a malloc'd array of
|
||||||
@@ -96,24 +129,38 @@ int delete_dir_entry_cmp_asc(const void* a, const void* b);
|
|||||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
const FilterRuleList* protect_rules, size_t* deleted_out,
|
const DeleteProtectRules* protect, size_t* deleted_out,
|
||||||
size_t* skipped_out);
|
size_t* skipped_out);
|
||||||
|
|
||||||
|
/* Entry kind of a removed path, reported to the delete observer so the receiver
|
||||||
|
can build rsync's `--stats` `Number of deleted files` per-type breakdown. The
|
||||||
|
four categories are a strict partition of every removed entry. */
|
||||||
|
typedef enum {
|
||||||
|
DELETE_ENTRY_REG = 0,
|
||||||
|
DELETE_ENTRY_DIR,
|
||||||
|
DELETE_ENTRY_LINK,
|
||||||
|
DELETE_ENTRY_SPECIAL
|
||||||
|
} DeleteEntryType;
|
||||||
|
|
||||||
/* Optional per-deletion observer: called for each destination-relative path
|
/* Optional per-deletion observer: called for each destination-relative path
|
||||||
actually removed (a file, symlink, or directory), in removal order, so the
|
actually removed (a file, symlink, or directory) with its entry kind, in
|
||||||
receiver can stream rsync's `--info=del`/`--info=remove` lines. */
|
removal order, so the receiver can stream rsync's `--info=del`/`--info=remove`
|
||||||
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
|
lines and tally the per-type `--stats` counters. */
|
||||||
|
typedef void (*DeletePathObserver)(void* context, const char* rel_path, DeleteEntryType type);
|
||||||
|
|
||||||
|
/* Classify a removed entry from its st_mode for the per-type delete counters. */
|
||||||
|
DeleteEntryType delete_entry_type_of_mode(mode_t mode);
|
||||||
|
|
||||||
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
|
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
|
||||||
* observer; the observer is invoked only for entries truly removed. When
|
* observer; the observer is invoked only for entries truly removed. When
|
||||||
* `protect_rules` is non-NULL its receiver-side verdict is evaluated for every
|
* `protect` is non-NULL its receiver-side verdict is evaluated for every
|
||||||
* candidate extra: a first-match PROTECT leaves the entry (and, for a
|
* candidate extra: a first-match PROTECT leaves the entry (and, for a
|
||||||
* directory, its whole subtree) in place, while RISK/NONE fall through to the
|
* directory, its whole subtree) in place, while RISK/NONE fall through to the
|
||||||
* ordinary skip-prefix/keep-set logic. */
|
* ordinary skip-prefix/keep-set logic. */
|
||||||
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
const FilterRuleList* protect_rules,
|
const DeleteProtectRules* protect,
|
||||||
size_t* deleted_out, size_t* skipped_out,
|
size_t* deleted_out, size_t* skipped_out,
|
||||||
DeletePathObserver observer,
|
DeletePathObserver observer,
|
||||||
void* observer_context);
|
void* observer_context);
|
||||||
@@ -124,7 +171,7 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
strings appended to `out` and receives their count in *count_out. */
|
strings appended to `out` and receives their count in *count_out. */
|
||||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||||
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out);
|
const DeleteProtectRules* protect, ArrayList* out, size_t* count_out);
|
||||||
bool delete_extras(const char* dest_root, const ArrayList* manifest);
|
bool delete_extras(const char* dest_root, const ArrayList* manifest);
|
||||||
|
|
||||||
/* Build the delete walk's skip-prefix set from the config's --delay-updates
|
/* Build the delete walk's skip-prefix set from the config's --delay-updates
|
||||||
|
|||||||
+181
-139
@@ -19,6 +19,7 @@
|
|||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "delay_updates.h"
|
#include "delay_updates.h"
|
||||||
#include "delete_commit.h"
|
#include "delete_commit.h"
|
||||||
|
#include "delete_plan.h"
|
||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "format.h"
|
#include "format.h"
|
||||||
@@ -102,6 +103,13 @@ DeleteManifest* receive_manifest_entries(int fd) {
|
|||||||
delete_manifest_free(manifest);
|
delete_manifest_free(manifest);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
|
/* Per-directory filter rules (protocol 2.30.0) follow the manifest sections
|
||||||
|
* with their own bounded self-describing format. */
|
||||||
|
if (!delete_filter_dir_rules_receive(fd, &manifest->per_dir_rules)) {
|
||||||
|
delete_manifest_free(manifest);
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
return manifest;
|
return manifest;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -112,6 +120,7 @@ void delete_manifest_free(DeleteManifest* manifest) {
|
|||||||
array_list_delete(manifest->protected);
|
array_list_delete(manifest->protected);
|
||||||
array_list_delete(manifest->missing);
|
array_list_delete(manifest->missing);
|
||||||
array_list_delete(manifest->dirs);
|
array_list_delete(manifest->dirs);
|
||||||
|
filter_rule_list_free(manifest->per_dir_rules);
|
||||||
free(manifest);
|
free(manifest);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -160,9 +169,12 @@ static bool delete_extras_budgeted_observed(const Config* config, const DeleteMa
|
|||||||
remaining = budget->max_delete - budget->deleted;
|
remaining = budget->max_delete - budget->deleted;
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
size_t skipped = 0;
|
size_t skipped = 0;
|
||||||
|
DeleteProtectRules protect = {.base_rules = config->protect_rules,
|
||||||
|
.dir_rules = manifest->per_dir_rules,
|
||||||
|
.backup_suffix = delete_backup_suffix(config)};
|
||||||
DeleteWalkResult result = delete_extras_limited_observed(
|
DeleteWalkResult result = delete_extras_limited_observed(
|
||||||
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips.entries,
|
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips.entries,
|
||||||
skips.count, config->protect_rules, &deleted, &skipped, observer, observer_context);
|
skips.count, &protect, &deleted, &skipped, observer, observer_context);
|
||||||
delete_skips_free(&skips);
|
delete_skips_free(&skips);
|
||||||
budget->deleted += deleted;
|
budget->deleted += deleted;
|
||||||
budget->skipped += skipped;
|
budget->skipped += skipped;
|
||||||
@@ -191,13 +203,13 @@ typedef struct {
|
|||||||
const char* prefix;
|
const char* prefix;
|
||||||
} PrefixedDeleteObserver;
|
} PrefixedDeleteObserver;
|
||||||
|
|
||||||
static void prefixed_delete_observer(void* context, const char* rel) {
|
static void prefixed_delete_observer(void* context, const char* rel, DeleteEntryType type) {
|
||||||
PrefixedDeleteObserver* prefixed = context;
|
PrefixedDeleteObserver* prefixed = context;
|
||||||
if (!prefixed->inner || !rel)
|
if (!prefixed->inner || !rel)
|
||||||
return;
|
return;
|
||||||
char* joined = path_cat((char*)prefixed->prefix, rel);
|
char* joined = path_cat((char*)prefixed->prefix, rel);
|
||||||
if (joined) {
|
if (joined) {
|
||||||
prefixed->inner(prefixed->inner_context, joined);
|
prefixed->inner(prefixed->inner_context, joined, type);
|
||||||
free(joined);
|
free(joined);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -215,6 +227,166 @@ static void prefixed_delete_observer(void* context, const char* rel) {
|
|||||||
--max-delete budget: once it is exhausted the remaining requests are skipped
|
--max-delete budget: once it is exhausted the remaining requests are skipped
|
||||||
and counted. Returns false only on a genuine error (a confinement failure on
|
and counted. Returns false only on a genuine error (a confinement failure on
|
||||||
a validated path or an I/O error), which fails the run. */
|
a validated path or an I/O error), which fails the run. */
|
||||||
|
|
||||||
|
/* How one missing-args request leaves the driver loop. The original walker
|
||||||
|
`continue`s past an invalid/protected/absent/budget-skipped request (without
|
||||||
|
breaking) but stops after a request that ran to completion while an error is
|
||||||
|
pending; NEXT/STOP preserve that control flow exactly. */
|
||||||
|
typedef enum { MISSING_ARG_NEXT, MISSING_ARG_STOP } MissingArgStep;
|
||||||
|
|
||||||
|
/* Remove a NON-empty missing-args directory recursively (--delete/--force in
|
||||||
|
effect): walk its contents through the budgeted extras walker so every removed
|
||||||
|
file/dir counts toward --max-delete (rsync parity), then remove the now-empty
|
||||||
|
directory itself, which costs one more budget unit. A run that hits the cap
|
||||||
|
leaves the remaining entries in place. The observer is wrapped so the nested
|
||||||
|
walk reports receive-root-relative paths. Sets the *removed and *ok outputs. */
|
||||||
|
static void delete_nonempty_missing_dir(const char* full, const char* rel,
|
||||||
|
DeleteBudgetState* budget, DeletePathObserver observer,
|
||||||
|
void* observer_context, bool* removed, bool* ok) {
|
||||||
|
ArrayList* no_keeps = array_list_create(free);
|
||||||
|
/* Never let an accounting slip (deleted > max_delete) underflow the remaining
|
||||||
|
budget into SIZE_MAX, which would grant unlimited deletions. */
|
||||||
|
size_t remaining =
|
||||||
|
budget->deleted >= budget->max_delete ? 0 : budget->max_delete - budget->deleted;
|
||||||
|
size_t contents_deleted = 0;
|
||||||
|
size_t contents_skipped = 0;
|
||||||
|
PrefixedDeleteObserver nested = {observer, observer_context, rel};
|
||||||
|
DeleteWalkResult walk =
|
||||||
|
no_keeps ? delete_extras_limited_observed(full, no_keeps, NULL, remaining, NULL, 0, NULL,
|
||||||
|
&contents_deleted, &contents_skipped,
|
||||||
|
observer ? prefixed_delete_observer : NULL,
|
||||||
|
observer ? &nested : NULL)
|
||||||
|
: DELETE_WALK_ERROR;
|
||||||
|
if (no_keeps)
|
||||||
|
array_list_delete(no_keeps);
|
||||||
|
budget->deleted += contents_deleted;
|
||||||
|
budget->skipped += contents_skipped;
|
||||||
|
if (walk == DELETE_WALK_LIMIT_REACHED) {
|
||||||
|
budget->limit_hit = true;
|
||||||
|
} else if (walk != DELETE_WALK_OK) {
|
||||||
|
*ok = false;
|
||||||
|
} else if (budget->deleted >= budget->max_delete) {
|
||||||
|
budget->limit_hit = true;
|
||||||
|
budget->skipped++;
|
||||||
|
} else if (file_remove_tree_secure(full)) {
|
||||||
|
/* The shared `if (removed)` tail charges this directory exactly once;
|
||||||
|
counting it here too would consume two budget units. */
|
||||||
|
*removed = true;
|
||||||
|
} else {
|
||||||
|
*ok = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Remove one missing-args destination mirror. `skips` holds the receiver
|
||||||
|
artifacts (staging directory, basis snapshots) that stay protected. Returns
|
||||||
|
MISSING_ARG_STOP when the driver loop must stop (a completed removal left a
|
||||||
|
genuine error pending) and MISSING_ARG_NEXT otherwise; *ok accumulates the
|
||||||
|
overall success across the whole run. */
|
||||||
|
static MissingArgStep delete_one_missing_arg(const Config* config, const char* rel,
|
||||||
|
const DeleteSkipSet* skips, DeleteBudgetState* budget,
|
||||||
|
DeletePathObserver observer, void* observer_context,
|
||||||
|
bool* ok) {
|
||||||
|
if (!rel || *rel == '\0' || *rel == '/' || has_path_traversal(rel)) {
|
||||||
|
/* Defensive only: receive_manifest_entries already validated every
|
||||||
|
section identically, so a controlled peer never reaches this branch. */
|
||||||
|
log_message(LOG_LEVEL_ERROR, "invalid missing-args delete path");
|
||||||
|
*ok = false;
|
||||||
|
return MISSING_ARG_NEXT;
|
||||||
|
}
|
||||||
|
bool at_root = strchr(rel, '/') == NULL;
|
||||||
|
if (path_under_skip_prefix(rel, at_root, skips->entries, skips->count)) {
|
||||||
|
char* escaped = output_escape(rel, log_get_8_bit_output());
|
||||||
|
log_message(LOG_LEVEL_WARNING,
|
||||||
|
"missing-args path '%s' is protected (staging directory or basis snapshot); "
|
||||||
|
"not deleting",
|
||||||
|
escaped ? escaped : "<allocation failed>");
|
||||||
|
free(escaped);
|
||||||
|
return MISSING_ARG_NEXT;
|
||||||
|
}
|
||||||
|
char* full = path_cat(config->receive_root_directory, rel);
|
||||||
|
if (!full) {
|
||||||
|
*ok = false;
|
||||||
|
return MISSING_ARG_NEXT;
|
||||||
|
}
|
||||||
|
char* leaf = NULL;
|
||||||
|
int parent_fd = file_open_secure_parent(full, &leaf, false);
|
||||||
|
if (parent_fd < 0) {
|
||||||
|
/* The mirror's parent directory may itself not exist on the destination
|
||||||
|
(a deeper missing entry whose leading directories were never created).
|
||||||
|
That is a no-op -- there is nothing to delete -- matching
|
||||||
|
file_remove_tree_secure's absent-path handling; only a genuine I/O
|
||||||
|
error (EACCES, a symlink loop, ...) fails the run. */
|
||||||
|
bool absent = errno == ENOENT || errno == ENOTDIR;
|
||||||
|
free(full);
|
||||||
|
free(leaf);
|
||||||
|
if (!absent)
|
||||||
|
*ok = false;
|
||||||
|
return MISSING_ARG_NEXT;
|
||||||
|
}
|
||||||
|
struct stat st;
|
||||||
|
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
||||||
|
/* Already absent: nothing to delete (a no-op, not a deletion). */
|
||||||
|
if (errno != ENOENT)
|
||||||
|
*ok = false;
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
free(full);
|
||||||
|
return MISSING_ARG_NEXT;
|
||||||
|
}
|
||||||
|
/* An entry that exists is one deletion: skip it (and count it) when the
|
||||||
|
shared --max-delete budget is already exhausted. */
|
||||||
|
if (budget->deleted >= budget->max_delete) {
|
||||||
|
budget->limit_hit = true;
|
||||||
|
budget->skipped++;
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
free(full);
|
||||||
|
return MISSING_ARG_NEXT;
|
||||||
|
}
|
||||||
|
bool removed = false;
|
||||||
|
if (S_ISDIR(st.st_mode)) {
|
||||||
|
if (unlinkat(parent_fd, leaf, AT_REMOVEDIR) == 0) {
|
||||||
|
removed = true;
|
||||||
|
} else if (errno == ENOTEMPTY || errno == EEXIST) {
|
||||||
|
close(parent_fd);
|
||||||
|
parent_fd = -1;
|
||||||
|
free(leaf);
|
||||||
|
leaf = NULL;
|
||||||
|
if (config->use_delete || config->force_delete) {
|
||||||
|
delete_nonempty_missing_dir(full, rel, budget, observer, observer_context, &removed, ok);
|
||||||
|
} else {
|
||||||
|
char* escaped = output_escape(rel, log_get_8_bit_output());
|
||||||
|
log_message(LOG_LEVEL_WARNING,
|
||||||
|
"missing-args destination '%s' is a non-empty directory; use --force or "
|
||||||
|
"--delete to remove it",
|
||||||
|
escaped ? escaped : "<allocation failed>");
|
||||||
|
free(escaped);
|
||||||
|
}
|
||||||
|
} else if (errno != ENOENT) {
|
||||||
|
*ok = false;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if (unlinkat(parent_fd, leaf, 0) == 0) {
|
||||||
|
removed = true;
|
||||||
|
} else if (errno != ENOENT) {
|
||||||
|
*ok = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (removed) {
|
||||||
|
budget->deleted++;
|
||||||
|
if (observer)
|
||||||
|
observer(observer_context, rel, delete_entry_type_of_mode(st.st_mode));
|
||||||
|
char* escaped = output_escape(rel, log_get_8_bit_output());
|
||||||
|
fprintf(stderr, " Deleted: %s\n", escaped ? escaped : "<allocation failed>");
|
||||||
|
free(escaped);
|
||||||
|
}
|
||||||
|
if (parent_fd >= 0)
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
free(full);
|
||||||
|
return *ok ? MISSING_ARG_NEXT : MISSING_ARG_STOP;
|
||||||
|
}
|
||||||
|
|
||||||
static bool delete_missing_args_budgeted_observed(const Config* config,
|
static bool delete_missing_args_budgeted_observed(const Config* config,
|
||||||
const DeleteManifest* manifest,
|
const DeleteManifest* manifest,
|
||||||
DeleteBudgetState* budget,
|
DeleteBudgetState* budget,
|
||||||
@@ -234,141 +406,8 @@ static bool delete_missing_args_budgeted_observed(const Config* config,
|
|||||||
bool ok = true;
|
bool ok = true;
|
||||||
for (int i = 0; i < manifest->missing->size; i++) {
|
for (int i = 0; i < manifest->missing->size; i++) {
|
||||||
const char* rel = (const char*)manifest->missing->items[i];
|
const char* rel = (const char*)manifest->missing->items[i];
|
||||||
if (!rel || *rel == '\0' || *rel == '/' || has_path_traversal(rel)) {
|
if (delete_one_missing_arg(config, rel, &skips, budget, observer, observer_context, &ok) ==
|
||||||
/* Defensive only: receive_manifest_entries already validated every
|
MISSING_ARG_STOP)
|
||||||
section identically, so a controlled peer never reaches this branch. */
|
|
||||||
log_message(LOG_LEVEL_ERROR, "invalid missing-args delete path");
|
|
||||||
ok = false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
bool at_root = strchr(rel, '/') == NULL;
|
|
||||||
if (path_under_skip_prefix(rel, at_root, skips.entries, skips.count)) {
|
|
||||||
char* escaped = output_escape(rel, log_get_8_bit_output());
|
|
||||||
log_message(LOG_LEVEL_WARNING,
|
|
||||||
"missing-args path '%s' is protected (staging directory or basis snapshot); "
|
|
||||||
"not deleting",
|
|
||||||
escaped ? escaped : "<allocation failed>");
|
|
||||||
free(escaped);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
char* full = path_cat(config->receive_root_directory, rel);
|
|
||||||
if (!full) {
|
|
||||||
ok = false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
char* leaf = NULL;
|
|
||||||
int parent_fd = file_open_secure_parent(full, &leaf, false);
|
|
||||||
if (parent_fd < 0) {
|
|
||||||
/* The mirror's parent directory may itself not exist on the destination
|
|
||||||
(a deeper missing entry whose leading directories were never created).
|
|
||||||
That is a no-op -- there is nothing to delete -- matching
|
|
||||||
file_remove_tree_secure's absent-path handling; only a genuine I/O
|
|
||||||
error (EACCES, a symlink loop, ...) fails the run. */
|
|
||||||
bool absent = errno == ENOENT || errno == ENOTDIR;
|
|
||||||
free(full);
|
|
||||||
free(leaf);
|
|
||||||
if (!absent)
|
|
||||||
ok = false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
struct stat st;
|
|
||||||
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
|
||||||
/* Already absent: nothing to delete (a no-op, not a deletion). */
|
|
||||||
if (errno != ENOENT)
|
|
||||||
ok = false;
|
|
||||||
close(parent_fd);
|
|
||||||
free(leaf);
|
|
||||||
free(full);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
/* An entry that exists is one deletion: skip it (and count it) when the
|
|
||||||
shared --max-delete budget is already exhausted. */
|
|
||||||
if (budget->deleted >= budget->max_delete) {
|
|
||||||
budget->limit_hit = true;
|
|
||||||
budget->skipped++;
|
|
||||||
close(parent_fd);
|
|
||||||
free(leaf);
|
|
||||||
free(full);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
bool removed = false;
|
|
||||||
if (S_ISDIR(st.st_mode)) {
|
|
||||||
if (unlinkat(parent_fd, leaf, AT_REMOVEDIR) == 0) {
|
|
||||||
removed = true;
|
|
||||||
} else if (errno == ENOTEMPTY || errno == EEXIST) {
|
|
||||||
close(parent_fd);
|
|
||||||
parent_fd = -1;
|
|
||||||
free(leaf);
|
|
||||||
leaf = NULL;
|
|
||||||
if (config->use_delete || config->force_delete) {
|
|
||||||
/* Remove the contents entry-by-entry through the budgeted extras
|
|
||||||
walker so every deleted file/dir counts toward --max-delete (rsync
|
|
||||||
parity); the now-empty directory itself costs one more. A run that
|
|
||||||
hits the cap leaves the remaining entries in place. */
|
|
||||||
ArrayList* no_keeps = array_list_create(free);
|
|
||||||
/* Never let an accounting slip (deleted > max_delete) underflow the
|
|
||||||
remaining budget into SIZE_MAX, which would grant unlimited
|
|
||||||
deletions. */
|
|
||||||
size_t remaining =
|
|
||||||
budget->deleted >= budget->max_delete ? 0 : budget->max_delete - budget->deleted;
|
|
||||||
size_t contents_deleted = 0;
|
|
||||||
size_t contents_skipped = 0;
|
|
||||||
PrefixedDeleteObserver nested = {observer, observer_context, rel};
|
|
||||||
DeleteWalkResult walk =
|
|
||||||
no_keeps ? delete_extras_limited_observed(full, no_keeps, NULL, remaining, NULL, 0,
|
|
||||||
NULL, &contents_deleted, &contents_skipped,
|
|
||||||
observer ? prefixed_delete_observer : NULL,
|
|
||||||
observer ? &nested : NULL)
|
|
||||||
: DELETE_WALK_ERROR;
|
|
||||||
if (no_keeps)
|
|
||||||
array_list_delete(no_keeps);
|
|
||||||
budget->deleted += contents_deleted;
|
|
||||||
budget->skipped += contents_skipped;
|
|
||||||
if (walk == DELETE_WALK_LIMIT_REACHED) {
|
|
||||||
budget->limit_hit = true;
|
|
||||||
} else if (walk != DELETE_WALK_OK) {
|
|
||||||
ok = false;
|
|
||||||
} else if (budget->deleted >= budget->max_delete) {
|
|
||||||
budget->limit_hit = true;
|
|
||||||
budget->skipped++;
|
|
||||||
} else if (file_remove_tree_secure(full)) {
|
|
||||||
/* The shared `if (removed)` tail charges this directory exactly
|
|
||||||
once; counting it here too would consume two budget units. */
|
|
||||||
removed = true;
|
|
||||||
} else {
|
|
||||||
ok = false;
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
char* escaped = output_escape(rel, log_get_8_bit_output());
|
|
||||||
log_message(LOG_LEVEL_WARNING,
|
|
||||||
"missing-args destination '%s' is a non-empty directory; use --force or "
|
|
||||||
"--delete to remove it",
|
|
||||||
escaped ? escaped : "<allocation failed>");
|
|
||||||
free(escaped);
|
|
||||||
}
|
|
||||||
} else if (errno != ENOENT) {
|
|
||||||
ok = false;
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if (unlinkat(parent_fd, leaf, 0) == 0) {
|
|
||||||
removed = true;
|
|
||||||
} else if (errno != ENOENT) {
|
|
||||||
ok = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (removed) {
|
|
||||||
budget->deleted++;
|
|
||||||
if (observer)
|
|
||||||
observer(observer_context, rel);
|
|
||||||
char* escaped = output_escape(rel, log_get_8_bit_output());
|
|
||||||
fprintf(stderr, " Deleted: %s\n", escaped ? escaped : "<allocation failed>");
|
|
||||||
free(escaped);
|
|
||||||
}
|
|
||||||
if (parent_fd >= 0)
|
|
||||||
close(parent_fd);
|
|
||||||
free(leaf);
|
|
||||||
free(full);
|
|
||||||
if (!ok)
|
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
delete_skips_free(&skips);
|
delete_skips_free(&skips);
|
||||||
@@ -386,8 +425,11 @@ bool manifest_would_delete_list(const Config* config, const DeleteManifest* mani
|
|||||||
DeleteSkipSet skips;
|
DeleteSkipSet skips;
|
||||||
if (!delete_skips_build(config, manifest->protected, NULL, true, &skips))
|
if (!delete_skips_build(config, manifest->protected, NULL, true, &skips))
|
||||||
return false;
|
return false;
|
||||||
|
DeleteProtectRules protect = {.base_rules = config->protect_rules,
|
||||||
|
.dir_rules = manifest->per_dir_rules,
|
||||||
|
.backup_suffix = delete_backup_suffix(config)};
|
||||||
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
|
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
|
||||||
skips.entries, skips.count, config->protect_rules, out, count_out);
|
skips.entries, skips.count, &protect, out, count_out);
|
||||||
delete_skips_free(&skips);
|
delete_skips_free(&skips);
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,7 @@
|
|||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
#include "delete.h"
|
#include "delete.h"
|
||||||
|
#include "filter.h"
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
|
|
||||||
/* Delete-commit module: delete-manifest receive plus the budgeted extras and
|
/* Delete-commit module: delete-manifest receive plus the budgeted extras and
|
||||||
@@ -30,6 +31,13 @@ typedef struct DeleteManifest {
|
|||||||
leave untransmitted directories and the unlisted parts of listed ones
|
leave untransmitted directories and the unlisted parts of listed ones
|
||||||
alone, matching rsync's "delete only in synchronized directories". */
|
alone, matching rsync's "delete only in synchronized directories". */
|
||||||
ArrayList* dirs;
|
ArrayList* dirs;
|
||||||
|
/* Per-directory filter rules the sender compiled while scanning (protocol
|
||||||
|
2.30.0), each carrying its owner directory and no-inherit flag. The
|
||||||
|
receiver evaluates them (deepest before ancestors, then the command-line
|
||||||
|
base rules) against every candidate extra so a destination-only entry that
|
||||||
|
matches ONLY a per-directory `.rsync-filter`/dir-merge rule is shielded.
|
||||||
|
NULL when the sender transmitted none. */
|
||||||
|
FilterRuleList* per_dir_rules;
|
||||||
} DeleteManifest;
|
} DeleteManifest;
|
||||||
|
|
||||||
void delete_manifest_free(DeleteManifest* manifest);
|
void delete_manifest_free(DeleteManifest* manifest);
|
||||||
|
|||||||
+209
-21
@@ -48,6 +48,7 @@ struct DeletePlanSender {
|
|||||||
const ArrayList* protected_prefixes;
|
const ArrayList* protected_prefixes;
|
||||||
const ArrayList* size_skipped;
|
const ArrayList* size_skipped;
|
||||||
const ArrayList* missing_args;
|
const ArrayList* missing_args;
|
||||||
|
const FilterRuleList* per_dir_rules;
|
||||||
size_t entries;
|
size_t entries;
|
||||||
/* Transmitted FILE entries only. The caller's "empty scan" safety guard keys
|
/* Transmitted FILE entries only. The caller's "empty scan" safety guard keys
|
||||||
off this (an I/O error that hid every file must refuse to delete even when
|
off this (an I/O error that hid every file must refuse to delete even when
|
||||||
@@ -284,12 +285,14 @@ bool delete_plan_sender_empty(const DeletePlanSender* sender) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
||||||
const ArrayList* size_skipped, const ArrayList* missing_args) {
|
const ArrayList* size_skipped, const ArrayList* missing_args,
|
||||||
|
const FilterRuleList* per_dir_rules) {
|
||||||
if (!sender)
|
if (!sender)
|
||||||
return;
|
return;
|
||||||
sender->protected_prefixes = protected_prefixes;
|
sender->protected_prefixes = protected_prefixes;
|
||||||
sender->size_skipped = size_skipped;
|
sender->size_skipped = size_skipped;
|
||||||
sender->missing_args = missing_args;
|
sender->missing_args = missing_args;
|
||||||
|
sender->per_dir_rules = per_dir_rules;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* True when `dir` is `root` itself or a descendant of it (path-component
|
/* True when `dir` is `root` itself or a descendant of it (path-component
|
||||||
@@ -320,6 +323,181 @@ static int send_str_section(int fd, const ArrayList* list) {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The directory a rule belongs to (its owner, or the transfer root for ""). */
|
||||||
|
static const char* filter_dir_rule_owner(const FilterRule* rule) {
|
||||||
|
return (rule && rule->owner) ? rule->owner : "";
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Transmit the received-side per-directory filter rules (protocol 2.30.0) as a
|
||||||
|
* self-describing list of directory groups: a group count, then for each group
|
||||||
|
* the relative owner directory followed by that directory's rule records (run
|
||||||
|
* order = the sender's traversal/rule order). Rules of one directory are
|
||||||
|
* appended to the sink contiguously, so runs reproduce the compilation order.
|
||||||
|
* Bounded by MAX_FILTER_RULES / MAX_FILTER_BYTES and MAX_PROTECT_PATTERN_LEN so
|
||||||
|
* the peer never sees a frame it would reject. The sender enforces exactly the
|
||||||
|
* receiver's limits (including the cumulative owner+pattern byte budget) and
|
||||||
|
* fails with a clear local error instead of emitting a frame that would abort
|
||||||
|
* the transfer with STATUS_ERROR. */
|
||||||
|
bool delete_filter_dir_rules_send(int fd, const FilterRuleList* rules) {
|
||||||
|
int count = rules ? rules->count : 0;
|
||||||
|
if (count < 0 || count > MAX_FILTER_RULES) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "too many per-directory filter rules: %d (maximum %d)", count,
|
||||||
|
MAX_FILTER_RULES);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
size_t bytes = 0;
|
||||||
|
for (int i = 0; i < count; i++) {
|
||||||
|
const FilterRule* rule = rules->items[i];
|
||||||
|
const char* owner = filter_dir_rule_owner(rule);
|
||||||
|
size_t owner_len = strlen(owner);
|
||||||
|
size_t pattern_len = rule && rule->pattern ? strlen(rule->pattern) : 0;
|
||||||
|
if (!rule || !rule->pattern || pattern_len == 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "invalid per-directory filter pattern");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (pattern_len > MAX_PROTECT_PATTERN_LEN) {
|
||||||
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"per-directory filter pattern exceeds %d bytes (use a shorter pattern)",
|
||||||
|
MAX_PROTECT_PATTERN_LEN);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!(owner_len == 0 || (owner[0] != '/' && !has_path_traversal(owner)))) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "invalid per-directory filter owner directory");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (owner_len + pattern_len > MAX_FILTER_BYTES - bytes) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "per-directory filter rules exceed %d bytes", MAX_FILTER_BYTES);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bytes += owner_len + pattern_len;
|
||||||
|
}
|
||||||
|
int groups = 0;
|
||||||
|
for (int i = 0; i < count;) {
|
||||||
|
const char* owner = filter_dir_rule_owner(rules->items[i]);
|
||||||
|
groups++;
|
||||||
|
i++;
|
||||||
|
while (i < count && strcmp(filter_dir_rule_owner(rules->items[i]), owner) == 0)
|
||||||
|
i++;
|
||||||
|
}
|
||||||
|
if (!send_int(fd, groups))
|
||||||
|
return false;
|
||||||
|
for (int i = 0; i < count;) {
|
||||||
|
const char* owner = filter_dir_rule_owner(rules->items[i]);
|
||||||
|
int start = i;
|
||||||
|
i++;
|
||||||
|
while (i < count && strcmp(filter_dir_rule_owner(rules->items[i]), owner) == 0)
|
||||||
|
i++;
|
||||||
|
if (!send_wire_str(fd, owner) || !send_int(fd, i - start))
|
||||||
|
return false;
|
||||||
|
for (int j = start; j < i; j++) {
|
||||||
|
const FilterRule* rule = rules->items[j];
|
||||||
|
if (!send_int(fd, (int)rule->action) || !send_int(fd, (int)rule->sides) ||
|
||||||
|
!send_int(fd, rule->anchored ? 1 : 0) || !send_int(fd, rule->dir_only ? 1 : 0) ||
|
||||||
|
!send_int(fd, rule->negate ? 1 : 0) || !send_int(fd, rule->no_inherit ? 1 : 0) ||
|
||||||
|
!send_wire_str(fd, rule->pattern))
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Read one wire flag (an int restricted to 0/1). */
|
||||||
|
static bool receive_flag(int fd, bool* value) {
|
||||||
|
int raw;
|
||||||
|
if (!receive_int(fd, &raw) || (raw != 0 && raw != 1))
|
||||||
|
return false;
|
||||||
|
*value = raw != 0;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Read the per-directory filter block emitted by delete_filter_dir_rules_send.
|
||||||
|
* Reconstructs a flat FilterRuleList whose rules carry their owner directory;
|
||||||
|
* `*out` is NULL when the sender transmitted no rules. Every bound is enforced
|
||||||
|
* (group/rule counts, owner/pattern bytes, pattern length, action/sides domain)
|
||||||
|
* so a malicious peer can neither overread nor allocate unboundedly. Returns
|
||||||
|
* false on a malformed frame (the caller signals STATUS_ERROR). */
|
||||||
|
bool delete_filter_dir_rules_receive(int fd, FilterRuleList** out) {
|
||||||
|
if (!out)
|
||||||
|
return false;
|
||||||
|
*out = NULL;
|
||||||
|
int groups;
|
||||||
|
if (!receive_int(fd, &groups) || groups < 0 || groups > MAX_FILTER_RULES)
|
||||||
|
return false;
|
||||||
|
if (groups == 0)
|
||||||
|
return true;
|
||||||
|
FilterRuleList* list = filter_rule_list_create();
|
||||||
|
if (!list)
|
||||||
|
return false;
|
||||||
|
int total_rules = 0;
|
||||||
|
size_t bytes = 0;
|
||||||
|
for (int g = 0; g < groups; g++) {
|
||||||
|
char* dir = receive_wire_str(fd);
|
||||||
|
if (!dir)
|
||||||
|
goto fail;
|
||||||
|
size_t dir_bytes = strlen(dir);
|
||||||
|
if (!(dir[0] == '\0' || (dir[0] != '/' && !has_path_traversal(dir))) ||
|
||||||
|
dir_bytes > MAX_FILTER_BYTES - bytes) {
|
||||||
|
free(dir);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
bytes += dir_bytes;
|
||||||
|
int rule_count;
|
||||||
|
if (!receive_int(fd, &rule_count) || rule_count < 0 || rule_count > MAX_FILTER_RULES ||
|
||||||
|
rule_count > MAX_FILTER_RULES - total_rules) {
|
||||||
|
free(dir);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
for (int r = 0; r < rule_count; r++) {
|
||||||
|
int action, sides;
|
||||||
|
bool anchored, dir_only, negate, no_inherit;
|
||||||
|
if (!receive_int(fd, &action) ||
|
||||||
|
(action != FILTER_ACTION_EXCLUDE && action != FILTER_ACTION_INCLUDE) ||
|
||||||
|
!receive_int(fd, &sides) || sides < (int)FILTER_SIDE_SENDER ||
|
||||||
|
sides > (int)(FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER) ||
|
||||||
|
!receive_flag(fd, &anchored) || !receive_flag(fd, &dir_only) ||
|
||||||
|
!receive_flag(fd, &negate) || !receive_flag(fd, &no_inherit)) {
|
||||||
|
free(dir);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
char* pattern = receive_wire_str(fd);
|
||||||
|
size_t pattern_bytes = pattern ? strlen(pattern) : 0;
|
||||||
|
if (!pattern || pattern_bytes == 0 || pattern_bytes > MAX_PROTECT_PATTERN_LEN ||
|
||||||
|
pattern_bytes > MAX_FILTER_BYTES - bytes) {
|
||||||
|
free(pattern);
|
||||||
|
free(dir);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
bytes += pattern_bytes;
|
||||||
|
FilterRule* rule = calloc(1, sizeof(FilterRule));
|
||||||
|
if (!rule) {
|
||||||
|
free(pattern);
|
||||||
|
free(dir);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
rule->action = (FilterAction)action;
|
||||||
|
rule->sides = (unsigned)sides;
|
||||||
|
rule->anchored = anchored;
|
||||||
|
rule->dir_only = dir_only;
|
||||||
|
rule->negate = negate;
|
||||||
|
rule->no_inherit = no_inherit;
|
||||||
|
rule->owner = str_dup(dir);
|
||||||
|
rule->pattern = pattern;
|
||||||
|
if (!rule->owner || !filter_rule_list_add(list, rule)) {
|
||||||
|
filter_rule_free(rule);
|
||||||
|
free(dir);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
total_rules++;
|
||||||
|
}
|
||||||
|
free(dir);
|
||||||
|
}
|
||||||
|
*out = list;
|
||||||
|
return true;
|
||||||
|
fail:
|
||||||
|
filter_rule_list_free(list);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
||||||
if (!send_status(fd, STATUS_DELETE_PLAN))
|
if (!send_status(fd, STATUS_DELETE_PLAN))
|
||||||
return -1;
|
return -1;
|
||||||
@@ -328,7 +506,8 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
|||||||
if (!sender->config_sent) {
|
if (!sender->config_sent) {
|
||||||
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
|
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
|
||||||
send_str_section(fd, sender->size_skipped) != 0 ||
|
send_str_section(fd, sender->size_skipped) != 0 ||
|
||||||
send_str_section(fd, sender->missing_args) != 0)
|
send_str_section(fd, sender->missing_args) != 0 ||
|
||||||
|
!delete_filter_dir_rules_send(fd, sender->per_dir_rules))
|
||||||
return -1;
|
return -1;
|
||||||
sender->config_sent = true;
|
sender->config_sent = true;
|
||||||
}
|
}
|
||||||
@@ -355,7 +534,8 @@ static int send_config_only(int fd, DeletePlanSender* sender) {
|
|||||||
return -1;
|
return -1;
|
||||||
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
|
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
|
||||||
send_str_section(fd, sender->size_skipped) != 0 ||
|
send_str_section(fd, sender->size_skipped) != 0 ||
|
||||||
send_str_section(fd, sender->missing_args) != 0)
|
send_str_section(fd, sender->missing_args) != 0 ||
|
||||||
|
!delete_filter_dir_rules_send(fd, sender->per_dir_rules))
|
||||||
return -1;
|
return -1;
|
||||||
sender->config_sent = true;
|
sender->config_sent = true;
|
||||||
if (!send_int(fd, 0)) /* apply = false */
|
if (!send_int(fd, 0)) /* apply = false */
|
||||||
@@ -472,15 +652,19 @@ struct DeletePlanSession {
|
|||||||
ArrayList* protected_prefixes;
|
ArrayList* protected_prefixes;
|
||||||
ArrayList* size_skipped;
|
ArrayList* size_skipped;
|
||||||
ArrayList* missing;
|
ArrayList* missing;
|
||||||
|
/* Received per-directory filter rules (protocol 2.30.0), or NULL. Evaluated
|
||||||
|
deepest-directory-first for every candidate extra so a destination-only
|
||||||
|
entry matching only a per-directory rule is protected. */
|
||||||
|
FilterRuleList* per_dir_rules;
|
||||||
ArrayList* deferred;
|
ArrayList* deferred;
|
||||||
DeletePathObserver observer;
|
DeletePathObserver observer;
|
||||||
void* observer_context;
|
void* observer_context;
|
||||||
};
|
};
|
||||||
|
|
||||||
/* Report one path the session truly removed (no-op without an observer). */
|
/* Report one path the session truly removed (no-op without an observer). */
|
||||||
static void notify_deleted(DeletePlanSession* session, const char* rel) {
|
static void notify_deleted(DeletePlanSession* session, const char* rel, DeleteEntryType type) {
|
||||||
if (session && session->observer && rel)
|
if (session && session->observer && rel)
|
||||||
session->observer(session->observer_context, rel);
|
session->observer(session->observer_context, rel, type);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* A removed directory is reported with rsync's trailing slash (`deleting dir/`)
|
/* A removed directory is reported with rsync's trailing slash (`deleting dir/`)
|
||||||
@@ -491,13 +675,13 @@ static void notify_deleted_dir(DeletePlanSession* session, const char* rel) {
|
|||||||
size_t len = strlen(rel);
|
size_t len = strlen(rel);
|
||||||
char* with_slash = malloc(len + 2);
|
char* with_slash = malloc(len + 2);
|
||||||
if (!with_slash) {
|
if (!with_slash) {
|
||||||
session->observer(session->observer_context, rel);
|
session->observer(session->observer_context, rel, DELETE_ENTRY_DIR);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
memcpy(with_slash, rel, len);
|
memcpy(with_slash, rel, len);
|
||||||
with_slash[len] = '/';
|
with_slash[len] = '/';
|
||||||
with_slash[len + 1] = '\0';
|
with_slash[len + 1] = '\0';
|
||||||
session->observer(session->observer_context, with_slash);
|
session->observer(session->observer_context, with_slash, DELETE_ENTRY_DIR);
|
||||||
free(with_slash);
|
free(with_slash);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -532,6 +716,7 @@ void delete_plan_session_destroy(DeletePlanSession* session) {
|
|||||||
array_list_delete(session->protected_prefixes);
|
array_list_delete(session->protected_prefixes);
|
||||||
array_list_delete(session->size_skipped);
|
array_list_delete(session->size_skipped);
|
||||||
array_list_delete(session->missing);
|
array_list_delete(session->missing);
|
||||||
|
filter_rule_list_free(session->per_dir_rules);
|
||||||
array_list_delete(session->deferred);
|
array_list_delete(session->deferred);
|
||||||
free(session);
|
free(session);
|
||||||
}
|
}
|
||||||
@@ -604,16 +789,19 @@ static int open_plan_dir(const Config* config, const char* dir) {
|
|||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
DeleteSkipSet set;
|
DeleteSkipSet set;
|
||||||
/* Receiver-side delete-protection rules received on the config frame (NULL
|
/* Receiver-side delete-protection rules. `base` is the config-frame
|
||||||
when the sender sent none). Evaluated per extra so a protect/risk rule is
|
command-line set and `dir` the received per-directory set (both NULL when
|
||||||
|
the sender sent none). Evaluated per extra so a protect/risk rule is
|
||||||
honored under --delete-during/--delete-delay exactly like the whole-tree
|
honored under --delete-during/--delete-delay exactly like the whole-tree
|
||||||
commit walker. */
|
commit walker. */
|
||||||
const FilterRuleList* protect_rules;
|
DeleteProtectRules protect;
|
||||||
} PlanSkips;
|
} PlanSkips;
|
||||||
|
|
||||||
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
|
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
|
||||||
PlanSkips* out) {
|
PlanSkips* out) {
|
||||||
out->protect_rules = config->protect_rules;
|
out->protect.base_rules = config->protect_rules;
|
||||||
|
out->protect.dir_rules = session->per_dir_rules;
|
||||||
|
out->protect.backup_suffix = delete_backup_suffix(config);
|
||||||
/* The per-directory plan walk keeps each basis path verbatim (it does not
|
/* The per-directory plan walk keeps each basis path verbatim (it does not
|
||||||
convert an absolute under-root path to its root-relative form, unlike the
|
convert an absolute under-root path to its root-relative form, unlike the
|
||||||
whole-tree commit walk). */
|
whole-tree commit walk). */
|
||||||
@@ -711,8 +899,8 @@ static bool process_extra_dir(int dirfd, const char* name, const char* child_rel
|
|||||||
return errno == ENOTEMPTY || errno == EEXIST;
|
return errno == ENOTEMPTY || errno == EEXIST;
|
||||||
}
|
}
|
||||||
|
|
||||||
static bool process_extra_file(int dirfd, const char* name, const char* child_rel, bool force_now,
|
static bool process_extra_file(int dirfd, const char* name, const char* child_rel, mode_t mode,
|
||||||
DeletePlanSession* session) {
|
bool force_now, DeletePlanSession* session) {
|
||||||
if (session->defer && !force_now) {
|
if (session->defer && !force_now) {
|
||||||
return defer_add(session, child_rel);
|
return defer_add(session, child_rel);
|
||||||
}
|
}
|
||||||
@@ -724,7 +912,7 @@ static bool process_extra_file(int dirfd, const char* name, const char* child_re
|
|||||||
session->deleted++;
|
session->deleted++;
|
||||||
session->planned++;
|
session->planned++;
|
||||||
log_deleted(child_rel);
|
log_deleted(child_rel);
|
||||||
notify_deleted(session, child_rel);
|
notify_deleted(session, child_rel, delete_entry_type_of_mode(mode));
|
||||||
} else if (errno != ENOENT) {
|
} else if (errno != ENOENT) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -778,10 +966,8 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
|
|||||||
bool is_dir = entries[i].is_dir;
|
bool is_dir = entries[i].is_dir;
|
||||||
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entries[i].name);
|
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entries[i].name);
|
||||||
bool in_keep_files = !is_dir && list_contains_str(keep_files, entries[i].name);
|
bool in_keep_files = !is_dir && list_contains_str(keep_files, entries[i].name);
|
||||||
bool rule_protected =
|
bool rule_protected = delete_protect_verdict(&skips->protect, child_rel, entries[i].name,
|
||||||
skips->protect_rules &&
|
is_dir) == FILTER_ACTION_PROTECT;
|
||||||
filter_rules_apply_side(skips->protect_rules, child_rel, entries[i].name, is_dir,
|
|
||||||
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT;
|
|
||||||
if (in_keep_dirs || in_keep_files || rule_protected) {
|
if (in_keep_dirs || in_keep_files || rule_protected) {
|
||||||
shielded[i] = true;
|
shielded[i] = true;
|
||||||
local_survives = true;
|
local_survives = true;
|
||||||
@@ -828,7 +1014,8 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
|
|||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (!process_extra_file(dirfd, entries[i].name, child_rel, force[i] || force_now, session))
|
if (!process_extra_file(dirfd, entries[i].name, child_rel, entries[i].mode,
|
||||||
|
force[i] || force_now, session))
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
free(child_rel);
|
free(child_rel);
|
||||||
}
|
}
|
||||||
@@ -902,7 +1089,8 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
|||||||
if (has_config) {
|
if (has_config) {
|
||||||
if (session->config_seen || !read_section(fd, session->protected_prefixes, true, &bytes) ||
|
if (session->config_seen || !read_section(fd, session->protected_prefixes, true, &bytes) ||
|
||||||
!read_section(fd, session->size_skipped, true, &bytes) ||
|
!read_section(fd, session->size_skipped, true, &bytes) ||
|
||||||
!read_section(fd, session->missing, true, &bytes)) {
|
!read_section(fd, session->missing, true, &bytes) ||
|
||||||
|
!delete_filter_dir_rules_receive(fd, &session->per_dir_rules)) {
|
||||||
send_status(fd, STATUS_ERROR);
|
send_status(fd, STATUS_ERROR);
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
@@ -1032,7 +1220,7 @@ static bool apply_deferred_path(DeletePlanSession* session, const Config* config
|
|||||||
session->deleted++;
|
session->deleted++;
|
||||||
session->planned++;
|
session->planned++;
|
||||||
log_deleted(rel);
|
log_deleted(rel);
|
||||||
notify_deleted(session, rel);
|
notify_deleted(session, rel, delete_entry_type_of_mode(st.st_mode));
|
||||||
} else if (errno != ENOENT) {
|
} else if (errno != ENOENT) {
|
||||||
close(parent_fd);
|
close(parent_fd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
|
|||||||
@@ -25,6 +25,19 @@
|
|||||||
* --delete-missing-args exact deletions, the shared --max-delete budget and,
|
* --delete-missing-args exact deletions, the shared --max-delete budget and,
|
||||||
* for --delete-delay, the snapshotted extras. */
|
* for --delete-delay, the snapshotted extras. */
|
||||||
|
|
||||||
|
/* Per-directory filter-rule block (protocol 2.30.0). The sender compiles the
|
||||||
|
* source's per-directory merge rules as it scans and streams them so the
|
||||||
|
* receiver can re-derive the receiver-side protect/risk verdicts for
|
||||||
|
* destination-only entries. The wire format is a group count, then for each
|
||||||
|
* directory group its relative owner path followed by that directory's rule
|
||||||
|
* records (action, sides, anchored, dir-only, negate, no-inherit, pattern).
|
||||||
|
* All bounds (MAX_FILTER_RULES, MAX_FILTER_BYTES, MAX_PROTECT_PATTERN_LEN) are
|
||||||
|
* enforced on both sides; a malformed receive frame signals STATUS_ERROR and
|
||||||
|
* returns false. Receive yields a flat FilterRuleList whose rules carry their
|
||||||
|
* owner, or NULL when no rules were sent. */
|
||||||
|
bool delete_filter_dir_rules_send(int fd, const FilterRuleList* rules);
|
||||||
|
bool delete_filter_dir_rules_receive(int fd, FilterRuleList** out);
|
||||||
|
|
||||||
/* ---- Sender: plan builder ---- */
|
/* ---- Sender: plan builder ---- */
|
||||||
|
|
||||||
typedef struct DeletePlanSender DeletePlanSender;
|
typedef struct DeletePlanSender DeletePlanSender;
|
||||||
@@ -53,7 +66,8 @@ bool delete_plan_sender_empty(const DeletePlanSender* sender);
|
|||||||
* block is always transmitted by delete_plan_send_root(), on a config-only
|
* block is always transmitted by delete_plan_send_root(), on a config-only
|
||||||
* carrier frame when the scope allows no directory plan. */
|
* carrier frame when the scope allows no directory plan. */
|
||||||
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
||||||
const ArrayList* size_skipped, const ArrayList* missing_args);
|
const ArrayList* size_skipped, const ArrayList* missing_args,
|
||||||
|
const FilterRuleList* per_dir_rules);
|
||||||
/* Send the root plan (even before any data, so root extras are handled like
|
/* Send the root plan (even before any data, so root extras are handled like
|
||||||
* rsync's first generator directory), after transmitting the per-run config
|
* rsync's first generator directory), after transmitting the per-run config
|
||||||
* block on its own carrier frame. Returns -1 on I/O error. */
|
* block on its own carrier frame. Returns -1 on I/O error. */
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
|
#include <errno.h>
|
||||||
#include <stdint.h>
|
#include <stdint.h>
|
||||||
#include <limits.h>
|
#include <limits.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
#define XXH_STATIC_LINKING_ONLY
|
#define XXH_STATIC_LINKING_ONLY
|
||||||
#define XXH_IMPLEMENTATION
|
#define XXH_IMPLEMENTATION
|
||||||
@@ -82,6 +84,64 @@ DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_
|
|||||||
return sig;
|
return sig;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Bounded read of exactly `len` bytes at `off`; retries on EINTR. */
|
||||||
|
static bool pread_all(int fd, void* buf, size_t len, uint64_t off) {
|
||||||
|
uint8_t* p = buf;
|
||||||
|
size_t done = 0;
|
||||||
|
while (done < len) {
|
||||||
|
ssize_t n = pread(fd, p + done, len - done, (off_t)(off + done));
|
||||||
|
if (n < 0 && errno == EINTR)
|
||||||
|
continue;
|
||||||
|
if (n <= 0)
|
||||||
|
return false;
|
||||||
|
done += (size_t)n;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
DeltaSignature* delta_signature_create_fd_seeded(int fd, uint64_t old_file_size,
|
||||||
|
uint32_t block_size, uint32_t seed) {
|
||||||
|
if (fd < 0 || old_file_size == 0 || block_size == 0 || block_size > DELTA_BLOCK_SIZE_MAX ||
|
||||||
|
old_file_size > UINT32_MAX * (uint64_t)block_size)
|
||||||
|
return NULL;
|
||||||
|
uint32_t block_count = (uint32_t)((old_file_size + block_size - 1) / block_size);
|
||||||
|
/* Bound the signature's own memory (block_count * sizeof(DeltaBlockSig)). */
|
||||||
|
if (block_count == 0 || block_count > MAX_DELTA_BLOCKS)
|
||||||
|
return NULL;
|
||||||
|
DeltaSignature* sig = protocol_alloc(sizeof(DeltaSignature));
|
||||||
|
if (!sig)
|
||||||
|
return NULL;
|
||||||
|
sig->file_size = old_file_size;
|
||||||
|
sig->block_size = block_size;
|
||||||
|
sig->block_count = block_count;
|
||||||
|
sig->blocks = protocol_alloc((size_t)block_count * sizeof(DeltaBlockSig));
|
||||||
|
if (!sig->blocks) {
|
||||||
|
free(sig);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
uint8_t* block = malloc(block_size);
|
||||||
|
if (!block) {
|
||||||
|
free(sig->blocks);
|
||||||
|
free(sig);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
for (uint32_t i = 0; i < block_count; i++) {
|
||||||
|
uint64_t offset = (uint64_t)i * block_size;
|
||||||
|
uint32_t len =
|
||||||
|
(uint32_t)((old_file_size - offset < block_size) ? (old_file_size - offset) : block_size);
|
||||||
|
if (!pread_all(fd, block, len, offset)) {
|
||||||
|
free(block);
|
||||||
|
free(sig->blocks);
|
||||||
|
free(sig);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
sig->blocks[i].adler32 = delta_adler32(block, len);
|
||||||
|
sig->blocks[i].xxhash = delta_xxhash32_seeded(block, len, seed);
|
||||||
|
}
|
||||||
|
free(block);
|
||||||
|
return sig;
|
||||||
|
}
|
||||||
|
|
||||||
Data* delta_signature_serialize(const DeltaSignature* sig) {
|
Data* delta_signature_serialize(const DeltaSignature* sig) {
|
||||||
if (!sig)
|
if (!sig)
|
||||||
return NULL;
|
return NULL;
|
||||||
@@ -687,6 +747,130 @@ void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta,
|
|||||||
return output;
|
return output;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void* delta_apply_fd(int src_fd, uint64_t old_size, const Delta* delta, uint32_t block_size) {
|
||||||
|
if (!delta || block_size == 0 || block_size > DELTA_BLOCK_SIZE_MAX ||
|
||||||
|
(delta->instruction_count > 0 && !delta->instructions) || delta->new_file_size == 0 ||
|
||||||
|
delta->new_file_size > SIZE_MAX)
|
||||||
|
return NULL;
|
||||||
|
|
||||||
|
void* output = protocol_alloc((size_t)delta->new_file_size);
|
||||||
|
if (!output)
|
||||||
|
return NULL;
|
||||||
|
|
||||||
|
uint8_t* out = (uint8_t*)output;
|
||||||
|
uint64_t out_pos = 0;
|
||||||
|
|
||||||
|
for (uint32_t i = 0; i < delta->instruction_count; i++) {
|
||||||
|
if (delta->instructions[i].type == DELTA_INSTR_BLOCK_MATCH) {
|
||||||
|
uint64_t src_offset = (uint64_t)delta->instructions[i].match.block_index * block_size;
|
||||||
|
if (src_offset > UINT64_MAX - delta->instructions[i].match.block_offset) {
|
||||||
|
free(output);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
src_offset += delta->instructions[i].match.block_offset;
|
||||||
|
uint32_t len = delta->instructions[i].match.length;
|
||||||
|
|
||||||
|
if (src_offset > old_size || (uint64_t)len > old_size - src_offset ||
|
||||||
|
out_pos > delta->new_file_size || (uint64_t)len > delta->new_file_size - out_pos) {
|
||||||
|
free(output);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
if (!pread_all(src_fd, out + out_pos, len, src_offset)) {
|
||||||
|
free(output);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
out_pos += len;
|
||||||
|
} else if (delta->instructions[i].type == DELTA_INSTR_LITERAL) {
|
||||||
|
uint32_t len = delta->instructions[i].literal.length;
|
||||||
|
if (out_pos > delta->new_file_size || (uint64_t)len > delta->new_file_size - out_pos) {
|
||||||
|
free(output);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
memcpy(out + out_pos, delta->instructions[i].literal.data, len);
|
||||||
|
out_pos += len;
|
||||||
|
} else {
|
||||||
|
free(output);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (out_pos != delta->new_file_size) {
|
||||||
|
free(output);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
return output;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool delta_apply_to_fd(const void* old_data, int src_fd, uint64_t old_size, const Delta* delta,
|
||||||
|
uint32_t block_size, int dst_fd) {
|
||||||
|
if (!delta || (old_data == NULL && src_fd < 0) || block_size == 0 ||
|
||||||
|
block_size > DELTA_BLOCK_SIZE_MAX || (delta->instruction_count > 0 && !delta->instructions))
|
||||||
|
return false;
|
||||||
|
const int chunk = 1 << 20;
|
||||||
|
uint8_t* buf = malloc((size_t)chunk);
|
||||||
|
if (!buf)
|
||||||
|
return false;
|
||||||
|
uint64_t out_pos = 0;
|
||||||
|
bool ok = true;
|
||||||
|
for (uint32_t i = 0; i < delta->instruction_count && ok; i++) {
|
||||||
|
uint64_t src_offset = 0;
|
||||||
|
uint64_t len = 0;
|
||||||
|
const uint8_t* lit = NULL;
|
||||||
|
if (delta->instructions[i].type == DELTA_INSTR_BLOCK_MATCH) {
|
||||||
|
src_offset = (uint64_t)delta->instructions[i].match.block_index * block_size;
|
||||||
|
if (src_offset > UINT64_MAX - delta->instructions[i].match.block_offset ||
|
||||||
|
src_offset + delta->instructions[i].match.block_offset > old_size) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
src_offset += delta->instructions[i].match.block_offset;
|
||||||
|
len = delta->instructions[i].match.length;
|
||||||
|
if (len > old_size - src_offset) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
} else if (delta->instructions[i].type == DELTA_INSTR_LITERAL) {
|
||||||
|
lit = delta->instructions[i].literal.data;
|
||||||
|
len = delta->instructions[i].literal.length;
|
||||||
|
} else {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (out_pos > delta->new_file_size || len > delta->new_file_size - out_pos) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
uint64_t done = 0;
|
||||||
|
while (ok && done < len) {
|
||||||
|
size_t want = (len - done) < (uint64_t)chunk ? (size_t)(len - done) : (size_t)chunk;
|
||||||
|
if (lit) {
|
||||||
|
memcpy(buf, lit + done, want);
|
||||||
|
} else if (old_data) {
|
||||||
|
memcpy(buf, (const uint8_t*)old_data + src_offset + done, want);
|
||||||
|
} else if (!pread_all(src_fd, buf, want, src_offset + done)) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
const uint8_t* p = buf;
|
||||||
|
size_t written = 0;
|
||||||
|
while (written < want) {
|
||||||
|
ssize_t n = write(dst_fd, p + written, want - written);
|
||||||
|
if (n < 0 && errno == EINTR)
|
||||||
|
continue;
|
||||||
|
if (n <= 0) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
written += (size_t)n;
|
||||||
|
}
|
||||||
|
done += want;
|
||||||
|
}
|
||||||
|
out_pos += len;
|
||||||
|
}
|
||||||
|
free(buf);
|
||||||
|
return ok && out_pos == delta->new_file_size;
|
||||||
|
}
|
||||||
|
|
||||||
void delta_destroy(Delta* delta) {
|
void delta_destroy(Delta* delta) {
|
||||||
if (!delta)
|
if (!delta)
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -61,6 +61,13 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
|
|||||||
* identical to the unseeded function. */
|
* identical to the unseeded function. */
|
||||||
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
|
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
|
||||||
uint32_t block_size, uint32_t seed);
|
uint32_t block_size, uint32_t seed);
|
||||||
|
/* Streaming equivalent of delta_signature_create_seeded: reads the basis blocks
|
||||||
|
* from `fd` in bounded chunks, so an arbitrarily large basis can be signed
|
||||||
|
* without materializing it. The signature itself is bounded (MAX_DELTA_BLOCKS
|
||||||
|
* entries); an over-large basis returns NULL and the caller falls back to a
|
||||||
|
* whole-file transfer. */
|
||||||
|
DeltaSignature* delta_signature_create_fd_seeded(int fd, uint64_t old_file_size,
|
||||||
|
uint32_t block_size, uint32_t seed);
|
||||||
Data* delta_signature_serialize(const DeltaSignature* sig);
|
Data* delta_signature_serialize(const DeltaSignature* sig);
|
||||||
DeltaSignature* delta_signature_deserialize(const Data* data);
|
DeltaSignature* delta_signature_deserialize(const Data* data);
|
||||||
void delta_signature_destroy(DeltaSignature* sig);
|
void delta_signature_destroy(DeltaSignature* sig);
|
||||||
@@ -75,6 +82,15 @@ Delta* delta_compute_seeded(const void* new_file_data, uint64_t new_file_size,
|
|||||||
Data* delta_serialize(const Delta* delta);
|
Data* delta_serialize(const Delta* delta);
|
||||||
Delta* delta_deserialize(const Data* data);
|
Delta* delta_deserialize(const Data* data);
|
||||||
void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta, uint32_t block_size);
|
void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta, uint32_t block_size);
|
||||||
|
/* Streaming equivalent of delta_apply: matched blocks are read from `src_fd` as
|
||||||
|
* they are emitted, so the basis never has to be resident. */
|
||||||
|
void* delta_apply_fd(int src_fd, uint64_t old_size, const Delta* delta, uint32_t block_size);
|
||||||
|
/* Fully streamed reconstruction: matched blocks come from `old_data` (when
|
||||||
|
* non-NULL) or are read from `src_fd`, and the reconstructed bytes are written
|
||||||
|
* straight to `dst_fd` in bounded chunks, so a reconstructed file larger than
|
||||||
|
* memory is never materialized. */
|
||||||
|
bool delta_apply_to_fd(const void* old_data, int src_fd, uint64_t old_size, const Delta* delta,
|
||||||
|
uint32_t block_size, int dst_fd);
|
||||||
void delta_destroy(Delta* delta);
|
void delta_destroy(Delta* delta);
|
||||||
|
|
||||||
bool delta_should_attempt(uint64_t old_size, uint64_t new_size, uint64_t max_file_size);
|
bool delta_should_attempt(uint64_t old_size, uint64_t new_size, uint64_t max_file_size);
|
||||||
|
|||||||
+292
-24
@@ -16,6 +16,8 @@
|
|||||||
|
|
||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "checksum.h"
|
#include "checksum.h"
|
||||||
|
#include "chunk.h"
|
||||||
|
#include "compression.h"
|
||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "file_store.h"
|
#include "file_store.h"
|
||||||
@@ -209,6 +211,7 @@ File* file_create(const char* path) {
|
|||||||
file->dir_time_only = false;
|
file->dir_time_only = false;
|
||||||
file->basis_link = NULL;
|
file->basis_link = NULL;
|
||||||
file->basis_copy = NULL;
|
file->basis_copy = NULL;
|
||||||
|
file->data_spool = false;
|
||||||
file->link_group = 0;
|
file->link_group = 0;
|
||||||
file->link_first = false;
|
file->link_first = false;
|
||||||
file->hardlink_target = NULL;
|
file->hardlink_target = NULL;
|
||||||
@@ -238,8 +241,11 @@ void file_destroy(void* item) {
|
|||||||
file->send_path = NULL;
|
file->send_path = NULL;
|
||||||
free(file->basis_link);
|
free(file->basis_link);
|
||||||
file->basis_link = NULL;
|
file->basis_link = NULL;
|
||||||
|
if (file->data_spool && file->basis_copy)
|
||||||
|
unlink(file->basis_copy);
|
||||||
free(file->basis_copy);
|
free(file->basis_copy);
|
||||||
file->basis_copy = NULL;
|
file->basis_copy = NULL;
|
||||||
|
file->data_spool = false;
|
||||||
free(file->hardlink_target);
|
free(file->hardlink_target);
|
||||||
file->hardlink_target = NULL;
|
file->hardlink_target = NULL;
|
||||||
free(file->symlink_target);
|
free(file->symlink_target);
|
||||||
@@ -381,6 +387,261 @@ size_t file_content_to_buffer(File* file) {
|
|||||||
return bytes_read;
|
return bytes_read;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ---- Streamed whole-file payload receive ----
|
||||||
|
*
|
||||||
|
* A whole-file data frame is a uint64 length followed by that many bytes. When
|
||||||
|
* the logical payload is at or below the receiver's streaming bound the
|
||||||
|
* historical charged whole-buffer path is kept (decompressing in one shot for
|
||||||
|
* `-z`). Above the bound the frame is read in bounded chunks and written into
|
||||||
|
* a spool temp file next to the destination, decompressing incrementally for
|
||||||
|
* zstd/zlib (lz4's block format cannot be streamed and keeps the buffered path).
|
||||||
|
* The spool is then installed by the existing bounded-buffer basis-copy path
|
||||||
|
* (file_copy_basis_stream_attrs), so the atomic temp+rename store, --partial/
|
||||||
|
* --partial-dir, --delay-updates, --preallocate and metadata/xattr application
|
||||||
|
* are all reused unchanged. Only bounded buffers (64 KiB read chunk + the
|
||||||
|
* decompressor's 256 KiB output window) are ever live. */
|
||||||
|
|
||||||
|
#define FILE_PAYLOAD_READ_CHUNK (64 * 1024)
|
||||||
|
#define FILE_PAYLOAD_PEEK_MAX 32
|
||||||
|
|
||||||
|
/* Create a confined spool temp file in the destination's directory. Returns an
|
||||||
|
* open write fd and an owned absolute path, or -1 (errno set). The parent walk
|
||||||
|
* creates missing directories exactly as a normal store would. */
|
||||||
|
static int file_spool_create(const char* dest_path, char** out_spool_path) {
|
||||||
|
*out_spool_path = NULL;
|
||||||
|
char* leaf = NULL;
|
||||||
|
int dirfd = file_open_secure_parent(dest_path, &leaf, true);
|
||||||
|
free(leaf);
|
||||||
|
if (dirfd < 0)
|
||||||
|
return -1;
|
||||||
|
char name[64];
|
||||||
|
for (unsigned int i = 0; i < 100; i++) {
|
||||||
|
snprintf(name, sizeof(name), ".fastsync-spool.%ld.%llu", (long)getpid(), next_temp_sequence());
|
||||||
|
int fd = openat(dirfd, name, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
|
||||||
|
if (fd < 0) {
|
||||||
|
if (errno != EEXIST)
|
||||||
|
break;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
char* copy = str_dup(dest_path);
|
||||||
|
const char* dir = copy ? dirname(copy) : NULL;
|
||||||
|
size_t need = dir ? strlen(dir) + 1 + strlen(name) + 1 : 0;
|
||||||
|
char* full = need ? malloc(need) : NULL;
|
||||||
|
if (!full) {
|
||||||
|
free(copy);
|
||||||
|
close(fd);
|
||||||
|
unlinkat(dirfd, name, 0);
|
||||||
|
close(dirfd);
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
snprintf(full, need, "%s/%s", dir, name);
|
||||||
|
free(copy);
|
||||||
|
close(dirfd);
|
||||||
|
*out_spool_path = full;
|
||||||
|
return fd;
|
||||||
|
}
|
||||||
|
close(dirfd);
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int file_spool_for_payload(const char* dest_path, char** out_spool_path) {
|
||||||
|
return file_spool_create(dest_path, out_spool_path);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool file_receive_payload(int fd, bool compress, unsigned long long expected_size,
|
||||||
|
const char* dest_path, unsigned long long stream_limit, Data** out_buffer,
|
||||||
|
char** out_spool, unsigned long long* out_size) {
|
||||||
|
if (out_buffer)
|
||||||
|
*out_buffer = NULL;
|
||||||
|
if (out_spool)
|
||||||
|
*out_spool = NULL;
|
||||||
|
if (out_size)
|
||||||
|
*out_size = 0;
|
||||||
|
if (!out_buffer || !out_spool || !out_size || !dest_path)
|
||||||
|
return false;
|
||||||
|
|
||||||
|
unsigned long long frame_size = 0;
|
||||||
|
if (!receive_n_data(fd, &frame_size, sizeof(frame_size)))
|
||||||
|
return false;
|
||||||
|
/* A frame larger than MAX_DATA_PAYLOAD_SIZE is allowed only on the streamed
|
||||||
|
path, which never materializes it; the buffered path's
|
||||||
|
receive_data_alloc/body enforces the historical bound itself. Only a size
|
||||||
|
unrepresentable on this platform is rejected here. */
|
||||||
|
if (frame_size > SIZE_MAX) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Data size %llu is not representable", frame_size);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
unsigned char prefix[FILE_PAYLOAD_PEEK_MAX];
|
||||||
|
size_t prefix_len = 0;
|
||||||
|
bool stream;
|
||||||
|
if (expected_size != 0) {
|
||||||
|
/* The check frame already told us the logical size: no need to peek. */
|
||||||
|
stream = expected_size > stream_limit;
|
||||||
|
} else if (!compress) {
|
||||||
|
stream = frame_size > stream_limit;
|
||||||
|
} else {
|
||||||
|
/* Non-incremental compressed frame with unknown logical size: peek the
|
||||||
|
header to decide, so a small compressed frame that expands past the bound
|
||||||
|
still streams instead of allocating the whole logical image. */
|
||||||
|
size_t want = frame_size < sizeof(prefix) ? (size_t)frame_size : sizeof(prefix);
|
||||||
|
if (want > 0 && !receive_n_data(fd, prefix, want))
|
||||||
|
return false;
|
||||||
|
prefix_len = want;
|
||||||
|
unsigned long long logical = compression_peek_frame_content_size(prefix, prefix_len);
|
||||||
|
stream = logical != 0 ? logical > stream_limit : frame_size > stream_limit;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!stream) {
|
||||||
|
Data* frame;
|
||||||
|
if (prefix_len > 0) {
|
||||||
|
frame = receive_data_alloc(fd, frame_size);
|
||||||
|
if (!frame)
|
||||||
|
return false;
|
||||||
|
memcpy(frame->data, prefix, prefix_len);
|
||||||
|
if (frame_size > prefix_len &&
|
||||||
|
!receive_n_data(fd, (char*)frame->data + prefix_len, (size_t)(frame_size - prefix_len))) {
|
||||||
|
data_destroy(frame);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
frame = receive_data_body(fd, frame_size);
|
||||||
|
if (!frame)
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (compress) {
|
||||||
|
unsigned long long bound =
|
||||||
|
expected_size != 0 ? expected_size : (unsigned long long)MAX_RECEIVE_WHOLE_FILE_SIZE;
|
||||||
|
Data* uncompressed = data_decompress_limited(frame, (size_t)bound);
|
||||||
|
ProtocolSession* owner = frame->owner;
|
||||||
|
data_destroy(frame);
|
||||||
|
if (!uncompressed)
|
||||||
|
return false;
|
||||||
|
if (expected_size != 0 && uncompressed->size != expected_size) {
|
||||||
|
data_destroy(uncompressed);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!data_charge_session(uncompressed, owner, uncompressed->size)) {
|
||||||
|
data_destroy(uncompressed);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
*out_buffer = uncompressed;
|
||||||
|
*out_size = uncompressed->size;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
*out_buffer = frame;
|
||||||
|
*out_size = frame->size;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Streamed path. */
|
||||||
|
int spool_fd = file_spool_create(dest_path, out_spool);
|
||||||
|
if (spool_fd < 0)
|
||||||
|
return false;
|
||||||
|
|
||||||
|
CompressionStreamDecompressor* dec = NULL;
|
||||||
|
size_t header_len = 0;
|
||||||
|
unsigned long long learned_size = expected_size;
|
||||||
|
if (compress) {
|
||||||
|
unsigned char hdr[1 + 4];
|
||||||
|
size_t hdr_have = 0;
|
||||||
|
if (prefix_len >= 1) {
|
||||||
|
hdr[0] = prefix[0];
|
||||||
|
hdr_have = 1;
|
||||||
|
} else {
|
||||||
|
if (!receive_n_data(fd, hdr, 1))
|
||||||
|
goto stream_fail;
|
||||||
|
hdr_have = 1;
|
||||||
|
}
|
||||||
|
CompressionAlgo algo = (CompressionAlgo)hdr[0];
|
||||||
|
if (!compression_algo_valid((int)algo) || algo == COMPRESSION_ALGO_LZ4)
|
||||||
|
goto stream_fail;
|
||||||
|
header_len = (algo == COMPRESSION_ALGO_ZLIB || algo == COMPRESSION_ALGO_ZLIBX) ? 5 : 1;
|
||||||
|
while (hdr_have < header_len) {
|
||||||
|
size_t need = header_len - hdr_have;
|
||||||
|
if (prefix_len > hdr_have) {
|
||||||
|
size_t avail = prefix_len - hdr_have;
|
||||||
|
size_t take = avail < need ? avail : need;
|
||||||
|
memcpy(hdr + hdr_have, prefix + hdr_have, take);
|
||||||
|
hdr_have += take;
|
||||||
|
} else {
|
||||||
|
if (!receive_n_data(fd, hdr + hdr_have, need))
|
||||||
|
goto stream_fail;
|
||||||
|
hdr_have = header_len;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (header_len == 5) {
|
||||||
|
uint32_t raw = 0;
|
||||||
|
for (int i = 0; i < 4; i++)
|
||||||
|
raw |= (uint32_t)hdr[1 + i] << (8 * i);
|
||||||
|
if (expected_size != 0 && raw != expected_size)
|
||||||
|
goto stream_fail;
|
||||||
|
if (learned_size == 0)
|
||||||
|
learned_size = raw;
|
||||||
|
}
|
||||||
|
dec = compression_stream_decompressor_create(algo, learned_size);
|
||||||
|
if (!dec)
|
||||||
|
goto stream_fail;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (frame_size < header_len)
|
||||||
|
goto stream_fail;
|
||||||
|
{
|
||||||
|
unsigned long long body_remaining = frame_size - header_len;
|
||||||
|
if (prefix_len > header_len) {
|
||||||
|
size_t avail = prefix_len - header_len;
|
||||||
|
if (compress) {
|
||||||
|
bool done = false;
|
||||||
|
if (!compression_stream_decompressor_feed(dec, prefix + header_len, avail, spool_fd, &done))
|
||||||
|
goto stream_fail;
|
||||||
|
} else if (!write_all(spool_fd, prefix + header_len, avail)) {
|
||||||
|
goto stream_fail;
|
||||||
|
}
|
||||||
|
body_remaining -= avail;
|
||||||
|
}
|
||||||
|
unsigned char buf[FILE_PAYLOAD_READ_CHUNK];
|
||||||
|
while (body_remaining > 0) {
|
||||||
|
size_t want = body_remaining < sizeof(buf) ? (size_t)body_remaining : (size_t)sizeof(buf);
|
||||||
|
if (!receive_n_data(fd, buf, want))
|
||||||
|
goto stream_fail;
|
||||||
|
if (compress) {
|
||||||
|
bool done = false;
|
||||||
|
if (!compression_stream_decompressor_feed(dec, buf, want, spool_fd, &done))
|
||||||
|
goto stream_fail;
|
||||||
|
} else if (!write_all(spool_fd, buf, want)) {
|
||||||
|
goto stream_fail;
|
||||||
|
}
|
||||||
|
body_remaining -= want;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
unsigned long long total = compress ? compression_stream_decompressor_total(dec) : frame_size;
|
||||||
|
if (learned_size != 0 && total != learned_size)
|
||||||
|
goto stream_fail;
|
||||||
|
*out_size = total;
|
||||||
|
}
|
||||||
|
if (dec)
|
||||||
|
compression_stream_decompressor_destroy(dec);
|
||||||
|
if (close(spool_fd) != 0) {
|
||||||
|
spool_fd = -1;
|
||||||
|
goto stream_fail;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
|
||||||
|
stream_fail:
|
||||||
|
if (dec)
|
||||||
|
compression_stream_decompressor_destroy(dec);
|
||||||
|
if (spool_fd >= 0)
|
||||||
|
close(spool_fd);
|
||||||
|
if (*out_spool) {
|
||||||
|
unlink(*out_spool);
|
||||||
|
free(*out_spool);
|
||||||
|
*out_spool = NULL;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
/* ---- Secure filesystem primitives ---- */
|
/* ---- Secure filesystem primitives ---- */
|
||||||
|
|
||||||
bool file_path_exists_secure(const char* path) {
|
bool file_path_exists_secure(const char* path) {
|
||||||
@@ -1182,21 +1443,24 @@ int file_open_temp_dir(const char* dir_path) {
|
|||||||
* destination file) and best-effort: a per-attribute or privilege failure is
|
* destination file) and best-effort: a per-attribute or privilege failure is
|
||||||
* logged and skipped, never fatal. */
|
* logged and skipped, never fatal. */
|
||||||
static void restore_extra_fd(int fd, const FileMetadata* metadata, const FileXattrList* xattrs,
|
static void restore_extra_fd(int fd, const FileMetadata* metadata, const FileXattrList* xattrs,
|
||||||
bool fake_super, FileAttrPolicy policy) {
|
bool fake_super, FileAttrPolicy policy, uint32_t fake_super_rdev_major,
|
||||||
|
uint32_t fake_super_rdev_minor) {
|
||||||
xattr_apply_fd(fd, xattrs);
|
xattr_apply_fd(fd, xattrs);
|
||||||
if (fake_super && metadata) {
|
if (fake_super && metadata) {
|
||||||
/* Record the ownership that WOULD have been applied: when an explicit
|
/* Record the ownership that WOULD have been applied: when an explicit
|
||||||
ownership request (--chown/--usermap/--groupmap/--copy-as or -o/-g) is
|
ownership request (--chown/--usermap/--groupmap/--copy-as or -o/-g) is
|
||||||
active, the resolved mapping; otherwise the source's own id. The real
|
active, the resolved mapping; otherwise the source's own id. The real
|
||||||
chown is suppressed (identity_apply_ownership early-returns under
|
chown is suppressed (identity_apply_ownership early-returns under
|
||||||
--fake-super) so recording never defeats the flag. Mode/mtime are still
|
--fake-super) so recording never defeats the flag. The recorded stat is
|
||||||
replayed (policy-gated) so unprivileged --fake-super keeps working. */
|
rsync's format; the permission bits are replayed (policy-gated) so
|
||||||
|
unprivileged --fake-super keeps working while mtime comes from the
|
||||||
|
normal metadata path above. */
|
||||||
uint32_t store_uid;
|
uint32_t store_uid;
|
||||||
uint32_t store_gid;
|
uint32_t store_gid;
|
||||||
identity_resolve_storage_ids((int32_t)metadata->uid, (int32_t)metadata->gid, &store_uid,
|
identity_resolve_storage_ids((int32_t)metadata->uid, (int32_t)metadata->gid, &store_uid,
|
||||||
&store_gid);
|
&store_gid);
|
||||||
fake_super_store_fd(fd, store_uid, store_gid, (uint32_t)metadata->mode, metadata->mtime_sec,
|
fake_super_store_fd(fd, store_uid, store_gid, (uint32_t)metadata->mode, fake_super_rdev_major,
|
||||||
metadata->mtime_nsec);
|
fake_super_rdev_minor);
|
||||||
fake_super_restore_fd(fd, policy);
|
fake_super_restore_fd(fd, policy);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1206,7 +1470,8 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
|
|||||||
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
|
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
|
||||||
FileAttrPolicy policy, bool update, bool no_replace, bool use_fsync,
|
FileAttrPolicy policy, bool update, bool no_replace, bool use_fsync,
|
||||||
const char* temp_dir, const FileXattrList* xattrs, bool fake_super,
|
const char* temp_dir, const FileXattrList* xattrs, bool fake_super,
|
||||||
bool keep_partial, unsigned* dirs_created, const char* count_floor) {
|
bool keep_partial, unsigned* dirs_created, const char* count_floor,
|
||||||
|
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor) {
|
||||||
char* leaf = NULL;
|
char* leaf = NULL;
|
||||||
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||||
if (dirfd < 0)
|
if (dirfd < 0)
|
||||||
@@ -1313,7 +1578,8 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (ok)
|
if (ok)
|
||||||
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
|
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, fake_super_rdev_major,
|
||||||
|
fake_super_rdev_minor);
|
||||||
if (ok && use_fsync)
|
if (ok && use_fsync)
|
||||||
ok = fsync(fd) == 0;
|
ok = fsync(fd) == 0;
|
||||||
}
|
}
|
||||||
@@ -1431,7 +1697,8 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (ok)
|
if (ok)
|
||||||
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
|
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, fake_super_rdev_major,
|
||||||
|
fake_super_rdev_minor);
|
||||||
if (ok && use_fsync)
|
if (ok && use_fsync)
|
||||||
ok = fsync(fd) == 0;
|
ok = fsync(fd) == 0;
|
||||||
}
|
}
|
||||||
@@ -1499,7 +1766,8 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
|
|||||||
"non-atomic copy into the destination directory");
|
"non-atomic copy into the destination directory");
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, update, no_replace, use_fsync, NULL, xattrs, fake_super,
|
policy, update, no_replace, use_fsync, NULL, xattrs, fake_super,
|
||||||
keep_partial, dirs_created, count_floor);
|
keep_partial, dirs_created, count_floor, fake_super_rdev_major,
|
||||||
|
fake_super_rdev_minor);
|
||||||
}
|
}
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
@@ -1509,7 +1777,7 @@ bool file_to_disk_secure(const char* path, const void* data, unsigned long long
|
|||||||
FileAttrPolicy policy, const char* temp_dir) {
|
FileAttrPolicy policy, const char* temp_dir) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, false, false, false, temp_dir, NULL, false, false, NULL,
|
policy, false, false, false, temp_dir, NULL, false, false, NULL,
|
||||||
NULL);
|
NULL, 0, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
|
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
|
||||||
@@ -1518,7 +1786,7 @@ bool file_to_disk_secure_update(const char* path, const void* data, unsigned lon
|
|||||||
const char* temp_dir) {
|
const char* temp_dir) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, true, false, false, temp_dir, NULL, false, false, NULL,
|
policy, true, false, false, temp_dir, NULL, false, false, NULL,
|
||||||
NULL);
|
NULL, 0, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
|
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
|
||||||
@@ -1527,7 +1795,7 @@ bool file_to_disk_secure_with_fsync(const char* path, const void* data,
|
|||||||
FileAttrPolicy policy, bool use_fsync, const char* temp_dir) {
|
FileAttrPolicy policy, bool use_fsync, const char* temp_dir) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, false, false, use_fsync, temp_dir, NULL, false, false,
|
policy, false, false, use_fsync, temp_dir, NULL, false, false,
|
||||||
NULL, NULL);
|
NULL, NULL, 0, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_to_disk_secure_no_replace(const char* path, const void* data,
|
bool file_to_disk_secure_no_replace(const char* path, const void* data,
|
||||||
@@ -1536,7 +1804,7 @@ bool file_to_disk_secure_no_replace(const char* path, const void* data,
|
|||||||
const char* temp_dir) {
|
const char* temp_dir) {
|
||||||
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
|
||||||
policy, false, true, false, temp_dir, NULL, false, false, NULL,
|
policy, false, true, false, temp_dir, NULL, false, false, NULL,
|
||||||
NULL);
|
NULL, 0, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
|
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
|
||||||
@@ -1551,19 +1819,19 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
|
|||||||
bool fake_super, bool keep_partial, const char* temp_dir) {
|
bool fake_super, bool keep_partial, const char* temp_dir) {
|
||||||
return file_to_disk_secure_attrs_counted(path, data, data_size, inplace, sparse, preallocate,
|
return file_to_disk_secure_attrs_counted(path, data, data_size, inplace, sparse, preallocate,
|
||||||
metadata, policy, update, no_replace, use_fsync, xattrs,
|
metadata, policy, update, no_replace, use_fsync, xattrs,
|
||||||
fake_super, keep_partial, temp_dir, NULL, NULL);
|
fake_super, keep_partial, temp_dir, NULL, NULL, 0, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
|
bool file_to_disk_secure_attrs_counted(
|
||||||
unsigned long long data_size, bool inplace, bool sparse,
|
const char* path, const void* data, unsigned long long data_size, bool inplace, bool sparse,
|
||||||
bool preallocate, const FileMetadata* metadata,
|
bool preallocate, const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||||
FileAttrPolicy policy, bool update, bool no_replace,
|
bool no_replace, bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
bool keep_partial, const char* temp_dir, unsigned* dirs_created, const char* count_floor,
|
||||||
bool keep_partial, const char* temp_dir,
|
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor) {
|
||||||
unsigned* dirs_created, const char* count_floor) {
|
|
||||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||||
policy, update, no_replace, use_fsync, temp_dir, xattrs,
|
policy, update, no_replace, use_fsync, temp_dir, xattrs,
|
||||||
fake_super, keep_partial, dirs_created, count_floor);
|
fake_super, keep_partial, dirs_created, count_floor,
|
||||||
|
fake_super_rdev_major, fake_super_rdev_minor);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Atomic --link-dest install. The destination is replaced (via a temporary
|
/* Atomic --link-dest install. The destination is replaced (via a temporary
|
||||||
@@ -1693,7 +1961,7 @@ static bool file_copy_basis_stream_impl(const char* path, const char* basis_path
|
|||||||
wrote = false;
|
wrote = false;
|
||||||
}
|
}
|
||||||
if (wrote)
|
if (wrote)
|
||||||
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
|
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, 0, 0);
|
||||||
if (wrote && use_fsync)
|
if (wrote && use_fsync)
|
||||||
wrote = fsync(fd) == 0;
|
wrote = fsync(fd) == 0;
|
||||||
if (close(fd) != 0)
|
if (close(fd) != 0)
|
||||||
@@ -1842,7 +2110,7 @@ static bool file_to_disk_secure_link_impl(const char* path, const char* basis_pa
|
|||||||
return true;
|
return true;
|
||||||
return file_to_disk_secure_attrs_counted(
|
return file_to_disk_secure_attrs_counted(
|
||||||
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
|
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
|
||||||
xattrs, fake_super, false, temp_dir, dirs_created, count_floor);
|
xattrs, fake_super, false, temp_dir, dirs_created, count_floor, 0, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (scratch_dirfd >= 0)
|
if (scratch_dirfd >= 0)
|
||||||
|
|||||||
+24
-7
@@ -177,18 +177,35 @@ bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
|
|||||||
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
const char* temp_dir);
|
const char* temp_dir);
|
||||||
|
/* Receive one length-prefixed whole-file data frame, streaming the payload
|
||||||
|
* through a bounded buffer when it (or its known logical size) exceeds
|
||||||
|
* `stream_limit`. On success exactly one of *out_buffer / *out_spool is set:
|
||||||
|
* - *out_buffer: the historical charged whole-buffer Data (caller destroys);
|
||||||
|
* - *out_spool: an owned temp path holding the payload, installed through the
|
||||||
|
* File's basis_copy field with File.data_spool set so file_destroy unlinks
|
||||||
|
* it. The destination policy/metadata/atomic-store handling is then the
|
||||||
|
* existing bounded-buffer basis install (file_copy_basis_stream_attrs).
|
||||||
|
* `expected_size` (0 = unknown) is the logical size from the check frame;
|
||||||
|
* `dest_path` locates the spool next to the destination; `compress` selects
|
||||||
|
* incremental decompression. Returns false on any framing/I/O/size error. */
|
||||||
|
bool file_receive_payload(int fd, bool compress, unsigned long long expected_size,
|
||||||
|
const char* dest_path, unsigned long long stream_limit, Data** out_buffer,
|
||||||
|
char** out_spool, unsigned long long* out_size);
|
||||||
|
/* Create a confined spool temp file next to `dest_path`; returns an open write
|
||||||
|
* fd and an owned absolute path (to be installed via File.basis_copy with
|
||||||
|
* File.data_spool set, and unlinked by file_destroy). */
|
||||||
|
int file_spool_for_payload(const char* dest_path, char** out_spool_path);
|
||||||
/* Protocol 2.28.0 receiver-stat variants: like the two above but additionally
|
/* Protocol 2.28.0 receiver-stat variants: like the two above but additionally
|
||||||
* report through `dirs_created` (when non-NULL) how many parent directories the
|
* report through `dirs_created` (when non-NULL) how many parent directories the
|
||||||
* confined secure walk had to create that lie strictly below `count_floor` (a
|
* confined secure walk had to create that lie strictly below `count_floor` (a
|
||||||
* receive-root-relative prefix, or NULL for all). Used to reproduce rsync's
|
* receive-root-relative prefix, or NULL for all). Used to reproduce rsync's
|
||||||
* `Number of created files` directory count on a fresh destination. */
|
* `Number of created files` directory count on a fresh destination. */
|
||||||
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
|
bool file_to_disk_secure_attrs_counted(
|
||||||
unsigned long long data_size, bool inplace, bool sparse,
|
const char* path, const void* data, unsigned long long data_size, bool inplace, bool sparse,
|
||||||
bool preallocate, const FileMetadata* metadata,
|
bool preallocate, const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||||
FileAttrPolicy policy, bool update, bool no_replace,
|
bool no_replace, bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
bool keep_partial, const char* temp_dir, unsigned* dirs_created, const char* count_floor,
|
||||||
bool keep_partial, const char* temp_dir,
|
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor);
|
||||||
unsigned* dirs_created, const char* count_floor);
|
|
||||||
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
|
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
|
||||||
const void* data, unsigned long long data_size,
|
const void* data, unsigned long long data_size,
|
||||||
bool preallocate, const FileMetadata* metadata,
|
bool preallocate, const FileMetadata* metadata,
|
||||||
|
|||||||
+72
-29
@@ -58,36 +58,41 @@ File* file_receive(const Config* config, int file_descriptor) {
|
|||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
Data* file_data = receive_data_limited(file_descriptor, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
bool compress =
|
||||||
if (file_data == NULL) {
|
config->use_compression && !compression_should_skip_with_suffixes(
|
||||||
|
file->path, config->skip_compress_suffixes,
|
||||||
|
config->skip_compress_set ? config->skip_compress_count : -1);
|
||||||
|
char* dest_path = path_cat(config->receive_root_directory, file->path);
|
||||||
|
if (!dest_path) {
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
if (config->use_compression &&
|
Data* buffer = NULL;
|
||||||
!compression_should_skip_with_suffixes(file->path, config->skip_compress_suffixes,
|
char* spool = NULL;
|
||||||
config->skip_compress_set ? config->skip_compress_count
|
unsigned long long size = 0;
|
||||||
: -1)) {
|
bool ok = file_receive_payload(file_descriptor, compress, 0, dest_path,
|
||||||
Data* file_data_uncompressed = data_decompress_limited(file_data, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
protocol_whole_file_receive_limit(), &buffer, &spool, &size);
|
||||||
ProtocolSession* owner = file_data->owner;
|
free(dest_path);
|
||||||
data_destroy(file_data);
|
if (!ok) {
|
||||||
if (file_data_uncompressed == NULL) {
|
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
if (!data_charge_session(file_data_uncompressed, owner, file_data_uncompressed->size)) {
|
if (spool) {
|
||||||
data_destroy(file_data_uncompressed);
|
Data* reserved = data_create_reserve((size_t)size);
|
||||||
|
if (!reserved) {
|
||||||
|
unlink(spool);
|
||||||
|
free(spool);
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
if (file_data_uncompressed->size > MAX_FILE_DATA_SIZE) {
|
|
||||||
data_destroy(file_data_uncompressed);
|
|
||||||
file_destroy(file);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
file_data = file_data_uncompressed;
|
|
||||||
}
|
|
||||||
data_destroy(file->data);
|
data_destroy(file->data);
|
||||||
file->data = file_data;
|
file->data = reserved;
|
||||||
|
file->basis_copy = spool;
|
||||||
|
file->data_spool = true;
|
||||||
|
} else {
|
||||||
|
data_destroy(file->data);
|
||||||
|
file->data = buffer;
|
||||||
|
}
|
||||||
return file;
|
return file;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -97,12 +102,14 @@ bool dir_metadata_should_capture(const Config* config) {
|
|||||||
/* Directory metadata is captured when a directory attribute is actually
|
/* Directory metadata is captured when a directory attribute is actually
|
||||||
* requested: -p/--perms (directory modes), -t/--times (directory mtimes,
|
* requested: -p/--perms (directory modes), -t/--times (directory mtimes,
|
||||||
* unless -O/--omit-dir-times suppresses them), -o/-g (directory ownership),
|
* unless -O/--omit-dir-times suppresses them), -o/-g (directory ownership),
|
||||||
* or -X/-A (directory xattrs/ACLs). --atimes/-U alone does not pull
|
* -X/-A (directory xattrs/ACLs), or --fake-super (whose reserved %stat record
|
||||||
* directory metadata (matching the original dir-time bundle). */
|
* is written on the directory itself, so its metadata must travel).
|
||||||
|
* --atimes/-U alone does not pull directory metadata (matching the original
|
||||||
|
* dir-time bundle). */
|
||||||
return config && config->use_metadata &&
|
return config && config->use_metadata &&
|
||||||
(config->preserve_perms || (config->preserve_times && !config->omit_dir_times) ||
|
(config->preserve_perms || (config->preserve_times && !config->omit_dir_times) ||
|
||||||
config->preserve_owner || config->preserve_group || config->preserve_xattrs ||
|
config->preserve_owner || config->preserve_group || config->preserve_xattrs ||
|
||||||
config->preserve_acls);
|
config->preserve_acls || config->fake_super);
|
||||||
}
|
}
|
||||||
|
|
||||||
void dir_time_list_init(DirTimeList* list) {
|
void dir_time_list_init(DirTimeList* list) {
|
||||||
@@ -200,12 +207,19 @@ void dir_metadata_list_apply(const DirTimeList* list, const char* root_directory
|
|||||||
bool apply_times = config->preserve_times && !config->omit_dir_times;
|
bool apply_times = config->preserve_times && !config->omit_dir_times;
|
||||||
bool apply_mode = config->preserve_perms;
|
bool apply_mode = config->preserve_perms;
|
||||||
bool apply_xattrs = config->use_xattrs;
|
bool apply_xattrs = config->use_xattrs;
|
||||||
|
bool apply_fake_super = config->fake_super;
|
||||||
/* Ownership is applied through the active identity snapshot (which no-ops
|
/* Ownership is applied through the active identity snapshot (which no-ops
|
||||||
* unless an ownership request is active), and xattrs only when -X/-A was
|
* unless an ownership request is active), xattrs only when -X/-A was
|
||||||
* negotiated. Times/mode keep their own per-attribute gates. */
|
* negotiated, and the --fake-super record whenever the flag is active.
|
||||||
bool have_any = apply_times || apply_mode || apply_xattrs || identity_active_enabled();
|
* Times/mode keep their own per-attribute gates. */
|
||||||
|
bool have_any =
|
||||||
|
apply_times || apply_mode || apply_xattrs || apply_fake_super || identity_active_enabled();
|
||||||
if (!have_any)
|
if (!have_any)
|
||||||
return;
|
return;
|
||||||
|
/* Built once: the --fake-super replay uses it to apply only the recorded
|
||||||
|
* permission bits (the special bits stay in the record, exactly like the
|
||||||
|
* regular-file fake-super receiver). */
|
||||||
|
FileAttrPolicy policy = file_attr_policy_from_config(config);
|
||||||
for (size_t i = 0; i < list->count; i++) {
|
for (size_t i = 0; i < list->count; i++) {
|
||||||
char* dir_path = path_cat(root_directory, list->paths[i]);
|
char* dir_path = path_cat(root_directory, list->paths[i]);
|
||||||
if (!dir_path)
|
if (!dir_path)
|
||||||
@@ -255,10 +269,12 @@ void dir_metadata_list_apply(const DirTimeList* list, const char* root_directory
|
|||||||
free(escaped_path);
|
free(escaped_path);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (apply_mode) {
|
/* The final directory mode (after any --chmod) is computed once so the
|
||||||
|
--fake-super record can carry it even when the on-disk replay is
|
||||||
|
restricted to the permission bits below. */
|
||||||
mode_t dir_mode = list->entries[i].mode;
|
mode_t dir_mode = list->entries[i].mode;
|
||||||
bool mode_ready = true;
|
bool mode_ready = true;
|
||||||
if (config->chmod_spec && *config->chmod_spec &&
|
if (apply_mode && config->chmod_spec && *config->chmod_spec &&
|
||||||
!chmod_apply(dir_mode, config->chmod_spec, &dir_mode)) {
|
!chmod_apply(dir_mode, config->chmod_spec, &dir_mode)) {
|
||||||
char* escaped_path = output_escape(dir_path, log_get_8_bit_output());
|
char* escaped_path = output_escape(dir_path, log_get_8_bit_output());
|
||||||
log_message(LOG_LEVEL_WARNING, "Failed to apply --chmod to directory %s",
|
log_message(LOG_LEVEL_WARNING, "Failed to apply --chmod to directory %s",
|
||||||
@@ -266,7 +282,13 @@ void dir_metadata_list_apply(const DirTimeList* list, const char* root_directory
|
|||||||
free(escaped_path);
|
free(escaped_path);
|
||||||
mode_ready = false;
|
mode_ready = false;
|
||||||
}
|
}
|
||||||
if (mode_ready) {
|
/* Under --fake-super the normal fchmod below still applies the mode, but
|
||||||
|
the fake-super replay that follows narrows the on-disk result to the
|
||||||
|
recorded permission bits (the full mode, including setuid/setgid/sticky,
|
||||||
|
lives only in the record). Keeping the normal fchmod first means a
|
||||||
|
filesystem without xattr support still gets the directory mode rather than
|
||||||
|
silently losing it. */
|
||||||
|
if (apply_mode && mode_ready) {
|
||||||
/* rsync -p copies the source directory mode exactly, including
|
/* rsync -p copies the source directory mode exactly, including
|
||||||
* group/other write and the setgid/sticky bits. Setuid/setgid/sticky
|
* group/other write and the setgid/sticky bits. Setuid/setgid/sticky
|
||||||
* are super-user activities: when the connection forbade them
|
* are super-user activities: when the connection forbade them
|
||||||
@@ -286,6 +308,19 @@ void dir_metadata_list_apply(const DirTimeList* list, const char* root_directory
|
|||||||
free(escaped_path);
|
free(escaped_path);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/* --fake-super: park the directory's full stat (rsync 3.4.1's exact
|
||||||
|
grammar) on the directory ITSELF, then replay only the recorded
|
||||||
|
permission bits fd-relative. The special bits live only in the record
|
||||||
|
and the recorded ownership is never real-chowned: the resolved ids are
|
||||||
|
stored for a later privileged restore, exactly like the file path. Runs
|
||||||
|
before the xattr apply so a mode change cannot clobber the ACL mask. */
|
||||||
|
if (apply_fake_super && dir_fd >= 0) {
|
||||||
|
uint32_t store_uid = 0;
|
||||||
|
uint32_t store_gid = 0;
|
||||||
|
identity_resolve_storage_ids((int32_t)list->entries[i].uid, (int32_t)list->entries[i].gid,
|
||||||
|
&store_uid, &store_gid);
|
||||||
|
fake_super_store_fd(dir_fd, store_uid, store_gid, (uint32_t)dir_mode, 0, 0);
|
||||||
|
fake_super_restore_fd(dir_fd, policy);
|
||||||
}
|
}
|
||||||
/* xattrs/ACLs last: a mode change can rewrite the ACL mask, so the ACL
|
/* xattrs/ACLs last: a mode change can rewrite the ACL mask, so the ACL
|
||||||
xattrs must be (re)applied after fchmod. */
|
xattrs must be (re)applied after fchmod. */
|
||||||
@@ -482,6 +517,14 @@ File* file_receive_symlink(int file_descriptor, const Config* config) {
|
|||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/* Symlink xattrs/ACLs (-X/-A) arrive in the same trailing block as the other
|
||||||
|
entry kinds; the block is present iff use_xattrs (which itself implies
|
||||||
|
use_metadata, so the metadata frame above is always consumed first). */
|
||||||
|
if (config && !receive_file_xattrs(file, file_descriptor, config)) {
|
||||||
|
file_destroy(file);
|
||||||
|
free(target);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
file->is_symlink = true;
|
file->is_symlink = true;
|
||||||
file->symlink_target = target;
|
file->symlink_target = target;
|
||||||
return file;
|
return file;
|
||||||
|
|||||||
+154
-39
@@ -2,6 +2,7 @@
|
|||||||
#include <ctype.h>
|
#include <ctype.h>
|
||||||
#include <dirent.h>
|
#include <dirent.h>
|
||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
|
#include <limits.h>
|
||||||
#include <libgen.h>
|
#include <libgen.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
@@ -198,6 +199,56 @@ static FileSaveResult hardlink_sibling_absent_first(const char* destination_path
|
|||||||
return FILE_SAVE_ERROR;
|
return FILE_SAVE_ERROR;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Resolve a user-supplied --temp-dir against the receive `root`.
|
||||||
|
*
|
||||||
|
* A relative, traversal-free name is joined below the root (the historical
|
||||||
|
* behavior). An absolute path is canonicalized with realpath(3) and accepted
|
||||||
|
* only when it lies inside the canonicalized receive root; this is the parity
|
||||||
|
* win over rejecting every absolute path, without weakening the confinement
|
||||||
|
* invariant: an absolute path that escapes the root (including one reached
|
||||||
|
* through a symlinked component) is still refused. A `..` component in a
|
||||||
|
* relative name is likewise refused. The root itself is treated as an
|
||||||
|
* absolute path free of `..`; its realpath() resolves any symlinks so the
|
||||||
|
* prefix comparison is against one canonical form.
|
||||||
|
*
|
||||||
|
* Logs a clear error on rejection (the scratch dir must stay confined) and
|
||||||
|
* returns a newly allocated scratch path, or NULL on rejection/allocation
|
||||||
|
* failure. */
|
||||||
|
static char* file_save_resolve_temp_dir(const char* root, const char* temp_dir) {
|
||||||
|
if (temp_dir[0] != '/') {
|
||||||
|
if (has_path_traversal(temp_dir)) {
|
||||||
|
log_message(
|
||||||
|
LOG_LEVEL_ERROR,
|
||||||
|
"receiver rejected --temp-dir '%s': a '..' component would escape the receive root",
|
||||||
|
temp_dir);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
return path_cat(root, temp_dir);
|
||||||
|
}
|
||||||
|
char canonical_temp[PATH_MAX];
|
||||||
|
char canonical_root[PATH_MAX];
|
||||||
|
if (!realpath(temp_dir, canonical_temp)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"receiver rejected --temp-dir '%s': could not resolve the absolute path (%s)",
|
||||||
|
temp_dir, strerror(errno));
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
if (!realpath(root, canonical_root)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"receiver rejected --temp-dir '%s': could not resolve the receive root (%s)",
|
||||||
|
temp_dir, strerror(errno));
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
if (strcmp(canonical_root, "/") != 0 && !path_is_within_root(canonical_root, canonical_temp)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"receiver rejected --temp-dir '%s': an absolute temp dir must be inside the "
|
||||||
|
"receive root '%s'",
|
||||||
|
temp_dir, canonical_root);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
return str_dup(canonical_temp);
|
||||||
|
}
|
||||||
|
|
||||||
/* Install a --hard-links/-H sibling: the destination entry is atomically
|
/* Install a --hard-links/-H sibling: the destination entry is atomically
|
||||||
replaced (temp + rename) with a hard link to the group's first member. The
|
replaced (temp + rename) with a hard link to the group's first member. The
|
||||||
first member is guaranteed already installed at `hardlink_target` under the
|
first member is guaranteed already installed at `hardlink_target` under the
|
||||||
@@ -303,17 +354,13 @@ static FileSaveResult file_save_hardlink_sibling(const char* root_directory, con
|
|||||||
free(destination_path);
|
free(destination_path);
|
||||||
return absent_result;
|
return absent_result;
|
||||||
}
|
}
|
||||||
/* Resolve a relative --temp-dir under the destination root, exactly as the
|
/* Resolve the --temp-dir under the destination root, exactly as the primary
|
||||||
* primary save path does; an absolute or `..`-escaping value is rejected. */
|
* save path does: a relative dir joins below the root, an absolute dir is
|
||||||
|
* accepted only when it canonicalizes inside the root, and any escaping value
|
||||||
|
* is rejected. */
|
||||||
char* resolved_temp = NULL;
|
char* resolved_temp = NULL;
|
||||||
if (cfg->temp_dir) {
|
if (cfg->temp_dir) {
|
||||||
if (cfg->temp_dir[0] == '/' || has_path_traversal(cfg->temp_dir)) {
|
resolved_temp = file_save_resolve_temp_dir(root_directory, cfg->temp_dir);
|
||||||
free(content);
|
|
||||||
free(first_disk);
|
|
||||||
free(destination_path);
|
|
||||||
return FILE_SAVE_ERROR;
|
|
||||||
}
|
|
||||||
resolved_temp = path_cat(root_directory, cfg->temp_dir);
|
|
||||||
if (!resolved_temp) {
|
if (!resolved_temp) {
|
||||||
free(content);
|
free(content);
|
||||||
free(first_disk);
|
free(first_disk);
|
||||||
@@ -353,11 +400,14 @@ bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode) {
|
|||||||
/* ---- Device/special node RECREATION (--devices/--specials), receiver side ----
|
/* ---- Device/special node RECREATION (--devices/--specials), receiver side ----
|
||||||
*
|
*
|
||||||
* Privilege gating: making a real device node requires CAP_MKNOD (root); making
|
* Privilege gating: making a real device node requires CAP_MKNOD (root); making
|
||||||
* a FIFO works unprivileged (mkfifo). When the receiver lacks the capability,
|
* a FIFO works unprivileged (mkfifo). A device node whose mknodat() fails with
|
||||||
* mknodat() fails with EPERM and the entry is SKIPPED with a warning -- the
|
* EPERM/EACCES is a PER-ENTRY failure (rsync parity: rsync reports the mknod
|
||||||
* whole transfer must NOT abort just because the environment cannot make the
|
* failure, still transfers the rest, and exits partial, code 23), reported as
|
||||||
* node. CI runs non-root, so device creation is expected to skip there and
|
* FILE_SAVE_FAILED so the receiver counts it and continues. Only the
|
||||||
* only a FIFO is honestly assertable unprivileged.
|
* unprivileged FIFO/socket (--specials) path keeps the best-effort skip,
|
||||||
|
* because those are normally creatable without privilege and a failure there is
|
||||||
|
* environmental. CI runs non-root, so device creation is expected to fail
|
||||||
|
* there; only a FIFO is honestly assertable unprivileged.
|
||||||
*
|
*
|
||||||
* Confinement: the parent directory is opened fd-relative below the receive
|
* Confinement: the parent directory is opened fd-relative below the receive
|
||||||
* root (file_open_secure_parent: O_NOFOLLOW, no "..", root-checked) and the
|
* root (file_open_secure_parent: O_NOFOLLOW, no "..", root-checked) and the
|
||||||
@@ -495,13 +545,32 @@ static FileSaveResult file_save_special_to_disk(const char* root_directory, cons
|
|||||||
node_kind, escaped_path ? escaped_path : "<allocation failed>");
|
node_kind, escaped_path ? escaped_path : "<allocation failed>");
|
||||||
free(escaped_path);
|
free(escaped_path);
|
||||||
} else if (errno == EPERM || errno == EACCES) {
|
} else if (errno == EPERM || errno == EACCES) {
|
||||||
/* Missing CAP_MKNOD / parent write permission: the environment cannot
|
|
||||||
create the node, so skip instead of failing the whole run. */
|
|
||||||
char* escaped_path = output_escape(file->path, log_get_8_bit_output());
|
char* escaped_path = output_escape(file->path, log_get_8_bit_output());
|
||||||
|
const char* shown_path = escaped_path ? escaped_path : "<allocation failed>";
|
||||||
|
if (is_char || is_blk) {
|
||||||
|
/* rsync parity: a device node that cannot be created (no CAP_MKNOD, or
|
||||||
|
* super-user activities not permitted) is a per-entry failure. rsync
|
||||||
|
* logs `mknod ".../node" failed: ...`, still transfers the remaining
|
||||||
|
* files, and exits partial (23); FastSync logs it, counts it, and
|
||||||
|
* continues rather than aborting the stream. FIFO/socket creation
|
||||||
|
* (--specials) keeps the best-effort skip path below. */
|
||||||
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"cannot create %s %s: %s\n"
|
||||||
|
" --devices node creation needs privilege (CAP_MKNOD)",
|
||||||
|
node_kind, shown_path, strerror(errno));
|
||||||
|
free(escaped_path);
|
||||||
|
close(parent_fd);
|
||||||
|
free(leaf);
|
||||||
|
free(destination);
|
||||||
|
return FILE_SAVE_FAILED;
|
||||||
|
}
|
||||||
|
/* Missing CAP_MKNOD / parent write permission for a FIFO/socket: the
|
||||||
|
environment cannot create the node, so skip instead of failing the
|
||||||
|
whole run. */
|
||||||
log_message(LOG_LEVEL_WARNING,
|
log_message(LOG_LEVEL_WARNING,
|
||||||
"skipping %s: cannot create %s node (%s)\n"
|
"skipping %s: cannot create %s node (%s)\n"
|
||||||
" --devices/--specials node creation needs privilege (CAP_MKNOD)",
|
" --specials node creation needs privilege (CAP_MKNOD)",
|
||||||
escaped_path ? escaped_path : "<allocation failed>", node_kind, strerror(errno));
|
shown_path, node_kind, strerror(errno));
|
||||||
free(escaped_path);
|
free(escaped_path);
|
||||||
} else {
|
} else {
|
||||||
char* escaped_path = output_escape(file->path, log_get_8_bit_output());
|
char* escaped_path = output_escape(file->path, log_get_8_bit_output());
|
||||||
@@ -748,6 +817,21 @@ static FileSaveResult file_save_directory_to_disk(const FileSavePlan* plan, bool
|
|||||||
} else if (ok && identity_copy_as_active()) {
|
} else if (ok && identity_copy_as_active()) {
|
||||||
ok = false;
|
ok = false;
|
||||||
}
|
}
|
||||||
|
/* --fake-super: park the directory's full stat in rsync's reserved
|
||||||
|
user.rsync.%stat xattr as soon as the directory exists. This makes even a
|
||||||
|
direct file_save_to_disk_full() caller -- which never runs the deferred
|
||||||
|
DirTimeList pass -- produce an rsync-readable fake-super record. The record
|
||||||
|
carries the full mode/uid/gid; the permission bits are replayed by the
|
||||||
|
deferred pass (never inline, so a restrictive mode cannot block child
|
||||||
|
creation) and the recorded ownership is never real-chowned. Best-effort:
|
||||||
|
fake_super_store_fd() logs and skips a failure, never failing the entry. */
|
||||||
|
if (ok && plan->config && plan->config->fake_super && file->metadata && dir_fd >= 0) {
|
||||||
|
uint32_t store_uid = 0;
|
||||||
|
uint32_t store_gid = 0;
|
||||||
|
identity_resolve_storage_ids((int32_t)file->metadata->uid, (int32_t)file->metadata->gid,
|
||||||
|
&store_uid, &store_gid);
|
||||||
|
fake_super_store_fd(dir_fd, store_uid, store_gid, (uint32_t)file->metadata->mode, 0, 0);
|
||||||
|
}
|
||||||
/* The final source MODE is deliberately NOT applied inline. A restrictive
|
/* The final source MODE is deliberately NOT applied inline. A restrictive
|
||||||
source mode (for example 0555) would make the directory unwritable before
|
source mode (for example 0555) would make the directory unwritable before
|
||||||
its children are created, so a non-root receiver fails each child with
|
its children are created, so a non-root receiver fails each child with
|
||||||
@@ -854,6 +938,21 @@ static FileSaveResult file_save_symlink_to_disk(const FileSavePlan* plan, bool*
|
|||||||
ok = file_restore_symlink_metadata(link_path, file->metadata, link_policy,
|
ok = file_restore_symlink_metadata(link_path, file->metadata, link_policy,
|
||||||
config->omit_link_times);
|
config->omit_link_times);
|
||||||
}
|
}
|
||||||
|
/* -X/-A: apply the symlink's OWN xattrs with a no-follow primitive. The
|
||||||
|
confined parent directory is the anchor and the final component is applied
|
||||||
|
with lsetxattr, so the referent is never touched. Best-effort: on Linux
|
||||||
|
the VFS refuses xattrs on symlinks, so this is normally a no-op. Hoist the
|
||||||
|
empty-list check so the common Linux case (NULL/empty xattrs) does not pay
|
||||||
|
an open/close of the parent per symlink. */
|
||||||
|
if (ok && config && config->use_xattrs && file->xattrs && file->xattrs->count > 0) {
|
||||||
|
char* leaf = NULL;
|
||||||
|
int parent_fd = file_open_secure_parent(link_path, &leaf, false);
|
||||||
|
if (parent_fd >= 0) {
|
||||||
|
xattr_apply_path_nofollow(parent_fd, leaf, file->xattrs, config->preserve_acls);
|
||||||
|
close(parent_fd);
|
||||||
|
}
|
||||||
|
free(leaf);
|
||||||
|
}
|
||||||
if (ok && created && !link_existed)
|
if (ok && created && !link_existed)
|
||||||
*created = true;
|
*created = true;
|
||||||
free(link_path);
|
free(link_path);
|
||||||
@@ -868,6 +967,14 @@ static bool file_save_try_special_dispatch(const FileSavePlan* plan, bool* creat
|
|||||||
/* Device/special node (--devices/--specials): recreate the node instead of
|
/* Device/special node (--devices/--specials): recreate the node instead of
|
||||||
writing content (privilege-gated, confined, rdev-validated). */
|
writing content (privilege-gated, confined, rdev-validated). */
|
||||||
if (file->is_special) {
|
if (file->is_special) {
|
||||||
|
/* Under --fake-super rsync never mknod()s a device: it writes a regular
|
||||||
|
empty file and records the real rdev in user.rsync.%stat. Fall through to
|
||||||
|
the ordinary writer so the device round-trips (its S_IFMT mode bits and
|
||||||
|
rdev are parked in the record). Without --fake-super the node is
|
||||||
|
recreated (or, when privilege is refused, handled per-entry). */
|
||||||
|
mode_t special_mode = file->metadata ? file->metadata->mode : 0;
|
||||||
|
if (config && config->fake_super && (S_ISCHR(special_mode) || S_ISBLK(special_mode)))
|
||||||
|
return false;
|
||||||
*out = file_save_special_to_disk(plan->root_directory, file, config, created);
|
*out = file_save_special_to_disk(plan->root_directory, file, config, created);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
@@ -896,17 +1003,17 @@ static bool file_save_try_special_dispatch(const FileSavePlan* plan, bool* creat
|
|||||||
and disk paths. Returns false on an invalid/escaping option or an
|
and disk paths. Returns false on an invalid/escaping option or an
|
||||||
allocation failure (the caller routes to the cleanup epilogue). */
|
allocation failure (the caller routes to the cleanup epilogue). */
|
||||||
static bool file_save_resolve_paths(FileSavePlan* plan) {
|
static bool file_save_resolve_paths(FileSavePlan* plan) {
|
||||||
/* These options arrive from the client. --backup-dir, --partial-dir and
|
/* These options arrive from the client. --backup-dir and --partial-dir are
|
||||||
--temp-dir are names below the server root, never independent filesystem
|
names below the server root, never independent filesystem roots: an
|
||||||
roots: an absolute or `..`-escaping value is rejected outright (rsync's
|
absolute or `..`-escaping value is rejected outright. --temp-dir is
|
||||||
daemon confines temp-dir to the module the same way). A relative temp dir
|
resolved by file_save_resolve_temp_dir below: a relative name joins below
|
||||||
is resolved under the receive root below; if that resolution still lands on
|
the root, an absolute name is accepted only when it canonicalizes inside
|
||||||
a different filesystem than the destination the install falls back to a
|
the root, and any escaping value is rejected. If the resolved scratch dir
|
||||||
non-atomic copy (see file_to_disk_secure_impl), never an abort. */
|
still lands on a different filesystem than the destination the install
|
||||||
|
falls back to a non-atomic copy (see file_to_disk_secure_impl), never an
|
||||||
|
abort. */
|
||||||
if ((plan->backup_dir && (plan->backup_dir[0] == '/' || has_path_traversal(plan->backup_dir))) ||
|
if ((plan->backup_dir && (plan->backup_dir[0] == '/' || has_path_traversal(plan->backup_dir))) ||
|
||||||
(plan->partial_dir &&
|
(plan->partial_dir && (plan->partial_dir[0] == '/' || has_path_traversal(plan->partial_dir))))
|
||||||
(plan->partial_dir[0] == '/' || has_path_traversal(plan->partial_dir))) ||
|
|
||||||
(plan->temp_dir && (plan->temp_dir[0] == '/' || has_path_traversal(plan->temp_dir))))
|
|
||||||
return false;
|
return false;
|
||||||
if (plan->backup_dir &&
|
if (plan->backup_dir &&
|
||||||
!(plan->confined_backup = path_cat(plan->root_directory, plan->backup_dir)))
|
!(plan->confined_backup = path_cat(plan->root_directory, plan->backup_dir)))
|
||||||
@@ -914,6 +1021,9 @@ static bool file_save_resolve_paths(FileSavePlan* plan) {
|
|||||||
if (plan->partial_dir &&
|
if (plan->partial_dir &&
|
||||||
!(plan->confined_partial = path_cat(plan->root_directory, plan->partial_dir)))
|
!(plan->confined_partial = path_cat(plan->root_directory, plan->partial_dir)))
|
||||||
return false;
|
return false;
|
||||||
|
if (plan->temp_dir &&
|
||||||
|
!(plan->confined_temp = file_save_resolve_temp_dir(plan->root_directory, plan->temp_dir)))
|
||||||
|
return false;
|
||||||
|
|
||||||
const char* actual_root = plan->use_partial_root ? plan->confined_partial : plan->root_directory;
|
const char* actual_root = plan->use_partial_root ? plan->confined_partial : plan->root_directory;
|
||||||
plan->destination_path = path_cat(plan->root_directory, plan->file->path);
|
plan->destination_path = path_cat(plan->root_directory, plan->file->path);
|
||||||
@@ -1039,7 +1149,7 @@ static bool file_save_install_data(FileSavePlan* plan, const FileMetadata* metad
|
|||||||
config && config->preallocate, metadata, plan->policy, config && config->update,
|
config && config->preallocate, metadata, plan->policy, config && config->update,
|
||||||
config && config->ignore_existing, config && config->use_fsync, file->xattrs,
|
config && config->ignore_existing, config && config->use_fsync, file->xattrs,
|
||||||
config ? config->fake_super : false, config ? config->partial : false, plan->confined_temp,
|
config ? config->fake_super : false, config ? config->partial : false, plan->confined_temp,
|
||||||
created_dirs, count_floor);
|
created_dirs, count_floor, (uint32_t)file->rdev_major, (uint32_t)file->rdev_minor);
|
||||||
}
|
}
|
||||||
free(count_floor);
|
free(count_floor);
|
||||||
return ok;
|
return ok;
|
||||||
@@ -1146,17 +1256,22 @@ FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File*
|
|||||||
|
|
||||||
/* A configured --temp-dir sends the temporary working copy to a scratch
|
/* A configured --temp-dir sends the temporary working copy to a scratch
|
||||||
directory; the engine then atomically renames the completed file into the
|
directory; the engine then atomically renames the completed file into the
|
||||||
final destination directory. A relative temp dir is resolved under the
|
final destination directory. The scratch path was confined to the receive
|
||||||
receive root and must already exist (an absolute or `..`-escaping value was
|
root (and canonicalized) in file_save_resolve_paths and must already exist;
|
||||||
rejected above); the engine falls back to a non-atomic copy on EXDEV. The
|
the engine falls back to a non-atomic copy on EXDEV. The partial-dir flow
|
||||||
partial-dir flow already keeps its working copy in a separate directory and
|
already keeps its working copy in a separate directory and --inplace writes
|
||||||
--inplace writes directly, so neither diverts through the scratch dir
|
directly, so neither diverts through the scratch dir (matching rsync, where
|
||||||
(matching rsync, where --inplace/--partial-dir supersede --temp-dir). */
|
--inplace/--partial-dir supersede --temp-dir). */
|
||||||
bool use_temp_dir = plan.temp_dir != NULL && !plan.inplace && !plan.use_partial_root;
|
/* --inplace and --partial-dir supersede --temp-dir in rsync, so the scratch
|
||||||
|
dir is not used on those paths. The value was still validated/confined by
|
||||||
|
file_save_resolve_paths; drop the resolved path so it is never handed to the
|
||||||
|
install engine. */
|
||||||
|
if (plan.confined_temp && (plan.inplace || plan.use_partial_root)) {
|
||||||
|
free(plan.confined_temp);
|
||||||
|
plan.confined_temp = NULL;
|
||||||
|
}
|
||||||
|
bool use_temp_dir = plan.confined_temp != NULL;
|
||||||
if (use_temp_dir) {
|
if (use_temp_dir) {
|
||||||
plan.confined_temp = path_cat(root_directory, plan.temp_dir);
|
|
||||||
if (!plan.confined_temp)
|
|
||||||
goto out;
|
|
||||||
/* A user-supplied trailing slash would leave the scratch path ending in
|
/* A user-supplied trailing slash would leave the scratch path ending in
|
||||||
"/", which has no final component to create/open. Normalize it away. */
|
"/", which has no final component to create/open. Normalize it away. */
|
||||||
size_t temp_len = strlen(plan.confined_temp);
|
size_t temp_len = strlen(plan.confined_temp);
|
||||||
|
|||||||
+10
-2
@@ -12,8 +12,16 @@
|
|||||||
|
|
||||||
/* Outcome of a single file_save_to_disk operation. The receiver needs to
|
/* Outcome of a single file_save_to_disk operation. The receiver needs to
|
||||||
distinguish "written" from "skipped" so --remove-source-files can be told
|
distinguish "written" from "skipped" so --remove-source-files can be told
|
||||||
which sources were actually stored. */
|
which sources were actually stored. FILE_SAVE_FAILED is a per-entry failure
|
||||||
typedef enum { FILE_SAVE_ERROR = 0, FILE_SAVE_WRITTEN = 1, FILE_SAVE_SKIPPED = 2 } FileSaveResult;
|
(for example a device node that mknodat() refused with EPERM/EACCES): it is
|
||||||
|
logged and counted by the receiver but does NOT abort the transfer, matching
|
||||||
|
rsync's continue-and-exit-partial behavior. */
|
||||||
|
typedef enum {
|
||||||
|
FILE_SAVE_ERROR = 0,
|
||||||
|
FILE_SAVE_WRITTEN = 1,
|
||||||
|
FILE_SAVE_SKIPPED = 2,
|
||||||
|
FILE_SAVE_FAILED = 3
|
||||||
|
} FileSaveResult;
|
||||||
|
|
||||||
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
|
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
|
||||||
|
|
||||||
|
|||||||
+133
-3
@@ -44,12 +44,138 @@ bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
|
|||||||
send_path, NULL, -1, 0, false);
|
send_path, NULL, -1, 0, false);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Stream-compress a whole source file into a temp file, then transmit it as the
|
||||||
|
* normal length-prefixed data frame. Used when the source was too large to
|
||||||
|
* load (File.data.data == NULL): the source is read in bounded chunks through a
|
||||||
|
* streaming codec, so neither the raw nor the compressed image is held in
|
||||||
|
* memory. The compressed length must be known before the frame is sent (the
|
||||||
|
* wire is length-prefixed), so the stream lands in a private temp file first. */
|
||||||
|
bool file_send_compressed_stream_with_skip(File* file, int file_descriptor, bool use_metadata,
|
||||||
|
int compression_level, bool send_path,
|
||||||
|
char* const* skip_suffixes, int skip_count,
|
||||||
|
int compression_threads, bool send_xattrs) {
|
||||||
|
/* The caller has already decided this file compresses; the skip list is part
|
||||||
|
of the public signature for symmetry with the buffered path. */
|
||||||
|
(void)skip_suffixes;
|
||||||
|
(void)skip_count;
|
||||||
|
if (!file || !file->path || !file->data || file->data->size == 0 || file->data->data != NULL)
|
||||||
|
return false;
|
||||||
|
CompressionAlgo algo = compression_get_algo();
|
||||||
|
CompressionStreamCompressor* compressor =
|
||||||
|
compression_stream_compressor_create(algo, compression_level, compression_threads);
|
||||||
|
if (!compressor) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "streaming compression is not available for this codec; "
|
||||||
|
"the source was not loaded for the buffered path");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
int src = file_open_for_read(file->path);
|
||||||
|
if (src < 0) {
|
||||||
|
compression_stream_compressor_destroy(compressor);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
struct stat src_st;
|
||||||
|
if (fstat(src, &src_st) != 0 || !S_ISREG(src_st.st_mode) ||
|
||||||
|
(unsigned long long)src_st.st_size < file->data->size) {
|
||||||
|
close(src);
|
||||||
|
compression_stream_compressor_destroy(compressor);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
const char* tmpdir = getenv("TMPDIR");
|
||||||
|
if (!tmpdir || tmpdir[0] == '\0')
|
||||||
|
tmpdir = "/tmp";
|
||||||
|
size_t tmplen = strlen(tmpdir) + strlen("/fastsync-z-XXXXXX") + 1;
|
||||||
|
char* tmpl = malloc(tmplen);
|
||||||
|
if (!tmpl) {
|
||||||
|
close(src);
|
||||||
|
compression_stream_compressor_destroy(compressor);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
snprintf(tmpl, tmplen, "%s/fastsync-z-XXXXXX", tmpdir);
|
||||||
|
int tmp_fd = mkstemp(tmpl);
|
||||||
|
if (tmp_fd < 0) {
|
||||||
|
log_perror("Could not create compression temp file");
|
||||||
|
free(tmpl);
|
||||||
|
close(src);
|
||||||
|
compression_stream_compressor_destroy(compressor);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
unlink(tmpl);
|
||||||
|
free(tmpl);
|
||||||
|
|
||||||
|
bool ok = compression_stream_compressor_begin(compressor, file->data->size, tmp_fd);
|
||||||
|
unsigned char buf[64 * 1024];
|
||||||
|
unsigned long long remaining = file->data->size;
|
||||||
|
while (ok && remaining > 0) {
|
||||||
|
size_t want = remaining < sizeof(buf) ? (size_t)remaining : sizeof(buf);
|
||||||
|
ssize_t got = read(src, buf, want);
|
||||||
|
if (got <= 0) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (!compression_stream_compressor_feed(compressor, buf, (size_t)got, tmp_fd))
|
||||||
|
ok = false;
|
||||||
|
remaining -= (unsigned long long)got;
|
||||||
|
}
|
||||||
|
if (ok)
|
||||||
|
ok = compression_stream_compressor_finish(compressor, tmp_fd);
|
||||||
|
close(src);
|
||||||
|
compression_stream_compressor_destroy(compressor);
|
||||||
|
if (!ok) {
|
||||||
|
close(tmp_fd);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
struct stat tmp_st;
|
||||||
|
if (fstat(tmp_fd, &tmp_st) != 0 || tmp_st.st_size < 0) {
|
||||||
|
close(tmp_fd);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
unsigned long long compressed_size = (unsigned long long)tmp_st.st_size;
|
||||||
|
if (lseek(tmp_fd, 0, SEEK_SET) == (off_t)-1) {
|
||||||
|
close(tmp_fd);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
ok = true;
|
||||||
|
if (send_path && !send_wire_str(file_descriptor, file_wire_path(file)))
|
||||||
|
ok = false;
|
||||||
|
if (ok && use_metadata && !metadata_send(file_descriptor, file->metadata))
|
||||||
|
ok = false;
|
||||||
|
if (ok && send_xattrs && !xattr_send(file_descriptor, file ? file->xattrs : NULL))
|
||||||
|
ok = false;
|
||||||
|
if (ok && !send_n_data(file_descriptor, &compressed_size, sizeof(compressed_size)))
|
||||||
|
ok = false;
|
||||||
|
unsigned long long left = compressed_size;
|
||||||
|
while (ok && left > 0) {
|
||||||
|
size_t want = left < sizeof(buf) ? (size_t)left : sizeof(buf);
|
||||||
|
ssize_t got = read(tmp_fd, buf, want);
|
||||||
|
if (got <= 0 || !send_n_data(file_descriptor, buf, (size_t)got)) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
left -= (unsigned long long)got;
|
||||||
|
}
|
||||||
|
close(tmp_fd);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
|
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
|
||||||
int compression_level, bool send_path,
|
int compression_level, bool send_path,
|
||||||
char* const* skip_suffixes, int skip_count,
|
char* const* skip_suffixes, int skip_count,
|
||||||
int compression_threads, bool send_xattrs) {
|
int compression_threads, bool send_xattrs) {
|
||||||
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data))
|
if (!file || !file->path || !file->data)
|
||||||
return false;
|
return false;
|
||||||
|
/* A source too large to load is streamed: compression streams through a
|
||||||
|
* temp file, no compression must have taken the sendfile path instead. */
|
||||||
|
if (file->data->size != 0 && file->data->data == NULL) {
|
||||||
|
if (compression_level <= 0 ||
|
||||||
|
compression_should_skip_with_suffixes(file->path, skip_suffixes, skip_count))
|
||||||
|
return false;
|
||||||
|
return file_send_compressed_stream_with_skip(file, file_descriptor, use_metadata,
|
||||||
|
compression_level, send_path, skip_suffixes,
|
||||||
|
skip_count, compression_threads, send_xattrs);
|
||||||
|
}
|
||||||
const Data* data_to_send = file->data;
|
const Data* data_to_send = file->data;
|
||||||
Data* compressed_data = NULL;
|
Data* compressed_data = NULL;
|
||||||
if (compression_level > 0 &&
|
if (compression_level > 0 &&
|
||||||
@@ -124,8 +250,12 @@ bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_meta
|
|||||||
}
|
}
|
||||||
|
|
||||||
/* sendfile cannot encrypt TLS records. Keep the framing identical but
|
/* sendfile cannot encrypt TLS records. Keep the framing identical but
|
||||||
route encrypted transfers through the deadline-aware IO layer. */
|
route encrypted transfers through the deadline-aware IO layer. Resolve
|
||||||
if (io_get_ssl() != NULL) {
|
the transport from the bound session, not the thread-local io_ssl: a
|
||||||
|
worker thread running a TLS transfer has its SSL only on the session it
|
||||||
|
bound, so io_get_ssl() would be NULL there and the raw sendfile() path
|
||||||
|
would be taken on an encrypted socket. */
|
||||||
|
if (protocol_current_ssl() != NULL) {
|
||||||
unsigned char buffer[64 * 1024];
|
unsigned char buffer[64 * 1024];
|
||||||
unsigned long long remaining = file_size;
|
unsigned long long remaining = file_size;
|
||||||
bool ok = true;
|
bool ok = true;
|
||||||
|
|||||||
@@ -13,6 +13,12 @@ bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_
|
|||||||
int compression_level, bool send_path,
|
int compression_level, bool send_path,
|
||||||
char* const* skip_suffixes, int skip_count,
|
char* const* skip_suffixes, int skip_count,
|
||||||
int compression_threads, bool send_xattrs);
|
int compression_threads, bool send_xattrs);
|
||||||
|
/* Stream-compress an unloaded whole source file into a temp file and send it as
|
||||||
|
* the usual data frame (see file_send.c). */
|
||||||
|
bool file_send_compressed_stream_with_skip(File* file, int file_descriptor, bool use_metadata,
|
||||||
|
int compression_level, bool send_path,
|
||||||
|
char* const* skip_suffixes, int skip_count,
|
||||||
|
int compression_threads, bool send_xattrs);
|
||||||
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
|
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
|
||||||
bool send_path);
|
bool send_path);
|
||||||
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
|
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
|
||||||
|
|||||||
@@ -62,6 +62,11 @@ typedef struct {
|
|||||||
* This lets a basis larger than any whole-file bound materialize without
|
* This lets a basis larger than any whole-file bound materialize without
|
||||||
* buffering it; the source metadata on `metadata` is applied afterwards. */
|
* buffering it; the source metadata on `metadata` is applied afterwards. */
|
||||||
char* basis_copy;
|
char* basis_copy;
|
||||||
|
/* Receiver-only. When set, `basis_copy` points at a receiver-created spool
|
||||||
|
* temp file holding a STREAMED whole-file payload (rather than a --copy-dest
|
||||||
|
* basis). file_destroy unlinks it after the install consumes it, so an
|
||||||
|
* over-limit file leaves no scratch behind. */
|
||||||
|
bool data_spool;
|
||||||
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
|
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
|
||||||
* id shared by every member of one source inode (0 = not part of a group).
|
* id shared by every member of one source inode (0 = not part of a group).
|
||||||
* The FIRST member (link_first == true) carries its data on the wire and is
|
* The FIRST member (link_first == true) carries its data on the wire and is
|
||||||
|
|||||||
+333
-90
@@ -21,6 +21,28 @@ void filter_rule_free(FilterRule* rule) {
|
|||||||
free(rule);
|
free(rule);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
FilterRule* filter_rule_clone(const FilterRule* rule) {
|
||||||
|
if (!rule)
|
||||||
|
return NULL;
|
||||||
|
FilterRule* copy = calloc(1, sizeof(FilterRule));
|
||||||
|
if (!copy)
|
||||||
|
return NULL;
|
||||||
|
copy->action = rule->action;
|
||||||
|
copy->sides = rule->sides;
|
||||||
|
copy->anchored = rule->anchored;
|
||||||
|
copy->dir_only = rule->dir_only;
|
||||||
|
copy->negate = rule->negate;
|
||||||
|
copy->perishable = rule->perishable;
|
||||||
|
copy->no_inherit = rule->no_inherit;
|
||||||
|
copy->owner = str_dup(rule->owner ? rule->owner : "");
|
||||||
|
copy->pattern = str_dup(rule->pattern ? rule->pattern : "");
|
||||||
|
if (!copy->owner || !copy->pattern) {
|
||||||
|
filter_rule_free(copy);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
return copy;
|
||||||
|
}
|
||||||
|
|
||||||
FilterRuleList* filter_rule_list_create(void) {
|
FilterRuleList* filter_rule_list_create(void) {
|
||||||
return calloc(1, sizeof(FilterRuleList));
|
return calloc(1, sizeof(FilterRuleList));
|
||||||
}
|
}
|
||||||
@@ -48,37 +70,94 @@ void filter_rule_list_free(FilterRuleList* list) {
|
|||||||
for (int i = 0; i < list->count; i++)
|
for (int i = 0; i < list->count; i++)
|
||||||
filter_rule_free(list->items[i]);
|
filter_rule_free(list->items[i]);
|
||||||
for (int i = 0; i < list->dir_merge_count; i++)
|
for (int i = 0; i < list->dir_merge_count; i++)
|
||||||
free(list->dir_merge_names[i]);
|
free(list->dir_merges[i].name);
|
||||||
free(list->dir_merge_names);
|
free(list->dir_merges);
|
||||||
free(list->items);
|
free(list->items);
|
||||||
free(list);
|
free(list);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Append an implicit exclude rule for the merge file itself (rsync's 'e'
|
||||||
|
* modifier). The rule is owned by the transfer root and matches the basename
|
||||||
|
* anywhere, exactly like rsync's EXCLUDE_SELF (a dual-sided exclude: it hides
|
||||||
|
* the file and protects its destination mirror from --delete). */
|
||||||
|
static bool filter_list_add_exclude_self(FilterRuleList* list, const char* name) {
|
||||||
|
const char* base = strrchr(name, '/');
|
||||||
|
base = base ? base + 1 : name;
|
||||||
|
if (base[0] == '\0')
|
||||||
|
return true;
|
||||||
|
FilterRule* rule = calloc(1, sizeof(FilterRule));
|
||||||
|
if (!rule)
|
||||||
|
return false;
|
||||||
|
rule->action = FILTER_ACTION_EXCLUDE;
|
||||||
|
rule->sides = FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER;
|
||||||
|
rule->pattern = str_dup(base);
|
||||||
|
if (!rule->pattern || !filter_rule_set_owner(rule, "")) {
|
||||||
|
filter_rule_free(rule);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!filter_rule_list_add(list, rule)) {
|
||||||
|
filter_rule_free(rule);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
/* Register a per-directory merge-file basename (for "dir-merge NAME"/": NAME"
|
/* Register a per-directory merge-file basename (for "dir-merge NAME"/": NAME"
|
||||||
* and -F's .rsync-filter). Duplicate names are ignored. */
|
* and -F's .rsync-filter). Duplicate names are ignored. */
|
||||||
bool filter_rule_list_add_dir_merge(FilterRuleList* list, const char* name) {
|
bool filter_rule_list_add_dir_merge(FilterRuleList* list, const char* name) {
|
||||||
|
return filter_rule_list_add_dir_merge_ex(list, name, false, false, false, false, false);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool filter_rule_list_add_dir_merge_ex(FilterRuleList* list, const char* name, bool no_prefixes,
|
||||||
|
bool include, bool word_split, bool no_inherit,
|
||||||
|
bool exclude_self) {
|
||||||
if (!list || !name || name[0] == '\0')
|
if (!list || !name || name[0] == '\0')
|
||||||
return false;
|
return false;
|
||||||
for (int i = 0; i < list->dir_merge_count; i++) {
|
for (int i = 0; i < list->dir_merge_count; i++) {
|
||||||
if (strcmp(list->dir_merge_names[i], name) == 0)
|
if (strcmp(list->dir_merges[i].name, name) == 0) {
|
||||||
|
/* rsync keeps the first registration (first-wins), but the 'e' modifier
|
||||||
|
is a list side effect, not a registration field: honor it on the
|
||||||
|
duplicate path too, adding the implicit exclude-self rule at most
|
||||||
|
once. */
|
||||||
|
if (exclude_self && !list->dir_merges[i].exclude_self) {
|
||||||
|
if (!filter_list_add_exclude_self(list, name))
|
||||||
|
return false;
|
||||||
|
list->dir_merges[i].exclude_self = true;
|
||||||
|
}
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
if (list->dir_merge_count == list->dir_merge_capacity) {
|
if (list->dir_merge_count == list->dir_merge_capacity) {
|
||||||
|
if (list->dir_merge_capacity > INT_MAX / 2)
|
||||||
|
return false;
|
||||||
int new_cap = list->dir_merge_capacity > 0 ? list->dir_merge_capacity * 2 : 4;
|
int new_cap = list->dir_merge_capacity > 0 ? list->dir_merge_capacity * 2 : 4;
|
||||||
char** grown = realloc(list->dir_merge_names, (size_t)new_cap * sizeof(char*));
|
FilterDirMerge* grown = realloc(list->dir_merges, (size_t)new_cap * sizeof(*grown));
|
||||||
if (!grown)
|
if (!grown)
|
||||||
return false;
|
return false;
|
||||||
list->dir_merge_names = grown;
|
list->dir_merges = grown;
|
||||||
list->dir_merge_capacity = new_cap;
|
list->dir_merge_capacity = new_cap;
|
||||||
}
|
}
|
||||||
char* dup = str_dup(name);
|
char* dup = str_dup(name);
|
||||||
if (!dup)
|
if (!dup)
|
||||||
return false;
|
return false;
|
||||||
list->dir_merge_names[list->dir_merge_count++] = dup;
|
if (exclude_self && !filter_list_add_exclude_self(list, name)) {
|
||||||
|
free(dup);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
FilterDirMerge* entry = &list->dir_merges[list->dir_merge_count];
|
||||||
|
entry->name = dup;
|
||||||
|
entry->no_prefixes = no_prefixes;
|
||||||
|
entry->include = include;
|
||||||
|
entry->word_split = word_split;
|
||||||
|
entry->no_inherit = no_inherit;
|
||||||
|
entry->exclude_self = exclude_self;
|
||||||
|
list->dir_merge_count++;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
static bool set_rule_owner(FilterRule* rule, const char* owner) {
|
bool filter_rule_set_owner(FilterRule* rule, const char* owner) {
|
||||||
|
if (!rule)
|
||||||
|
return false;
|
||||||
char* dup = str_dup(owner ? owner : "");
|
char* dup = str_dup(owner ? owner : "");
|
||||||
if (!dup)
|
if (!dup)
|
||||||
return false;
|
return false;
|
||||||
@@ -177,10 +256,11 @@ static bool is_unsupported_modifier_char(char c) {
|
|||||||
return c == 'e' || c == 'n' || c == 'w';
|
return c == 'e' || c == 'n' || c == 'w';
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Merge-file modifiers rsync accepts on merge/dir-merge rules: 'e', 'n', 'w'
|
/* Merge-file modifiers rsync accepts on merge/dir-merge rules: 'e' (exclude
|
||||||
* and '-' (do not transfer the merge file). */
|
* self), 'n' (no inherit), 'w' (word split), '-' (bare excludes) and '+'
|
||||||
|
* (bare includes). */
|
||||||
static bool is_merge_modifier_char(char c) {
|
static bool is_merge_modifier_char(char c) {
|
||||||
return c == 'e' || c == 'n' || c == 'w' || c == '-';
|
return c == 'e' || c == 'n' || c == 'w' || c == '-' || c == '+';
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Characters that count as part of a modifier run for `kind` when deciding
|
/* Characters that count as part of a modifier run for `kind` when deciding
|
||||||
@@ -228,8 +308,10 @@ static char unsupported_modifier_in_token(const char* tok, RuleKind kind) {
|
|||||||
* generic syntax error so callers can emit a precise diagnostic. */
|
* generic syntax error so callers can emit a precise diagnostic. */
|
||||||
static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
|
static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
|
||||||
bool* sides_explicit, bool* negate, bool* anchored_mod,
|
bool* sides_explicit, bool* negate, bool* anchored_mod,
|
||||||
bool* perishable, bool* xattr, bool* cvs_inject,
|
bool* perishable, bool* xattr, bool* cvs_inject, bool* no_prefixes,
|
||||||
const char** pat_start, size_t* pat_len, char* bad_mod) {
|
bool* include_defaults, bool* word_split, bool* no_inherit,
|
||||||
|
bool* exclude_self, const char** pat_start, size_t* pat_len,
|
||||||
|
char* bad_mod) {
|
||||||
const char* p = text;
|
const char* p = text;
|
||||||
*sides = FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER;
|
*sides = FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER;
|
||||||
*sides_explicit = false;
|
*sides_explicit = false;
|
||||||
@@ -238,6 +320,11 @@ static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
|
|||||||
*perishable = false;
|
*perishable = false;
|
||||||
*xattr = false;
|
*xattr = false;
|
||||||
*cvs_inject = false;
|
*cvs_inject = false;
|
||||||
|
*no_prefixes = false;
|
||||||
|
*include_defaults = false;
|
||||||
|
*word_split = false;
|
||||||
|
*no_inherit = false;
|
||||||
|
*exclude_self = false;
|
||||||
*pat_start = NULL;
|
*pat_start = NULL;
|
||||||
*pat_len = 0;
|
*pat_len = 0;
|
||||||
*bad_mod = '\0';
|
*bad_mod = '\0';
|
||||||
@@ -312,6 +399,21 @@ static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
|
|||||||
case 'C':
|
case 'C':
|
||||||
*cvs_inject = true;
|
*cvs_inject = true;
|
||||||
break;
|
break;
|
||||||
|
case '-':
|
||||||
|
*no_prefixes = true;
|
||||||
|
break;
|
||||||
|
case '+':
|
||||||
|
*include_defaults = true;
|
||||||
|
break;
|
||||||
|
case 'e':
|
||||||
|
*exclude_self = true;
|
||||||
|
break;
|
||||||
|
case 'n':
|
||||||
|
*no_inherit = true;
|
||||||
|
break;
|
||||||
|
case 'w':
|
||||||
|
*word_split = true;
|
||||||
|
break;
|
||||||
default:
|
default:
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -347,11 +449,13 @@ FilterRule* filter_rule_parse(const char* line, const FilterParseOptions* opts,
|
|||||||
RuleKind kind = RULE_KIND_UNKNOWN;
|
RuleKind kind = RULE_KIND_UNKNOWN;
|
||||||
unsigned sides;
|
unsigned sides;
|
||||||
bool sides_explicit, negate, anchored_mod, perishable, xattr, cvs_inject;
|
bool sides_explicit, negate, anchored_mod, perishable, xattr, cvs_inject;
|
||||||
|
bool no_prefixes, include_defaults, word_split, no_inherit, exclude_self;
|
||||||
const char* pat;
|
const char* pat;
|
||||||
size_t pat_len;
|
size_t pat_len;
|
||||||
char bad_mod;
|
char bad_mod;
|
||||||
if (!parse_rule_syntax(p, &kind, &sides, &sides_explicit, &negate, &anchored_mod, &perishable,
|
if (!parse_rule_syntax(p, &kind, &sides, &sides_explicit, &negate, &anchored_mod, &perishable,
|
||||||
&xattr, &cvs_inject, &pat, &pat_len, &bad_mod)) {
|
&xattr, &cvs_inject, &no_prefixes, &include_defaults, &word_split,
|
||||||
|
&no_inherit, &exclude_self, &pat, &pat_len, &bad_mod)) {
|
||||||
if (bad_mod != '\0')
|
if (bad_mod != '\0')
|
||||||
filter_set_error(err, err_size, "unsupported filter modifier '%c'", bad_mod);
|
filter_set_error(err, err_size, "unsupported filter modifier '%c'", bad_mod);
|
||||||
else
|
else
|
||||||
@@ -504,7 +608,7 @@ static bool filter_list_append_cvs(FilterRuleList* list, unsigned sides) {
|
|||||||
}
|
}
|
||||||
memcpy(rule->pattern, CVS_DEFAULTS[i].pattern, plen);
|
memcpy(rule->pattern, CVS_DEFAULTS[i].pattern, plen);
|
||||||
rule->pattern[plen] = '\0';
|
rule->pattern[plen] = '\0';
|
||||||
if (!set_rule_owner(rule, "")) {
|
if (!filter_rule_set_owner(rule, "")) {
|
||||||
filter_rule_free(rule);
|
filter_rule_free(rule);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -522,16 +626,138 @@ static bool filter_list_parse_append_depth(FilterRuleList* list, const char* lin
|
|||||||
const FilterParseOptions* opts, const char* base_dir,
|
const FilterParseOptions* opts, const char* base_dir,
|
||||||
int depth, char* err, size_t err_size);
|
int depth, char* err, size_t err_size);
|
||||||
|
|
||||||
/* Read a merge file and splice its rules into `list`. A relative path is
|
/* Append one merge-file token/line to `list`, honoring the merge rule's
|
||||||
* resolved below `base_dir` when given, else used as-is (rsync resolves a
|
* no-prefix/include mode. In no-prefix mode the token is a bare pattern whose
|
||||||
* command-line merge file relative to the current directory). */
|
* include/exclude default comes from the merge rule (rsync's "-"/"+" merge
|
||||||
|
* modifiers); otherwise the token is parsed as a full filter rule. */
|
||||||
|
static bool filter_merge_append_token(FilterRuleList* list, const char* token,
|
||||||
|
const FilterDirMerge* spec, const FilterParseOptions* opts,
|
||||||
|
const char* base_dir, int depth, char* err, size_t err_size) {
|
||||||
|
if (spec->no_prefixes || spec->include) {
|
||||||
|
size_t tlen = strlen(token);
|
||||||
|
char* text = malloc(tlen + 3);
|
||||||
|
if (!text) {
|
||||||
|
filter_set_error(err, err_size, "memory allocation failed");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
text[0] = spec->include ? '+' : '-';
|
||||||
|
text[1] = ' ';
|
||||||
|
memcpy(text + 2, token, tlen + 1);
|
||||||
|
bool ok = filter_list_parse_append_depth(list, text, opts, base_dir, depth + 1, err, err_size);
|
||||||
|
free(text);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
return filter_list_parse_append_depth(list, token, opts, base_dir, depth + 1, err, err_size);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Read a merge file's tokens/lines into `list` for `spec`. A `w` merge rule
|
||||||
|
* word-splits on whitespace (turning comments off); otherwise lines are parsed
|
||||||
|
* and whole-line `#` comments skipped. When `owner_rel` is non-NULL the newly
|
||||||
|
* added rules are owned by that directory; a no-inherit spec marks them so they
|
||||||
|
* apply only there. Returns false on parse/allocation failure. */
|
||||||
|
static bool filter_merge_read(FilterRuleList* list, FILE* fp, const char* display_path,
|
||||||
|
const FilterDirMerge* spec, const FilterParseOptions* opts,
|
||||||
|
const char* base_dir, const char* owner_rel, int depth, char* err,
|
||||||
|
size_t err_size) {
|
||||||
|
/* Lowest list index this read is responsible for. A "clear"/"!" inside the
|
||||||
|
* file resets list->count to 0 (freeing the caller's earlier rules too), so
|
||||||
|
* the base must follow it down: otherwise post-clear rules sit below the
|
||||||
|
* original count and never receive an owner (nor no-inherit) and are missed
|
||||||
|
* by the rollback. */
|
||||||
|
int floor = list->count;
|
||||||
|
char* line = NULL;
|
||||||
|
size_t cap = 0;
|
||||||
|
bool ok = true;
|
||||||
|
while (ok) {
|
||||||
|
ssize_t n = utils_getdelim_bounded(fp, &line, &cap, '\n', UTILS_MAX_LINE_LEN);
|
||||||
|
if (n < 0) {
|
||||||
|
if (errno == EFBIG)
|
||||||
|
filter_set_error(err, err_size, "line in %s exceeds %d bytes", display_path,
|
||||||
|
(int)UTILS_MAX_LINE_LEN);
|
||||||
|
else
|
||||||
|
filter_set_error(err, err_size, "error reading %s: %s", display_path, strerror(errno));
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (n == 0)
|
||||||
|
break;
|
||||||
|
if (spec->word_split) {
|
||||||
|
/* Whitespace-separated tokens; newlines are ordinary separators and
|
||||||
|
* comments are disabled. */
|
||||||
|
const char* s = line;
|
||||||
|
while (*s) {
|
||||||
|
while (*s == ' ' || *s == '\t' || *s == '\n' || *s == '\r')
|
||||||
|
s++;
|
||||||
|
if (*s == '\0')
|
||||||
|
break;
|
||||||
|
const char* start = s;
|
||||||
|
while (*s != '\0' && *s != ' ' && *s != '\t' && *s != '\n' && *s != '\r')
|
||||||
|
s++;
|
||||||
|
size_t tlen = (size_t)(s - start);
|
||||||
|
char* token = malloc(tlen + 1);
|
||||||
|
if (!token) {
|
||||||
|
filter_set_error(err, err_size, "memory allocation failed");
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
memcpy(token, start, tlen);
|
||||||
|
token[tlen] = '\0';
|
||||||
|
if (!filter_merge_append_token(list, token, spec, opts, base_dir, depth, err, err_size))
|
||||||
|
ok = false;
|
||||||
|
if (list->count < floor)
|
||||||
|
floor = list->count; /* a "clear" reset the list below this read's base */
|
||||||
|
free(token);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const char* lp = line;
|
||||||
|
while (*lp == ' ' || *lp == '\t')
|
||||||
|
lp++;
|
||||||
|
if (*lp == '\0' || *lp == '\n' || *lp == '\r' || *lp == '#')
|
||||||
|
continue;
|
||||||
|
if (!filter_merge_append_token(list, lp, spec, opts, base_dir, depth, err, err_size))
|
||||||
|
ok = false;
|
||||||
|
if (list->count < floor)
|
||||||
|
floor = list->count; /* a "clear" reset the list below this read's base */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
free(line);
|
||||||
|
if (!ok) {
|
||||||
|
/* Drop every live rule this read is responsible for. After a "clear" that
|
||||||
|
* base is 0, so the post-clear rules are freed too instead of leaking. */
|
||||||
|
for (int i = floor; i < list->count; i++)
|
||||||
|
filter_rule_free(list->items[i]);
|
||||||
|
list->count = floor;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
for (int i = floor; i < list->count; i++) {
|
||||||
|
FilterRule* rule = list->items[i];
|
||||||
|
if (spec->no_inherit)
|
||||||
|
rule->no_inherit = true;
|
||||||
|
if (owner_rel && !filter_rule_set_owner(rule, owner_rel)) {
|
||||||
|
filter_set_error(err, err_size, "memory allocation failed");
|
||||||
|
for (int j = floor; j < list->count; j++)
|
||||||
|
filter_rule_free(list->items[j]);
|
||||||
|
list->count = floor;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Read a single-instance merge file and splice its rules into `list`. A
|
||||||
|
* relative path is resolved below `base_dir` when given, else used as-is (rsync
|
||||||
|
* resolves a command-line merge file relative to the current directory). */
|
||||||
static bool filter_list_merge_file(FilterRuleList* list, const char* name,
|
static bool filter_list_merge_file(FilterRuleList* list, const char* name,
|
||||||
const FilterParseOptions* opts, const char* base_dir, int depth,
|
const FilterDirMerge* spec, const FilterParseOptions* opts,
|
||||||
char* err, size_t err_size) {
|
const char* base_dir, int depth, char* err, size_t err_size) {
|
||||||
if (name[0] == '\0') {
|
if (name[0] == '\0') {
|
||||||
filter_set_error(err, err_size, "merge requires a filename");
|
filter_set_error(err, err_size, "merge requires a filename");
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
if (spec->exclude_self && !filter_list_add_exclude_self(list, name)) {
|
||||||
|
filter_set_error(err, err_size, "memory allocation failed");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
char* path =
|
char* path =
|
||||||
(base_dir && base_dir[0] && name[0] != '/') ? path_cat(base_dir, name) : str_dup(name);
|
(base_dir && base_dir[0] && name[0] != '/') ? path_cat(base_dir, name) : str_dup(name);
|
||||||
if (!path) {
|
if (!path) {
|
||||||
@@ -544,29 +770,7 @@ static bool filter_list_merge_file(FilterRuleList* list, const char* name,
|
|||||||
free(path);
|
free(path);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
char* line = NULL;
|
bool ok = filter_merge_read(list, fp, path, spec, opts, base_dir, NULL, depth, err, err_size);
|
||||||
size_t cap = 0;
|
|
||||||
bool ok = true;
|
|
||||||
while (true) {
|
|
||||||
ssize_t n = utils_getdelim_bounded(fp, &line, &cap, '\n', UTILS_MAX_LINE_LEN);
|
|
||||||
if (n < 0) {
|
|
||||||
filter_set_error(err, err_size, "error reading merge file '%s'", path);
|
|
||||||
ok = false;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
if (n == 0)
|
|
||||||
break;
|
|
||||||
const char* lp = line;
|
|
||||||
while (*lp == ' ' || *lp == '\t')
|
|
||||||
lp++;
|
|
||||||
if (*lp == '\0' || *lp == '\n' || *lp == '\r' || *lp == '#')
|
|
||||||
continue;
|
|
||||||
if (!filter_list_parse_append_depth(list, lp, opts, base_dir, depth + 1, err, err_size)) {
|
|
||||||
ok = false;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
free(line);
|
|
||||||
fclose(fp);
|
fclose(fp);
|
||||||
free(path);
|
free(path);
|
||||||
return ok;
|
return ok;
|
||||||
@@ -590,11 +794,13 @@ static bool filter_list_parse_append_depth(FilterRuleList* list, const char* lin
|
|||||||
RuleKind kind = RULE_KIND_UNKNOWN;
|
RuleKind kind = RULE_KIND_UNKNOWN;
|
||||||
unsigned sides;
|
unsigned sides;
|
||||||
bool sides_explicit, negate, anchored_mod, perishable, xattr, cvs_inject;
|
bool sides_explicit, negate, anchored_mod, perishable, xattr, cvs_inject;
|
||||||
|
bool no_prefixes, include_defaults, word_split, no_inherit, exclude_self;
|
||||||
const char* pat;
|
const char* pat;
|
||||||
size_t pat_len;
|
size_t pat_len;
|
||||||
char bad_mod;
|
char bad_mod;
|
||||||
if (!parse_rule_syntax(p, &kind, &sides, &sides_explicit, &negate, &anchored_mod, &perishable,
|
if (!parse_rule_syntax(p, &kind, &sides, &sides_explicit, &negate, &anchored_mod, &perishable,
|
||||||
&xattr, &cvs_inject, &pat, &pat_len, &bad_mod)) {
|
&xattr, &cvs_inject, &no_prefixes, &include_defaults, &word_split,
|
||||||
|
&no_inherit, &exclude_self, &pat, &pat_len, &bad_mod)) {
|
||||||
if (bad_mod != '\0')
|
if (bad_mod != '\0')
|
||||||
filter_set_error(err, err_size, "unsupported filter modifier '%c': %s", bad_mod, p);
|
filter_set_error(err, err_size, "unsupported filter modifier '%c': %s", bad_mod, p);
|
||||||
else
|
else
|
||||||
@@ -640,7 +846,15 @@ static bool filter_list_parse_append_depth(FilterRuleList* list, const char* lin
|
|||||||
}
|
}
|
||||||
memcpy(name, pat, pat_len);
|
memcpy(name, pat, pat_len);
|
||||||
name[pat_len] = '\0';
|
name[pat_len] = '\0';
|
||||||
bool ok = filter_list_merge_file(list, name, opts, base_dir, depth, err, err_size);
|
/* A single-instance merge has no inheritance, so 'n' is meaningless; the
|
||||||
|
* other merge modifiers still shape how the file is read. */
|
||||||
|
FilterDirMerge spec = {.name = name,
|
||||||
|
.no_prefixes = no_prefixes,
|
||||||
|
.include = include_defaults,
|
||||||
|
.word_split = word_split,
|
||||||
|
.no_inherit = false,
|
||||||
|
.exclude_self = exclude_self};
|
||||||
|
bool ok = filter_list_merge_file(list, name, &spec, opts, base_dir, depth, err, err_size);
|
||||||
free(name);
|
free(name);
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
@@ -656,7 +870,8 @@ static bool filter_list_parse_append_depth(FilterRuleList* list, const char* lin
|
|||||||
}
|
}
|
||||||
memcpy(name, pat, pat_len);
|
memcpy(name, pat, pat_len);
|
||||||
name[pat_len] = '\0';
|
name[pat_len] = '\0';
|
||||||
bool ok = filter_rule_list_add_dir_merge(list, name);
|
bool ok = filter_rule_list_add_dir_merge_ex(list, name, no_prefixes, include_defaults,
|
||||||
|
word_split, no_inherit, exclude_self);
|
||||||
free(name);
|
free(name);
|
||||||
if (!ok) {
|
if (!ok) {
|
||||||
filter_set_error(err, err_size, "memory allocation failed");
|
filter_set_error(err, err_size, "memory allocation failed");
|
||||||
@@ -717,27 +932,30 @@ FilterRuleList* filter_base_build(const char* const* rule_texts, int rule_count,
|
|||||||
/* Undo the rules and dir-merge registrations that one merge file appended,
|
/* Undo the rules and dir-merge registrations that one merge file appended,
|
||||||
* leaving the caller's earlier content intact. A "clear" rule inside the file
|
* leaving the caller's earlier content intact. A "clear" rule inside the file
|
||||||
* frees every rule, including the caller's; clamp to the surviving count so
|
* frees every rule, including the caller's; clamp to the surviving count so
|
||||||
* those already-freed rules are never resurrected and freed a second time. */
|
* those already-freed rules are never resurrected and freed a second time.
|
||||||
|
* (filter_merge_read() has already rolled its own range back by the time this
|
||||||
|
* runs, so on a post-clear failure `list->count` is below `rules_before` and
|
||||||
|
* this is a no-op for the rules.) */
|
||||||
static void filter_file_rollback(FilterRuleList* list, int rules_before, int dir_merges_before) {
|
static void filter_file_rollback(FilterRuleList* list, int rules_before, int dir_merges_before) {
|
||||||
int first = rules_before < list->count ? rules_before : list->count;
|
int first = rules_before < list->count ? rules_before : list->count;
|
||||||
for (int i = first; i < list->count; i++)
|
for (int i = first; i < list->count; i++)
|
||||||
filter_rule_free(list->items[i]);
|
filter_rule_free(list->items[i]);
|
||||||
list->count = first;
|
list->count = first;
|
||||||
for (int i = dir_merges_before; i < list->dir_merge_count; i++)
|
for (int i = dir_merges_before; i < list->dir_merge_count; i++)
|
||||||
free(list->dir_merge_names[i]);
|
free(list->dir_merges[i].name);
|
||||||
list->dir_merge_count = dir_merges_before;
|
list->dir_merge_count = dir_merges_before;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool filter_file_append(FilterRuleList* list, const char* dir_path, const char* name,
|
bool filter_dir_merge_append(FilterRuleList* list, const char* dir_path, const FilterDirMerge* spec,
|
||||||
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||||
char* err, size_t err_size) {
|
char* err, size_t err_size) {
|
||||||
if (err && err_size > 0)
|
if (err && err_size > 0)
|
||||||
err[0] = '\0';
|
err[0] = '\0';
|
||||||
if (exists)
|
if (exists)
|
||||||
*exists = false;
|
*exists = false;
|
||||||
if (!list)
|
if (!list || !spec || !spec->name)
|
||||||
return false;
|
return false;
|
||||||
char* filter_path = path_cat(dir_path, name);
|
char* filter_path = path_cat(dir_path, spec->name);
|
||||||
if (!filter_path) {
|
if (!filter_path) {
|
||||||
filter_set_error(err, err_size, "memory allocation failed");
|
filter_set_error(err, err_size, "memory allocation failed");
|
||||||
return false;
|
return false;
|
||||||
@@ -748,7 +966,7 @@ bool filter_file_append(FilterRuleList* list, const char* dir_path, const char*
|
|||||||
if (errno == ENOENT || errno == ENOTDIR)
|
if (errno == ENOENT || errno == ENOTDIR)
|
||||||
return true;
|
return true;
|
||||||
char* escaped_dir = output_escape(dir_path, log_get_8_bit_output());
|
char* escaped_dir = output_escape(dir_path, log_get_8_bit_output());
|
||||||
log_message(LOG_LEVEL_WARNING, "Could not read %s in %s: %s", name,
|
log_message(LOG_LEVEL_WARNING, "Could not read %s in %s: %s", spec->name,
|
||||||
escaped_dir ? escaped_dir : "<allocation failed>", strerror(errno));
|
escaped_dir ? escaped_dir : "<allocation failed>", strerror(errno));
|
||||||
free(escaped_dir);
|
free(escaped_dir);
|
||||||
return true;
|
return true;
|
||||||
@@ -757,51 +975,25 @@ bool filter_file_append(FilterRuleList* list, const char* dir_path, const char*
|
|||||||
*exists = true;
|
*exists = true;
|
||||||
int rules_before = list->count;
|
int rules_before = list->count;
|
||||||
int dir_merges_before = list->dir_merge_count;
|
int dir_merges_before = list->dir_merge_count;
|
||||||
char* line = NULL;
|
|
||||||
size_t line_cap = 0;
|
|
||||||
bool ok = true;
|
|
||||||
while (true) {
|
|
||||||
ssize_t n = utils_getdelim_bounded(fp, &line, &line_cap, '\n', UTILS_MAX_LINE_LEN);
|
|
||||||
if (n < 0) {
|
|
||||||
if (errno == EFBIG) {
|
|
||||||
filter_set_error(err, err_size, "line in %s exceeds %d bytes", name,
|
|
||||||
(int)UTILS_MAX_LINE_LEN);
|
|
||||||
} else {
|
|
||||||
filter_set_error(err, err_size, "error reading %s: %s", name, strerror(errno));
|
|
||||||
}
|
|
||||||
ok = false;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
if (n == 0)
|
|
||||||
break;
|
|
||||||
const char* p = line;
|
|
||||||
while (*p == ' ' || *p == '\t')
|
|
||||||
p++;
|
|
||||||
if (*p == '\0' || *p == '\n' || *p == '\r' || *p == '#')
|
|
||||||
continue;
|
|
||||||
/* Merge files inside a per-directory file resolve relative to that
|
/* Merge files inside a per-directory file resolve relative to that
|
||||||
directory. */
|
directory. */
|
||||||
if (!filter_list_parse_append_depth(list, p, opts, dir_path, 0, err, err_size)) {
|
bool ok =
|
||||||
ok = false;
|
filter_merge_read(list, fp, spec->name, spec, opts, dir_path, owner_rel, 0, err, err_size);
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
free(line);
|
|
||||||
fclose(fp);
|
fclose(fp);
|
||||||
if (!ok) {
|
if (!ok) {
|
||||||
filter_file_rollback(list, rules_before, dir_merges_before);
|
filter_file_rollback(list, rules_before, dir_merges_before);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
for (int i = rules_before; i < list->count; i++) {
|
|
||||||
if (!set_rule_owner(list->items[i], owner_rel)) {
|
|
||||||
filter_set_error(err, err_size, "memory allocation failed");
|
|
||||||
filter_file_rollback(list, rules_before, dir_merges_before);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool filter_file_append(FilterRuleList* list, const char* dir_path, const char* name,
|
||||||
|
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||||
|
char* err, size_t err_size) {
|
||||||
|
FilterDirMerge spec = {.name = (char*)name};
|
||||||
|
return filter_dir_merge_append(list, dir_path, &spec, owner_rel, opts, exists, err, err_size);
|
||||||
|
}
|
||||||
|
|
||||||
FilterRuleList* filter_file_read_named(const char* dir_path, const char* name,
|
FilterRuleList* filter_file_read_named(const char* dir_path, const char* name,
|
||||||
const char* owner_rel, const FilterParseOptions* opts,
|
const char* owner_rel, const FilterParseOptions* opts,
|
||||||
bool* exists, char* err, size_t err_size) {
|
bool* exists, char* err, size_t err_size) {
|
||||||
@@ -843,11 +1035,16 @@ static FilterAction rule_matches(const FilterRule* rule, const char* rel_path, c
|
|||||||
return FILTER_ACTION_NONE;
|
return FILTER_ACTION_NONE;
|
||||||
if (!(rule->sides & side))
|
if (!(rule->sides & side))
|
||||||
return FILTER_ACTION_NONE;
|
return FILTER_ACTION_NONE;
|
||||||
/* A rule applies only to entries below its owner directory. */
|
/* A rule applies only to entries below its owner directory. The receive
|
||||||
|
* root's destination-relative coordinate may be written as "." (the
|
||||||
|
* synced-directory sentinel), which is the same scope as the empty owner. */
|
||||||
|
const char* owner = rule->owner;
|
||||||
|
if (owner && strcmp(owner, ".") == 0)
|
||||||
|
owner = "";
|
||||||
const char* rel2 = rel_path;
|
const char* rel2 = rel_path;
|
||||||
if (rule->owner && rule->owner[0] != '\0') {
|
if (owner && owner[0] != '\0') {
|
||||||
size_t owner_len = strlen(rule->owner);
|
size_t owner_len = strlen(owner);
|
||||||
if (strncmp(rule->owner, rel_path, owner_len) != 0)
|
if (strncmp(owner, rel_path, owner_len) != 0)
|
||||||
return FILTER_ACTION_NONE;
|
return FILTER_ACTION_NONE;
|
||||||
if (rel_path[owner_len] != '/')
|
if (rel_path[owner_len] != '/')
|
||||||
return FILTER_ACTION_NONE;
|
return FILTER_ACTION_NONE;
|
||||||
@@ -855,6 +1052,10 @@ static FilterAction rule_matches(const FilterRule* rule, const char* rel_path, c
|
|||||||
}
|
}
|
||||||
if (rel2[0] == '\0')
|
if (rel2[0] == '\0')
|
||||||
return FILTER_ACTION_NONE;
|
return FILTER_ACTION_NONE;
|
||||||
|
/* A no-inherit rule ('n' on its dir-merge) applies only to direct children of
|
||||||
|
* its owner directory, never to deeper entries. */
|
||||||
|
if (rule->no_inherit && strchr(rel2, '/') != NULL)
|
||||||
|
return FILTER_ACTION_NONE;
|
||||||
bool matched;
|
bool matched;
|
||||||
if (rule->dir_only && !is_dir)
|
if (rule->dir_only && !is_dir)
|
||||||
matched = false;
|
matched = false;
|
||||||
@@ -884,3 +1085,45 @@ FilterAction filter_rules_apply_side(const FilterRuleList* list, const char* rel
|
|||||||
}
|
}
|
||||||
return FILTER_ACTION_NONE;
|
return FILTER_ACTION_NONE;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
FilterAction filter_dir_rules_apply_side(const FilterRuleList* dir_rules, const char* rel_path,
|
||||||
|
const char* leaf, bool is_dir) {
|
||||||
|
if (!dir_rules || !rel_path)
|
||||||
|
return FILTER_ACTION_NONE;
|
||||||
|
size_t len = strlen(rel_path);
|
||||||
|
if (len == 0)
|
||||||
|
return FILTER_ACTION_NONE;
|
||||||
|
/* The containing directory of rel_path is the prefix before its final '/'. */
|
||||||
|
size_t owner_len = 0;
|
||||||
|
for (size_t i = 0; i < len; i++) {
|
||||||
|
if (rel_path[i] == '/')
|
||||||
|
owner_len = i;
|
||||||
|
}
|
||||||
|
for (;;) {
|
||||||
|
for (int i = 0; i < dir_rules->count; i++) {
|
||||||
|
const FilterRule* rule = dir_rules->items[i];
|
||||||
|
const char* rule_owner = rule && rule->owner ? rule->owner : "";
|
||||||
|
/* "." is the receive root's coordinate (see rule_matches). */
|
||||||
|
if (strcmp(rule_owner, ".") == 0)
|
||||||
|
rule_owner = "";
|
||||||
|
size_t rule_owner_len = strlen(rule_owner);
|
||||||
|
if (rule_owner_len != owner_len)
|
||||||
|
continue;
|
||||||
|
if (owner_len != 0 && memcmp(rule_owner, rel_path, owner_len) != 0)
|
||||||
|
continue;
|
||||||
|
FilterAction action = rule_matches(rule, rel_path, leaf, is_dir, FILTER_SIDE_RECEIVER);
|
||||||
|
if (action != FILTER_ACTION_NONE)
|
||||||
|
return action;
|
||||||
|
}
|
||||||
|
if (owner_len == 0)
|
||||||
|
break;
|
||||||
|
/* Move to the parent directory: the last '/' before owner_len. */
|
||||||
|
size_t parent = 0;
|
||||||
|
for (size_t j = 0; j < owner_len; j++) {
|
||||||
|
if (rel_path[j] == '/')
|
||||||
|
parent = j;
|
||||||
|
}
|
||||||
|
owner_len = parent;
|
||||||
|
}
|
||||||
|
return FILTER_ACTION_NONE;
|
||||||
|
}
|
||||||
+61
-11
@@ -25,11 +25,12 @@
|
|||||||
* Modifiers: '/' absolute anchor, '!' negate match, 'C' inject CVS defaults,
|
* Modifiers: '/' absolute anchor, '!' negate match, 'C' inject CVS defaults,
|
||||||
* 's' sender side, 'r' receiver side, 'p' perishable. The rsync 'x'
|
* 's' sender side, 'r' receiver side, 'p' perishable. The rsync 'x'
|
||||||
* (xattr-name) modifier is not implemented and is rejected explicitly
|
* (xattr-name) modifier is not implemented and is rejected explicitly
|
||||||
* everywhere. The merge-file modifiers 'e' (exclude the merge file itself),
|
* everywhere. The merge-only modifiers are accepted only on merge/dir-merge
|
||||||
* 'n' (do not inherit the merge file), 'w' (word-split the merge file) and '-'
|
* rules (rejected on every other rule, matching rsync): 'e' excludes the merge
|
||||||
* (do not transfer the merge file) are accepted and consumed only on merge/
|
* file itself, 'n' makes the merged rules non-inheriting (they apply only to
|
||||||
* dir-merge rules (rejected on every other rule, matching rsync); their
|
* the directory that holds the merge file), 'w' word-splits the merge file on
|
||||||
* semantics are not implemented and they are otherwise ignored.
|
* whitespace instead of lines, and '-' reads the merge file as a list of bare
|
||||||
|
* exclude patterns with no rule prefixes.
|
||||||
* A trailing '/' makes a pattern match directories only. A leading '/' anchors
|
* A trailing '/' makes a pattern match directories only. A leading '/' anchors
|
||||||
* the pattern to its owner directory.
|
* the pattern to its owner directory.
|
||||||
*/
|
*/
|
||||||
@@ -55,18 +56,32 @@ typedef struct {
|
|||||||
bool dir_only; /* pattern had a trailing '/': matches directories only */
|
bool dir_only; /* pattern had a trailing '/': matches directories only */
|
||||||
bool negate; /* '!' modifier: match succeeds when the pattern does not */
|
bool negate; /* '!' modifier: match succeeds when the pattern does not */
|
||||||
bool perishable; /* 'p' modifier (ignored in deleted directories) */
|
bool perishable; /* 'p' modifier (ignored in deleted directories) */
|
||||||
|
bool no_inherit; /* 'n' on the owning dir-merge: applies only in `owner` */
|
||||||
char* owner; /* owning directory rel path ("" == transfer root) */
|
char* owner; /* owning directory rel path ("" == transfer root) */
|
||||||
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
|
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
|
||||||
} FilterRule;
|
} FilterRule;
|
||||||
|
|
||||||
|
/* One per-directory merge-file registration ("dir-merge NAME"/": NAME", "merge
|
||||||
|
* NAME"/". NAME" and -F's .rsync-filter) together with the merge-only modifiers
|
||||||
|
* parsed from the rule. The scanner reads `name` in every directory it
|
||||||
|
* traverses and applies `no_prefixes`/`include`/`word_split`/`no_inherit`/
|
||||||
|
* `exclude_self` while merging the file's rules. */
|
||||||
|
typedef struct {
|
||||||
|
char* name;
|
||||||
|
bool no_prefixes; /* '-' : file holds only bare exclude patterns */
|
||||||
|
bool include; /* '+' : file holds only bare include patterns */
|
||||||
|
bool word_split; /* 'w' : split the file on whitespace, not lines */
|
||||||
|
bool no_inherit; /* 'n' : the merged rules do not inherit below their dir */
|
||||||
|
bool exclude_self; /* 'e' : exclude the merge file itself from the transfer */
|
||||||
|
} FilterDirMerge;
|
||||||
|
|
||||||
typedef struct FilterRuleList {
|
typedef struct FilterRuleList {
|
||||||
FilterRule** items; /* owned array of rule pointers */
|
FilterRule** items; /* owned array of rule pointers */
|
||||||
int count;
|
int count;
|
||||||
int capacity;
|
int capacity;
|
||||||
/* Per-directory merge-file basenames registered by "dir-merge NAME"/": NAME"
|
/* Per-directory merge-file registrations. Owned; the scanner reads each
|
||||||
* or by -F (.rsync-filter). Owned strings; the scanner reads each name in
|
* name in every directory it traverses. */
|
||||||
* every directory it traverses. */
|
FilterDirMerge* dir_merges;
|
||||||
char** dir_merge_names;
|
|
||||||
int dir_merge_count;
|
int dir_merge_count;
|
||||||
int dir_merge_capacity;
|
int dir_merge_capacity;
|
||||||
} FilterRuleList;
|
} FilterRuleList;
|
||||||
@@ -83,13 +98,28 @@ typedef struct {
|
|||||||
FilterRule* filter_rule_parse(const char* line, const FilterParseOptions* opts, char* err,
|
FilterRule* filter_rule_parse(const char* line, const FilterParseOptions* opts, char* err,
|
||||||
size_t err_size);
|
size_t err_size);
|
||||||
void filter_rule_free(FilterRule* rule);
|
void filter_rule_free(FilterRule* rule);
|
||||||
|
/* Deep-copy a rule (owned pattern/owner). Returns NULL on allocation failure. */
|
||||||
|
FilterRule* filter_rule_clone(const FilterRule* rule);
|
||||||
|
/* Replace a rule's owner directory (owned copy of `owner`, "" for the transfer
|
||||||
|
* root). Returns false on allocation failure, leaving the rule unchanged.
|
||||||
|
* Used to re-express a mirrored per-directory rule in the receiver's
|
||||||
|
* destination-relative coordinate system. */
|
||||||
|
bool filter_rule_set_owner(FilterRule* rule, const char* owner);
|
||||||
|
|
||||||
FilterRuleList* filter_rule_list_create(void);
|
FilterRuleList* filter_rule_list_create(void);
|
||||||
/* Append a fully-parsed rule (takes ownership). Returns false on OOM. */
|
/* Append a fully-parsed rule (takes ownership). Returns false on OOM. */
|
||||||
bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule);
|
bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule);
|
||||||
/* Register a per-directory merge-file basename (idempotent). Returns false on
|
/* Register a per-directory merge-file basename (idempotent, no modifiers).
|
||||||
* OOM. Used by the scanner to read custom "dir-merge" files. */
|
* Returns false on OOM. Used by the scanner to read custom "dir-merge" files. */
|
||||||
bool filter_rule_list_add_dir_merge(FilterRuleList* list, const char* name);
|
bool filter_rule_list_add_dir_merge(FilterRuleList* list, const char* name);
|
||||||
|
/* Register a per-directory merge file with its merge-only modifiers. On a
|
||||||
|
* duplicate name the existing registration is kept (rsync's first wins) and true
|
||||||
|
* is returned. When `exclude_self` is set an implicit exclude rule for the
|
||||||
|
* merge file's basename is appended to the list at this position, matching
|
||||||
|
* rsync's `e` modifier. Returns false on OOM. */
|
||||||
|
bool filter_rule_list_add_dir_merge_ex(FilterRuleList* list, const char* name, bool no_prefixes,
|
||||||
|
bool include, bool word_split, bool no_inherit,
|
||||||
|
bool exclude_self);
|
||||||
/* Parse `line` and append it. Handles "clear"/"!" (resets the list), "merge
|
/* Parse `line` and append it. Handles "clear"/"!" (resets the list), "merge
|
||||||
* FILE"/". FILE" (splices the file's rules) and "dir-merge NAME"/": NAME"
|
* FILE"/". FILE" (splices the file's rules) and "dir-merge NAME"/": NAME"
|
||||||
* (registers a per-directory filename). Returns false and fills `err` on bad
|
* (registers a per-directory filename). Returns false and fills `err` on bad
|
||||||
@@ -122,6 +152,15 @@ bool filter_file_append(FilterRuleList* list, const char* dir_path, const char*
|
|||||||
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||||
char* err, size_t err_size);
|
char* err, size_t err_size);
|
||||||
|
|
||||||
|
/* Append a per-directory merge file honoring its merge-only modifiers: `-`
|
||||||
|
* reads every (word-split, when `w`) token as a bare exclude, `+` as a bare
|
||||||
|
* include, and `n` marks each read rule non-inheriting. A plain name behaves
|
||||||
|
* like filter_file_append. A missing file yields *exists=false with no error;
|
||||||
|
* returns false only on a parse/allocation failure (message in `err`). */
|
||||||
|
bool filter_dir_merge_append(FilterRuleList* list, const char* dir_path, const FilterDirMerge* spec,
|
||||||
|
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||||
|
char* err, size_t err_size);
|
||||||
|
|
||||||
/* filter_file_read_named with the default ".rsync-filter" name. */
|
/* filter_file_read_named with the default ".rsync-filter" name. */
|
||||||
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
|
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
|
||||||
char* err, size_t err_size);
|
char* err, size_t err_size);
|
||||||
@@ -135,4 +174,15 @@ FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bo
|
|||||||
FilterAction filter_rules_apply_side(const FilterRuleList* list, const char* rel_path,
|
FilterAction filter_rules_apply_side(const FilterRuleList* list, const char* rel_path,
|
||||||
const char* leaf, bool is_dir, unsigned side);
|
const char* leaf, bool is_dir, unsigned side);
|
||||||
|
|
||||||
|
/* Evaluate a received per-directory rule set for the receiver side, using
|
||||||
|
* rsync's per-directory-before-ancestors order: the entry's containing
|
||||||
|
* directory's rules are tried first, then each ancestor's, then the receive
|
||||||
|
* root's. `dir_rules` is a flat list whose rules carry `owner`; a rule applies
|
||||||
|
* only when `owner` is exactly the directory being examined (a no-inherit rule
|
||||||
|
* therefore applies only to that directory's direct children). Returns the
|
||||||
|
* first matching rule's receiver verdict (PROTECT/RISK) or FILTER_ACTION_NONE.
|
||||||
|
* The caller evaluates the command-line base rules after this chain. */
|
||||||
|
FilterAction filter_dir_rules_apply_side(const FilterRuleList* dir_rules, const char* rel_path,
|
||||||
|
const char* leaf, bool is_dir);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
+16
-6
@@ -62,14 +62,16 @@ bool format_dest_state_send(int fd, const OutputDestState* state) {
|
|||||||
if (!state)
|
if (!state)
|
||||||
return false;
|
return false;
|
||||||
int32_t has_old = state->existed ? 1 : 0;
|
int32_t has_old = state->existed ? 1 : 0;
|
||||||
|
int32_t target_matches = state->target_matches ? 1 : 0;
|
||||||
uint64_t size = (uint64_t)state->size;
|
uint64_t size = (uint64_t)state->size;
|
||||||
int64_t mtime = (int64_t)state->mtime_sec;
|
int64_t mtime = (int64_t)state->mtime_sec;
|
||||||
int64_t mtime_nsec = state->mtime_nsec;
|
int64_t mtime_nsec = state->mtime_nsec;
|
||||||
uint32_t mode = state->mode;
|
uint32_t mode = state->mode;
|
||||||
int32_t uid = state->uid;
|
int32_t uid = state->uid;
|
||||||
int32_t gid = state->gid;
|
int32_t gid = state->gid;
|
||||||
return send_n_data(fd, &has_old, sizeof(has_old)) && send_n_data(fd, &size, sizeof(size)) &&
|
return send_n_data(fd, &has_old, sizeof(has_old)) &&
|
||||||
send_n_data(fd, &mtime, sizeof(mtime)) &&
|
send_n_data(fd, &target_matches, sizeof(target_matches)) &&
|
||||||
|
send_n_data(fd, &size, sizeof(size)) && send_n_data(fd, &mtime, sizeof(mtime)) &&
|
||||||
send_n_data(fd, &mtime_nsec, sizeof(mtime_nsec)) && send_n_data(fd, &mode, sizeof(mode)) &&
|
send_n_data(fd, &mtime_nsec, sizeof(mtime_nsec)) && send_n_data(fd, &mode, sizeof(mode)) &&
|
||||||
send_n_data(fd, &uid, sizeof(uid)) && send_n_data(fd, &gid, sizeof(gid));
|
send_n_data(fd, &uid, sizeof(uid)) && send_n_data(fd, &gid, sizeof(gid));
|
||||||
}
|
}
|
||||||
@@ -78,14 +80,16 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
|
|||||||
if (!state)
|
if (!state)
|
||||||
return false;
|
return false;
|
||||||
int32_t has_old = 0;
|
int32_t has_old = 0;
|
||||||
|
int32_t target_matches = 0;
|
||||||
uint64_t size = 0;
|
uint64_t size = 0;
|
||||||
int64_t mtime = 0;
|
int64_t mtime = 0;
|
||||||
int64_t mtime_nsec = 0;
|
int64_t mtime_nsec = 0;
|
||||||
uint32_t mode = 0;
|
uint32_t mode = 0;
|
||||||
int32_t uid = 0;
|
int32_t uid = 0;
|
||||||
int32_t gid = 0;
|
int32_t gid = 0;
|
||||||
if (!receive_n_data(fd, &has_old, sizeof(has_old)) || !receive_n_data(fd, &size, sizeof(size)) ||
|
if (!receive_n_data(fd, &has_old, sizeof(has_old)) ||
|
||||||
!receive_n_data(fd, &mtime, sizeof(mtime)) ||
|
!receive_n_data(fd, &target_matches, sizeof(target_matches)) ||
|
||||||
|
!receive_n_data(fd, &size, sizeof(size)) || !receive_n_data(fd, &mtime, sizeof(mtime)) ||
|
||||||
!receive_n_data(fd, &mtime_nsec, sizeof(mtime_nsec)) ||
|
!receive_n_data(fd, &mtime_nsec, sizeof(mtime_nsec)) ||
|
||||||
!receive_n_data(fd, &mode, sizeof(mode)) || !receive_n_data(fd, &uid, sizeof(uid)) ||
|
!receive_n_data(fd, &mode, sizeof(mode)) || !receive_n_data(fd, &uid, sizeof(uid)) ||
|
||||||
!receive_n_data(fd, &gid, sizeof(gid)))
|
!receive_n_data(fd, &gid, sizeof(gid)))
|
||||||
@@ -93,6 +97,7 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
|
|||||||
memset(state, 0, sizeof(*state));
|
memset(state, 0, sizeof(*state));
|
||||||
state->known = true;
|
state->known = true;
|
||||||
state->existed = has_old != 0;
|
state->existed = has_old != 0;
|
||||||
|
state->target_matches = target_matches != 0;
|
||||||
state->size = size;
|
state->size = size;
|
||||||
state->mtime_sec = mtime;
|
state->mtime_sec = mtime;
|
||||||
state->mtime_nsec = mtime_nsec;
|
state->mtime_nsec = mtime_nsec;
|
||||||
@@ -105,9 +110,10 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
|
|||||||
bool format_stats_send(int fd, const ReceiverStats* stats) {
|
bool format_stats_send(int fd, const ReceiverStats* stats) {
|
||||||
if (!stats)
|
if (!stats)
|
||||||
return false;
|
return false;
|
||||||
unsigned long long fields[8] = {
|
unsigned long long fields[12] = {
|
||||||
stats->matched_data, stats->deleted_files, stats->would_delete_count, stats->literal_bytes,
|
stats->matched_data, stats->deleted_files, stats->would_delete_count, stats->literal_bytes,
|
||||||
stats->created_reg, stats->created_dir, stats->created_link, stats->created_special,
|
stats->created_reg, stats->created_dir, stats->created_link, stats->created_special,
|
||||||
|
stats->deleted_reg, stats->deleted_dir, stats->deleted_link, stats->deleted_special,
|
||||||
};
|
};
|
||||||
return send_n_data(fd, fields, sizeof(fields));
|
return send_n_data(fd, fields, sizeof(fields));
|
||||||
}
|
}
|
||||||
@@ -115,7 +121,7 @@ bool format_stats_send(int fd, const ReceiverStats* stats) {
|
|||||||
bool format_stats_receive(int fd, ReceiverStats* stats) {
|
bool format_stats_receive(int fd, ReceiverStats* stats) {
|
||||||
if (!stats)
|
if (!stats)
|
||||||
return false;
|
return false;
|
||||||
unsigned long long fields[8] = {0};
|
unsigned long long fields[12] = {0};
|
||||||
if (!receive_n_data(fd, fields, sizeof(fields)))
|
if (!receive_n_data(fd, fields, sizeof(fields)))
|
||||||
return false;
|
return false;
|
||||||
memset(stats, 0, sizeof(*stats));
|
memset(stats, 0, sizeof(*stats));
|
||||||
@@ -127,5 +133,9 @@ bool format_stats_receive(int fd, ReceiverStats* stats) {
|
|||||||
stats->created_dir = fields[5];
|
stats->created_dir = fields[5];
|
||||||
stats->created_link = fields[6];
|
stats->created_link = fields[6];
|
||||||
stats->created_special = fields[7];
|
stats->created_special = fields[7];
|
||||||
|
stats->deleted_reg = fields[8];
|
||||||
|
stats->deleted_dir = fields[9];
|
||||||
|
stats->deleted_link = fields[10];
|
||||||
|
stats->deleted_special = fields[11];
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
+25
-6
@@ -17,10 +17,18 @@
|
|||||||
/* Pre-transfer destination snapshot, reported by the receiver when the wire
|
/* Pre-transfer destination snapshot, reported by the receiver when the wire
|
||||||
* config carries report_dest_info. `known` distinguishes "no report was
|
* config carries report_dest_info. `known` distinguishes "no report was
|
||||||
* requested/received" from "the destination did not exist" (`existed == false`
|
* requested/received" from "the destination did not exist" (`existed == false`
|
||||||
* with `known == true`). */
|
* with `known == true`).
|
||||||
|
*
|
||||||
|
* `target_matches` is meaningful only for a symlink destination (protocol
|
||||||
|
* 2.30.0): the receiver compares its on-disk link target with the incoming
|
||||||
|
* target and reports whether they are equal, so the sender can render rsync's
|
||||||
|
* `cLc........` (target changed) versus `.L..t......` (attributes only) and
|
||||||
|
* suppress an unchanged symlink's line entirely. It is always false for every
|
||||||
|
* other entry kind. */
|
||||||
typedef struct {
|
typedef struct {
|
||||||
bool known;
|
bool known;
|
||||||
bool existed;
|
bool existed;
|
||||||
|
bool target_matches;
|
||||||
unsigned long long size;
|
unsigned long long size;
|
||||||
long long mtime_sec;
|
long long mtime_sec;
|
||||||
long long mtime_nsec;
|
long long mtime_nsec;
|
||||||
@@ -57,17 +65,24 @@ bool format_dest_state_send(int fd, const OutputDestState* state);
|
|||||||
bool format_dest_state_receive(int fd, OutputDestState* state);
|
bool format_dest_state_receive(int fd, OutputDestState* state);
|
||||||
|
|
||||||
/* End-of-transfer receiver counters reported through STATUS_STATS (protocol
|
/* End-of-transfer receiver counters reported through STATUS_STATS (protocol
|
||||||
* 2.25.0, extended in 2.28.0) when the wire config carries report_stats.
|
* 2.25.0, extended in 2.28.0 and 2.30.0) when the wire config carries
|
||||||
* `would_delete_count` is the number of destination-relative paths the receiver
|
* report_stats. `would_delete_count` is the number of destination-relative
|
||||||
* would have deleted in a -n/--dry-run --delete run; that many wire strings
|
* paths the receiver would have deleted in a -n/--dry-run --delete run; that
|
||||||
* immediately follow the fixed record (sent/read by the caller).
|
* many wire strings immediately follow the fixed record (sent/read by the
|
||||||
|
* caller).
|
||||||
*
|
*
|
||||||
* Protocol 2.28.0 adds the receiver-observed counters the sender cannot see:
|
* Protocol 2.28.0 adds the receiver-observed counters the sender cannot see:
|
||||||
* `literal_bytes` is the file data the receiver actually stored literally
|
* `literal_bytes` is the file data the receiver actually stored literally
|
||||||
* (whole files plus the literal fragments of a delta) and the four `created_*`
|
* (whole files plus the literal fragments of a delta) and the four `created_*`
|
||||||
* counters split the destination entries the receiver newly created by type,
|
* counters split the destination entries the receiver newly created by type,
|
||||||
* reproducing rsync's `Number of created files` breakdown and an exact
|
* reproducing rsync's `Number of created files` breakdown and an exact
|
||||||
* `Literal data` for a delta run. */
|
* `Literal data` for a delta run.
|
||||||
|
*
|
||||||
|
* Protocol 2.30.0 appends the four `deleted_*` counters: the same reg/dir/link/
|
||||||
|
* special split for the entries the receiver ACTUALLY removed, so `--stats` can
|
||||||
|
* render rsync's `Number of deleted files: X (reg: A, dir: B, link: C,
|
||||||
|
* special: D)` parenthetical. The scalar `deleted_files` stays the authoritative
|
||||||
|
* total (the breakdown is a strict partition of it). */
|
||||||
typedef struct {
|
typedef struct {
|
||||||
unsigned long long matched_data;
|
unsigned long long matched_data;
|
||||||
unsigned long long deleted_files;
|
unsigned long long deleted_files;
|
||||||
@@ -77,6 +92,10 @@ typedef struct {
|
|||||||
unsigned long long created_dir;
|
unsigned long long created_dir;
|
||||||
unsigned long long created_link;
|
unsigned long long created_link;
|
||||||
unsigned long long created_special;
|
unsigned long long created_special;
|
||||||
|
unsigned long long deleted_reg;
|
||||||
|
unsigned long long deleted_dir;
|
||||||
|
unsigned long long deleted_link;
|
||||||
|
unsigned long long deleted_special;
|
||||||
} ReceiverStats;
|
} ReceiverStats;
|
||||||
|
|
||||||
/* Fixed-width STATUS_STATS counter record. The status frame and the optional
|
/* Fixed-width STATUS_STATS counter record. The status frame and the optional
|
||||||
|
|||||||
@@ -276,7 +276,7 @@ static bool identity_wire_map_valid(const IdentityMap* map) {
|
|||||||
}
|
}
|
||||||
if (map->to < IDENTITY_CURRENT)
|
if (map->to < IDENTITY_CURRENT)
|
||||||
return false;
|
return false;
|
||||||
if (map->to_name && strlen(map->to_name) > 255)
|
if (map->to_name && strlen(map->to_name) > IDENTITY_MAX_NAME_LEN)
|
||||||
return false;
|
return false;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,11 @@
|
|||||||
#include <stdint.h>
|
#include <stdint.h>
|
||||||
#include <sys/types.h>
|
#include <sys/types.h>
|
||||||
|
|
||||||
|
/* Maximum length of a receiver-resolved identity name in a FROM:TO map's TO
|
||||||
|
* field. Bounded so a malicious/huge name can never cross the wire (see
|
||||||
|
* identity_wire_map_valid). */
|
||||||
|
#define IDENTITY_MAX_NAME_LEN 255
|
||||||
|
|
||||||
/*
|
/*
|
||||||
* Identity mapping: --numeric-ids / --usermap / --groupmap / --chown / --copy-as.
|
* Identity mapping: --numeric-ids / --usermap / --groupmap / --chown / --copy-as.
|
||||||
*
|
*
|
||||||
|
|||||||
+269
-251
@@ -44,99 +44,53 @@ bool receive_file_xattrs(File* file, int fd, const Config* config) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
static File* receive_delta_file(int fd, const Config* config, const char* check_path,
|
static bool receive_file_payload_into(File* file, int fd, const Config* config,
|
||||||
void* old_data, unsigned long long old_size, bool* failed) {
|
const char* dest_path, unsigned long long expected_size);
|
||||||
if (!old_data) {
|
|
||||||
free(old_data); /* defensive: old_data is always non-NULL today */
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
DeltaSignature* sig = delta_signature_create_seeded(old_data, old_size, config->delta_block_size,
|
/* Receive a STATUS_DELTA_DATA response: the sender's delta against the basis we
|
||||||
(uint32_t)config->checksum_seed);
|
signed. Deserializes, decompresses and applies the delta (in memory or
|
||||||
if (!sig) {
|
through a spool temp file), then receives the metadata/xattr block and
|
||||||
free(old_data);
|
installs the reconstructed payload. Takes ownership of `old_data` and `sig`,
|
||||||
*failed = true;
|
releasing both on every path. */
|
||||||
return NULL;
|
static File* receive_delta_data_branch(int fd, const Config* config, const char* check_path,
|
||||||
}
|
void* old_data, unsigned long long old_size, int basis_fd,
|
||||||
|
DeltaSignature* sig, bool* failed) {
|
||||||
|
Data* raw_delta = NULL;
|
||||||
|
Delta* delta = NULL;
|
||||||
|
void* new_data = NULL;
|
||||||
|
char* spool = NULL;
|
||||||
|
File* file = NULL;
|
||||||
|
|
||||||
Data* sig_data = delta_signature_serialize(sig);
|
raw_delta = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
||||||
if (!sig_data) {
|
if (!raw_delta)
|
||||||
delta_signature_destroy(sig);
|
goto fail;
|
||||||
free(old_data);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool sig_sent = send_status(fd, STATUS_DELTA_SIGNATURE) && send_data(fd, sig_data);
|
|
||||||
data_destroy(sig_data);
|
|
||||||
|
|
||||||
if (!sig_sent) {
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
free(old_data);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
Status resp;
|
|
||||||
if (!receive_status(fd, &resp)) {
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
free(old_data);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (resp == STATUS_DELTA_DATA) {
|
|
||||||
Data* delta_data = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
|
||||||
if (!delta_data) {
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
free(old_data);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
Data* raw_delta = delta_data;
|
|
||||||
if (config->use_compression &&
|
if (config->use_compression &&
|
||||||
!compression_should_skip_with_suffixes(
|
!compression_should_skip_with_suffixes(check_path, config->skip_compress_suffixes,
|
||||||
check_path, config->skip_compress_suffixes,
|
config->skip_compress_set ? config->skip_compress_count
|
||||||
config->skip_compress_set ? config->skip_compress_count : -1)) {
|
: -1)) {
|
||||||
ProtocolSession* owner = delta_data->owner;
|
ProtocolSession* owner = raw_delta->owner;
|
||||||
raw_delta = data_decompress_limited(delta_data, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
Data* decompressed = data_decompress_limited(raw_delta, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
||||||
data_destroy(delta_data);
|
data_destroy(raw_delta);
|
||||||
if (!raw_delta) {
|
raw_delta = decompressed;
|
||||||
free(old_data);
|
if (!raw_delta)
|
||||||
delta_signature_destroy(sig);
|
goto fail;
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
/* Charge the decompressed delta to the connection budget (the paired
|
/* Charge the decompressed delta to the connection budget (the paired
|
||||||
wire buffer's charge was just released). */
|
wire buffer's charge was just released). */
|
||||||
if (!data_charge_session(raw_delta, owner, raw_delta->size)) {
|
if (!data_charge_session(raw_delta, owner, raw_delta->size))
|
||||||
data_destroy(raw_delta);
|
goto fail;
|
||||||
free(old_data);
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
Delta* delta = delta_deserialize(raw_delta);
|
delta = delta_deserialize(raw_delta);
|
||||||
data_destroy(raw_delta);
|
data_destroy(raw_delta);
|
||||||
if (!delta) {
|
raw_delta = NULL;
|
||||||
free(old_data);
|
if (!delta)
|
||||||
delta_signature_destroy(sig);
|
goto fail;
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
uint64_t new_size = delta->new_file_size;
|
uint64_t new_size = delta->new_file_size;
|
||||||
if (new_size > MAX_RECEIVE_WHOLE_FILE_SIZE || new_size > SIZE_MAX) {
|
if (new_size > SIZE_MAX) {
|
||||||
delta_destroy(delta);
|
|
||||||
free(old_data);
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
send_status(fd, STATUS_ERROR);
|
send_status(fd, STATUS_ERROR);
|
||||||
*failed = true;
|
goto fail;
|
||||||
return NULL;
|
|
||||||
}
|
}
|
||||||
/* Wire-stats tally: bytes taken straight from the basis file (matched
|
/* Wire-stats tally: bytes taken straight from the basis file (matched
|
||||||
delta blocks) and bytes shipped literally (protocol 2.28.0). Computed
|
delta blocks) and bytes shipped literally (protocol 2.28.0). Computed
|
||||||
@@ -149,134 +103,180 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
|
|||||||
else if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
|
else if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
|
||||||
literal += delta->instructions[k].literal.length;
|
literal += delta->instructions[k].literal.length;
|
||||||
}
|
}
|
||||||
void* new_data = delta_apply(old_data, old_size, delta, config->delta_block_size);
|
|
||||||
|
/* A reconstructed file above the streaming bound is written into a spool
|
||||||
|
temp file through delta_apply_to_fd; a smaller one keeps the historical
|
||||||
|
in-memory reconstruction. */
|
||||||
|
if (new_size > protocol_whole_file_receive_limit()) {
|
||||||
|
char* dest_path = path_cat(config->receive_root_directory, check_path);
|
||||||
|
int spool_fd = dest_path ? file_spool_for_payload(dest_path, &spool) : -1;
|
||||||
|
free(dest_path);
|
||||||
|
if (spool_fd < 0) {
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
bool applied =
|
||||||
|
delta_apply_to_fd(old_data, basis_fd, old_size, delta, config->delta_block_size, spool_fd);
|
||||||
|
if (close(spool_fd) != 0)
|
||||||
|
applied = false;
|
||||||
delta_destroy(delta);
|
delta_destroy(delta);
|
||||||
|
delta = NULL;
|
||||||
if (!new_data) {
|
if (!applied) {
|
||||||
free(old_data);
|
send_status(fd, STATUS_ERROR);
|
||||||
delta_signature_destroy(sig);
|
goto fail;
|
||||||
*failed = true;
|
}
|
||||||
return NULL;
|
} else {
|
||||||
|
new_data = old_data ? delta_apply(old_data, old_size, delta, config->delta_block_size)
|
||||||
|
: delta_apply_fd(basis_fd, old_size, delta, config->delta_block_size);
|
||||||
|
delta_destroy(delta);
|
||||||
|
delta = NULL;
|
||||||
|
if (!new_data)
|
||||||
|
goto fail;
|
||||||
}
|
}
|
||||||
|
|
||||||
File* file = file_create(check_path);
|
file = file_create(check_path);
|
||||||
if (!file) {
|
if (!file)
|
||||||
free(new_data);
|
goto fail;
|
||||||
free(old_data);
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
file->matched_bytes = matched;
|
file->matched_bytes = matched;
|
||||||
file->literal_bytes = literal;
|
file->literal_bytes = literal;
|
||||||
|
|
||||||
if (config->use_metadata) {
|
if (config->use_metadata) {
|
||||||
int meta_ok = 1;
|
int meta_ok = 1;
|
||||||
file->metadata = metadata_receive(fd, &meta_ok);
|
file->metadata = metadata_receive(fd, &meta_ok);
|
||||||
if (!meta_ok) {
|
if (!meta_ok)
|
||||||
file_destroy(file);
|
goto fail;
|
||||||
free(new_data);
|
|
||||||
free(old_data);
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (!receive_file_xattrs(file, fd, config)) {
|
|
||||||
file_destroy(file);
|
|
||||||
free(new_data);
|
|
||||||
free(old_data);
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
}
|
||||||
|
if (!receive_file_xattrs(file, fd, config))
|
||||||
|
goto fail;
|
||||||
|
|
||||||
Data* replacement = data_create(new_data, (size_t)new_size);
|
if (spool) {
|
||||||
if (replacement == NULL) {
|
Data* reserved = data_create_reserve((size_t)new_size);
|
||||||
file_destroy(file);
|
if (reserved == NULL) {
|
||||||
free(old_data);
|
|
||||||
delta_signature_destroy(sig);
|
|
||||||
send_status(fd, STATUS_ERROR);
|
send_status(fd, STATUS_ERROR);
|
||||||
*failed = true;
|
goto fail;
|
||||||
return NULL;
|
}
|
||||||
|
data_destroy(file->data);
|
||||||
|
file->data = reserved;
|
||||||
|
file->basis_copy = spool;
|
||||||
|
spool = NULL; /* ownership moved into file->basis_copy */
|
||||||
|
file->data_spool = true;
|
||||||
|
} else {
|
||||||
|
Data* replacement = data_create(new_data, (size_t)new_size);
|
||||||
|
new_data = NULL; /* data_create owns, and frees, the buffer on failure */
|
||||||
|
if (replacement == NULL) {
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
|
goto fail;
|
||||||
}
|
}
|
||||||
data_destroy(file->data);
|
data_destroy(file->data);
|
||||||
file->data = replacement;
|
file->data = replacement;
|
||||||
|
}
|
||||||
|
|
||||||
free(old_data);
|
free(old_data);
|
||||||
delta_signature_destroy(sig);
|
delta_signature_destroy(sig);
|
||||||
return file;
|
return file;
|
||||||
}
|
|
||||||
|
|
||||||
if (resp == STATUS_NEXT) {
|
fail:
|
||||||
|
free(new_data);
|
||||||
|
if (spool) {
|
||||||
|
unlink(spool);
|
||||||
|
free(spool);
|
||||||
|
}
|
||||||
|
file_destroy(file);
|
||||||
|
delta_destroy(delta);
|
||||||
|
data_destroy(raw_delta);
|
||||||
|
free(old_data);
|
||||||
|
delta_signature_destroy(sig);
|
||||||
|
*failed = true;
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Receive a STATUS_NEXT response: the sender declined the delta and will send
|
||||||
|
the whole file. Releases the basis signature and snapshot, then receives the
|
||||||
|
metadata/xattr block and the full payload. Takes ownership of `old_data` and
|
||||||
|
`sig`, releasing both immediately. */
|
||||||
|
static File* receive_next_branch(int fd, const Config* config, const char* check_path,
|
||||||
|
unsigned long long expected_size, void* old_data,
|
||||||
|
DeltaSignature* sig, bool* failed) {
|
||||||
delta_signature_destroy(sig);
|
delta_signature_destroy(sig);
|
||||||
free(old_data);
|
free(old_data);
|
||||||
|
|
||||||
File* file = file_create(check_path);
|
File* file = file_create(check_path);
|
||||||
if (!file) {
|
if (!file)
|
||||||
*failed = true;
|
goto fail;
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (config->use_metadata) {
|
if (config->use_metadata) {
|
||||||
int meta_ok = 1;
|
int meta_ok = 1;
|
||||||
file->metadata = metadata_receive(fd, &meta_ok);
|
file->metadata = metadata_receive(fd, &meta_ok);
|
||||||
if (!meta_ok) {
|
if (!meta_ok)
|
||||||
file_destroy(file);
|
goto fail;
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (!receive_file_xattrs(file, fd, config)) {
|
|
||||||
file_destroy(file);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
}
|
||||||
|
if (!receive_file_xattrs(file, fd, config))
|
||||||
|
goto fail;
|
||||||
|
|
||||||
Data* file_data = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
char* dest_path = path_cat(config->receive_root_directory, check_path);
|
||||||
if (file_data == NULL) {
|
if (!dest_path)
|
||||||
file_destroy(file);
|
goto fail;
|
||||||
*failed = true;
|
bool payload_ok = receive_file_payload_into(file, fd, config, dest_path, expected_size);
|
||||||
return NULL;
|
free(dest_path);
|
||||||
}
|
if (!payload_ok)
|
||||||
|
goto fail;
|
||||||
if (config->use_compression &&
|
|
||||||
!compression_should_skip_with_suffixes(
|
|
||||||
file->path, config->skip_compress_suffixes,
|
|
||||||
config->skip_compress_set ? config->skip_compress_count : -1)) {
|
|
||||||
Data* uncompressed = data_decompress_limited(file_data, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
|
||||||
ProtocolSession* owner = file_data->owner;
|
|
||||||
data_destroy(file_data);
|
|
||||||
if (uncompressed == NULL) {
|
|
||||||
file_destroy(file);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
if (!data_charge_session(uncompressed, owner, uncompressed->size)) {
|
|
||||||
data_destroy(uncompressed);
|
|
||||||
file_destroy(file);
|
|
||||||
send_status(fd, STATUS_ERROR);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
if (uncompressed->size > MAX_FILE_DATA_SIZE) {
|
|
||||||
data_destroy(uncompressed);
|
|
||||||
file_destroy(file);
|
|
||||||
send_status(fd, STATUS_ERROR);
|
|
||||||
*failed = true;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
file_data = uncompressed;
|
|
||||||
}
|
|
||||||
|
|
||||||
data_destroy(file->data);
|
|
||||||
file->data = file_data;
|
|
||||||
return file;
|
return file;
|
||||||
}
|
|
||||||
|
|
||||||
|
fail:
|
||||||
|
file_destroy(file);
|
||||||
|
*failed = true;
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Delta handshake dispatcher: sign the basis, ship the signature, then hand the
|
||||||
|
response off to the matching branch helper. Takes ownership of `old_data`
|
||||||
|
(and, once created, `sig`); sets `*failed` on every error path. */
|
||||||
|
static File* receive_delta_file(int fd, const Config* config, const char* check_path,
|
||||||
|
void* old_data, unsigned long long old_size,
|
||||||
|
unsigned long long expected_size, int basis_fd, bool* failed) {
|
||||||
|
/* The basis is either an in-memory snapshot (the destination file, bounded) or
|
||||||
|
* a confined descriptor (a --fuzzy sibling, possibly larger than memory) that
|
||||||
|
* is signed/applied in bounded chunks. */
|
||||||
|
Data* sig_data = NULL;
|
||||||
|
Status resp = STATUS_ERROR;
|
||||||
|
bool sig_sent = false;
|
||||||
|
/* A delta check needs at least one basis source: the in-memory destination
|
||||||
|
* snapshot or a confined basis descriptor. */
|
||||||
|
if (!old_data && basis_fd < 0) {
|
||||||
|
*failed = true;
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
DeltaSignature* sig =
|
||||||
|
old_data ? delta_signature_create_seeded(old_data, old_size, config->delta_block_size,
|
||||||
|
(uint32_t)config->checksum_seed)
|
||||||
|
: delta_signature_create_fd_seeded(basis_fd, old_size, config->delta_block_size,
|
||||||
|
(uint32_t)config->checksum_seed);
|
||||||
|
if (!sig)
|
||||||
|
goto fail;
|
||||||
|
|
||||||
|
sig_data = delta_signature_serialize(sig);
|
||||||
|
if (!sig_data)
|
||||||
|
goto fail;
|
||||||
|
|
||||||
|
sig_sent = send_status(fd, STATUS_DELTA_SIGNATURE) && send_data(fd, sig_data);
|
||||||
|
data_destroy(sig_data);
|
||||||
|
sig_data = NULL;
|
||||||
|
|
||||||
|
if (!sig_sent || !receive_status(fd, &resp))
|
||||||
|
goto fail;
|
||||||
|
|
||||||
|
if (resp == STATUS_DELTA_DATA)
|
||||||
|
return receive_delta_data_branch(fd, config, check_path, old_data, old_size, basis_fd, sig,
|
||||||
|
failed);
|
||||||
|
|
||||||
|
if (resp == STATUS_NEXT)
|
||||||
|
return receive_next_branch(fd, config, check_path, expected_size, old_data, sig, failed);
|
||||||
|
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
|
fail:
|
||||||
|
data_destroy(sig_data);
|
||||||
delta_signature_destroy(sig);
|
delta_signature_destroy(sig);
|
||||||
free(old_data);
|
free(old_data);
|
||||||
send_status(fd, STATUS_ERROR);
|
|
||||||
*failed = true;
|
*failed = true;
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
@@ -640,28 +640,29 @@ static bool fuzzy_candidate_better(const FuzzyCandidate* cand, const FuzzyCandid
|
|||||||
return strcmp(cand->name, best->name) < 0;
|
return strcmp(cand->name, best->name) < 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Search the destination directory that will contain `check_path` for a
|
/* Search the destination directory that will contain `check_path` for a similar
|
||||||
* similar regular file usable as a --fuzzy delta basis and return its full
|
* regular file usable as a --fuzzy delta basis and return an open, confined
|
||||||
* content in a malloc'd (protocol_alloc) buffer. Returns NULL (with *out_size
|
* read descriptor to it (with *out_size set). Returns -1 (with *out_size 0)
|
||||||
* = 0) when no candidate qualifies, which means the caller performs the normal
|
* when no candidate qualifies, which means the caller performs the normal
|
||||||
* whole-file transfer. */
|
* whole-file transfer. The basis is signed/applied by streaming its descriptor,
|
||||||
static void* fuzzy_basis_find_and_load(const Config* config, const char* check_path,
|
* so no whole-basis buffer is ever needed and its size is not capped. */
|
||||||
|
static int fuzzy_basis_find_and_open(const Config* config, const char* check_path,
|
||||||
unsigned long long check_size, time_t check_mtime,
|
unsigned long long check_size, time_t check_mtime,
|
||||||
long check_mtime_nsec, unsigned long long* out_size) {
|
long check_mtime_nsec, unsigned long long* out_size) {
|
||||||
*out_size = 0;
|
*out_size = 0;
|
||||||
if (!config || !config->receive_root_directory || !config->fuzzy || !config->use_delta ||
|
if (!config || !config->receive_root_directory || !config->fuzzy || !config->use_delta ||
|
||||||
!check_path || check_size > MAX_RECEIVE_WHOLE_FILE_SIZE)
|
!check_path)
|
||||||
return NULL;
|
return -1;
|
||||||
|
|
||||||
char* full_path = path_cat(config->receive_root_directory, check_path);
|
char* full_path = path_cat(config->receive_root_directory, check_path);
|
||||||
if (!full_path)
|
if (!full_path)
|
||||||
return NULL;
|
return -1;
|
||||||
char* leaf = NULL;
|
char* leaf = NULL;
|
||||||
int dir_fd = file_open_secure_parent(full_path, &leaf, false);
|
int dir_fd = file_open_secure_parent(full_path, &leaf, false);
|
||||||
if (dir_fd < 0 || !leaf) {
|
if (dir_fd < 0 || !leaf) {
|
||||||
free(leaf);
|
free(leaf);
|
||||||
free(full_path);
|
free(full_path);
|
||||||
return NULL;
|
return -1;
|
||||||
}
|
}
|
||||||
size_t target_len = strlen(leaf);
|
size_t target_len = strlen(leaf);
|
||||||
/* A target basename longer than FUZZY_NAME_LIMIT can never pass the name gate
|
/* A target basename longer than FUZZY_NAME_LIMIT can never pass the name gate
|
||||||
@@ -670,7 +671,7 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
|||||||
close(dir_fd);
|
close(dir_fd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
free(full_path);
|
free(full_path);
|
||||||
return NULL;
|
return -1;
|
||||||
}
|
}
|
||||||
|
|
||||||
int scanfd = dup(dir_fd);
|
int scanfd = dup(dir_fd);
|
||||||
@@ -678,7 +679,7 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
|||||||
close(dir_fd);
|
close(dir_fd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
free(full_path);
|
free(full_path);
|
||||||
return NULL;
|
return -1;
|
||||||
}
|
}
|
||||||
DIR* dir = fdopendir(scanfd);
|
DIR* dir = fdopendir(scanfd);
|
||||||
if (!dir) {
|
if (!dir) {
|
||||||
@@ -686,7 +687,7 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
|||||||
close(dir_fd);
|
close(dir_fd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
free(full_path);
|
free(full_path);
|
||||||
return NULL;
|
return -1;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* The weighted-distance scratch row is allocated once per scan (not once per
|
/* The weighted-distance scratch row is allocated once per scan (not once per
|
||||||
@@ -697,7 +698,7 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
|||||||
close(dir_fd);
|
close(dir_fd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
free(full_path);
|
free(full_path);
|
||||||
return NULL;
|
return -1;
|
||||||
}
|
}
|
||||||
int fname_suf_len = 0;
|
int fname_suf_len = 0;
|
||||||
const char* fname_suf = fuzzy_find_suffix(leaf, (int)target_len, &fname_suf_len);
|
const char* fname_suf = fuzzy_find_suffix(leaf, (int)target_len, &fname_suf_len);
|
||||||
@@ -727,7 +728,7 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
|||||||
if (fstatat(dir_fd, name, &st, AT_SYMLINK_NOFOLLOW) != 0 || !S_ISREG(st.st_mode))
|
if (fstatat(dir_fd, name, &st, AT_SYMLINK_NOFOLLOW) != 0 || !S_ISREG(st.st_mode))
|
||||||
continue;
|
continue;
|
||||||
unsigned long long cand_size = (unsigned long long)st.st_size;
|
unsigned long long cand_size = (unsigned long long)st.st_size;
|
||||||
if (cand_size == 0 || cand_size > MAX_RECEIVE_WHOLE_FILE_SIZE)
|
if (cand_size == 0)
|
||||||
continue;
|
continue;
|
||||||
long cand_nsec = 0;
|
long cand_nsec = 0;
|
||||||
#ifdef __linux__
|
#ifdef __linux__
|
||||||
@@ -771,7 +772,7 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
|||||||
if (exact.name[0])
|
if (exact.name[0])
|
||||||
best = exact;
|
best = exact;
|
||||||
|
|
||||||
void* basis = NULL;
|
int basis_fd = -1;
|
||||||
if (best.name[0]) {
|
if (best.name[0]) {
|
||||||
/* O_NONBLOCK: a name raced to a FIFO between the fstatat gate and this open
|
/* O_NONBLOCK: a name raced to a FIFO between the fstatat gate and this open
|
||||||
would otherwise block the receive thread forever on open(2); with it the
|
would otherwise block the receive thread forever on open(2); with it the
|
||||||
@@ -781,29 +782,55 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
|||||||
if (fd >= 0) {
|
if (fd >= 0) {
|
||||||
struct stat st;
|
struct stat st;
|
||||||
if (fstat(fd, &st) == 0 && S_ISREG(st.st_mode) &&
|
if (fstat(fd, &st) == 0 && S_ISREG(st.st_mode) &&
|
||||||
(unsigned long long)st.st_size == best.size && best.size <= SIZE_MAX) {
|
(unsigned long long)st.st_size == best.size) {
|
||||||
basis = protocol_alloc((size_t)best.size);
|
basis_fd = fd;
|
||||||
if (basis) {
|
} else {
|
||||||
size_t got = 0;
|
|
||||||
while (got < (size_t)best.size) {
|
|
||||||
ssize_t n = read(fd, (char*)basis + got, (size_t)best.size - got);
|
|
||||||
if (n <= 0) {
|
|
||||||
free(basis);
|
|
||||||
basis = NULL;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
got += (size_t)n;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
close(fd);
|
close(fd);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
close(dir_fd);
|
close(dir_fd);
|
||||||
free(full_path);
|
free(full_path);
|
||||||
if (basis)
|
if (basis_fd >= 0)
|
||||||
*out_size = best.size;
|
*out_size = best.size;
|
||||||
return basis;
|
return basis_fd;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Receive one whole-file data frame into `file`. A payload at or below the
|
||||||
|
* receiver's streaming bound keeps the historical charged whole-buffer path; a
|
||||||
|
* larger one is streamed into a spool temp file (decompressing incrementally)
|
||||||
|
* and installed through the File's basis_copy field. `expected_size` is the
|
||||||
|
* logical size from the check frame (0 when unknown, e.g. the non-incremental
|
||||||
|
* path). */
|
||||||
|
static bool receive_file_payload_into(File* file, int fd, const Config* config,
|
||||||
|
const char* dest_path, unsigned long long expected_size) {
|
||||||
|
bool compress =
|
||||||
|
config->use_compression && !compression_should_skip_with_suffixes(
|
||||||
|
file->path, config->skip_compress_suffixes,
|
||||||
|
config->skip_compress_set ? config->skip_compress_count : -1);
|
||||||
|
Data* buffer = NULL;
|
||||||
|
char* spool = NULL;
|
||||||
|
unsigned long long size = 0;
|
||||||
|
if (!file_receive_payload(fd, compress, expected_size, dest_path,
|
||||||
|
protocol_whole_file_receive_limit(), &buffer, &spool, &size)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (spool) {
|
||||||
|
Data* reserved = data_create_reserve((size_t)size);
|
||||||
|
if (!reserved) {
|
||||||
|
unlink(spool);
|
||||||
|
free(spool);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
data_destroy(file->data);
|
||||||
|
file->data = reserved;
|
||||||
|
file->basis_copy = spool;
|
||||||
|
file->data_spool = true;
|
||||||
|
} else {
|
||||||
|
data_destroy(file->data);
|
||||||
|
file->data = buffer;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Read the remainder of a full-file transfer after the receiver has already
|
/* Read the remainder of a full-file transfer after the receiver has already
|
||||||
@@ -811,7 +838,8 @@ static void* fuzzy_basis_find_and_load(const Config* config, const char* check_p
|
|||||||
* data frame, and return an owned File. Shared by the plain full-transfer path
|
* data frame, and return an owned File. Shared by the plain full-transfer path
|
||||||
* and the --append-verify prefix-mismatch fallback (a clean full transfer
|
* and the --append-verify prefix-mismatch fallback (a clean full transfer
|
||||||
* instead of a corrupt prefix+tail blend). */
|
* instead of a corrupt prefix+tail blend). */
|
||||||
static File* receive_full_file(int fd, const Config* config, const char* path) {
|
static File* receive_full_file(int fd, const Config* config, const char* path,
|
||||||
|
unsigned long long expected_size) {
|
||||||
File* file = file_create(path);
|
File* file = file_create(path);
|
||||||
if (!file)
|
if (!file)
|
||||||
return NULL;
|
return NULL;
|
||||||
@@ -827,36 +855,17 @@ static File* receive_full_file(int fd, const Config* config, const char* path) {
|
|||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
Data* file_data = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
char* dest_path = path_cat(config->receive_root_directory, path);
|
||||||
if (file_data == NULL) {
|
if (!dest_path) {
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
if (config->use_compression &&
|
bool ok = receive_file_payload_into(file, fd, config, dest_path, expected_size);
|
||||||
!compression_should_skip_with_suffixes(file->path, config->skip_compress_suffixes,
|
free(dest_path);
|
||||||
config->skip_compress_set ? config->skip_compress_count
|
if (!ok) {
|
||||||
: -1)) {
|
|
||||||
Data* uncompressed = data_decompress_limited(file_data, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
|
||||||
ProtocolSession* owner = file_data->owner;
|
|
||||||
data_destroy(file_data);
|
|
||||||
if (uncompressed == NULL) {
|
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
if (!data_charge_session(uncompressed, owner, uncompressed->size)) {
|
|
||||||
data_destroy(uncompressed);
|
|
||||||
file_destroy(file);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
if (uncompressed->size > MAX_FILE_DATA_SIZE) {
|
|
||||||
data_destroy(uncompressed);
|
|
||||||
file_destroy(file);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
file_data = uncompressed;
|
|
||||||
}
|
|
||||||
data_destroy(file->data);
|
|
||||||
file->data = file_data;
|
|
||||||
return file;
|
return file;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -974,13 +983,12 @@ static IncrementalCheckOutcome incremental_check_receive_request(IncrementalChec
|
|||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* A basis-configured run may materialize a file larger than the whole-file
|
/* No file-size refusal: a whole-file payload larger than the historical
|
||||||
payload bound: a basis hit is streamed from the basis path (bounded
|
whole-file bound is streamed through a bounded buffer (see
|
||||||
buffers), so the check size is not itself an allocation. Every other
|
file_receive_payload). Only a size that cannot be represented on this
|
||||||
path (delta/append/full) still applies MAX_RECEIVE_WHOLE_FILE_SIZE, and a
|
platform is rejected. */
|
||||||
miss simply falls through to the normal transfer with its own bound. */
|
if (state->check_size > SIZE_MAX) {
|
||||||
if (!config_has_basis(config) && state->check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
|
send_error_detail(fd, "check size is not representable");
|
||||||
send_error_detail(fd, "check size exceeds receiver limit");
|
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1411,7 +1419,7 @@ static IncrementalCheckOutcome incremental_check_try_append_resume(IncrementalCh
|
|||||||
close(state->old_fd);
|
close(state->old_fd);
|
||||||
state->old_fd = -1;
|
state->old_fd = -1;
|
||||||
}
|
}
|
||||||
*out_file = receive_full_file(fd, config, check_path);
|
*out_file = receive_full_file(fd, config, check_path, check_size);
|
||||||
return INCREMENTAL_FILE;
|
return INCREMENTAL_FILE;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1428,20 +1436,24 @@ static IncrementalCheckOutcome incremental_check_try_append_resume(IncrementalCh
|
|||||||
if (config->use_metadata) {
|
if (config->use_metadata) {
|
||||||
int meta_ok = 1;
|
int meta_ok = 1;
|
||||||
meta = metadata_receive(fd, &meta_ok);
|
meta = metadata_receive(fd, &meta_ok);
|
||||||
if (!meta_ok)
|
if (!meta_ok) {
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
if (config->use_xattrs) {
|
if (config->use_xattrs) {
|
||||||
int xok = 0;
|
int xok = 0;
|
||||||
append_xattrs = xattr_receive(fd, &xok, config->preserve_acls);
|
append_xattrs = xattr_receive(fd, &xok, config->preserve_acls);
|
||||||
if (!xok) {
|
if (!xok) {
|
||||||
xattr_list_free(append_xattrs);
|
xattr_list_free(append_xattrs);
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Data* tail = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
Data* tail = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
|
||||||
if (tail == NULL) {
|
if (tail == NULL) {
|
||||||
xattr_list_free(append_xattrs);
|
xattr_list_free(append_xattrs);
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
if (config->use_compression &&
|
if (config->use_compression &&
|
||||||
@@ -1453,16 +1465,19 @@ static IncrementalCheckOutcome incremental_check_try_append_resume(IncrementalCh
|
|||||||
data_destroy(tail);
|
data_destroy(tail);
|
||||||
if (uncompressed == NULL) {
|
if (uncompressed == NULL) {
|
||||||
xattr_list_free(append_xattrs);
|
xattr_list_free(append_xattrs);
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
if (!data_charge_session(uncompressed, owner, uncompressed->size)) {
|
if (!data_charge_session(uncompressed, owner, uncompressed->size)) {
|
||||||
data_destroy(uncompressed);
|
data_destroy(uncompressed);
|
||||||
xattr_list_free(append_xattrs);
|
xattr_list_free(append_xattrs);
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
if (uncompressed->size > MAX_FILE_DATA_SIZE) {
|
if (uncompressed->size > MAX_FILE_DATA_SIZE) {
|
||||||
data_destroy(uncompressed);
|
data_destroy(uncompressed);
|
||||||
xattr_list_free(append_xattrs);
|
xattr_list_free(append_xattrs);
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
tail = uncompressed;
|
tail = uncompressed;
|
||||||
@@ -1475,6 +1490,7 @@ static IncrementalCheckOutcome incremental_check_try_append_resume(IncrementalCh
|
|||||||
send_status(fd, STATUS_ERROR);
|
send_status(fd, STATUS_ERROR);
|
||||||
data_destroy(tail);
|
data_destroy(tail);
|
||||||
xattr_list_free(append_xattrs);
|
xattr_list_free(append_xattrs);
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
size_t full_size = (size_t)check_size;
|
size_t full_size = (size_t)check_size;
|
||||||
@@ -1482,6 +1498,7 @@ static IncrementalCheckOutcome incremental_check_try_append_resume(IncrementalCh
|
|||||||
if (!full) {
|
if (!full) {
|
||||||
data_destroy(tail);
|
data_destroy(tail);
|
||||||
xattr_list_free(append_xattrs);
|
xattr_list_free(append_xattrs);
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
if (old_size > 0 && state->old_data)
|
if (old_size > 0 && state->old_data)
|
||||||
@@ -1496,6 +1513,7 @@ static IncrementalCheckOutcome incremental_check_try_append_resume(IncrementalCh
|
|||||||
if (!file) {
|
if (!file) {
|
||||||
free(full);
|
free(full);
|
||||||
xattr_list_free(append_xattrs);
|
xattr_list_free(append_xattrs);
|
||||||
|
file_metadata_destroy(meta);
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
file->metadata = meta;
|
file->metadata = meta;
|
||||||
@@ -1516,8 +1534,9 @@ static IncrementalCheckOutcome incremental_check_try_delta(IncrementalCheckState
|
|||||||
bool try_delta, File** out_file) {
|
bool try_delta, File** out_file) {
|
||||||
if (try_delta && state->old_data != NULL) {
|
if (try_delta && state->old_data != NULL) {
|
||||||
bool delta_failed = false;
|
bool delta_failed = false;
|
||||||
File* delta_file = receive_delta_file(state->fd, state->config, state->check_path,
|
File* delta_file =
|
||||||
state->old_data, state->old_size, &delta_failed);
|
receive_delta_file(state->fd, state->config, state->check_path, state->old_data,
|
||||||
|
state->old_size, state->check_size, -1, &delta_failed);
|
||||||
state->old_data = NULL; /* receive_delta_file consumes the snapshot on every path */
|
state->old_data = NULL; /* receive_delta_file consumes the snapshot on every path */
|
||||||
if (delta_file) {
|
if (delta_file) {
|
||||||
*out_file = delta_file;
|
*out_file = delta_file;
|
||||||
@@ -1540,14 +1559,14 @@ static IncrementalCheckOutcome incremental_check_try_fuzzy(IncrementalCheckState
|
|||||||
if (!config->fuzzy || !config->use_delta)
|
if (!config->fuzzy || !config->use_delta)
|
||||||
return INCREMENTAL_CONTINUE;
|
return INCREMENTAL_CONTINUE;
|
||||||
unsigned long long fuzzy_size = 0;
|
unsigned long long fuzzy_size = 0;
|
||||||
void* fuzzy_basis = fuzzy_basis_find_and_load(config, state->check_path, state->check_size,
|
int fuzzy_fd = fuzzy_basis_find_and_open(config, state->check_path, state->check_size,
|
||||||
(time_t)state->check_mtime,
|
(time_t)state->check_mtime,
|
||||||
(long)state->check_mtime_nsec, &fuzzy_size);
|
(long)state->check_mtime_nsec, &fuzzy_size);
|
||||||
if (fuzzy_basis != NULL) {
|
if (fuzzy_fd >= 0) {
|
||||||
bool fuzzy_failed = false;
|
bool fuzzy_failed = false;
|
||||||
File* fuzzy_file = receive_delta_file(state->fd, config, state->check_path, fuzzy_basis,
|
File* fuzzy_file = receive_delta_file(state->fd, config, state->check_path, NULL, fuzzy_size,
|
||||||
fuzzy_size, &fuzzy_failed);
|
state->check_size, fuzzy_fd, &fuzzy_failed);
|
||||||
fuzzy_basis = NULL; /* receive_delta_file consumes the buffer on every path */
|
close(fuzzy_fd);
|
||||||
if (fuzzy_file) {
|
if (fuzzy_file) {
|
||||||
*out_file = fuzzy_file;
|
*out_file = fuzzy_file;
|
||||||
return INCREMENTAL_FILE;
|
return INCREMENTAL_FILE;
|
||||||
@@ -1555,7 +1574,6 @@ static IncrementalCheckOutcome incremental_check_try_fuzzy(IncrementalCheckState
|
|||||||
if (fuzzy_failed)
|
if (fuzzy_failed)
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
free(fuzzy_basis);
|
|
||||||
return INCREMENTAL_CONTINUE;
|
return INCREMENTAL_CONTINUE;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1567,7 +1585,7 @@ static File* incremental_check_receive_full(IncrementalCheckState* state) {
|
|||||||
close(state->old_fd);
|
close(state->old_fd);
|
||||||
state->old_fd = -1;
|
state->old_fd = -1;
|
||||||
}
|
}
|
||||||
return receive_full_file(state->fd, state->config, state->check_path);
|
return receive_full_file(state->fd, state->config, state->check_path, state->check_size);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Core implementation. `would_transfer` (may be NULL) is set true only on the
|
/* Core implementation. `would_transfer` (may be NULL) is set true only on the
|
||||||
|
|||||||
+87
-20
@@ -1,4 +1,5 @@
|
|||||||
#include "log.h"
|
#include "log.h"
|
||||||
|
#include "utils.h"
|
||||||
#include <errno.h>
|
#include <errno.h>
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
#include <stdarg.h>
|
#include <stdarg.h>
|
||||||
@@ -16,6 +17,7 @@ static bool info_flags_explicit = false;
|
|||||||
static FILE* log_fp = NULL;
|
static FILE* log_fp = NULL;
|
||||||
static _Thread_local bool eight_bit_output;
|
static _Thread_local bool eight_bit_output;
|
||||||
static LogStderrMode stderr_mode = LOG_STDERR_ERRORS;
|
static LogStderrMode stderr_mode = LOG_STDERR_ERRORS;
|
||||||
|
static LogClientMsgSink client_msg_sink = NULL;
|
||||||
|
|
||||||
/* Serializes access to log_fp and makes each emitted line atomic: the
|
/* Serializes access to log_fp and makes each emitted line atomic: the
|
||||||
* timestamp prefix, formatted body, and trailing newline are written as one
|
* timestamp prefix, formatted body, and trailing newline are written as one
|
||||||
@@ -77,6 +79,51 @@ LogStderrMode log_get_stderr_mode(void) {
|
|||||||
return stderr_mode;
|
return stderr_mode;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void log_set_client_msg_sink(LogClientMsgSink sink) {
|
||||||
|
client_msg_sink = sink;
|
||||||
|
}
|
||||||
|
|
||||||
|
LogClientMsgSink log_get_client_msg_sink(void) {
|
||||||
|
return client_msg_sink;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Format just the message body (no prefix/newline) into a freshly allocated
|
||||||
|
* buffer. Shared by log_message (which may hand the body to a client-message
|
||||||
|
* sink) and log_client_message. Returns NULL on allocation/format failure. */
|
||||||
|
static char* format_log_body(const char* format, va_list args) {
|
||||||
|
va_list copy;
|
||||||
|
va_copy(copy, args);
|
||||||
|
int body_len = vsnprintf(NULL, 0, format, copy);
|
||||||
|
va_end(copy);
|
||||||
|
if (body_len < 0)
|
||||||
|
return NULL;
|
||||||
|
char* body = malloc((size_t)body_len + 1);
|
||||||
|
if (!body)
|
||||||
|
return NULL;
|
||||||
|
vsnprintf(body, (size_t)body_len + 1, format, args);
|
||||||
|
return body;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Assemble a complete log line (prefix + body + newline) from an already
|
||||||
|
* formatted body. Returns NULL on allocation failure. */
|
||||||
|
static char* format_log_line_from_body(LogLevel log_level, const struct tm* t, const char* body) {
|
||||||
|
char prefix[64];
|
||||||
|
int prefix_len = snprintf(
|
||||||
|
prefix, sizeof(prefix), "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t->tm_year + 1900,
|
||||||
|
t->tm_mon + 1, t->tm_mday, t->tm_hour, t->tm_min, t->tm_sec, log_level_strings[log_level]);
|
||||||
|
if (prefix_len < 0 || prefix_len >= (int)sizeof(prefix))
|
||||||
|
return NULL;
|
||||||
|
size_t body_len = strlen(body);
|
||||||
|
char* line = malloc((size_t)prefix_len + body_len + 2); /* body + '\n' + NUL */
|
||||||
|
if (!line)
|
||||||
|
return NULL;
|
||||||
|
memcpy(line, prefix, (size_t)prefix_len);
|
||||||
|
memcpy(line + prefix_len, body, body_len);
|
||||||
|
line[(size_t)prefix_len + body_len] = '\n';
|
||||||
|
line[(size_t)prefix_len + body_len + 1] = '\0';
|
||||||
|
return line;
|
||||||
|
}
|
||||||
|
|
||||||
/* Format one complete log line (timestamp prefix + body + newline) into a
|
/* Format one complete log line (timestamp prefix + body + newline) into a
|
||||||
* freshly allocated buffer. This is pure CPU/malloc work and must happen
|
* freshly allocated buffer. This is pure CPU/malloc work and must happen
|
||||||
* OUTSIDE the log mutex: the mutex only guards the log_fp pointer, so a
|
* OUTSIDE the log mutex: the mutex only guards the log_fp pointer, so a
|
||||||
@@ -84,26 +131,11 @@ LogStderrMode log_get_stderr_mode(void) {
|
|||||||
* on allocation/formatting failure. */
|
* on allocation/formatting failure. */
|
||||||
static char* format_log_line(LogLevel log_level, const struct tm* t, const char* format,
|
static char* format_log_line(LogLevel log_level, const struct tm* t, const char* format,
|
||||||
va_list args) {
|
va_list args) {
|
||||||
char prefix[64];
|
char* body = format_log_body(format, args);
|
||||||
int prefix_len = snprintf(
|
if (!body)
|
||||||
prefix, sizeof(prefix), "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t->tm_year + 1900,
|
|
||||||
t->tm_mon + 1, t->tm_mday, t->tm_hour, t->tm_min, t->tm_sec, log_level_strings[log_level]);
|
|
||||||
if (prefix_len < 0 || prefix_len >= (int)sizeof(prefix))
|
|
||||||
return NULL;
|
return NULL;
|
||||||
va_list copy;
|
char* line = format_log_line_from_body(log_level, t, body);
|
||||||
va_copy(copy, args);
|
free(body);
|
||||||
int body_len = vsnprintf(NULL, 0, format, copy);
|
|
||||||
va_end(copy);
|
|
||||||
if (body_len < 0)
|
|
||||||
return NULL;
|
|
||||||
size_t total = (size_t)prefix_len + (size_t)body_len;
|
|
||||||
char* line = malloc(total + 2); /* body bytes + '\n' + NUL */
|
|
||||||
if (!line)
|
|
||||||
return NULL;
|
|
||||||
memcpy(line, prefix, (size_t)prefix_len);
|
|
||||||
vsnprintf(line + prefix_len, (size_t)body_len + 1, format, args);
|
|
||||||
line[total] = '\n';
|
|
||||||
line[total + 1] = '\0';
|
|
||||||
return line;
|
return line;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -121,6 +153,26 @@ static void emit_log_line(FILE* console, const char* line) {
|
|||||||
mtx_unlock(&log_mutex);
|
mtx_unlock(&log_mutex);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void log_client_message(const char* message) {
|
||||||
|
if (!message)
|
||||||
|
return;
|
||||||
|
/* The body is peer-controlled: escape every non-printable byte (newlines,
|
||||||
|
CR, ANSI ESC, ...) so a hostile client cannot forge log lines or inject
|
||||||
|
terminal control sequences. output_escape() is the codebase's canonical
|
||||||
|
escaper and leaves printable text untouched. */
|
||||||
|
char* escaped = output_escape(message, log_get_8_bit_output());
|
||||||
|
if (!escaped)
|
||||||
|
return;
|
||||||
|
/* Route through the ordinary log level / destination gate (log_message):
|
||||||
|
this respects --log-file, the configured stderr mode and the level
|
||||||
|
threshold instead of always writing to stderr. The wire body carries no
|
||||||
|
severity, so the forwarded diagnostic is emitted as a warning -- the
|
||||||
|
lowest level the default gate admits, which keeps the peer's messages
|
||||||
|
visible without bypassing --quiet. */
|
||||||
|
log_message(LOG_LEVEL_WARNING, "%s", escaped);
|
||||||
|
free(escaped);
|
||||||
|
}
|
||||||
|
|
||||||
void log_message(LogLevel log_level, const char* format, ...) {
|
void log_message(LogLevel log_level, const char* format, ...) {
|
||||||
if (log_level < current_log_level)
|
if (log_level < current_log_level)
|
||||||
return;
|
return;
|
||||||
@@ -138,8 +190,23 @@ void log_message(LogLevel log_level, const char* format, ...) {
|
|||||||
|
|
||||||
va_list args;
|
va_list args;
|
||||||
va_start(args, format);
|
va_start(args, format);
|
||||||
char* line = format_log_line(log_level, &t, format, args);
|
char* body = format_log_body(format, args);
|
||||||
va_end(args);
|
va_end(args);
|
||||||
|
if (!body)
|
||||||
|
return;
|
||||||
|
/* LOG_STDERR_CLIENT: hand the diagnostic to the client-message channel. A
|
||||||
|
sink that takes ownership suppresses the local write; otherwise (no sink
|
||||||
|
yet, or the peer connection is not up) fall through to local output so the
|
||||||
|
diagnostic is never lost. */
|
||||||
|
if (stderr_mode == LOG_STDERR_CLIENT) {
|
||||||
|
LogClientMsgSink sink = client_msg_sink;
|
||||||
|
if (sink && sink(body)) {
|
||||||
|
free(body);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
char* line = format_log_line_from_body(log_level, &t, body);
|
||||||
|
free(body);
|
||||||
if (!line)
|
if (!line)
|
||||||
return;
|
return;
|
||||||
emit_log_line(dest_io, line);
|
emit_log_line(dest_io, line);
|
||||||
|
|||||||
+20
-1
@@ -15,7 +15,11 @@
|
|||||||
#endif
|
#endif
|
||||||
|
|
||||||
typedef enum { LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_WARNING, LOG_LEVEL_ERROR } LogLevel;
|
typedef enum { LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_WARNING, LOG_LEVEL_ERROR } LogLevel;
|
||||||
typedef enum { LOG_STDERR_ERRORS, LOG_STDERR_ALL } LogStderrMode;
|
/* --stderr=MODE destinations. ERRORS keeps errors on stderr and everything
|
||||||
|
* else on stdout; ALL sends every message to stderr; CLIENT routes the client's
|
||||||
|
* own diagnostics over the protocol stream to the peer's stderr (rsync's
|
||||||
|
* --stderr=client / --no-msgs2stderr). */
|
||||||
|
typedef enum { LOG_STDERR_ERRORS, LOG_STDERR_ALL, LOG_STDERR_CLIENT } LogStderrMode;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LOG_DEBUG_IO = 1u << 0,
|
LOG_DEBUG_IO = 1u << 0,
|
||||||
@@ -86,5 +90,20 @@ void log_set_8_bit_output(bool enabled);
|
|||||||
bool log_get_8_bit_output(void);
|
bool log_get_8_bit_output(void);
|
||||||
void log_set_stderr_mode(LogStderrMode mode);
|
void log_set_stderr_mode(LogStderrMode mode);
|
||||||
LogStderrMode log_get_stderr_mode(void);
|
LogStderrMode log_get_stderr_mode(void);
|
||||||
|
/* Write a message a peer forwarded over the client-message channel to this
|
||||||
|
* process's stderr (and log file), with the standard log prefix. Used by the
|
||||||
|
* server side of rsync's --stderr=client. */
|
||||||
|
void log_client_message(const char* message);
|
||||||
|
|
||||||
|
/* Sink for LOG_STDERR_CLIENT. log_message() passes the un-prefixed message
|
||||||
|
* body to the installed sink; a `true` return means the sink took ownership
|
||||||
|
* (e.g. queued it for protocol transmission) and the message must NOT also be
|
||||||
|
* written locally. A `false` return (or a NULL sink) makes log_message fall
|
||||||
|
* back to the normal local destination, so a diagnostic emitted before the peer
|
||||||
|
* connection exists is never lost (rsync's documented fallback). The sink may
|
||||||
|
* be called from any thread and must be tolerant of that. */
|
||||||
|
typedef bool (*LogClientMsgSink)(const char* message);
|
||||||
|
void log_set_client_msg_sink(LogClientMsgSink sink);
|
||||||
|
LogClientMsgSink log_get_client_msg_sink(void);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
@@ -34,9 +34,11 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
|
|||||||
context->synced_dirs = NULL;
|
context->synced_dirs = NULL;
|
||||||
context->plan_dirs = NULL;
|
context->plan_dirs = NULL;
|
||||||
context->missing_args = NULL;
|
context->missing_args = NULL;
|
||||||
|
context->per_dir_rules = NULL;
|
||||||
context->scan_had_io_error = false;
|
context->scan_had_io_error = false;
|
||||||
context->remove_source_files = NULL;
|
context->remove_source_files = NULL;
|
||||||
context->early_delete = false;
|
context->early_delete = false;
|
||||||
|
context->prescan_chunks = NULL;
|
||||||
context->delete_plans = NULL;
|
context->delete_plans = NULL;
|
||||||
context->delete_suppressed = false;
|
context->delete_suppressed = false;
|
||||||
context->scan_stopped_early = false;
|
context->scan_stopped_early = false;
|
||||||
@@ -52,6 +54,7 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
|
|||||||
context->dir_entries_mutex_init = false;
|
context->dir_entries_mutex_init = false;
|
||||||
atomic_init(&context->dir_count, 0);
|
atomic_init(&context->dir_count, 0);
|
||||||
context->delete_limit = false;
|
context->delete_limit = false;
|
||||||
|
context->partial = false;
|
||||||
int init = 0;
|
int init = 0;
|
||||||
if (config->use_metadata) {
|
if (config->use_metadata) {
|
||||||
context->dir_entries = array_list_create(file_destroy);
|
context->dir_entries = array_list_create(file_destroy);
|
||||||
@@ -194,6 +197,8 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
|
|||||||
if (context->manifest) {
|
if (context->manifest) {
|
||||||
array_list_delete(context->manifest);
|
array_list_delete(context->manifest);
|
||||||
}
|
}
|
||||||
|
if (context->prescan_chunks)
|
||||||
|
array_list_delete(context->prescan_chunks);
|
||||||
if (context->delete_plans)
|
if (context->delete_plans)
|
||||||
delete_plan_sender_destroy(context->delete_plans);
|
delete_plan_sender_destroy(context->delete_plans);
|
||||||
if (context->excluded_paths)
|
if (context->excluded_paths)
|
||||||
@@ -206,6 +211,8 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
|
|||||||
array_list_delete(context->plan_dirs);
|
array_list_delete(context->plan_dirs);
|
||||||
if (context->missing_args)
|
if (context->missing_args)
|
||||||
array_list_delete(context->missing_args);
|
array_list_delete(context->missing_args);
|
||||||
|
if (context->per_dir_rules)
|
||||||
|
filter_rule_list_free(context->per_dir_rules);
|
||||||
if (context->remove_source_files)
|
if (context->remove_source_files)
|
||||||
array_list_delete(context->remove_source_files);
|
array_list_delete(context->remove_source_files);
|
||||||
if (context->dir_entries)
|
if (context->dir_entries)
|
||||||
|
|||||||
@@ -67,6 +67,12 @@ typedef struct {
|
|||||||
them in the manifest frame's third section and the receiver deletes each as
|
them in the manifest frame's third section and the receiver deletes each as
|
||||||
an explicit request. */
|
an explicit request. */
|
||||||
ArrayList* missing_args;
|
ArrayList* missing_args;
|
||||||
|
/* Per-directory filter rules the source scan compiled (protocol 2.30.0),
|
||||||
|
sent with the delete manifest/plan config so the receiver can re-derive the
|
||||||
|
per-directory protect/risk set. NULL when --delete is off. Populated by
|
||||||
|
the scanner (parallel workers append under mutex_scanner) or the early
|
||||||
|
pre-scan. */
|
||||||
|
FilterRuleList* per_dir_rules;
|
||||||
/* A source I/O error (unreadable directory) was recorded during the scan.
|
/* A source I/O error (unreadable directory) was recorded during the scan.
|
||||||
Set by the pre-scan (before the threads start) or by the scanner thread
|
Set by the pre-scan (before the threads start) or by the scanner thread
|
||||||
under mutex_scanner; the caller turns it into a non-zero exit when
|
under mutex_scanner; the caller turns it into a non-zero exit when
|
||||||
@@ -78,6 +84,13 @@ typedef struct {
|
|||||||
path-only pre-scan on the calling thread and the pipeline scanner must not
|
path-only pre-scan on the calling thread and the pipeline scanner must not
|
||||||
append to it. Set once before the worker threads start. */
|
append to it. Set once before the worker threads start. */
|
||||||
bool early_delete;
|
bool early_delete;
|
||||||
|
/* --delete-before: the path-only pre-scan that built the early keep-set,
|
||||||
|
retained as the pipeline's file list (owning Chunk*; consumed and NULLed by
|
||||||
|
the scanner thread) so the data pass replays rsync's single file list
|
||||||
|
instead of re-reading the source. NULL in every other mode, where the
|
||||||
|
scanner thread scans normally. Set once before the worker threads start
|
||||||
|
and freed with the context. */
|
||||||
|
ArrayList* prescan_chunks;
|
||||||
/* Non-NULL for --delete-during/--delete-delay: the per-directory plan set
|
/* Non-NULL for --delete-during/--delete-delay: the per-directory plan set
|
||||||
prebuilt by the path-only pre-scan on the calling thread. The sender
|
prebuilt by the path-only pre-scan on the calling thread. The sender
|
||||||
thread transmits the root plan before any data and the remaining plans
|
thread transmits the root plan before any data and the remaining plans
|
||||||
@@ -127,6 +140,11 @@ typedef struct {
|
|||||||
deletion (STATUS_DELETE_LIMIT): the transfer succeeded and the process must
|
deletion (STATUS_DELETE_LIMIT): the transfer succeeded and the process must
|
||||||
exit 25 like rsync. Read by the caller after the sender thread is joined. */
|
exit 25 like rsync. Read by the caller after the sender thread is joined. */
|
||||||
bool delete_limit;
|
bool delete_limit;
|
||||||
|
/* Set by the sender thread when the receiver reported STATUS_PARTIAL (a
|
||||||
|
per-entry receiver failure that did not abort the stream): the transfer
|
||||||
|
otherwise succeeded, successfully stored --remove-source-files sources were
|
||||||
|
removed, and the process must exit 23 like rsync. Read after join. */
|
||||||
|
bool partial;
|
||||||
} PipelineContextSender;
|
} PipelineContextSender;
|
||||||
|
|
||||||
/* `config` is borrowed and must outlive the context: destroy does NOT free it,
|
/* `config` is borrowed and must outlive the context: destroy does NOT free it,
|
||||||
|
|||||||
+244
-91
@@ -38,6 +38,150 @@ static atomic_ullong io_bytes_read = 0;
|
|||||||
|
|
||||||
static unsigned long long global_bwlimit(void);
|
static unsigned long long global_bwlimit(void);
|
||||||
|
|
||||||
|
/* Runtime whole-file receive bound (see protocol.h). Resolved once; an
|
||||||
|
* override can only LOWER the ceiling, never raise it above the protocol
|
||||||
|
* constant, so the wire/security bound is unchanged. A parse failure or a
|
||||||
|
* non-positive value leaves the default in place. */
|
||||||
|
unsigned long long protocol_whole_file_receive_limit(void) {
|
||||||
|
static atomic_ullong cached = 0;
|
||||||
|
unsigned long long value = atomic_load_explicit(&cached, memory_order_relaxed);
|
||||||
|
if (value != 0)
|
||||||
|
return value;
|
||||||
|
value = MAX_RECEIVE_WHOLE_FILE_SIZE;
|
||||||
|
const char* env = getenv("FASTSYNC_MAX_WHOLE_FILE_SIZE");
|
||||||
|
if (env && env[0] != '\0') {
|
||||||
|
char* end = NULL;
|
||||||
|
unsigned long long parsed = strtoull(env, &end, 10);
|
||||||
|
if (end && *end == '\0' && parsed > 0 && parsed < value)
|
||||||
|
value = parsed;
|
||||||
|
}
|
||||||
|
atomic_store_explicit(&cached, value, memory_order_relaxed);
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------------- *
|
||||||
|
* Transport vtable implementations.
|
||||||
|
*
|
||||||
|
* Each op performs exactly one transfer attempt. WANT_READ/WANT_WRITE and an
|
||||||
|
* EINTR-interrupted syscall are reported as PROTOCOL_IO_RETRY (with
|
||||||
|
* *wait_events set to the poll event the caller must wait on); a clean peer
|
||||||
|
* close is PROTOCOL_IO_CLOSED and anything else is PROTOCOL_IO_ERROR. This
|
||||||
|
* keeps every WANT_READ/WANT_WRITE and EINTR retry exactly where it was before
|
||||||
|
* the vtable was introduced, just moved behind the function pointer.
|
||||||
|
* ------------------------------------------------------------------------- */
|
||||||
|
|
||||||
|
static ssize_t plain_io_send(ProtocolSession* session, const void* data, size_t size,
|
||||||
|
short* wait_events) {
|
||||||
|
ssize_t written = write(session->write_fd, data, size);
|
||||||
|
if (written < 0) {
|
||||||
|
if (errno == EINTR)
|
||||||
|
return PROTOCOL_IO_RETRY;
|
||||||
|
return PROTOCOL_IO_ERROR;
|
||||||
|
}
|
||||||
|
if (written == 0)
|
||||||
|
return PROTOCOL_IO_ERROR;
|
||||||
|
*wait_events = POLLOUT;
|
||||||
|
return written;
|
||||||
|
}
|
||||||
|
|
||||||
|
static ssize_t plain_io_recv(ProtocolSession* session, void* data, size_t size,
|
||||||
|
short* wait_events) {
|
||||||
|
ssize_t received = read(session->read_fd, data, size);
|
||||||
|
if (received < 0) {
|
||||||
|
if (errno == EINTR)
|
||||||
|
return PROTOCOL_IO_RETRY;
|
||||||
|
return PROTOCOL_IO_ERROR;
|
||||||
|
}
|
||||||
|
if (received == 0)
|
||||||
|
return PROTOCOL_IO_CLOSED;
|
||||||
|
*wait_events = POLLIN;
|
||||||
|
return received;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool plain_io_has_pending(const ProtocolSession* session) {
|
||||||
|
(void)session;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
static ssize_t tls_io_send(ProtocolSession* session, const void* data, size_t size,
|
||||||
|
short* wait_events) {
|
||||||
|
/* SSL_write takes an int length; clamp a >INT_MAX request into chunks so the
|
||||||
|
* size_t downcast can never truncate into a negative/partial write. */
|
||||||
|
size_t chunk = size > (size_t)INT_MAX ? (size_t)INT_MAX : size;
|
||||||
|
ssize_t written = SSL_write(session->ssl, data, (int)chunk);
|
||||||
|
if (written <= 0) {
|
||||||
|
int ssl_err = SSL_get_error(session->ssl, (int)written);
|
||||||
|
if (ssl_err == SSL_ERROR_WANT_WRITE) {
|
||||||
|
*wait_events = POLLOUT;
|
||||||
|
return PROTOCOL_IO_RETRY;
|
||||||
|
}
|
||||||
|
if (ssl_err == SSL_ERROR_WANT_READ) {
|
||||||
|
*wait_events = POLLIN;
|
||||||
|
return PROTOCOL_IO_RETRY;
|
||||||
|
}
|
||||||
|
/* A signal (e.g. Ctrl-C) interrupts the blocking TLS write: retry so the
|
||||||
|
* send loop can observe the abort flag at the next checkpoint. Only an
|
||||||
|
* actual negative return is an interrupted syscall; a 0-byte SSL_write is
|
||||||
|
* not a valid EINTR retry. */
|
||||||
|
if (written < 0 && ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
|
||||||
|
return PROTOCOL_IO_RETRY;
|
||||||
|
return PROTOCOL_IO_ERROR;
|
||||||
|
}
|
||||||
|
*wait_events = POLLOUT;
|
||||||
|
return written;
|
||||||
|
}
|
||||||
|
|
||||||
|
static ssize_t tls_io_recv(ProtocolSession* session, void* data, size_t size, short* wait_events) {
|
||||||
|
/* SSL_read takes an int length; clamp a >INT_MAX request into chunks
|
||||||
|
* (mirrors the send path) so the size_t downcast can never truncate into a
|
||||||
|
* negative/partial read. */
|
||||||
|
size_t chunk = size > (size_t)INT_MAX ? (size_t)INT_MAX : size;
|
||||||
|
ssize_t received = SSL_read(session->ssl, data, (int)chunk);
|
||||||
|
if (received <= 0) {
|
||||||
|
int ssl_err = SSL_get_error(session->ssl, (int)received);
|
||||||
|
if (ssl_err == SSL_ERROR_WANT_WRITE) {
|
||||||
|
*wait_events = POLLOUT;
|
||||||
|
return PROTOCOL_IO_RETRY;
|
||||||
|
}
|
||||||
|
if (ssl_err == SSL_ERROR_WANT_READ) {
|
||||||
|
*wait_events = POLLIN;
|
||||||
|
return PROTOCOL_IO_RETRY;
|
||||||
|
}
|
||||||
|
/* A signal interrupts the blocking TLS read: retry (mirrors the send path)
|
||||||
|
* so the loop reaches its next abort/deadline checkpoint. Only an actual
|
||||||
|
* negative return is an interrupted syscall: a 0-byte SSL_read is an
|
||||||
|
* unexpected EOF (the peer closed without close_notify), which OpenSSL also
|
||||||
|
* reports as SSL_ERROR_SYSCALL with errno possibly still EINTR from an
|
||||||
|
* earlier interrupted poll/read. Retrying that would busy-spin the
|
||||||
|
* status-read loop until its deadline, so classify it as closed instead. */
|
||||||
|
if (received < 0 && ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
|
||||||
|
return PROTOCOL_IO_RETRY;
|
||||||
|
/* A zero-length SSL_read is the peer's clean close_notify (or EOF without
|
||||||
|
* one); report it distinctly so the caller can log it as a close. */
|
||||||
|
if (received == 0)
|
||||||
|
return PROTOCOL_IO_CLOSED;
|
||||||
|
return PROTOCOL_IO_ERROR;
|
||||||
|
}
|
||||||
|
*wait_events = POLLIN;
|
||||||
|
return received;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool tls_io_has_pending(const ProtocolSession* session) {
|
||||||
|
return session->ssl != NULL && SSL_pending(session->ssl) > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static const ProtocolIoOps plain_io_ops = {
|
||||||
|
.send = plain_io_send,
|
||||||
|
.recv = plain_io_recv,
|
||||||
|
.has_pending = plain_io_has_pending,
|
||||||
|
};
|
||||||
|
|
||||||
|
static const ProtocolIoOps tls_io_ops = {
|
||||||
|
.send = tls_io_send,
|
||||||
|
.recv = tls_io_recv,
|
||||||
|
.has_pending = tls_io_has_pending,
|
||||||
|
};
|
||||||
|
|
||||||
static bool protocol_reserve_memory(ProtocolSession* session, size_t charge) {
|
static bool protocol_reserve_memory(ProtocolSession* session, size_t charge) {
|
||||||
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
|
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
|
||||||
while (true) {
|
while (true) {
|
||||||
@@ -79,6 +223,7 @@ void io_set_fds(int read_fd, int write_fd) {
|
|||||||
legacy_io_session.read_fd = read_fd;
|
legacy_io_session.read_fd = read_fd;
|
||||||
legacy_io_session.write_fd = write_fd;
|
legacy_io_session.write_fd = write_fd;
|
||||||
legacy_io_session.ssl = NULL;
|
legacy_io_session.ssl = NULL;
|
||||||
|
legacy_io_session.ops = &plain_io_ops;
|
||||||
legacy_io_session.eight_bit_output = false;
|
legacy_io_session.eight_bit_output = false;
|
||||||
atomic_store(&legacy_io_session.total_allocated_bytes, 0);
|
atomic_store(&legacy_io_session.total_allocated_bytes, 0);
|
||||||
legacy_io_session.max_alloc = DEFAULT_MAX_ALLOC;
|
legacy_io_session.max_alloc = DEFAULT_MAX_ALLOC;
|
||||||
@@ -91,6 +236,7 @@ void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd)
|
|||||||
memset(session, 0, sizeof(*session));
|
memset(session, 0, sizeof(*session));
|
||||||
session->read_fd = read_fd;
|
session->read_fd = read_fd;
|
||||||
session->write_fd = write_fd;
|
session->write_fd = write_fd;
|
||||||
|
session->ops = &plain_io_ops;
|
||||||
session->max_alloc = DEFAULT_MAX_ALLOC;
|
session->max_alloc = DEFAULT_MAX_ALLOC;
|
||||||
session->io_timeout_sec = RECEIVE_TIMEOUT_SEC;
|
session->io_timeout_sec = RECEIVE_TIMEOUT_SEC;
|
||||||
atomic_init(&session->total_allocated_bytes, 0);
|
atomic_init(&session->total_allocated_bytes, 0);
|
||||||
@@ -158,8 +304,12 @@ void protocol_session_unbind(void) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
void protocol_session_set_ssl(ProtocolSession* session, SSL* ssl) {
|
void protocol_session_set_ssl(ProtocolSession* session, SSL* ssl) {
|
||||||
if (session)
|
if (!session)
|
||||||
|
return;
|
||||||
session->ssl = ssl;
|
session->ssl = ssl;
|
||||||
|
/* Select the transport dispatch once, here, instead of branching on the SSL
|
||||||
|
* pointer inside every I/O loop. */
|
||||||
|
session->ops = ssl ? &tls_io_ops : &plain_io_ops;
|
||||||
}
|
}
|
||||||
|
|
||||||
static void bw_mutex_init(void) {
|
static void bw_mutex_init(void) {
|
||||||
@@ -270,6 +420,20 @@ SSL* io_get_ssl(void) {
|
|||||||
return io_ssl;
|
return io_ssl;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
SSL* protocol_current_ssl(void) {
|
||||||
|
/* The bound session is the authoritative transport for a worker thread: it
|
||||||
|
* was explicitly handed to protocol_session_bind() and carries its own SSL,
|
||||||
|
* whereas io_ssl is thread-local and NULL in a thread that never performed
|
||||||
|
* the handshake. Only a session whose selected dispatch is TLS may supply
|
||||||
|
* the SSL: a bound plaintext session has ssl == NULL and must not shadow a
|
||||||
|
* live thread-local io_ssl, or file_send.c would take the raw sendfile(2)
|
||||||
|
* path on a socket this thread is driving with TLS. With no TLS session
|
||||||
|
* bound (plaintext session, or the fd-shim path), fall back to io_ssl. */
|
||||||
|
if (bound_session && bound_session->ops == &tls_io_ops && bound_session->ssl)
|
||||||
|
return bound_session->ssl;
|
||||||
|
return io_ssl;
|
||||||
|
}
|
||||||
|
|
||||||
unsigned long long protocol_bytes_written(void) {
|
unsigned long long protocol_bytes_written(void) {
|
||||||
return atomic_load(&io_bytes_written);
|
return atomic_load(&io_bytes_written);
|
||||||
}
|
}
|
||||||
@@ -298,6 +462,7 @@ static ProtocolSession* legacy_session(int read_fd, int write_fd) {
|
|||||||
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
|
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
|
||||||
}
|
}
|
||||||
legacy_io_session.ssl = io_ssl;
|
legacy_io_session.ssl = io_ssl;
|
||||||
|
legacy_io_session.ops = io_ssl ? &tls_io_ops : &plain_io_ops;
|
||||||
return &legacy_io_session;
|
return &legacy_io_session;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -335,8 +500,9 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
|
|||||||
if (!data && data_size != 0)
|
if (!data && data_size != 0)
|
||||||
return false;
|
return false;
|
||||||
log_debug_message(LOG_DEBUG_IO, " Sending n Data: %zu", data_size);
|
log_debug_message(LOG_DEBUG_IO, " Sending n Data: %zu", data_size);
|
||||||
if (!session)
|
if (!session || !session->ops)
|
||||||
return false;
|
return false;
|
||||||
|
const ProtocolIoOps* ops = session->ops;
|
||||||
/* A non-positive session timeout disables the deadline entirely (rsync's
|
/* A non-positive session timeout disables the deadline entirely (rsync's
|
||||||
* --timeout=0 default); poll then blocks until the socket becomes writable. */
|
* --timeout=0 default); poll then blocks until the socket becomes writable. */
|
||||||
int timeout_sec = session->io_timeout_sec > 0 ? session->io_timeout_sec : 0;
|
int timeout_sec = session->io_timeout_sec > 0 ? session->io_timeout_sec : 0;
|
||||||
@@ -362,36 +528,16 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
|
|||||||
continue;
|
continue;
|
||||||
if (pfd.revents & (POLLERR | POLLNVAL))
|
if (pfd.revents & (POLLERR | POLLNVAL))
|
||||||
return false;
|
return false;
|
||||||
ssize_t bytes_send;
|
ssize_t bytes_send =
|
||||||
if (session->ssl) {
|
ops->send(session, (const char*)data + total_bytes_send, chunk, &wait_events);
|
||||||
/* SSL_write takes an int length; clamp a >INT_MAX request into chunks so
|
if (bytes_send == PROTOCOL_IO_RETRY)
|
||||||
* the size_t downcast can never truncate into a negative/partial write. */
|
continue;
|
||||||
size_t ssl_chunk = chunk > (size_t)INT_MAX ? (size_t)INT_MAX : chunk;
|
|
||||||
bytes_send = SSL_write(session->ssl, (const char*)data + total_bytes_send, (int)ssl_chunk);
|
|
||||||
} else {
|
|
||||||
bytes_send = write(fd, (const char*)data + total_bytes_send, chunk);
|
|
||||||
}
|
|
||||||
if (bytes_send <= 0) {
|
if (bytes_send <= 0) {
|
||||||
if (session->ssl) {
|
|
||||||
int ssl_err = SSL_get_error(session->ssl, (int)bytes_send);
|
|
||||||
if (ssl_err == SSL_ERROR_WANT_WRITE || ssl_err == SSL_ERROR_WANT_READ) {
|
|
||||||
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
/* A signal (e.g. Ctrl-C) interrupts the blocking TLS write: retry so
|
|
||||||
the send loop can observe the abort flag at the next checkpoint. */
|
|
||||||
if (ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
|
|
||||||
continue;
|
|
||||||
} else if (errno == EINTR) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Could not send data");
|
log_message(LOG_LEVEL_ERROR, "Could not send data");
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
bw_throttle_session(session, (size_t)bytes_send);
|
bw_throttle_session(session, (size_t)bytes_send);
|
||||||
total_bytes_send += bytes_send;
|
total_bytes_send += bytes_send;
|
||||||
if (session->ssl)
|
|
||||||
wait_events = POLLOUT;
|
|
||||||
}
|
}
|
||||||
log_debug_message(LOG_DEBUG_IO, " Send n Data: %zd", total_bytes_send);
|
log_debug_message(LOG_DEBUG_IO, " Send n Data: %zd", total_bytes_send);
|
||||||
atomic_fetch_add(&io_bytes_written, (unsigned long long)total_bytes_send);
|
atomic_fetch_add(&io_bytes_written, (unsigned long long)total_bytes_send);
|
||||||
@@ -424,14 +570,15 @@ bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_s
|
|||||||
static bool protocol_receive_n_data_until(ProtocolSession* session, void* data, size_t data_size,
|
static bool protocol_receive_n_data_until(ProtocolSession* session, void* data, size_t data_size,
|
||||||
const struct timespec* deadline) {
|
const struct timespec* deadline) {
|
||||||
log_debug_message(LOG_DEBUG_IO, " Receiving n Data: %zu", data_size);
|
log_debug_message(LOG_DEBUG_IO, " Receiving n Data: %zu", data_size);
|
||||||
if (!session)
|
if (!session || !session->ops)
|
||||||
return false;
|
return false;
|
||||||
|
const ProtocolIoOps* ops = session->ops;
|
||||||
int fd = session->read_fd;
|
int fd = session->read_fd;
|
||||||
|
|
||||||
size_t total_bytes_received = 0;
|
size_t total_bytes_received = 0;
|
||||||
short wait_events = POLLIN;
|
short wait_events = POLLIN;
|
||||||
while (total_bytes_received < data_size) {
|
while (total_bytes_received < data_size) {
|
||||||
if (!session->ssl || SSL_pending(session->ssl) == 0) {
|
if (!ops->has_pending(session)) {
|
||||||
struct pollfd pfd = {.fd = fd, .events = wait_events};
|
struct pollfd pfd = {.fd = fd, .events = wait_events};
|
||||||
/* A NULL deadline means "wait indefinitely" (timeout disabled). */
|
/* A NULL deadline means "wait indefinitely" (timeout disabled). */
|
||||||
int poll_result = poll(&pfd, 1, deadline ? deadline_remaining_ms(deadline) : -1);
|
int poll_result = poll(&pfd, 1, deadline ? deadline_remaining_ms(deadline) : -1);
|
||||||
@@ -449,43 +596,19 @@ static bool protocol_receive_n_data_until(ProtocolSession* session, void* data,
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
ssize_t bytes_received;
|
ssize_t bytes_received = ops->recv(session, (char*)data + total_bytes_received,
|
||||||
if (session->ssl) {
|
data_size - total_bytes_received, &wait_events);
|
||||||
/* SSL_read takes an int length; clamp a >INT_MAX request into chunks
|
if (bytes_received == PROTOCOL_IO_RETRY)
|
||||||
* (mirrors the send path) so the size_t downcast can never truncate into
|
continue;
|
||||||
* a negative/partial read. */
|
if (bytes_received == PROTOCOL_IO_CLOSED) {
|
||||||
size_t ssl_chunk = data_size - total_bytes_received > (size_t)INT_MAX
|
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
|
||||||
? (size_t)INT_MAX
|
return false;
|
||||||
: data_size - total_bytes_received;
|
|
||||||
bytes_received = SSL_read(session->ssl, (char*)data + total_bytes_received, (int)ssl_chunk);
|
|
||||||
} else {
|
|
||||||
bytes_received =
|
|
||||||
read(fd, (char*)data + total_bytes_received, data_size - total_bytes_received);
|
|
||||||
}
|
}
|
||||||
if (bytes_received <= 0) {
|
if (bytes_received <= 0) {
|
||||||
if (session->ssl) {
|
|
||||||
int ssl_err = SSL_get_error(session->ssl, (int)bytes_received);
|
|
||||||
if (ssl_err == SSL_ERROR_WANT_WRITE || ssl_err == SSL_ERROR_WANT_READ) {
|
|
||||||
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
/* A signal interrupts the blocking TLS read: retry (mirrors the send
|
|
||||||
path and protocol_read_status_until) so the loop reaches its next
|
|
||||||
abort/deadline checkpoint instead of failing spuriously. */
|
|
||||||
if (ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
|
|
||||||
continue;
|
|
||||||
} else if (errno == EINTR) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (bytes_received == 0)
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
|
|
||||||
else
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Could not receive bytes");
|
log_message(LOG_LEVEL_ERROR, "Could not receive bytes");
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
total_bytes_received += (size_t)bytes_received;
|
total_bytes_received += (size_t)bytes_received;
|
||||||
if (session->ssl)
|
|
||||||
wait_events = POLLIN;
|
|
||||||
}
|
}
|
||||||
log_debug_message(LOG_DEBUG_IO, " Received n Data: %zu", total_bytes_received);
|
log_debug_message(LOG_DEBUG_IO, " Received n Data: %zu", total_bytes_received);
|
||||||
atomic_fetch_add(&io_bytes_read, (unsigned long long)total_bytes_received);
|
atomic_fetch_add(&io_bytes_read, (unsigned long long)total_bytes_received);
|
||||||
@@ -562,6 +685,10 @@ static const char* status_to_string(Status status) {
|
|||||||
return "DELETE_LIMIT";
|
return "DELETE_LIMIT";
|
||||||
case STATUS_DEST_INFO:
|
case STATUS_DEST_INFO:
|
||||||
return "DEST_INFO";
|
return "DEST_INFO";
|
||||||
|
case STATUS_CLIENT_MSG:
|
||||||
|
return "CLIENT_MSG";
|
||||||
|
case STATUS_PARTIAL:
|
||||||
|
return "PARTIAL";
|
||||||
default:
|
default:
|
||||||
return "UNKNOWN";
|
return "UNKNOWN";
|
||||||
}
|
}
|
||||||
@@ -570,10 +697,10 @@ static const char* status_to_string(Status status) {
|
|||||||
/* Reject a raw wire status outside the known enum range before it is handed to
|
/* Reject a raw wire status outside the known enum range before it is handed to
|
||||||
* callers, so an unknown/corrupt frame fails as a protocol error instead of
|
* callers, so an unknown/corrupt frame fails as a protocol error instead of
|
||||||
* being silently interpreted as an unexpected-but-valid verdict. STATUS_OK is
|
* being silently interpreted as an unexpected-but-valid verdict. STATUS_OK is
|
||||||
* the first enumerator and STATUS_STATS the last, so the range check accepts
|
* the first enumerator and STATUS_PARTIAL the last, so the range check accepts
|
||||||
* every status the protocol defines. */
|
* every status the protocol defines. */
|
||||||
static bool status_is_valid(Status status) {
|
static bool status_is_valid(Status status) {
|
||||||
return status >= STATUS_OK && status <= STATUS_STATS;
|
return status >= STATUS_OK && status <= STATUS_PARTIAL;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Shared string send/receive implementation. `redact` selects whether the
|
/* Shared string send/receive implementation. `redact` selects whether the
|
||||||
@@ -663,17 +790,11 @@ bool protocol_send_data(ProtocolSession* session, const Data* data) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long maximum_size) {
|
Data* protocol_receive_data_alloc(ProtocolSession* session, unsigned long long size) {
|
||||||
if (!session)
|
if (!session)
|
||||||
return NULL;
|
return NULL;
|
||||||
unsigned long long size = 0;
|
if (size > MAX_DATA_PAYLOAD_SIZE)
|
||||||
if (!protocol_receive_n_data(session, &size, sizeof(unsigned long long)))
|
|
||||||
return NULL;
|
return NULL;
|
||||||
if (size > MAX_DATA_PAYLOAD_SIZE || size > maximum_size) {
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Data size %llu exceeds maximum %llu", size,
|
|
||||||
(unsigned long long)MAX_DATA_PAYLOAD_SIZE);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
if (size > SIZE_MAX)
|
if (size > SIZE_MAX)
|
||||||
return NULL;
|
return NULL;
|
||||||
size_t allocation_size = size == 0 ? 1 : (size_t)size;
|
size_t allocation_size = size == 0 ? 1 : (size_t)size;
|
||||||
@@ -688,12 +809,6 @@ Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long
|
|||||||
protocol_release_memory_for_session(session, allocation_size);
|
protocol_release_memory_for_session(session, allocation_size);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
if (!protocol_receive_n_data(session, data, (size_t)size)) {
|
|
||||||
free(data);
|
|
||||||
protocol_release_memory_for_session(session, allocation_size);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
log_debug_message(LOG_DEBUG_PROTO, "Received %llu data", size);
|
|
||||||
Data* result = data_create(data, (size_t)size);
|
Data* result = data_create(data, (size_t)size);
|
||||||
if (!result) {
|
if (!result) {
|
||||||
protocol_release_memory_for_session(session, allocation_size);
|
protocol_release_memory_for_session(session, allocation_size);
|
||||||
@@ -704,6 +819,32 @@ Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long
|
|||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
Data* protocol_receive_data_body(ProtocolSession* session, unsigned long long size) {
|
||||||
|
Data* result = protocol_receive_data_alloc(session, size);
|
||||||
|
if (!result)
|
||||||
|
return NULL;
|
||||||
|
if (!protocol_receive_n_data(session, result->data, (size_t)size)) {
|
||||||
|
data_destroy(result);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
log_debug_message(LOG_DEBUG_PROTO, "Received %llu data", size);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long maximum_size) {
|
||||||
|
if (!session)
|
||||||
|
return NULL;
|
||||||
|
unsigned long long size = 0;
|
||||||
|
if (!protocol_receive_n_data(session, &size, sizeof(unsigned long long)))
|
||||||
|
return NULL;
|
||||||
|
if (size > MAX_DATA_PAYLOAD_SIZE || size > maximum_size) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Data size %llu exceeds maximum %llu", size,
|
||||||
|
(unsigned long long)MAX_DATA_PAYLOAD_SIZE);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
return protocol_receive_data_body(session, size);
|
||||||
|
}
|
||||||
|
|
||||||
bool protocol_send_int(ProtocolSession* session, int data) {
|
bool protocol_send_int(ProtocolSession* session, int data) {
|
||||||
if (!protocol_send_n_data(session, &data, sizeof(int)))
|
if (!protocol_send_n_data(session, &data, sizeof(int)))
|
||||||
return false;
|
return false;
|
||||||
@@ -845,11 +986,14 @@ bool protocol_receive_status_timed(ProtocolSession* session, Status* status, int
|
|||||||
* reply across a frame boundary. Returns false on timeout/EOF/error. */
|
* reply across a frame boundary. Returns false on timeout/EOF/error. */
|
||||||
static bool protocol_read_status_until(ProtocolSession* session, Status* status,
|
static bool protocol_read_status_until(ProtocolSession* session, Status* status,
|
||||||
const struct timespec* deadline) {
|
const struct timespec* deadline) {
|
||||||
|
if (!session || !session->ops)
|
||||||
|
return false;
|
||||||
|
const ProtocolIoOps* ops = session->ops;
|
||||||
Status received = STATUS_ERROR;
|
Status received = STATUS_ERROR;
|
||||||
size_t got = 0;
|
size_t got = 0;
|
||||||
short wait_events = POLLIN;
|
short wait_events = POLLIN;
|
||||||
while (got < sizeof(Status)) {
|
while (got < sizeof(Status)) {
|
||||||
if (!session->ssl || SSL_pending(session->ssl) == 0) {
|
if (!ops->has_pending(session)) {
|
||||||
int remaining_ms = deadline ? deadline_remaining_ms(deadline) : -1;
|
int remaining_ms = deadline ? deadline_remaining_ms(deadline) : -1;
|
||||||
if (remaining_ms == 0) {
|
if (remaining_ms == 0) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Receive timeout while reading status");
|
log_message(LOG_LEVEL_ERROR, "Receive timeout while reading status");
|
||||||
@@ -869,21 +1013,11 @@ static bool protocol_read_status_until(ProtocolSession* session, Status* status,
|
|||||||
if (pfd.revents & (POLLERR | POLLNVAL))
|
if (pfd.revents & (POLLERR | POLLNVAL))
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
ssize_t bytes_received;
|
ssize_t bytes_received =
|
||||||
if (session->ssl)
|
ops->recv(session, (char*)&received + got, sizeof(Status) - got, &wait_events);
|
||||||
bytes_received = SSL_read(session->ssl, (char*)&received + got, sizeof(Status) - got);
|
if (bytes_received == PROTOCOL_IO_RETRY)
|
||||||
else
|
continue;
|
||||||
bytes_received = read(session->read_fd, (char*)&received + got, sizeof(Status) - got);
|
|
||||||
if (bytes_received <= 0) {
|
if (bytes_received <= 0) {
|
||||||
if (session->ssl) {
|
|
||||||
int ssl_err = SSL_get_error(session->ssl, (int)bytes_received);
|
|
||||||
if (ssl_err == SSL_ERROR_WANT_READ || ssl_err == SSL_ERROR_WANT_WRITE) {
|
|
||||||
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (bytes_received < 0 && errno == EINTR)
|
|
||||||
continue;
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving status");
|
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving status");
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -912,7 +1046,7 @@ bool protocol_receive_status_keepalive(ProtocolSession* session, Status* status,
|
|||||||
while (true) {
|
while (true) {
|
||||||
if (abort_check && abort_check())
|
if (abort_check && abort_check())
|
||||||
return false;
|
return false;
|
||||||
if (!session->ssl || SSL_pending(session->ssl) == 0) {
|
if (!session->ops || !session->ops->has_pending(session)) {
|
||||||
int remaining_ms = deadline_remaining_ms(&deadline);
|
int remaining_ms = deadline_remaining_ms(&deadline);
|
||||||
if (remaining_ms <= 0) {
|
if (remaining_ms <= 0) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", timeout_sec);
|
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", timeout_sec);
|
||||||
@@ -1015,6 +1149,12 @@ Data* receive_data(int fd) {
|
|||||||
Data* receive_data_limited(int fd, unsigned long long maximum_size) {
|
Data* receive_data_limited(int fd, unsigned long long maximum_size) {
|
||||||
return protocol_receive_data_limited(legacy_session(fd, -1), maximum_size);
|
return protocol_receive_data_limited(legacy_session(fd, -1), maximum_size);
|
||||||
}
|
}
|
||||||
|
Data* receive_data_body(int fd, unsigned long long size) {
|
||||||
|
return protocol_receive_data_body(legacy_session(fd, -1), size);
|
||||||
|
}
|
||||||
|
Data* receive_data_alloc(int fd, unsigned long long size) {
|
||||||
|
return protocol_receive_data_alloc(legacy_session(fd, -1), size);
|
||||||
|
}
|
||||||
bool send_int(int fd, int data) {
|
bool send_int(int fd, int data) {
|
||||||
return protocol_send_int(legacy_session(-1, fd), data);
|
return protocol_send_int(legacy_session(-1, fd), data);
|
||||||
}
|
}
|
||||||
@@ -1049,6 +1189,19 @@ bool send_error_detail(int fd, const char* message) {
|
|||||||
return send_status(fd, STATUS_ERROR_DETAIL) && send_str(fd, message);
|
return send_status(fd, STATUS_ERROR_DETAIL) && send_str(fd, message);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool send_client_message(int fd, const char* message) {
|
||||||
|
if (!message)
|
||||||
|
message = "";
|
||||||
|
char bounded[MAX_CLIENT_MSG_BYTES + 1];
|
||||||
|
size_t len = strlen(message);
|
||||||
|
if (len > MAX_CLIENT_MSG_BYTES) {
|
||||||
|
memcpy(bounded, message, MAX_CLIENT_MSG_BYTES);
|
||||||
|
bounded[MAX_CLIENT_MSG_BYTES] = '\0';
|
||||||
|
message = bounded;
|
||||||
|
}
|
||||||
|
return send_status(fd, STATUS_CLIENT_MSG) && send_str(fd, message);
|
||||||
|
}
|
||||||
|
|
||||||
const char* protocol_last_error(void) {
|
const char* protocol_last_error(void) {
|
||||||
return io_error_detail;
|
return io_error_detail;
|
||||||
}
|
}
|
||||||
|
|||||||
+121
-17
@@ -15,6 +15,12 @@
|
|||||||
* this for a rejection and the detail frame stays a small, fixed bound. */
|
* this for a rejection and the detail frame stays a small, fixed bound. */
|
||||||
#define MAX_ERROR_DETAIL_BYTES 4096
|
#define MAX_ERROR_DETAIL_BYTES 4096
|
||||||
|
|
||||||
|
/* Hard cap on a client diagnostic forwarded over the STATUS_CLIENT_MSG channel
|
||||||
|
* (protocol 2.30.0, rsync's --stderr=client). The body is reused from the
|
||||||
|
* bounded-string wire helper and sliced to this many bytes before it is sent,
|
||||||
|
* so a peer can never be made to retain more than this per message. */
|
||||||
|
#define MAX_CLIENT_MSG_BYTES 4096
|
||||||
|
|
||||||
/* Maximum uncompressed file payload accepted by the receiver's whole-file
|
/* Maximum uncompressed file payload accepted by the receiver's whole-file
|
||||||
* paths. A single whole file is charged against the per-connection memory
|
* paths. A single whole file is charged against the per-connection memory
|
||||||
* reservation (MAX_CONNECTION_MEMORY) and against the server allocation
|
* reservation (MAX_CONNECTION_MEMORY) and against the server allocation
|
||||||
@@ -26,6 +32,16 @@
|
|||||||
/* Maximum allowed data payload size for receive_data (whole-file bound) */
|
/* Maximum allowed data payload size for receive_data (whole-file bound) */
|
||||||
#define MAX_DATA_PAYLOAD_SIZE MAX_RECEIVE_WHOLE_FILE_SIZE
|
#define MAX_DATA_PAYLOAD_SIZE MAX_RECEIVE_WHOLE_FILE_SIZE
|
||||||
|
|
||||||
|
/* Runtime whole-file receive bound. It defaults to MAX_RECEIVE_WHOLE_FILE_SIZE
|
||||||
|
* and exists so the test suite can lower the ceiling (via the
|
||||||
|
* FASTSYNC_MAX_WHOLE_FILE_SIZE environment variable, a byte count) and exercise
|
||||||
|
* the streaming path with a small, fast transfer. A payload at or below the
|
||||||
|
* bound keeps the historical whole-buffer path; a larger one is streamed
|
||||||
|
* through a bounded buffer. The value is resolved once per process and never
|
||||||
|
* exceeds the compile-time ceiling, so a malicious environment cannot raise it
|
||||||
|
* beyond the protocol limit. */
|
||||||
|
unsigned long long protocol_whole_file_receive_limit(void);
|
||||||
|
|
||||||
/* Maximum chunk size (64 MB) — prevents unbounded allocation from the wire */
|
/* Maximum chunk size (64 MB) — prevents unbounded allocation from the wire */
|
||||||
#define MAX_CHUNK_SIZE (64ULL * 1024 * 1024)
|
#define MAX_CHUNK_SIZE (64ULL * 1024 * 1024)
|
||||||
/* Files larger than this are not kept fully in memory while loading: the
|
/* Files larger than this are not kept fully in memory while loading: the
|
||||||
@@ -50,16 +66,48 @@
|
|||||||
|
|
||||||
typedef struct ssl_st SSL;
|
typedef struct ssl_st SSL;
|
||||||
|
|
||||||
|
typedef struct ProtocolSession ProtocolSession;
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Transport vtable: the per-session set of I/O primitives the three protocol
|
||||||
|
* loops (send, receive, status-read) dispatch through. The ops are selected
|
||||||
|
* once, when the session is initialized or its SSL is installed, so the loops
|
||||||
|
* never branch on the transport at runtime. A plaintext session uses the
|
||||||
|
* read()/write() ops; a TLS session uses the SSL_read()/SSL_write() ops.
|
||||||
|
*
|
||||||
|
* `send`/`recv` attempt exactly one transfer and return:
|
||||||
|
* > 0 bytes transferred,
|
||||||
|
* PROTOCOL_IO_RETRY no progress; poll on *wait_events and retry,
|
||||||
|
* PROTOCOL_IO_CLOSED peer closed the stream,
|
||||||
|
* PROTOCOL_IO_ERROR fatal transport error.
|
||||||
|
* `has_pending` reports bytes already buffered by the transport (a TLS record
|
||||||
|
* residue); the receive loops skip the poll() gate when it is true.
|
||||||
|
*/
|
||||||
|
typedef struct ProtocolIoOps {
|
||||||
|
ssize_t (*send)(ProtocolSession* session, const void* data, size_t size, short* wait_events);
|
||||||
|
ssize_t (*recv)(ProtocolSession* session, void* data, size_t size, short* wait_events);
|
||||||
|
bool (*has_pending)(const ProtocolSession* session);
|
||||||
|
} ProtocolIoOps;
|
||||||
|
|
||||||
|
/* Negative sentinels returned by ProtocolIoOps.send/recv (see above). */
|
||||||
|
enum {
|
||||||
|
PROTOCOL_IO_RETRY = -1,
|
||||||
|
PROTOCOL_IO_CLOSED = -2,
|
||||||
|
PROTOCOL_IO_ERROR = -3,
|
||||||
|
};
|
||||||
|
|
||||||
/*
|
/*
|
||||||
* Explicit owner of protocol I/O. A session does not own the descriptors or
|
* Explicit owner of protocol I/O. A session does not own the descriptors or
|
||||||
* SSL object; it only describes the transport used by a transfer. This makes
|
* SSL object; it only describes the transport used by a transfer. This makes
|
||||||
* it safe to pass the transport to a worker without relying on inherited
|
* it safe to pass the transport to a worker without relying on inherited
|
||||||
* thread-local state.
|
* thread-local state.
|
||||||
*/
|
*/
|
||||||
typedef struct ProtocolSession {
|
struct ProtocolSession {
|
||||||
int read_fd;
|
int read_fd;
|
||||||
int write_fd;
|
int write_fd;
|
||||||
SSL* ssl;
|
SSL* ssl;
|
||||||
|
/* Transport dispatch selected by protocol_session_init()/set_ssl(). */
|
||||||
|
const ProtocolIoOps* ops;
|
||||||
unsigned long long bwlimit;
|
unsigned long long bwlimit;
|
||||||
long long bw_tokens;
|
long long bw_tokens;
|
||||||
long long bw_last_refill_sec;
|
long long bw_last_refill_sec;
|
||||||
@@ -75,7 +123,7 @@ typedef struct ProtocolSession {
|
|||||||
* SO_RCVTIMEO/SO_SNDTIMEO. The server does not propagate a client 0 here: it
|
* SO_RCVTIMEO/SO_SNDTIMEO. The server does not propagate a client 0 here: it
|
||||||
* installs protocol_server_io_timeout_sec() so its sessions keep a floor. */
|
* installs protocol_server_io_timeout_sec() so its sessions keep a floor. */
|
||||||
int io_timeout_sec;
|
int io_timeout_sec;
|
||||||
} ProtocolSession;
|
};
|
||||||
|
|
||||||
typedef int Status;
|
typedef int Status;
|
||||||
enum NET_STATUS {
|
enum NET_STATUS {
|
||||||
@@ -91,8 +139,13 @@ enum NET_STATUS {
|
|||||||
STATUS_KEEPALIVE,
|
STATUS_KEEPALIVE,
|
||||||
STATUS_ABORT,
|
STATUS_ABORT,
|
||||||
STATUS_CHECK_BATCH,
|
STATUS_CHECK_BATCH,
|
||||||
/* An explicit directory entry (--dirs): the sender transmits only the path;
|
/* An explicit directory entry (--dirs / an empty source directory): the sender
|
||||||
* the receiver creates the directory below the receive root. */
|
* transmits the path and, when metadata/xattrs are negotiated, their blocks;
|
||||||
|
* the receiver creates the directory below the receive root. Protocol 2.30.0
|
||||||
|
* inserts an int32 probe flag right after the status when report_dest_info is
|
||||||
|
* negotiated: probe=1 is a report-only frame (path only; the receiver answers
|
||||||
|
* STATUS_DEST_INFO and creates nothing), probe=0 is a real create that is
|
||||||
|
* answered with the directory's pre-transfer state before it is created. */
|
||||||
STATUS_MKDIR,
|
STATUS_MKDIR,
|
||||||
/* --append / --append-verify tail resume. STATUS_APPEND is sent by the
|
/* --append / --append-verify tail resume. STATUS_APPEND is sent by the
|
||||||
* receiver after a per-file STATUS_CHECK when the existing destination file
|
* receiver after a per-file STATUS_CHECK when the existing destination file
|
||||||
@@ -175,16 +228,22 @@ enum NET_STATUS {
|
|||||||
* limit stopped deletions"). Appended after STATUS_DRY_RUN_TRANSFER so no
|
* limit stopped deletions"). Appended after STATUS_DRY_RUN_TRANSFER so no
|
||||||
* existing status is renumbered. */
|
* existing status is renumbered. */
|
||||||
STATUS_DELETE_LIMIT,
|
STATUS_DELETE_LIMIT,
|
||||||
/* Destination-state report for output parity (protocol 2.23.0). When the
|
/* Destination-state report for output parity (protocol 2.23.0; extended to
|
||||||
* wire config carries report_dest_info=true, the receiver answers every
|
* directories/symlinks in 2.30.0). When the wire config carries
|
||||||
* per-file STATUS_CHECK request with STATUS_DEST_INFO FIRST, followed by a
|
* report_dest_info=true, the receiver answers every per-file STATUS_CHECK
|
||||||
* fixed record describing the pre-transfer destination entry
|
* request with STATUS_DEST_INFO FIRST, followed by a fixed record describing
|
||||||
* (int32 has_old; uint64 size; int64 mtime; int64 mtime_nsec; uint32 mode;
|
* the pre-transfer destination entry (int32 has_old; int32 target_matches;
|
||||||
* int32 uid; int32 gid). The ordinary STATUS_OK/STATUS_NEXT/... verdict
|
* uint64 size; int64 mtime; int64 mtime_nsec; uint32 mode; int32 uid;
|
||||||
* follows, so the sender can render rsync-accurate -i/--out-format columns
|
* int32 gid). The ordinary STATUS_OK/STATUS_NEXT/... verdict follows, so the
|
||||||
* (new vs modified, and which of size/time/perms/owner/group differ) without
|
* sender can render rsync-accurate -i/--out-format columns (new vs modified,
|
||||||
* changing the transfer decision itself. Appended after
|
* and which of size/time/perms/owner/group differ) without changing the
|
||||||
* STATUS_DELETE_LIMIT so no existing status is renumbered. */
|
* transfer decision itself. Protocol 2.30.0 also uses this record for
|
||||||
|
* STATUS_MKDIR and STATUS_SYMLINK: the sender consumes it into the entry's
|
||||||
|
* dest_state before emitting its change line, and target_matches reports
|
||||||
|
* whether an existing symlink's on-disk target already equals the incoming
|
||||||
|
* one (so the sender can render `cLc........` vs `.L..t......` and suppress
|
||||||
|
* an unchanged symlink). Appended after STATUS_DELETE_LIMIT so no existing
|
||||||
|
* status is renumbered. */
|
||||||
STATUS_DEST_INFO,
|
STATUS_DEST_INFO,
|
||||||
/* Per-directory delete plan (protocol 2.24.0). The sender of a
|
/* Per-directory delete plan (protocol 2.24.0). The sender of a
|
||||||
* --delete-during/--delete-delay transfer streams one frame per source
|
* --delete-during/--delete-delay transfer streams one frame per source
|
||||||
@@ -206,9 +265,29 @@ enum NET_STATUS {
|
|||||||
* config carries report_stats=true, the receiver sends this status once,
|
* config carries report_stats=true, the receiver sends this status once,
|
||||||
* immediately before its terminal success status, followed by a fixed stats
|
* immediately before its terminal success status, followed by a fixed stats
|
||||||
* record (see format_stats_send/receive in format.h) and, when the run is a
|
* record (see format_stats_send/receive in format.h) and, when the run is a
|
||||||
* --dry-run with --delete, the would-delete path list. Appended after
|
* --dry-run with --delete, the would-delete path list. Protocol 2.30.0
|
||||||
* STATUS_DELETE_PLAN so no existing status is renumbered. */
|
* appends the four deleted_reg/dir/link/special counters to that record, so
|
||||||
STATUS_STATS
|
* --stats can render rsync's `Number of deleted files` per-type breakdown.
|
||||||
|
* Appended after STATUS_DELETE_PLAN so no existing status is renumbered. */
|
||||||
|
STATUS_STATS,
|
||||||
|
/* Client diagnostic channel (protocol 2.30.0, rsync's --stderr=client /
|
||||||
|
* --no-msgs2stderr). When the client's --stderr mode is `client`, the
|
||||||
|
* client forwards its own diagnostics over this client->server frame
|
||||||
|
* (STATUS_CLIENT_MSG followed by a bounded length-prefixed string, capped at
|
||||||
|
* MAX_CLIENT_MSG_BYTES) instead of writing them to its local stderr. The
|
||||||
|
* receiver reads the string and writes it to the server's stderr (respecting
|
||||||
|
* the server log destination). Appended after STATUS_STATS so no existing
|
||||||
|
* status is renumbered. */
|
||||||
|
STATUS_CLIENT_MSG,
|
||||||
|
/* Receiver-side partial transfer (protocol 2.30.0). Sent by the receiver as
|
||||||
|
* the terminal status INSTEAD of STATUS_OK when one or more entries failed
|
||||||
|
* per-entry without aborting the stream (currently a --devices mknod
|
||||||
|
* EPERM/EACCES). The transfer otherwise succeeded and every successfully
|
||||||
|
* stored file was acknowledged, so the sender may still remove
|
||||||
|
* --remove-source-files sources; the sender maps this to rsync's exit code
|
||||||
|
* 23 ("partial transfer due to error"), distinct from a fatal STATUS_ERROR.
|
||||||
|
* Appended after STATUS_CLIENT_MSG so no existing status is renumbered. */
|
||||||
|
STATUS_PARTIAL
|
||||||
};
|
};
|
||||||
|
|
||||||
void io_set_fds(int read_fd, int write_fd);
|
void io_set_fds(int read_fd, int write_fd);
|
||||||
@@ -216,6 +295,17 @@ void io_set_bwlimit(unsigned long long bytes_per_sec);
|
|||||||
unsigned long long io_get_bwlimit(void);
|
unsigned long long io_get_bwlimit(void);
|
||||||
void io_set_ssl(SSL* ssl);
|
void io_set_ssl(SSL* ssl);
|
||||||
SSL* io_get_ssl(void);
|
SSL* io_get_ssl(void);
|
||||||
|
/* SSL object of the transport in effect on this thread: the currently bound
|
||||||
|
* session's SSL when a TLS session is bound, otherwise the legacy thread-local
|
||||||
|
* io_ssl. NULL for a plaintext transport. Unlike io_get_ssl(), this resolves
|
||||||
|
* worker threads that bound a TLS session via protocol_session_set_ssl()/
|
||||||
|
* protocol_session_bind() but never called io_set_ssl() themselves (C11
|
||||||
|
* _Thread_local state is not inherited by a new thread). A bound session only
|
||||||
|
* wins when its selected dispatch is TLS; a bound plaintext session (ssl ==
|
||||||
|
* NULL) falls back to io_ssl so it can never mask a live encrypted transport.
|
||||||
|
* Callers that must choose a TLS-only code path (e.g. file_send.c's sendfile
|
||||||
|
* fallback) must use this instead of io_get_ssl(). */
|
||||||
|
SSL* protocol_current_ssl(void);
|
||||||
|
|
||||||
/* Process-wide wire byte counters. protocol_send_n_data/protocol_receive_n_data
|
/* Process-wide wire byte counters. protocol_send_n_data/protocol_receive_n_data
|
||||||
* update them; the zero-copy sendfile path reports through
|
* update them; the zero-copy sendfile path reports through
|
||||||
@@ -290,6 +380,15 @@ char* receive_str_redacted(int file_descriptor);
|
|||||||
bool send_data(int file_descriptor, const Data* data);
|
bool send_data(int file_descriptor, const Data* data);
|
||||||
Data* receive_data(int file_descriptor);
|
Data* receive_data(int file_descriptor);
|
||||||
Data* receive_data_limited(int file_descriptor, unsigned long long maximum_size);
|
Data* receive_data_limited(int file_descriptor, unsigned long long maximum_size);
|
||||||
|
/* Read exactly `size` bytes as a charged Data body. The length-prefixed
|
||||||
|
* receive_data_limited() reads the header itself; this variant is for callers
|
||||||
|
* that must inspect the declared size (and possibly stream the body instead)
|
||||||
|
* before allocating. `size` must already be within MAX_DATA_PAYLOAD_SIZE. */
|
||||||
|
Data* receive_data_body(int file_descriptor, unsigned long long size);
|
||||||
|
/* Allocate (and charge) a `size`-byte Data body without reading it; the caller
|
||||||
|
* fills `result->data` itself. Used when a frame's leading bytes must be
|
||||||
|
* inspected before the rest of the body is read. */
|
||||||
|
Data* receive_data_alloc(int file_descriptor, unsigned long long size);
|
||||||
bool send_int(int file_descriptor, int data);
|
bool send_int(int file_descriptor, int data);
|
||||||
bool receive_int(int file_descriptor, int* data);
|
bool receive_int(int file_descriptor, int* data);
|
||||||
bool send_status(int file_descriptor, Status status);
|
bool send_status(int file_descriptor, Status status);
|
||||||
@@ -298,6 +397,11 @@ bool receive_status(int file_descriptor, Status* status);
|
|||||||
* length-prefixed string. Over-long messages are sliced and NULL is treated
|
* length-prefixed string. Over-long messages are sliced and NULL is treated
|
||||||
* as "". Returns false if the status or the string could not be sent. */
|
* as "". Returns false if the status or the string could not be sent. */
|
||||||
bool send_error_detail(int file_descriptor, const char* message);
|
bool send_error_detail(int file_descriptor, const char* message);
|
||||||
|
/* Send STATUS_CLIENT_MSG followed by a bounded (<= MAX_CLIENT_MSG_BYTES)
|
||||||
|
* length-prefixed string carrying a client diagnostic. Over-long messages are
|
||||||
|
* sliced and NULL is treated as "". Returns false if the status or the string
|
||||||
|
* could not be sent. */
|
||||||
|
bool send_client_message(int file_descriptor, const char* message);
|
||||||
/* Human-readable reason captured from the most recent STATUS_ERROR_DETAIL
|
/* Human-readable reason captured from the most recent STATUS_ERROR_DETAIL
|
||||||
* received on this thread, or "" when the last status was a bare STATUS_ERROR
|
* received on this thread, or "" when the last status was a bare STATUS_ERROR
|
||||||
* (or no detail was seen). Thread-local, and valid until the next non-keepalive
|
* (or no detail was seen). Thread-local, and valid until the next non-keepalive
|
||||||
|
|||||||
@@ -134,7 +134,7 @@ Server* server_create_ex(int port, const ServerBindOptions* bind_opts) {
|
|||||||
|
|
||||||
server->file_descriptor = file_descriptor;
|
server->file_descriptor = file_descriptor;
|
||||||
server->ssl_ctx = NULL;
|
server->ssl_ctx = NULL;
|
||||||
server->max_connections = 100;
|
server->max_connections = SERVER_DEFAULT_MAX_CONNECTIONS;
|
||||||
server->active_connections = 0;
|
server->active_connections = 0;
|
||||||
server->limit_registry = NULL;
|
server->limit_registry = NULL;
|
||||||
|
|
||||||
|
|||||||
@@ -11,6 +11,10 @@
|
|||||||
* stored here so the transport layer does not depend on daemon config. */
|
* stored here so the transport layer does not depend on daemon config. */
|
||||||
struct DaemonLimitRegistry;
|
struct DaemonLimitRegistry;
|
||||||
|
|
||||||
|
/* Connection cap applied by server_create_ex() until the daemon's configured
|
||||||
|
* `max connections` overrides it via server_set_max_connections(). */
|
||||||
|
#define SERVER_DEFAULT_MAX_CONNECTIONS 100
|
||||||
|
|
||||||
typedef struct Server {
|
typedef struct Server {
|
||||||
struct sockaddr_storage address;
|
struct sockaddr_storage address;
|
||||||
unsigned int address_length;
|
unsigned int address_length;
|
||||||
|
|||||||
+161
-40
@@ -7,6 +7,7 @@
|
|||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include "file_types.h"
|
#include "file_types.h"
|
||||||
#include <errno.h>
|
#include <errno.h>
|
||||||
|
#include <limits.h>
|
||||||
#include <stdint.h>
|
#include <stdint.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
@@ -133,16 +134,23 @@ static bool xattr_name_is_posix_acl(const char* name) {
|
|||||||
|
|
||||||
/* ---- SENDER: capture ---- */
|
/* ---- SENDER: capture ---- */
|
||||||
|
|
||||||
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
|
/* The two syscall families differ only in whether the FINAL component is
|
||||||
|
* followed (`listxattr`/`getxattr` follow; `llistxattr`/`lgetxattr` do not), so
|
||||||
|
* one common implementation backs both public entry points. */
|
||||||
|
typedef ssize_t (*XattrListFn)(const char* path, char* list, size_t size);
|
||||||
|
typedef ssize_t (*XattrGetFn)(const char* path, const char* name, void* value, size_t size);
|
||||||
|
|
||||||
|
static FileXattrList* xattr_capture_common(const char* path, bool preserve_acls,
|
||||||
|
XattrListFn list_fn, XattrGetFn get_fn) {
|
||||||
if (!path)
|
if (!path)
|
||||||
return NULL;
|
return NULL;
|
||||||
ssize_t list_size = listxattr(path, NULL, 0);
|
ssize_t list_size = list_fn(path, NULL, 0);
|
||||||
if (list_size <= 0)
|
if (list_size <= 0)
|
||||||
return NULL; /* no xattrs, ENOTSUP, or error: nothing appliable */
|
return NULL; /* no xattrs, ENOTSUP, or error: nothing appliable */
|
||||||
char* names = malloc((size_t)list_size);
|
char* names = malloc((size_t)list_size);
|
||||||
if (!names)
|
if (!names)
|
||||||
return NULL;
|
return NULL;
|
||||||
ssize_t got = listxattr(path, names, (size_t)list_size);
|
ssize_t got = list_fn(path, names, (size_t)list_size);
|
||||||
if (got < 0) {
|
if (got < 0) {
|
||||||
free(names);
|
free(names);
|
||||||
return NULL;
|
return NULL;
|
||||||
@@ -165,7 +173,7 @@ FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
|
|||||||
negotiated. Without it a plain -X capture never carries an ACL. */
|
negotiated. Without it a plain -X capture never carries an ACL. */
|
||||||
if (!xattr_name_appliable(name, preserve_acls))
|
if (!xattr_name_appliable(name, preserve_acls))
|
||||||
continue;
|
continue;
|
||||||
ssize_t value_size = getxattr(path, name, NULL, 0);
|
ssize_t value_size = get_fn(path, name, NULL, 0);
|
||||||
if (value_size < 0)
|
if (value_size < 0)
|
||||||
continue;
|
continue;
|
||||||
if (value_size > XATTR_VALUE_MAX)
|
if (value_size > XATTR_VALUE_MAX)
|
||||||
@@ -178,7 +186,7 @@ FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
|
|||||||
free(names);
|
free(names);
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
ssize_t read_len = getxattr(path, name, buffer, (size_t)value_size);
|
ssize_t read_len = get_fn(path, name, buffer, (size_t)value_size);
|
||||||
if (read_len < 0 || read_len != value_size) {
|
if (read_len < 0 || read_len != value_size) {
|
||||||
free(buffer);
|
free(buffer);
|
||||||
continue;
|
continue;
|
||||||
@@ -200,6 +208,14 @@ FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
|
|||||||
return list;
|
return list;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
|
||||||
|
return xattr_capture_common(path, preserve_acls, listxattr, getxattr);
|
||||||
|
}
|
||||||
|
|
||||||
|
FileXattrList* xattr_capture_path_nofollow(const char* path, bool preserve_acls) {
|
||||||
|
return xattr_capture_common(path, preserve_acls, llistxattr, lgetxattr);
|
||||||
|
}
|
||||||
|
|
||||||
/* ---- WIRE ---- */
|
/* ---- WIRE ---- */
|
||||||
|
|
||||||
bool xattr_send(int fd, const FileXattrList* list) {
|
bool xattr_send(int fd, const FileXattrList* list) {
|
||||||
@@ -363,29 +379,140 @@ bool xattr_apply_fd(int fd, const FileXattrList* list) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ---- --fake-super: park ownership/mode/mtime in a reserved xattr ---- */
|
/* Symlink counterpart of xattr_apply_fd(): target the link ITSELF, never its
|
||||||
|
* referent. fsetxattr cannot be used (no *at xattr syscall exists, and the
|
||||||
|
* kernel rejects xattr syscalls on an O_PATH descriptor), so the already-open,
|
||||||
|
* confinement-checked parent directory is addressed through /proc/self/fd and
|
||||||
|
* the final component is applied with lsetxattr, which does not follow it.
|
||||||
|
*
|
||||||
|
* The list is trusted to come from xattr_receive() (already whitelisted), but
|
||||||
|
* every name is re-validated here so this path-based primitive is confined on
|
||||||
|
* its own -- this is the only apply primitive that addresses a path, and the
|
||||||
|
* header promises a whitelisted apply. The apply is best-effort: if /proc is
|
||||||
|
* not mounted (the anchor cannot be formed) or the kernel refuses the set, the
|
||||||
|
* failure is skipped and never fails the transfer. See xattr.h for the bounded
|
||||||
|
* residual TOCTOU between link creation and lsetxattr. */
|
||||||
|
bool xattr_apply_path_nofollow(int parent_fd, const char* leaf, const FileXattrList* list,
|
||||||
|
bool preserve_acls) {
|
||||||
|
if (parent_fd < 0 || !leaf || leaf[0] == '\0' || strchr(leaf, '/') != NULL || !list)
|
||||||
|
return false;
|
||||||
|
if (list->count == 0)
|
||||||
|
return true;
|
||||||
|
char prefix[64];
|
||||||
|
int prefix_len = snprintf(prefix, sizeof(prefix), "/proc/self/fd/%d/", parent_fd);
|
||||||
|
if (prefix_len < 0 || (size_t)prefix_len >= sizeof(prefix))
|
||||||
|
return false;
|
||||||
|
size_t leaf_len = strlen(leaf);
|
||||||
|
char* path = malloc((size_t)prefix_len + leaf_len + 1);
|
||||||
|
if (!path)
|
||||||
|
return false;
|
||||||
|
memcpy(path, prefix, (size_t)prefix_len);
|
||||||
|
memcpy(path + prefix_len, leaf, leaf_len + 1);
|
||||||
|
bool warned = false;
|
||||||
|
int first_errno = 0;
|
||||||
|
for (int i = 0; i < list->count; i++) {
|
||||||
|
const FileXattr* xa = &list->items[i];
|
||||||
|
/* Defense in depth: re-validate against the receiver's full whitelist, so a
|
||||||
|
hand-crafted list can never apply a privileged namespace or the reserved
|
||||||
|
--fake-super key through this path-based primitive. */
|
||||||
|
if (!xattr_name_appliable(xa->name, preserve_acls))
|
||||||
|
continue;
|
||||||
|
if (lsetxattr(path, xa->name, xa->value, xa->value_len, 0) != 0) {
|
||||||
|
if (!warned) {
|
||||||
|
warned = true;
|
||||||
|
first_errno = errno;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (warned)
|
||||||
|
log_message(LOG_LEVEL_WARNING,
|
||||||
|
"could not set one or more xattrs on the destination symlink: %s",
|
||||||
|
strerror(first_errno));
|
||||||
|
free(path);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, int64_t mtime_sec,
|
/* ---- --fake-super: park ownership/mode/rdev in a reserved xattr ---- */
|
||||||
int64_t mtime_nsec) {
|
|
||||||
|
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, uint32_t rdev_major,
|
||||||
|
uint32_t rdev_minor) {
|
||||||
if (fd < 0)
|
if (fd < 0)
|
||||||
return;
|
return;
|
||||||
char record[128];
|
/* rsync 3.4.1's exact grammar: "<octal full st_mode> <rdev_major>,<rdev_minor>
|
||||||
int len =
|
* <uid>:<gid>". The octal mode carries the S_IFMT bits (e.g. 0104711 for a
|
||||||
snprintf(record, sizeof(record), "%lu:%lu:%03o:%lld:%ld", (unsigned long)uid,
|
* setuid regular file, 020644 for a char device); the rdev pair is 0,0 for a
|
||||||
(unsigned long)gid, (unsigned)mode & 0777U, (long long)mtime_sec, (long)mtime_nsec);
|
* non-device. No mtime field: rsync leaves the file's own timestamp in
|
||||||
|
* charge of mtime. This value is what rsync reads back to restore a
|
||||||
|
* fake-super tree, so the field order and separators must not change. */
|
||||||
|
char record[96];
|
||||||
|
int len = snprintf(record, sizeof(record), "%o %u,%u %u:%u", (unsigned)mode, (unsigned)rdev_major,
|
||||||
|
(unsigned)rdev_minor, (unsigned)uid, (unsigned)gid);
|
||||||
if (len <= 0 || (size_t)len >= sizeof(record))
|
if (len <= 0 || (size_t)len >= sizeof(record))
|
||||||
return;
|
return;
|
||||||
if (fsetxattr(fd, FAKESUPER_XATTR, record, (size_t)len, 0) != 0) {
|
if (fsetxattr(fd, FAKESUPER_XATTR, record, (size_t)len, 0) != 0) {
|
||||||
log_message(LOG_LEVEL_WARNING, "--fake-super: could not store %s on destination file: %s",
|
log_message(LOG_LEVEL_WARNING, "--fake-super: could not store %s on destination entry: %s",
|
||||||
FAKESUPER_XATTR, strerror(errno));
|
FAKESUPER_XATTR, strerror(errno));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/* --fake-super replay: read the freshly-stored record and re-apply mode/mtime
|
/* Parse rsync's `user.rsync.%stat` grammar strictly:
|
||||||
* fd-relative. The recorded uid/gid are retained for a later privileged
|
* "<octal st_mode> <rdev_major>,<rdev_minor> <uid>:<gid>"
|
||||||
* restore but are NEVER chowned here: --fake-super only RECORDS ownership, it
|
* Every field is parsed with strtoul() so an out-of-range value is a clean
|
||||||
* must not real-chown the recorded (resolved) owner. Mode/mtime still apply so
|
* rejection rather than the undefined behavior sscanf("%u") exhibited, each
|
||||||
* unprivileged --fake-super keeps working. */
|
* field is range-checked against the same bounds the wire validator uses, and
|
||||||
|
* the whole record must be consumed (only trailing whitespace is tolerated) so
|
||||||
|
* trailing garbage is refused. Returns false on any malformed input. */
|
||||||
|
static bool fake_super_parse_stat(const char* record, unsigned* mode_out, unsigned* rdev_major_out,
|
||||||
|
unsigned* rdev_minor_out, unsigned* uid_out, unsigned* gid_out) {
|
||||||
|
if (!record)
|
||||||
|
return false;
|
||||||
|
char* end = NULL;
|
||||||
|
const char* p = record;
|
||||||
|
errno = 0;
|
||||||
|
unsigned long mode = strtoul(p, &end, 8);
|
||||||
|
if (errno != 0 || end == p || mode > (unsigned long)UINT_MAX || *end != ' ')
|
||||||
|
return false;
|
||||||
|
p = end + 1;
|
||||||
|
errno = 0;
|
||||||
|
unsigned long rdev_major = strtoul(p, &end, 10);
|
||||||
|
if (errno != 0 || end == p || rdev_major > 0xffffUL || *end != ',')
|
||||||
|
return false;
|
||||||
|
p = end + 1;
|
||||||
|
errno = 0;
|
||||||
|
unsigned long rdev_minor = strtoul(p, &end, 10);
|
||||||
|
if (errno != 0 || end == p || rdev_minor > 0x00ffffffUL || *end != ' ')
|
||||||
|
return false;
|
||||||
|
p = end + 1;
|
||||||
|
errno = 0;
|
||||||
|
unsigned long uid = strtoul(p, &end, 10);
|
||||||
|
if (errno != 0 || end == p || uid > (unsigned long)UINT_MAX || *end != ':')
|
||||||
|
return false;
|
||||||
|
p = end + 1;
|
||||||
|
errno = 0;
|
||||||
|
unsigned long gid = strtoul(p, &end, 10);
|
||||||
|
if (errno != 0 || end == p || gid > (unsigned long)UINT_MAX)
|
||||||
|
return false;
|
||||||
|
p = end;
|
||||||
|
while (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r')
|
||||||
|
p++;
|
||||||
|
if (*p != '\0')
|
||||||
|
return false;
|
||||||
|
*mode_out = (unsigned)mode;
|
||||||
|
*rdev_major_out = (unsigned)rdev_major;
|
||||||
|
*rdev_minor_out = (unsigned)rdev_minor;
|
||||||
|
*uid_out = (unsigned)uid;
|
||||||
|
*gid_out = (unsigned)gid;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* --fake-super replay: read the freshly-stored record and re-apply its
|
||||||
|
* permission bits fd-relative. The recorded uid/gid are retained for a later
|
||||||
|
* privileged restore but are NEVER chowned here: --fake-super only RECORDS
|
||||||
|
* ownership, it must not real-chown the recorded (resolved) owner. The
|
||||||
|
* recorded rdev is likewise parsed for grammar compatibility but is not acted
|
||||||
|
* on (device recreation is a separate, privilege-gated path). mtime is not in
|
||||||
|
* the record: the normal metadata path applies it (policy.times), exactly as
|
||||||
|
* rsync relies on the file's own timestamp. */
|
||||||
bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
|
bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
|
||||||
if (fd < 0)
|
if (fd < 0)
|
||||||
return false;
|
return false;
|
||||||
@@ -394,24 +521,25 @@ bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
|
|||||||
if (len < 0)
|
if (len < 0)
|
||||||
return false; /* absent or filesystem without xattrs: silent no-op */
|
return false; /* absent or filesystem without xattrs: silent no-op */
|
||||||
record[len] = '\0';
|
record[len] = '\0';
|
||||||
unsigned long ul_uid, ul_gid, ul_mode;
|
unsigned ul_mode, rdev_major, rdev_minor, ul_uid, ul_gid;
|
||||||
long long mtime_sec;
|
if (!fake_super_parse_stat(record, &ul_mode, &rdev_major, &rdev_minor, &ul_uid, &ul_gid))
|
||||||
long mtime_nsec;
|
|
||||||
if (sscanf(record, "%lu:%lu:%lo:%lld:%ld", &ul_uid, &ul_gid, &ul_mode, &mtime_sec, &mtime_nsec) !=
|
|
||||||
5)
|
|
||||||
return false; /* malformed record: skip, never fatal */
|
return false; /* malformed record: skip, never fatal */
|
||||||
|
|
||||||
/* --fake-super NEVER performs a real chown: that would defeat the whole
|
/* --fake-super NEVER performs a real chown: that would defeat the whole point
|
||||||
point of the flag (record privileged ownership on an unprivileged receiver
|
of the flag (record privileged ownership on an unprivileged receiver for a
|
||||||
for a later privileged restore). The uid/gid parsed above are retained in
|
later privileged restore). The uid/gid parsed above are retained in the
|
||||||
the record for that later restore, but no ownership change happens here. */
|
record for that later restore, but no ownership change happens here. The
|
||||||
|
rdev is retained for the same reason. */
|
||||||
|
(void)rdev_major;
|
||||||
|
(void)rdev_minor;
|
||||||
(void)ul_uid;
|
(void)ul_uid;
|
||||||
(void)ul_gid;
|
(void)ul_gid;
|
||||||
/* Mode is applied only when the per-attribute policy asks for it, through the
|
/* Mode is applied only when the per-attribute policy asks for it, through the
|
||||||
SAME shared helper the normal metadata path uses (metadata_mode_for_policy):
|
SAME shared helper the normal metadata path uses (metadata_mode_for_policy).
|
||||||
under --perms the recorded source mode is copied exactly, including
|
The recorded special bits are stripped first: rsync's fake-super receiver
|
||||||
group/other write and setuid/setgid/sticky bits (rsync parity), and the -E
|
stores the full mode in the xattr but never installs setuid/setgid/sticky on
|
||||||
rule derives exec bits from the destination's read bits exactly like
|
the real file, so only the 0777 permission bits may be replayed. The -E
|
||||||
|
rule then derives exec bits from the destination's read bits exactly like
|
||||||
file_restore_metadata_fd. */
|
file_restore_metadata_fd. */
|
||||||
if (policy.perms || policy.executability) {
|
if (policy.perms || policy.executability) {
|
||||||
struct stat cur;
|
struct stat cur;
|
||||||
@@ -419,19 +547,12 @@ bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
|
|||||||
if (fstat(fd, &cur) != 0) {
|
if (fstat(fd, &cur) != 0) {
|
||||||
log_message(LOG_LEVEL_WARNING, "--fake-super: could not read destination mode: %s",
|
log_message(LOG_LEVEL_WARNING, "--fake-super: could not read destination mode: %s",
|
||||||
strerror(errno));
|
strerror(errno));
|
||||||
} else if (metadata_mode_for_policy((mode_t)ul_mode, cur.st_mode, policy, &want)) {
|
} else if (metadata_mode_for_policy((mode_t)(ul_mode & 0777U), cur.st_mode, policy, &want)) {
|
||||||
if (fchmod(fd, want) != 0)
|
if (fchmod(fd, want) != 0)
|
||||||
log_message(LOG_LEVEL_WARNING,
|
log_message(LOG_LEVEL_WARNING,
|
||||||
"--fake-super: could not restore mode on destination file: %s",
|
"--fake-super: could not restore mode on destination entry: %s",
|
||||||
strerror(errno));
|
strerror(errno));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (policy.times) {
|
|
||||||
struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
|
|
||||||
{.tv_sec = (time_t)mtime_sec, .tv_nsec = mtime_nsec}};
|
|
||||||
if (futimens(fd, times) != 0)
|
|
||||||
log_message(LOG_LEVEL_WARNING,
|
|
||||||
"--fake-super: could not restore mtime on destination file: %s", strerror(errno));
|
|
||||||
}
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
+77
-23
@@ -26,16 +26,22 @@
|
|||||||
* and total bytes) on BOTH ends to prevent OOM/memory abuse; an oversized
|
* and total bytes) on BOTH ends to prevent OOM/memory abuse; an oversized
|
||||||
* or malformed frame is a clean protocol rejection, never an allocation
|
* or malformed frame is a clean protocol rejection, never an allocation
|
||||||
* blowup.
|
* blowup.
|
||||||
* * Application is confined to the exact destination file descriptor
|
* * Application is confined to the exact destination entry: fsetxattr on the
|
||||||
* (fsetxattr on the just-written fd), never a caller-controlled path.
|
* just-written fd for regular files/directories, and for a symlink an
|
||||||
|
* lsetxattr on "/proc/self/fd/<parent_fd>/<leaf>" reached through the
|
||||||
|
* already-opened, confinement-checked parent directory -- never a
|
||||||
|
* caller-controlled path, and never following the link.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
/* Reserved key used by --fake-super to park the source's privileged ownership
|
/* Reserved key used by --fake-super to park the source's privileged ownership
|
||||||
* / mode / mtime on the destination file as an unprivileged user.* xattr, so a
|
* / mode / rdev on the destination file as an unprivileged user.* xattr, so the
|
||||||
* later privileged restore could re-apply them. Exact documented format:
|
* tree is interoperable with rsync 3.4.1 and a later privileged restore can
|
||||||
* uid:gid:mode:mtime_sec:mtime_nsec (decimal, decimal, octal, dec, dec)
|
* re-apply them. This is rsync's own key and value grammar exactly:
|
||||||
* e.g. "1000:1000:644:1765238400:0". */
|
* <octal st_mode with S_IFMT> <rdev_major>,<rdev_minor> <uid>:<gid>
|
||||||
#define FAKESUPER_XATTR "user.fastsync.stat"
|
* e.g. "104711 0,0 1234:5678" for a setuid regular file owned by 1234:5678,
|
||||||
|
* or "20644 1,3 111:222" for a char device. mtime is deliberately NOT part of
|
||||||
|
* the record: exactly like rsync, the file's own timestamp carries it. */
|
||||||
|
#define FAKESUPER_XATTR "user.rsync.%stat"
|
||||||
|
|
||||||
/* --- bounds --- */
|
/* --- bounds --- */
|
||||||
#define XATTR_NAME_MAX 255 /* xattr names are limited to 255 bytes */
|
#define XATTR_NAME_MAX 255 /* xattr names are limited to 255 bytes */
|
||||||
@@ -76,6 +82,17 @@ bool xattr_name_appliable(const char* name, bool preserve_acls);
|
|||||||
* distinct from NULL. */
|
* distinct from NULL. */
|
||||||
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls);
|
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls);
|
||||||
|
|
||||||
|
/* Sender: like xattr_capture_path() but reads the xattrs of `path` ITSELF,
|
||||||
|
* never following a final symlink (llistxattr/lgetxattr). A symlink entry must
|
||||||
|
* use this so the scanner never captures the REFERENT's attributes onto the
|
||||||
|
* link (the path-following variant would). On Linux the VFS refuses to
|
||||||
|
* associate xattrs with symlinks at all, so this normally returns NULL; it is
|
||||||
|
* still correct and portable for a filesystem/platform that supports them.
|
||||||
|
* The same whitelist/bounds as xattr_capture_path() apply. Returns NULL when
|
||||||
|
* the link has no appliable xattrs (or the filesystem does not support them);
|
||||||
|
* an empty-but-valid list is never returned distinct from NULL. */
|
||||||
|
FileXattrList* xattr_capture_path_nofollow(const char* path, bool preserve_acls);
|
||||||
|
|
||||||
/* Wire: bounded serialization. xattr_send returns false on write failure; an
|
/* Wire: bounded serialization. xattr_send returns false on write failure; an
|
||||||
* empty/NULL list transmits a zero-count block. xattr_receive returns NULL and
|
* empty/NULL list transmits a zero-count block. xattr_receive returns NULL and
|
||||||
* sets *ok = 0 on any malformed / oversized / non-whitelisted entry. When
|
* sets *ok = 0 on any malformed / oversized / non-whitelisted entry. When
|
||||||
@@ -91,24 +108,61 @@ FileXattrList* xattr_receive(int fd, int* ok, bool preserve_acls);
|
|||||||
* true when apply was attempted (allowing callers to treat it as best-effort). */
|
* true when apply was attempted (allowing callers to treat it as best-effort). */
|
||||||
bool xattr_apply_fd(int fd, const FileXattrList* list);
|
bool xattr_apply_fd(int fd, const FileXattrList* list);
|
||||||
|
|
||||||
/* --fake-super: write the source uid/gid/mode/mtime record into the reserved
|
/* Receiver: apply every entry to the symlink named by (parent_fd, leaf) WITHOUT
|
||||||
* FAKESUPER_XATTR on `fd`. Best-effort (logged, never fatal). Only meaningful
|
* following it, via lsetxattr() on the confined path
|
||||||
* when metadata was transmitted so the values exist. */
|
* "/proc/self/fd/<parent_fd>/<leaf>". Every incoming name is independently
|
||||||
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, int64_t mtime_sec,
|
* re-validated against xattr_name_appliable() with `preserve_acls`, exactly like
|
||||||
int64_t mtime_nsec);
|
* xattr_apply_fd(): a non-whitelisted namespace (including the reserved
|
||||||
|
* --fake-super key) is skipped, so this primitive stays confined even if handed
|
||||||
|
* a hand-crafted list. A symlink cannot be targeted by the fd-relative
|
||||||
|
* fsetxattr() path: there is no *at() xattr syscall and the kernel rejects
|
||||||
|
* xattr syscalls on an O_PATH descriptor, so the already-opened,
|
||||||
|
* confinement-checked parent directory is the anchor and only the final
|
||||||
|
* component is the (no-follow) link. `leaf` must be a single path component.
|
||||||
|
*
|
||||||
|
* Portability: the "/proc/self/fd/<parent_fd>" anchor requires a mounted /proc.
|
||||||
|
* Where /proc is unavailable (or the fd cannot be addressed that way) the
|
||||||
|
* lsetxattr simply fails and is skipped -- the apply is best-effort exactly like
|
||||||
|
* xattr_apply_fd(), so no error is propagated and the transfer continues. A
|
||||||
|
* per-attribute failure (on Linux every set on a symlink fails with EPERM) is
|
||||||
|
* logged once and skipped, never fatal. Returns false only for an invalid
|
||||||
|
* anchor/list; true when an apply was attempted.
|
||||||
|
*
|
||||||
|
* Residual TOCTOU: `leaf` is a caller-supplied name resolved by path in the
|
||||||
|
* parent, so a local writer could replace the just-created symlink between its
|
||||||
|
* creation and lsetxattr(). This is bounded: it requires write access to the
|
||||||
|
* confinement-checked destination directory (already trusted), can only install
|
||||||
|
* a whitelisted user namespace or POSIX-ACL name, and never follows the link (a
|
||||||
|
* replacement symlink is still applied to as the final, no-follow component). */
|
||||||
|
bool xattr_apply_path_nofollow(int parent_fd, const char* leaf, const FileXattrList* list,
|
||||||
|
bool preserve_acls);
|
||||||
|
|
||||||
|
/* --fake-super: write the source uid/gid/mode/rdev record into the reserved
|
||||||
|
* FAKESUPER_XATTR on `fd`, using rsync 3.4.1's exact grammar (see the key
|
||||||
|
* comment above). `mode` is the full st_mode including its S_IFMT bits.
|
||||||
|
* `fd` may be a regular file, a faked char/block device (written as a regular
|
||||||
|
* file), or a DIRECTORY: rsync stores a directory's faked mode/uid/gid in the
|
||||||
|
* reserved xattr on the directory itself. Best-effort (logged, never fatal).
|
||||||
|
* Only meaningful when metadata was transmitted so the values exist. */
|
||||||
|
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, uint32_t rdev_major,
|
||||||
|
uint32_t rdev_minor);
|
||||||
|
|
||||||
/* --fake-super replay: parse the FAKESUPER_XATTR record previously written on
|
/* --fake-super replay: parse the FAKESUPER_XATTR record previously written on
|
||||||
* `fd` by fake_super_store_fd and re-apply mode/mtime fd-relative. The
|
* `fd` by fake_super_store_fd and re-apply the recorded permission bits
|
||||||
* recorded uid/gid are deliberately NOT chowned for real: --fake-super only
|
* fd-relative. `fd` may be a regular file, a faked device, or a DIRECTORY;
|
||||||
* RECORDS ownership (the caller stores the resolved mapping via
|
* fgetxattr/fchmod work identically on a directory descriptor. The recorded
|
||||||
* identity_resolve_storage_ids), it never performs a real chown. Best-effort:
|
* uid/gid are deliberately NOT chowned for real: --fake-super only RECORDS
|
||||||
* absence of the xattr or a malformed record is a silent no-op that never fails
|
* ownership (the caller stores the resolved mapping via
|
||||||
* the transfer. The MODE leg is applied only when policy.perms||policy.
|
* identity_resolve_storage_ids), it never performs a real chown. The
|
||||||
* executability and the MTIME leg only when policy.times, so the fake-super
|
* recorded rdev is retained for a later privileged restore but is not acted on
|
||||||
* replay cannot bypass the per-attribute split; the mode follows the normal
|
* here. Best-effort: absence of the xattr or a malformed record is a silent
|
||||||
* metadata path exactly (under --perms the source mode is copied verbatim,
|
* no-op that never fails the transfer. The MODE leg is applied only when
|
||||||
* special and group/other write bits included).
|
* policy.perms||policy.executability, and the recorded special bits
|
||||||
* Returns true when the xattr was present and parsed. */
|
* (setuid/setgid/sticky) are NOT applied to the real entry -- exactly like
|
||||||
|
* rsync's fake-super receiver, which stores the full mode in the xattr but
|
||||||
|
* strips the special bits on disk. mtime is not part of the record; the normal
|
||||||
|
* metadata path carries it (policy.times) exactly as rsync sets the file's own
|
||||||
|
* timestamp. Returns true when the xattr was present and parsed. */
|
||||||
bool fake_super_restore_fd(int fd, FileAttrPolicy policy);
|
bool fake_super_restore_fd(int fd, FileAttrPolicy policy);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
hello
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
x
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
y
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
a.txt
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
b.txt
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
world
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
deep
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
../a.txt
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
hello
|
||||||
Symlink
+1
@@ -0,0 +1 @@
|
|||||||
|
b.txt
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
world
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
deep
|
||||||
Symlink
+1
@@ -0,0 +1 @@
|
|||||||
|
../a.txt
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
hello
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
world
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
hello
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
world
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
hello
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
world
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
hello
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
world
|
||||||
@@ -34,7 +34,7 @@ class ServerManager:
|
|||||||
self._proc = None
|
self._proc = None
|
||||||
self._port = None
|
self._port = None
|
||||||
|
|
||||||
def start(self, extra_args=None):
|
def start(self, extra_args=None, env=None):
|
||||||
self.stop()
|
self.stop()
|
||||||
self._port = _find_free_port()
|
self._port = _find_free_port()
|
||||||
# Plain TCP is intentionally explicit in the server; integration tests
|
# Plain TCP is intentionally explicit in the server; integration tests
|
||||||
@@ -42,7 +42,11 @@ class ServerManager:
|
|||||||
cmd = SERVER_CMD + ["-p", str(self._port), "--allow-unauthenticated"]
|
cmd = SERVER_CMD + ["-p", str(self._port), "--allow-unauthenticated"]
|
||||||
if extra_args:
|
if extra_args:
|
||||||
cmd += extra_args
|
cmd += extra_args
|
||||||
self._proc = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
proc_env = dict(os.environ)
|
||||||
|
if env:
|
||||||
|
proc_env.update(env)
|
||||||
|
self._proc = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||||
|
env=proc_env)
|
||||||
_wait_for_port(self._port, timeout=5)
|
_wait_for_port(self._port, timeout=5)
|
||||||
|
|
||||||
def stop(self):
|
def stop(self):
|
||||||
|
|||||||
@@ -141,6 +141,7 @@ class DaemonManager:
|
|||||||
def __init__(self):
|
def __init__(self):
|
||||||
self._proc = None
|
self._proc = None
|
||||||
self._port = None
|
self._port = None
|
||||||
|
self.log_path = None
|
||||||
|
|
||||||
def start(self, config_path, port_override=None, extra_args=None, log_path=None):
|
def start(self, config_path, port_override=None, extra_args=None, log_path=None):
|
||||||
self.stop()
|
self.stop()
|
||||||
@@ -154,7 +155,11 @@ class DaemonManager:
|
|||||||
if extra_args:
|
if extra_args:
|
||||||
cmd += extra_args
|
cmd += extra_args
|
||||||
if log_path is None:
|
if log_path is None:
|
||||||
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
|
# A unique log per manager: several managers run in one xdist
|
||||||
|
# worker, and a shared log lets one daemon's truncate/write offset
|
||||||
|
# corrupt the other's appended lines (a flaky log assertion).
|
||||||
|
log_path = os.path.join(TEST_DATA_DIR, f"fastsyncd_{id(self):x}.log")
|
||||||
|
self.log_path = log_path
|
||||||
log = open(log_path, "w")
|
log = open(log_path, "w")
|
||||||
self._proc = subprocess.Popen(
|
self._proc = subprocess.Popen(
|
||||||
cmd, stdout=log, stderr=log, stdin=subprocess.DEVNULL, start_new_session=True)
|
cmd, stdout=log, stderr=log, stdin=subprocess.DEVNULL, start_new_session=True)
|
||||||
@@ -224,6 +229,7 @@ def daemon_env():
|
|||||||
"\n"
|
"\n"
|
||||||
"[files]\n"
|
"[files]\n"
|
||||||
"path = %s\n"
|
"path = %s\n"
|
||||||
|
"read only = no\n"
|
||||||
"\n"
|
"\n"
|
||||||
"[readonly]\n"
|
"[readonly]\n"
|
||||||
"path = %s\n"
|
"path = %s\n"
|
||||||
@@ -231,18 +237,22 @@ def daemon_env():
|
|||||||
"\n"
|
"\n"
|
||||||
"[locked]\n"
|
"[locked]\n"
|
||||||
"path = %s\n"
|
"path = %s\n"
|
||||||
|
"read only = no\n"
|
||||||
"auth users = alice\n"
|
"auth users = alice\n"
|
||||||
"\n"
|
"\n"
|
||||||
"[team]\n"
|
"[team]\n"
|
||||||
"path = %s\n"
|
"path = %s\n"
|
||||||
|
"read only = no\n"
|
||||||
"auth users = alice,bob\n"
|
"auth users = alice,bob\n"
|
||||||
"\n"
|
"\n"
|
||||||
"[owner]\n"
|
"[owner]\n"
|
||||||
"path = %s\n"
|
"path = %s\n"
|
||||||
|
"read only = no\n"
|
||||||
"client owner = yes\n"
|
"client owner = yes\n"
|
||||||
"\n"
|
"\n"
|
||||||
"[denied]\n"
|
"[denied]\n"
|
||||||
"path = %s\n"
|
"path = %s\n"
|
||||||
|
"read only = no\n"
|
||||||
"hosts deny = 127.0.0.1\n"
|
"hosts deny = 127.0.0.1\n"
|
||||||
% (config_port, FILES_MODULE, READONLY_MODULE, AUTH_MODULE, TEAM_MODULE, OWNER_MODULE,
|
% (config_port, FILES_MODULE, READONLY_MODULE, AUTH_MODULE, TEAM_MODULE, OWNER_MODULE,
|
||||||
DENIED_MODULE))
|
DENIED_MODULE))
|
||||||
@@ -261,7 +271,8 @@ def daemon_env():
|
|||||||
global DETACH_PORT
|
global DETACH_PORT
|
||||||
DETACH_PORT = _find_free_port()
|
DETACH_PORT = _find_free_port()
|
||||||
with open(DETACH_CONF, "w") as f:
|
with open(DETACH_CONF, "w") as f:
|
||||||
f.write("port = %d\n\n[detach]\npath = %s\n" % (DETACH_PORT, DETACH_MODULE))
|
f.write("port = %d\n\n[detach]\npath = %s\nread only = no\n"
|
||||||
|
% (DETACH_PORT, DETACH_MODULE))
|
||||||
|
|
||||||
yield
|
yield
|
||||||
_kill_by_cmdline_marker(DETACH_CONF)
|
_kill_by_cmdline_marker(DETACH_CONF)
|
||||||
@@ -348,7 +359,8 @@ class TestDaemonModuleSelection:
|
|||||||
the fix regresses."""
|
the fix regresses."""
|
||||||
port = _find_free_port()
|
port = _find_free_port()
|
||||||
with open(UMASK_CONF, "w") as f:
|
with open(UMASK_CONF, "w") as f:
|
||||||
f.write("port = %d\n\n[files]\npath = %s\n" % (port, FILES_MODULE))
|
f.write("port = %d\n\n[files]\npath = %s\nread only = no\n"
|
||||||
|
% (port, FILES_MODULE))
|
||||||
sub = os.path.join(FILES_MODULE, "umask_check")
|
sub = os.path.join(FILES_MODULE, "umask_check")
|
||||||
shutil.rmtree(sub, ignore_errors=True)
|
shutil.rmtree(sub, ignore_errors=True)
|
||||||
os.makedirs(sub, exist_ok=True)
|
os.makedirs(sub, exist_ok=True)
|
||||||
@@ -375,6 +387,58 @@ class TestDaemonModuleSelection:
|
|||||||
proc.kill()
|
proc.kill()
|
||||||
|
|
||||||
|
|
||||||
|
class TestRsyncConfigCompat:
|
||||||
|
"""A real rsyncd.conf can be pointed at FastSync: the common rsync GLOBAL
|
||||||
|
and MODULE keys are accepted, the ones with a FastSync equivalent (port,
|
||||||
|
path, read only, max connections) take effect, and the inert ones (pid
|
||||||
|
file, log file, comment, use chroot, uid, gid, exclude, timeout, ...) are
|
||||||
|
documented no-ops. --dparam accepts the same expanded key set."""
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_rsync_style_config_round_trip(self):
|
||||||
|
module = os.path.join(MODULE_ROOT, "rsync_style")
|
||||||
|
shutil.rmtree(module, ignore_errors=True)
|
||||||
|
os.makedirs(module, exist_ok=True)
|
||||||
|
port = _find_free_port()
|
||||||
|
conf = os.path.join(TEST_DATA_DIR, "fastsyncd_rsync_style.conf")
|
||||||
|
with open(conf, "w") as f:
|
||||||
|
f.write(
|
||||||
|
"# an rsync 3.4.1-style rsyncd.conf\n"
|
||||||
|
"pid file = /tmp/fastsyncd_rsync_style.pid\n"
|
||||||
|
"log file = /tmp/fastsyncd_rsync_style.log\n"
|
||||||
|
"socket options = TCP_NODELAY\n"
|
||||||
|
"use chroot = no\n"
|
||||||
|
"uid = nobody\n"
|
||||||
|
"gid = nogroup\n"
|
||||||
|
"timeout = 600\n"
|
||||||
|
"max verbosity = 2\n"
|
||||||
|
"transfer logging = yes\n"
|
||||||
|
"port = %d\n"
|
||||||
|
"\n"
|
||||||
|
"[rsync_style]\n"
|
||||||
|
"path = %s\n"
|
||||||
|
"comment = rsync-style module\n"
|
||||||
|
"use chroot = no\n"
|
||||||
|
"exclude = *.tmp\n"
|
||||||
|
"read only = no\n"
|
||||||
|
"max connections = 4\n"
|
||||||
|
% (port, module))
|
||||||
|
d = DaemonManager()
|
||||||
|
# --dparam borrows rsync's compact spelling; `pidfile` is inert but must
|
||||||
|
# not be rejected, proving dparam reuses the expanded global key set.
|
||||||
|
d.start(conf, extra_args=["--dparam", "pidfile=/tmp/rsync_style.pid"],
|
||||||
|
log_path=os.path.join(TEST_DATA_DIR, "fastsyncd_rsync_style.log"))
|
||||||
|
try:
|
||||||
|
result = _push("127.0.0.1::rsync_style", d.port)
|
||||||
|
assert result.returncode == 0, result.stderr or result.stdout
|
||||||
|
received = get_dest_received_dir(module, SOURCE_DIR)
|
||||||
|
mismatches, missing = verify_transfer(SOURCE_DIR, received)
|
||||||
|
assert not missing, f"missing: {missing[:5]}"
|
||||||
|
assert not mismatches, f"mismatch: {mismatches[:5]}"
|
||||||
|
finally:
|
||||||
|
d.stop()
|
||||||
|
|
||||||
|
|
||||||
class TestDaemonRejection:
|
class TestDaemonRejection:
|
||||||
def _tree_files(self):
|
def _tree_files(self):
|
||||||
"""Snapshot every file path (module-relative) currently under the module
|
"""Snapshot every file path (module-relative) currently under the module
|
||||||
@@ -477,7 +541,7 @@ class TestDaemonRejection:
|
|||||||
before any data lands. `accept` lists the log phrases that count as the
|
before any data lands. `accept` lists the log phrases that count as the
|
||||||
refusal (a non-root daemon refuses --copy-as earlier, at the privilege
|
refusal (a non-root daemon refuses --copy-as earlier, at the privilege
|
||||||
check, so the caller accepts that phrase too)."""
|
check, so the caller accepts that phrase too)."""
|
||||||
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
|
log_path = daemon.log_path
|
||||||
before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
|
before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
|
||||||
before_files = self._tree_files()
|
before_files = self._tree_files()
|
||||||
result, _ = run_client(SOURCE_DIR, f"127.0.0.1::{module}", port=daemon.port, flags=flags)
|
result, _ = run_client(SOURCE_DIR, f"127.0.0.1::{module}", port=daemon.port, flags=flags)
|
||||||
@@ -514,14 +578,13 @@ class TestDaemonRejection:
|
|||||||
the refusal into a silent accept."""
|
the refusal into a silent accept."""
|
||||||
port = _find_free_port()
|
port = _find_free_port()
|
||||||
d = DaemonManager()
|
d = DaemonManager()
|
||||||
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
|
|
||||||
try:
|
try:
|
||||||
d.start(CONF_FILE, port_override=port,
|
d.start(CONF_FILE, port_override=port,
|
||||||
extra_args=["--password-file", CRED_FILE, "--no-super"])
|
extra_args=["--password-file", CRED_FILE, "--no-super"])
|
||||||
result, _ = run_client(SOURCE_DIR, "127.0.0.1::files", port=d.port,
|
result, _ = run_client(SOURCE_DIR, "127.0.0.1::files", port=d.port,
|
||||||
flags=["--super", "--preserve"])
|
flags=["--super", "--preserve"])
|
||||||
assert result.returncode != 0, "the --no-super daemon must refuse --super"
|
assert result.returncode != 0, "the --no-super daemon must refuse --super"
|
||||||
with open(log_path, "rb") as f:
|
with open(d.log_path, "rb") as f:
|
||||||
tail = f.read().decode("utf-8", "replace")
|
tail = f.read().decode("utf-8", "replace")
|
||||||
assert "client-chosen ownership" in tail, (
|
assert "client-chosen ownership" in tail, (
|
||||||
f"daemon did not log the --super refusal: {tail[-400:]!r}"
|
f"daemon did not log the --super refusal: {tail[-400:]!r}"
|
||||||
@@ -1010,7 +1073,7 @@ class TestDaemonAuthentication:
|
|||||||
|
|
||||||
def test_auth_log_does_not_leak_password(self, daemon):
|
def test_auth_log_does_not_leak_password(self, daemon):
|
||||||
"""The daemon log must never contain the password or the store verifier."""
|
"""The daemon log must never contain the password or the store verifier."""
|
||||||
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
|
log_path = daemon.log_path
|
||||||
before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
|
before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
|
||||||
_push_with_creds("127.0.0.1::locked", daemon.port, "alice", WRONG_PASS)
|
_push_with_creds("127.0.0.1::locked", daemon.port, "alice", WRONG_PASS)
|
||||||
_push_with_creds("127.0.0.1::locked", daemon.port, "alice", ALICE_PASS)
|
_push_with_creds("127.0.0.1::locked", daemon.port, "alice", ALICE_PASS)
|
||||||
@@ -1037,7 +1100,7 @@ class TestDaemonAuthentication:
|
|||||||
_push_with_creds("127.0.0.1::locked", port, "alice", ALICE_PASS)
|
_push_with_creds("127.0.0.1::locked", port, "alice", ALICE_PASS)
|
||||||
_push_with_creds("127.0.0.1::locked", port, "alice", WRONG_PASS)
|
_push_with_creds("127.0.0.1::locked", port, "alice", WRONG_PASS)
|
||||||
time.sleep(0.3)
|
time.sleep(0.3)
|
||||||
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
|
log_path = d.log_path
|
||||||
with open(log_path, "rb") as f:
|
with open(log_path, "rb") as f:
|
||||||
log = f.read().decode("utf-8", "replace")
|
log = f.read().decode("utf-8", "replace")
|
||||||
finally:
|
finally:
|
||||||
@@ -1072,7 +1135,8 @@ class TestDaemonMotd:
|
|||||||
motd_line = "motd file = %s\n" % motd_path if motd_path else ""
|
motd_line = "motd file = %s\n" % motd_path if motd_path else ""
|
||||||
os.makedirs(self.MOTD_MODULE, exist_ok=True)
|
os.makedirs(self.MOTD_MODULE, exist_ok=True)
|
||||||
with open(self.MOTD_CONF, "w") as f:
|
with open(self.MOTD_CONF, "w") as f:
|
||||||
f.write("port = %d\n%s\n[files]\npath = %s\n" % (port, motd_line, self.MOTD_MODULE))
|
f.write("port = %d\n%s\n[files]\npath = %s\nread only = no\n"
|
||||||
|
% (port, motd_line, self.MOTD_MODULE))
|
||||||
d = DaemonManager()
|
d = DaemonManager()
|
||||||
d.start(self.MOTD_CONF, port_override=port)
|
d.start(self.MOTD_CONF, port_override=port)
|
||||||
return d, port
|
return d, port
|
||||||
@@ -1256,12 +1320,12 @@ class TestDaemonTLSAuth:
|
|||||||
_write_client_password_file(client_creds, "alice", ALICE_PASS)
|
_write_client_password_file(client_creds, "alice", ALICE_PASS)
|
||||||
d = DaemonManager()
|
d = DaemonManager()
|
||||||
port = _find_free_port()
|
port = _find_free_port()
|
||||||
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
|
|
||||||
try:
|
try:
|
||||||
d.start(CONF_FILE, port_override=port, extra_args=[
|
d.start(CONF_FILE, port_override=port, extra_args=[
|
||||||
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
|
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
|
||||||
"--ca", certs["ca"], "--client-cn", "fastsync-client",
|
"--ca", certs["ca"], "--client-cn", "fastsync-client",
|
||||||
"--password-file", CRED_FILE])
|
"--password-file", CRED_FILE])
|
||||||
|
log_path = d.log_path
|
||||||
before_files = _tree_file_count(AUTH_MODULE)
|
before_files = _tree_file_count(AUTH_MODULE)
|
||||||
log_before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
|
log_before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
|
||||||
tls_flags = ["--tls",
|
tls_flags = ["--tls",
|
||||||
@@ -1309,6 +1373,7 @@ class TestDaemonConnectionLimits:
|
|||||||
"\n"
|
"\n"
|
||||||
"[locked]\n"
|
"[locked]\n"
|
||||||
"path = %s\n"
|
"path = %s\n"
|
||||||
|
"read only = no\n"
|
||||||
"auth users = alice\n"
|
"auth users = alice\n"
|
||||||
% (port, AUTH_MODULE))
|
% (port, AUTH_MODULE))
|
||||||
d = DaemonManager()
|
d = DaemonManager()
|
||||||
@@ -1346,6 +1411,7 @@ class TestDaemonConnectionLimits:
|
|||||||
"\n"
|
"\n"
|
||||||
"[files]\n"
|
"[files]\n"
|
||||||
"path = %s\n"
|
"path = %s\n"
|
||||||
|
"read only = no\n"
|
||||||
"max connections = 2\n"
|
"max connections = 2\n"
|
||||||
% (port, FILES_MODULE))
|
% (port, FILES_MODULE))
|
||||||
d = DaemonManager()
|
d = DaemonManager()
|
||||||
|
|||||||
@@ -111,13 +111,20 @@ class _SlicingProxy:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, target_port, forward_limit=None, hook=None, hook_after=0,
|
def __init__(self, target_port, forward_limit=None, hook=None, hook_after=0,
|
||||||
throttle=0.0, wait_for_reply=False):
|
throttle=0.0, wait_for_reply=False, hook_after_config_ack=False):
|
||||||
self.target = ("127.0.0.1", target_port)
|
self.target = ("127.0.0.1", target_port)
|
||||||
self.forward_limit = forward_limit
|
self.forward_limit = forward_limit
|
||||||
self.hook = hook
|
self.hook = hook
|
||||||
self.hook_after = hook_after
|
self.hook_after = hook_after
|
||||||
self.throttle = throttle
|
self.throttle = throttle
|
||||||
self.wait_for_reply = wait_for_reply
|
self.wait_for_reply = wait_for_reply
|
||||||
|
# When set, the hook fires on the FIRST client->server bytes that follow
|
||||||
|
# the config-frame ack, BEFORE they are forwarded. For --delete-before
|
||||||
|
# those bytes are the keep-set manifest, so this runs the hook after the
|
||||||
|
# client's source pre-scan but before the receiver's delete ack releases
|
||||||
|
# the client into its data pass -- a deterministic late-file window.
|
||||||
|
self.hook_after_config_ack = hook_after_config_ack
|
||||||
|
self.config_acked = False
|
||||||
self.server_replied = threading.Event()
|
self.server_replied = threading.Event()
|
||||||
self.hook_called = threading.Event()
|
self.hook_called = threading.Event()
|
||||||
self.listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
self.listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||||
@@ -165,6 +172,13 @@ class _SlicingProxy:
|
|||||||
socks = []
|
socks = []
|
||||||
break
|
break
|
||||||
data = data[:room]
|
data = data[:room]
|
||||||
|
if (self.hook_after_config_ack and self.config_acked and self.hook is not None
|
||||||
|
and not self.hook_called.is_set()):
|
||||||
|
# The first client bytes after the config ack are the
|
||||||
|
# pre-scan keep-set manifest: run the injection before
|
||||||
|
# forwarding so it is causally after the source scan.
|
||||||
|
self.hook()
|
||||||
|
self.hook_called.set()
|
||||||
backend.sendall(data)
|
backend.sendall(data)
|
||||||
forwarded += len(data)
|
forwarded += len(data)
|
||||||
self._maybe_hook(forwarded)
|
self._maybe_hook(forwarded)
|
||||||
@@ -177,6 +191,7 @@ class _SlicingProxy:
|
|||||||
client.sendall(data)
|
client.sendall(data)
|
||||||
# Any server reply proves the receiver consumed the
|
# Any server reply proves the receiver consumed the
|
||||||
# frames that precede it, so the hook barrier is met.
|
# frames that precede it, so the hook barrier is met.
|
||||||
|
self.config_acked = True
|
||||||
self.server_replied.set()
|
self.server_replied.set()
|
||||||
self._maybe_hook(forwarded)
|
self._maybe_hook(forwarded)
|
||||||
except OSError:
|
except OSError:
|
||||||
@@ -198,8 +213,11 @@ class _SlicingProxy:
|
|||||||
def _maybe_hook(self, forwarded):
|
def _maybe_hook(self, forwarded):
|
||||||
"""Fire the one-shot hook once its barrier is satisfied: enough client
|
"""Fire the one-shot hook once its barrier is satisfied: enough client
|
||||||
bytes have been forwarded and, when ``wait_for_reply`` is set, the
|
bytes have been forwarded and, when ``wait_for_reply`` is set, the
|
||||||
server has sent a reply proving it processed the preceding frames."""
|
server has sent a reply proving it processed the preceding frames.
|
||||||
if self.hook is None or self.hook_called.is_set():
|
|
||||||
|
``hook_after_config_ack`` uses its own barrier (see ``_serve``), so the
|
||||||
|
byte/reply heuristic is bypassed entirely."""
|
||||||
|
if self.hook is None or self.hook_called.is_set() or self.hook_after_config_ack:
|
||||||
return
|
return
|
||||||
if forwarded < self.hook_after:
|
if forwarded < self.hook_after:
|
||||||
return
|
return
|
||||||
@@ -537,3 +555,76 @@ class TestDeleteDelayMaxDeleteRefilledDir:
|
|||||||
assert os.path.isdir(later_dir), "later extra was not skipped by the budget"
|
assert os.path.isdir(later_dir), "later extra was not skipped by the budget"
|
||||||
# The one actual removal is reported.
|
# The one actual removal is reported.
|
||||||
assert _deleted_count(result.stdout) == 1, result.stdout
|
assert _deleted_count(result.stdout) == 1, result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
class TestDeleteBeforeLateFileParity:
|
||||||
|
"""rsync builds its file list once, so a source file created after that scan
|
||||||
|
is NOT transferred and its destination extra is deleted. FastSync used to
|
||||||
|
re-scan the source in its data pass (single-threaded) or pipeline a fresh
|
||||||
|
re-scan against the pre-scan keep-set (``--threads``) and would transfer the
|
||||||
|
late file (a safe superset); both paths now replay the pre-scan file list
|
||||||
|
instead, matching rsync.
|
||||||
|
|
||||||
|
The late file is injected through the config-ack barrier: the first client
|
||||||
|
bytes after the config ack are the pre-scan keep-set manifest, so the hook
|
||||||
|
runs causally after the source scan and before the receiver's delete ack
|
||||||
|
releases the client into its data pass.
|
||||||
|
|
||||||
|
For ``--threads`` the pipeline scanner runs concurrently with the sender, so
|
||||||
|
the injection must land while that re-scan is still in flight to be observed
|
||||||
|
by it. The source is therefore a tree of ``_N_DIRS`` directories: the
|
||||||
|
injection writes the late file into EVERY directory, so it is enough that
|
||||||
|
any one directory is still unscanned when the hook fires. The tree is sized
|
||||||
|
so the hook (a localhost round trip) lands long before a full scan finishes;
|
||||||
|
a re-scanning pipeline then transfers the late files for the directories it
|
||||||
|
has not yet reached, which the tree comparison catches.
|
||||||
|
"""
|
||||||
|
|
||||||
|
_N_DIRS = 2000
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
|
def test_late_source_file_not_transferred_and_extra_deleted(self, mt):
|
||||||
|
tag = f"dblate_mt{int(mt)}"
|
||||||
|
source = os.path.join(TEST_DATA_DIR, f"{tag}_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, f"{tag}_dst")
|
||||||
|
rsync_dst = os.path.join(TEST_DATA_DIR, f"{tag}_rsync_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
clean_dir(rsync_dst)
|
||||||
|
for i in range(self._N_DIRS):
|
||||||
|
_write(os.path.join(source, f"dir{i:05d}", "keep.txt"), b"kept payload\n")
|
||||||
|
# Both destinations carry the would-be late file as an extra.
|
||||||
|
for root in (dest, rsync_dst):
|
||||||
|
received = get_dest_received_dir(root, source)
|
||||||
|
for i in range(self._N_DIRS):
|
||||||
|
_write(os.path.join(received, f"dir{i:05d}", "late.txt"), b"stale extra\n")
|
||||||
|
|
||||||
|
# rsync reference: the same source with no late file; the extras are
|
||||||
|
# removed and nothing is transferred for the (never-scanned) late paths.
|
||||||
|
rsync_result = _rsync(["-a", "--delete-before", source + "/", rsync_dst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
rsync_tree = _tree(rsync_dst)
|
||||||
|
assert "dir00000/late.txt" not in rsync_tree
|
||||||
|
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
|
||||||
|
def hook():
|
||||||
|
# Runs after the pre-scan and before the receiver's delete ack.
|
||||||
|
for i in range(self._N_DIRS):
|
||||||
|
_write(os.path.join(source, f"dir{i:05d}", "late.txt"),
|
||||||
|
b"created after the scan\n")
|
||||||
|
|
||||||
|
with ServerManager() as server:
|
||||||
|
server.start(extra_args=["--allow-delete"])
|
||||||
|
proxy = _SlicingProxy(server.port, hook=hook, hook_after_config_ack=True)
|
||||||
|
flags = ["--delete-before"] + (["--threads=4"] if mt else [])
|
||||||
|
result, _ = run_client(source, dest, flags=flags, port=proxy.port)
|
||||||
|
proxy.finish()
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||||
|
assert proxy.hook_called.is_set(), "late-file hook never fired"
|
||||||
|
assert _tree(received) == rsync_tree, (
|
||||||
|
f"late source {'multithreaded' if mt else 'single-threaded'} data pass re-scanned: "
|
||||||
|
f"{sum(1 for p in _tree(received) if p.endswith('late.txt'))} late files were "
|
||||||
|
"transferred"
|
||||||
|
)
|
||||||
@@ -17,6 +17,7 @@ Run locally::
|
|||||||
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity
|
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import sys
|
import sys
|
||||||
import warnings
|
import warnings
|
||||||
@@ -105,6 +106,15 @@ def seed_backup(_src, rroot, froot):
|
|||||||
_mk(os.path.join(root, "a.txt"), b"OLD-CONTENT\n", _OLD_MTIME)
|
_mk(os.path.join(root, "a.txt"), b"OLD-CONTENT\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def seed_delay_updates(_src, rroot, froot):
|
||||||
|
"""A changed file plus an extra, so --delay-updates (and its implied
|
||||||
|
--delete-after) has both a publication and a deletion to order."""
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "a.txt"), b"OLD-CONTENT\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "extra.txt"), b"extra\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "extradir", "z.txt"), b"z\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
def seed_size_only(_src, rroot, froot):
|
def seed_size_only(_src, rroot, froot):
|
||||||
for root in (rroot, froot):
|
for root in (rroot, froot):
|
||||||
_mk(os.path.join(root, "a.txt"), b"XXXXXXXXXXX\n", _OLD_MTIME)
|
_mk(os.path.join(root, "a.txt"), b"XXXXXXXXXXX\n", _OLD_MTIME)
|
||||||
@@ -127,6 +137,66 @@ def seed_filter_protect(_src, rroot, froot):
|
|||||||
_mk(os.path.join(root, "sub", "other2.txt"), b"nested dest-only other\n", _OLD_MTIME)
|
_mk(os.path.join(root, "sub", "other2.txt"), b"nested dest-only other\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def seed_perdir_protect(_src, rroot, froot):
|
||||||
|
"""Per-directory `.rsync-filter` carrying `P` rules on the source and both
|
||||||
|
destinations, plus destination-only extras. The `.log` extras must survive
|
||||||
|
--delete under every timing while the other extras go; the source's
|
||||||
|
`.rsync-filter` (which FastSync carries to the receiver) and the seeded
|
||||||
|
destination one (which rsync's receiver reads) are byte-identical."""
|
||||||
|
for root in (_src, rroot, froot):
|
||||||
|
_mk(os.path.join(root, ".rsync-filter"), b"P extra.log\nP nested.log\n")
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "extra.log"), b"dest-only protected\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "other.txt"), b"dest-only deleted\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "nested.log"), b"nested protected\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "other2.txt"), b"nested deleted\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def seed_perdir_exclude(_src, rroot, froot):
|
||||||
|
"""Per-directory unqualified exclude (`-`): dual-sided, so it protects the
|
||||||
|
matching destination-only extra (and is opted back in by
|
||||||
|
--delete-excluded)."""
|
||||||
|
for root in (_src, rroot, froot):
|
||||||
|
_mk(os.path.join(root, ".rsync-filter"), b"- extra.log\n")
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "extra.log"), b"dest-only excluded\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "other.txt"), b"dest-only deleted\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def seed_perdir_subdir_protect(_src, rroot, froot):
|
||||||
|
"""A SUBDIRECTORY-owned `.rsync-filter` (its owner is not the transfer root):
|
||||||
|
the receiver must re-derive the `P` rules in the destination-relative
|
||||||
|
coordinate system, otherwise the destination-only nested extras are wrongly
|
||||||
|
deleted (silent data loss). A root-level extra is included so a too-broad
|
||||||
|
rule would over-protect. The file is seeded on both destinations because
|
||||||
|
rsync's receiver reads the per-directory file locally for delete-during."""
|
||||||
|
for root in (_src, rroot, froot):
|
||||||
|
_mk(os.path.join(root, "sub", ".rsync-filter"), b"P nested.log\nP extra.log\n")
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "sub", "nested.log"), b"dest-only protected\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "extra.log"), b"dest-only protected 2\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "other.txt"), b"dest-only deleted\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "root_extra.txt"), b"root dest-only deleted\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def seed_perdir_subdir_exclude(_src, rroot, froot):
|
||||||
|
"""A SUBDIRECTORY-owned unqualified exclude (`-`): dual-sided, so it protects
|
||||||
|
the matching destination-only nested extra under plain --delete and is opted
|
||||||
|
back in by --delete-excluded."""
|
||||||
|
for root in (_src, rroot, froot):
|
||||||
|
_mk(os.path.join(root, "sub", ".rsync-filter"), b"- nested.log\n")
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "sub", "nested.log"), b"dest-only excluded\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "other.txt"), b"dest-only deleted\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_rules(content):
|
||||||
|
def seed(_src, _rroot, _froot):
|
||||||
|
_mk(os.path.join(_src, ".rules"), content)
|
||||||
|
|
||||||
|
return seed
|
||||||
|
|
||||||
|
|
||||||
def seed_max_delete(_src, rroot, froot):
|
def seed_max_delete(_src, rroot, froot):
|
||||||
for root in (rroot, froot):
|
for root in (rroot, froot):
|
||||||
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
|
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
|
||||||
@@ -244,6 +314,14 @@ _CASES = [
|
|||||||
H.Case("delete_commit", "basic", ["-a", "--delete-after"], seed=seed_extras,
|
H.Case("delete_commit", "basic", ["-a", "--delete-after"], seed=seed_extras,
|
||||||
fastsync_flags=["-a", "--delete-commit"], server_args=DELETE,
|
fastsync_flags=["-a", "--delete-commit"], server_args=DELETE,
|
||||||
ref="FastSync-only --delete-commit == rsync --delete-after"),
|
ref="FastSync-only --delete-commit == rsync --delete-after"),
|
||||||
|
# #317: --delay-updates stages under a per-run unique name and publishes
|
||||||
|
# every update before the implied --delete-after removes extras.
|
||||||
|
H.Case("delay_updates", "basic", ["-a", "--delay-updates"],
|
||||||
|
seed=seed_delay_updates, ref="--delay-updates stages then publishes"),
|
||||||
|
H.Case("delay_updates_delete", "basic",
|
||||||
|
["-a", "--delay-updates", "--delete"], seed=seed_delay_updates,
|
||||||
|
server_args=DELETE, ci=True,
|
||||||
|
ref="--delay-updates implies --delete-after (publish before delete)"),
|
||||||
H.Case("delete_excluded", "filters",
|
H.Case("delete_excluded", "filters",
|
||||||
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
|
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
|
||||||
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
|
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
|
||||||
@@ -271,6 +349,81 @@ _CASES = [
|
|||||||
["-a", "--delete-after", "--filter=P *.log"],
|
["-a", "--delete-after", "--filter=P *.log"],
|
||||||
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
ref="--filter P/--protect under the whole-tree --delete-after commit"),
|
ref="--filter P/--protect under the whole-tree --delete-after commit"),
|
||||||
|
# Per-directory merge rules (#315): the receiver must re-derive the
|
||||||
|
# protect/risk verdict from the carried per-directory rules, so a
|
||||||
|
# destination-only entry matching ONLY a per-directory rule is shielded.
|
||||||
|
H.Case("filter_perdir_protect", "filters",
|
||||||
|
["-a", "-F", "--delete"],
|
||||||
|
seed=seed_perdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F per-directory P rule under the default --delete timing"),
|
||||||
|
H.Case("filter_perdir_protect_during", "filters",
|
||||||
|
["-a", "-F", "--delete-during"],
|
||||||
|
seed=seed_perdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F per-directory P rule under --delete-during"),
|
||||||
|
H.Case("filter_perdir_protect_delay", "filters",
|
||||||
|
["-a", "-F", "--delete-delay"],
|
||||||
|
seed=seed_perdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F per-directory P rule under --delete-delay"),
|
||||||
|
H.Case("filter_perdir_protect_before", "filters",
|
||||||
|
["-a", "-F", "--delete-before"],
|
||||||
|
seed=seed_perdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F per-directory P rule under the whole-tree --delete-before commit"),
|
||||||
|
H.Case("filter_perdir_protect_after", "filters",
|
||||||
|
["-a", "-F", "--delete-after"],
|
||||||
|
seed=seed_perdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F per-directory P rule under the whole-tree --delete-after commit"),
|
||||||
|
H.Case("filter_perdir_exclude_protect", "filters",
|
||||||
|
["-a", "-F", "--delete"],
|
||||||
|
seed=seed_perdir_exclude, server_args=DELETE, ci=True,
|
||||||
|
ref="-F per-directory exclude protects its destination mirror"),
|
||||||
|
H.Case("filter_perdir_exclude_deleted", "filters",
|
||||||
|
["-a", "-F", "--delete", "--delete-excluded"],
|
||||||
|
seed=seed_perdir_exclude, server_args=DELETE, ci=True,
|
||||||
|
ref="-F per-directory exclude under --delete-excluded is at risk"),
|
||||||
|
# #316: a rule owned by a SUBDIRECTORY (not the transfer root) must be
|
||||||
|
# re-expressed in the receiver's destination-relative coordinate system, or
|
||||||
|
# the dest-only extras it protects are silently deleted.
|
||||||
|
H.Case("filter_perdir_subdir_protect", "filters",
|
||||||
|
["-a", "-F", "--delete"],
|
||||||
|
seed=seed_perdir_subdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F subdirectory-owned P rule under the default --delete timing"),
|
||||||
|
H.Case("filter_perdir_subdir_protect_during", "filters",
|
||||||
|
["-a", "-F", "--delete-during"],
|
||||||
|
seed=seed_perdir_subdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F subdirectory-owned P rule under --delete-during"),
|
||||||
|
H.Case("filter_perdir_subdir_protect_delay", "filters",
|
||||||
|
["-a", "-F", "--delete-delay"],
|
||||||
|
seed=seed_perdir_subdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F subdirectory-owned P rule under --delete-delay"),
|
||||||
|
H.Case("filter_perdir_subdir_protect_before", "filters",
|
||||||
|
["-a", "-F", "--delete-before"],
|
||||||
|
seed=seed_perdir_subdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F subdirectory-owned P rule under the whole-tree --delete-before commit"),
|
||||||
|
H.Case("filter_perdir_subdir_protect_after", "filters",
|
||||||
|
["-a", "-F", "--delete-after"],
|
||||||
|
seed=seed_perdir_subdir_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="-F subdirectory-owned P rule under the whole-tree --delete-after commit"),
|
||||||
|
H.Case("filter_perdir_subdir_exclude_protect", "filters",
|
||||||
|
["-a", "-F", "--delete"],
|
||||||
|
seed=seed_perdir_subdir_exclude, server_args=DELETE, ci=True,
|
||||||
|
ref="-F subdirectory-owned exclude protects its destination mirror"),
|
||||||
|
H.Case("filter_perdir_subdir_exclude_deleted", "filters",
|
||||||
|
["-a", "-F", "--delete", "--delete-excluded"],
|
||||||
|
seed=seed_perdir_subdir_exclude, server_args=DELETE, ci=True,
|
||||||
|
ref="-F subdirectory-owned exclude under --delete-excluded is at risk"),
|
||||||
|
# Merge-file modifiers (#315): e/n/w/- semantics match rsync 3.4.1.
|
||||||
|
H.Case("dir_merge_e", "filters", ["-a", "--filter=:e .rules"],
|
||||||
|
seed=_seed_rules(b"- *.log\n"), ci=True,
|
||||||
|
ref="dir-merge,e excludes the merge file itself"),
|
||||||
|
H.Case("dir_merge_n", "filters", ["-a", "--filter=:n .rules"],
|
||||||
|
seed=_seed_rules(b"- *.log\n"), ci=True,
|
||||||
|
ref="dir-merge,n does not inherit into subdirectories"),
|
||||||
|
H.Case("dir_merge_dash", "filters", ["-a", "--filter=:- .rules"],
|
||||||
|
seed=_seed_rules(b"*.log\n*.bin\n"), ci=True,
|
||||||
|
ref="dir-merge,- reads the file as bare exclude patterns"),
|
||||||
|
H.Case("dir_merge_w", "filters", ["-a", "--filter=:-w .rules"],
|
||||||
|
seed=_seed_rules(b"*.log *.bin\n"), ci=True,
|
||||||
|
ref="dir-merge,w word-splits bare patterns on whitespace"),
|
||||||
|
|
||||||
# --- relative / dirs --------------------------------------------------
|
# --- relative / dirs --------------------------------------------------
|
||||||
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
|
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
|
||||||
@@ -666,6 +819,157 @@ def test_added_and_deleted_between_runs(parity_server_factory):
|
|||||||
_run_and_check(case_id, result)
|
_run_and_check(case_id, result)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_dest_tree(src, root):
|
||||||
|
"""Copy `src`'s tree into `root` (the transfer mirror), preserving symlinks
|
||||||
|
and directory mtimes, so a second differential run starts from an existing
|
||||||
|
destination exactly like a seeded rsync run."""
|
||||||
|
os.makedirs(root, exist_ok=True)
|
||||||
|
for dirpath, dirnames, filenames in os.walk(src):
|
||||||
|
rel = os.path.relpath(dirpath, src)
|
||||||
|
for name in dirnames:
|
||||||
|
s = os.path.join(dirpath, name)
|
||||||
|
d = os.path.join(root, rel, name) if rel != "." else os.path.join(root, name)
|
||||||
|
if os.path.islink(s):
|
||||||
|
continue
|
||||||
|
os.makedirs(d, exist_ok=True)
|
||||||
|
for name in filenames:
|
||||||
|
s = os.path.join(dirpath, name)
|
||||||
|
d = os.path.join(root, rel, name) if rel != "." else os.path.join(root, name)
|
||||||
|
os.makedirs(os.path.dirname(d), exist_ok=True)
|
||||||
|
if os.path.islink(s):
|
||||||
|
if os.path.lexists(d):
|
||||||
|
os.remove(d)
|
||||||
|
os.symlink(os.readlink(s), d)
|
||||||
|
else:
|
||||||
|
shutil.copy2(s, d)
|
||||||
|
if rel != ".":
|
||||||
|
os.utime(os.path.join(root, rel), None)
|
||||||
|
os.utime(root, None)
|
||||||
|
|
||||||
|
|
||||||
|
# A full rsync itemize code (11 columns) followed by the name. H._ITEMIZE_RE
|
||||||
|
# only matches created (`+`) entries, so the changed-attribute codes this test
|
||||||
|
# asserts need their own matcher.
|
||||||
|
_ITEMIZE_LINE_RE = re.compile(r"^[<>ch.*][fdLDS].{9} ")
|
||||||
|
|
||||||
|
|
||||||
|
def _itemize_dir_link_lines(text):
|
||||||
|
"""The itemize lines for directory and symlink entries, excluding the
|
||||||
|
transfer-root `./` line (FastSync emits it unconditionally; a documented
|
||||||
|
residual)."""
|
||||||
|
out = []
|
||||||
|
for line in (text or "").splitlines():
|
||||||
|
line = line.rstrip()
|
||||||
|
if not line or not _ITEMIZE_LINE_RE.match(line):
|
||||||
|
continue
|
||||||
|
name = line.rsplit(" ", 1)[-1]
|
||||||
|
if name == "./":
|
||||||
|
continue
|
||||||
|
if name.endswith("/") or " -> " in line:
|
||||||
|
out.append(line)
|
||||||
|
return sorted(out)
|
||||||
|
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@parity
|
||||||
|
def test_itemize_rerun_dirs_symlinks_matches_rsync(parity_server_factory):
|
||||||
|
"""#314: a re-run reports directory/symlink destination state like rsync.
|
||||||
|
|
||||||
|
On an unchanged tree FastSync emits no per-directory `cd+++++++++` (or
|
||||||
|
symlink) lines, and after a changed directory mtime / symlink target it
|
||||||
|
renders rsync's `.d..t......` / `cLc........` instead of `cd`/`cL`."""
|
||||||
|
case_id = "itemize_rerun_dirs_symlinks"
|
||||||
|
src = os.path.join(TEST_DATA_DIR, "parity_itemds_src")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "parity_itemds_rdst")
|
||||||
|
fdst = os.path.join(TEST_DATA_DIR, "parity_itemds_fdst")
|
||||||
|
clean_dir(src)
|
||||||
|
_mk(os.path.join(src, "sub", "b.txt"), b"nested\n")
|
||||||
|
os.makedirs(os.path.join(src, "emptydir"), exist_ok=True)
|
||||||
|
os.symlink("a.txt", os.path.join(src, "link"))
|
||||||
|
_mk(os.path.join(src, "a.txt"), b"top\n")
|
||||||
|
clean_dir(rdst)
|
||||||
|
clean_dir(fdst)
|
||||||
|
server = parity_server_factory(SUPER)
|
||||||
|
rroot = rdst
|
||||||
|
froot = get_dest_received_dir(fdst, src)
|
||||||
|
_seed_dest_tree(src, rroot)
|
||||||
|
_seed_dest_tree(src, froot)
|
||||||
|
|
||||||
|
# Unchanged re-run: no directory or symlink itemize lines from either tool.
|
||||||
|
rs = H.run_rsync(src, rdst, ["-a", "-i"])
|
||||||
|
fs, _ = H.run_fastsync(src, fdst, ["-a", "-i", "--incremental"], server.port)
|
||||||
|
assert rs.returncode == 0, rs.stderr
|
||||||
|
assert fs.returncode == 0, fs.stderr
|
||||||
|
assert _itemize_dir_link_lines(rs.stdout) == []
|
||||||
|
fast_unchanged = _itemize_dir_link_lines(fs.stdout)
|
||||||
|
assert fast_unchanged == [], f"unchanged re-run itemized dirs/links: {fast_unchanged}"
|
||||||
|
|
||||||
|
# Change the directory mtime and the symlink target, then re-run.
|
||||||
|
_pin(os.path.join(src, "sub"), _OLD_MTIME)
|
||||||
|
os.remove(os.path.join(src, "link"))
|
||||||
|
os.symlink("b.txt", os.path.join(src, "link"))
|
||||||
|
rs = H.run_rsync(src, rdst, ["-a", "-i"])
|
||||||
|
fs, _ = H.run_fastsync(src, fdst, ["-a", "-i", "--incremental"], server.port)
|
||||||
|
assert rs.returncode == 0, rs.stderr
|
||||||
|
assert fs.returncode == 0, fs.stderr
|
||||||
|
expected = _itemize_dir_link_lines(rs.stdout)
|
||||||
|
actual = _itemize_dir_link_lines(fs.stdout)
|
||||||
|
assert actual == expected, f"rsync={rs.stdout!r} fastsync={fs.stdout!r}"
|
||||||
|
assert any(line.endswith(" sub/") and line.startswith(".d..t") for line in actual), actual
|
||||||
|
assert any(line.startswith("cLc") and " -> b.txt" in line for line in actual), actual
|
||||||
|
|
||||||
|
|
||||||
|
def _deleted_breakdown_line(text):
|
||||||
|
for line in (text or "").splitlines():
|
||||||
|
if line.startswith("Number of deleted files:"):
|
||||||
|
return " ".join(line.split())
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@parity
|
||||||
|
def test_stats_deleted_breakdown_matches_rsync(parity_server_factory):
|
||||||
|
"""#316: `--stats` renders rsync's per-type `Number of deleted files`
|
||||||
|
breakdown for removed regular files, directories, symlinks and a special."""
|
||||||
|
case_id = "stats_deleted_breakdown"
|
||||||
|
src = os.path.join(TEST_DATA_DIR, "parity_delbd_src")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "parity_delbd_rdst")
|
||||||
|
fdst = os.path.join(TEST_DATA_DIR, "parity_delbd_fdst")
|
||||||
|
clean_dir(src)
|
||||||
|
_mk(os.path.join(src, "keep.txt"), b"keep\n")
|
||||||
|
server = parity_server_factory(DELETE)
|
||||||
|
|
||||||
|
def seed(_src, rroot, froot):
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "extradir", "inside.txt"), b"e2\n", _OLD_MTIME)
|
||||||
|
os.makedirs(os.path.join(root, "extradir"), exist_ok=True)
|
||||||
|
link = os.path.join(root, "extralink")
|
||||||
|
if not os.path.lexists(link):
|
||||||
|
os.symlink("keep.txt", link)
|
||||||
|
fifo = os.path.join(root, "extrafifo")
|
||||||
|
if not os.path.exists(fifo):
|
||||||
|
os.mkfifo(fifo)
|
||||||
|
|
||||||
|
def extra(_src, _rroot, _froot, rs, fs):
|
||||||
|
rs_line = _deleted_breakdown_line(rs.stdout)
|
||||||
|
fs_line = _deleted_breakdown_line(fs.stdout)
|
||||||
|
if not rs_line:
|
||||||
|
return ["rsync printed no deleted-files line"]
|
||||||
|
if rs_line != fs_line:
|
||||||
|
return [f"deleted breakdown rsync={rs_line!r} fastsync={fs_line!r}"]
|
||||||
|
if "reg:" not in rs_line or "dir:" not in rs_line or \
|
||||||
|
"link:" not in rs_line or "special:" not in rs_line:
|
||||||
|
return [f"breakdown missing a category: {rs_line!r}"]
|
||||||
|
return []
|
||||||
|
|
||||||
|
result = H.run_differential(
|
||||||
|
src, rdst, fdst, ["-a", "--delete", "--stats"],
|
||||||
|
["-a", "--delete", "--stats", "--incremental"], server,
|
||||||
|
seed=seed, extra_check=extra)
|
||||||
|
_run_and_check(case_id, result, ref="--stats deleted per-type breakdown")
|
||||||
|
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
@parity
|
@parity
|
||||||
def test_one_file_system(parity_server_factory):
|
def test_one_file_system(parity_server_factory):
|
||||||
|
|||||||
@@ -36,7 +36,7 @@ from common import ( # noqa: E402
|
|||||||
verify_transfer,
|
verify_transfer,
|
||||||
)
|
)
|
||||||
|
|
||||||
PROTOCOL_VERSION = b"2.28.0"
|
PROTOCOL_VERSION = b"2.30.0"
|
||||||
STATUS_MANIFEST = 5
|
STATUS_MANIFEST = 5
|
||||||
STATUS_OK = 0
|
STATUS_OK = 0
|
||||||
|
|
||||||
|
|||||||
@@ -183,10 +183,11 @@ class TestDeviceSpecial:
|
|||||||
)
|
)
|
||||||
|
|
||||||
@pytest.mark.setpriv
|
@pytest.mark.setpriv
|
||||||
def test_devices_nonroot_receiver_skips_safely(self):
|
def test_devices_nonroot_receiver_errors_like_rsync(self):
|
||||||
"""A receiver without CAP_MKNOD must skip a device entry with a warning
|
"""A receiver without CAP_MKNOD must report the failed device mknod as a
|
||||||
and never abort. A root runner drops the receiver (server) to nobody
|
transfer error (rsync parity, partial failure) instead of silently
|
||||||
via setpriv; on a non-root runner (or without setpriv) the test skips."""
|
succeeding. A root runner drops the receiver (server) to nobody via
|
||||||
|
setpriv; on a non-root runner (or without setpriv) the test skips."""
|
||||||
if os.geteuid() != 0 or shutil.which("setpriv") is None:
|
if os.geteuid() != 0 or shutil.which("setpriv") is None:
|
||||||
pytest.skip("requires root + setpriv to run the receiver unprivileged")
|
pytest.skip("requires root + setpriv to run the receiver unprivileged")
|
||||||
self._setup()
|
self._setup()
|
||||||
@@ -202,16 +203,48 @@ class TestDeviceSpecial:
|
|||||||
flags=["--devices"], port=port)
|
flags=["--devices"], port=port)
|
||||||
finally:
|
finally:
|
||||||
out, err = _stop_captured_server(server)
|
out, err = _stop_captured_server(server)
|
||||||
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:300]}"
|
assert result.returncode == 23, (
|
||||||
received = get_dest_received_dir(DEVICE_DEST, DEVICE_SOURCE)
|
f"a failed device mknod must exit 23 (rsync partial transfer), got "
|
||||||
with open(os.path.join(received, "plain.txt")) as f:
|
f"{result.returncode}: {(out + err)[:300]}"
|
||||||
assert f.read() == "regular content\n"
|
|
||||||
assert not os.path.lexists(os.path.join(received, "chardev")), (
|
|
||||||
"a receiver without CAP_MKNOD must skip the device node, not create it"
|
|
||||||
)
|
)
|
||||||
assert ("cannot create device node" in (out + err)
|
received = get_dest_received_dir(DEVICE_DEST, DEVICE_SOURCE)
|
||||||
or "device-node creation is not permitted" in (out + err)), (
|
assert not os.path.lexists(os.path.join(received, "chardev")), (
|
||||||
f"receiver did not log the documented device skip: out={out!r} err={err!r}"
|
"a receiver without CAP_MKNOD must not create the device node"
|
||||||
|
)
|
||||||
|
assert "cannot create device" in (out + err), (
|
||||||
|
f"receiver did not log the device creation error: out={out!r} err={err!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
@pytest.mark.skipif(os.geteuid() != 0, reason="requires root to create device nodes")
|
||||||
|
def test_devices_nonroot_partial_removes_transferred_sources(self):
|
||||||
|
"""rsync parity for a partial receiver run under --remove-source-files:
|
||||||
|
the successfully transferred regular source is still removed, the
|
||||||
|
un-creatable device source is kept, and the client exits 23 (verified
|
||||||
|
against rsync 3.4.1: it removes ok.txt/ok2.txt, keeps the device, and
|
||||||
|
exits 23)."""
|
||||||
|
if os.geteuid() != 0 or shutil.which("setpriv") is None:
|
||||||
|
pytest.skip("requires root + setpriv to run the receiver unprivileged")
|
||||||
|
self._setup()
|
||||||
|
os.mknod(os.path.join(DEVICE_SOURCE, "chardev"), stat.S_IFCHR | 0o666,
|
||||||
|
os.makedev(1, 3))
|
||||||
|
os.makedirs(DEVICE_DEST, exist_ok=True)
|
||||||
|
os.chmod(DEVICE_DEST, 0o777)
|
||||||
|
server, port = _start_captured_server(
|
||||||
|
prefix=["setpriv", "--reuid=65534", "--regid=65534", "--clear-groups"])
|
||||||
|
try:
|
||||||
|
result, _ = run_client(DEVICE_SOURCE, DEVICE_DEST,
|
||||||
|
flags=["--devices", "--remove-source-files"], port=port)
|
||||||
|
finally:
|
||||||
|
out, err = _stop_captured_server(server)
|
||||||
|
assert result.returncode == 23, (
|
||||||
|
f"a partial receiver run must exit 23, got {result.returncode}: "
|
||||||
|
f"{(out + err)[:300]}"
|
||||||
|
)
|
||||||
|
assert not os.path.exists(os.path.join(DEVICE_SOURCE, "plain.txt")), (
|
||||||
|
"a successfully transferred source must be removed even on a partial run"
|
||||||
|
)
|
||||||
|
assert os.path.exists(os.path.join(DEVICE_SOURCE, "chardev")), (
|
||||||
|
"the source device that failed to materialize must be kept"
|
||||||
)
|
)
|
||||||
|
|
||||||
@pytest.mark.skipif(os.geteuid() != 0, reason="requires root to create device nodes")
|
@pytest.mark.skipif(os.geteuid() != 0, reason="requires root to create device nodes")
|
||||||
@@ -333,6 +366,57 @@ def setup_test_data():
|
|||||||
shutil.rmtree(DEST_DIR, ignore_errors=True)
|
shutil.rmtree(DEST_DIR, ignore_errors=True)
|
||||||
|
|
||||||
|
|
||||||
|
class TestClientStderrChannel:
|
||||||
|
"""--stderr=client: the client's own diagnostics go to the peer's stderr."""
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_client_diagnostic_reaches_server_stderr(self):
|
||||||
|
"""A client-side warning emitted during the transfer is forwarded over
|
||||||
|
the STATUS_CLIENT_MSG channel and printed on the server's stderr, not the
|
||||||
|
client's. A dangling symlink under -L is the deterministic trigger."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "client_msg_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "client_msg_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
with open(os.path.join(source, "plain.txt"), "wb") as f:
|
||||||
|
f.write(b"payload\n")
|
||||||
|
os.symlink("no-such-referent", os.path.join(source, "dangling"))
|
||||||
|
server, port = _start_captured_server()
|
||||||
|
try:
|
||||||
|
result, _ = run_client(source, dest, flags=["-L", "--stderr=client"],
|
||||||
|
port=port)
|
||||||
|
finally:
|
||||||
|
out, err = _stop_captured_server(server)
|
||||||
|
assert "symlink has no referent" in (out + err), (
|
||||||
|
f"client diagnostic did not reach the server stderr: out={out!r} err={err!r}"
|
||||||
|
)
|
||||||
|
assert "symlink has no referent" not in (result.stderr or ""), (
|
||||||
|
f"client diagnostic must not also be written locally: {result.stderr!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_no_msgs2stderr_alias_uses_client_channel(self):
|
||||||
|
"""--no-msgs2stderr is rsync's spelling of --stderr=client and now
|
||||||
|
forwards the client's diagnostics to the server too."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "client_msg_alias_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "client_msg_alias_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
with open(os.path.join(source, "plain.txt"), "wb") as f:
|
||||||
|
f.write(b"payload\n")
|
||||||
|
os.symlink("no-such-referent", os.path.join(source, "dangling"))
|
||||||
|
server, port = _start_captured_server()
|
||||||
|
try:
|
||||||
|
result, _ = run_client(source, dest, flags=["-L", "--no-msgs2stderr"],
|
||||||
|
port=port)
|
||||||
|
finally:
|
||||||
|
out, err = _stop_captured_server(server)
|
||||||
|
assert "symlink has no referent" in (out + err), (
|
||||||
|
f"--no-msgs2stderr did not route to the server: out={out!r} err={err!r}"
|
||||||
|
)
|
||||||
|
assert "symlink has no referent" not in (result.stderr or "")
|
||||||
|
|
||||||
|
|
||||||
class TestDryRun:
|
class TestDryRun:
|
||||||
def test_trust_sender_transfer_completes(self, shared_server):
|
def test_trust_sender_transfer_completes(self, shared_server):
|
||||||
"""--trust-sender is a receiver-local policy (never sent to the peer).
|
"""--trust-sender is a receiver-local policy (never sent to the peer).
|
||||||
@@ -2809,6 +2893,37 @@ class TestItemizeChanges:
|
|||||||
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-i", "--dry-run"])
|
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-i", "--dry-run"])
|
||||||
assert result.returncode == 0, f"dry-run -i failed: {result.stderr[:200]}"
|
assert result.returncode == 0, f"dry-run -i failed: {result.stderr[:200]}"
|
||||||
|
|
||||||
|
def test_chunk_serialization_probes_ancestor_dir_state(self, shared_server):
|
||||||
|
"""#314: --chunk-serialization + -i must probe ancestor directory state
|
||||||
|
so a pre-existing directory with a changed mtime itemizes as an
|
||||||
|
attribute change (`.d..t......`) instead of being rendered as created
|
||||||
|
(`cd+++++++++`)."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "itemize_chunk_serial_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "itemize_chunk_serial_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
subdir = os.path.join(source, "sub")
|
||||||
|
os.makedirs(subdir)
|
||||||
|
with open(os.path.join(subdir, "file.txt"), "wb") as fh:
|
||||||
|
fh.write(b"payload\n")
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest, flags=["--preserve"], port=shared_server.port)
|
||||||
|
assert result.returncode == 0, f"seed sync failed: {result.stderr[:200]}"
|
||||||
|
|
||||||
|
# Change only the source directory's mtime; its contents stay identical
|
||||||
|
# so only the directory's time attribute differs on the rerun.
|
||||||
|
os.utime(subdir, (1_000_000_000, 1_000_000_000))
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=["--preserve", "-i", "--chunk-serialization"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, f"chunk-serialization -i failed: {result.stderr[:200]}"
|
||||||
|
dir_lines = [line for line in result.stdout.splitlines() if line.endswith(" sub/")]
|
||||||
|
assert dir_lines == [".d..t...... sub/"], (
|
||||||
|
f"expected an attribute-change dir line, got {dir_lines!r}; "
|
||||||
|
f"full stdout={result.stdout!r}"
|
||||||
|
)
|
||||||
|
|
||||||
def test_changed_file_on_second_incremental_run_prints_exactly_one_line(self, shared_server):
|
def test_changed_file_on_second_incremental_run_prints_exactly_one_line(self, shared_server):
|
||||||
"""A changed file itemizes exactly once on an incremental rerun while
|
"""A changed file itemizes exactly once on an incremental rerun while
|
||||||
unchanged files print nothing (no double emission)."""
|
unchanged files print nothing (no double emission)."""
|
||||||
@@ -2974,12 +3089,12 @@ class TestDelayUpdates:
|
|||||||
"staging directory left behind after a successful delayed transfer"
|
"staging directory left behind after a successful delayed transfer"
|
||||||
|
|
||||||
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
|
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
|
||||||
def test_delay_updates_staging_name_collision_residual(self):
|
def test_delay_updates_staging_name_collision_preserved(self):
|
||||||
"""Documented residual (RSYNC_COMPAT.md `--delay-updates` row): FastSync
|
"""rsync parity (RSYNC_COMPAT.md `--delay-updates` row): the receiver
|
||||||
uses a fixed `.fastsync-stage` staging name and wipes a pre-existing tree
|
stages under a per-run unique name, so a genuine pre-existing
|
||||||
of that name at the start of a delayed run (crash-leftover cleanup),
|
destination entry named like the reserved staging prefix (`.fastsync-
|
||||||
even without `--delete`; rsync leaves a genuine destination entry of that
|
stage`) is never wiped -- even without `--delete`. rsync likewise
|
||||||
name untouched. Pins the divergence that keeps the row Divergent."""
|
leaves a real destination entry of its own temp name untouched."""
|
||||||
source = self._make_source("delay_collide_src")
|
source = self._make_source("delay_collide_src")
|
||||||
rdst = os.path.join(TEST_DATA_DIR, "delay_collide_rdst")
|
rdst = os.path.join(TEST_DATA_DIR, "delay_collide_rdst")
|
||||||
fdst = os.path.join(TEST_DATA_DIR, "delay_collide_fdst")
|
fdst = os.path.join(TEST_DATA_DIR, "delay_collide_fdst")
|
||||||
@@ -3005,8 +3120,11 @@ class TestDelayUpdates:
|
|||||||
result, _ = run_client(source, fdst, flags=["--delay-updates"],
|
result, _ = run_client(source, fdst, flags=["--delay-updates"],
|
||||||
port=server.port)
|
port=server.port)
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
assert not os.path.exists(os.path.join(fdst, self.STAGING)), \
|
assert _read_file(os.path.join(fdst, self.STAGING, "keepme.txt")) == b"genuine user data\n", \
|
||||||
"FastSync did not wipe the reserved staging name (residual changed)"
|
"FastSync destroyed a genuine destination entry named like the staging prefix"
|
||||||
|
# The per-run staging directory itself is removed after a clean run.
|
||||||
|
leftovers = [n for n in os.listdir(fdst) if n.startswith(self.STAGING + ".")]
|
||||||
|
assert leftovers == [], f"per-run staging directories left behind: {leftovers}"
|
||||||
|
|
||||||
@pytest.mark.parametrize("mt", [False, True])
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
def test_delay_updates_incremental_rerun_no_leftovers(self, shared_server, mt):
|
def test_delay_updates_incremental_rerun_no_leftovers(self, shared_server, mt):
|
||||||
@@ -3046,10 +3164,11 @@ class TestDelayUpdates:
|
|||||||
|
|
||||||
@pytest.mark.parametrize("mt", [False, True])
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
def test_delete_with_delay_updates(self, mt):
|
def test_delete_with_delay_updates(self, mt):
|
||||||
"""--delete runs before publication, so the delete walker must not treat
|
"""rsync parity: --delay-updates implies --delete-after, so every staged
|
||||||
the staging directory as a set of extras: a changed file must still be
|
update is published first and the genuine extras are removed only after
|
||||||
published after genuine extras are removed. Uses its own server started
|
that (the delete walker must never treat the staging directory as a set
|
||||||
with --allow-delete (the shared session server refuses deletion)."""
|
of extras). Uses its own server started with --allow-delete (the shared
|
||||||
|
session server refuses deletion)."""
|
||||||
source = os.path.join(TEST_DATA_DIR, "delay_delete_src")
|
source = os.path.join(TEST_DATA_DIR, "delay_delete_src")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "delay_delete_dst")
|
dest = os.path.join(TEST_DATA_DIR, "delay_delete_dst")
|
||||||
clean_dir(source)
|
clean_dir(source)
|
||||||
@@ -3079,6 +3198,65 @@ class TestDelayUpdates:
|
|||||||
assert not os.path.exists(os.path.join(received, "extra.txt")), \
|
assert not os.path.exists(os.path.join(received, "extra.txt")), \
|
||||||
"genuine extra file was not deleted"
|
"genuine extra file was not deleted"
|
||||||
assert not os.path.isdir(os.path.join(dest, self.STAGING))
|
assert not os.path.isdir(os.path.join(dest, self.STAGING))
|
||||||
|
assert [n for n in os.listdir(dest) if n.startswith(self.STAGING + ".")] == []
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
|
def test_delay_updates_delete_keeps_backup(self, mt):
|
||||||
|
"""The --backup/--delay-updates interplay: the old destination file is
|
||||||
|
moved aside at publication, and that backup survives the implied
|
||||||
|
--delete-after pass (rsync never treats a backup file as an extra)."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "delay_bak_del_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "delay_bak_del_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
||||||
|
fh.write(b"NEW")
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
os.makedirs(received, exist_ok=True)
|
||||||
|
with open(os.path.join(received, "f.txt"), "wb") as fh:
|
||||||
|
fh.write(b"OLD")
|
||||||
|
os.utime(os.path.join(received, "f.txt"), (1_500_000_000, 1_500_000_000))
|
||||||
|
# A pre-existing backup-looking extra must also be shielded.
|
||||||
|
with open(os.path.join(received, "stale.txt~"), "wb") as fh:
|
||||||
|
fh.write(b"stale backup")
|
||||||
|
with ServerManager() as server:
|
||||||
|
server.start(extra_args=["--allow-delete"])
|
||||||
|
flags = ["--delete", "--backup", "--delay-updates"] + (["--threads"] if mt else [])
|
||||||
|
result, _ = run_client(source, dest, flags=flags, port=server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
assert _read_file(os.path.join(received, "f.txt")) == b"NEW"
|
||||||
|
assert _read_file(os.path.join(received, "f.txt~")) == b"OLD", \
|
||||||
|
"the publication backup was removed by the delete-after pass"
|
||||||
|
assert os.path.exists(os.path.join(received, "stale.txt~")), \
|
||||||
|
"a pre-existing backup-suffixed entry was deleted"
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
|
def test_delay_updates_failed_run_leaves_no_staged_files(self, shared_server, mt):
|
||||||
|
"""A run that fails before publication installs nothing and removes the
|
||||||
|
per-run staging directory (no staged leftovers)."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "delay_fail_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "delay_fail_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
with open(os.path.join(source, "top.txt"), "wb") as fh:
|
||||||
|
fh.write(b"top\n")
|
||||||
|
os.makedirs(os.path.join(source, "sub"))
|
||||||
|
with open(os.path.join(source, "sub", "deep.txt"), "wb") as fh:
|
||||||
|
fh.write(b"deep\n")
|
||||||
|
# Plant a regular file where the "sub" directory must be created so the
|
||||||
|
# nested publish fails (the top-level file still publishes first).
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
os.makedirs(received)
|
||||||
|
with open(os.path.join(received, "sub"), "wb") as fh:
|
||||||
|
fh.write(b"blocker")
|
||||||
|
|
||||||
|
flags = ["--delay-updates"] + (["--threads"] if mt else [])
|
||||||
|
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
|
||||||
|
assert result.returncode != 0, "a blocked nested publish must fail the run"
|
||||||
|
assert not os.path.lexists(os.path.join(received, "sub", "deep.txt")), \
|
||||||
|
"a staged file appeared despite the failed run"
|
||||||
|
assert [n for n in os.listdir(dest) if n.startswith(self.STAGING + ".")] == [], \
|
||||||
|
"the per-run staging directory survived a failed run"
|
||||||
|
|
||||||
def test_delay_updates_rejects_reserved_backup_dir(self):
|
def test_delay_updates_rejects_reserved_backup_dir(self):
|
||||||
"""--backup-dir equal to the internal staging name must be rejected so
|
"""--backup-dir equal to the internal staging name must be rejected so
|
||||||
@@ -6599,7 +6777,7 @@ class TestExtendedAttributes:
|
|||||||
os.getxattr(os.path.join(received, "data.txt"), "user.foo")
|
os.getxattr(os.path.join(received, "data.txt"), "user.foo")
|
||||||
|
|
||||||
def test_reserved_fake_super_key_not_forwarded(self, shared_server):
|
def test_reserved_fake_super_key_not_forwarded(self, shared_server):
|
||||||
"""A source file that already carries the reserved user.fastsync.stat
|
"""A source file that already carries the reserved user.rsync.%stat
|
||||||
record must NOT have it planted on the receiver during a plain -X run
|
record must NOT have it planted on the receiver during a plain -X run
|
||||||
(it is receiver-only, so it cannot be spoofed for a later privileged
|
(it is receiver-only, so it cannot be spoofed for a later privileged
|
||||||
restore)."""
|
restore)."""
|
||||||
@@ -6609,7 +6787,7 @@ class TestExtendedAttributes:
|
|||||||
fh.write(b"reserved\n")
|
fh.write(b"reserved\n")
|
||||||
if not _xattr_supported(f):
|
if not _xattr_supported(f):
|
||||||
pytest.skip("filesystem does not support user xattrs")
|
pytest.skip("filesystem does not support user xattrs")
|
||||||
os.setxattr(f, "user.fastsync.stat", b"0:0:644:0:0")
|
os.setxattr(f, "user.rsync.%stat", b"100644 0,0 0:0")
|
||||||
# A normal user.* attr still travels alongside.
|
# A normal user.* attr still travels alongside.
|
||||||
os.setxattr(f, "user.keep", b"yes")
|
os.setxattr(f, "user.keep", b"yes")
|
||||||
|
|
||||||
@@ -6619,7 +6797,7 @@ class TestExtendedAttributes:
|
|||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
assert os.getxattr(os.path.join(received, "data.txt"), "user.keep") == b"yes"
|
assert os.getxattr(os.path.join(received, "data.txt"), "user.keep") == b"yes"
|
||||||
with pytest.raises(OSError):
|
with pytest.raises(OSError):
|
||||||
os.getxattr(os.path.join(received, "data.txt"), "user.fastsync.stat")
|
os.getxattr(os.path.join(received, "data.txt"), "user.rsync.%stat")
|
||||||
|
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_xattrs_multithreaded(self, shared_server):
|
def test_xattrs_multithreaded(self, shared_server):
|
||||||
@@ -6636,6 +6814,68 @@ class TestExtendedAttributes:
|
|||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
assert os.getxattr(os.path.join(received, "data.txt"), "user.k") == b"v"
|
assert os.getxattr(os.path.join(received, "data.txt"), "user.k") == b"v"
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_symlink_own_xattrs_never_referent(self, shared_server):
|
||||||
|
"""Protocol 2.29.0: a symlink's STATUS_SYMLINK frame carries a trailing
|
||||||
|
xattr block captured with llistxattr/lgetxattr (no follow) and applied
|
||||||
|
with lsetxattr on the link itself. Linux's VFS refuses to associate
|
||||||
|
xattrs with a symlink at all, so the portable guarantee asserted here is
|
||||||
|
the no-follow one: a referent that carries user.* must NOT have those
|
||||||
|
attributes appear on the destination symlink entry (the old
|
||||||
|
path-following capture would have copied the referent's attrs onto the
|
||||||
|
link). On a platform/filesystem that does support symlink xattrs the
|
||||||
|
full round-trip of the link's own attribute is asserted too."""
|
||||||
|
source, dest = self._source_and_dest("symlink_xattr")
|
||||||
|
target = os.path.join(source, "target.txt")
|
||||||
|
with open(target, "wb") as fh:
|
||||||
|
fh.write(b"referent payload\n")
|
||||||
|
if not _xattr_supported(target):
|
||||||
|
pytest.skip("filesystem does not support user xattrs")
|
||||||
|
os.setxattr(target, "user.referent-only", b"referent-value")
|
||||||
|
|
||||||
|
link = os.path.join(source, "link")
|
||||||
|
os.symlink("target.txt", link)
|
||||||
|
link_xattr_supported = False
|
||||||
|
try:
|
||||||
|
os.setxattr(link, "user.link-own", b"link-value", follow_symlinks=False)
|
||||||
|
link_xattr_supported = os.getxattr(
|
||||||
|
link, "user.link-own", follow_symlinks=False
|
||||||
|
) == b"link-value"
|
||||||
|
except (OSError, AttributeError, NotImplementedError):
|
||||||
|
link_xattr_supported = False
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest, flags=["-aX"], port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"-aX symlink sync failed: {(result.stderr or result.stdout)[:300]}"
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
dst_link = os.path.join(received, "link")
|
||||||
|
assert os.path.islink(dst_link), "destination link entry is not a symlink"
|
||||||
|
assert os.readlink(dst_link) == "target.txt"
|
||||||
|
|
||||||
|
# The no-follow guarantee. Checking only the link's own xattr list is
|
||||||
|
# vacuous on Linux (lsetxattr on a symlink always fails EPERM), so also
|
||||||
|
# prove the apply never followed the link: the destination REFERENT must
|
||||||
|
# keep its own user.* value untouched.
|
||||||
|
dst_target = os.path.join(received, "target.txt")
|
||||||
|
assert os.getxattr(dst_target, "user.referent-only") == b"referent-value", (
|
||||||
|
"the destination symlink apply followed the link and rewrote the "
|
||||||
|
"referent's xattr"
|
||||||
|
)
|
||||||
|
link_names = os.listxattr(dst_link, follow_symlinks=False)
|
||||||
|
assert "user.referent-only" not in link_names, (
|
||||||
|
"the destination symlink captured its REFERENT's xattr "
|
||||||
|
"(path-following capture bug)"
|
||||||
|
)
|
||||||
|
if sys.platform.startswith("linux"):
|
||||||
|
assert link_names == [], (
|
||||||
|
"Linux associates no xattrs with a symlink; the link entry must "
|
||||||
|
"carry none"
|
||||||
|
)
|
||||||
|
if link_xattr_supported:
|
||||||
|
assert os.getxattr(
|
||||||
|
dst_link, "user.link-own", follow_symlinks=False
|
||||||
|
) == b"link-value", "the symlink's own xattr did not round-trip"
|
||||||
|
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_acls_via_posix_acl_xattr(self, shared_server):
|
def test_acls_via_posix_acl_xattr(self, shared_server):
|
||||||
source, dest = self._source_and_dest("acl")
|
source, dest = self._source_and_dest("acl")
|
||||||
@@ -6708,10 +6948,17 @@ class TestExtendedAttributes:
|
|||||||
assert result.returncode == 0, \
|
assert result.returncode == 0, \
|
||||||
f"--fake-super sync failed: {(result.stderr or result.stdout)[:300]}"
|
f"--fake-super sync failed: {(result.stderr or result.stdout)[:300]}"
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
record = os.getxattr(os.path.join(received, "data.txt"), "user.fastsync.stat").decode()
|
record = os.getxattr(os.path.join(received, "data.txt"), "user.rsync.%stat").decode()
|
||||||
fields = record.split(":")
|
# rsync 3.4.1 grammar: "<octal st_mode> <rdev_major>,<rdev_minor> <uid>:<gid>".
|
||||||
assert len(fields) == 5
|
fields = record.split()
|
||||||
assert fields[0] == str(uid), f"reserved uid field {fields[0]} != source uid {uid}"
|
assert len(fields) == 3, f"unexpected rsync fake-super record {record!r}"
|
||||||
|
mode_field, rdev_field, owner_field = fields
|
||||||
|
assert rdev_field == "0,0", f"regular file rdev must be 0,0, got {rdev_field!r}"
|
||||||
|
assert int(mode_field, 8) & 0o170000 == stat.S_IFREG, (
|
||||||
|
f"recorded mode {mode_field!r} must carry S_IFREG"
|
||||||
|
)
|
||||||
|
assert owner_field.split(":")[0] == str(uid), \
|
||||||
|
f"recorded uid {owner_field!r} != source uid {uid}"
|
||||||
|
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_fake_super_records_resolved_chown_without_real_chown(self, shared_server):
|
def test_fake_super_records_resolved_chown_without_real_chown(self, shared_server):
|
||||||
@@ -6730,12 +6977,137 @@ class TestExtendedAttributes:
|
|||||||
assert result.returncode == 0, \
|
assert result.returncode == 0, \
|
||||||
f"--fake-super --chown sync failed: {(result.stderr or result.stdout)[:300]}"
|
f"--fake-super --chown sync failed: {(result.stderr or result.stdout)[:300]}"
|
||||||
dst = os.path.join(get_dest_received_dir(dest, source), "data.txt")
|
dst = os.path.join(get_dest_received_dir(dest, source), "data.txt")
|
||||||
record = os.getxattr(dst, "user.fastsync.stat").decode().split(":")
|
record = os.getxattr(dst, "user.rsync.%stat").decode().split()
|
||||||
assert record[0] == "33333", f"recorded owner {record[0]} != resolved 33333"
|
owner = record[2].split(":")
|
||||||
assert record[1] == "44444", f"recorded group {record[1]} != resolved 44444"
|
assert owner == ["33333", "44444"], (
|
||||||
|
f"recorded owner {record[2]!r} != resolved 33333:44444"
|
||||||
|
)
|
||||||
st = os.stat(dst)
|
st = os.stat(dst)
|
||||||
assert st.st_uid != 33333, "--fake-super must not real-chown the recorded owner"
|
assert st.st_uid != 33333, "--fake-super must not real-chown the recorded owner"
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_fake_super_rsync_interop(self, shared_server):
|
||||||
|
"""A fake-super tree written by FastSync is readable by rsync 3.4.1:
|
||||||
|
rsync reads the `user.rsync.%stat` record (mode/rdev/uid:gid) and, when
|
||||||
|
it re-emits a fake-super tree, reproduces the same record. This pins
|
||||||
|
the on-disk key and value grammar against the real tool."""
|
||||||
|
rsync = shutil.which("rsync")
|
||||||
|
if rsync is None:
|
||||||
|
pytest.skip("rsync not installed")
|
||||||
|
source, dest = self._source_and_dest("fakesuper_interop")
|
||||||
|
f = os.path.join(source, "data.txt")
|
||||||
|
with open(f, "wb") as fh:
|
||||||
|
fh.write(b"interop\n")
|
||||||
|
if not _xattr_supported(f):
|
||||||
|
pytest.skip("filesystem does not support user xattrs")
|
||||||
|
# rsync's fake-super receiver only writes a %stat% record when it has
|
||||||
|
# something to fake; a root-owned file with a matching root stat is
|
||||||
|
# a no-op. When privileged, record a non-root owner so the round-trip
|
||||||
|
# actually exercises the parser (non-root CI already has a non-zero uid).
|
||||||
|
if os.geteuid() == 0:
|
||||||
|
try:
|
||||||
|
os.chown(f, 12345, 12346)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
# A setuid bit exercises the full st_mode encoding; set it AFTER any
|
||||||
|
# chown (chown clears setuid/setgid), and note that neither tool installs
|
||||||
|
# it on the real destination file.
|
||||||
|
os.chmod(f, 0o4711)
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest, flags=["--fake-super"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"--fake-super sync failed: {(result.stderr or result.stdout)[:300]}"
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
rec = os.getxattr(os.path.join(received, "data.txt"), "user.rsync.%stat").decode()
|
||||||
|
rec_fields = rec.split()
|
||||||
|
assert len(rec_fields) == 3 and rec_fields[1] == "0,0", (
|
||||||
|
f"FastSync did not write rsync's stat grammar: {rec!r}"
|
||||||
|
)
|
||||||
|
assert int(rec_fields[0], 8) & 0o7777 == 0o4711, (
|
||||||
|
f"FastSync did not record the source mode in rsync's grammar: {rec!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
out = os.path.join(TEST_DATA_DIR, "fakesuper_interop_rsync")
|
||||||
|
clean_dir(out)
|
||||||
|
rs = subprocess.run([rsync, "-aX", "--fake-super",
|
||||||
|
received + "/", out + "/"],
|
||||||
|
capture_output=True, text=True, timeout=120)
|
||||||
|
assert rs.returncode == 0, (
|
||||||
|
f"rsync could not read FastSync's fake-super tree: {rs.stderr[:300]}"
|
||||||
|
)
|
||||||
|
out_rec = os.getxattr(os.path.join(out, "data.txt"), "user.rsync.%stat").decode()
|
||||||
|
assert out_rec == rec, (
|
||||||
|
"rsync re-emitted a different fake-super record; FastSync's grammar "
|
||||||
|
f"is not interoperable: ours={rec!r} rsync={out_rec!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_fake_super_directory_rsync_interop(self, shared_server):
|
||||||
|
"""#319: --fake-super fakes DIRECTORIES too. A recursive -a
|
||||||
|
--fake-super run must write rsync 3.4.1's `user.rsync.%stat` record on
|
||||||
|
the directory itself (full mode with S_IFDIR + special bits, rdev 0,0,
|
||||||
|
uid:gid), replay only the permission bits on disk, and real rsync must
|
||||||
|
read the tree and re-emit the identical record."""
|
||||||
|
rsync = shutil.which("rsync")
|
||||||
|
if rsync is None:
|
||||||
|
pytest.skip("rsync not installed")
|
||||||
|
source, dest = self._source_and_dest("fakesuper_dir_interop")
|
||||||
|
sub = os.path.join(source, "subdir")
|
||||||
|
os.makedirs(sub)
|
||||||
|
with open(os.path.join(sub, "f.txt"), "wb") as fh:
|
||||||
|
fh.write(b"dir interop\n")
|
||||||
|
if not _xattr_supported(sub):
|
||||||
|
pytest.skip("filesystem does not support user xattrs")
|
||||||
|
# A special bit (setgid) is exactly what a fake-super record exists to
|
||||||
|
# carry: rsync only re-emits a directory record when there is something
|
||||||
|
# it cannot represent on disk (a special bit, or a mode it would widen
|
||||||
|
# to keep the owner's rwx). Skip cleanly when the filesystem drops it.
|
||||||
|
os.chmod(sub, 0o2751)
|
||||||
|
if stat.S_IMODE(os.stat(sub).st_mode) & 0o7000 == 0:
|
||||||
|
pytest.skip("filesystem drops directory special bits")
|
||||||
|
uid = os.stat(sub).st_uid
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest, flags=["-a", "--fake-super"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"-a --fake-super dir sync failed: {(result.stderr or result.stdout)[:300]}"
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
dst_sub = os.path.join(received, "subdir")
|
||||||
|
assert os.path.isdir(dst_sub), "the directory entry was not transferred"
|
||||||
|
|
||||||
|
rec = os.getxattr(dst_sub, "user.rsync.%stat").decode()
|
||||||
|
fields = rec.split()
|
||||||
|
assert len(fields) == 3, f"unexpected rsync fake-super record {rec!r}"
|
||||||
|
mode_field, rdev_field, owner_field = fields
|
||||||
|
assert rdev_field == "0,0", f"directory rdev must be 0,0, got {rdev_field!r}"
|
||||||
|
assert int(mode_field, 8) & 0o170000 == stat.S_IFDIR, (
|
||||||
|
f"recorded mode {mode_field!r} must carry S_IFDIR"
|
||||||
|
)
|
||||||
|
assert int(mode_field, 8) & 0o7777 == 0o2751, (
|
||||||
|
f"recorded mode {mode_field!r} must carry the full source mode 02751"
|
||||||
|
)
|
||||||
|
assert owner_field.split(":")[0] == str(uid), \
|
||||||
|
f"recorded uid {owner_field!r} != source uid {uid}"
|
||||||
|
# Permission bits only on disk: the setgid bit stays in the record.
|
||||||
|
assert stat.S_IMODE(os.stat(dst_sub).st_mode) == 0o751, (
|
||||||
|
"the directory's special bits must not be installed on disk"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Real rsync reads FastSync's directory record and re-emits it verbatim.
|
||||||
|
out = os.path.join(TEST_DATA_DIR, "fakesuper_dir_interop_rsync")
|
||||||
|
clean_dir(out)
|
||||||
|
rs = subprocess.run([rsync, "-aX", "--fake-super", received + "/", out + "/"],
|
||||||
|
capture_output=True, text=True, timeout=120)
|
||||||
|
assert rs.returncode == 0, (
|
||||||
|
f"rsync could not read FastSync's fake-super directory tree: {rs.stderr[:300]}"
|
||||||
|
)
|
||||||
|
out_rec = os.getxattr(os.path.join(out, "subdir"), "user.rsync.%stat").decode()
|
||||||
|
assert out_rec == rec, (
|
||||||
|
"rsync re-emitted a different directory fake-super record; FastSync's "
|
||||||
|
f"grammar is not interoperable: ours={rec!r} rsync={out_rec!r}"
|
||||||
|
)
|
||||||
|
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_directory_xattrs_preserved(self, shared_server):
|
def test_directory_xattrs_preserved(self, shared_server):
|
||||||
"""#286.3: -aX must preserve user.* xattrs on DIRECTORIES, not just files."""
|
"""#286.3: -aX must preserve user.* xattrs on DIRECTORIES, not just files."""
|
||||||
@@ -7255,9 +7627,10 @@ class TestCopyAs:
|
|||||||
)
|
)
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
dst = os.path.join(received, "mixed.txt")
|
dst = os.path.join(received, "mixed.txt")
|
||||||
record = os.getxattr(dst, "user.fastsync.stat").decode().split(":")
|
record = os.getxattr(dst, "user.rsync.%stat").decode().split()
|
||||||
assert (record[0], record[1]) == ("65534", "65534"), (
|
owner = record[2].split(":")
|
||||||
f"fake-super must record the resolved copy-as ownership: {record[:2]}"
|
assert owner == ["65534", "65534"], (
|
||||||
|
f"fake-super must record the resolved copy-as ownership: {owner}"
|
||||||
)
|
)
|
||||||
st = os.lstat(dst)
|
st = os.lstat(dst)
|
||||||
assert (st.st_uid, st.st_gid) != (12345, 12346), (
|
assert (st.st_uid, st.st_gid) != (12345, 12346), (
|
||||||
|
|||||||
@@ -0,0 +1,222 @@
|
|||||||
|
"""#318: whole-file streaming above the receiver's 256 MiB ceiling.
|
||||||
|
|
||||||
|
The receiver's historical whole-file bound (``MAX_RECEIVE_WHOLE_FILE_SIZE``,
|
||||||
|
256 MiB) refused any single-file payload above it. The transfer engine now
|
||||||
|
streams such a payload (and the basis read/verify/hash) through a bounded buffer
|
||||||
|
and spools it to a temp file, so arbitrarily large single files transfer without
|
||||||
|
being materialized in memory.
|
||||||
|
|
||||||
|
To exercise the streaming path deterministically and quickly, these tests lower
|
||||||
|
the receiver bound with the test-only ``FASTSYNC_MAX_WHOLE_FILE_SIZE`` hook (it
|
||||||
|
can only lower, never raise, the protocol ceiling) and transfer a file a few
|
||||||
|
times larger than the lowered bound. A real >256 MiB transfer is covered once,
|
||||||
|
unmarked, so it runs in the full suite but not the fast PR gate.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(__file__))
|
||||||
|
from common import ( # noqa: E402
|
||||||
|
ServerManager,
|
||||||
|
TEST_DATA_DIR,
|
||||||
|
clean_dir,
|
||||||
|
get_dest_received_dir,
|
||||||
|
run_client,
|
||||||
|
)
|
||||||
|
|
||||||
|
LOW_BOUND = 1024 * 1024
|
||||||
|
FILE_SIZE = 3 * 1024 * 1024
|
||||||
|
OLD_MTIME = 1_500_000_000
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def small_bound_server():
|
||||||
|
"""A server whose whole-file streaming bound is 1 MiB."""
|
||||||
|
server = ServerManager()
|
||||||
|
server.start(extra_args=["--allow-super"],
|
||||||
|
env={"FASTSYNC_MAX_WHOLE_FILE_SIZE": str(LOW_BOUND)})
|
||||||
|
yield server
|
||||||
|
server.stop()
|
||||||
|
|
||||||
|
|
||||||
|
def _payload(n):
|
||||||
|
rng = random.Random(0xC0FFEE)
|
||||||
|
return rng.randbytes(n)
|
||||||
|
|
||||||
|
|
||||||
|
def _write(path, data, mtime=None):
|
||||||
|
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||||
|
with open(path, "wb") as fh:
|
||||||
|
fh.write(data)
|
||||||
|
if mtime is not None:
|
||||||
|
os.utime(path, (mtime, mtime))
|
||||||
|
|
||||||
|
|
||||||
|
def _resolved(dest, source, rel):
|
||||||
|
return os.path.join(get_dest_received_dir(dest, source), rel)
|
||||||
|
|
||||||
|
|
||||||
|
def _no_spool_leftovers(dest):
|
||||||
|
leftovers = []
|
||||||
|
for root, _dirs, files in os.walk(dest):
|
||||||
|
leftovers += [os.path.join(root, f) for f in files if ".fastsync-spool." in f]
|
||||||
|
return leftovers
|
||||||
|
|
||||||
|
|
||||||
|
class TestStreamedWholeFile:
|
||||||
|
"""A file above the (lowered) bound transfers correctly in every mode."""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"flags",
|
||||||
|
[
|
||||||
|
["-a"],
|
||||||
|
["-a", "--incremental"],
|
||||||
|
["-a", "-z"],
|
||||||
|
["-a", "--incremental", "-z"],
|
||||||
|
["-a", "--threads", "--incremental"],
|
||||||
|
["-a", "--inplace"],
|
||||||
|
["-a", "--partial"],
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_above_bound_transfers(self, small_bound_server, flags):
|
||||||
|
tag = "_".join(f.strip("-") for f in flags) or "default"
|
||||||
|
source = os.path.join(TEST_DATA_DIR, f"stream_src_{tag}")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, f"stream_dst_{tag}")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
data = _payload(FILE_SIZE)
|
||||||
|
_write(os.path.join(source, "big.bin"), data, OLD_MTIME)
|
||||||
|
if "--inplace" in flags:
|
||||||
|
# --inplace only matters when the destination already exists.
|
||||||
|
_write(_resolved(dest, source, "big.bin"), b"stale", OLD_MTIME)
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest, flags=flags, port=small_bound_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||||
|
|
||||||
|
got = os.path.join(get_dest_received_dir(dest, source), "big.bin")
|
||||||
|
assert os.path.exists(got), "streamed file was not written"
|
||||||
|
with open(got, "rb") as fh:
|
||||||
|
assert fh.read() == data, "streamed file content mismatch"
|
||||||
|
assert _no_spool_leftovers(dest) == [], "a spool temp file leaked"
|
||||||
|
|
||||||
|
|
||||||
|
class TestStreamedBasis:
|
||||||
|
"""A basis above the bound is streamed, not refused (compare/copy/link)."""
|
||||||
|
|
||||||
|
def _seed(self, dest, source, data):
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
_write(os.path.join(source, "big.bin"), data, OLD_MTIME)
|
||||||
|
# FastSync resolves a relative basis DIR against the destination and
|
||||||
|
# appends the transfer-relative name.
|
||||||
|
_write(os.path.join(dest, "basis", "big.bin"), data, OLD_MTIME)
|
||||||
|
|
||||||
|
def test_compare_dest_above_bound(self, small_bound_server):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "sbasis_cmp_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "sbasis_cmp_dst")
|
||||||
|
data = _payload(FILE_SIZE)
|
||||||
|
self._seed(dest, source, data)
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=["-a", "--compare-dest=basis", "--incremental"],
|
||||||
|
port=small_bound_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||||
|
# compare-dest never copies: an already-present destination stays sparse.
|
||||||
|
assert not os.path.exists(_resolved(dest, source, "big.bin"))
|
||||||
|
|
||||||
|
def test_copy_dest_above_bound(self, small_bound_server):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "sbasis_cpy_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "sbasis_cpy_dst")
|
||||||
|
data = _payload(FILE_SIZE)
|
||||||
|
self._seed(dest, source, data)
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=["-a", "--copy-dest=basis", "--incremental"],
|
||||||
|
port=small_bound_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||||
|
got = _resolved(dest, source, "big.bin")
|
||||||
|
assert os.path.exists(got)
|
||||||
|
with open(got, "rb") as fh:
|
||||||
|
assert fh.read() == data
|
||||||
|
assert os.stat(got).st_ino != os.stat(os.path.join(dest, "basis", "big.bin")).st_ino
|
||||||
|
assert _no_spool_leftovers(dest) == []
|
||||||
|
|
||||||
|
def test_link_dest_above_bound(self, small_bound_server):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "sbasis_lnk_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "sbasis_lnk_dst")
|
||||||
|
data = _payload(FILE_SIZE)
|
||||||
|
self._seed(dest, source, data)
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=["-a", "--link-dest=basis", "--incremental"],
|
||||||
|
port=small_bound_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||||
|
got = _resolved(dest, source, "big.bin")
|
||||||
|
assert os.path.exists(got)
|
||||||
|
with open(got, "rb") as fh:
|
||||||
|
assert fh.read() == data
|
||||||
|
assert os.stat(got).st_ino == os.stat(os.path.join(dest, "basis", "big.bin")).st_ino
|
||||||
|
|
||||||
|
|
||||||
|
class TestFuzzyAboveBound:
|
||||||
|
"""-y/--fuzzy reuses a basis above the bound by streaming its signature."""
|
||||||
|
|
||||||
|
def test_fuzzy_oversized_sibling(self, small_bound_server):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "sfuzzy_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "sfuzzy_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
base = _payload(FILE_SIZE)
|
||||||
|
sibling = bytearray(base)
|
||||||
|
sibling[FILE_SIZE // 2:FILE_SIZE // 2 + 4096] = bytes(
|
||||||
|
(b + 1) % 256 for b in sibling[FILE_SIZE // 2:FILE_SIZE // 2 + 4096])
|
||||||
|
_write(os.path.join(source, "report_v2.txt"), base)
|
||||||
|
_write(os.path.join(get_dest_received_dir(dest, source), "report_v1.txt"), bytes(sibling))
|
||||||
|
result, _ = run_client(
|
||||||
|
source, dest,
|
||||||
|
flags=["-a", "--incremental", "--delta", "--fuzzy", "--stats"],
|
||||||
|
port=small_bound_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||||
|
got = _resolved(dest, source, "report_v2.txt")
|
||||||
|
with open(got, "rb") as fh:
|
||||||
|
assert fh.read() == base, "fuzzy reconstruction mismatch"
|
||||||
|
assert _no_spool_leftovers(dest) == []
|
||||||
|
|
||||||
|
|
||||||
|
class TestRealLargeFile:
|
||||||
|
"""A real >256 MiB transfer, run only in the full (non-PR-gate) suite."""
|
||||||
|
|
||||||
|
def test_real_300mib_transfer(self, shared_server):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "real_large_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "real_large_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
clean_dir(dest)
|
||||||
|
n = 300 * 1024 * 1024
|
||||||
|
# Deterministic, compressible pattern written in bounded chunks.
|
||||||
|
chunk = bytes(range(256)) * 4096
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with open(os.path.join(source, "big.bin"), "wb") as fh:
|
||||||
|
written = 0
|
||||||
|
while written < n:
|
||||||
|
piece = chunk[: min(len(chunk), n - written)]
|
||||||
|
fh.write(piece)
|
||||||
|
digest.update(piece)
|
||||||
|
written += len(piece)
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest, flags=["-a", "--incremental"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
|
||||||
|
got = os.path.join(get_dest_received_dir(dest, source), "big.bin")
|
||||||
|
assert os.path.getsize(got) == n
|
||||||
|
got_digest = hashlib.sha256()
|
||||||
|
with open(got, "rb") as fh:
|
||||||
|
while True:
|
||||||
|
block = fh.read(1 << 20)
|
||||||
|
if not block:
|
||||||
|
break
|
||||||
|
got_digest.update(block)
|
||||||
|
assert got_digest.hexdigest() == digest.hexdigest()
|
||||||
|
shutil.rmtree(source, ignore_errors=True)
|
||||||
|
shutil.rmtree(dest, ignore_errors=True)
|
||||||
@@ -536,3 +536,78 @@ class TestFilterProtect:
|
|||||||
assert "extra.log" not in result.stdout, result.stdout
|
assert "extra.log" not in result.stdout, result.stdout
|
||||||
assert os.path.exists(os.path.join(received, "extra.log"))
|
assert os.path.exists(os.path.join(received, "extra.log"))
|
||||||
assert os.path.exists(os.path.join(received, "other.txt"))
|
assert os.path.exists(os.path.join(received, "other.txt"))
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_perdir_protect_dest_only_matches_rsync(self):
|
||||||
|
"""#315: a `P` rule inside a per-directory `.rsync-filter` is carried to
|
||||||
|
the receiver, so a destination-only extra matching ONLY that rule is
|
||||||
|
shielded under --delete. Both roots carry the same filter file (rsync's
|
||||||
|
receiver reads the destination one; FastSync carries the source's)."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "fpdp_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "fpdp_dst")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "fpdp_rdst")
|
||||||
|
clean_dir(source)
|
||||||
|
_write(os.path.join(source, "keep.txt"), b"keep\n")
|
||||||
|
_write(os.path.join(source, ".rsync-filter"), b"P extra.log\n")
|
||||||
|
clean_dir(rdst)
|
||||||
|
_write(os.path.join(rdst, ".rsync-filter"), b"P extra.log\n")
|
||||||
|
_write(os.path.join(rdst, "extra.log"), b"extra\n")
|
||||||
|
_write(os.path.join(rdst, "other.txt"), b"other\n")
|
||||||
|
|
||||||
|
rsync_result = _rsync(["-aF", "--delete", source + "/", rdst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
assert os.path.exists(os.path.join(rdst, "extra.log")), "rsync did not protect extra.log"
|
||||||
|
assert not os.path.exists(os.path.join(rdst, "other.txt"))
|
||||||
|
|
||||||
|
clean_dir(dest)
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
os.makedirs(received, exist_ok=True)
|
||||||
|
_write(os.path.join(received, ".rsync-filter"), b"P extra.log\n")
|
||||||
|
_write(os.path.join(received, "extra.log"), b"extra\n")
|
||||||
|
_write(os.path.join(received, "other.txt"), b"other\n")
|
||||||
|
with ServerManager() as server:
|
||||||
|
server.start(extra_args=["--allow-delete"])
|
||||||
|
result, _ = run_client(source, dest, flags=["-aF", "--delete"], port=server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
assert os.path.exists(os.path.join(received, "extra.log")), (
|
||||||
|
"FastSync must protect a destination-only per-directory P match like rsync")
|
||||||
|
assert not os.path.exists(os.path.join(received, "other.txt"))
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_perdir_protect_dry_run_enumeration(self, shared_server):
|
||||||
|
"""#315: the -n/--dry-run would-delete enumeration also honors the
|
||||||
|
carried per-directory rules, matching rsync's `*deleting` set: a
|
||||||
|
destination-only entry matching only a `.rsync-filter` P rule is not
|
||||||
|
reported (nor removed)."""
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "fpdp_nd_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "fpdp_nd_dst")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "fpdp_nd_rdst")
|
||||||
|
clean_dir(source)
|
||||||
|
_write(os.path.join(source, "keep.txt"), b"keep\n")
|
||||||
|
_write(os.path.join(source, ".rsync-filter"), b"P extra.log\n")
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
clean_dir(rdst)
|
||||||
|
clean_dir(received)
|
||||||
|
for root in (rdst, received):
|
||||||
|
_write(os.path.join(root, "keep.txt"), b"keep\n")
|
||||||
|
_write(os.path.join(root, ".rsync-filter"), b"P extra.log\n")
|
||||||
|
_write(os.path.join(root, "extra.log"), b"extra\n")
|
||||||
|
_write(os.path.join(root, "other.txt"), b"other\n")
|
||||||
|
|
||||||
|
rsync_result = _rsync(["-an", "-i", "-F", "--delete", source + "/", rdst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
rsync_del = sorted(l for l in rsync_result.stdout.splitlines()
|
||||||
|
if l.startswith("*deleting"))
|
||||||
|
assert rsync_del == ["*deleting other.txt"], f"unexpected rsync set: {rsync_del}"
|
||||||
|
|
||||||
|
with ServerManager() as server:
|
||||||
|
server.start(extra_args=["--allow-delete"])
|
||||||
|
result, _ = run_client(source, dest, flags=["-aF", "-n", "-i", "--delete"],
|
||||||
|
port=server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
fs_del = sorted(l for l in (result.stdout or "").splitlines()
|
||||||
|
if l.startswith("*deleting"))
|
||||||
|
assert fs_del == rsync_del, f"rsync={rsync_del}\nfastsync={fs_del}"
|
||||||
|
assert os.path.exists(os.path.join(received, "extra.log"))
|
||||||
|
assert os.path.exists(os.path.join(received, "other.txt"))
|
||||||
@@ -133,14 +133,14 @@ def _seed_protocol_source(source):
|
|||||||
class TestProtocol:
|
class TestProtocol:
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_protocol_current_version_accepted(self, shared_server):
|
def test_protocol_current_version_accepted(self, shared_server):
|
||||||
"""--protocol=2.28.0 (the current PROTOCOL_VERSION) is accepted and the
|
"""--protocol=2.30.0 (the current PROTOCOL_VERSION) is accepted and the
|
||||||
transfer completes normally."""
|
transfer completes normally."""
|
||||||
source = os.path.join(TEST_DATA_DIR, "proto_ok_src")
|
source = os.path.join(TEST_DATA_DIR, "proto_ok_src")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "proto_ok_dst")
|
dest = os.path.join(TEST_DATA_DIR, "proto_ok_dst")
|
||||||
shutil.rmtree(dest, ignore_errors=True)
|
shutil.rmtree(dest, ignore_errors=True)
|
||||||
os.makedirs(dest)
|
os.makedirs(dest)
|
||||||
_seed_protocol_source(source)
|
_seed_protocol_source(source)
|
||||||
result, _ = run_client(source, dest, flags=["--protocol=2.28.0"],
|
result, _ = run_client(source, dest, flags=["--protocol=2.30.0"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode == 0, \
|
assert result.returncode == 0, \
|
||||||
f"--protocol current run failed: {(result.stderr or result.stdout)[:400]}"
|
f"--protocol current run failed: {(result.stderr or result.stdout)[:400]}"
|
||||||
@@ -157,7 +157,7 @@ class TestProtocol:
|
|||||||
shutil.rmtree(dest, ignore_errors=True)
|
shutil.rmtree(dest, ignore_errors=True)
|
||||||
os.makedirs(dest)
|
os.makedirs(dest)
|
||||||
_seed_protocol_source(source)
|
_seed_protocol_source(source)
|
||||||
for bad in ("2.27.0", "2.26.0", "2.25.0", "2.24.0", "2.23.0", "2.22.0", "2.21.0", "2.20.0",
|
for bad in ("2.29.0", "2.28.0", "2.27.0", "2.26.0", "2.25.0", "2.24.0", "2.23.0", "2.22.0", "2.21.0", "2.20.0",
|
||||||
"2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0", "216", "31"):
|
"2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0", "216", "31"):
|
||||||
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
|
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
|
|||||||
Loaded 100 of 117 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user