Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 TapTap
|
ef76c9034d |
@@ -49,47 +49,6 @@ jobs:
|
||||
if: github.event_name == 'push'
|
||||
run: python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv" --durations=25 --tb=short -q
|
||||
|
||||
# Differential rsync-parity gate: runs real rsync 3.4.1 and FastSync over the
|
||||
# same corpora and compares destinations + normalized output. The fast subset
|
||||
# guards the ✅ surface on every PR; the full set (with FASTSYNC_PARITY_STRICT
|
||||
# so a fixed caveat must be removed from the allowlist) burns the documented
|
||||
# ⚠️/❌ residuals down on push. See tests/integration/README.md.
|
||||
parity-fast:
|
||||
runs-on: ubuntu-latest
|
||||
container: gitea.tap-tap.win/taptap/fastsync-ci:v11
|
||||
needs: lint
|
||||
if: github.event_name == 'pull_request'
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Configure
|
||||
run: cmake -B build -S . -DSTRICT_WARNINGS=ON
|
||||
|
||||
- name: Build
|
||||
run: cmake --build build -j$(nproc)
|
||||
|
||||
- name: Differential parity (fast subset)
|
||||
run: python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci -q
|
||||
|
||||
parity-full:
|
||||
runs-on: ubuntu-latest
|
||||
container: gitea.tap-tap.win/taptap/fastsync-ci:v11
|
||||
needs: lint
|
||||
if: github.event_name == 'push'
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Configure
|
||||
run: cmake -B build -S . -DSTRICT_WARNINGS=ON
|
||||
|
||||
- name: Build
|
||||
run: cmake --build build -j$(nproc)
|
||||
|
||||
- name: Differential parity (full set)
|
||||
run: FASTSYNC_PARITY_STRICT=1 python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity -q
|
||||
|
||||
sanitizers:
|
||||
runs-on: ubuntu-latest
|
||||
container: gitea.tap-tap.win/taptap/fastsync-ci:v11
|
||||
|
||||
@@ -12,12 +12,3 @@ build_docker2/
|
||||
# Test/run artifacts
|
||||
root/
|
||||
test_partial_install_tmp/
|
||||
|
||||
# Editor/tooling + test caches/artifacts
|
||||
.pytest_cache/
|
||||
*.gcda
|
||||
*.gcno
|
||||
*.gcov
|
||||
di/
|
||||
test_data-manual/
|
||||
*.log
|
||||
|
||||
@@ -16,7 +16,7 @@ Ask the user or determine from context:
|
||||
- **Minor** (x.Y.0) — new features, backward compatible
|
||||
- **Patch** (x.y.Z) — bug fixes, no protocol changes
|
||||
|
||||
Current version: `PROTOCOL_VERSION "2.29.0"` in `src/shared/config.h`
|
||||
Current version: `PROTOCOL_VERSION "2.26.0"` in `src/shared/config.h`
|
||||
|
||||
### Step 2: Check Protocol Version
|
||||
|
||||
|
||||
@@ -4,9 +4,9 @@ FastSync is a high-performance file synchronization system written in C11. It su
|
||||
|
||||
## Dependency installation
|
||||
|
||||
**CI rule:** never add `apt-get install` / `pip install` steps to CI workflows — use the custom Docker image instead. The image is built from the repo-root `Dockerfile` and is the same image CI uses: `gitea.tap-tap.win/taptap/fastsync-ci:v11`. It contains the full toolchain: gcc/g++, CMake, libzstd-dev, zlib1g-dev, liblz4-dev, libxxhash-dev, libssl-dev, make, git, cppcheck, clang-format, python3 + pytest + pytest-xdist, openssh-client, Node.js, plus `rsync` 3.4.1 (with zstd/xxhash/lz4), `acl` and `attr` (setfacl/getfacl, setfattr/getfattr) for drop-in parity tests. (CMake hard-requires zstd, zlib, and lz4; xxHash is fetched via `FetchContent`.)
|
||||
**CI rule:** never add `apt-get install` / `pip install` steps to CI workflows — use the custom Docker image instead. The image is built from the repo-root `Dockerfile` and is the same image CI uses: `gitea.tap-tap.win/taptap/fastsync-ci:v11`. It contains the full toolchain: gcc/g++, CMake, libzstd-dev, libssl-dev, make, git, cppcheck, clang-format, python3 + pytest + pytest-xdist, openssh-client, Node.js, plus `rsync` 3.4.1 (with zstd/xxhash/lz4), `acl` and `attr` (setfacl/getfacl, setfattr/getfattr) for drop-in parity tests.
|
||||
|
||||
**Host rule:** for local development, use `nix-shell` (see `README.md`) which provides zstd, zlib, lz4, OpenSSL, CMake, and gcc. The Docker image can also be used locally for CI parity.
|
||||
**Host rule:** for local development, use `nix-shell` (see `README.md`) which provides zstd, OpenSSL, CMake, and gcc. The Docker image can also be used locally for CI parity.
|
||||
|
||||
```bash
|
||||
# Use the prebuilt CI image directly (faster, guaranteed CI parity)
|
||||
@@ -38,12 +38,11 @@ If a dependency is missing from the CI image, add it to the `Dockerfile` (and re
|
||||
When configuring for CI parity, use:
|
||||
```bash
|
||||
cmake -B build -S . -DSTRICT_WARNINGS=ON # -Wextra -Wpedantic -Werror
|
||||
cmake -B build -S . -DSANITIZER=address # AddressSanitizer (ASan); in the CI matrix
|
||||
cmake -B build -S . -DSANITIZER=undefined # UndefinedBehaviorSanitizer (UBSan); in the CI matrix
|
||||
cmake -B build -S . -DSANITIZER=thread # ThreadSanitizer (TSan); local-only, NOT in CI
|
||||
cmake -B build -S . -DSANITIZER=address # AddressSanitizer (ASan)
|
||||
cmake -B build -S . -DSANITIZER=thread # ThreadSanitizer (TSan)
|
||||
```
|
||||
|
||||
The CI workflow (`.gitea/workflows/ci.yaml`) runs lint (clang-format, cppcheck), then a **fast PR gate** — build + unit + a representative subset of integration tests marked `@pytest.mark.ci`, parallelized with pytest-xdist (`-n 4 --dist=load`). The full coverage jobs (full integration suite as `-m "not setpriv"`, the `address`+`undefined` sanitizer matrix, fuzz, coverage, valgrind) run **only on push to `dev`/`main`**; pull requests skip them to keep PR CI under ~3 minutes. TSan is not part of the CI matrix and is a local-only configuration. The `setpriv`-marked privilege tests (four decorated functions, collecting to eight instances because two are parametrized) are excluded from CI via a marker because their result depends on the runner/container uid and host mount permissions.
|
||||
The CI workflow (`.gitea/workflows/ci.yaml`) runs lint (clang-format, cppcheck), then a **fast PR gate** — build + unit + a representative subset of integration tests marked `@pytest.mark.ci`, parallelized with pytest-xdist (`-n 4 --dist=load`). The full coverage jobs (full integration suite as `-m "not setpriv"`, sanitizer, fuzz, coverage, valgrind) run **only on push to `dev`/`main`**; pull requests skip them to keep PR CI under ~3 minutes. The two `setpriv` privilege tests are excluded from CI via a marker because their result depends on the runner/container uid and host mount permissions.
|
||||
|
||||
## Build
|
||||
|
||||
@@ -57,21 +56,6 @@ cmake -B build -S . && cmake --build build -j$(nproc)
|
||||
./build/tests # unit tests
|
||||
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv" # full integration suite (CI excludes env-dependent privilege tests)
|
||||
python3 -m pytest tests/integration/ -n 4 --dist=load -m ci # PR-gate subset only
|
||||
|
||||
# Differential rsync-parity gate (real rsync 3.4.1 vs FastSync)
|
||||
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci # fast PR subset
|
||||
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity # full set
|
||||
```
|
||||
|
||||
See `tests/integration/README.md` for the differential parity gate and its
|
||||
`parity_caveats.py` allowlist (the residual burn-down mechanism).
|
||||
|
||||
Unit tests under valgrind must set `FASTSYNC_UNDER_VALGRIND=1` (CI does): the
|
||||
tests use it to skip fork-based tests, because valgrind 3.22 does not expose
|
||||
`vgpreload` in the guest's `/proc/self/maps`.
|
||||
|
||||
```bash
|
||||
FASTSYNC_UNDER_VALGRIND=1 valgrind --leak-check=full --show-leak-kinds=definite --error-exitcode=1 ./build/tests
|
||||
```
|
||||
|
||||
## CI Workflow — Waiting for Results
|
||||
@@ -106,7 +90,7 @@ Two main branches: `dev` (integration) and `main` (stable releases).
|
||||
|
||||
### Rules
|
||||
- **All PRs target `dev`** — never target `main` directly
|
||||
- **`dev` is intended to be the default branch** in Gitea repo settings — verify in the repo settings, since this clone's `origin/HEAD` still points at `main`
|
||||
- **`dev` is the default branch** in Gitea repo settings
|
||||
- **`main` is protected** — only merged from `dev` via PR with 2 approvals + full CI pass
|
||||
- **Feature/bug branches** branch from `dev`, PR back to `dev`
|
||||
- **`dev` → `main` merges** happen on-demand or weekly, requiring full CI + review
|
||||
|
||||
-315
@@ -4,323 +4,8 @@ All notable changes to FastSync are documented here. Versions match
|
||||
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
|
||||
run the same version because the handshake is strict.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
Wire backlog cycle (protocol 2.29.0 → 2.30.0; config-frame layout unchanged).
|
||||
|
||||
- **`--stderr=client` client-message channel (#313):** the client now accepts
|
||||
`--stderr=client` (and maps the deprecated `--no-msgs2stderr` to it), routing
|
||||
its own diagnostics over the new bounded `STATUS_CLIENT_MSG` client->server
|
||||
frame instead of writing them locally; the server writes each received
|
||||
message to its stderr (respecting the server log destination). `errors`/`all`
|
||||
behavior is unchanged.
|
||||
- **Receiver partial failures exit 23 (#320):** a per-entry receiver failure
|
||||
that does not abort the stream (e.g. an unprivileged `--devices` mknod) now
|
||||
sends the terminal `STATUS_PARTIAL`; the client exits 23 like rsync and, under
|
||||
`--remove-source-files`, still removes the sources it successfully
|
||||
transferred. A clean run stays 0 and a fatal/connection error stays non-23.
|
||||
- **Directory/symlink destination-state itemize (#314):** when `report_dest_info`
|
||||
is negotiated (now also for `--progress`), the receiver answers `STATUS_MKDIR`
|
||||
and `STATUS_SYMLINK` with the entry's pre-transfer destination snapshot
|
||||
(existence, type, perms/owner/group/time, and whether an existing symlink's
|
||||
target already matches), and the sender probes every ancestor directory before
|
||||
the receiver creates it implicitly. A re-run over an unchanged tree no longer
|
||||
emits per-directory `cd+++++++++` or unchanged-symlink lines, a changed
|
||||
directory renders rsync's `.d..t......`, and a changed symlink renders
|
||||
`cLc........` / `.L..t......`. Directory/symlink time comparison uses whole
|
||||
seconds (rsync's `cmp_time`). The `STATUS_MKDIR` body gains a probe flag and
|
||||
the `STATUS_DEST_INFO` record gains a symlink-target-match field; the
|
||||
config-frame layout is unchanged. Differential-tested against rsync 3.4.1.
|
||||
- **`--stats` deleted per-type breakdown (#316):** `STATUS_STATS` gains
|
||||
`deleted_reg/dir/link/special`, tallied by the delete observers and rendered
|
||||
as rsync's `Number of deleted files: X (reg: A, dir: B, link: C, special: D)`.
|
||||
Differential-tested against rsync 3.4.1 for a mixed-type `--delete` tree.
|
||||
|
||||
## [2.29.0] - 2026-09-23
|
||||
|
||||
The rsync-parity cycle 2.29 (no wire change; `PROTOCOL_VERSION` stays 2.28.0).
|
||||
`RSYNC_COMPAT.md` moves from **116 ✅ / 14 ⚠️ / 27 ❌** to
|
||||
**120 ✅ / 10 ⚠️ / 27 ❌** of 157 rows.
|
||||
|
||||
An audit cycle follows on the same wire version (`PROTOCOL_VERSION` stays
|
||||
2.28.0): a security-and-correctness pass over the parity-2.29 baseline, plus a
|
||||
set of audit follow-ups (filter merge modifiers, the `--inplace`/`--partial-dir`
|
||||
conflict, credential-file hardening, and small leak/log/test fixes). It fixes
|
||||
a `--temp-dir` symlink escape, gates client-controlled special permission bits,
|
||||
corrects `--partial-dir`/`--bwlimit`/`-z` behavior, handles unsupported filter
|
||||
modifiers, and tightens client and wire validation. The only parity
|
||||
reclassification is `--filter=RULE` moving ✅ → ⚠️, because its merge-only
|
||||
`e`/`n`/`w`/`-` modifiers are now accepted and consumed but their semantics
|
||||
remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ /
|
||||
11 ⚠️ / 27 ❌** of 157 rows. The affected rows' notes and the summary tally in
|
||||
`RSYNC_COMPAT.md` were updated. A following triage-fix cycle (see **Triage
|
||||
fixes** below) moves `-F` and `-i` to ⚠️, for a final **117 ✅ / 13 ⚠️ / 27 ❌**
|
||||
of 157 rows.
|
||||
|
||||
A no-wire parity burn-down cycle follows on 2.28.0: it accepts
|
||||
`--inc-recursive`/`--no-inc-recursive` as inert no-ops, accepts an absolute
|
||||
`--temp-dir` that canonicalizes inside the receive root, closes the
|
||||
`--delete-before` phase-0 divergence (both the single-threaded and `--threads`
|
||||
data passes replay the pre-scan list), makes `--fake-super` interoperable with
|
||||
rsync's `user.rsync.%stat` key/grammar (regular files and char/block devices
|
||||
faked as regular files), turns a failed device `mknod` into a continuing
|
||||
per-entry failure, and accepts a practical subset of rsync's `rsyncd.conf`
|
||||
grammar (modules are read-only by default, and accepted-but-unenforced
|
||||
access-control keys emit a startup warning). The matrix moves to **119 ✅ /
|
||||
14 ⚠️ / 24 ❌** of 157 rows.
|
||||
|
||||
A structural cycle then lands a transport I/O vtable over TCP/TLS (fixing the
|
||||
TLS-multithreaded sendfile path and making the per-thread SSL resolution
|
||||
explicit) and bumps the wire to **2.29.0**: the `STATUS_SYMLINK` frame grows an
|
||||
optional symlink-xattr block (captured no-follow with `llistxattr`/`lgetxattr`,
|
||||
applied no-follow with `lsetxattr`). Because the handshake is strict, 2.28.0 and
|
||||
2.29.0 peers are incompatible. Note: Linux refuses to associate xattrs with a
|
||||
symlink at all, so the symlink-xattr block is a no-op on Linux and is carried
|
||||
for correctness on platforms/filesystems that do support it; the config-frame
|
||||
layout is unchanged (golden length still 886).
|
||||
|
||||
### Changed
|
||||
|
||||
- **rsync-exact traversal order.** The sequential scanner now walks each
|
||||
directory's entries in rsync 3.4.1's flist order (non-directories ascending,
|
||||
then directories ascending, depth-first), so `--info=name`, the
|
||||
`--delete-during`/`--delete-delay`/`-n` would-delete order and the partial
|
||||
`--max-delete` survivor set match rsync byte-for-byte. `--threads` has no
|
||||
rsync analogue and stays unordered.
|
||||
- **Delete timing.** The complete `--delete-during`/`--delete-delay`
|
||||
per-directory plan set is transmitted before the first data frame, so a
|
||||
mid-transfer abort has already removed every planned extra like rsync's
|
||||
generator; `-d/--dirs` uses per-directory plans (shielded untraversed
|
||||
subdirectories) instead of the end-of-transfer commit. `-n`, `--delete`,
|
||||
`--del`/`--delete-during` and `--delete-delay` are now ✅ Parity.
|
||||
- **Basis directories.** A relative `--compare-dest`/`--copy-dest`/`--link-dest`
|
||||
DIR resolves against the destination directory with the transfer-relative
|
||||
name appended, exactly like rsync 3.4.1.
|
||||
- **`-y`/`--fuzzy`.** The candidate search no longer inherits the ordinary delta
|
||||
engine's 16 KiB minimum or 10× size-ratio bound, so an oversized or
|
||||
sub-16-KiB sibling is reused exactly as rsync reuses it.
|
||||
- `--info=mount` prints rsync's mount-point skip line (repeated `-xx` drops the
|
||||
mount-point directory); `--info=stats` enables the `--stats` block; `-x` is
|
||||
repeatable. `--stats` counts traversed directories for the `Number of files`
|
||||
breakdown under a plain `-r` scan. `--debug` emits real output for
|
||||
`flist`/`del`/`hash`/`deltasum`/`recv`/`filter`/`send`.
|
||||
|
||||
### Known residuals
|
||||
|
||||
- `--progress` and `--info` still need a receiver→sender event channel for the
|
||||
root `./` line, ancestor-directory suppression, receiver-side `skip`/`backup`
|
||||
wording, and symlink/empty-directory quick-checks.
|
||||
- `--delete-before`'s phase-0 late-file divergence remains (rsync's pre-scan
|
||||
fixes the file list before the data pass).
|
||||
- A whole-file sender that cannot stream its codec (lz4's one-shot block
|
||||
format) or a `--append`/delta source above the bound still buffers; the
|
||||
default zstd/zlib and the uncompressed paths stream (see #318).
|
||||
- `--stats` byte totals and `--msgs2stderr` stay documented divergences.
|
||||
|
||||
### Security
|
||||
|
||||
- **`--temp-dir` symlink escape fixed.** The receiver's scratch directory was
|
||||
opened with a bare `open()`, so a symlink planted under the receive root could
|
||||
redirect receiver scratch files outside the authorized root. The opened
|
||||
directory is now judged by the real path of its fd (`/proc/self/fd` via
|
||||
`realpath`) and an escaping target is refused (`EACCES`, logged); an in-root
|
||||
link to another filesystem (the `EXDEV` fallback case) still works.
|
||||
- **Client-controlled special bits masked when super-user activities are not
|
||||
permitted.** Setuid/setgid/sticky bits (`--perms`, `--chmod`, the symlink and
|
||||
special-node paths, and deferred directory modes) are now stripped when the
|
||||
connection forbids super activities (`--no-super`, a non-opted daemon module,
|
||||
a privileged listener without `--allow-super`); exact rsync semantics are
|
||||
preserved wherever super activities are permitted.
|
||||
- **Daemon umask no longer forced to `0`.** `daemonize()` now sets the
|
||||
conventional `022`, so implied parent directories created without `-p` are no
|
||||
longer world-writable `0777`.
|
||||
- **Daemon modules are read-only by default.** A `--daemon` module is now
|
||||
served read-only unless it sets `read only = no` (or rsync's `write only =
|
||||
yes`), matching rsync: a real `rsyncd.conf` that omits `read only` is no
|
||||
longer silently writable. A global `read only` still sets the default for
|
||||
later modules, and an explicit module value wins. This is a behavior change
|
||||
for existing FastSync-native configs that relied on the old writable default;
|
||||
add `read only = no` to keep them writable. An rsync `write only = yes` is
|
||||
mapped to writability (FastSync is push-only, so a module can never be read
|
||||
from the network).
|
||||
- **Accepted-but-unenforced rsync security keys now warn at startup.** The
|
||||
rsync keys FastSync recognizes but does not implement — `secrets file`,
|
||||
`refuse options`, `exclude`/`include`/`filter`, `max size`/`min size`,
|
||||
`pre-xfer exec`/`post-xfer exec`, `incoming chmod`/`outgoing chmod`,
|
||||
`name converter`, `use chroot`, `uid`/`gid`, and the rest of the
|
||||
access-control set — load for migration compatibility but now emit a
|
||||
`WARN` naming the key (and module) so an operator does not believe the
|
||||
restriction is enforced. `auth users`/`secrets file` stay fail-closed: a
|
||||
module declaring `auth users` still requires a FastSync credential store.
|
||||
- **Credentials and signal handling hardened.** Secret files are opened with
|
||||
`O_NOFOLLOW|O_NONBLOCK` (while allowing fd-backed store paths and bound-waiting
|
||||
a FIFO read for ~3 s so a slow process substitution works but a connected-but-
|
||||
silent FIFO cannot hang), and signal handlers use `sigaction` with
|
||||
async-signal-safe bodies.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`-z` on 100–256 MiB files.** The decompressor's internal ceiling was 100 MiB
|
||||
while the receiver advertises and the sender compresses whole files up to
|
||||
`MAX_RECEIVE_WHOLE_FILE_SIZE` (256 MiB), so `-z` on a 100–256 MiB regular file
|
||||
failed with `Declared decompressed size exceeds 104857600 bytes`. The ceiling
|
||||
is now defined in terms of the protocol whole-file bound (still an
|
||||
allocation-clamped bomb guard).
|
||||
- **`--bwlimit` now paces `--sendfile`.** The plaintext-TCP `--sendfile` fast
|
||||
path bypassed the protocol's token bucket, so the limit was ignored there. It
|
||||
now throttles through the same per-session leaky bucket as the TLS path.
|
||||
- **`--partial-dir` implies `--partial`.** Matching rsync 3.4.1 (which sets
|
||||
`keep_partial` after option parsing), `--partial-dir=DIR` alone retains an
|
||||
interrupted transfer's partial and wins over an explicit `--no-partial`;
|
||||
`--inplace` still bypasses the partial machinery, and combining `--inplace`
|
||||
with `--partial-dir` is now rejected up front with rsync's message
|
||||
(`--inplace cannot be used with --partial-dir`).
|
||||
- **Filter modifiers handled.** The `x` xattr-name modifier is rejected with a
|
||||
clear error everywhere. The merge-only `e`/`n`/`w` and `-` modifiers are now
|
||||
accepted and consumed on `merge`/`dir-merge` rules (so they no longer leak
|
||||
into the merge filename) while still being rejected on non-merge rules,
|
||||
matching rsync; their semantics remain unimplemented (accepted-but-ignored).
|
||||
Glued patterns (`-newfile`, `-e2e`) and mixed tokens (`H,!secret`) keep their
|
||||
historical parsing.
|
||||
- **Credential-file reads hardened.** Secret files (`--password-file`/
|
||||
`--early-input`/`--hash-credentials` input) are opened with `O_NOFOLLOW`, so a
|
||||
symlinked credential path now fails closed (`ELOOP`) instead of being followed
|
||||
before the owner/mode gate; literal fd-backed paths (`/dev/fd/<digits>`,
|
||||
`/proc/self/fd/<digits>`) are exempt so process substitution still works. A
|
||||
FIFO/process-substitution read now waits under a bounded ~3 s deadline for its
|
||||
writer, so a slow producer works while a connected-but-silent FIFO fails
|
||||
instead of hanging.
|
||||
- **Miscellaneous correctness fixes:** `--filter` rule count is checked
|
||||
client-side against `MAX_FILTER_RULES` before any network I/O (the receiver
|
||||
still re-checks the expanded count); unknown wire `Status` values are rejected
|
||||
as protocol errors; a mutex leak on an init-failure path, an `errno` read
|
||||
after `free()` in deferred delete application, `log_perror` misuse for
|
||||
non-`errno` conditions, and a `NULL` `server_host`/`ssh_destination`
|
||||
allocation path were fixed (the `config_create` failure now releases through
|
||||
`config_delete`); the decompression-limit log now prints the effective bound
|
||||
rather than the compile-time ceiling; the daemon umask and root test fixtures
|
||||
were hardened; `SSL_read` length is clamped and `sendfile` `poll()` retries on
|
||||
`EINTR`.
|
||||
|
||||
### Refactored / Docs
|
||||
|
||||
- Dropped dead `filter_rules_apply` and dead `--old-args` plumbing, unified
|
||||
`set_error`, deduplicated `path_is_within` and shared constants, and added
|
||||
printf format attributes (fixing format mismatches). `RSYNC_COMPAT.md`,
|
||||
`CHANGELOG.md` and `HANDOFF.md` were updated for the audit cycle; the
|
||||
`RSYNC_COMPAT.md` summary tally was corrected to match the rows.
|
||||
|
||||
### Triage fixes
|
||||
|
||||
- **`--dirs` directory xattrs applied inline.** A `-d/--dirs` transfer now
|
||||
applies captured directory `-X`/`-A` xattrs fd-relative on the directory entry
|
||||
instead of dropping them, so directory xattrs survive the non-recursive path
|
||||
(`src/shared/file_save.c`, `tests/test_xattr.c`).
|
||||
- **Directory/root itemize and `--out-format` lines.** `-i`/`--itemize-changes`
|
||||
and `--out-format` now emit the transfer-root `./` line and per-directory
|
||||
`cd...`/`.d..t...` lines, rendered by the shared itemize code. This matches
|
||||
rsync's fresh-transfer output; because the root line is unconditional and an
|
||||
incremental re-run may itemize directories/symlinks that rsync's quick-check
|
||||
leaves silent, `-i` is now a ⚠️ Caveat row.
|
||||
- **FROM name globs for identity maps.** `--usermap`/`--groupmap` `FROM` tokens
|
||||
now accept `*`/`?`/`[...]` globs, expanded sender-side against the passwd/group
|
||||
database and collapsed into bounded numeric ranges (`MAX_IDENTITY_MAP`),
|
||||
matching rsync.
|
||||
- **Transport fallback unit tests.** Added unit coverage for the TCP/TLS
|
||||
transport fallback paths (`tests/test_transport_tcp.c`,
|
||||
`tests/test_transport_tls.c`).
|
||||
- **Docs corrections.** `RSYNC_COMPAT.md`/`README.md` corrected stale parity
|
||||
claims for issues #286–#297: the `-F` and `-i` reclassifications, the
|
||||
`--munge-links` direction, the accepted checksum/compression name sets,
|
||||
`--bwlimit` parsing, `--stop-at` grammar, `--trust-sender`, symlink xattrs, and
|
||||
the native/non-interoperable batch and credential notes. The summary tally is
|
||||
now **117 ✅ / 13 ⚠️ / 27 ❌** of 157 rows.
|
||||
|
||||
## [2.28.0] - 2026-09-20
|
||||
|
||||
The rsync-parity cycle. `PROTOCOL_VERSION` moves `2.26.0 → 2.27.0 → 2.28.0`;
|
||||
client and server must run the same version (the handshake is strict). See
|
||||
`RSYNC_COMPAT.md` for the per-option matrix, now **116 ✅ / 14 ⚠️ / 27 ❌** of
|
||||
157 rows.
|
||||
|
||||
### Added
|
||||
|
||||
- **Differential rsync 3.4.1 parity gate** (`tests/integration/
|
||||
test_differential_parity.py`, `parity_harness.py`, `parity_caveats.py`): runs
|
||||
real `rsync` and FastSync over generated corpora and diffs the destination
|
||||
tree, normalized stdout and exit code. A fast subset runs on pull requests and
|
||||
the full strict set on push; the residual allowlist is empty.
|
||||
- FastSync-only long option **`--verify-basis`**: require a
|
||||
`--compare-dest`/`--copy-dest`/`--link-dest` hit to match the source by
|
||||
whole-file digest instead of trusting the size+mtime quick-check.
|
||||
- FastSync-only long option **`--delete-commit`** (implies `--delete`): the old
|
||||
atomic late whole-tree commit.
|
||||
- `--bwlimit` now parses rsync's units exactly and paces like rsync's leaky
|
||||
bucket; `--ignore-errors` reproduces rsync's skip-unreadable-subdir and
|
||||
IO-error-suppressed deletion (exit 23).
|
||||
- `--info=name/flist/del/remove/nonreg/progress` emit rsync's line format,
|
||||
including real-run `deleting`/`*deleting` lines carried by a new
|
||||
`report_deletes` wire bool.
|
||||
- Receiver-observed `--stats` counters: `Number of created files` now carries
|
||||
rsync's `(reg/dir/link/special)` breakdown and `Literal data` is exact for a
|
||||
delta transfer (extended `STATUS_STATS`).
|
||||
- `--progress` uses an opt-in paths-only pre-count so the `to-chk` denominator
|
||||
counts every entry like rsync, and emits per-directory/symlink/special names.
|
||||
- Receiver-side `protect`/`risk` filter engine (new bounded filter-rule wire
|
||||
block): `--filter='P ...'` now shields a destination-only entry like rsync.
|
||||
- `auto` for `--compress-choice`/`--checksum-choice` honors
|
||||
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST`, and per-codec compression-level
|
||||
defaults match rsync.
|
||||
- Empty source directories are recreated recursively; `-R --no-implied-dirs
|
||||
--files-from` places listed files under missing implied parents; `--iconv`
|
||||
matches rsync's push direction; `--delete-delay` reports actual removals and
|
||||
recursively removes a refilled deferred directory.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`--delete` now defaults to delete-during (rsync `--del`) timing.** With no
|
||||
explicit timing flag, a plain `--delete` removes each directory's extras as
|
||||
that directory is processed instead of committing one whole-tree deletion only
|
||||
after the entire transfer succeeds. This matches rsync, frees destination
|
||||
space progressively, and avoids the whole-old+new-tree peak that could
|
||||
`ENOSPC` a tight destination. The client maps the default onto the existing
|
||||
`delete_during` wire boolean, so `PROTOCOL_VERSION` stays `2.28.0`.
|
||||
- Basis directories (`--compare-dest`/`--copy-dest`/`--link-dest`) now default
|
||||
to rsync's metadata quick-check (equal size and mtime; `--size-only` drops the
|
||||
mtime leg) instead of FastSync's historical always-verify content hash.
|
||||
`--copy-dest` re-applies the source attributes, and basis materialization is
|
||||
streamed so the 256 MiB whole-file cap no longer applies to a basis hit.
|
||||
- The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag:
|
||||
the one-shot per-run config block (protected prefixes, size-pruned mirrors,
|
||||
`--delete-missing-args` exact paths) is now always transmitted first on a
|
||||
config-only carrier (`apply=false`), fixing a latent bug where a
|
||||
`--delete-missing-args` run whose `--files-from` list synchronized no directory
|
||||
never sent its exact deletions.
|
||||
|
||||
### Notes
|
||||
|
||||
- `--delete`/`--delete-during` remain caveats for the mid-transfer abort
|
||||
boundary (rsync's generator removes all planned extras ahead of its throttled
|
||||
sender; FastSync removes only reached directories — final trees agree).
|
||||
`--delete-before`, `--progress`, `--stats`, `--fuzzy` and the basis rows keep
|
||||
their documented residuals in `RSYNC_COMPAT.md`; `--filter` and
|
||||
`--delete-excluded` are now parity, including protection of a destination-only
|
||||
excluded entry under default `--delete`.
|
||||
|
||||
### Migration
|
||||
|
||||
- Scripts that relied on plain `--delete` deleting nothing until the transfer
|
||||
fully succeeded must pass **`--delete-commit`** (or `--delete-after`) to keep
|
||||
that behavior. Plain `--delete` now removes reached directories' extras during
|
||||
the transfer, exactly like rsync's default; on a completed run the final tree
|
||||
is unchanged.
|
||||
- Deployments that relied on FastSync's stricter basis verification should pass
|
||||
**`--verify-basis`**; the default now trusts the size+mtime quick-check like
|
||||
rsync.
|
||||
|
||||
## [2.26.0] - 2026-09-17
|
||||
|
||||
|
||||
### Added
|
||||
|
||||
- **Parity-completion wave.** Closed the remaining rsync-parity gaps against
|
||||
|
||||
+3
-14
@@ -1,6 +1,6 @@
|
||||
cmake_minimum_required(VERSION 3.22)
|
||||
|
||||
project(FastFileTransfer VERSION 2.30.0)
|
||||
project(FastFileTransfer VERSION 2.26.0)
|
||||
|
||||
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
|
||||
set(CMAKE_C_STANDARD 11)
|
||||
@@ -26,9 +26,9 @@ elseif(NOT SANITIZER STREQUAL "none")
|
||||
endif()
|
||||
|
||||
# --- Strict warnings option ---
|
||||
option(STRICT_WARNINGS "Enable strict warnings (Wextra, Wpedantic, Wformat-signedness, Werror)" OFF)
|
||||
option(STRICT_WARNINGS "Enable strict warnings (Wextra, Wpedantic, Werror)" OFF)
|
||||
if(STRICT_WARNINGS)
|
||||
add_compile_options(-Wextra -Wpedantic -Wformat-signedness -Werror)
|
||||
add_compile_options(-Wextra -Wpedantic -Werror)
|
||||
endif()
|
||||
|
||||
# --- Coverage option ---
|
||||
@@ -99,21 +99,17 @@ set(SHARED_SRCS
|
||||
src/shared/daemon_limits.c
|
||||
src/shared/data.c
|
||||
src/shared/delay_updates.c
|
||||
src/shared/delete.c
|
||||
src/shared/delete_commit.c
|
||||
src/shared/delete_plan.c
|
||||
src/shared/delta.c
|
||||
src/shared/file.c
|
||||
src/shared/file_list.c
|
||||
src/shared/file_receive.c
|
||||
src/shared/file_save.c
|
||||
src/shared/file_send.c
|
||||
src/shared/file_store.c
|
||||
src/shared/filter.c
|
||||
src/shared/format.c
|
||||
src/shared/hardlink.c
|
||||
src/shared/identity.c
|
||||
src/shared/incremental_check.c
|
||||
src/shared/log.c
|
||||
src/shared/metadata.c
|
||||
src/shared/motd.c
|
||||
@@ -140,14 +136,9 @@ set(SERVER_MAIN_SRCS src/server/server.c)
|
||||
# Client implementation (no main): everything except the CLI entry point.
|
||||
set(CLIENT_CORE_SRCS
|
||||
src/client/change_list.c
|
||||
src/client/client_manifest.c
|
||||
src/client/client_report.c
|
||||
src/client/client_scan.c
|
||||
src/client/client_send.c
|
||||
src/client/client_validation.c
|
||||
src/client/scanner.c
|
||||
src/client/scanner_filter.c
|
||||
src/client/scanner_parallel.c
|
||||
src/client/usage.c
|
||||
)
|
||||
set(CLIENT_MAIN_SRCS src/client/client_cli.c)
|
||||
@@ -230,12 +221,10 @@ set(TEST_SRCS
|
||||
tests/test_daemon_limits.c
|
||||
tests/test_data.c
|
||||
tests/test_delay_updates.c
|
||||
tests/test_delete_plan.c
|
||||
tests/test_delta.c
|
||||
tests/test_file.c
|
||||
tests/test_file_list.c
|
||||
tests/test_file_sendfile.c
|
||||
tests/test_filter.c
|
||||
tests/test_format.c
|
||||
tests/test_fuzz_smoke.c
|
||||
tests/test_glob.c
|
||||
|
||||
+22
-217
@@ -1,31 +1,16 @@
|
||||
# FastSync — Session Handoff (2026-09-21)
|
||||
# FastSync — Session Handoff (2026-09-17)
|
||||
|
||||
## Current status
|
||||
- **Release `v2.28.0`** is tagged and merged to `main`: tag `v2.28.0` points at
|
||||
`ee6523a`, and the PR #304 merge commit `b4d54504` is on `main`.
|
||||
- **`dev` is at `0fbb9de`** — the merge of parity cycle 2.29 (PR #305). The old
|
||||
`558782d` (incremental-check flake fix) is an ancestor.
|
||||
- **`PROTOCOL_VERSION` = `"2.28.0"`** (`src/shared/config.h`); CMake
|
||||
`project(FastFileTransfer VERSION 2.28.0)`.
|
||||
- **Parity cycle 2.29 is merged to `dev`** (PR #305), no wire change. It closed
|
||||
the scanner-order, delete-timing, relative-basis and fuzzy-eligibility
|
||||
residuals and improved the `--info`/`--stats`/`--debug` partials. Parity
|
||||
matrix: **120 ✅ / 10 ⚠️ / 27 ❌ = 157**. Remaining ⚠️ rows: `--info`,
|
||||
`--debug`, `--msgs2stderr`, `--stats`, `--progress`, `--delete-before`, the
|
||||
three basis-dir options, and `-y`/`--fuzzy`.
|
||||
- **Audit cycle complete on branch `fix/audit-cycle`** (branched from `dev` @
|
||||
`0fbb9de`), integration PR to `dev` pending. No wire change
|
||||
(`PROTOCOL_VERSION` stays 2.28.0). It lands the receiver/client security and
|
||||
correctness fixes — `--temp-dir` symlink-escape confinement, special-bit
|
||||
masking under a super-off policy, daemon `umask(022)`, the `-z` decompression
|
||||
ceiling raised to the 256 MiB whole-file bound, `--bwlimit` pacing the
|
||||
plaintext `--sendfile` path, `--partial-dir` implying `--partial`, rejection
|
||||
of unsupported filter modifiers (`x`/`e`/`n`/`w`), client-side
|
||||
`MAX_FILTER_RULES` enforcement, unknown wire `Status` rejection, and the
|
||||
accompanying refactors/docs. The parity matrix is unchanged at
|
||||
**120 ✅ / 10 ⚠️ / 27 ❌ = 157**; this docs pass (worktree `fix/audit-docs2`)
|
||||
corrects the `RSYNC_COMPAT.md` summary tally to match the rows.
|
||||
|
||||
- **Release `v2.21.0`** tagged (`919a729`, "Release v2.21.0"); full CI green
|
||||
(run 552: lint, build-and-test, ASan, UBSan, fuzz-build, coverage, valgrind).
|
||||
`dev` has the release commit plus later doc-only merges (a README refresh and
|
||||
this handoff).
|
||||
- **Release PR #284 (`dev` -> `main`)** open, CI green (run 553).
|
||||
`main` is protected: it needs review/approval to merge.
|
||||
https://gitea.tap-tap.win/TapTap/FastSync/pulls/284
|
||||
- **`PROTOCOL_VERSION` = `"2.26.0"`** (`src/shared/config.h`); CMake
|
||||
`project(FastFileTransfer VERSION 2.26.0)`.
|
||||
- Working tree clean; no wave worktrees remain.
|
||||
|
||||
## What landed this session
|
||||
1. **Wave 8 (refactors):** Config X-macro wire table; single-owner `authorized_root`;
|
||||
@@ -54,199 +39,19 @@
|
||||
docs state push-only / remote-source unsupported.
|
||||
5. **Preserve-attribute split (protocol 2.22.0)** landed on `feat/preserve-attr-split`: per-attribute `-p/-t/-o/-g` + `--no-*` negations, `-a` = `-rlptgoD`, and the 2.21.0 → 2.22.0 wire bump.
|
||||
6. **Rsync-parity wave (protocol 2.23.0)** on `feat/rsync-parity`: rsync short options/clustering/attached values (`-r`/`-b`/`-L`/`-B`, `-av`, `-aAX`, `-B1000`, `-essh`, `-MOPT`), `-c` checksum quick-check, `--checksum-choice`/`--compress-choice` validation and seed randomization, rsync timeout/max-alloc defaults, temp-dir confinement + `EXDEV` fallback, ownership/mapping parity (numeric-ids modifier, map ranges/`*`/empty-FROM, `--chown`+map conflicts, fake-super resolved-owner record), verbatim symlink storage with rsync `--safe-links`/`--munge-links`, socket recreation under `--specials`, `--chmod` 3.4.1 semantics, and delete scoping + `--max-delete` partial/exit-25. Wire: appended delete-manifest synchronized-directory section and `STATUS_DELETE_LIMIT`.
|
||||
7. **Parity-completion wave (protocol 2.24.0 → 2.26.0)** on `feat/parity-completion`: per-directory delete plans (`STATUS_DELETE_PLAN`) for `--delete-during`/`--delete-delay`; receiver `STATUS_STATS` counters feeding `--stats`/`--progress` and `--out-format %b/%c/%C`, plus `-n --delete` lines; `lz4`/`zlib`/`zlibx` compression and `md4`/`sha1`/`none` checksums with `auto` negotiation (default `xxh128`/`zstd`); general `-R`/`--no-implied-dirs`/`-d`; the full filter grammar (`merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` + modifiers) and corrected `-F`/`-FF`; receiver-side `--chown`/map TO-name resolution; absolute basis dirs + `--link-dest` relink; receiver-side `--ignore-existing` short-circuit; `--preallocate` over `--sparse` via `fallocate(2)`; `--iconv=.`/`-`/`--no-iconv`; lone `-h` help; aliases `--ignore-non-existing`/`--protect-args`/`--msgs2stderr`; and the full `--info`/`--debug` vocabulary. `RSYNC_COMPAT.md` reclassifies the matrix to 106 ✅ / 27 ⚠️ / 23 ❌; the later rsync-parity-stats pass (`fix/parity-stats`) moves it to 107 ✅ / 25 ⚠️ / 24 ❌ (see item 8).
|
||||
8. **rsync-parity-stats pass** on `fix/parity-stats` (no wire change, `PROTOCOL_VERSION` stays `2.26.0`): `--delete-delay` now reports only entries it actually removes, while the `--max-delete` budget is charged at plan/snapshot time (`planned`, via `defer_add`) to bound the deferred list (a refilled deferred directory that survives `ENOTEMPTY` is not reported but still consumes budget); `--stats` gained the `(reg/dir/link/special)` `Number of files` breakdown and now counts only regular files actually stored for `Number of regular files transferred`/transferred size/literal data (up-to-date re-runs report 0); `Total file size` includes symlink target lengths; `--progress` prints the leading `./` root line and counts it in `to-chk` so a single-file transfer matches rsync; and `%C` uses the selected transfer checksum with `checksum_digest_file` supporting md4/sha1/none, byte-identical to rsync for every algorithm. `--out-format` reclassified ❌ (`%b`/delta-`%c` are protocol-specific). Differential + regression tests added; full suite + ASan + clang-format + cppcheck clean.
|
||||
9. **Option-parity wave (protocol 2.26.0 → 2.27.0, on `fix/parity-options`):**
|
||||
`--bwlimit` now ports rsync 3.4.1's units/quantization and paces like its
|
||||
leaky bucket; `--ignore-errors` reproduces rsync's default (an I/O error
|
||||
skips deletion unless the flag is set; the readable tree still transfers and
|
||||
the run exits 23) across every delete timing; the `--info` categories with a
|
||||
FastSync event (`name`/`flist`/`del`/`remove`/`nonreg`/`progress`) emit
|
||||
rsync's line format, with real-run `deleting`/`*deleting` lines carried over
|
||||
the new trailing config bool `report_deletes` (golden wire updated by
|
||||
`tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
|
||||
over daemon/TCP (no argv channel in FastSync's binary config handshake;
|
||||
rsync-daemon differential pins the rsync behavior) and receiver-side
|
||||
`protect`/`risk` re-derivation for destination-only entries (would need a
|
||||
receiver filter engine; differential pins the divergence — **reversed by
|
||||
track 4a below**, which adds that engine). The options pass
|
||||
stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
|
||||
holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
|
||||
`--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
|
||||
|
||||
10. **rsync-parity-fs pass** on `fix/parity-fs` (no wire change of its own; integrated
|
||||
on top of the 2.27.0 options wave): recursive transfers now recreate empty source directories (and
|
||||
`-m/--prune-empty-dirs` still suppresses them), a directory entry replaces a
|
||||
blocking destination regular file, and `-R --no-implied-dirs --files-from`
|
||||
places a listed file under a missing implied parent with default attributes
|
||||
instead of refusing (real rsync 3.4.1 parity, differential-tested). `--iconv`
|
||||
now reproduces rsync's push direction (destination charset = the spec's REMOTE
|
||||
half; a server `--iconv` overrides), and `-T/--temp-dir` relative semantics are
|
||||
confirmed identical while the absolute-path confinement is a deliberate
|
||||
divergence. The basis-dir options, `--delay-updates` and `--dry-run` were
|
||||
reclassified to ❌ after a differential test reproduced each exact residual
|
||||
(basis content verification, fixed staging-name collision, and dry-run
|
||||
would-delete over-report). `--fuzzy` was also reclassified to ❌ (deterministic
|
||||
heuristic with a 10× size window, not rsync's matcher), but its residual is the
|
||||
candidate-selection heuristic itself: the final tree is byte-exact by design, so
|
||||
it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
|
||||
differential. (Track 5b later found the name heuristic is rsync's own and moved
|
||||
the row ❌ → ⚠️, leaving only the narrower delta size window; see entry 15.) The parity-review pass then moved `--delete-delay` to ⚠️ (the
|
||||
plan-time `--max-delete` charge and non-recursive deferred removal differ from
|
||||
rsync when a snapshotted entry fails removal). Differential-gate allowlist
|
||||
entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
|
||||
integrated stats+options+fs branch stands at **111 ✅ / 13 ⚠️ / 33 ❌ = 157**;
|
||||
full suite + ASan + clang-format + cppcheck clean.
|
||||
|
||||
11. **No-wire parity track 1** on `feat/parity-2.28` (no protocol change):
|
||||
`-n --delete` now sends the same filter-excluded + size-pruned protected
|
||||
prefixes and synchronized-directory scope as a real run (dry-run would-delete
|
||||
matches rsync for source-derived protections; the destination-only exclude
|
||||
residual was later closed by track 4a, readdir ordering remains);
|
||||
`--delete-delay` now charges
|
||||
`--max-delete` on actual removals and re-scans a queued directory at commit
|
||||
to remove content created after the plan, with an independent deferred-list
|
||||
cap (only partial-delete ordering remains); and `--info=name2` emits `NAME is
|
||||
uptodate` plus the leading `./` root name line for `--info=name` (only the
|
||||
root-line trigger condition and receiver-side `skip` wording remain). Matrix
|
||||
now **111 ✅ / 14 ⚠️ / 32 ❌ = 157**; differential + unit tests added in
|
||||
`test_features.py`, `test_option_parity.py`, the unit test
|
||||
`tests/test_delete_plan.c`,
|
||||
`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`.
|
||||
12. **No-wire parity track 2b** on `feat/parity-2.28` (no protocol change):
|
||||
`--progress`/`-P`/`--info=progress` (when not `--quiet`) now run an opt-in
|
||||
paths-only metadata pre-count (no file reads/hashing) that supplies rsync's
|
||||
full file-list total for the `to-chk` denominator and the directory names,
|
||||
and emits per-directory/symlink/special name lines, in both the sequential
|
||||
and `--threads` paths. `--delete-during`/`--delete-delay` reuse their
|
||||
keep-set pre-scan instead of a second walk; non-progress runs are
|
||||
unaffected. Differential tests (`progress`/`progress_threads` over a new
|
||||
`multidir` corpus) match rsync's name set and `to-chk` denominator on a
|
||||
fresh transfer, and the single-file byte-identical test still passes;
|
||||
emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream)
|
||||
plus re-run over-naming (unconditional `./`, ancestor dirs named with a
|
||||
transferred child, and no quick-check for symlinks/empty dirs) remain the
|
||||
caveats, so the row stays ⚠️ and the matrix is unchanged at
|
||||
**111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
|
||||
|
||||
13. **Wire parity track 4a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||
`2.28.0`): the receiver now has a delete-time filter engine. The sender
|
||||
compiles its root-level selection rules exactly as the scanner does
|
||||
(`filter_base_build`) and streams them as one bounded, self-describing
|
||||
config-frame block (action, sides, anchored, dir-only, negate, owner,
|
||||
pattern; bounded rule count and pattern bytes, unknown action/sides is a
|
||||
protocol error). The receiver reconstructs `protect_rules` and applies them
|
||||
first-match-wins to each extraneous destination path in every delete timing
|
||||
(the whole-tree commit walker, the `--delete-during`/`--delete-delay`
|
||||
per-directory plans, and the `-n` would-delete enumeration), so a
|
||||
`P *.log` rule protects a destination-only `extra.log` like rsync (with
|
||||
`risk` cancelling); the sender-derived protected-prefix behavior is
|
||||
preserved when no rules are sent and `--delete-excluded` semantics are
|
||||
unchanged. Per-directory merge (`:`/`.`) receiver re-derivation remains the
|
||||
residual. `TestFilterProtect` (real + dry-run) plus differential cases
|
||||
`filter_protect`, `filter_protect_during`, `filter_protect_delay` added and
|
||||
the `--filter=RULE` row moves ❌ → ✅: matrix now
|
||||
**115 ✅ / 11 ⚠️ / 31 ❌ = 157**; unit tests, the three named integration
|
||||
files, clang-format and cppcheck clean.
|
||||
|
||||
14. **Wire parity track 5a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||
`2.28.0` by project decision): the three basis-dir options now default to
|
||||
rsync's metadata quick-check (equal size + equal mtime, or size alone under
|
||||
`--size-only`; `-I` disables matching) instead of FastSync's historical
|
||||
xxHash64 content equality, so a same-size/different-content basis is trusted
|
||||
exactly as rsync trusts it. A new FastSync-only, long-only `--verify-basis`
|
||||
flag restores the strict whole-file content equality; its bool is appended to
|
||||
the basis block of the config frame (golden wire frame 882 → 886 bytes).
|
||||
`--verify-basis` streams the confined basis descriptor to hash it, and a
|
||||
basis hit is no longer capped at the 256 MiB whole-file payload bound:
|
||||
`--copy-dest` streams the basis through a bounded buffer and `--link-dest`'s
|
||||
copy fallback streams from the basis, so an over-limit hit materializes (a
|
||||
basis MISS still falls back to the normal transfer and keeps its own bound).
|
||||
A `--copy-dest` hit re-applies the SOURCE attributes (the sender transmits
|
||||
the source metadata with the basis check frame), matching rsync's
|
||||
"copy then fix attributes"; a `--link-dest` success keeps the shared inode's
|
||||
attributes (writing through it would mutate the basis). Differential cases
|
||||
`copy_dest` and `verify_basis` added; `test_basis_dir_size_only_content_residual`
|
||||
converted to a passing parity assertion; `TestBasisDestDirs` updated for the
|
||||
new default + `--verify-basis`; unit tests cover the quick-check/verify
|
||||
decision and the same-size/different-content handshake. The
|
||||
`--compare-dest`/`--copy-dest`/`--link-dest` rows move ❌ → ⚠️ (relative-DIR
|
||||
resolution base and over-limit MISS refusal): matrix now
|
||||
**116 ✅ / 13 ⚠️ / 28 ❌ = 157**.
|
||||
|
||||
15. **No-wire parity track 5b** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||
`2.28.0` by project decision): `-y`/`--fuzzy` reclassified ❌ → ⚠️. A probe
|
||||
against real rsync 3.4.1 (pinned `-B8192`, repeated-content 64 KiB corpus)
|
||||
showed the name heuristic is already rsync's (`util1.c fuzzy_distance` /
|
||||
`find_filename_suffix` + the exact size+mtime pass) and the output is always
|
||||
byte-exact; the only residual is candidate ELIGIBILITY, because FastSync's
|
||||
`delta_should_attempt` gate caps the size ratio at 10× and requires both
|
||||
files ≥ 16 KiB while rsync will reuse a basis from 0.25× to 10000× and below
|
||||
16 KiB. The choice is observable only as `--stats` bandwidth counters. Added
|
||||
differential case `fuzzy_basis` (same-suffix sibling, one name edit,
|
||||
identical content, block size pinned) asserting tree **and** normalized
|
||||
`--stats` parity where the choices coincide, plus `TestFuzzy` pinning the
|
||||
window boundary on both sides (>10× and <16 KiB siblings declined by
|
||||
FastSync while rsync uses them, both trees byte-identical). Matrix now
|
||||
**116 ✅ / 14 ⚠️ / 27 ❌ = 157**.
|
||||
|
||||
16. **Lockstep delete-default track 6** on `feat/parity-2.28` (`PROTOCOL_VERSION`
|
||||
stays `2.28.0`): plain `--delete` now defaults to rsync's delete-during
|
||||
(`--del`) timing, normalized on the client onto the existing `delete_during`
|
||||
wire bool. The old late whole-tree commit is opt-in via `--delete-after` or
|
||||
the FastSync-only long `--delete-commit` (identical `delete_after` timing).
|
||||
`-d/--dirs` still falls back to the end commit, `--delay-updates` still
|
||||
deletes before publication, and `--files-from`/`-R` scope is unchanged. The
|
||||
`STATUS_DELETE_PLAN` frame gained a one-int `apply` flag so the per-run
|
||||
config block (including `--delete-missing-args` exact paths) is always
|
||||
transmitted, on a config-only carrier when the scope allows no directory
|
||||
plan — fixing a latent bug with a file-only `--files-from` list. Differential
|
||||
cases `delete`/`delete_commit`/`filter_protect_after` plus the extended
|
||||
`test_delete_timing_parity.py` (plain `--delete` mid-abort removes reached
|
||||
extras, `--delete-commit` defers) pass; full `-m "not setpriv"` suite,
|
||||
clang-format and cppcheck clean. Matrix unchanged at
|
||||
**116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay
|
||||
⚠️ for the abort boundary; `--delete-after` stays ✅).
|
||||
|
||||
17. **Audit cycle** on `fix/audit-cycle` (from `dev` @ `0fbb9de`;
|
||||
`PROTOCOL_VERSION` stays `2.28.0`): a security/correctness pass over the
|
||||
parity-2.29 baseline. It raises the decompression ceiling to the 256 MiB
|
||||
protocol whole-file bound (`-z` on 100–256 MiB files now works), paces the
|
||||
plaintext-TCP `--sendfile` path with `--bwlimit`, confines the `--temp-dir`
|
||||
scratch dir by the fd's real path (symlink escape refused), masks
|
||||
client-controlled setuid/setgid/sticky bits when super activities are not
|
||||
permitted, sets the daemon umask to `022`, makes `--partial-dir` imply
|
||||
`--partial`, rejects the unsupported filter modifiers (`x`/`e`/`n`/`w`),
|
||||
enforces `MAX_FILTER_RULES` client-side, rejects unknown wire `Status`
|
||||
values, and hardens credentials/signal handling (with the accompanying
|
||||
refactors and docs). No row changes classification, so the matrix stays
|
||||
**120 ✅ / 10 ⚠️ / 27 ❌ = 157**. This docs pass is on `fix/audit-docs2`.
|
||||
7. **Parity-completion wave (protocol 2.24.0 → 2.26.0)** on `feat/parity-completion`: per-directory delete plans (`STATUS_DELETE_PLAN`) for `--delete-during`/`--delete-delay`; receiver `STATUS_STATS` counters feeding `--stats`/`--progress` and `--out-format %b/%c/%C`, plus `-n --delete` lines; `lz4`/`zlib`/`zlibx` compression and `md4`/`sha1`/`none` checksums with `auto` negotiation (default `xxh128`/`zstd`); general `-R`/`--no-implied-dirs`/`-d`; the full filter grammar (`merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` + modifiers) and corrected `-F`/`-FF`; receiver-side `--chown`/map TO-name resolution; absolute basis dirs + `--link-dest` relink; receiver-side `--ignore-existing` short-circuit; `--preallocate` over `--sparse` via `fallocate(2)`; `--iconv=.`/`-`/`--no-iconv`; lone `-h` help; aliases `--ignore-non-existing`/`--protect-args`/`--msgs2stderr`; and the full `--info`/`--debug` vocabulary. `RSYNC_COMPAT.md` reclassifies the matrix to 106 ✅ / 27 ⚠️ / 23 ❌.
|
||||
|
||||
## Next steps
|
||||
1. **Open and merge the audit-cycle PR** (`fix/audit-cycle`, including this
|
||||
`fix/audit-docs2` docs pass) into `dev` once reviewed. `dev` is the default
|
||||
branch; all PRs target `dev`, never `main` directly.
|
||||
2. **Remaining deferred items:**
|
||||
- **Large structural refactors:** delete-engine consolidation
|
||||
(`delete_extras_fd`/`manifest_delete_extras`/the delete-plan path),
|
||||
god-function splits, and translation-unit splits.
|
||||
- **`--progress`/`--info` receiver→sender event channel:** the root `./`
|
||||
line, ancestor-directory suppression, receiver-side `skip`/`backup` echo,
|
||||
and symlink/empty-dir quick-check feedback.
|
||||
- **`--delete-before` phase-0 keep-set** (rsync fixes the file list before
|
||||
the data pass; FastSync keeps its pre-scan snapshot race).
|
||||
- ~~**>256 MiB single-file streaming** (B4, the general whole-file limit).~~
|
||||
Closed by #318: the whole-file payload, the basis read/verify and the fuzzy
|
||||
basis are streamed through bounded buffers (lz4/append remain buffered).
|
||||
- **Wire native-size framing:** lengths are native `size_t` and the protocol
|
||||
assumes homogeneous word size/endianness — document or move to fixed-width
|
||||
framing.
|
||||
- **SCRAM-like daemon auth channel binding:** no TLS channel binding today
|
||||
(and it is not RFC 5802).
|
||||
- Still-open security nits: the pre-auth config/daemon-auth handshake has no
|
||||
aggregate wall-clock deadline (per-message timeout only — slowloris holds
|
||||
connection slots); the per-source registry fails open when the shared table
|
||||
is full (per-module/global caps and host ACLs still apply).
|
||||
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
|
||||
2. **Deferred security items** (documented, not implemented):
|
||||
- Pre-auth config/daemon-auth handshake has no aggregate wall-clock deadline
|
||||
(per-message timeout only) — slowloris holds connection slots.
|
||||
- Per-source registry fails open when the shared table is full (per-module/global
|
||||
caps and host ACLs still apply); consider fail-closed or larger/evicting table.
|
||||
- SCRAM-like daemon auth has no TLS channel binding (and is not RFC 5802).
|
||||
- `cleanup()` signal handler calls non-async-signal-safe teardown; daemon `umask(0)`.
|
||||
- Wire protocol assumes homogeneous word size/endianness (lengths are native
|
||||
`size_t`) — document or move to fixed-width framing.
|
||||
3. **Out of scope / intentional:** pull (remote source) mode is **not** planned —
|
||||
FastSync is push-only; see `RSYNC_COMPAT.md#direction`.
|
||||
|
||||
|
||||
@@ -75,9 +75,8 @@ matrix is classified as parity, caveat, or divergent in
|
||||
owner, group, devices, and special files — and does not imply compression or
|
||||
multithreading (see [Client](#client)). Ownership application is still
|
||||
privilege-gated: a receiver that cannot `chown` logs a warning and skips it.
|
||||
Under `-p` the source mode is copied exactly, including group/other-write
|
||||
bits; setuid/setgid/sticky bits are copied only when super-user activities are
|
||||
permitted, and are masked under `SUPER_MODE_OFF`/`--no-super` (see
|
||||
Under `-p` the source mode is copied exactly, including setuid/setgid/sticky
|
||||
and group/other-write bits (strict rsync parity; see
|
||||
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)).
|
||||
- Symlink transfer stores targets **verbatim** (`-l`/`--links`), including
|
||||
absolute and `..`-bearing targets, matching rsync. The receiver does not
|
||||
@@ -114,18 +113,13 @@ matrix is classified as parity, caveat, or divergent in
|
||||
(`-B1000`, `-essh`, `-MOPT`, `--opt=value`) are accepted, matching rsync.
|
||||
- `-r`, `-b`, `-L`, and `-B` are parsed with the rsync short names.
|
||||
- `--stats` prints the counters FastSync can observe plus the receiver-only
|
||||
counters reported over the wire (`Matched data`, deleted files, and the
|
||||
created/literal counters); `Number of files` and `Number of created files`
|
||||
carry rsync's per-type breakdown. `--progress` prints rsync-style per-file
|
||||
blocks including the leading `./` line, and (when progress is requested) a
|
||||
paths-only pre-count supplies rsync's `to-chk` denominator.
|
||||
counters (`Matched data`, deleted files) reported over the wire; rsync's
|
||||
per-type `Number of files` breakdown is not reproduced. `--progress` prints
|
||||
rsync-style per-file blocks (without rsync's leading `./` line).
|
||||
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
|
||||
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums. `auto` honors
|
||||
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` and otherwise follows rsync's
|
||||
compiled-in order. An omitted `--compress-level` uses the codec's rsync
|
||||
default (zstd 3, zlib/zlibx 6, lz4 ignored); `zlib`/`zlibx` share the
|
||||
literal-only zlib path (rsync's zlibx semantics), and the transfer checksum is
|
||||
not separately selectable.
|
||||
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums, negotiated with
|
||||
`auto`; `zlibx` behaves as `zlib`, and the transfer checksum is not separately
|
||||
selectable.
|
||||
|
||||
The detailed flag matrix is maintained in
|
||||
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It reports each row as **parity**,
|
||||
@@ -173,7 +167,7 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
||||
| `--preserve` | Preserve mode and mtime (`-p` + `-t`; add `-o`/`-g` for owner/group or `-U`/`--atimes` for atime; `-N`/`--crtimes` captures birth time but cannot apply it) |
|
||||
| `-U, --atimes` | Preserve access times. Captured with the metadata payload; does not enable ownership. |
|
||||
| `-N, --crtimes` | Capture birth time; cannot be applied (documented divergence) |
|
||||
| `-p, --perms` | Preserve permission bits. The source mode is copied exactly, including group/other-write bits; setuid/setgid/sticky are copied only when super-user activities are permitted (`SUPER_MODE_OFF`/`--no-super` masks them) |
|
||||
| `-p, --perms` | Preserve permission bits. Strict rsync parity: the source mode is copied exactly, including setuid/setgid/sticky and group/other-write bits |
|
||||
| `-t, --times` | Preserve modification times |
|
||||
| `-o, --owner` | Preserve the source owner (privilege-gated; mapped by name on the receiver with a numeric fallback) |
|
||||
| `-g, --group` | Preserve the source group (privilege-gated; mapped by name on the receiver with a numeric fallback) |
|
||||
@@ -187,7 +181,7 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
||||
| `--groupmap=MAP` | Map group names when applying ownership |
|
||||
| `--numeric-ids` | Apply source numeric uid/gid directly instead of mapping by name |
|
||||
| `--copy-as=USER[:GROUP]` | Force every written entry to USER[:GROUP] (requires a privileged receiver) |
|
||||
| `--fake-super` | Record the resolved owner plus full mode/rdev in rsync's reserved `user.rsync.%stat` xattr (rsync 3.4.1 grammar) and replay the permission bits; never performs a real chown |
|
||||
| `--fake-super` | Record the resolved owner plus mode/time in a reserved `user.fastsync.stat` xattr and replay mode/time; never performs a real chown |
|
||||
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes) |
|
||||
| `-D` | Preserve device and special files (implies `--devices --specials`) |
|
||||
| `--devices` | Recreate device nodes on the destination (privileged; skipped without `CAP_MKNOD`) |
|
||||
@@ -205,37 +199,34 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
||||
| `-u, --update` | Skip files newer than the source on the receiver |
|
||||
| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `--chunk-serialization`. |
|
||||
| `--existing` | Skip files not already present at the destination; update existing files normally. |
|
||||
| `--ignore-existing` | Skip files that already exist on the receiver; like rsync it does not apply to directories or symlinks. |
|
||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`; a basis of any size is supported, streamed in bounded chunks — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination (the copy is streamed, so a basis of any size works; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win; a basis of any size works; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
||||
| `--verify-basis` | FastSync-only: require a basis hit (`--compare-dest`/`--copy-dest`/`--link-dest`) to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync) |
|
||||
| `--delete` | Delete files on receiver not present in source (default timing: delete-during, matching rsync, so destination space is freed progressively). Scoped to the synchronized directories, so `--files-from` subsets are safe |
|
||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`) |
|
||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination |
|
||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win) |
|
||||
| `--delete` | Delete files on receiver not present in source (default timing: delete-after, i.e. only after the whole transfer succeeded). Scoped to the synchronized directories, so `--files-from` subsets are safe |
|
||||
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
|
||||
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
|
||||
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`) |
|
||||
| `--delete-after` | Explicit delete-after timing (implies `--delete`) |
|
||||
| `--delete-commit` | FastSync-only: keep the pre-2.28 atomic timing — delete only after the whole transfer succeeded (identical timing to `--delete-after`) |
|
||||
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
|
||||
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
|
||||
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication; the fixed `.fastsync-stage` staging name diverges from rsync — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
|
||||
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install; confined to the receive root (a relative path resolves below it; an absolute path is accepted only when it canonicalizes inside it), with an `EXDEV` non-atomic copy fallback |
|
||||
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication) |
|
||||
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install; confined to the receive root (relative only), with an `EXDEV` non-atomic copy fallback |
|
||||
| `-n, --dry-run` | Report what would be transferred without mutating the destination. Since protocol 2.21.0 a server-routed target contacts the receiver and reports would-transfer based on receiver state; a plain local destination keeps the client-side scan. Never mutates or deletes. |
|
||||
| `-v, --verbose` | Enable debug logging |
|
||||
| `-q, --quiet` | Suppress non-error output |
|
||||
| `--progress` | Show rsync-style per-file progress blocks from the receiver's wire counters; the root `./` line is printed whenever progress is active (rsync prints it only when the transfer root is created) |
|
||||
| `--progress` | Show rsync-style per-file progress blocks from the receiver's wire counters (FastSync does not print rsync's leading `./` line) |
|
||||
| `-P` | Enables partial-transfer mode + progress output; interrupted writes retain the already-written temp for resumption |
|
||||
| `--stats` | Print transfer statistics at end (bytes, files, timing), including the receiver-only counters reported over the wire; `Number of files` and `Number of created files` carry rsync's per-type breakdown (deleted files are reported as a single total) |
|
||||
| `--stats` | Print transfer statistics at end (bytes, files, timing), including the receiver-only counters reported over the wire; rsync's per-type `Number of files` breakdown is not reproduced |
|
||||
| `-i, --itemize-changes` | Print an rsync-style per-file change line |
|
||||
| `--out-format=FORMAT` | Output format for changed files (`%f %n %l %b %c %C %i %M %%`) |
|
||||
| `--out-format=FORMAT` | Output format for changed files (`%f %n %l %b %M %%`) |
|
||||
| `--list-only` | List source files instead of transferring |
|
||||
| `--fsync` | Fsync every written file before publication |
|
||||
| `-h, --human-readable` | Format transfer byte/rate counts with rsync's decimal (base-1000) units |
|
||||
| `--max-depth <n>` | Maximum directory depth to recurse (0 = unlimited, default: 0) |
|
||||
| `--log-file <path>` | Write log messages to file instead of stderr |
|
||||
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree (FastSync-native format, not rsync-interoperable) |
|
||||
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server); FastSync-native format, not rsync-interoperable |
|
||||
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server); FastSync-native format, not rsync-interoperable |
|
||||
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree |
|
||||
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server) |
|
||||
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server) |
|
||||
| `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
|
||||
| `--dest-dir <path>` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) |
|
||||
| `--save-to-disk` | Write received files to disk |
|
||||
@@ -248,12 +239,12 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
||||
| `-4, --ipv4` | Force IPv4 for destination resolution |
|
||||
| `-6, --ipv6` | Force IPv6 for destination resolution |
|
||||
| `--sockopts=OPTS` | Comma-separated OPT=VAL socket options applied before connect (`TCP_NODELAY`, `SO_KEEPALIVE`, `SO_RCVBUF`, `SO_SNDBUF`, `SO_REUSEADDR`) |
|
||||
| `--bwlimit <RATE>` | Bandwidth limit, using rsync's exact `parse_size_arg` grammar: a bare value is KiB/s; `K`/`M`/`G`/`T`/`P` are binary suffixes; `KB`/`MB` are decimal and `KiB`/`MiB` binary; decimals are accepted and quantized to whole KiB; `0` (or empty) means no limit. Also paces `--sendfile` transfers |
|
||||
| `--bwlimit <KB/s>` | Bandwidth limit in kilobytes per second |
|
||||
| `--chunk-size <n>` | Chunk size in bytes (default: 10485760) |
|
||||
| `--timeout <sec>` | I/O timeout in seconds, applied to both the socket (`SO_RCVTIMEO`/`SO_SNDTIMEO`) and the per-message protocol poll deadline. Default `0` = disabled (matching rsync); `0` disables it. `--no-timeout` is the negation. The value is not sent on the wire; the server side keeps its own safe floor. |
|
||||
| `--contimeout <sec>` | Connection timeout in seconds (default: 60, matching rsync); `0` disables it (`--no-contimeout` is the negation) |
|
||||
| `--stop-after=MINS` | Stop the transfer after MINS minutes (a positive integer); whatever was already transferred is kept |
|
||||
| `--stop-at=TIME` | Stop at an absolute time. Accepts rsync's `parse_time` forms (`Y-M-DTh:m`, `Y/M/DTh:m`, `Y-M-D`, `M-D`, `D`, `h:m`, `:m`, `T h:m`; omitted fields resolve to the next matching point in the local timezone), plus `now+N[smhd]` and FastSync's `HH:MM`/`HH:MM:SS` clock-time spelling. An early stop skips the late `--delete` keep-set |
|
||||
| `--stop-at=TIME` | Stop at an absolute time (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`); an early stop skips the late `--delete` keep-set |
|
||||
| `-b, --backup` | Backup existing destination files before overwriting |
|
||||
| `--backup-dir <dir>` | Target directory for backups (requires `--backup`) |
|
||||
| `--tls` | Enable TLS encryption |
|
||||
@@ -300,7 +291,6 @@ transfer is never aborted.
|
||||
| `FASTSYNC_SOURCE_DIR` | — | Source directory fallback |
|
||||
| `FASTSYNC_DEST_DIR` | — | Destination directory fallback |
|
||||
| `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback |
|
||||
| `FASTSYNC_MAX_WHOLE_FILE_SIZE` | `268435456` | Receiver-only test hook: a byte count that lowers the whole-file streaming bound. Payloads above it are streamed through a bounded buffer. Values are clamped to the 256 MiB protocol ceiling, so it can only lower, never raise, the bound. |
|
||||
|
||||
## Implementation Details
|
||||
|
||||
@@ -317,22 +307,17 @@ transfer is never aborted.
|
||||
`timeout`, `contimeout`, `quiet`, `stats`, `max_depth`, and `log_file` are
|
||||
client-only.
|
||||
5. **Queue** — thread-safe bounded queue with condition variables.
|
||||
6. **DirectoryScanner** — recursive traversal that buffers and sorts each
|
||||
directory (non-directories ascending, then directories ascending) and walks
|
||||
depth-first in rsync flist order, with exclude and include pattern support and
|
||||
max-depth enforcement.
|
||||
6. **DirectoryScanner** — recursive BFS traversal with exclude and include
|
||||
pattern support, max-depth enforcement.
|
||||
|
||||
### Key Algorithms
|
||||
|
||||
1. **File scanning** — sorted depth-first traversal in rsync flist order (each
|
||||
directory's non-directories ascending, then its directories ascending);
|
||||
entries matched against exclude and include patterns, with max-depth
|
||||
enforced. The `--threads` parallel scanner remains unordered.
|
||||
1. **File scanning** — BFS directory traversal; entries matched against exclude
|
||||
and include patterns, with max-depth enforced.
|
||||
2. **Chunking** — files accumulated until the `chunk_size` threshold (default
|
||||
10 MiB) is reached, then flushed.
|
||||
3. **Compression** — streaming zstd via `ZSTD_compressStream2()` /
|
||||
`ZSTD_decompressStream()`, with lz4 and zlib/zlibx codecs also supported
|
||||
(selectable with `--compress-choice`).
|
||||
`ZSTD_decompressStream()`.
|
||||
4. **Network protocol** — status-code-driven exchange with metadata packing,
|
||||
keep-alive, and abort support.
|
||||
5. **Incremental check** — the client sends `STATUS_CHECK` + path + size +
|
||||
@@ -387,8 +372,6 @@ Received files are written to a temporary path (suffixed with `.tmp`) and then a
|
||||
- C11 compiler
|
||||
- CMake >= 3.22
|
||||
- zstd library
|
||||
- zlib library
|
||||
- lz4 library
|
||||
- OpenSSL (development headers and libraries)
|
||||
- pthreads
|
||||
- SSH client (for SSH transport mode only)
|
||||
@@ -397,12 +380,12 @@ Received files are written to a temporary path (suffixed with `.tmp`) and then a
|
||||
|
||||
**Ubuntu/Debian:**
|
||||
```bash
|
||||
sudo apt install cmake build-essential libzstd-dev zlib1g-dev liblz4-dev libssl-dev openssh-client
|
||||
sudo apt install cmake build-essential libzstd-dev libssl-dev openssh-client
|
||||
```
|
||||
|
||||
**Nix:**
|
||||
```bash
|
||||
nix-shell # provides zstd, zlib, lz4, openssl, cmake, gcc
|
||||
nix-shell # provides zstd, openssl, cmake, gcc
|
||||
```
|
||||
|
||||
## Building
|
||||
@@ -522,8 +505,8 @@ features without changing the meaning of ordinary compatibility options.
|
||||
|---|---|
|
||||
| `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. |
|
||||
| `-z [level]`, `--compress [level]` | Enable streaming compression (default `zstd`), levels 1-22. |
|
||||
| `--compress-level <n>` | Set the compression level (1-22). Omitted, each codec uses its rsync default: zstd 3, zlib/zlibx 6, lz4 ignored. |
|
||||
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlib`/`zlibx` share the same literal-only zlib path. |
|
||||
| `--compress-level <n>` | Set the compression level. |
|
||||
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlibx` behaves as `zlib`. |
|
||||
| `--zl <n>` | Alias for `--compress-level`. |
|
||||
| `--skip-compress <list>` | Skip compression for `/`- or `,`-separated suffixes; defaults to rsync 3.4.1's built-in list. Incompatible with `--chunk-serialization`. |
|
||||
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
|
||||
@@ -536,9 +519,9 @@ features without changing the meaning of ordinary compatibility options.
|
||||
| `--server-host <host>` | Select the TCP server host. |
|
||||
| `--server-port <port>` | Select the TCP server port (`--port <port>` and `--port=<port>` are rsync-friendly aliases). |
|
||||
| `--tls` | Enable TLS for TCP transport. |
|
||||
| `--bwlimit <RATE>` | Apply token-bucket bandwidth limiting with rsync's exact `parse_size_arg` grammar (bare = KiB/s, `K`/`M`/`G`/`T`/`P` binary, `KB`/`MB` decimal, `KiB`/`MiB` binary, decimals quantized to whole KiB, `0`/empty = no limit; also paces `--sendfile` transfers). |
|
||||
| `--progress` | Show rsync-style per-file progress blocks from the receiver's wire counters; the root `./` line is printed whenever progress is active (rsync prints it only when the transfer root is created). |
|
||||
| `--stats` | Print transfer statistics, including the receiver-only counters reported over the wire; `Number of files`/`Number of created files` carry rsync's per-type breakdown (deleted files are a single total). |
|
||||
| `--bwlimit <KB/s>` | Apply token-bucket bandwidth limiting. |
|
||||
| `--progress` | Show rsync-style per-file progress blocks from the receiver's wire counters (FastSync omits rsync's leading `./` line). |
|
||||
| `--stats` | Print transfer statistics, including the receiver-only counters reported over the wire; rsync's per-type `Number of files` breakdown is not reproduced. |
|
||||
| `--timeout <seconds>` | Set the socket **and** per-message protocol I/O timeout. Default `0` = disabled (matching rsync); `0` disables it. |
|
||||
| `--contimeout <seconds>` | Connection timeout (default 60, matching rsync); `0` disables it. |
|
||||
|
||||
@@ -572,27 +555,22 @@ remote SSH argv is already built injection-safe.
|
||||
| `--size-only` | Skip incremental files matching in size, ignoring mtime. |
|
||||
| `-I, --ignore-times` | Transfer files even when size and mtime match. |
|
||||
| `-u, --update` | Skip files newer than the source on the receiver. |
|
||||
| `--ignore-existing` | Skip files that already exist on the receiver; like rsync it does not apply to directories or symlinks. |
|
||||
| `-@, --modify-window <sec>` | Modification-time tolerance (seconds) for the incremental/basis quick-check; `0` requires an exact mtime match. |
|
||||
| `-W, --whole-file` | Transfer changed files without delta processing (`--no-whole-file` clears it). |
|
||||
| `-B <n>, --block-size <n>` | Delta block size in bytes (alias `--delta-block`). |
|
||||
| `-d, --dirs` | Transfer the named directory entries without recursing into their contents (aliases `--old-dirs`/`--old-d`). |
|
||||
| `-R, --relative` | Use rsync's relative path semantics (including the `/./` cut); with `--files-from`, preserve each listed entry's relative path below the destination root. |
|
||||
| `--files-from <file>` | Read the source file list from FILE (paths relative to the source root). |
|
||||
| `-0, --from0` | Treat entries in `--files-from` files as NUL-delimited instead of newline-delimited. |
|
||||
| `--delay-updates` | Put updated files into place only at the end of the transfer (the fixed `.fastsync-stage` staging name diverges from rsync; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`; a basis of any size is supported, streamed in bounded chunks — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination (the copy is streamed, so a basis of any size works; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win; a basis of any size works; see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)). |
|
||||
| `--verify-basis` | FastSync-only: require a basis hit to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync). |
|
||||
| `--delay-updates` | Put updated files into place only at the end of the transfer. |
|
||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`). |
|
||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination. |
|
||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win). |
|
||||
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
|
||||
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
|
||||
| `--append-verify` | Like `--append`, but verifies the retained prefix checksum first (falls back to a full transfer on mismatch). |
|
||||
| `--delete` | Request removal of destination entries absent from the source. The server must allow deletion. Default timing is delete-during (matching rsync's `--del`): extras are removed per directory as the transfer proceeds, so destination space is freed progressively. Scoped to the synchronized directories, so `--files-from` subsets are safe. |
|
||||
| `--delete` | Request removal of destination entries absent from the source. The server must allow deletion. Default timing is delete-after: extras are removed only after the whole transfer succeeded. Scoped to the synchronized directories, so `--files-from` subsets are safe. |
|
||||
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
|
||||
| `--delete-during`, `--del` | Delete each directory's extras as that directory is processed (implies `--delete`). Since protocol 2.24.0 the sender streams a per-directory `STATUS_DELETE_PLAN` frame as it reaches each source directory; this is also the default timing of a plain `--delete`. |
|
||||
| `--delete-delay` | Record extras per directory during the scan but remove them only after a successful transfer (implies `--delete`). Uses the same per-directory `STATUS_DELETE_PLAN` frames as `--delete-during`, applied late. |
|
||||
| `--delete-commit` | FastSync-only: atomic delete-after timing (only after the whole transfer succeeded). |
|
||||
| `--delete-during`, `--del` | Delete extras once the keep-set manifest is known, before data is applied (implies `--delete`; early mode, same engine behaviour as `--delete-before`). |
|
||||
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`; commit mode, same behaviour as `--delete-after`). |
|
||||
| `--delete-after` | Explicit delete-after timing: delete only after the transfer succeeded (implies `--delete`). |
|
||||
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected). |
|
||||
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync. |
|
||||
@@ -607,18 +585,18 @@ remote SSH argv is already built injection-safe.
|
||||
| `--max-alloc <SIZE>` | Maximum single allocation (binary units; default 1G; `0` = no local limit). |
|
||||
| `--max-depth <n>` | Limit recursive scanning depth; zero means unlimited. |
|
||||
| `-b, --backup` | Back up overwritten files. |
|
||||
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install (confined to the receive root: relative resolves below it, absolute must canonicalize inside it; `EXDEV` falls back to a non-atomic copy). |
|
||||
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install (confined to the receive root; `EXDEV` falls back to a non-atomic copy). |
|
||||
| `--backup-dir <dir>` | Store backups under a separate directory (requires `--backup`). |
|
||||
| `--suffix <suffix>` | Set the backup filename suffix (default: `~`). |
|
||||
| `--partial` | Select partial-transfer handling. On failed/interrupted writes the already-written temp file is retained (best-effort) for resumption. With `--partial --partial-dir <dir>`, completed files are written under the partial directory and installed atomically. |
|
||||
| `--partial-dir <dir>` | Set a relative partial-transfer directory below the server destination root. Implies `--partial`. Rejected together with `--inplace` (`--inplace cannot be used with --partial-dir`, matching rsync), because the inplace path bypasses partial/temp staging. |
|
||||
| `--inplace` | Write directly to the destination instead of using a temporary file. Cannot be combined with `--partial-dir`. |
|
||||
| `--partial-dir <dir>` | Set a relative partial-transfer directory below the server destination root. Use with `--partial`. |
|
||||
| `--inplace` | Write directly to the destination instead of using a temporary file. |
|
||||
| `--fsync` | Fsync every written file before publication. |
|
||||
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree (FastSync-native format, not rsync-interoperable). |
|
||||
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server); FastSync-native format, not rsync-interoperable. |
|
||||
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server); FastSync-native format, not rsync-interoperable. |
|
||||
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree. |
|
||||
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server). |
|
||||
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server). |
|
||||
| `--stop-after=MINS` | Stop the transfer after MINS minutes; whatever was already transferred is kept. |
|
||||
| `--stop-at=TIME` | Stop at an absolute time. Accepts rsync's `parse_time` forms (`Y-M-DTh:m`, `Y/M/DTh:m`, `Y-M-D`, `M-D`, `D`, `h:m`, `:m`, `T h:m`; omitted fields resolve to the next matching point in the local timezone), plus `now+N[smhd]` and FastSync's `HH:MM`/`HH:MM:SS` clock-time spelling. An early stop skips the late `--delete` keep-set. |
|
||||
| `--stop-at=TIME` | Stop at an absolute time (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`). An early stop skips the late `--delete` keep-set. |
|
||||
|
||||
### Metadata and links
|
||||
|
||||
@@ -627,11 +605,8 @@ remote SSH argv is already built injection-safe.
|
||||
| `--preserve` | Preserve mode and mtime (long form only; equivalent to `-p` + `-t`). Add `-o`/`-g` for owner/group, `-U`/`--atimes` for atime, or an identity flag (`--chown`/`--usermap`/`--groupmap`/`--numeric-ids`/`--copy-as`) for mapped ownership. |
|
||||
| `-U`, `--atimes` | Preserve access times. Captured with the metadata payload; does not enable ownership. |
|
||||
| `-N`, `--crtimes` | Capture birth time and transmit it; it cannot be applied because no portable filesystem call can set a birth time (documented divergence). |
|
||||
| `-p`, `--perms` | Preserve permission bits. One of the four per-attribute preserve flags (with `-t`/`-o`/`-g`); under `-p` the source mode is copied exactly (group/other-write included; setuid/setgid/sticky included only when super-user activities are permitted, masked under `SUPER_MODE_OFF`/`--no-super`), matching rsync otherwise. |
|
||||
| `-p`, `--perms` | Preserve permission bits. One of the four per-attribute preserve flags (with `-t`/`-o`/`-g`); under `-p` the source mode is copied exactly (setuid/setgid/sticky and group/other-write included), matching rsync. |
|
||||
| `-t`, `--times` | Preserve modification times. Independent of the other attributes; `-O`/`--omit-dir-times` suppresses directories only. |
|
||||
| `-O`, `--omit-dir-times` | Do not apply modification times to directories. |
|
||||
| `-J`, `--omit-link-times` | Do not apply times to symlinks. |
|
||||
| `--open-noatime` | Open source files with `O_NOATIME` so reading for a transfer does not update their access time (client-only). |
|
||||
| `-o`, `--owner` | Preserve the source owner (uid). Mapped by name on the receiver with a raw-numeric fallback (only numeric ids cross the wire); application is privilege-gated. |
|
||||
| `-g`, `--group` | Preserve the source group (gid). Same name-mapping/numeric-fallback and privilege gating as `-o`. |
|
||||
| `--no-perms`, `--no-times`, `--no-owner`, `--no-group` | Negate each per-attribute flag (also `--no-p`/`--no-t`/`--no-o`/`--no-g`); `--no-preserve` clears all four. |
|
||||
@@ -644,7 +619,7 @@ remote SSH argv is already built injection-safe.
|
||||
| `--groupmap=MAP` | Map group names when applying ownership (same syntax as `--usermap`). |
|
||||
| `--numeric-ids` | Mapping modifier: apply the source numeric uid/gid directly instead of mapping by name (combine with `-o`/`-g`, `-a`, or a map). |
|
||||
| `--copy-as=USER[:GROUP]` | Force every written entry to USER[:GROUP]; requires a privileged receiver. |
|
||||
| `--fake-super` | Record the resolved owner plus full mode/rdev in rsync's reserved `user.rsync.%stat` xattr (rsync 3.4.1 grammar) and replay the permission bits; never performs a real chown. |
|
||||
| `--fake-super` | Record the resolved owner plus mode/time in a reserved `user.fastsync.stat` xattr and replay mode/time; never performs a real chown. |
|
||||
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes). |
|
||||
| `--no-super` | Forbid those super-user activities even when the receiver is root. |
|
||||
| `-l`, `--links` | Copy symlinks as symlinks; the target is stored verbatim (absolute and `..`-bearing targets included), matching rsync. |
|
||||
@@ -658,7 +633,6 @@ remote SSH argv is already built injection-safe.
|
||||
| `-D` | Preserve device and special files (implies `--devices --specials`). |
|
||||
| `--devices` | Recreate device nodes on the destination (privileged; skipped without `CAP_MKNOD`). |
|
||||
| `--specials` | Recreate special files: FIFOs and unix sockets. |
|
||||
| `--copy-devices` | Copy a source device's content as an ordinary regular file on the destination (rsync's non-privileged safe mode) instead of recreating the device node. |
|
||||
| `-S`, `--sparse` | Sparse-file handling: receiver preserves holes (zero runs are written as holes; no wire change). |
|
||||
|
||||
### Output and logging
|
||||
@@ -667,17 +641,12 @@ remote SSH argv is already built injection-safe.
|
||||
|---|---|
|
||||
| `-v`, `--verbose` | Enable debug logging. |
|
||||
| `-q`, `--quiet` | Suppress non-error output. |
|
||||
| `--progress` | Show rsync-style per-file progress blocks; the root `./` line is printed whenever progress is active (rsync prints it only when the transfer root is created). |
|
||||
| `--stats` | Print transfer statistics, including the receiver-only counters reported over the wire; `Number of files`/`Number of created files` carry rsync's per-type breakdown (deleted files are a single total). |
|
||||
| `-i, --itemize-changes` | Print an rsync-style per-file change line. |
|
||||
| `--out-format=FORMAT` | Output format for changed files (`%f %n %l %b %c %C %i %M %%`). |
|
||||
| `--progress` | Show rsync-style per-file progress blocks (not rsync's leading `./` line). |
|
||||
| `--stats` | Print transfer statistics, including the receiver-only counters reported over the wire. |
|
||||
| `-i`, `--itemize-changes` | Print an rsync-style per-file change line. |
|
||||
| `--out-format=FORMAT` | Output format for changed files (`%f %n %l %b %M %%`). |
|
||||
| `--list-only` | List source files instead of transferring. |
|
||||
| `--outbuf=MODE` | stdout/stderr buffering: `N` (none/unbuffered), `L` (line-buffered), or `B` (block-buffered, default). |
|
||||
| `--log-file <path>` | Write log output to a file. |
|
||||
| `--log-file-format=FORMAT` | Per-file log-line format (requires `--log-file`). |
|
||||
| `--stderr=MODE` | Route logging: `errors` (default), `all`, or `client` (forward the client's diagnostics to the server's stderr over the client-message channel). |
|
||||
| `--msgs2stderr` | Route all messages to stderr (deprecated spelling of `--stderr=all`). |
|
||||
| `--no-msgs2stderr` | Forward the client's diagnostics to the server (deprecated spelling of `--stderr=client`). |
|
||||
| `-V`, `--version` | Print the FastSync protocol version. |
|
||||
| `--help` | Print command usage. |
|
||||
|
||||
@@ -690,7 +659,7 @@ remote SSH argv is already built injection-safe.
|
||||
| `--fastsync-server-path <path>` | Remote FastSync server path for SSH mode (client-only; never crosses the wire). |
|
||||
| `--rsync-path <path>` | Alias for `--fastsync-server-path`. |
|
||||
| `-M`, `--remote-option=OPT` | Append OPT to the remote server invocation over SSH (repeatable; rejected for daemon/TCP destinations). |
|
||||
| `--trust-sender` | Receiver-local: trust the remote sender's file list and skip path re-validation (does not affect symlink targets). **On the client this flag alone is inert** — it is never sent on the wire; the server must be started with its own `--trust-sender`, or the client must forward it with `-M--trust-sender` (SSH only). |
|
||||
| `--trust-sender` | Receiver-local: trust the remote sender's file list and skip path re-validation (does not affect symlink targets). |
|
||||
| `--timeout <sec>` | Socket + per-message I/O timeout; default `0` = disabled. |
|
||||
| `--contimeout <sec>` | Connection timeout; default 60; `0` disables. |
|
||||
| `--source-dir <path>` | Set the source directory explicitly. |
|
||||
@@ -702,11 +671,6 @@ remote SSH argv is already built injection-safe.
|
||||
| `-4`, `--ipv4` | Force IPv4 for destination resolution. |
|
||||
| `-6`, `--ipv6` | Force IPv6 for destination resolution. |
|
||||
| `--sockopts=OPTS` | Comma-separated OPT=VAL socket options applied before connect. |
|
||||
| `--blocking-io` | SSH transport only: leave the socket without read/write timeouts so it blocks naturally (no effect on TCP). |
|
||||
| `--protocol=NUM` | Force the wire protocol version; must equal the current `PROTOCOL_VERSION` (FastSync cannot speak older/virtual wire formats). |
|
||||
| `--old-args` | Accepted for rsync CLI compatibility; no effect (the remote server path is always safely quoted). |
|
||||
| `--iconv=LOCAL[,REMOTE]` | Convert file-name charsets at the wire boundary (`LOCAL` is our names' charset, `REMOTE` the peer's, defaulting to `LOCAL`). |
|
||||
| `--no-iconv` | Disable `--iconv` charset conversion (same as `--iconv=-`). |
|
||||
| `--tls` | Enable TLS. Requires `--cert`, `--key`, and `--ca`. |
|
||||
| `--cert <path>` | TLS certificate file. |
|
||||
| `--key <path>` | TLS private key file. |
|
||||
@@ -733,14 +697,14 @@ remote SSH argv is already built injection-safe.
|
||||
| `-6`, `--ipv6` | Bind an IPv6 socket. |
|
||||
| `--allow-delete` | Permit client delete manifests. Deletion is refused by default. This also gates `--force` (which can recursively replace/remove a destination directory tree). |
|
||||
| `--allow-super` | Standalone TCP listener only: keep super-user activities enabled for a **root** receiver. Without it a root standalone server forces `SUPER_MODE_OFF`, so client `--devices`/`--write-devices`/`--super` and client-chosen ownership requests are skipped/refused. Rejected with `--stdio` (the SSH remote argv is client-composed; use a forced command if the default must hold). No effect when not root. Daemon modules opt in per module with `client owner = yes`. |
|
||||
| `--trust-sender` | Trust the remote sender's file list: skip the receiver's up-front path-traversal re-validation (fewer checks, faster, potentially unsafe; off by default). It does not affect symlink targets, which are stored verbatim either way. A client `--trust-sender` is never sent over the wire — the server must set this flag itself, or the client must forward it via `-M--trust-sender`. |
|
||||
| `--trust-sender` | Trust the remote sender's file list: skip the receiver's up-front path-traversal re-validation (fewer checks, faster, potentially unsafe; off by default). It does not affect symlink targets, which are stored verbatim either way. |
|
||||
| `--no-super` | Operator veto: never attempt super-user activities (ownership, device nodes) even as root, and refuse any client `--copy-as`/`--super` request. |
|
||||
| `--allow-unauthenticated` | Permit plaintext/anonymous network clients; an auth-required module still accepts only opted-in loopback plaintext. |
|
||||
| `--iconv=LOCAL[,REMOTE]` | Declare this server's LOCAL charset for file-name conversion. |
|
||||
| `--password-file=FILE` | Credential store for modules that declare `auth users`. Requires `--daemon`. FastSync-native SCRAM/PBKDF2 format, not rsync-interoperable. |
|
||||
| `--early-input=FILE` | Second credential store layered over `--password-file`. Requires `--daemon`. FastSync-native format, not rsync-interoperable. |
|
||||
| `--hash-credentials <file>` | Read `<file>`'s `user:password` lines and print PBKDF2 credential-store lines to stdout, then exit. Cannot be combined with `--daemon` or `--stdio`. FastSync-native, not rsync-interoperable. |
|
||||
| `--iterations N` | PBKDF2 iteration count for `--hash-credentials` (default 600000, range 100000–10000000). Requires `--hash-credentials`. FastSync-native, not rsync-interoperable. |
|
||||
| `--password-file=FILE` | Credential store for modules that declare `auth users`. Requires `--daemon`. |
|
||||
| `--early-input=FILE` | Second credential store layered over `--password-file`. Requires `--daemon`. |
|
||||
| `--hash-credentials <file>` | Read `<file>`'s `user:password` lines and print PBKDF2 credential-store lines to stdout, then exit. Cannot be combined with `--daemon` or `--stdio`. |
|
||||
| `--iterations N` | PBKDF2 iteration count for `--hash-credentials` (default 600000, range 100000–10000000). Requires `--hash-credentials`. |
|
||||
| `-v`, `--verbose` | Enable debug logging. |
|
||||
| `--help` | Print server usage. |
|
||||
|
||||
@@ -769,17 +733,9 @@ and `address`, the global section accepts:
|
||||
- `hosts allow` / `hosts deny` — comma- and/or whitespace-separated host access
|
||||
patterns.
|
||||
|
||||
A `[module]` requires `path`, and may also set `read only`, `write only`,
|
||||
`client owner`, `auth users`, `max connections` (0 = unlimited; enforced per
|
||||
module across all connection children), and its own `hosts allow`/`hosts deny`.
|
||||
|
||||
Like rsync, a module is **read-only by default**: a bare `[module]` with only a
|
||||
`path` refuses a write transfer. Opt a module into writability explicitly with
|
||||
`read only = no` or `write only = yes`; a global `read only` value in the
|
||||
section before the first `[module]` sets the default for later modules, and a
|
||||
module's own `read only`/`write only = yes` always wins over it. An
|
||||
rsync-style `write only = yes` is mapped to writability because FastSync is
|
||||
push-only (a module can never be read from the network).
|
||||
A `[module]` requires `path`, and may also set `read only`, `client owner`,
|
||||
`auth users`, `max connections` (0 = unlimited; enforced per module across all
|
||||
connection children), and its own `hosts allow`/`hosts deny`.
|
||||
|
||||
The per-host cap and the shared auth lockout identify a source by its numeric
|
||||
peer IP. **Loopback peers (127.0.0.0/8, IPv6 `::1`) are exempt**: every local
|
||||
@@ -833,7 +789,7 @@ before the module list, before authentication, and the connecting peer address
|
||||
|
||||
## Protocol and Security
|
||||
|
||||
FastSync protocol version `2.30.0` is shared by the client and server. The
|
||||
FastSync protocol version `2.26.0` is shared by the client and server. The
|
||||
current protocol is sender-driven and includes configuration negotiation,
|
||||
including the maximum allocation limit, incremental checks, checksums,
|
||||
manifests, keep-alives, abort handling, per-file remove-source results, and
|
||||
@@ -906,10 +862,8 @@ The project will reach the drop-in replacement goal in stages:
|
||||
completion wave's scope; the tests live in `tests/integration/` and skip
|
||||
cleanly when rsync is unavailable.
|
||||
3. `-a` implements full rsync `-rlptgoD`; under `-p` the source mode is copied
|
||||
exactly, including group/other-write bits, with setuid/setgid/sticky copied
|
||||
only when super-user activities are permitted (masked under
|
||||
`SUPER_MODE_OFF`/`--no-super`). Ownership application stays privilege-gated,
|
||||
as in rsync.
|
||||
exactly (no masking). Ownership application stays privilege-gated, as in
|
||||
rsync.
|
||||
4. Symlink (verbatim storage), sparse-file, metadata, delete-policy (including
|
||||
`--max-delete` partial + exit 25, per-directory `--delete-during`/
|
||||
`--delete-delay`), codecs, and resumable-write semantics are implemented;
|
||||
|
||||
+153
-288
File diff suppressed because one or more lines are too long
@@ -8,6 +8,3 @@ markers =
|
||||
daemon_detach: real double-fork backgrounding path (--daemon without
|
||||
--no-detach); slower/fragile, so it runs in the full suite but not the
|
||||
fast PR gate
|
||||
parity: differential rsync-parity case (full set; runs on push to
|
||||
dev/main)
|
||||
parity_ci: fast differential rsync-parity subset (runs on the PR gate)
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
nested
|
||||
@@ -1 +0,0 @@
|
||||
nested
|
||||
+29
-198
@@ -1,6 +1,5 @@
|
||||
#include "change_list.h"
|
||||
#include "checksum.h"
|
||||
#include "log.h"
|
||||
#include "utils.h"
|
||||
#include <fcntl.h>
|
||||
#include <limits.h>
|
||||
@@ -71,16 +70,7 @@ static bool strbuf_append(StrBuf* buf, const char* text) {
|
||||
|
||||
bool change_list_enabled(const Config* config) {
|
||||
return config != NULL && (config->itemize_changes || config->out_format != NULL ||
|
||||
(config->log_file != NULL && config->log_file_format != NULL) ||
|
||||
(config->info_level & LOG_INFO_NAME) != 0);
|
||||
}
|
||||
|
||||
/* Emitted once, lazily, ahead of the first --info=name entry: rsync prints the
|
||||
* transfer-root `./` name line when the root directory is (re)created. */
|
||||
static bool name_root_printed = false;
|
||||
|
||||
void change_reset_name_root(void) {
|
||||
name_root_printed = false;
|
||||
(config->log_file != NULL && config->log_file_format != NULL));
|
||||
}
|
||||
|
||||
/* ---- Itemize code ---- */
|
||||
@@ -123,15 +113,8 @@ static char itemize_type_char(const ChangeEvent* event) {
|
||||
static bool times_match(const Config* config, const ChangeEvent* event) {
|
||||
if (!event->dest.known || !event->dest.existed)
|
||||
return false;
|
||||
if (event->mtime_sec == event->dest.mtime_sec) {
|
||||
/* A regular file's sub-second mtime IS preserved by the receiver, so an nsec
|
||||
difference is a real change. A directory or symlink has no preserved
|
||||
sub-second mtime (rsync's quick-check compares whole seconds there), so a
|
||||
nanosecond-only difference must not render a spurious `.d..t` / `.L..t`. */
|
||||
if (event->is_directory || event->is_symlink || event->is_special)
|
||||
return true;
|
||||
if (event->mtime_sec == event->dest.mtime_sec)
|
||||
return event->mtime_nsec == event->dest.mtime_nsec;
|
||||
}
|
||||
long long delta = (long long)event->mtime_sec - (long long)event->dest.mtime_sec;
|
||||
if (delta < 0)
|
||||
delta = -delta;
|
||||
@@ -148,14 +131,6 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
|
||||
update = 'h';
|
||||
else if (created)
|
||||
update = (event->is_directory || event->is_symlink || event->is_special) ? 'c' : '>';
|
||||
else if (event->is_directory)
|
||||
/* rsync: an existing directory that only has attribute changes carries no
|
||||
transfer, so the update column is `.` rather than `>`. */
|
||||
update = '.';
|
||||
else if (event->is_symlink)
|
||||
/* rsync: an existing symlink whose target is unchanged is a `.` update
|
||||
(attributes only); a changed target is `c` (the link value changed). */
|
||||
update = event->dest.target_matches ? '.' : 'c';
|
||||
else
|
||||
update = '>';
|
||||
code[0] = update;
|
||||
@@ -166,20 +141,12 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
|
||||
code[11] = '\0';
|
||||
return;
|
||||
}
|
||||
/* rsync's value/checksum column: `c` for a symlink whose target changed (the
|
||||
link value is the compared content); no destination digest is available for
|
||||
a regular file. */
|
||||
bool value_diff = event->is_symlink && !event->dest.target_matches;
|
||||
/* rsync itemizes size only for regular files: a directory's st_size and a
|
||||
symlink's target length are not compared. */
|
||||
bool size_diff = !event->is_directory && !event->is_symlink && !event->is_special &&
|
||||
event->size != event->dest.size;
|
||||
/* rsync itemizes the time column only when -t/--times is in effect. */
|
||||
bool time_diff = config->preserve_times && !times_match(config, event);
|
||||
bool size_diff = event->size != event->dest.size;
|
||||
bool time_diff = !times_match(config, event);
|
||||
bool perms_diff = (event->mode & 07777) != (event->dest.mode & 07777);
|
||||
bool owner_diff = event->uid != (uid_t)event->dest.uid;
|
||||
bool group_diff = event->gid != (gid_t)event->dest.gid;
|
||||
code[2] = value_diff ? 'c' : '.';
|
||||
code[2] = '.'; /* checksum: no destination digest available */
|
||||
code[3] = size_diff ? 's' : '.';
|
||||
code[4] = time_diff ? 't' : '.';
|
||||
code[5] = (config->preserve_perms && perms_diff) ? 'p' : '.';
|
||||
@@ -191,33 +158,12 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
|
||||
code[11] = '\0';
|
||||
}
|
||||
|
||||
/* True when the itemized destination entry is unchanged, i.e. rsync would print
|
||||
* no line at all. Reuses itemize_code so suppression is exactly consistent
|
||||
* with what would have been rendered: the update column must be `.` and every
|
||||
* attribute column must be `.`. */
|
||||
static bool itemize_is_unchanged(const Config* config, const ChangeEvent* event) {
|
||||
if (!event->dest.known || !event->dest.existed)
|
||||
return false;
|
||||
char code[12];
|
||||
itemize_code(config, event, code);
|
||||
if (code[0] != '.')
|
||||
return false;
|
||||
for (int i = 2; i < 11; i++) {
|
||||
if (code[i] != '.')
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* rsync %n: the transfer-relative name, with a trailing slash for directories.
|
||||
* The transfer root is `.` (so `%n` renders `./`), matching rsync's root entry. */
|
||||
/* rsync %n: the transfer-relative name, with a trailing slash for directories. */
|
||||
static bool append_name(StrBuf* buf, const ChangeEvent* event) {
|
||||
const char* name = event->name != NULL ? event->name : "";
|
||||
if (event->is_directory && name[0] == '\0')
|
||||
return strbuf_append(buf, "./");
|
||||
if (!strbuf_append(buf, name))
|
||||
if (!strbuf_append(buf, event->name != NULL ? event->name : ""))
|
||||
return false;
|
||||
if (event->is_directory && name[strlen(name) - 1] != '/')
|
||||
if (event->is_directory && (event->name == NULL || event->name[0] == '\0' ||
|
||||
event->name[strlen(event->name) - 1] != '/'))
|
||||
return strbuf_append_char(buf, '/');
|
||||
return true;
|
||||
}
|
||||
@@ -246,55 +192,27 @@ char* change_render_itemize(const Config* config, const ChangeEvent* event) {
|
||||
return line.data;
|
||||
}
|
||||
|
||||
/* rsync's `--info=name` line for an updated entry: the transfer-relative name
|
||||
* (trailing slash for directories) plus the ` -> target` / ` => target` link
|
||||
* suffix. `--info=name` does not alter an itemize/out-format run. */
|
||||
static char* change_render_name(const ChangeEvent* event) {
|
||||
StrBuf line = {0};
|
||||
bool ok = append_name(&line, event) && append_link_suffix(&line, event);
|
||||
if (!ok) {
|
||||
strbuf_free(&line);
|
||||
return NULL;
|
||||
}
|
||||
if (line.data == NULL) {
|
||||
line.data = str_dup("");
|
||||
if (!line.data)
|
||||
return NULL;
|
||||
}
|
||||
return line.data;
|
||||
}
|
||||
|
||||
/* rsync's `--info=name2` line for an unchanged entry: `NAME is uptodate`. */
|
||||
static char* change_render_name_uptodate(const ChangeEvent* event) {
|
||||
char* name = change_render_name(event);
|
||||
if (name == NULL)
|
||||
return NULL;
|
||||
size_t length = strlen(name);
|
||||
char* line = malloc(length + sizeof(" is uptodate"));
|
||||
if (line == NULL) {
|
||||
free(name);
|
||||
return NULL;
|
||||
}
|
||||
memcpy(line, name, length);
|
||||
memcpy(line + length, " is uptodate", sizeof(" is uptodate"));
|
||||
free(name);
|
||||
return line;
|
||||
}
|
||||
|
||||
/* ---- --out-format / --log-file-format ---- */
|
||||
|
||||
/* rsync 3.4.1's `%C` uses the negotiated TRANSFER checksum (the first name of a
|
||||
* two-name "transfer,pre-transfer" --checksum-choice), not the pre-transfer
|
||||
* whole-file digest FastSync compares against on the wire. The default "auto"
|
||||
* resolves to xxh128, so an explicit selection and the default both render the
|
||||
* selected algorithm's digest. */
|
||||
/* rsync 3.4.1's `%C` uses the negotiated transfer checksum; with the default
|
||||
* "auto" choice on both ends that is xxh128. FastSync's internal XXH64 default
|
||||
* is not an rsync algorithm, so map it to xxh128 for parity. */
|
||||
static ChecksumAlgo out_format_checksum_algo(const Config* config) {
|
||||
return (ChecksumAlgo)config->cli.checksum_transfer_algo;
|
||||
switch ((ChecksumAlgo)config->checksum_algo) {
|
||||
case CHECKSUM_ALGO_MD5:
|
||||
return CHECKSUM_ALGO_MD5;
|
||||
case CHECKSUM_ALGO_XXH3:
|
||||
return CHECKSUM_ALGO_XXH3;
|
||||
case CHECKSUM_ALGO_XXH128:
|
||||
return CHECKSUM_ALGO_XXH128;
|
||||
case CHECKSUM_ALGO_XXH64:
|
||||
default:
|
||||
return CHECKSUM_ALGO_XXH128;
|
||||
}
|
||||
}
|
||||
|
||||
/* Render a digest as rsync's sum_as_hex: xxh128 prints the HIGH 64-bit half
|
||||
* before the low half, and xxh64/xxh3 print their 64-bit value big-endian; every
|
||||
* other algorithm prints its bytes in order. */
|
||||
/* Render a digest as rsync's sum_as_hex: for xxh128 the HIGH 64-bit half is
|
||||
* printed before the low half; every other algorithm prints its bytes in order. */
|
||||
static void digest_to_hex(ChecksumAlgo algo, const uint8_t* digest, size_t len, char* out) {
|
||||
if (algo == CHECKSUM_ALGO_XXH128 && len == 16) {
|
||||
uint64_t low = 0;
|
||||
@@ -304,12 +222,6 @@ static void digest_to_hex(ChecksumAlgo algo, const uint8_t* digest, size_t len,
|
||||
snprintf(out, len * 2 + 1, "%016llx%016llx", (unsigned long long)high, (unsigned long long)low);
|
||||
return;
|
||||
}
|
||||
if ((algo == CHECKSUM_ALGO_XXH64 || algo == CHECKSUM_ALGO_XXH3) && len == 8) {
|
||||
uint64_t value = 0;
|
||||
memcpy(&value, digest, sizeof(value));
|
||||
snprintf(out, len * 2 + 1, "%016llx", (unsigned long long)value);
|
||||
return;
|
||||
}
|
||||
static const char hex[] = "0123456789abcdef";
|
||||
for (size_t i = 0; i < len; i++) {
|
||||
out[i * 2] = hex[(digest[i] >> 4) & 0xf];
|
||||
@@ -348,9 +260,6 @@ static void fill_event_checksum(const Config* config, const File* file, ChangeEv
|
||||
if (file->path == NULL)
|
||||
return;
|
||||
ChecksumAlgo algo = out_format_checksum_algo(config);
|
||||
/* rsync renders `--checksum-choice=none` as a blank 2-character column. */
|
||||
if (algo == CHECKSUM_ALGO_NONE)
|
||||
return;
|
||||
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||
size_t len = 0;
|
||||
/* rsync's %C is the transfer checksum, which is always seeded with 0 (it is
|
||||
@@ -420,10 +329,9 @@ char* change_render_format(const char* format, const Config* config, const Chang
|
||||
if (event->checksum_known) {
|
||||
ok = strbuf_append(&line, event->checksum);
|
||||
} else {
|
||||
/* rsync pads a non-regular / untransferred / `none` entry with spaces;
|
||||
`none` renders as a blank 2-character column. */
|
||||
/* rsync pads a non-regular / untransferred entry with spaces. */
|
||||
ChecksumAlgo algo = out_format_checksum_algo(config);
|
||||
int width = algo == CHECKSUM_ALGO_NONE ? 2 : checksum_digest_len(algo) * 2;
|
||||
int width = checksum_digest_len(algo) * 2;
|
||||
for (int i = 0; i < width && ok; i++)
|
||||
ok = strbuf_append_char(&line, ' ');
|
||||
}
|
||||
@@ -530,24 +438,10 @@ static void print_escaped_line(FILE* stream, const char* line, bool eight_bit_ou
|
||||
void change_emit(const Config* config, const ChangeEvent* event) {
|
||||
if (event == NULL || !change_list_enabled(config))
|
||||
return;
|
||||
if (event->decision == CHANGE_UP_TO_DATE)
|
||||
return;
|
||||
bool to_stdout = config->itemize_changes || config->out_format != NULL;
|
||||
bool to_log = config->log_file != NULL && config->log_file_format != NULL;
|
||||
bool progress_active = config->show_progress || (config->info_level & LOG_INFO_PROGRESS);
|
||||
if (event->decision == CHANGE_UP_TO_DATE) {
|
||||
/* --info=name2 prints `NAME is uptodate` for entries the receiver already
|
||||
had. An itemize/out-format run reports them through its own format (or
|
||||
not at all), the progress stream has no frame for them, and neither the
|
||||
itemize nor the log-file stream previously reported an up-to-date entry,
|
||||
so nothing else here changes. */
|
||||
if (!to_stdout && (config->info_level & LOG_INFO_NAME_UPTODATE) != 0 && !progress_active) {
|
||||
char* line = change_render_name_uptodate(event);
|
||||
if (line != NULL) {
|
||||
print_escaped_line(stdout, line, config->eight_bit_output);
|
||||
free(line);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (to_stdout) {
|
||||
char* line = config->out_format != NULL
|
||||
? change_render_format(config->out_format, config, event)
|
||||
@@ -556,20 +450,6 @@ void change_emit(const Config* config, const ChangeEvent* event) {
|
||||
print_escaped_line(stdout, line, config->eight_bit_output);
|
||||
free(line);
|
||||
}
|
||||
} else if ((config->info_level & LOG_INFO_NAME) != 0 && !progress_active) {
|
||||
/* --info=name without -i/--out-format: print the updated entry's name. The
|
||||
--progress path owns the name line when progress output is active (it
|
||||
emits the same names before the progress frames), so do not duplicate.
|
||||
The transfer-root `./` line precedes the first such name. */
|
||||
if (!name_root_printed) {
|
||||
name_root_printed = true;
|
||||
fputs("./\n", stdout);
|
||||
}
|
||||
char* line = change_render_name(event);
|
||||
if (line != NULL) {
|
||||
print_escaped_line(stdout, line, config->eight_bit_output);
|
||||
free(line);
|
||||
}
|
||||
}
|
||||
if (to_log) {
|
||||
char* line = change_render_format(config->log_file_format, config, event);
|
||||
@@ -715,19 +595,6 @@ void change_emit_file_sent_bytes(const Config* config, const File* file,
|
||||
char* name = NULL;
|
||||
char* path = NULL;
|
||||
fill_event_from_file(config, file, &event, &name, &path);
|
||||
if (file->is_symlink) {
|
||||
/* Output parity (protocol 2.30.0): an unchanged symlink is silent, like
|
||||
rsync's quick check. The itemize/log stream suppresses it only when every
|
||||
attribute matches; the name stream suppresses it whenever the link target
|
||||
is unchanged (rsync names a symlink only when it relinks or creates it). */
|
||||
bool itemize_output = config->itemize_changes || config->out_format != NULL ||
|
||||
(config->log_file != NULL && config->log_file_format != NULL);
|
||||
bool suppress = itemize_output
|
||||
? itemize_is_unchanged(config, &event)
|
||||
: (event.dest.known && event.dest.existed && event.dest.target_matches);
|
||||
if (suppress)
|
||||
event.decision = CHANGE_UP_TO_DATE;
|
||||
}
|
||||
if (name != NULL && path != NULL) {
|
||||
fill_event_checksum(config, file, &event);
|
||||
change_emit(config, &event);
|
||||
@@ -743,29 +610,6 @@ void change_emit_file_sent(const Config* config, const File* file) {
|
||||
change_emit_file_sent_bytes(config, file, payload, 0);
|
||||
}
|
||||
|
||||
void change_emit_file_uptodate(const Config* config, const File* file) {
|
||||
if (file == NULL || !change_list_enabled(config))
|
||||
return;
|
||||
ChangeEvent event;
|
||||
memset(&event, 0, sizeof(event));
|
||||
event.decision = CHANGE_UP_TO_DATE;
|
||||
event.is_directory = false;
|
||||
event.is_symlink = file->is_symlink;
|
||||
event.is_special = file->is_special;
|
||||
event.is_hardlink = file->link_group != 0 && !file->link_first;
|
||||
event.symlink_target = file->symlink_target;
|
||||
event.hardlink_target = file->hardlink_target;
|
||||
event.size = file->data != NULL ? file->data->size : 0;
|
||||
event.dest = file->dest_state;
|
||||
char* name = NULL;
|
||||
char* path = NULL;
|
||||
fill_event_from_file(config, file, &event, &name, &path);
|
||||
if (name != NULL && path != NULL)
|
||||
change_emit(config, &event);
|
||||
free(name);
|
||||
free(path);
|
||||
}
|
||||
|
||||
void change_emit_dir_sent(const Config* config, const File* file) {
|
||||
if (file == NULL || !change_list_enabled(config))
|
||||
return;
|
||||
@@ -779,19 +623,6 @@ void change_emit_dir_sent(const Config* config, const File* file) {
|
||||
char* name = NULL;
|
||||
char* path = NULL;
|
||||
fill_event_from_file(config, file, &event, &name, &path);
|
||||
/* Output parity (protocol 2.30.0): suppress a directory rsync would leave
|
||||
silent. The itemize/log stream suppresses it only when every attribute
|
||||
matches (`.d.........`); the name stream suppresses any pre-existing
|
||||
directory (rsync names a directory only when it is created). */
|
||||
bool itemize_output = config->itemize_changes || config->out_format != NULL ||
|
||||
(config->log_file != NULL && config->log_file_format != NULL);
|
||||
bool suppress = itemize_output ? itemize_is_unchanged(config, &event)
|
||||
: (event.dest.known && event.dest.existed);
|
||||
/* The transfer root's line is an unconditional FastSync residual (rsync keys
|
||||
it off the root's own attribute change); keep emitting it. */
|
||||
bool is_root = event.name != NULL && event.name[0] == '\0';
|
||||
if (suppress && !is_root)
|
||||
event.decision = CHANGE_UP_TO_DATE;
|
||||
if (name != NULL && path != NULL)
|
||||
change_emit(config, &event);
|
||||
free(name);
|
||||
|
||||
@@ -102,13 +102,4 @@ void change_emit_file_sent(const Config* config, const File* file);
|
||||
/* Build and emit a CHANGE_SENT event for an explicit directory entry (-d). */
|
||||
void change_emit_dir_sent(const Config* config, const File* file);
|
||||
|
||||
/* Build and emit a CHANGE_UP_TO_DATE event for a file the receiver already had.
|
||||
* With --info=name2 it renders rsync's "NAME is uptodate" line (no output
|
||||
* otherwise). */
|
||||
void change_emit_file_uptodate(const Config* config, const File* file);
|
||||
|
||||
/* Reset the lazy transfer-root `./` line emitted ahead of the first
|
||||
* --info=name entry. Call once at the start of a transfer. */
|
||||
void change_reset_name_root(void);
|
||||
|
||||
#endif
|
||||
|
||||
+69
-365
@@ -23,7 +23,6 @@
|
||||
#include <langinfo.h>
|
||||
#include <limits.h>
|
||||
#include <locale.h>
|
||||
#include <math.h>
|
||||
#include <time.h>
|
||||
#include <signal.h>
|
||||
#include <stdbool.h>
|
||||
@@ -54,26 +53,13 @@ bool client_abort_pending(void) {
|
||||
}
|
||||
|
||||
#ifndef FASTSYNC_TEST_BUILD
|
||||
/* SIG_DFL disposition used by the handler's "not armed" fallback. It is built
|
||||
* once at load time so the handler can restore the default action with
|
||||
* sigaction(2) -- which is async-signal-safe -- instead of signal(3), which is
|
||||
* not. The zero-initialized sa_mask is the empty set. */
|
||||
static const struct sigaction client_default_action = {
|
||||
.sa_handler = SIG_DFL,
|
||||
.sa_flags = 0,
|
||||
};
|
||||
|
||||
/* Signal handler: perform NO work beyond storing the flag. Logging, protocol
|
||||
* I/O and the STATUS_ABORT frame are all done later on the normal send path,
|
||||
* which is not async-signal-safe. When no transfer is armed, restore the
|
||||
* default disposition (async-signal-safe sigaction) and re-raise so local-only
|
||||
* modes remain interruptible. The handler deliberately stays installed while a
|
||||
* transfer is armed -- rather than using SA_RESETHAND -- so a second Ctrl-C
|
||||
* during the graceful abort keeps setting the flag instead of hard-killing the
|
||||
* process mid-cleanup. */
|
||||
* which is not async-signal-safe. When no transfer is armed, fall back to the
|
||||
* default action so local-only modes remain interruptible. */
|
||||
static void client_signal_handler(int signo) {
|
||||
if (!client_abort_armed) {
|
||||
sigaction(signo, &client_default_action, NULL);
|
||||
signal(signo, SIG_DFL);
|
||||
raise(signo);
|
||||
return;
|
||||
}
|
||||
@@ -170,26 +156,20 @@ static int set_positive_int_option(int* dest, const char* value, const char* opt
|
||||
* name is a hard error with rsync's exit code 4, never a silent no-op. */
|
||||
static int set_compression_choice(Config* config, const char* value) {
|
||||
if (!value) {
|
||||
config->cli.cli_exit_code = 4;
|
||||
config->cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
int algo;
|
||||
if (strcasecmp(value, "auto") == 0) {
|
||||
algo = compression_choice_resolve();
|
||||
if (algo < 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
|
||||
config->cli.cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
} else {
|
||||
if (strcasecmp(value, "auto") == 0)
|
||||
algo = (int)compression_negotiate_default();
|
||||
else
|
||||
algo = compression_algo_from_name(value);
|
||||
}
|
||||
if (algo < 0) {
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, "
|
||||
"lz4, zlib, zlibx, none or auto",
|
||||
value);
|
||||
config->cli.cli_exit_code = 4;
|
||||
config->cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
const char* canonical = compression_algo_name((CompressionAlgo)algo);
|
||||
@@ -226,7 +206,7 @@ static int resolve_checksum_name(const char* name, size_t len, int* out) {
|
||||
* resolves to FastSync's negotiated default (xxh128). */
|
||||
static int set_checksum_choice(Config* config, const char* value) {
|
||||
if (!value) {
|
||||
config->cli.cli_exit_code = 4;
|
||||
config->cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
const char* comma = strchr(value, ',');
|
||||
@@ -244,28 +224,19 @@ static int set_checksum_choice(Config* config, const char* value) {
|
||||
"--checksum-choice '%s' is invalid; FastSync supports xxh64 (or xxhash), xxh128, "
|
||||
"xxh3, md5, md4, sha1, none or auto, optionally as 'transfer,pre-transfer'",
|
||||
value);
|
||||
config->cli.cli_exit_code = 4;
|
||||
config->cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
int negotiated = -1;
|
||||
if (rc1 == 1 || rc2 == 1) {
|
||||
negotiated = checksum_choice_resolve();
|
||||
if (negotiated < 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
|
||||
config->cli.cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
ChecksumAlgo negotiated = checksum_negotiate_default();
|
||||
if (rc1 == 1)
|
||||
transfer = negotiated;
|
||||
transfer = (int)negotiated;
|
||||
if (!name2)
|
||||
pre = transfer;
|
||||
else if (rc2 == 1)
|
||||
pre = negotiated;
|
||||
pre = (int)negotiated;
|
||||
|
||||
config->checksum_algo = pre;
|
||||
config->cli.checksum_transfer_algo = transfer;
|
||||
config->cli.checksum_choice_set = true;
|
||||
config->checksum_transfer_algo = transfer;
|
||||
/* rsync: "none" for the transfer checksum forces --whole-file. */
|
||||
if (transfer == (int)CHECKSUM_ALGO_NONE)
|
||||
config->whole_file = true;
|
||||
@@ -433,10 +404,12 @@ static int set_stderr_mode(const char* value) {
|
||||
log_set_stderr_mode(LOG_STDERR_ERRORS);
|
||||
else if (strcmp(value, "all") == 0 || strcmp(value, "a") == 0)
|
||||
log_set_stderr_mode(LOG_STDERR_ALL);
|
||||
else if (strcmp(value, "client") == 0 || strcmp(value, "c") == 0)
|
||||
log_set_stderr_mode(LOG_STDERR_CLIENT);
|
||||
else {
|
||||
log_message(LOG_LEVEL_ERROR, "--stderr must be errors, all, or client");
|
||||
else if (strcmp(value, "client") == 0 || strcmp(value, "c") == 0) {
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"--stderr=client is not supported: FastSync has no client message channel");
|
||||
return -1;
|
||||
} else {
|
||||
log_message(LOG_LEVEL_ERROR, "--stderr must be errors or all");
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
@@ -515,8 +488,9 @@ static bool split_flag_level(const char* token, char* name, size_t name_size, in
|
||||
* of rsync's `symsafe`, `hlink`, and `own`. */
|
||||
static bool is_accepted_debug_category(const char* name) {
|
||||
static const char* const categories[] = {
|
||||
"acl", "backup", "bind", "chdir", "cmd", "connect", "dup", "exit", "fuzzy",
|
||||
"genr", "hl", "hlink", "iconv", "nstr", "own", "owner", "time",
|
||||
"acl", "backup", "bind", "chdir", "cmd", "connect", "del", "deltasum",
|
||||
"dup", "exit", "filter", "flist", "fuzzy", "genr", "hash", "hl",
|
||||
"hlink", "iconv", "nstr", "own", "owner", "recv", "send", "time",
|
||||
};
|
||||
for (size_t i = 0; i < sizeof(categories) / sizeof(categories[0]); i++) {
|
||||
if (strcmp(name, categories[i]) == 0)
|
||||
@@ -527,9 +501,7 @@ static bool is_accepted_debug_category(const char* name) {
|
||||
|
||||
static bool is_accepted_info_category(const char* name) {
|
||||
static const char* const categories[] = {
|
||||
"backup",
|
||||
"syms",
|
||||
"symsafe",
|
||||
"backup", "del", "flist", "mount", "nonreg", "progress", "remove", "syms", "symsafe",
|
||||
};
|
||||
for (size_t i = 0; i < sizeof(categories) / sizeof(categories[0]); i++) {
|
||||
if (strcmp(name, categories[i]) == 0)
|
||||
@@ -580,18 +552,6 @@ static int parse_debug_flags(const char* value, Config* config) {
|
||||
flag = LOG_DEBUG_PACK;
|
||||
} else if (strcmp(name, "util") == 0) {
|
||||
flag = LOG_DEBUG_UTIL;
|
||||
} else if (strcmp(name, "flist") == 0) {
|
||||
flag = LOG_DEBUG_FLIST;
|
||||
} else if (strcmp(name, "del") == 0) {
|
||||
flag = LOG_DEBUG_DEL;
|
||||
} else if (strcmp(name, "hash") == 0 || strcmp(name, "deltasum") == 0) {
|
||||
flag = LOG_DEBUG_HASH;
|
||||
} else if (strcmp(name, "recv") == 0) {
|
||||
flag = LOG_DEBUG_RECV;
|
||||
} else if (strcmp(name, "filter") == 0) {
|
||||
flag = LOG_DEBUG_FILTER;
|
||||
} else if (strcmp(name, "send") == 0) {
|
||||
flag = LOG_DEBUG_SEND;
|
||||
} else if (is_accepted_debug_category(name)) {
|
||||
continue;
|
||||
} else {
|
||||
@@ -648,41 +608,14 @@ static int parse_info_flags(const char* value, Config* config) {
|
||||
free(flags);
|
||||
return 1;
|
||||
}
|
||||
if (strcmp(name, "copy") == 0)
|
||||
if (strcmp(name, "copy") == 0 || strcmp(name, "name") == 0)
|
||||
flag = LOG_INFO_COPY;
|
||||
else if (strcmp(name, "name") == 0) {
|
||||
/* name level 2 adds rsync's "is uptodate" lines. */
|
||||
if (level == 0)
|
||||
parsed &= ~(uint32_t)(LOG_INFO_NAME | LOG_INFO_NAME_UPTODATE);
|
||||
else {
|
||||
parsed |= LOG_INFO_NAME;
|
||||
if (level >= 2)
|
||||
parsed |= LOG_INFO_NAME_UPTODATE;
|
||||
else
|
||||
parsed &= ~(uint32_t)LOG_INFO_NAME_UPTODATE;
|
||||
}
|
||||
continue;
|
||||
} else if (strcmp(name, "misc") == 0)
|
||||
else if (strcmp(name, "misc") == 0)
|
||||
flag = LOG_INFO_MISC;
|
||||
else if (strcmp(name, "skip") == 0)
|
||||
flag = LOG_INFO_SKIP;
|
||||
else if (strcmp(name, "stats") == 0) {
|
||||
else if (strcmp(name, "stats") == 0)
|
||||
flag = LOG_INFO_STATS;
|
||||
/* `--info=stats` requests the same transfer-statistics block as
|
||||
`--stats`; `--info=stats0` turns it back off. */
|
||||
config->stats = level > 0;
|
||||
} else if (strcmp(name, "del") == 0)
|
||||
flag = LOG_INFO_DEL;
|
||||
else if (strcmp(name, "remove") == 0)
|
||||
flag = LOG_INFO_REMOVE;
|
||||
else if (strcmp(name, "flist") == 0)
|
||||
flag = LOG_INFO_FLIST;
|
||||
else if (strcmp(name, "nonreg") == 0)
|
||||
flag = LOG_INFO_NONREG;
|
||||
else if (strcmp(name, "mount") == 0)
|
||||
flag = LOG_INFO_MOUNT;
|
||||
else if (strcmp(name, "progress") == 0)
|
||||
flag = LOG_INFO_PROGRESS;
|
||||
else if (is_accepted_info_category(name))
|
||||
continue;
|
||||
else {
|
||||
@@ -960,18 +893,8 @@ static const OptionEntry OPTION_TABLE[] = {
|
||||
/* rsync -r/--recursive: FastSync is always recursive, so this is a
|
||||
* faithful no-op (accepted silently, never consumes an argument). */
|
||||
{"--recursive", "-r", OPT_NOOP, 0},
|
||||
/* rsync's incremental-recursion scan-mode switch. FastSync always performs
|
||||
* a single full recursive scan, so both spellings are accepted as no-ops:
|
||||
* the destination is identical whichever mode the caller requests.
|
||||
* --no-inc-recursive is handled before the generic --no-* negation branch
|
||||
* (see cli_handle_pre_negation) but is registered here for discoverability. */
|
||||
{"--inc-recursive", NULL, OPT_NOOP, 0},
|
||||
{"--no-inc-recursive", NULL, OPT_NOOP, 0},
|
||||
{"--update", "-u", OPT_FLAG, offsetof(Config, update)},
|
||||
/* rsync's --old-args: accepted for CLI compatibility as a documented no-op
|
||||
* (the remote server path is always safely quoted; see usage.c). It is
|
||||
* recognized but stores no Config field. */
|
||||
{"--old-args", NULL, OPT_NOOP, 0},
|
||||
{"--old-args", NULL, OPT_FLAG, offsetof(Config, old_args)},
|
||||
{"--rsh", "-e", OPT_STRING, offsetof(Config, rsh_command)},
|
||||
{"--blocking-io", NULL, OPT_FLAG, offsetof(Config, blocking_io)},
|
||||
{"--links", "-l", OPT_FLAG, offsetof(Config, follow_symlinks)},
|
||||
@@ -1033,12 +956,6 @@ static const OptionEntry OPTION_TABLE[] = {
|
||||
{"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)},
|
||||
{"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)},
|
||||
{"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)},
|
||||
/* FastSync-only long spelling of the late whole-tree commit, which selects
|
||||
the same timing as rsync's --delete-after in FastSync (the whole-tree
|
||||
keep-set manifest is committed only after the entire transfer succeeded).
|
||||
Plain --delete now defaults to delete-during, so this restores the old
|
||||
FastSync behavior; it maps onto the same delete_after wire field. */
|
||||
{"--delete-commit", NULL, OPT_FLAG, offsetof(Config, delete_after)},
|
||||
{"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)},
|
||||
{"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)},
|
||||
{"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)},
|
||||
@@ -1096,11 +1013,6 @@ static const OptionEntry OPTION_TABLE[] = {
|
||||
* --remote-option is parsed. --trust-sender is a local receiver policy and
|
||||
* never travels to the remote peer. */
|
||||
{"--trust-sender", NULL, OPT_FLAG, offsetof(Config, trust_sender)},
|
||||
/* FastSync-only (not an rsync option): require a basis-hit's content to
|
||||
* match the source by whole-file digest instead of trusting rsync's
|
||||
* size+mtime quick-check. Long-only; crosses the wire so the receiver
|
||||
* performs the extra read/hash. */
|
||||
{"--verify-basis", NULL, OPT_FLAG, offsetof(Config, verify_basis)},
|
||||
};
|
||||
|
||||
/* Only boolean options with no required argument are safe to negate. */
|
||||
@@ -1143,7 +1055,6 @@ static const NegatableOption NEGATABLE_OPTIONS[] = {
|
||||
{"xattrs", "X", offsetof(Config, preserve_xattrs)},
|
||||
{"acls", "A", offsetof(Config, preserve_acls)},
|
||||
{"fake-super", NULL, offsetof(Config, fake_super)},
|
||||
{"verify-basis", NULL, offsetof(Config, verify_basis)},
|
||||
};
|
||||
|
||||
static bool opt_is(const char* arg, const char* name, const char* alias) {
|
||||
@@ -1204,12 +1115,12 @@ static int apply_negation(Config* config, const char* arg) {
|
||||
config->preserve_times = false;
|
||||
config->preserve_owner = false;
|
||||
config->preserve_group = false;
|
||||
config->cli.metadata_explicitly_disabled = true;
|
||||
config->metadata_explicitly_disabled = true;
|
||||
/* --no-preserve is an explicit opt-out of the whole bundle: record it so
|
||||
* the --incremental/--delta auto-preserve in cli_finalize_config does not
|
||||
* silently re-enable perms/times. */
|
||||
config->cli.preserve_perms_explicit_off = true;
|
||||
config->cli.preserve_times_explicit_off = true;
|
||||
config->preserve_perms_explicit_off = true;
|
||||
config->preserve_times_explicit_off = true;
|
||||
return 0;
|
||||
}
|
||||
*(bool*)((char*)config + entry->offset) = false;
|
||||
@@ -1217,9 +1128,9 @@ static int apply_negation(Config* config, const char* arg) {
|
||||
* auto-preserve the OTHER attribute without undoing this one. A later
|
||||
* -p/-t sets the attribute directly; this flag only gates the implication. */
|
||||
if (entry->offset == offsetof(Config, preserve_perms))
|
||||
config->cli.preserve_perms_explicit_off = true;
|
||||
config->preserve_perms_explicit_off = true;
|
||||
else if (entry->offset == offsetof(Config, preserve_times))
|
||||
config->cli.preserve_times_explicit_off = true;
|
||||
config->preserve_times_explicit_off = true;
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -1249,13 +1160,7 @@ static int apply_table_option(Config* config, const OptionEntry* entry, const ch
|
||||
void* field = (char*)config + entry->offset;
|
||||
switch (entry->kind) {
|
||||
case OPT_FLAG:
|
||||
/* -x/--one-file-system is repeatable in rsync: `-xx` increments the level so
|
||||
the scanner drops mount-point directories instead of recreating them
|
||||
empty. Everything else is a plain boolean. */
|
||||
if (entry->offset == offsetof(Config, one_file_system))
|
||||
(*(int*)field)++;
|
||||
else
|
||||
*(bool*)field = true;
|
||||
*(bool*)field = true;
|
||||
return 0;
|
||||
case OPT_NOOP:
|
||||
return 0;
|
||||
@@ -1351,9 +1256,10 @@ static bool cli_handle_pre_negation(CliParseCtx* ctx) {
|
||||
return true;
|
||||
}
|
||||
/* "--no-msgs2stderr" is the deprecated spelling of --stderr=client (rsync
|
||||
* 3.4.1); the client-message channel now exists, so it maps to `client`. */
|
||||
* 3.4.1). FastSync has no separate client message channel, so the closest
|
||||
* supported mode is the errors-only default. */
|
||||
if (strcmp(arg, "--no-msgs2stderr") == 0)
|
||||
return set_stderr_mode("client") == 0;
|
||||
return set_stderr_mode("errors") == 0;
|
||||
/* "--no-motd" is a real rsync option name (client-side daemon MOTD display
|
||||
* suppression), not a negation of a "--motd" flag, so it is handled before
|
||||
* the generic --no-* negation branch. */
|
||||
@@ -1391,12 +1297,6 @@ static bool cli_handle_pre_negation(CliParseCtx* ctx) {
|
||||
ctx->no_delta = true;
|
||||
else if (strcmp(arg, "--no-incremental") == 0)
|
||||
ctx->no_incremental = true;
|
||||
/* Real rsync option names that merely start with "--no-" and are inert
|
||||
* no-ops (e.g. --no-inc-recursive) are registered as OPT_NOOP entries;
|
||||
* accept them before the generic negation table would reject the name. */
|
||||
const OptionEntry* noop = find_table_option(arg);
|
||||
if (noop && noop->kind == OPT_NOOP)
|
||||
return true;
|
||||
if (apply_negation(config, arg) != 0) {
|
||||
ctx->exit_code = -1;
|
||||
return true;
|
||||
@@ -1453,7 +1353,7 @@ static bool cli_handle_range_time_options(CliParseCtx* ctx) {
|
||||
ctx->exit_code = -1;
|
||||
return true;
|
||||
}
|
||||
config->cli.stop_at_set = true;
|
||||
config->stop_at_set = true;
|
||||
return true;
|
||||
}
|
||||
if (strcmp(arg, "--stop-at") == 0) {
|
||||
@@ -1468,7 +1368,7 @@ static bool cli_handle_range_time_options(CliParseCtx* ctx) {
|
||||
ctx->exit_code = -1;
|
||||
return true;
|
||||
}
|
||||
config->cli.stop_at_set = true;
|
||||
config->stop_at_set = true;
|
||||
return true;
|
||||
}
|
||||
const char* threads_prefix = "--compress-threads=";
|
||||
@@ -1538,8 +1438,6 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
|
||||
ctx->exit_code = -1;
|
||||
return true;
|
||||
}
|
||||
if (entry->offset == offsetof(Config, compression_level))
|
||||
config->cli.compression_level_set = true;
|
||||
if (entry->offset == offsetof(Config, chmod_spec)) {
|
||||
mode_t ignored;
|
||||
if (!chmod_apply(0, config->chmod_spec, &ignored)) {
|
||||
@@ -1552,7 +1450,7 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
|
||||
defaults to 127.0.0.1, so a value check cannot distinguish it). Used
|
||||
by --dry-run to route an explicit remote target to the server. */
|
||||
if (entry->offset == offsetof(Config, server_host))
|
||||
config->cli.server_host_set = true;
|
||||
config->server_host_set = true;
|
||||
}
|
||||
} else if (apply_table_option(config, entry, NULL) != 0) {
|
||||
ctx->exit_code = -1;
|
||||
@@ -1566,9 +1464,7 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
|
||||
if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX)
|
||||
config->per_dir_filter_count++;
|
||||
/* A delete-timing flag selects when --delete removes extras, so it
|
||||
implies --delete exactly like the rsync options do. --delete-commit (the
|
||||
FastSync-only late-commit spelling) is mapped onto delete_after and so is
|
||||
covered here too. */
|
||||
implies --delete exactly like the rsync options do. */
|
||||
if (entry->offset == offsetof(Config, delete_before) ||
|
||||
entry->offset == offsetof(Config, delete_during) ||
|
||||
entry->offset == offsetof(Config, delete_delay) ||
|
||||
@@ -1826,7 +1722,6 @@ static bool cli_handle_transfer_flags(CliParseCtx* ctx) {
|
||||
return true;
|
||||
}
|
||||
config->compression_level = (int)level;
|
||||
config->cli.compression_level_set = true;
|
||||
log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level);
|
||||
ctx->i++;
|
||||
}
|
||||
@@ -1890,7 +1785,7 @@ static int set_server_port_option(Config* config, const char* value, const char*
|
||||
return -1;
|
||||
}
|
||||
config->server_port = port;
|
||||
config->cli.server_port_set = true;
|
||||
config->server_port_set = true;
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -1924,130 +1819,22 @@ static int set_log_file_option(Config* config, const char* log_path) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Faithful port of rsync 3.4.1's `parse_size_arg(bwlimit_arg, 'K', "bwlimit",
|
||||
* 512, -1, True)`: a default KiB suffix, binary (1024) multipliers unless a
|
||||
* `b`/`B` decimal suffix or explicit `iB` is given, an optional decimal
|
||||
* fraction, the P/T/G/M/K suffixes, and the special rules that a value of 0
|
||||
* means "no limit" while any other value below 512 bytes is rejected. The
|
||||
* parsed byte count is then quantized to whole KiB exactly like rsync's
|
||||
* `bwlimit = (size + 512) / 1024`. Returns 0 on success, -1 on a parse error. */
|
||||
static int parse_bwlimit_value(const char* value, unsigned long long* bytes_per_sec_out) {
|
||||
if (!value || !bytes_per_sec_out)
|
||||
return -1;
|
||||
const char* arg = value;
|
||||
int reps;
|
||||
long long mult;
|
||||
while (*arg >= '0' && *arg <= '9')
|
||||
arg++;
|
||||
if (*arg != '\0' && (*arg == '.' || *arg == localeconv()->decimal_point[0]))
|
||||
for (arg++; *arg >= '0' && *arg <= '9'; arg++) {
|
||||
}
|
||||
|
||||
char suffix = *arg && *arg != '+' && *arg != '-' ? *arg++ : 'K';
|
||||
switch (suffix) {
|
||||
case 'b':
|
||||
case 'B':
|
||||
reps = 0;
|
||||
break;
|
||||
case 'k':
|
||||
case 'K':
|
||||
reps = 1;
|
||||
break;
|
||||
case 'm':
|
||||
case 'M':
|
||||
reps = 2;
|
||||
break;
|
||||
case 'g':
|
||||
case 'G':
|
||||
reps = 3;
|
||||
break;
|
||||
case 't':
|
||||
case 'T':
|
||||
reps = 4;
|
||||
break;
|
||||
case 'p':
|
||||
case 'P':
|
||||
reps = 5;
|
||||
break;
|
||||
default:
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is invalid", value);
|
||||
return -1;
|
||||
}
|
||||
if (*arg == 'b' || *arg == 'B') {
|
||||
mult = 1000;
|
||||
arg++;
|
||||
} else if (*arg == '\0' || *arg == '+' || *arg == '-') {
|
||||
mult = 1024;
|
||||
} else if ((arg[0] == 'i' || arg[0] == 'I') && (arg[1] == 'b' || arg[1] == 'B')) {
|
||||
mult = 1024;
|
||||
arg += 2;
|
||||
} else {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is invalid", value);
|
||||
return -1;
|
||||
}
|
||||
|
||||
long long base = 1;
|
||||
for (int i = 0; i < reps; i++) {
|
||||
if (base > LLONG_MAX / mult) {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too large", value);
|
||||
return -1;
|
||||
}
|
||||
base *= mult;
|
||||
}
|
||||
/* rsync multiplies the numeric prefix (atof) by mult^reps in a signed
|
||||
* ssize_t, which is undefined on overflow. Scale in double and range-check
|
||||
* before converting, so a huge value is rejected as "too large" (where
|
||||
* rsync's overflow happens to land on a negative result) without invoking
|
||||
* signed-overflow UB. */
|
||||
double scaled = (double)base * strtod(value, NULL);
|
||||
/* (double)LLONG_MAX rounds up to 2^63, which is itself out of range for the
|
||||
* cast, so reject at >= that bound; LLONG_MIN == -2^63 is exactly
|
||||
* representable and thus castable, so the lower bound stays strict. */
|
||||
if (!isfinite(scaled) || scaled >= (double)LLONG_MAX || scaled < (double)LLONG_MIN) {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too large", value);
|
||||
return -1;
|
||||
}
|
||||
long long size = (long long)scaled;
|
||||
if ((*arg == '+' || *arg == '-') && arg[1] == '1' && arg != value) {
|
||||
/* The only form accepted here is "+1"/"-1" (a longer number leaves a
|
||||
trailing byte and is rejected below), so apply the delta directly and
|
||||
guard the one overflow direction. */
|
||||
if (*arg == '+') {
|
||||
if (size == LLONG_MAX) {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too large", value);
|
||||
return -1;
|
||||
}
|
||||
size += 1;
|
||||
} else {
|
||||
size -= 1;
|
||||
}
|
||||
arg += 2;
|
||||
}
|
||||
if (*arg != '\0' || size < 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is %s", value, size < 0 ? "too large" : "invalid");
|
||||
return -1;
|
||||
}
|
||||
if (size != 0 && size < 512) {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too small (min: 512 or 0 for unlimited)", value);
|
||||
return -1;
|
||||
}
|
||||
long long kib = size == 0 ? 0 : (size + 512) / 1024;
|
||||
if (kib > (long long)(ULLONG_MAX / 1024)) {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too large", value);
|
||||
return -1;
|
||||
}
|
||||
*bytes_per_sec_out = (unsigned long long)kib * 1024;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Apply a --bwlimit value using rsync 3.4.1's units/semantics. Returns 0 on
|
||||
* success, -1 on error. */
|
||||
/* Apply a --bwlimit value (kilobytes per second). Returns 0 on success, -1 on
|
||||
* error. */
|
||||
static int set_bwlimit_option(const char* value) {
|
||||
unsigned long long bytes_per_sec;
|
||||
if (parse_bwlimit_value(value, &bytes_per_sec) != 0)
|
||||
unsigned long long kbps;
|
||||
if (parse_ull_arg(value, &kbps, "--bwlimit") != 0)
|
||||
return -1;
|
||||
io_set_bwlimit(bytes_per_sec);
|
||||
log_info_message(LOG_INFO_MISC, "Set bandwidth limit to %llu KB/s", bytes_per_sec / 1024);
|
||||
if (kbps == 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit must be a positive integer");
|
||||
return -1;
|
||||
}
|
||||
if (kbps > ULLONG_MAX / 1024) {
|
||||
log_message(LOG_LEVEL_ERROR, "--bwlimit value too large");
|
||||
return -1;
|
||||
}
|
||||
io_set_bwlimit(kbps * 1024);
|
||||
log_info_message(LOG_INFO_MISC, "Set bandwidth limit to %llu KB/s", kbps);
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -2621,43 +2408,7 @@ static bool cli_handle_outbuf_option(CliParseCtx* ctx) {
|
||||
* load --files-from once every argument has been seen. Returns 0 on success,
|
||||
* -1 on error. */
|
||||
static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) {
|
||||
/* An explicit --debug=FLAGS enables the debug log level by itself (rsync
|
||||
behaviour); -v enables every other INFO-level message. */
|
||||
bool debug_enabled = verbose || config->debug_level != 0;
|
||||
set_log_level(config->quiet ? LOG_LEVEL_ERROR
|
||||
: (debug_enabled ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
|
||||
/* rsync parity: --delay-updates implies --delete-after. Every staged file is
|
||||
published first and only then are extras removed. Normalize onto the
|
||||
existing delete_after wire bool (no new wire field), overriding any other
|
||||
explicit timing exactly as rsync does; without --delete there is no
|
||||
deletion, so no timing is set (and the wire config stays valid). */
|
||||
if (config->delay_updates && config->use_delete) {
|
||||
config->delete_before = false;
|
||||
config->delete_during = false;
|
||||
config->delete_delay = false;
|
||||
config->delete_after = true;
|
||||
}
|
||||
/* rsync's plain --delete defaults to delete-during (--del): each directory's
|
||||
extras are removed as that directory is processed, so space is freed
|
||||
progressively and a tight destination never has to hold the whole old+new
|
||||
tree at once. The late whole-tree commit FastSync historically used is
|
||||
still selected explicitly by --delete-after or by the FastSync-only long
|
||||
spelling --delete-commit (an exact alias for --delete-after). Resolve the
|
||||
default on the client, before validation and before the config crosses the
|
||||
wire, so exactly one timing flag is ever set; an explicit timing (including
|
||||
--delete-commit) always wins. */
|
||||
if (config->use_delete && !config->delete_before && !config->delete_during &&
|
||||
!config->delete_delay && !config->delete_after)
|
||||
config->delete_during = true;
|
||||
/* rsync parity: --partial-dir=DIR chooses where an interrupted transfer's
|
||||
partial file is kept, so it implies --partial. rsync applies the
|
||||
implication after option parsing, so it wins over an explicit --no-partial
|
||||
regardless of the order the two options appear in (verified on rsync
|
||||
3.4.1). --inplace is the exception: the destination file is written in
|
||||
place with no partial/temp staging, so the partial machinery is bypassed
|
||||
and the implication is skipped to leave --inplace behavior untouched. */
|
||||
if (config->partial_dir && !config->inplace)
|
||||
config->partial = true;
|
||||
set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
|
||||
if (config->compress_choice) {
|
||||
int algo = compression_algo_from_name(config->compress_choice);
|
||||
if (algo >= 0) {
|
||||
@@ -2665,49 +2416,14 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
|
||||
config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE);
|
||||
}
|
||||
}
|
||||
/* A bare -z (no --compress-choice) resolves like rsync's "auto": the
|
||||
* RSYNC_COMPRESS_LIST preference list first, then the compiled-in order. A
|
||||
* list that names no supported codec is rsync's failed negotiation (exit 4). */
|
||||
if (config->use_compression && !config->compress_choice) {
|
||||
int resolved = compression_choice_resolve();
|
||||
if (resolved < 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
|
||||
config->cli.cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
config->compression_algo = resolved;
|
||||
if (resolved == (int)COMPRESSION_ALGO_NONE)
|
||||
config->use_compression = false;
|
||||
}
|
||||
/* Apply rsync's per-codec compression level: an explicit --compress-level is
|
||||
* clamped to the codec's range, otherwise the codec's own default is used. */
|
||||
if (config->use_compression) {
|
||||
CompressionAlgo algo = (CompressionAlgo)config->compression_algo;
|
||||
config->compression_level = config->cli.compression_level_set
|
||||
? compression_clamp_level(algo, config->compression_level)
|
||||
: compression_default_level(algo);
|
||||
log_debug_message(LOG_DEBUG_UTIL, "Client compression: %s (level %d)",
|
||||
compression_algo_name(algo), config->compression_level);
|
||||
}
|
||||
/* The negotiated checksum is always resolved (rsync negotiates one for the
|
||||
* delta strong sum even without --checksum): RSYNC_CHECKSUM_LIST first, then
|
||||
* the compiled-in order. An explicit --checksum-choice already set it. */
|
||||
if (!config->cli.checksum_choice_set) {
|
||||
int resolved = checksum_choice_resolve();
|
||||
if (resolved < 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
|
||||
config->cli.cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
config->checksum_algo = resolved;
|
||||
config->cli.checksum_transfer_algo = resolved;
|
||||
}
|
||||
if (config->use_compression && config->compression_algo == (int)COMPRESSION_ALGO_NONE)
|
||||
config->compression_algo = (int)compression_negotiate_default();
|
||||
/* rsync parity: "none" as the pre-transfer checksum cannot be combined with
|
||||
* --checksum (exit 4). The check runs here because --checksum may appear on
|
||||
* either side of --checksum-choice. */
|
||||
if (config->checksum && config->checksum_algo == (int)CHECKSUM_ALGO_NONE) {
|
||||
log_message(LOG_LEVEL_ERROR, "Invalid checksum-choice for --checksum: none");
|
||||
config->cli.cli_exit_code = 4;
|
||||
config->cli_exit_code = 4;
|
||||
return -1;
|
||||
}
|
||||
|
||||
@@ -2786,11 +2502,11 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
|
||||
* explicitly negated them (--no-perms/--no-times/--no-preserve). This runs
|
||||
* BEFORE the derived use_metadata bit so the transport frame is still sent
|
||||
* for the incremental/delta handshake even when both attributes were negated
|
||||
* via --no-preserve (cli.metadata_explicitly_disabled handles that opt-out). */
|
||||
if (preserve_implied && !config->cli.metadata_explicitly_disabled) {
|
||||
if (!config->cli.preserve_perms_explicit_off)
|
||||
* via --no-preserve (metadata_explicitly_disabled handles that opt-out). */
|
||||
if (preserve_implied && !config->metadata_explicitly_disabled) {
|
||||
if (!config->preserve_perms_explicit_off)
|
||||
config->preserve_perms = true;
|
||||
if (!config->cli.preserve_times_explicit_off)
|
||||
if (!config->preserve_times_explicit_off)
|
||||
config->preserve_times = true;
|
||||
}
|
||||
|
||||
@@ -2818,11 +2534,8 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
|
||||
* need the pre-transfer destination snapshot (new vs modified and which
|
||||
* attributes differ), so ask the receiver to report it on every per-file
|
||||
* check. This is a wire field. */
|
||||
bool progress_active =
|
||||
!config->quiet && (config->show_progress || (config->info_level & LOG_INFO_PROGRESS) != 0);
|
||||
config->report_dest_info = config->itemize_changes || config->out_format != NULL ||
|
||||
(config->log_file != NULL && config->log_file_format != NULL) ||
|
||||
progress_active;
|
||||
(config->log_file != NULL && config->log_file_format != NULL);
|
||||
/* Wire-stats parity: --stats, --progress/-P, an --out-format token that needs
|
||||
* a wire counter (%b/%c), or a dry-run --delete need the receiver's
|
||||
* end-of-transfer STATUS_STATS report. This is a wire field (protocol
|
||||
@@ -2836,17 +2549,8 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
|
||||
}
|
||||
}
|
||||
}
|
||||
/* --info=del on a real --delete run asks the receiver to report the paths it
|
||||
actually removed; the report rides the STATUS_STATS path list, so the wire
|
||||
stats frame must be negotiated too. --debug=del needs the same paths, so
|
||||
it opts into the existing report (no new wire field). */
|
||||
config->report_deletes =
|
||||
config->use_delete && !config->dry_run &&
|
||||
((config->info_level & LOG_INFO_DEL) != 0 || config->itemize_changes ||
|
||||
config->out_format != NULL || (config->debug_level & LOG_DEBUG_DEL) != 0);
|
||||
config->report_stats = config->stats || config->show_progress ||
|
||||
(config->info_level & LOG_INFO_PROGRESS) || format_needs_wire ||
|
||||
config->report_deletes || (config->dry_run && config->use_delete);
|
||||
config->report_stats = config->stats || config->show_progress || format_needs_wire ||
|
||||
(config->dry_run && config->use_delete);
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -3180,7 +2884,7 @@ int main(int argc, char* argv[]) {
|
||||
int parse_ret = parse_args(config, argc, argv, positional_args, &positional_count);
|
||||
if (parse_ret != 0) {
|
||||
if (parse_ret < 0)
|
||||
exit_code = config->cli.cli_exit_code ? config->cli.cli_exit_code : 1;
|
||||
exit_code = config->cli_exit_code ? config->cli_exit_code : 1;
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
@@ -3323,7 +3027,7 @@ int main(int argc, char* argv[]) {
|
||||
exit_code = 1;
|
||||
}
|
||||
} else if (config->use_multithreading) {
|
||||
exit_code = send_files_multithreaded(config);
|
||||
exit_code = send_files_multithreaded(&config);
|
||||
} else {
|
||||
exit_code = send_files(config);
|
||||
}
|
||||
|
||||
@@ -1,679 +0,0 @@
|
||||
#include "client_send_internal.h"
|
||||
#include "array_list.h"
|
||||
#include "change_list.h"
|
||||
#include "charset.h"
|
||||
#include "config.h"
|
||||
#include "data.h"
|
||||
#include "delta.h"
|
||||
#include "file.h"
|
||||
#include "format.h"
|
||||
#include "log.h"
|
||||
#include "protocol.h"
|
||||
#include "scanner.h"
|
||||
#include "transport_tls.h"
|
||||
#include "utils.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <time.h>
|
||||
|
||||
/* True when --dry-run should contact a receiver rather than running the
|
||||
* client-side local manifest. Any target a real run would reach over the wire
|
||||
* selects the server-contacting path: a remote (SSH host:path), a daemon
|
||||
* (host::module/path), an explicit --server-host, --server-port/--port, TLS, or
|
||||
* a source-bind --address. A plain local destination (none of these) keeps the
|
||||
* original client-side behavior, which never dials the default 127.0.0.1:8080. */
|
||||
bool dry_run_targets_server(const Config* config) {
|
||||
if (!config)
|
||||
return false;
|
||||
if (config->transport == TRANSPORT_SSH)
|
||||
return true;
|
||||
if (config->module && config->module[0] != '\0')
|
||||
return true;
|
||||
if (config->cli.server_host_set || config->cli.server_port_set)
|
||||
return true;
|
||||
if (config->use_tls)
|
||||
return true;
|
||||
if (config->address != NULL)
|
||||
return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
bool add_chunk_to_manifest(ArrayList* manifest, const Chunk* chunk) {
|
||||
if (!manifest)
|
||||
return true;
|
||||
for (int i = 0; i < chunk->element_count; i++) {
|
||||
const char* path = file_wire_path(chunk->items[i]);
|
||||
if (*path == '/')
|
||||
path++;
|
||||
char* entry = str_dup(path);
|
||||
if (!entry) {
|
||||
log_message(LOG_LEVEL_ERROR, "Failed to allocate manifest entry");
|
||||
return false;
|
||||
}
|
||||
if (!array_list_add(manifest, entry)) {
|
||||
free(entry);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Print dry-run manifest showing files that would be transferred. Returns 0 on success. */
|
||||
int send_dry_run_manifest(const Config* config) {
|
||||
int skipped = 0;
|
||||
ArrayList* missing_dest = NULL;
|
||||
if (!client_prepare_files_from(config, &missing_dest, &skipped))
|
||||
return -1;
|
||||
PreparedScanner prepared;
|
||||
if (!prepare_scanner(config, 0, &prepared)) {
|
||||
if (missing_dest)
|
||||
array_list_delete(missing_dest);
|
||||
return -1;
|
||||
}
|
||||
DirectoryScanner* scanner =
|
||||
directory_scanner_create_with_options(config->send_directory, &prepared.options);
|
||||
if (!scanner) {
|
||||
prepared_scanner_destroy(&prepared);
|
||||
if (missing_dest)
|
||||
array_list_delete(missing_dest);
|
||||
return -1;
|
||||
}
|
||||
Chunk* chunk;
|
||||
int file_count = 0;
|
||||
unsigned long long total_bytes = 0;
|
||||
char size_buffer[32];
|
||||
if (!config->quiet)
|
||||
printf("Dry run: files to be transferred\n");
|
||||
while ((chunk = directory_scanner_next(scanner)) != NULL) {
|
||||
for (int i = 0; i < chunk->element_count; i++) {
|
||||
if (!config->quiet) {
|
||||
char* escaped_path =
|
||||
output_escape(file_wire_path(chunk->items[i]), config->eight_bit_output);
|
||||
if (!escaped_path) {
|
||||
chunk_destroy(chunk);
|
||||
directory_scanner_destroy(scanner);
|
||||
prepared_scanner_destroy(&prepared);
|
||||
if (missing_dest)
|
||||
array_list_delete(missing_dest);
|
||||
return -1;
|
||||
}
|
||||
if (config->human_readable)
|
||||
printf(
|
||||
" %s (%s)\n", escaped_path,
|
||||
display_bytes(chunk->items[i]->data->size, true, size_buffer, sizeof(size_buffer)));
|
||||
else
|
||||
printf(" %s (%zu bytes)\n", escaped_path, chunk->items[i]->data->size);
|
||||
free(escaped_path);
|
||||
}
|
||||
total_bytes += chunk->items[i]->data->size;
|
||||
file_count++;
|
||||
}
|
||||
chunk_destroy(chunk);
|
||||
}
|
||||
directory_scanner_destroy(scanner);
|
||||
prepared_scanner_destroy(&prepared);
|
||||
/* --delete-missing-args: the missing entries' destination mirrors render as
|
||||
would-be deletions (rsync's dry-run also lists its *deleting lines). */
|
||||
if (missing_dest && !config->quiet) {
|
||||
for (int i = 0; i < missing_dest->size; i++) {
|
||||
char* escaped = output_escape((char*)missing_dest->items[i], config->eight_bit_output);
|
||||
printf(" %s (missing; would be deleted)\n", escaped ? escaped : "<allocation failed>");
|
||||
free(escaped);
|
||||
}
|
||||
}
|
||||
if (missing_dest)
|
||||
array_list_delete(missing_dest);
|
||||
if (!config->quiet) {
|
||||
if (config->human_readable)
|
||||
printf("Total: %d files, %s\n", file_count,
|
||||
display_bytes(total_bytes, true, size_buffer, sizeof(size_buffer)));
|
||||
else
|
||||
printf("Total: %d files, %.1f MB\n", file_count, (double)total_bytes / (double)BYTES_PER_MIB);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
char* name; /* transfer-relative name ("" == the source root) */
|
||||
mode_t mode;
|
||||
unsigned long long size;
|
||||
time_t mtime;
|
||||
long mtime_nsec;
|
||||
bool is_dir;
|
||||
bool is_symlink;
|
||||
char* link_target;
|
||||
} ListEntry;
|
||||
|
||||
static void list_entries_destroy(ListEntry* entries, size_t count) {
|
||||
if (entries == NULL)
|
||||
return;
|
||||
for (size_t i = 0; i < count; i++) {
|
||||
free(entries[i].name);
|
||||
free(entries[i].link_target);
|
||||
}
|
||||
free(entries);
|
||||
}
|
||||
|
||||
static int compare_list_entries(const void* left, const void* right) {
|
||||
const ListEntry* a = (const ListEntry*)left;
|
||||
const ListEntry* b = (const ListEntry*)right;
|
||||
return strcmp(a->name, b->name);
|
||||
}
|
||||
|
||||
/* Relative path of an entry below `root` ("" for the root itself). Mirrors
|
||||
* change_list's relative_name for list-only rendering. */
|
||||
static char* list_relative_name(const char* root, const char* full) {
|
||||
if (root == NULL || full == NULL)
|
||||
return str_dup(full != NULL ? full : "");
|
||||
size_t root_len = strlen(root);
|
||||
while (root_len > 1 && root[root_len - 1] == '/')
|
||||
root_len--;
|
||||
if (strncmp(root, full, root_len) == 0) {
|
||||
if (full[root_len] == '\0')
|
||||
return str_dup("");
|
||||
if (full[root_len] == '/')
|
||||
return str_dup(full + root_len + 1);
|
||||
}
|
||||
return str_dup(full);
|
||||
}
|
||||
|
||||
/* --list-only: print an ls-style listing of the entries that WOULD be
|
||||
* transferred and exit without contacting the server or writing anything.
|
||||
* Names are transfer-relative (rsync prints `a.txt`, `sub/b.txt`, `.`) and
|
||||
* directory entries are included. Returns 0 on success, 1 on error. */
|
||||
int send_list_only(const Config* config) {
|
||||
int skipped = 0;
|
||||
if (!files_from_list_check(config, NULL, &skipped))
|
||||
return 1;
|
||||
PreparedScanner prepared;
|
||||
if (!prepare_scanner(config, 0, &prepared))
|
||||
return 1;
|
||||
prepared.options.use_metadata = true; /* capture mode + mtime for the listing */
|
||||
prepared.options.list_dirs = true;
|
||||
DirectoryScanner* scanner =
|
||||
directory_scanner_create_with_options(config->send_directory, &prepared.options);
|
||||
if (!scanner) {
|
||||
prepared_scanner_destroy(&prepared);
|
||||
return 1;
|
||||
}
|
||||
ListEntry* entries = NULL;
|
||||
size_t count = 0;
|
||||
size_t capacity = 0;
|
||||
bool oom = false;
|
||||
|
||||
/* rsync lists the source root itself (as "."). Only when the source is a
|
||||
* directory and no --files-from subset is in effect. */
|
||||
if (config->files_from_set == NULL && config->send_directory != NULL) {
|
||||
struct stat st;
|
||||
if (stat(config->send_directory, &st) == 0 && S_ISDIR(st.st_mode)) {
|
||||
capacity = 64;
|
||||
entries = calloc(capacity, sizeof(ListEntry));
|
||||
if (entries == NULL) {
|
||||
oom = true;
|
||||
} else if ((entries[0].name = str_dup("")) == NULL) {
|
||||
/* A NULL name would be dereferenced by qsort/render: fail the listing. */
|
||||
oom = true;
|
||||
} else {
|
||||
entries[0].mode = st.st_mode;
|
||||
entries[0].mtime = st.st_mtime;
|
||||
entries[0].mtime_nsec = st.st_mtim.tv_nsec;
|
||||
entries[0].size = (unsigned long long)st.st_size;
|
||||
entries[0].is_dir = true;
|
||||
count = 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Chunk* chunk;
|
||||
while (!oom && (chunk = directory_scanner_next(scanner)) != NULL) {
|
||||
for (int i = 0; i < chunk->element_count; i++) {
|
||||
File* f = chunk->items[i];
|
||||
if (f == NULL)
|
||||
continue;
|
||||
if (count == capacity) {
|
||||
size_t new_capacity = capacity > 0 ? capacity * 2 : 64;
|
||||
if (new_capacity <= capacity) {
|
||||
oom = true;
|
||||
break;
|
||||
}
|
||||
ListEntry* grown = realloc(entries, new_capacity * sizeof(ListEntry));
|
||||
if (!grown) {
|
||||
oom = true;
|
||||
break;
|
||||
}
|
||||
entries = grown;
|
||||
memset(entries + capacity, 0, (new_capacity - capacity) * sizeof(ListEntry));
|
||||
capacity = new_capacity;
|
||||
}
|
||||
char* name = list_relative_name(config->send_directory, file_wire_path(f));
|
||||
if (!name) {
|
||||
oom = true;
|
||||
break;
|
||||
}
|
||||
mode_t mode = 0;
|
||||
time_t mtime = 0;
|
||||
long mtime_nsec = 0;
|
||||
if (f->metadata != NULL) {
|
||||
mode = f->metadata->mode;
|
||||
mtime = f->metadata->mtime_sec;
|
||||
mtime_nsec = f->metadata->mtime_nsec;
|
||||
} else {
|
||||
struct stat st;
|
||||
if (lstat(f->path, &st) == 0) {
|
||||
mode = st.st_mode;
|
||||
mtime = st.st_mtime;
|
||||
mtime_nsec = st.st_mtim.tv_nsec;
|
||||
}
|
||||
}
|
||||
entries[count].name = name;
|
||||
entries[count].mode = mode;
|
||||
entries[count].mtime = mtime;
|
||||
entries[count].mtime_nsec = mtime_nsec;
|
||||
if (f->is_symlink)
|
||||
entries[count].size = f->symlink_target != NULL ? strlen(f->symlink_target) : 0;
|
||||
else if (f->is_dir) {
|
||||
struct stat dir_st;
|
||||
entries[count].size = stat(f->path, &dir_st) == 0 ? (unsigned long long)dir_st.st_size : 0;
|
||||
} else
|
||||
entries[count].size = f->data != NULL ? f->data->size : 0;
|
||||
entries[count].is_dir = f->is_dir;
|
||||
entries[count].is_symlink = f->is_symlink;
|
||||
entries[count].link_target =
|
||||
f->is_symlink && f->symlink_target ? str_dup(f->symlink_target) : NULL;
|
||||
count++;
|
||||
}
|
||||
chunk_destroy(chunk);
|
||||
}
|
||||
bool failed = oom || directory_scanner_failed(scanner) || directory_scanner_had_io_error(scanner);
|
||||
directory_scanner_destroy(scanner);
|
||||
prepared_scanner_destroy(&prepared);
|
||||
if (failed) {
|
||||
list_entries_destroy(entries, count);
|
||||
if (oom)
|
||||
log_message(LOG_LEVEL_ERROR, "memory allocation failed while listing");
|
||||
return 1;
|
||||
}
|
||||
if (count > 1)
|
||||
qsort(entries, count, sizeof(ListEntry), compare_list_entries);
|
||||
for (size_t i = 0; i < count; i++) {
|
||||
ChangeEvent event;
|
||||
memset(&event, 0, sizeof(event));
|
||||
event.name = entries[i].name;
|
||||
event.path = entries[i].name;
|
||||
event.mode = entries[i].mode;
|
||||
event.size = entries[i].size;
|
||||
event.mtime_sec = entries[i].mtime;
|
||||
event.mtime_nsec = entries[i].mtime_nsec;
|
||||
event.is_directory = entries[i].is_dir;
|
||||
event.is_symlink = entries[i].is_symlink;
|
||||
event.symlink_target = entries[i].link_target;
|
||||
char* line = change_render_list_line(config, &event);
|
||||
if (line != NULL) {
|
||||
char* escaped = output_escape(line, config->eight_bit_output);
|
||||
printf("%s\n", escaped != NULL ? escaped : line);
|
||||
free(escaped);
|
||||
free(line);
|
||||
}
|
||||
}
|
||||
list_entries_destroy(entries, count);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Send the delete manifest to the server. Returns 0 on success, -1 on
|
||||
failure. It carries FOUR path sections (keep-set paths, protected excluded
|
||||
prefixes, --delete-missing-args exact-delete paths, and the destination-
|
||||
relative directories the sender synchronized this run) followed by the
|
||||
protocol-2.30.0 per-directory filter-rule block (`per_dir_rules`, the rules
|
||||
the scan compiled from each directory's merge files).
|
||||
When --delete-excluded is given `protected` is empty: excluded destination
|
||||
mirrors are then ordinary extras and are removed. When
|
||||
--delete-missing-args is active `missing_args` holds the destination mirrors
|
||||
of missing --files-from entries: each is an explicit receiver-side deletion
|
||||
request, independent of the extras walk. `synced_dirs` confines the extras
|
||||
walk to entries directly inside a synchronized directory. A NULL
|
||||
keep-set / protected / missing / dirs list transmits an empty section. All
|
||||
four sections are unbounded on the sender; the receiver enforces
|
||||
MAX_MANIFEST_ENTRIES per section and a single MAX_MANIFEST_BYTES budget
|
||||
shared across the sections, rejecting (with STATUS_ERROR) an over-budget
|
||||
frame. A heavily filtered source whose exclusion list is large therefore
|
||||
fails the run cleanly on the receiver rather than being truncated. */
|
||||
int send_delete_manifest(int fd, ArrayList* manifest, ArrayList* protected_prefixes,
|
||||
ArrayList* size_skipped, ArrayList* missing_args, ArrayList* synced_dirs,
|
||||
const FilterRuleList* per_dir_rules) {
|
||||
if (!send_status(fd, STATUS_MANIFEST))
|
||||
return -1;
|
||||
int keep_count = manifest ? manifest->size : 0;
|
||||
if (!send_int(fd, keep_count))
|
||||
return -1;
|
||||
for (int i = 0; i < keep_count; i++) {
|
||||
if (!send_wire_str(fd, (char*)manifest->items[i]))
|
||||
return -1;
|
||||
}
|
||||
/* The receiver has ONE protected-prefix section; filter-excluded prefixes
|
||||
(dropped under --delete-excluded) and size-pruned prefixes (always
|
||||
protected) are concatenated into it. */
|
||||
int protected_count =
|
||||
(protected_prefixes ? protected_prefixes->size : 0) + (size_skipped ? size_skipped->size : 0);
|
||||
if (!send_int(fd, protected_count))
|
||||
return -1;
|
||||
if (protected_prefixes) {
|
||||
for (int i = 0; i < protected_prefixes->size; i++) {
|
||||
if (!send_wire_str(fd, (char*)protected_prefixes->items[i]))
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
if (size_skipped) {
|
||||
for (int i = 0; i < size_skipped->size; i++) {
|
||||
if (!send_wire_str(fd, (char*)size_skipped->items[i]))
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
int missing_count = missing_args ? missing_args->size : 0;
|
||||
if (!send_int(fd, missing_count))
|
||||
return -1;
|
||||
for (int i = 0; i < missing_count; i++) {
|
||||
if (!send_wire_str(fd, (char*)missing_args->items[i]))
|
||||
return -1;
|
||||
}
|
||||
int dirs_count = synced_dirs ? synced_dirs->size : 0;
|
||||
if (!send_int(fd, dirs_count))
|
||||
return -1;
|
||||
for (int i = 0; i < dirs_count; i++) {
|
||||
if (!send_wire_str(fd, (char*)synced_dirs->items[i]))
|
||||
return -1;
|
||||
}
|
||||
/* Protocol 2.30.0: the receiver-side per-directory filter rules discovered by
|
||||
the sender's scan, so the whole-tree commit walker can shield a
|
||||
destination-only entry that matches only a per-directory merge rule. */
|
||||
if (!delete_filter_dir_rules_send(fd, per_dir_rules))
|
||||
return -1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Transmit the keep-set manifest and wait for the receiver's verdict. Used by
|
||||
--delete-before/--delete-during, where the extras are removed on the receiver
|
||||
BEFORE the first byte of file data is sent: the receiver acknowledges with
|
||||
STATUS_OK once the bounded delete committed, or STATUS_ERROR if it could not
|
||||
(in which case the sender aborts without streaming any data). The ACK may
|
||||
take much longer than an ordinary per-message round trip because the receiver
|
||||
performs the whole bounded deletion walk (up to MAX_SERVER_DELETE_COUNT
|
||||
unlinks) before replying, so the wait uses a generous explicit deadline
|
||||
instead of the default 60 s receive window. */
|
||||
#define DELETE_ACK_TIMEOUT_SEC 3600
|
||||
/* While waiting for the (potentially slow) receiver-side deletion, send a
|
||||
* STATUS_KEEPALIVE at most this often so the connection is demonstrably alive
|
||||
* and neither side's per-message timeout trips. */
|
||||
#define DELETE_ACK_KEEPALIVE_SEC 10
|
||||
|
||||
bool send_delete_manifest_early(Client* client, ArrayList* manifest, ArrayList* protected_prefixes,
|
||||
ArrayList* size_skipped, ArrayList* missing_args,
|
||||
ArrayList* synced_dirs, const FilterRuleList* per_dir_rules) {
|
||||
if (!client || !manifest)
|
||||
return false;
|
||||
if (send_delete_manifest(client->file_descriptor, manifest, protected_prefixes, size_skipped,
|
||||
missing_args, synced_dirs, per_dir_rules) != 0)
|
||||
return false;
|
||||
Status ack;
|
||||
/* The wait is long (up to an hour) and runs inline on this thread: a helper
|
||||
* thread would race the non-thread-safe protocol send path, so keepalives are
|
||||
* emitted from this wait loop itself. A Ctrl-C/SIGTERM abort flag also ends
|
||||
* the wait; the caller then best-effort sends STATUS_ABORT. */
|
||||
if (!receive_status_keepalive(client->file_descriptor, &ack, DELETE_ACK_TIMEOUT_SEC,
|
||||
DELETE_ACK_KEEPALIVE_SEC, client_abort_pending)) {
|
||||
/* A Ctrl-C/SIGTERM abort ends the wait above; tell the receiver before the
|
||||
caller tears the connection down (best-effort). */
|
||||
if (client_abort_pending()) {
|
||||
log_info_message(LOG_INFO_MISC,
|
||||
"Abort requested while awaiting delete ack; sending STATUS_ABORT");
|
||||
send_status(client->file_descriptor, STATUS_ABORT);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
if (ack != STATUS_OK) {
|
||||
log_server_rejection("Server failed to delete files before the transfer");
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Server-contacting --dry-run. Connects to the configured remote/daemon and
|
||||
* runs the normal per-file incremental decision WITHOUT transmitting any file
|
||||
* data: the receiver (which also sees dry_run=true on the wire) answers
|
||||
* STATUS_OK for an up-to-date file and STATUS_DRY_RUN_TRANSFER for a file it
|
||||
* would otherwise write, mutating nothing on either side. The would-transfer
|
||||
* set and the same trailer as the local dry-run are printed. A
|
||||
* --compare-dest exact basis hit with no destination copy is reported as a
|
||||
* skip by the receiver.
|
||||
*
|
||||
* Only regular files take the receiver-consulted check; directory / symlink /
|
||||
* special / hard-link-sibling entries have no per-file content check, so they
|
||||
* are reported conservatively as would-transfer and their frames are never
|
||||
* sent (which is what keeps the receiver mutation-free). --delete* is
|
||||
* deliberately NOT transmitted in dry-run, so no deletion can occur; the
|
||||
* would-delete manifest report is a documented follow-up.
|
||||
*
|
||||
* Returns 0 on success, 1 on error. */
|
||||
int send_dry_run_remote(Config* config) {
|
||||
int from_skipped = 0;
|
||||
ArrayList* missing_args = NULL;
|
||||
if (!client_prepare_files_from(config, &missing_args, &from_skipped))
|
||||
return 1;
|
||||
if (missing_args)
|
||||
array_list_delete(missing_args);
|
||||
/* A live session may follow, so arm graceful abort handling. */
|
||||
client_set_abort_armed(true);
|
||||
ProtocolSession session;
|
||||
Client* client = client_connect_and_bind_session(config, &session);
|
||||
if (!client) {
|
||||
client_set_abort_armed(false);
|
||||
return 1;
|
||||
}
|
||||
|
||||
int ret = 1;
|
||||
bool partial = false;
|
||||
time_t dry_start = time(NULL);
|
||||
ReceiverStats dry_stats;
|
||||
memset(&dry_stats, 0, sizeof(dry_stats));
|
||||
PreparedScanner prepared;
|
||||
memset(&prepared, 0, sizeof(prepared));
|
||||
DirectoryScanner* scanner = NULL;
|
||||
ArrayList* dry_manifest = NULL;
|
||||
ArrayList* dry_dirs = NULL;
|
||||
ArrayList* dry_excluded = NULL;
|
||||
ArrayList* dry_size_skipped = NULL;
|
||||
FilterRuleList* dry_per_dir = NULL;
|
||||
if (!config_send(client->file_descriptor, config))
|
||||
goto dry_fail;
|
||||
receive_daemon_motd(client, config);
|
||||
if (!prepare_scanner(config, 0, &prepared))
|
||||
goto dry_fail;
|
||||
/* -n --delete: build the same keep-set manifest, protected prefixes, and
|
||||
synchronized-directory scope a real run would send, so the receiver's
|
||||
read-only extras walk enumerates exactly the deletions a real run makes. */
|
||||
if (config->use_delete) {
|
||||
dry_manifest = array_list_create(free);
|
||||
dry_dirs = array_list_create(free);
|
||||
dry_size_skipped = array_list_create(free);
|
||||
dry_per_dir = filter_rule_list_create();
|
||||
if (!dry_manifest || !dry_dirs || !dry_size_skipped || !dry_per_dir)
|
||||
goto dry_fail;
|
||||
prepared.options.per_dir_rules = dry_per_dir;
|
||||
if (!config->delete_excluded) {
|
||||
dry_excluded = array_list_create(free);
|
||||
if (!dry_excluded)
|
||||
goto dry_fail;
|
||||
prepared.options.excluded_paths = dry_excluded;
|
||||
}
|
||||
prepared.options.size_skipped_paths = dry_size_skipped;
|
||||
/* A --files-from subset confines the extras walk to the directories the
|
||||
scan synchronized; a full recursive transfer marks the root itself. */
|
||||
if (config->files_from_set == NULL) {
|
||||
char* root_marker = delete_scope_root_marker(config);
|
||||
if (!root_marker || !array_list_add(dry_dirs, root_marker)) {
|
||||
free(root_marker);
|
||||
goto dry_fail;
|
||||
}
|
||||
} else {
|
||||
prepared.options.synced_dirs = dry_dirs;
|
||||
}
|
||||
}
|
||||
scanner = directory_scanner_create_with_options(config->send_directory, &prepared.options);
|
||||
if (!scanner)
|
||||
goto dry_fail;
|
||||
|
||||
int file_count = 0;
|
||||
unsigned long long total_bytes = 0;
|
||||
char size_buffer[32];
|
||||
if (!config->quiet)
|
||||
printf("Dry run: files to be transferred\n");
|
||||
Chunk* chunk;
|
||||
while ((chunk = directory_scanner_next(scanner)) != NULL) {
|
||||
if (dry_manifest && !add_chunk_to_manifest(dry_manifest, chunk)) {
|
||||
chunk_destroy(chunk);
|
||||
goto dry_fail;
|
||||
}
|
||||
for (int i = 0; i < chunk->element_count; i++) {
|
||||
File* f = chunk->items[i];
|
||||
if (!f)
|
||||
continue;
|
||||
unsigned long long fsize = f->data ? f->data->size : 0;
|
||||
bool would;
|
||||
if (f->is_dir || f->is_symlink || f->is_special ||
|
||||
(f->link_group != 0 && !f->link_first && f->hardlink_target != NULL)) {
|
||||
/* No receiver-side content check exists for these frame types; a real
|
||||
run would (re)create them, so report would-transfer and send no
|
||||
frame (the receiver must stay mutation-free). */
|
||||
would = true;
|
||||
} else if (fsize > MAX_RECEIVE_WHOLE_FILE_SIZE && !config->use_incremental &&
|
||||
!config_has_basis(config)) {
|
||||
/* A non-incremental run streams a >whole-file-limit source without the
|
||||
STATUS_CHECK handshake, so no read-only receiver decision is possible
|
||||
(and none is needed: a real run would transfer it). */
|
||||
would = true;
|
||||
} else {
|
||||
DeltaSignature* sig = NULL;
|
||||
unsigned long long resume_offset = 0;
|
||||
int rc = incremental_check(client, f, config, &sig, &resume_offset);
|
||||
delta_signature_destroy(sig);
|
||||
if (rc < 0) {
|
||||
chunk_destroy(chunk);
|
||||
goto dry_fail;
|
||||
}
|
||||
if (rc == 1)
|
||||
continue; /* up to date; nothing to report */
|
||||
if (rc != 4) {
|
||||
log_message(LOG_LEVEL_ERROR, "Unexpected receiver reply during dry-run");
|
||||
chunk_destroy(chunk);
|
||||
goto dry_fail;
|
||||
}
|
||||
would = true;
|
||||
}
|
||||
if (would) {
|
||||
if (!config->quiet) {
|
||||
char* escaped_path = output_escape(file_wire_path(f), config->eight_bit_output);
|
||||
if (!escaped_path) {
|
||||
chunk_destroy(chunk);
|
||||
goto dry_fail;
|
||||
}
|
||||
if (config->human_readable)
|
||||
printf(" %s (%s)\n", escaped_path,
|
||||
display_bytes(fsize, true, size_buffer, sizeof(size_buffer)));
|
||||
else
|
||||
printf(" %s (%llu bytes)\n", escaped_path, fsize);
|
||||
free(escaped_path);
|
||||
}
|
||||
total_bytes += fsize;
|
||||
file_count++;
|
||||
}
|
||||
}
|
||||
chunk_destroy(chunk);
|
||||
}
|
||||
bool io_error = directory_scanner_had_io_error(scanner);
|
||||
if (directory_scanner_failed(scanner))
|
||||
goto dry_fail;
|
||||
if (io_error)
|
||||
log_message(LOG_LEVEL_WARNING, "source scan hit an unreadable directory");
|
||||
/* Send the keep-set manifest (no data frames) so the receiver can enumerate
|
||||
the destination extras; an early-timing delete ACKs before it will accept
|
||||
the terminal FINISHED. */
|
||||
bool early_delete = config->use_delete && config_delete_timing_early(config);
|
||||
if (dry_manifest) {
|
||||
if (send_delete_manifest(client->file_descriptor, dry_manifest, dry_excluded, dry_size_skipped,
|
||||
NULL, dry_dirs, dry_per_dir) != 0)
|
||||
goto dry_fail;
|
||||
if (early_delete) {
|
||||
Status ack;
|
||||
if (!receive_status_keepalive(client->file_descriptor, &ack, DELETE_ACK_TIMEOUT_SEC,
|
||||
DELETE_ACK_KEEPALIVE_SEC, client_abort_pending) ||
|
||||
ack != STATUS_OK)
|
||||
goto dry_fail;
|
||||
}
|
||||
}
|
||||
/* Terminate the stream so the receiver emits its success frame; no data frame
|
||||
is ever sent in dry-run. */
|
||||
if (!send_status(client->file_descriptor, STATUS_FINISHED))
|
||||
goto dry_fail;
|
||||
Status status;
|
||||
if (!receive_status(client->file_descriptor, &status))
|
||||
goto dry_fail;
|
||||
if (status == STATUS_STATS) {
|
||||
ArrayList* would_delete = array_list_create(free);
|
||||
if (!would_delete)
|
||||
goto dry_fail;
|
||||
if (!receive_stats_record(client->file_descriptor, &dry_stats, would_delete)) {
|
||||
array_list_delete(would_delete);
|
||||
goto dry_fail;
|
||||
}
|
||||
print_delete_reports(config, would_delete);
|
||||
array_list_delete(would_delete);
|
||||
if (!receive_status(client->file_descriptor, &status))
|
||||
goto dry_fail;
|
||||
}
|
||||
/* A per-entry receiver failure is rsync's PARTIAL transfer (exit 23), not a
|
||||
hard failure: a dry run transfers nothing, but keep the verdict consistent
|
||||
with the normal path instead of treating it as a protocol error. */
|
||||
if (status == STATUS_PARTIAL) {
|
||||
partial = true;
|
||||
} else if (status != STATUS_OK) {
|
||||
goto dry_fail;
|
||||
}
|
||||
if (!config->quiet) {
|
||||
if (config->human_readable)
|
||||
printf("Total: %d files, %s\n", file_count,
|
||||
display_bytes(total_bytes, true, size_buffer, sizeof(size_buffer)));
|
||||
else
|
||||
printf("Total: %d files, %.1f MB\n", file_count, (double)total_bytes / (double)BYTES_PER_MIB);
|
||||
}
|
||||
{
|
||||
TransferStats dry_transfer;
|
||||
memset(&dry_transfer, 0, sizeof(dry_transfer));
|
||||
dry_transfer.flist_reg = (unsigned long long)file_count;
|
||||
dry_transfer.total_file_size = total_bytes;
|
||||
dry_transfer.transferred_regular = (unsigned long long)file_count;
|
||||
dry_transfer.transferred_file_size = total_bytes;
|
||||
dry_transfer.literal_data = total_bytes;
|
||||
report_transfer_stats(config, &dry_transfer, dry_start, &dry_stats);
|
||||
}
|
||||
ret = io_error ? 1 : (partial ? 23 : 0);
|
||||
|
||||
dry_fail:
|
||||
if (dry_manifest)
|
||||
array_list_delete(dry_manifest);
|
||||
if (dry_dirs)
|
||||
array_list_delete(dry_dirs);
|
||||
if (dry_excluded)
|
||||
array_list_delete(dry_excluded);
|
||||
if (dry_size_skipped)
|
||||
array_list_delete(dry_size_skipped);
|
||||
if (dry_per_dir)
|
||||
filter_rule_list_free(dry_per_dir);
|
||||
if (scanner)
|
||||
directory_scanner_destroy(scanner);
|
||||
prepared_scanner_destroy(&prepared);
|
||||
disconnect_transfer_client(client);
|
||||
protocol_session_unbind();
|
||||
client_set_abort_armed(false);
|
||||
return ret;
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,470 +0,0 @@
|
||||
#include "client_send_internal.h"
|
||||
#include "array_list.h"
|
||||
#include "charset.h"
|
||||
#include "config.h"
|
||||
#include "delete_plan.h"
|
||||
#include "file.h"
|
||||
#include "file_list.h"
|
||||
#include "filter.h"
|
||||
#include "hardlink.h"
|
||||
#include "log.h"
|
||||
#include "scanner.h"
|
||||
#include "utils.h"
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
/* Build the scanner options for one scan. Returns false and logs on failure. */
|
||||
bool prepare_scanner(const Config* config, int num_threads, PreparedScanner* out) {
|
||||
if (!out)
|
||||
return false;
|
||||
out->base_filters = NULL;
|
||||
out->hardlinks = NULL;
|
||||
out->relative_prefix = NULL;
|
||||
memset(&out->options, 0, sizeof(out->options));
|
||||
|
||||
int rule_count = config->filters ? config->filters->size : 0;
|
||||
const char** texts = NULL;
|
||||
if (rule_count > 0) {
|
||||
texts = malloc((size_t)rule_count * sizeof(char*));
|
||||
if (!texts) {
|
||||
log_message(LOG_LEVEL_ERROR, "memory allocation failed for filter rules");
|
||||
return false;
|
||||
}
|
||||
for (int i = 0; i < rule_count; i++)
|
||||
texts[i] = (const char*)config->filters->items[i];
|
||||
}
|
||||
if (rule_count > 0 || config->cvs_exclude) {
|
||||
char err[160];
|
||||
out->base_filters = filter_base_build(texts, rule_count, config->cvs_exclude,
|
||||
config->delete_excluded, err, sizeof(err));
|
||||
free(texts);
|
||||
if (!out->base_filters) {
|
||||
log_message(LOG_LEVEL_ERROR, "invalid filter rule: %s", err);
|
||||
return false;
|
||||
}
|
||||
} else {
|
||||
free(texts);
|
||||
}
|
||||
|
||||
ScannerOptions* options = &out->options;
|
||||
options->use_metadata = config->use_metadata;
|
||||
options->preserve_atimes = config->preserve_atimes;
|
||||
options->preserve_crtimes = config->preserve_crtimes;
|
||||
options->preserve_xattrs = config->preserve_xattrs;
|
||||
options->preserve_acls = config->preserve_acls;
|
||||
options->chunk_size = config->chunk_size;
|
||||
/* --exclude/--include are compiled, in command-line order, into the SAME
|
||||
* ordered filter rule list as --filter/-f (see config_add_selection_rule), so
|
||||
* the legacy per-kind arrays are deliberately NOT passed to the scanner:
|
||||
* doing so would re-apply them with the old "excludes first, then includes as
|
||||
* a mandatory whitelist" precedence and defeat rsync's first-match-wins
|
||||
* ordering. The arrays remain populated purely for the Config API surface. */
|
||||
options->exclude_patterns = NULL;
|
||||
options->exclude_count = 0;
|
||||
options->include_patterns = NULL;
|
||||
options->include_count = 0;
|
||||
options->max_size = config->max_size;
|
||||
options->min_size = config->min_size;
|
||||
options->max_depth = config->max_depth;
|
||||
options->num_threads = num_threads;
|
||||
options->follow_symlinks = config->follow_symlinks;
|
||||
options->copy_links = config->copy_links;
|
||||
options->safe_links = config->safe_links;
|
||||
options->copy_unsafe_links = config->copy_unsafe_links;
|
||||
options->copy_dirlinks = config->copy_dirlinks;
|
||||
options->munge_links = config->munge_links;
|
||||
options->checksum = config->checksum;
|
||||
options->one_file_system = config->one_file_system;
|
||||
options->preserve_devices = config->preserve_devices;
|
||||
options->preserve_specials = config->preserve_specials;
|
||||
options->copy_devices = config->copy_devices;
|
||||
options->file_list = (const FileListSet*)config->files_from_set;
|
||||
options->base_filters = out->base_filters;
|
||||
options->per_dir_filters = config->per_dir_filter;
|
||||
options->delete_excluded = config->delete_excluded;
|
||||
options->exclude_per_dir_filter_files = config->per_dir_filter_count >= 2;
|
||||
options->dirs = config->dirs;
|
||||
options->relative = config->relative;
|
||||
/* A real recursive transfer recreates empty source directories (rsync
|
||||
parity); low-level scanner users leave this off. */
|
||||
options->emit_empty_dirs = true;
|
||||
/* --no-implied-dirs only has meaning with -R (rsync): without it the option
|
||||
is a documented no-op, so the scanner must not suppress directory
|
||||
metadata. */
|
||||
options->no_implied_dirs = config->no_implied_dirs && config->relative;
|
||||
/* -R/--relative outside --files-from reconstructs every destination path from
|
||||
* the source spec (rsync's '/./' cut point). With --files-from the listed
|
||||
* entry already supplies the bare relative path, so no prefix is built. */
|
||||
if (config->relative && config->files_from_set == NULL && config->send_directory) {
|
||||
out->relative_prefix = scanner_relative_prefix(config->send_directory);
|
||||
if (!out->relative_prefix) {
|
||||
log_message(LOG_LEVEL_ERROR, "memory allocation failed building --relative path prefix");
|
||||
filter_rule_list_free(out->base_filters);
|
||||
out->base_filters = NULL;
|
||||
return false;
|
||||
}
|
||||
options->relative_prefix = out->relative_prefix;
|
||||
}
|
||||
options->prune_empty_dirs = config->prune_empty_dirs;
|
||||
options->ignore_io_errors = config->ignore_errors;
|
||||
options->ignore_missing_args = config->ignore_missing_args || config->delete_missing_args;
|
||||
options->note_nonreg = (config->info_level & LOG_INFO_NONREG) != 0 && !config->quiet;
|
||||
options->note_mount = (config->info_level & LOG_INFO_MOUNT) != 0 && !config->quiet;
|
||||
options->send_directory = config->send_directory;
|
||||
options->eight_bit_output = config->eight_bit_output;
|
||||
options->excluded_paths = NULL;
|
||||
options->excluded_mutex = NULL;
|
||||
options->size_skipped_paths = NULL;
|
||||
options->synced_dirs = NULL;
|
||||
options->hardlinks = NULL;
|
||||
/* Set by the real send paths; NULL for the metadata-only scans (progress
|
||||
pre-count, batch) that must not perturb the sender's --stats counter. */
|
||||
options->dir_count = NULL;
|
||||
/* P7 Wave D: capture source directory metadata when a directory attribute is
|
||||
requested (-p for modes, -t for times unless -O omits them). Whether they
|
||||
are APPLIED is decided receiver-side. */
|
||||
options->capture_dir_times = dir_metadata_should_capture(config);
|
||||
options->dir_entries = NULL;
|
||||
options->dir_entries_mutex = NULL;
|
||||
if (config->preserve_hard_links) {
|
||||
out->hardlinks = hardlink_table_create();
|
||||
if (!out->hardlinks) {
|
||||
filter_rule_list_free(out->base_filters);
|
||||
out->base_filters = NULL;
|
||||
return false;
|
||||
}
|
||||
options->hardlinks = out->hardlinks;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void prepared_scanner_destroy(PreparedScanner* prepared) {
|
||||
if (!prepared)
|
||||
return;
|
||||
filter_rule_list_free(prepared->base_filters);
|
||||
prepared->base_filters = NULL;
|
||||
hardlink_table_destroy(prepared->hardlinks);
|
||||
prepared->hardlinks = NULL;
|
||||
free(prepared->relative_prefix);
|
||||
prepared->relative_prefix = NULL;
|
||||
}
|
||||
|
||||
/* -R/--relative implied directories: rsync transmits the metadata of the
|
||||
* parent directories implied by the source path (every prefix component above
|
||||
* the source root) so the receiver applies their attributes to the created
|
||||
* parents. FastSync's scan only covers the source root and below, so append
|
||||
* one metadata-only directory entry per implied ancestor. --no-implied-dirs
|
||||
* suppresses this exactly like rsync. A missing ancestor is never fatal. */
|
||||
bool append_implied_dir_times(const Config* config, ArrayList* dir_entries) {
|
||||
if (!dir_entries || !config->relative || config->files_from_set != NULL ||
|
||||
config->no_implied_dirs || !config->send_directory)
|
||||
return true;
|
||||
char* prefix = scanner_relative_prefix(config->send_directory);
|
||||
if (!prefix)
|
||||
return true;
|
||||
int ncomp = 0;
|
||||
for (const char* s = prefix; *s;) {
|
||||
while (*s == '/')
|
||||
s++;
|
||||
if (!*s)
|
||||
break;
|
||||
while (*s && *s != '/')
|
||||
s++;
|
||||
ncomp++;
|
||||
}
|
||||
if (ncomp <= 1) {
|
||||
free(prefix);
|
||||
return true;
|
||||
}
|
||||
char* fs = str_dup(config->send_directory);
|
||||
if (!fs) {
|
||||
free(prefix);
|
||||
return true;
|
||||
}
|
||||
size_t flen = strlen(fs);
|
||||
while (flen > 1 && fs[flen - 1] == '/')
|
||||
fs[--flen] = '\0';
|
||||
bool ok = true;
|
||||
/* Walk the source path upwards one component at a time (fs is truncated in
|
||||
place, so each step targets the next implied ancestor). */
|
||||
for (int depth = ncomp - 2; depth >= 0 && ok; depth--) {
|
||||
char* slash = strrchr(fs, '/');
|
||||
if (!slash || slash == fs)
|
||||
break;
|
||||
*slash = '\0';
|
||||
char* p = prefix;
|
||||
int c = 0;
|
||||
while (c <= depth) {
|
||||
while (*p == '/')
|
||||
p++;
|
||||
while (*p && *p != '/')
|
||||
p++;
|
||||
c++;
|
||||
}
|
||||
char saved = *p;
|
||||
*p = '\0';
|
||||
struct stat st;
|
||||
if (stat(fs, &st) == 0 && S_ISDIR(st.st_mode)) {
|
||||
File* file = file_create(fs);
|
||||
if (!file) {
|
||||
ok = false;
|
||||
} else {
|
||||
file->is_dir = true;
|
||||
file->metadata =
|
||||
file_metadata_create(fs, &st, config->preserve_atimes, config->preserve_crtimes);
|
||||
file->send_path = str_dup(prefix);
|
||||
if (!file->metadata || !file->send_path || !array_list_add(dir_entries, file)) {
|
||||
file_destroy(file);
|
||||
ok = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
*p = saved;
|
||||
}
|
||||
free(fs);
|
||||
free(prefix);
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* The delete-walk root scope for a full (non---files-from) transfer: rsync
|
||||
* confines --delete to the directories it actually transferred. A plain
|
||||
* recursive run mirrors the source under the receive root, so "." (the whole
|
||||
* tree) is correct; an -R run transfers only the reconstructed prefix subtree,
|
||||
* so the walk is scoped to that prefix instead. Returns a malloc'd wire path
|
||||
* (or "."), or NULL on allocation failure. */
|
||||
char* delete_scope_root_marker(const Config* config) {
|
||||
if (config->relative && config->files_from_set == NULL && config->send_directory) {
|
||||
char* prefix = scanner_relative_prefix(config->send_directory);
|
||||
if (!prefix)
|
||||
return NULL;
|
||||
if (prefix[0] != '\0')
|
||||
return prefix;
|
||||
free(prefix);
|
||||
}
|
||||
return str_dup(".");
|
||||
}
|
||||
|
||||
/* The -R destination prefix that confines a per-directory delete walk, or NULL
|
||||
* when the whole receive root is in scope. The marker was installed into
|
||||
* `synced_dirs` by delete_scope_root_marker(); for a plain recursive transfer
|
||||
* it is "." (whole root) and for --files-from the list is not a single prefix. */
|
||||
const char* delete_plan_walk_root(const Config* config, const ArrayList* synced_dirs) {
|
||||
if (!config || config->files_from_set != NULL || !config->relative || !config->send_directory)
|
||||
return NULL;
|
||||
if (!synced_dirs || synced_dirs->size != 1)
|
||||
return NULL;
|
||||
const char* marker = (const char*)synced_dirs->items[0];
|
||||
if (marker[0] == '\0' || strcmp(marker, ".") == 0)
|
||||
return NULL;
|
||||
return marker;
|
||||
}
|
||||
|
||||
/* The destination-relative mirror path for a missing --files-from entry: where
|
||||
a PRESENT entry with the same name would have been written. With -R that is
|
||||
the entry's bare relative path (the bare wire path the receiver uses);
|
||||
otherwise it is the full source mirror below the destination root
|
||||
(`send_directory` joined to the entry, leading '/' stripped), exactly the
|
||||
path the manifest records for a present sibling. Returns an owned string, or
|
||||
NULL on allocation failure. */
|
||||
static char* files_from_missing_dest_path(const Config* config, const char* entry) {
|
||||
if (config->relative)
|
||||
return str_dup(entry);
|
||||
char* joined = path_cat(config->send_directory, entry);
|
||||
if (!joined)
|
||||
return NULL;
|
||||
const char* rel = *joined == '/' ? joined + 1 : joined;
|
||||
char* dup = str_dup(rel);
|
||||
free(joined);
|
||||
return dup;
|
||||
}
|
||||
|
||||
/* --files-from semantics: every listed entry must resolve under the source
|
||||
* root, otherwise rsync reports a hard error instead of silently transferring
|
||||
* nothing. An entry of "." (the whole tree) and listed-but-empty directories
|
||||
* are valid. An empty list is valid too: rsync transfers nothing and exits 0.
|
||||
* With --ignore-missing-args
|
||||
* (implied by --delete-missing-args) a listed-but-missing entry is instead
|
||||
* skipped: nothing is transferred for it, it never enters the keep-set and the
|
||||
* run succeeds for the rest (an all-missing non-empty list succeeds
|
||||
* transferring nothing, matching rsync). With --delete-missing-args
|
||||
* `missing_dest` (when non-NULL) collects the entry's destination-relative
|
||||
* mirror for the receiver's exact-deletion request. Runs before any
|
||||
* transfer so the failure/skip is surfaced uniformly in the single-threaded,
|
||||
* -m, dry-run and --list-only paths. */
|
||||
bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* skipped_out) {
|
||||
*skipped_out = 0;
|
||||
const FileListSet* set = (const FileListSet*)config->files_from_set;
|
||||
if (!set)
|
||||
return true;
|
||||
if (!config->send_directory) {
|
||||
log_message(LOG_LEVEL_ERROR, "--files-from requires a source directory");
|
||||
return false;
|
||||
}
|
||||
if (set->count == 0) {
|
||||
/* rsync treats an empty --files-from list as "nothing to transfer" and
|
||||
exits 0 (the source directory is still a valid source arg), so this is
|
||||
not an error. Nothing passes the (empty) allow-set, so no file is sent
|
||||
and no keep-set entry is produced. */
|
||||
return true;
|
||||
}
|
||||
bool ignore = config->ignore_missing_args || config->delete_missing_args;
|
||||
for (int i = 0; i < set->count; i++) {
|
||||
const char* entry = set->entries[i];
|
||||
if (entry[0] == '\0')
|
||||
continue; /* "." == list the whole tree */
|
||||
char* full = path_cat(config->send_directory, entry);
|
||||
if (!full) {
|
||||
log_message(LOG_LEVEL_ERROR, "memory allocation failed while validating --files-from");
|
||||
return false;
|
||||
}
|
||||
struct stat st;
|
||||
if (lstat(full, &st) != 0) {
|
||||
free(full);
|
||||
if (ignore) {
|
||||
(*skipped_out)++;
|
||||
char* escaped_entry = output_escape(entry, log_get_8_bit_output());
|
||||
log_info_message(LOG_INFO_MISC, "skipping missing --files-from entry '%s'",
|
||||
escaped_entry ? escaped_entry : "<allocation failed>");
|
||||
free(escaped_entry);
|
||||
if (config->delete_missing_args && missing_dest) {
|
||||
char* mirror = files_from_missing_dest_path(config, entry);
|
||||
if (!mirror || !array_list_add(missing_dest, mirror)) {
|
||||
free(mirror);
|
||||
log_message(LOG_LEVEL_ERROR, "memory allocation failed while validating --files-from");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
char* escaped_entry = output_escape(entry, log_get_8_bit_output());
|
||||
char* escaped_src = output_escape(config->send_directory, log_get_8_bit_output());
|
||||
log_message(LOG_LEVEL_ERROR, "--files-from entry '%s' not found in source '%s'",
|
||||
escaped_entry ? escaped_entry : "<allocation failed>",
|
||||
escaped_src ? escaped_src : "<allocation failed>");
|
||||
free(escaped_entry);
|
||||
free(escaped_src);
|
||||
return false;
|
||||
}
|
||||
free(full);
|
||||
}
|
||||
if (*skipped_out > 0) {
|
||||
if (config->delete_missing_args) {
|
||||
/* --list-only never deletes and a --dry-run only shows intent, so the
|
||||
summary must not claim a real deletion happened in those modes. */
|
||||
if (config->list_only)
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"--delete-missing-args: %d missing --files-from entr%s skipped (--list-only "
|
||||
"never deletes)",
|
||||
*skipped_out, *skipped_out == 1 ? "y" : "ies");
|
||||
else if (config->dry_run)
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"--delete-missing-args: %d missing --files-from entr%s would be deleted from "
|
||||
"the destination (dry run)",
|
||||
*skipped_out, *skipped_out == 1 ? "y" : "ies");
|
||||
else
|
||||
log_message(
|
||||
LOG_LEVEL_WARNING,
|
||||
"--delete-missing-args: %d missing --files-from entr%s will be deleted from the "
|
||||
"destination",
|
||||
*skipped_out, *skipped_out == 1 ? "y" : "ies");
|
||||
} else if (config->ignore_missing_args)
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"--ignore-missing-args: ignored %d missing --files-from entr%s", *skipped_out,
|
||||
*skipped_out == 1 ? "y" : "ies");
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Walk the whole source tree once collecting only destination-relative wire
|
||||
paths, loading and sending nothing. --delete-before/--delete-during need the
|
||||
complete keep-set manifest before the first data byte, so it is built by a
|
||||
dedicated pre-scan pass and transmitted early; the data pass then re-scans
|
||||
with a fresh scanner. --delete-before additionally replays this very scan as
|
||||
its data pass (rsync's single file list), so `chunks_out` (optional) retains
|
||||
the scanned Chunk objects for the caller to send instead of destroying them;
|
||||
the caller owns the list and must give it a chunk_destroy destructor. A
|
||||
source I/O error is fatal unless the options carry --ignore-errors, in which
|
||||
case the scan continues past the unreadable directory and *io_error_out
|
||||
reports it (the caller still performs the deletion but reports the run as
|
||||
errored). */
|
||||
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
|
||||
DeletePlanSender* plans, bool* io_error_out,
|
||||
unsigned long long* non_dir_count_out, ArrayList* chunks_out,
|
||||
bool emit_nonreg) {
|
||||
if (io_error_out)
|
||||
*io_error_out = false;
|
||||
if (non_dir_count_out)
|
||||
*non_dir_count_out = 0;
|
||||
ScannerOptions local = *options;
|
||||
/* The pre-scan is normally a paths-only pass with no client output: it must
|
||||
not emit --info=nonreg lines because the data pass re-scans and emits them
|
||||
once. When the caller replays this scan as the data pass (--delete-before)
|
||||
there is no later scan, so it opts in and the lines are emitted here. */
|
||||
local.note_nonreg = emit_nonreg && options->note_nonreg;
|
||||
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
|
||||
if (!scanner)
|
||||
return false;
|
||||
bool ok = true;
|
||||
Chunk* chunk;
|
||||
while ((chunk = directory_scanner_next(scanner)) != NULL) {
|
||||
if (non_dir_count_out) {
|
||||
for (int i = 0; i < chunk->element_count; i++) {
|
||||
const File* f = chunk->items[i];
|
||||
if (f && !f->is_dir)
|
||||
(*non_dir_count_out)++;
|
||||
}
|
||||
}
|
||||
if (manifest && !add_chunk_to_manifest(manifest, chunk)) {
|
||||
ok = false;
|
||||
chunk_destroy(chunk);
|
||||
break;
|
||||
}
|
||||
if (plans) {
|
||||
for (int i = 0; i < chunk->element_count; i++) {
|
||||
File* f = chunk->items[i];
|
||||
if (!f)
|
||||
continue;
|
||||
const char* path = file_wire_path(f);
|
||||
if (!delete_plan_sender_add(plans, path, f->is_dir)) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!ok) {
|
||||
chunk_destroy(chunk);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (chunks_out) {
|
||||
/* Retain the chunk for the caller's data pass; ownership moves with it. */
|
||||
if (!array_list_add(chunks_out, chunk)) {
|
||||
ok = false;
|
||||
chunk_destroy(chunk);
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
chunk_destroy(chunk);
|
||||
}
|
||||
}
|
||||
if (ok) {
|
||||
/* Keep every traversed source directory, including empty ones, so a plan
|
||||
no longer removes the destination directory itself. Their own plans are
|
||||
emitted after the data stream (no file frame triggers them). */
|
||||
if (plans && options->plan_dirs) {
|
||||
for (int i = 0; i < options->plan_dirs->size; i++) {
|
||||
if (!delete_plan_sender_add(plans, (const char*)options->plan_dirs->items[i], true)) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (ok && directory_scanner_failed(scanner))
|
||||
ok = false;
|
||||
if (io_error_out)
|
||||
*io_error_out = directory_scanner_had_io_error(scanner);
|
||||
directory_scanner_destroy(scanner);
|
||||
return ok;
|
||||
}
|
||||
+1802
-755
File diff suppressed because it is too large
Load Diff
@@ -22,13 +22,7 @@ void client_set_abort_armed(bool armed);
|
||||
* never free it, and the caller retains ownership (freeing it with
|
||||
* config_delete() once the call returns). */
|
||||
int send_files(Config* config);
|
||||
int send_files_multithreaded(Config* config);
|
||||
/* rsync's --ignore-errors deletion gate: with no I/O error during the scan the
|
||||
* deletion phase always proceeds; with one it is suppressed unless
|
||||
* `--ignore-errors` was given. Exposed so the decision can be unit-tested
|
||||
* without a privileged (mode-000) source directory. See client_send.c. */
|
||||
bool ignore_errors_allows_delete(const Config* config, bool had_io_error);
|
||||
|
||||
int send_files_multithreaded(Config** config);
|
||||
/* Phase 6 residual-batch (client-only). See client_send.c. */
|
||||
int write_batch_from_source(const Config* config, const char* batch_path);
|
||||
int apply_batch_to_dest(const Config* config, const char* batch_path, const char* dest_root);
|
||||
|
||||
@@ -1,126 +0,0 @@
|
||||
#ifndef CLIENT_SEND_INTERNAL_H
|
||||
#define CLIENT_SEND_INTERNAL_H
|
||||
|
||||
/* Declarations shared between the client_send.c transfer orchestration and the
|
||||
* reporting (client_report.c), scanner-preparation (client_scan.c) and
|
||||
* manifest/list/dry-run (client_manifest.c) translation units that were split
|
||||
* out of it. Nothing here is part of the public client_send.h facade. */
|
||||
|
||||
#include "array_list.h"
|
||||
#include "client_send.h"
|
||||
#include "config.h"
|
||||
#include "delete_plan.h"
|
||||
#include "delta.h"
|
||||
#include "format.h"
|
||||
#include "log.h"
|
||||
#include "protocol.h"
|
||||
#include "scanner.h"
|
||||
#include <stdatomic.h>
|
||||
#include <stdbool.h>
|
||||
#include <stddef.h>
|
||||
#include <time.h>
|
||||
|
||||
/* One mebibyte in bytes; the unit used by the --stats/--progress lines.
|
||||
Always cast to double when dividing so the output stays fractional. */
|
||||
#define BYTES_PER_MIB (1024ULL * 1024ULL)
|
||||
|
||||
/* Compiled scanner inputs that are shared read-only across scanner instances
|
||||
* and, in -m mode, across worker threads. `base_filters` owns the compiled
|
||||
* command-line + -C rules; the FileListSet allow-set lives in the Config.
|
||||
* `hardlinks` owns the --hard-links/-H link-group detection table (NULL when
|
||||
* off) and is shared (mutex-guarded) across every scanner/worker of one scan. */
|
||||
typedef struct {
|
||||
ScannerOptions options;
|
||||
FilterRuleList* base_filters; /* owned; may be NULL */
|
||||
HardLinkTable* hardlinks; /* owned; may be NULL */
|
||||
char* relative_prefix; /* owned -R prefix; may be NULL */
|
||||
} PreparedScanner;
|
||||
|
||||
/* client_scan.c */
|
||||
bool prepare_scanner(const Config* config, int num_threads, PreparedScanner* out);
|
||||
void prepared_scanner_destroy(PreparedScanner* prepared);
|
||||
bool append_implied_dir_times(const Config* config, ArrayList* dir_entries);
|
||||
char* delete_scope_root_marker(const Config* config);
|
||||
const char* delete_plan_walk_root(const Config* config, const ArrayList* synced_dirs);
|
||||
bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* skipped_out);
|
||||
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
|
||||
DeletePlanSender* plans, bool* io_error_out,
|
||||
unsigned long long* non_dir_count_out, ArrayList* chunks_out,
|
||||
bool emit_nonreg);
|
||||
|
||||
/* client_report.c */
|
||||
void log_server_rejection(const char* context);
|
||||
const char* display_bytes(unsigned long long bytes, bool human_readable, char* buffer,
|
||||
size_t buffer_size);
|
||||
unsigned long long dir_count_for_stats(const Config* config, const ArrayList* dir_entries,
|
||||
atomic_ullong* counter);
|
||||
void report_transfer_stats(const Config* config, const TransferStats* stats, time_t start,
|
||||
const ReceiverStats* recv);
|
||||
void transfer_stats_note_entry(TransferStats* stats, const File* file);
|
||||
void transfer_stats_note_transferred(TransferStats* stats, const File* file);
|
||||
bool info_flag_enabled(const Config* config, LogInfoFlag flag);
|
||||
void print_delete_reports(const Config* config, const ArrayList* paths);
|
||||
const char* delete_display_path(const Config* config, const char* path);
|
||||
bool progress_requested(const Config* config);
|
||||
void client_progress_cleanup(void);
|
||||
void client_progress_begin(const Config* config);
|
||||
void client_progress_file(const Config* config, const File* file);
|
||||
void client_progress_name(const Config* config, const File* file);
|
||||
/* Emit a transferred entry's ancestor directories (as -i/--out-format change
|
||||
* lines or --progress name lines) before the entry's own line. */
|
||||
void client_change_emit_ancestors(const Config* config, const File* file);
|
||||
/* Output parity (protocol 2.30.0): probe each not-yet-known ancestor directory's
|
||||
* pre-transfer destination state before the entry that first triggers it is
|
||||
* sent. Returns false on a protocol/transport error. */
|
||||
bool client_change_probe_ancestors(const Config* config, const File* file, int fd);
|
||||
/* Mark a transferred directory entry as already reported, and flush the
|
||||
* itemize lines for changed directories that had no transferred child. */
|
||||
void client_change_mark_dir(const Config* config, const File* file);
|
||||
void client_change_emit_pending_dirs(const Config* config, int fd);
|
||||
void client_progress_uptodate(const Config* config, const File* file);
|
||||
void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
|
||||
unsigned long long plan_non_dir_count);
|
||||
bool receive_stats_record(int fd, ReceiverStats* stats, ArrayList* would_delete);
|
||||
/* --stderr=client diagnostic channel (client_report.c): install the queueing
|
||||
* log sink for a transfer, mark the session live, flush queued diagnostics over
|
||||
* the wire at a frame boundary, and tear the sink down. */
|
||||
void client_messages_install(void);
|
||||
void client_messages_activate(bool active);
|
||||
void client_flush_client_messages(int fd);
|
||||
void client_messages_end(void);
|
||||
|
||||
/* client_send.c */
|
||||
void receive_daemon_motd(Client* client, const Config* config);
|
||||
Client* connect_transfer_client(const Config* config);
|
||||
/* Connect the configured transport and install `session` on it: init with the
|
||||
* socket fd pair, apply the I/O timeout and (when negotiated) the TLS object,
|
||||
* then bind the session to this thread. Returns the connected client, or NULL
|
||||
* after logging the connect failure. The caller owns the client and must keep
|
||||
* `session` alive until it calls protocol_session_unbind(). */
|
||||
Client* client_connect_and_bind_session(const Config* config, ProtocolSession* session);
|
||||
/* Shared --files-from/--delete-missing-args preamble for the send entry points:
|
||||
* when --delete-missing-args is set, allocate the list that
|
||||
* files_from_list_check fills with the destination mirrors of missing entries;
|
||||
* then validate the --files-from list. On success returns true and stores the
|
||||
* (possibly NULL) owned list in *missing_args_out plus the skipped count; on
|
||||
* failure returns false after freeing the list. */
|
||||
bool client_prepare_files_from(const Config* config, ArrayList** missing_args_out,
|
||||
int* skipped_out);
|
||||
void disconnect_transfer_client(Client* client);
|
||||
int incremental_check(Client* client, File* file, const Config* config, DeltaSignature** out_sig,
|
||||
unsigned long long* resume_offset);
|
||||
|
||||
/* client_manifest.c */
|
||||
bool dry_run_targets_server(const Config* config);
|
||||
bool add_chunk_to_manifest(ArrayList* manifest, const Chunk* chunk);
|
||||
int send_dry_run_manifest(const Config* config);
|
||||
int send_list_only(const Config* config);
|
||||
int send_dry_run_remote(Config* config);
|
||||
int send_delete_manifest(int fd, ArrayList* manifest, ArrayList* protected_prefixes,
|
||||
ArrayList* size_skipped, ArrayList* missing_args, ArrayList* synced_dirs,
|
||||
const FilterRuleList* per_dir_rules);
|
||||
bool send_delete_manifest_early(Client* client, ArrayList* manifest, ArrayList* protected_prefixes,
|
||||
ArrayList* size_skipped, ArrayList* missing_args,
|
||||
ArrayList* synced_dirs, const FilterRuleList* per_dir_rules);
|
||||
|
||||
#endif
|
||||
@@ -53,7 +53,7 @@ bool validate_config(const Config* config) {
|
||||
return false;
|
||||
}
|
||||
if (config->compression_threads > 0 && !config->use_compression) {
|
||||
log_message(LOG_LEVEL_ERROR, "--compress-threads requires compression (-z/--compress)");
|
||||
log_message(LOG_LEVEL_ERROR, "--compress-threads requires compression (-c or -z)");
|
||||
return false;
|
||||
}
|
||||
if (config->transport == TRANSPORT_SSH && config->use_sendfile) {
|
||||
@@ -75,13 +75,6 @@ bool validate_config(const Config* config) {
|
||||
log_message(LOG_LEVEL_ERROR, "-4/--ipv4 and -6/--ipv6 are mutually exclusive");
|
||||
return false;
|
||||
}
|
||||
/* rsync 3.4.1 rejects --inplace together with --partial-dir (exit 1): the
|
||||
inplace write path bypasses partial staging, so a partial-dir name would be
|
||||
silently ignored. Match rsync's message and refuse before any I/O. */
|
||||
if (config->inplace && config->partial_dir) {
|
||||
log_message(LOG_LEVEL_ERROR, "--inplace cannot be used with --partial-dir");
|
||||
return false;
|
||||
}
|
||||
if (config->log_file_format && !config->log_file) {
|
||||
log_message(LOG_LEVEL_ERROR, "--log-file-format requires --log-file");
|
||||
return false;
|
||||
@@ -109,16 +102,6 @@ bool validate_config(const Config* config) {
|
||||
log_message(LOG_LEVEL_ERROR, "%s", invariants_error);
|
||||
return false;
|
||||
}
|
||||
/* The receiver rejects a protect-rule block with more than MAX_FILTER_RULES
|
||||
entries as an opaque protocol error; reject an over-limit --filter set here,
|
||||
before any network I/O, with an actionable message. send_protect_entries()
|
||||
re-checks the final built count because cvs-exclude / merge rules can
|
||||
expand it beyond config->filters->size. */
|
||||
if (config->filters && config->filters->size > MAX_FILTER_RULES) {
|
||||
log_message(LOG_LEVEL_ERROR, "too many filter rules: %d (maximum %d)", config->filters->size,
|
||||
MAX_FILTER_RULES);
|
||||
return false;
|
||||
}
|
||||
/* --protocol: FastSync has exactly one wire format, so the forced version
|
||||
must equal the current PROTOCOL_VERSION exactly. Rejected here, before any
|
||||
network I/O, rather than letting the server hit its own mismatch check. */
|
||||
|
||||
+1546
-473
File diff suppressed because it is too large
Load Diff
+5
-70
@@ -10,7 +10,6 @@
|
||||
#include "stop_condition.h"
|
||||
#include <dirent.h>
|
||||
#include <stdbool.h>
|
||||
#include <stddef.h>
|
||||
#include <stdatomic.h>
|
||||
#include <sys/types.h>
|
||||
#include <threads.h>
|
||||
@@ -19,11 +18,6 @@
|
||||
* keeps one transfer from spawning an unbounded pool on a very large machine. */
|
||||
#define MAX_SCANNER_THREADS 256
|
||||
|
||||
/* Depth of the scanner's work queues: the sequential scanner's pending-directory
|
||||
* stack and the parallel scanner's result queue. Bounds memory for a very wide
|
||||
* or very deep tree while leaving ample headroom for normal scans. */
|
||||
#define SCANNER_RESULT_QUEUE_CAP 100
|
||||
|
||||
typedef struct {
|
||||
bool use_metadata;
|
||||
/* Phase 4 metadata capture: -U/--atimes and -N/--crtimes tell the scanner to
|
||||
@@ -56,7 +50,7 @@ typedef struct {
|
||||
bool copy_dirlinks;
|
||||
bool munge_links;
|
||||
bool checksum;
|
||||
int one_file_system;
|
||||
bool one_file_system;
|
||||
/* Phase 4 special/devices: whether device nodes (--devices) and special files
|
||||
* (--specials) are preserved via recreation, and whether --copy-devices
|
||||
* copies a device's content as an ordinary regular file. */
|
||||
@@ -120,13 +114,6 @@ typedef struct {
|
||||
* directories, exactly like rsync; the receive root is the "." sentinel.
|
||||
* Guarded by `excluded_mutex`. */
|
||||
ArrayList* synced_dirs;
|
||||
/* Per-directory filter-rule sink (optional): when non-NULL the scanner appends
|
||||
* a deep copy of every rule it reads from a per-directory merge file, each
|
||||
* carrying its owner directory and no-inherit flag (see filter.h). The delete
|
||||
* carriers transmit them so the receiver re-derives the per-directory
|
||||
* protect/risk set for destination-only entries. Guarded by `excluded_mutex`
|
||||
* like the other sinks. */
|
||||
FilterRuleList* per_dir_rules;
|
||||
/* Delete-plan directory sink (optional): when non-NULL the scanner appends
|
||||
* the destination-relative path of every directory it traverses (except the
|
||||
* receive root). The per-directory --delete-during/--delete-delay plan
|
||||
@@ -134,29 +121,9 @@ typedef struct {
|
||||
* it) and to emit its plan after the data stream, when no file frame would
|
||||
* otherwise trigger it. Guarded by `excluded_mutex`. */
|
||||
ArrayList* plan_dirs;
|
||||
/* --ignore-errors: an unreadable subdirectory no longer aborts the scan (it
|
||||
* is always skipped so the rest of the tree transfers); this flag is kept so
|
||||
* the client can distinguish the option state when deciding deletion policy.
|
||||
* Client-only. */
|
||||
/* --ignore-errors: an unreadable directory during the scan is recorded as an
|
||||
* I/O error and skipped instead of aborting the scan. Client-only. */
|
||||
bool ignore_io_errors;
|
||||
/* --info=nonreg: print rsync's `skipping non-regular file "NAME"` line for a
|
||||
* non-regular entry that is not being preserved. Client-only. */
|
||||
bool note_nonreg;
|
||||
/* --info=mount: print rsync's `[sender] skipping mount-point dir NAME` when
|
||||
* -xx drops a mount-point directory. Client-only. */
|
||||
bool note_mount;
|
||||
/* --stats directory accounting for a `-r` run (no -t/-p): a shared counter of
|
||||
* traversed directories that are NOT otherwise represented by an inline
|
||||
* directory entry (rsync still counts every directory in `Number of files`).
|
||||
* Incremented when a directory is opened and decremented when an empty
|
||||
* directory is emitted inline (so it is counted exactly once). Atomic
|
||||
* because the parallel scanner's workers share it; NULL disables the
|
||||
* accounting. Client-only. */
|
||||
atomic_ullong* dir_count;
|
||||
/* Source root and 8-bit-output policy used to render a `--info=nonreg` name
|
||||
* relative to the transfer root. Borrowed read-only. */
|
||||
const char* send_directory;
|
||||
bool eight_bit_output;
|
||||
/* --ignore-missing-args (implied by --delete-missing-args): an explicitly
|
||||
* --files-from-listed entry that does not exist under the source is skipped
|
||||
* instead of failing (the --dirs generator is the only scanner path that
|
||||
@@ -184,17 +151,6 @@ typedef struct {
|
||||
bool capture_dir_times;
|
||||
ArrayList* dir_entries;
|
||||
mtx_t* dir_entries_mutex;
|
||||
/* Recreate empty source directories on a recursive transfer: emit a
|
||||
* payload-less directory entry for every traversed directory that produced
|
||||
* no transferred/descended child. Off by default so low-level scanner users
|
||||
* (unit helpers, --list-only) see only the historical file list; the real
|
||||
* sender sets it in prepare_scanner. */
|
||||
bool emit_empty_dirs;
|
||||
/* --no-implied-dirs with -R + --files-from: a directory that is only an
|
||||
* implied parent of a listed entry (not itself listed, nor below a listed
|
||||
* directory) must not carry source metadata; it is created with default
|
||||
* attributes at the destination, matching rsync. */
|
||||
bool no_implied_dirs;
|
||||
} ScannerOptions;
|
||||
|
||||
/* Internal per-scanner filter state. FilterNode chains represent the ordered
|
||||
@@ -212,22 +168,6 @@ typedef struct {
|
||||
int current_depth;
|
||||
dev_t root_dev;
|
||||
bool failed;
|
||||
/* rsync-order traversal: each opened directory's entries are inspected once
|
||||
and buffered (an internal SortedEntry[] owned here) sorted as rsync's flist
|
||||
orders them -- non-directories ascending, then directories ascending. The
|
||||
entries are walked in order and child directories are collected in
|
||||
`pending_dirs` (an ArrayList of DirEntry*, owned here) and pushed onto the
|
||||
LIFO `directories` stack in reverse at directory exhaustion, so the emitted
|
||||
stream is depth-first like rsync. `sorted_*` are reset per directory. */
|
||||
void* sorted_entries;
|
||||
size_t sorted_count;
|
||||
size_t sorted_index;
|
||||
void* pending_dirs;
|
||||
/* Recursive scan: whether the open directory yielded any transferred or
|
||||
descended entry. When it did not, closing it emits a directory entry so
|
||||
the empty source directory is recreated at the destination (rsync
|
||||
parity). */
|
||||
bool current_dir_produced;
|
||||
/* Phase 2 (files-from / filter layer). */
|
||||
char* root_path; /* transfer root (fs path) for rel computation */
|
||||
char* current_rel; /* rel path of the open directory ("" == root) */
|
||||
@@ -246,10 +186,6 @@ typedef struct {
|
||||
--ignore-errors the scan continues past it and the caller decides what to
|
||||
do; `failed` is reserved for fatal errors that always abort the scan. */
|
||||
bool io_error;
|
||||
/* The transfer ROOT could not be opened. It is always fatal, even under
|
||||
--ignore-errors, but the client still maps it to rsync's partial-transfer
|
||||
exit (23) rather than a generic failure. */
|
||||
bool root_io_error;
|
||||
} DirectoryScanner;
|
||||
|
||||
typedef struct {
|
||||
@@ -269,8 +205,7 @@ typedef struct {
|
||||
int completed;
|
||||
Chunk* initial_chunk;
|
||||
ProtocolSession* allocation_session;
|
||||
FilterNode* root_filter_node; /* root .rsync-filter context (owned by ps) */
|
||||
const ScannerOptions* options; /* borrowed scan options (--info=nonreg output) */
|
||||
FilterNode* root_filter_node; /* root .rsync-filter context (owned by ps) */
|
||||
} ParallelScanner;
|
||||
|
||||
DirectoryScanner* directory_scanner_create(const char* root_directory, bool use_metadata,
|
||||
@@ -289,7 +224,7 @@ void directory_scanner_destroy(DirectoryScanner* scanner);
|
||||
/* --one-file-system (-x) decision: a directory entry may be descended into
|
||||
* only when the option is disabled or the entry lives on the same device as
|
||||
* the transfer root. Exposed so tests can exercise the rule directly. */
|
||||
bool scanner_same_filesystem(int one_file_system, dev_t root_device, dev_t entry_device);
|
||||
bool scanner_same_filesystem(bool one_file_system, dev_t root_device, dev_t entry_device);
|
||||
|
||||
/* Relative path of an on-disk path below `root` ("" == the root itself, NULL
|
||||
* when `fs_path` is not under `root`). Handles trailing slashes and a root of
|
||||
|
||||
@@ -1,793 +0,0 @@
|
||||
#include "log.h"
|
||||
#include "scanner.h"
|
||||
#include "scanner_internal.h"
|
||||
#include "array_list.h"
|
||||
#include "chunk.h"
|
||||
#include "file.h"
|
||||
#include "queue.h"
|
||||
#include "utils.h"
|
||||
#include <dirent.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <sys/sysmacros.h>
|
||||
#include <threads.h>
|
||||
#include <unistd.h>
|
||||
#include <limits.h>
|
||||
|
||||
#include "xattr.h"
|
||||
|
||||
/* A chain node: `own` holds the .rsync-filter rules of one directory, `parent`
|
||||
* the context that directory inherited (nearest ancestor with a filter file).
|
||||
* The chain for a directory's contents runs from that directory's own node up
|
||||
* to the root; the command-line base rules are evaluated after the whole
|
||||
* chain. */
|
||||
struct FilterNode {
|
||||
FilterNode* parent;
|
||||
FilterRuleList* own;
|
||||
};
|
||||
|
||||
void filter_node_destroy(void* item) {
|
||||
if (item) {
|
||||
FilterNode* node = (FilterNode*)item;
|
||||
if (node->own)
|
||||
filter_rule_list_free(node->own);
|
||||
free(node);
|
||||
}
|
||||
}
|
||||
|
||||
FilterNode* filter_node_alloc(FilterNode* parent, FilterRuleList* own) {
|
||||
FilterNode* node = malloc(sizeof(FilterNode));
|
||||
if (!node)
|
||||
return NULL;
|
||||
node->parent = parent;
|
||||
node->own = own;
|
||||
return node;
|
||||
}
|
||||
|
||||
/* Evaluate a rule chain for one entry. rsync precedence, highest first: the
|
||||
* innermost (current) directory's .rsync-filter rules, then each ancestor's,
|
||||
* then the root's, and finally the command-line base rules (--filter/-C). The
|
||||
* sender-side verdict decides whether the entry is hidden from the transfer;
|
||||
* the receiver-side verdict decides whether its destination mirror is protected
|
||||
* from --delete. Each side takes the FIRST matching rule independently. */
|
||||
typedef struct {
|
||||
bool hide; /* sender-side exclude matched */
|
||||
bool protect; /* receiver-side exclude matched */
|
||||
} FilterOutcome;
|
||||
|
||||
static void chain_rules_outcome(const FilterRuleList* base, const FilterNode* node, const char* rel,
|
||||
const char* leaf, bool is_dir, FilterOutcome* out) {
|
||||
memset(out, 0, sizeof(*out));
|
||||
bool sender_decided = false;
|
||||
bool receiver_decided = false;
|
||||
const FilterNode* n = node;
|
||||
while (!sender_decided || !receiver_decided) {
|
||||
const FilterRuleList* list = n ? n->own : base;
|
||||
if (list) {
|
||||
if (!sender_decided) {
|
||||
FilterAction action = filter_rules_apply_side(list, rel, leaf, is_dir, FILTER_SIDE_SENDER);
|
||||
if (action != FILTER_ACTION_NONE) {
|
||||
out->hide = action == FILTER_ACTION_EXCLUDE;
|
||||
sender_decided = true;
|
||||
}
|
||||
}
|
||||
if (!receiver_decided) {
|
||||
FilterAction action =
|
||||
filter_rules_apply_side(list, rel, leaf, is_dir, FILTER_SIDE_RECEIVER);
|
||||
if (action != FILTER_ACTION_NONE) {
|
||||
out->protect = action == FILTER_ACTION_PROTECT;
|
||||
receiver_decided = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!n)
|
||||
break;
|
||||
n = n->parent;
|
||||
}
|
||||
}
|
||||
|
||||
static bool entry_allowed(const FilterRuleList* base, const FilterNode* node, const char* rel,
|
||||
const char* leaf, bool is_dir, bool exclude_filter_files,
|
||||
bool* protect_out) {
|
||||
/* -FF: per-directory .rsync-filter files are never transferred (single -F
|
||||
transfers them, matching rsync). */
|
||||
if (exclude_filter_files && !is_dir && strcmp(leaf, ".rsync-filter") == 0) {
|
||||
if (protect_out)
|
||||
*protect_out = false;
|
||||
return false;
|
||||
}
|
||||
FilterOutcome outcome;
|
||||
chain_rules_outcome(base, node, rel, leaf, is_dir, &outcome);
|
||||
if (protect_out)
|
||||
*protect_out = outcome.protect;
|
||||
return !outcome.hide;
|
||||
}
|
||||
|
||||
void dir_entry_destroy(void* item) {
|
||||
if (item) {
|
||||
DirEntry* de = (DirEntry*)item;
|
||||
free(de->path);
|
||||
free(de);
|
||||
}
|
||||
}
|
||||
|
||||
DirEntry* dir_entry_create(const char* path, int depth, FilterNode* context) {
|
||||
DirEntry* de = malloc(sizeof(DirEntry));
|
||||
if (!de)
|
||||
return NULL;
|
||||
de->path = str_dup(path);
|
||||
if (!de->path) {
|
||||
free(de);
|
||||
return NULL;
|
||||
}
|
||||
de->depth = depth;
|
||||
de->context = context;
|
||||
return de;
|
||||
}
|
||||
|
||||
/* Apply rsync's symlink-resolution precedence to one S_ISLNK entry:
|
||||
* --copy-links dereferences every symlink;
|
||||
* --copy-unsafe-links dereferences only targets unsafe_symlink() flags;
|
||||
* -k/--copy-dirlinks dereferences only a symlink whose referent is a dir;
|
||||
* --safe-links (receiver-side in rsync; modelled here) ignores an unsafe
|
||||
* target that would otherwise be carried; with --munge-links
|
||||
* every stored target becomes absolute, so --safe-links then
|
||||
* ignores every symlink, exactly as rsync documents;
|
||||
* -l/--links carries the link.
|
||||
* `link_rel` is the symlink's transfer-relative path (incl. name) and is used
|
||||
* only for the lexical unsafe test. `target` receives the raw link value. */
|
||||
LinkAction scanner_link_action(const ScannerOptions* options, const char* path,
|
||||
const char* link_rel, char* target, size_t target_size) {
|
||||
if (!options->follow_symlinks && !options->copy_links && !options->safe_links &&
|
||||
!options->copy_unsafe_links && !options->copy_dirlinks)
|
||||
return LINK_ACTION_SKIP;
|
||||
ssize_t length = readlink(path, target, target_size - 1);
|
||||
if (length < 0)
|
||||
return LINK_ACTION_SKIP;
|
||||
target[length] = '\0';
|
||||
|
||||
bool unsafe = file_symlink_unsafe(target, link_rel);
|
||||
if (options->copy_links || (options->copy_unsafe_links && unsafe))
|
||||
return LINK_ACTION_DEREF;
|
||||
if (options->copy_dirlinks) {
|
||||
struct stat ref;
|
||||
if (stat(path, &ref) == 0 && S_ISDIR(ref.st_mode))
|
||||
return LINK_ACTION_DEREF;
|
||||
}
|
||||
if (options->safe_links && (unsafe || options->munge_links))
|
||||
return LINK_ACTION_SKIP_PROTECTED;
|
||||
if (!options->follow_symlinks || target[0] == '\0')
|
||||
return LINK_ACTION_SKIP;
|
||||
return LINK_ACTION_CARRY;
|
||||
}
|
||||
|
||||
/* --one-file-system (-x) decision. Only directories can carry a different
|
||||
* device than their parent (mount points), so this is checked when a child
|
||||
* directory is about to be descended into. */
|
||||
bool scanner_same_filesystem(int one_file_system, dev_t root_device, dev_t entry_device) {
|
||||
return one_file_system <= 0 || entry_device == root_device;
|
||||
}
|
||||
|
||||
/* Build a payload-less directory File carrying the captured metadata (when
|
||||
* requested). Used by -x mount-point emission and --list-only directory
|
||||
* entries. Returns NULL on allocation failure. */
|
||||
File* scanner_build_dir_file(const char* path, const struct stat* stats,
|
||||
const ScannerOptions* options) {
|
||||
File* dir = file_create(path);
|
||||
if (dir == NULL)
|
||||
return NULL;
|
||||
dir->is_dir = true;
|
||||
if (options->use_metadata) {
|
||||
dir->metadata =
|
||||
file_metadata_create(dir->path, stats, options->preserve_atimes, options->preserve_crtimes);
|
||||
if (!dir->metadata) {
|
||||
file_destroy(dir);
|
||||
return NULL;
|
||||
}
|
||||
}
|
||||
return dir;
|
||||
}
|
||||
|
||||
/* Relative path of an on-disk path below `root`. The transfer root may be
|
||||
* given with a trailing slash; the returned rel path never has one and is ""
|
||||
* for the root itself. A root of "/" is handled (its children start at "/").
|
||||
* Exposed so tests can exercise the mapping directly. */
|
||||
char* scanner_path_relative(const char* root, const char* fs_path) {
|
||||
size_t root_len = strlen(root);
|
||||
while (root_len > 1 && root[root_len - 1] == '/')
|
||||
root_len--;
|
||||
if (strncmp(root, fs_path, root_len) != 0)
|
||||
return NULL;
|
||||
if (root_len == 1 && root[0] == '/') {
|
||||
if (fs_path[1] == '\0')
|
||||
return str_dup("");
|
||||
return str_dup(fs_path + 1);
|
||||
}
|
||||
if (fs_path[root_len] == '\0')
|
||||
return str_dup("");
|
||||
if (fs_path[root_len] != '/')
|
||||
return NULL;
|
||||
return str_dup(fs_path + root_len + 1);
|
||||
}
|
||||
|
||||
/* -R/--relative destination-relative prefix reconstructed from a source spec:
|
||||
* everything after the first '.' path component (rsync's '/./' cut point),
|
||||
* with leading/trailing slashes removed; or the whole spec (normalized) when
|
||||
* there is no cut. Returns "" for the receive root. Exposed for tests. */
|
||||
char* scanner_relative_prefix(const char* spec) {
|
||||
if (!spec || spec[0] == '\0')
|
||||
return NULL;
|
||||
const char* after = spec;
|
||||
if (spec[0] == '.' && spec[1] == '/') {
|
||||
after = spec + 2;
|
||||
} else {
|
||||
const char* cut = strstr(spec, "/./");
|
||||
if (cut)
|
||||
after = cut + 3;
|
||||
}
|
||||
size_t cap = strlen(spec) + 1;
|
||||
char* out = malloc(cap);
|
||||
if (!out)
|
||||
return NULL;
|
||||
size_t len = 0;
|
||||
for (const char* s = after; *s;) {
|
||||
while (*s == '/')
|
||||
s++;
|
||||
const char* comp = s;
|
||||
while (*s && *s != '/')
|
||||
s++;
|
||||
size_t clen = (size_t)(s - comp);
|
||||
if (clen == 0 || (clen == 1 && comp[0] == '.'))
|
||||
continue;
|
||||
if (len)
|
||||
out[len++] = '/';
|
||||
memcpy(out + len, comp, clen);
|
||||
len += clen;
|
||||
}
|
||||
out[len] = '\0';
|
||||
return out;
|
||||
}
|
||||
|
||||
/* Relative path of a child entry below the current directory. */
|
||||
char* child_rel_path(const char* parent_rel, const char* name) {
|
||||
if (!parent_rel || parent_rel[0] == '\0')
|
||||
return str_dup(name);
|
||||
return path_cat(parent_rel, name);
|
||||
}
|
||||
|
||||
/* Destination-relative wire path for an entry under an -R prefix. */
|
||||
char* scanner_prefix_send_path(const char* prefix, const char* rel) {
|
||||
if (prefix[0] == '\0')
|
||||
return str_dup(rel);
|
||||
if (rel[0] == '\0')
|
||||
return str_dup(prefix);
|
||||
return path_cat(prefix, rel);
|
||||
}
|
||||
|
||||
/* Apply the --files-from allow-set and the filter layer to one entry. On
|
||||
* return `*protect_out` is true when a receiver-side rule protects the entry's
|
||||
* destination mirror from deletion. */
|
||||
bool entry_passes_selection(const FileListSet* file_list, const FilterRuleList* base,
|
||||
const FilterNode* node, const char* rel, const char* leaf, bool is_dir,
|
||||
bool per_dir_filters, bool exclude_filter_files, bool* protect_out) {
|
||||
if (protect_out)
|
||||
*protect_out = false;
|
||||
if (file_list && !file_list_affects(file_list, rel))
|
||||
return false;
|
||||
if (base || per_dir_filters)
|
||||
return entry_allowed(base, node, rel, leaf, is_dir, exclude_filter_files, protect_out);
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Best-effort capture of the file's whitelisted xattrs (-X/-A). A failure to
|
||||
* read xattrs is non-fatal: the file is transferred without them. A symlink
|
||||
* entry reads the LINK's own xattrs (never the referent's) with the no-follow
|
||||
* variant; on Linux the VFS refuses xattrs on symlinks, so that yields NULL. */
|
||||
void scanner_capture_xattrs_opts(const ScannerOptions* options, File* file) {
|
||||
if (!options || !file || !(options->preserve_xattrs || options->preserve_acls))
|
||||
return;
|
||||
file->xattrs = file->is_symlink ? xattr_capture_path_nofollow(file->path, options->preserve_acls)
|
||||
: xattr_capture_path(file->path, options->preserve_acls);
|
||||
}
|
||||
|
||||
void scanner_capture_xattrs(const DirectoryScanner* scanner, File* file) {
|
||||
if (!scanner)
|
||||
return;
|
||||
scanner_capture_xattrs_opts(&scanner->options, file);
|
||||
}
|
||||
|
||||
/* Apply --hard-links (-H) detection to one regular File. On a sibling (a
|
||||
* later member of an already-seen source inode) the File keeps the group id
|
||||
* and the first member's wire path but carries NO data payload (size 0); the
|
||||
* first member is left untouched (data present, link_first). Returns false on
|
||||
* allocation failure (the caller marks the scan failed); the File stays usable
|
||||
* either way. */
|
||||
bool scanner_assign_hardlink(HardLinkTable* table, File* file, const struct stat* stats) {
|
||||
if (!table || !file || !stats)
|
||||
return true;
|
||||
int gid;
|
||||
bool is_first;
|
||||
char* first_path = NULL;
|
||||
if (!hardlink_table_assign(table, file_wire_path(file), stats->st_dev, stats->st_ino, &gid,
|
||||
&is_first, &first_path))
|
||||
return false;
|
||||
file->link_group = gid;
|
||||
file->link_first = is_first;
|
||||
if (!is_first) {
|
||||
file->hardlink_target = first_path;
|
||||
file->data->size = 0;
|
||||
} else {
|
||||
free(first_path);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Phase 4 special/devices decision for one non-regular entry, matching rsync:
|
||||
- a char/block device is RECREATED as a node under -D/--devices, unless
|
||||
--copy-devices asks for its content to be copied into a regular file;
|
||||
- a FIFO/socket is RECREATED under --specials;
|
||||
- when the matching flag is absent the entry is SKIPPED ("skipping
|
||||
non-regular file"), exactly like rsync's default, instead of being
|
||||
silently copied as a zero-length regular file;
|
||||
- anything else (regular/directory) is left to the normal data path. */
|
||||
ScannerSpecial scanner_prepare_special(bool preserve_devices, bool preserve_specials,
|
||||
bool copy_devices, File* file, const struct stat* stats) {
|
||||
if (!file || !stats)
|
||||
return SCANNER_SPECIAL_REGULAR;
|
||||
bool is_device = S_ISCHR(stats->st_mode) || S_ISBLK(stats->st_mode);
|
||||
bool is_fifo = S_ISFIFO(stats->st_mode);
|
||||
bool is_socket = S_ISSOCK(stats->st_mode);
|
||||
if (!is_device && !is_fifo && !is_socket)
|
||||
return SCANNER_SPECIAL_REGULAR;
|
||||
if (is_device && copy_devices)
|
||||
return SCANNER_SPECIAL_REGULAR; /* copy device content as a regular file */
|
||||
bool preserve = is_device ? preserve_devices : preserve_specials;
|
||||
if (!preserve)
|
||||
return SCANNER_SPECIAL_SKIP;
|
||||
file->is_special = true;
|
||||
file->data->size = 0;
|
||||
file->data->data = NULL;
|
||||
if (is_device) {
|
||||
file->rdev_major = (int32_t)major(stats->st_rdev);
|
||||
file->rdev_minor = (int32_t)minor(stats->st_rdev);
|
||||
}
|
||||
return SCANNER_SPECIAL_RECREATE;
|
||||
}
|
||||
|
||||
/* Append `rel` to the caller's exclusion sink, taking `mtx` when shared across
|
||||
parallel worker threads. Returns false on allocation failure (list left
|
||||
unchanged). */
|
||||
bool excluded_sink_append(ArrayList* list, mtx_t* mtx, const char* rel) {
|
||||
if (!list)
|
||||
return true;
|
||||
char* dup = str_dup(rel);
|
||||
if (!dup)
|
||||
return false;
|
||||
if (mtx)
|
||||
mtx_lock(mtx);
|
||||
bool ok = array_list_add(list, dup);
|
||||
if (mtx)
|
||||
mtx_unlock(mtx);
|
||||
if (!ok)
|
||||
free(dup);
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Record one pruned filesystem path in a delete-protection sink. The stored
|
||||
form is the entry's wire/destination-relative path (a single leading '/'
|
||||
removed, exactly how manifest keep entries are stored), so the receiver's
|
||||
walker prefixes match the destination layout. An allocation failure is a
|
||||
fatal scan error. */
|
||||
static void scanner_record_protected(DirectoryScanner* scanner, const char* fs_path,
|
||||
ArrayList* sink) {
|
||||
if (!sink || !fs_path)
|
||||
return;
|
||||
const char* rel = *fs_path == '/' ? fs_path + 1 : fs_path;
|
||||
if (!excluded_sink_append(sink, scanner->options.excluded_mutex, rel))
|
||||
scanner->failed = true;
|
||||
}
|
||||
|
||||
/* rsync's `--info=nonreg` line for a non-regular entry that is not being
|
||||
* preserved: `skipping non-regular file "NAME"`. The name is the path relative
|
||||
* to the transfer root, so it matches rsync's displayed name. */
|
||||
void scanner_note_nonreg(const ScannerOptions* options, const char* fs_path) {
|
||||
if (!options || !options->note_nonreg || !fs_path)
|
||||
return;
|
||||
const char* rel = utils_strip_transfer_root(fs_path, options->send_directory);
|
||||
char* escaped = output_escape(rel, options->eight_bit_output);
|
||||
printf("skipping non-regular file \"%s\"\n", escaped ? escaped : rel);
|
||||
free(escaped);
|
||||
fflush(stdout);
|
||||
}
|
||||
|
||||
/* Construct one non-directory File from an inspected entry. Shared by the
|
||||
* sequential and parallel scanners so entry construction has a single
|
||||
* implementation: data size (or carried symlink), -R wire path, special/devices
|
||||
* classification, hardlink group, metadata and xattr capture all happen here in
|
||||
* the same order for both. See the declaration for the ownership contract. */
|
||||
ScannerBuildStatus scanner_build_file_entry(const ScannerOptions* options, ScannerEntry* inspected,
|
||||
const char* rel, File** out_file, bool* failed) {
|
||||
*out_file = NULL;
|
||||
if (failed)
|
||||
*failed = false;
|
||||
File* file = file_create(inspected->path);
|
||||
if (!file) {
|
||||
/* The File never existed, so drop the not-yet-transferred symlink target
|
||||
here; the caller's entry teardown would otherwise double-free it. */
|
||||
free(inspected->link_target);
|
||||
inspected->link_target = NULL;
|
||||
return SCANNER_BUILD_FAIL_CONTINUE;
|
||||
}
|
||||
if (inspected->is_symlink) {
|
||||
file->is_symlink = true;
|
||||
file->symlink_target = inspected->link_target;
|
||||
inspected->link_target = NULL;
|
||||
} else {
|
||||
file->data->size = inspected->stats.st_size;
|
||||
}
|
||||
/* -R + --files-from uses the bare transfer-relative path; -R without
|
||||
--files-from prefixes it. Plain scans keep the source path. */
|
||||
bool relative_mode = options->relative && options->file_list != NULL;
|
||||
if (relative_mode) {
|
||||
file->send_path = str_dup(rel);
|
||||
} else if (options->relative_prefix) {
|
||||
file->send_path = scanner_prefix_send_path(options->relative_prefix, rel);
|
||||
}
|
||||
if ((relative_mode || options->relative_prefix) && !file->send_path) {
|
||||
file_destroy(file);
|
||||
return SCANNER_BUILD_FAIL_BREAK;
|
||||
}
|
||||
/* --devices/--specials: a device/FIFO/socket entry marked for preservation
|
||||
becomes a node to recreate (is_special, no data, rdev captured); an
|
||||
unrequested non-regular entry is skipped (rsync default). */
|
||||
ScannerSpecial special =
|
||||
scanner_prepare_special(options->preserve_devices, options->preserve_specials,
|
||||
options->copy_devices, file, &inspected->stats);
|
||||
if (special == SCANNER_SPECIAL_SKIP) {
|
||||
scanner_note_nonreg(options, file->path);
|
||||
file_destroy(file);
|
||||
return SCANNER_BUILD_SKIP;
|
||||
}
|
||||
if (options->hardlinks && S_ISREG(inspected->stats.st_mode) &&
|
||||
!scanner_assign_hardlink(options->hardlinks, file, &inspected->stats)) {
|
||||
/* Allocation failure is non-fatal to this entry (it is still emitted) but
|
||||
marks the scan failed, matching the historical inlined behaviour. */
|
||||
if (failed)
|
||||
*failed = true;
|
||||
}
|
||||
if (options->use_metadata) {
|
||||
file->metadata = file_metadata_create(file->path, &inspected->stats, options->preserve_atimes,
|
||||
options->preserve_crtimes);
|
||||
if (!file->metadata) {
|
||||
file_destroy(file);
|
||||
return SCANNER_BUILD_FAIL_BREAK;
|
||||
}
|
||||
}
|
||||
/* A hardlink sibling carries no data, so it carries no xattrs. */
|
||||
if (!(file->link_group != 0 && !file->link_first))
|
||||
scanner_capture_xattrs_opts(options, file);
|
||||
*out_file = file;
|
||||
return SCANNER_BUILD_OK;
|
||||
}
|
||||
|
||||
/* rsync 3.4.1's `--info=mount` line, emitted when `-xx` drops a mount-point
|
||||
* directory: `[sender] skipping mount-point dir NAME` (the client is the
|
||||
* sender). Plain `-x` keeps the empty directory and prints nothing, matching
|
||||
* rsync. */
|
||||
void scanner_note_mount(const ScannerOptions* options, const char* fs_path) {
|
||||
if (!options || !options->note_mount || !fs_path)
|
||||
return;
|
||||
const char* rel = utils_strip_transfer_root(fs_path, options->send_directory);
|
||||
char* escaped = output_escape(rel, options->eight_bit_output);
|
||||
printf("[sender] skipping mount-point dir %s\n", escaped ? escaped : rel);
|
||||
free(escaped);
|
||||
fflush(stdout);
|
||||
}
|
||||
|
||||
/* --debug=filter: a selection/filter decision dropped an entry. */
|
||||
void scanner_note_filter(const ScannerOptions* options, const char* name) {
|
||||
if (!options || !log_debug_enabled(LOG_DEBUG_FILTER) || !name)
|
||||
return;
|
||||
log_debug_message(LOG_DEBUG_FILTER, "filter: excluded %s", name);
|
||||
}
|
||||
|
||||
/* Account for a directory that will not be represented by an inline directory
|
||||
* entry. Paired with scanner_dir_count_uncount for empty directories that are
|
||||
* emitted inline, so every traversed directory is counted exactly once. */
|
||||
void scanner_dir_count_count(const ScannerOptions* options) {
|
||||
if (options && options->dir_count)
|
||||
atomic_fetch_add(options->dir_count, 1);
|
||||
}
|
||||
|
||||
void scanner_dir_count_uncount(const ScannerOptions* options) {
|
||||
if (options && options->dir_count)
|
||||
atomic_fetch_sub(options->dir_count, 1);
|
||||
}
|
||||
|
||||
/* A user-selection exclusion (--filter/-C/per-dir or --exclude/--include). */
|
||||
void scanner_record_excluded(DirectoryScanner* scanner, const char* fs_path) {
|
||||
scanner_record_protected(scanner, fs_path, scanner->options.excluded_paths);
|
||||
}
|
||||
|
||||
/* A --max-size/--min-size prune (always protected, even under --delete-excluded). */
|
||||
void scanner_record_size_skipped(DirectoryScanner* scanner, const char* fs_path) {
|
||||
scanner_record_protected(scanner, fs_path, scanner->options.size_skipped_paths);
|
||||
}
|
||||
|
||||
/* The destination-relative coordinate the receiver's delete walkers match
|
||||
against for an entry at `fs_path` (with `rel` its path relative to the
|
||||
transfer root, "" for the root): `relative_prefix + rel` under -R+--relative,
|
||||
the bare relative path under -R+--files-from, else the source path with a
|
||||
leading '/' removed, with "." for the receive root. Shared by the
|
||||
synchronized-directory sink and the mirrored per-directory rule owners so
|
||||
both live in the same coordinate system. Returns an owned string, or NULL on
|
||||
allocation failure. */
|
||||
char* scanner_dest_rel_path(const ScannerOptions* options, const char* fs_path, const char* rel,
|
||||
bool relative_mode) {
|
||||
char* prefixed = NULL;
|
||||
const char* dest;
|
||||
if (relative_mode) {
|
||||
dest = rel;
|
||||
} else if (options->relative_prefix) {
|
||||
prefixed = scanner_prefix_send_path(options->relative_prefix, rel);
|
||||
if (!prefixed)
|
||||
return NULL;
|
||||
dest = prefixed;
|
||||
} else {
|
||||
dest = fs_path;
|
||||
}
|
||||
if (dest[0] == '/')
|
||||
dest++;
|
||||
if (dest[0] == '\0')
|
||||
dest = ".";
|
||||
char* out = str_dup(dest);
|
||||
free(prefixed);
|
||||
return out;
|
||||
}
|
||||
|
||||
/* Record a directory the scan synchronized. `fs_path` is its absolute path and
|
||||
`rel` its path relative to the transfer root ("" for the root); the stored
|
||||
form matches the wire layout (see scanner_dest_rel_path). Returns false on
|
||||
allocation failure. */
|
||||
bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_path, const char* rel,
|
||||
bool relative_mode) {
|
||||
if (!options->synced_dirs && !options->plan_dirs)
|
||||
return true;
|
||||
if (!file_list_dir_in_scope(options->file_list, rel))
|
||||
return true;
|
||||
char* dest = scanner_dest_rel_path(options, fs_path, rel, relative_mode);
|
||||
if (!dest)
|
||||
return false;
|
||||
bool ok = true;
|
||||
if (options->synced_dirs)
|
||||
ok = excluded_sink_append(options->synced_dirs, options->excluded_mutex, dest);
|
||||
/* The delete-plan keep set needs an entry for every traversed source
|
||||
directory, including empty ones, so its destination mirror is kept rather
|
||||
than deleted as an extra; the receive root (".") is implicit. */
|
||||
if (ok && options->plan_dirs && strcmp(dest, ".") != 0)
|
||||
ok = excluded_sink_append(options->plan_dirs, options->excluded_mutex, dest);
|
||||
free(dest);
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Read every per-directory filter file that applies to `dir_path` (its
|
||||
* .rsync-filter when -F is active, plus each registered "dir-merge NAME") into a
|
||||
* fresh list. Returns NULL on allocation/parse failure (message in `err`);
|
||||
* returns an empty list (and *any_exists=false) when no file exists. */
|
||||
FilterRuleList* read_dir_filters(const ScannerOptions* options, const char* dir_path,
|
||||
const char* rel, bool relative_mode, bool* any_exists, char* err,
|
||||
size_t err_size) {
|
||||
if (err && err_size > 0)
|
||||
err[0] = '\0';
|
||||
const FilterRuleList* base = options->base_filters;
|
||||
bool have_names = options->per_dir_filters || (base && base->dir_merge_count > 0);
|
||||
if (any_exists)
|
||||
*any_exists = false;
|
||||
if (!have_names)
|
||||
return NULL;
|
||||
FilterRuleList* own = filter_rule_list_create();
|
||||
if (!own) {
|
||||
snprintf(err, err_size, "memory allocation failed");
|
||||
return NULL;
|
||||
}
|
||||
FilterParseOptions opts = {.delete_excluded = options->delete_excluded, .cvs_exclude = false};
|
||||
bool exists = false;
|
||||
if (options->per_dir_filters) {
|
||||
if (!filter_file_append(own, dir_path, ".rsync-filter", rel, &opts, &exists, err, err_size))
|
||||
goto fail;
|
||||
if (exists && any_exists)
|
||||
*any_exists = true;
|
||||
}
|
||||
if (base) {
|
||||
for (int i = 0; i < base->dir_merge_count; i++) {
|
||||
if (!filter_dir_merge_append(own, dir_path, &base->dir_merges[i], rel, &opts, &exists, err,
|
||||
err_size))
|
||||
goto fail;
|
||||
if (exists && any_exists)
|
||||
*any_exists = true;
|
||||
}
|
||||
}
|
||||
/* Mirror the directory's rules into the delete-carrier sink so the receiver
|
||||
* can reconstruct its per-directory protect/risk set. The mirrored rules
|
||||
* carry the destination-relative owner coordinate (not the transfer-root-
|
||||
* relative one the sender's own evaluation uses) so the receiver's delete
|
||||
* walkers, which match against receive-root-relative paths, find them. */
|
||||
if (options->per_dir_rules && own->count > 0) {
|
||||
char* owner = scanner_dest_rel_path(options, dir_path, rel, relative_mode);
|
||||
if (!owner)
|
||||
goto fail;
|
||||
mtx_t* mtx = options->excluded_mutex;
|
||||
if (mtx)
|
||||
mtx_lock(mtx);
|
||||
for (int i = 0; i < own->count; i++) {
|
||||
FilterRule* copy = filter_rule_clone(own->items[i]);
|
||||
if (!copy || !filter_rule_set_owner(copy, owner) ||
|
||||
!filter_rule_list_add(options->per_dir_rules, copy)) {
|
||||
filter_rule_free(copy);
|
||||
if (mtx)
|
||||
mtx_unlock(mtx);
|
||||
free(owner);
|
||||
goto fail;
|
||||
}
|
||||
}
|
||||
if (mtx)
|
||||
mtx_unlock(mtx);
|
||||
free(owner);
|
||||
}
|
||||
return own;
|
||||
fail:
|
||||
filter_rule_list_free(own);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* Merge the open directory's own per-directory filter files (the default
|
||||
* .rsync-filter when -F is active, plus every "dir-merge NAME" registered on the
|
||||
* base rule list) into the inherited context, returning the context used for
|
||||
* this directory's entries. On a parse error the scanner is marked failed.
|
||||
* Returns 0 on success, -1 on failure. */
|
||||
int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* inherited) {
|
||||
char err[256];
|
||||
bool any_exists = false;
|
||||
FilterRuleList* own = read_dir_filters(&scanner->options, scanner->current_path,
|
||||
scanner->current_rel ? scanner->current_rel : "",
|
||||
scanner->relative_mode, &any_exists, err, sizeof(err));
|
||||
if (!own) {
|
||||
/* read_dir_filters() leaves `err` set on a parse/allocation failure even
|
||||
when an earlier merge file in the same directory existed (any_exists true);
|
||||
key off the error text rather than any_exists so an invalid per-directory
|
||||
filter file can never be silently ignored. */
|
||||
if (err[0] == '\0') {
|
||||
scanner->current_node = (FilterNode*)inherited;
|
||||
return 0;
|
||||
}
|
||||
char* escaped_path = output_escape(scanner->current_path, log_get_8_bit_output());
|
||||
log_message(LOG_LEVEL_ERROR, "invalid per-directory filter in %s: %s",
|
||||
escaped_path ? escaped_path : "<allocation failed>", err);
|
||||
free(escaped_path);
|
||||
scanner->failed = true;
|
||||
return -1;
|
||||
}
|
||||
if (any_exists && (own->count > 0 || own->dir_merge_count > 0)) {
|
||||
FilterNode* node = filter_node_alloc((FilterNode*)inherited, own);
|
||||
if (!node || !array_list_add(scanner->filter_nodes, node)) {
|
||||
filter_node_destroy(node);
|
||||
scanner->failed = true;
|
||||
return -1;
|
||||
}
|
||||
scanner->current_node = node;
|
||||
} else {
|
||||
filter_rule_list_free(own);
|
||||
scanner->current_node = (FilterNode*)inherited;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Inspect symlinks, resolve the entry type, and apply file filters once for both scanners.
|
||||
* `link_rel` is the entry's path relative to the transfer root (including its
|
||||
* name), used for the lexical rsync unsafe-symlink test. */
|
||||
int scanner_inspect_entry(const ScannerOptions* options, const char* containing_dir,
|
||||
const char* link_rel, const char* name, ScannerEntry* entry) {
|
||||
entry->excluded = false;
|
||||
entry->size_excluded = false;
|
||||
entry->referent_error = false;
|
||||
entry->is_symlink = false;
|
||||
entry->link_target = NULL;
|
||||
entry->path = path_cat(containing_dir, name);
|
||||
if (!entry->path)
|
||||
return -1;
|
||||
|
||||
struct stat link_stats;
|
||||
if (lstat(entry->path, &link_stats) != 0) {
|
||||
free(entry->path);
|
||||
return 0;
|
||||
}
|
||||
if (!S_ISLNK(link_stats.st_mode))
|
||||
goto regular;
|
||||
|
||||
char link_target[4096];
|
||||
switch (scanner_link_action(options, entry->path, link_rel, link_target, sizeof(link_target))) {
|
||||
case LINK_ACTION_SKIP:
|
||||
goto skip;
|
||||
case LINK_ACTION_SKIP_PROTECTED:
|
||||
/* --safe-links ignored the link, but rsync still counts it as present in
|
||||
the transfer, so its destination mirror survives --delete. Record it as
|
||||
an excluded path (the same delete-protection channel as a filter prune). */
|
||||
entry->excluded = true;
|
||||
goto skip;
|
||||
case LINK_ACTION_DEREF:
|
||||
if (stat(entry->path, &entry->stats) != 0) {
|
||||
/* rsync reports "symlink has no referent" and continues with a partial
|
||||
transfer (exit 23); record the error so the run exits 23 too. */
|
||||
char* escaped = output_escape(entry->path, log_get_8_bit_output());
|
||||
log_message(LOG_LEVEL_WARNING, "symlink has no referent: %s",
|
||||
escaped ? escaped : "<allocation failed>");
|
||||
free(escaped);
|
||||
entry->referent_error = true;
|
||||
goto skip;
|
||||
}
|
||||
entry->is_directory = S_ISDIR(entry->stats.st_mode);
|
||||
if (entry->is_directory)
|
||||
return 1;
|
||||
goto apply_filters;
|
||||
case LINK_ACTION_CARRY:
|
||||
break;
|
||||
}
|
||||
|
||||
/* Carry the link as a symlink. --munge-links is applied by the RECEIVER (it
|
||||
prefixes every stored target with /rsyncd-munged/); when the SOURCE already
|
||||
holds a munged value the sender strips it so the receiver re-munges a clean
|
||||
target, round-tripping a munged tree exactly like rsync. */
|
||||
entry->is_symlink = true;
|
||||
entry->stats = link_stats;
|
||||
entry->is_directory = false;
|
||||
entry->link_target = str_dup(link_target);
|
||||
if (!entry->link_target)
|
||||
goto skip;
|
||||
if (options->munge_links)
|
||||
file_symlink_unmunge(entry->link_target);
|
||||
goto apply_filters;
|
||||
|
||||
regular:
|
||||
/* Not a symlink: the lstat() above already described this entry, and lstat
|
||||
and stat are identical for every non-symlink, so reuse that result instead
|
||||
of issuing a redundant stat() on the scanner hot path. stat() is still
|
||||
used on the dereference paths above/below for actual symlinks (copy-links,
|
||||
safe/copy-unsafe links, and -k symlinks-to-directories). */
|
||||
entry->stats = link_stats;
|
||||
entry->is_directory = S_ISDIR(link_stats.st_mode);
|
||||
if (entry->is_directory)
|
||||
return 1;
|
||||
|
||||
apply_filters:
|
||||
for (int i = 0; i < options->exclude_count; i++)
|
||||
if (glob_match(options->exclude_patterns[i], name)) {
|
||||
entry->excluded = true;
|
||||
goto skip;
|
||||
}
|
||||
if (options->include_count > 0) {
|
||||
bool included = false;
|
||||
for (int i = 0; i < options->include_count; i++)
|
||||
if (glob_match(options->include_patterns[i], name))
|
||||
included = true;
|
||||
if (!included) {
|
||||
entry->excluded = true;
|
||||
goto skip;
|
||||
}
|
||||
}
|
||||
if ((options->max_size > 0 && (unsigned long long)entry->stats.st_size > options->max_size) ||
|
||||
(options->min_size > 0 && (unsigned long long)entry->stats.st_size < options->min_size)) {
|
||||
entry->excluded = true;
|
||||
entry->size_excluded = true;
|
||||
goto skip;
|
||||
}
|
||||
return 1;
|
||||
|
||||
skip:
|
||||
free(entry->path);
|
||||
entry->path = NULL;
|
||||
free(entry->link_target);
|
||||
entry->link_target = NULL;
|
||||
return 0;
|
||||
}
|
||||
@@ -1,132 +0,0 @@
|
||||
#ifndef SCANNER_INTERNAL_H
|
||||
#define SCANNER_INTERNAL_H
|
||||
|
||||
/* Internal declarations shared between the scanner translation units
|
||||
* (scanner_filter.c, scanner.c, scanner_parallel.c). Nothing here is part of
|
||||
* the public scanner façade (scanner.h); every symbol stays internal to the
|
||||
* client module. */
|
||||
|
||||
#include "array_list.h"
|
||||
#include "file.h"
|
||||
#include "scanner.h"
|
||||
#include <stdbool.h>
|
||||
#include <stddef.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
typedef struct {
|
||||
char* path;
|
||||
int depth;
|
||||
FilterNode* context; /* inherited per-directory filter context */
|
||||
} DirEntry;
|
||||
|
||||
/* How rsync's readlink_stat()/generator resolves one source symlink. */
|
||||
typedef enum {
|
||||
LINK_ACTION_SKIP, /* not transferred (no link option) */
|
||||
LINK_ACTION_SKIP_PROTECTED, /* ignored as unsafe by --safe-links; rsync keeps
|
||||
it in the transfer, so its destination mirror
|
||||
must be protected from --delete */
|
||||
LINK_ACTION_DEREF, /* follow the referent (--copy-links, an unsafe
|
||||
target under --copy-unsafe-links, or -k dir) */
|
||||
LINK_ACTION_CARRY, /* transmit the link itself (-l) */
|
||||
} LinkAction;
|
||||
|
||||
typedef struct {
|
||||
char* path;
|
||||
struct stat stats;
|
||||
bool is_directory;
|
||||
/* True when the entry should be carried through as a SYMLINK (is_symlink)
|
||||
rather than a dereferenced file/directory. When true, `link_target` holds
|
||||
the owned target string to transmit (sender-munged under --munge-links);
|
||||
ownership transfers to the File built from this entry. */
|
||||
bool is_symlink;
|
||||
char* link_target;
|
||||
/* True when the entry was pruned by a user selection rule (--filter/-C/per-dir
|
||||
rules or the --exclude/--include layer) rather than skipped for another
|
||||
reason (unreadable, symlink policy, not applicable). */
|
||||
bool excluded;
|
||||
/* True when the entry was skipped specifically by --max-size/--min-size.
|
||||
Size pruning protects the destination mirror even under --delete-excluded,
|
||||
so it is recorded into a separate sink from `excluded`. */
|
||||
bool size_excluded;
|
||||
/* True when a symlink selected for dereferencing (-L/--copy-links or an
|
||||
unsafe target under --copy-unsafe-links) had no usable referent (a broken
|
||||
link or a stat() failure). rsync still reports this as a partial transfer
|
||||
(exit 23) even though the entry is skipped, so the scanner records it as a
|
||||
non-fatal I/O error. */
|
||||
bool referent_error;
|
||||
} ScannerEntry;
|
||||
|
||||
typedef enum {
|
||||
SCANNER_SPECIAL_REGULAR, /* ordinary file: transfer content */
|
||||
SCANNER_SPECIAL_RECREATE, /* is_special node to recreate on the receiver */
|
||||
SCANNER_SPECIAL_SKIP, /* non-regular entry not requested: skip */
|
||||
} ScannerSpecial;
|
||||
|
||||
/* Result of scanner_build_file_entry(). The two failure variants preserve the
|
||||
* sequential scanner's historical distinction between a failure before the
|
||||
* File existed (which kept walking the directory) and one afterwards (which cut
|
||||
* the chunk short); both mark the scan failed. */
|
||||
typedef enum {
|
||||
SCANNER_BUILD_OK, /* File built; caller owns it */
|
||||
SCANNER_BUILD_SKIP, /* non-regular entry not preserved; no File */
|
||||
SCANNER_BUILD_FAIL_CONTINUE, /* failed before the File existed */
|
||||
SCANNER_BUILD_FAIL_BREAK, /* failed after the File existed */
|
||||
} ScannerBuildStatus;
|
||||
|
||||
/* scanner_filter.c */
|
||||
void filter_node_destroy(void* item);
|
||||
FilterNode* filter_node_alloc(FilterNode* parent, FilterRuleList* own);
|
||||
void dir_entry_destroy(void* item);
|
||||
DirEntry* dir_entry_create(const char* path, int depth, FilterNode* context);
|
||||
LinkAction scanner_link_action(const ScannerOptions* options, const char* path,
|
||||
const char* link_rel, char* target, size_t target_size);
|
||||
File* scanner_build_dir_file(const char* path, const struct stat* stats,
|
||||
const ScannerOptions* options);
|
||||
char* child_rel_path(const char* parent_rel, const char* name);
|
||||
char* scanner_prefix_send_path(const char* prefix, const char* rel);
|
||||
char* scanner_dest_rel_path(const ScannerOptions* options, const char* fs_path, const char* rel,
|
||||
bool relative_mode);
|
||||
bool entry_passes_selection(const FileListSet* file_list, const FilterRuleList* base,
|
||||
const FilterNode* node, const char* rel, const char* leaf, bool is_dir,
|
||||
bool per_dir_filters, bool exclude_filter_files, bool* protect_out);
|
||||
void scanner_capture_xattrs(const DirectoryScanner* scanner, File* file);
|
||||
void scanner_capture_xattrs_opts(const ScannerOptions* options, File* file);
|
||||
bool scanner_assign_hardlink(HardLinkTable* table, File* file, const struct stat* stats);
|
||||
ScannerSpecial scanner_prepare_special(bool preserve_devices, bool preserve_specials,
|
||||
bool copy_devices, File* file, const struct stat* stats);
|
||||
/* Build one non-directory transfer File from an inspected entry. `rel` is the
|
||||
* entry's transfer-root-relative path (used for the -R wire path); `inspected`
|
||||
* supplies the on-disk path, stats and (for a carried symlink) the target whose
|
||||
* ownership transfers to the File. Populates data size, send_path, special-node
|
||||
* state, hardlink group, metadata and xattrs. On SCANNER_BUILD_OK the caller
|
||||
* owns *out_file; on SCANNER_BUILD_SKIP it is NULL and the entry is dropped; on
|
||||
* either failure it is NULL and the caller must mark the scan failed. `*failed`
|
||||
* additionally reports a non-fatal hardlink-table allocation failure, in which
|
||||
* case a usable File is still returned. */
|
||||
ScannerBuildStatus scanner_build_file_entry(const ScannerOptions* options, ScannerEntry* inspected,
|
||||
const char* rel, File** out_file, bool* failed);
|
||||
bool excluded_sink_append(ArrayList* list, mtx_t* mtx, const char* rel);
|
||||
void scanner_note_nonreg(const ScannerOptions* options, const char* fs_path);
|
||||
void scanner_note_mount(const ScannerOptions* options, const char* fs_path);
|
||||
void scanner_note_filter(const ScannerOptions* options, const char* name);
|
||||
void scanner_dir_count_count(const ScannerOptions* options);
|
||||
void scanner_dir_count_uncount(const ScannerOptions* options);
|
||||
void scanner_record_excluded(DirectoryScanner* scanner, const char* fs_path);
|
||||
void scanner_record_size_skipped(DirectoryScanner* scanner, const char* fs_path);
|
||||
bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_path, const char* rel,
|
||||
bool relative_mode);
|
||||
FilterRuleList* read_dir_filters(const ScannerOptions* options, const char* dir_path,
|
||||
const char* rel, bool relative_mode, bool* any_exists, char* err,
|
||||
size_t err_size);
|
||||
int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* inherited);
|
||||
int scanner_inspect_entry(const ScannerOptions* options, const char* containing_dir,
|
||||
const char* link_rel, const char* name, ScannerEntry* entry);
|
||||
|
||||
/* scanner.c */
|
||||
bool scanner_capture_dir_time(ArrayList* dir_entries, mtx_t* mutex, const char* root_path,
|
||||
const char* fs_path, bool relative_mode, const char* relative_prefix,
|
||||
bool preserve_atimes, bool preserve_crtimes, bool preserve_xattrs,
|
||||
bool preserve_acls, bool no_implied_dirs,
|
||||
const FileListSet* file_list);
|
||||
|
||||
#endif
|
||||
@@ -1,681 +0,0 @@
|
||||
#include "log.h"
|
||||
#include "scanner.h"
|
||||
#include "scanner_internal.h"
|
||||
#include "array_list.h"
|
||||
#include "chunk.h"
|
||||
#include "file.h"
|
||||
#include "queue.h"
|
||||
#include "utils.h"
|
||||
#include <dirent.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <sys/sysmacros.h>
|
||||
#include <threads.h>
|
||||
#include <unistd.h>
|
||||
#include <limits.h>
|
||||
|
||||
#include "xattr.h"
|
||||
|
||||
typedef struct {
|
||||
ParallelScanner* ps;
|
||||
char** dirs;
|
||||
int dir_count;
|
||||
char* root_dir; /* the transfer root, for relative-path computation */
|
||||
ScannerOptions options;
|
||||
ProtocolSession* allocation_session;
|
||||
} ParallelWorkerArg;
|
||||
|
||||
static int parallel_worker_thread(void* arg) {
|
||||
ParallelWorkerArg* wa = (ParallelWorkerArg*)arg;
|
||||
ProtocolSession* allocation_session = wa->allocation_session;
|
||||
if (allocation_session)
|
||||
protocol_session_bind(allocation_session);
|
||||
for (int i = 0; i < wa->dir_count; i++) {
|
||||
DirectoryScanner* ds = directory_scanner_create_with_options(wa->dirs[i], &wa->options);
|
||||
if (!ds) {
|
||||
mtx_lock(&wa->ps->result_mutex);
|
||||
wa->ps->failed = true;
|
||||
atomic_store(&wa->ps->cancelled, true);
|
||||
cnd_broadcast(&wa->ps->result_not_empty);
|
||||
cnd_broadcast(&wa->ps->result_not_full);
|
||||
mtx_unlock(&wa->ps->result_mutex);
|
||||
for (int j = i; j < wa->dir_count; j++)
|
||||
free(wa->dirs[j]);
|
||||
break;
|
||||
}
|
||||
/* Root .rsync-filter rules (parsed by the parallel scanner) apply to the
|
||||
* contents of every assigned subdirectory. Relative paths (used by the
|
||||
* allow-set and per-directory rules) are computed against the transfer
|
||||
* root, not the subdirectory the worker is seeded with. Exclusion
|
||||
* recording shares one caller-owned list across the workers. */
|
||||
free(ds->root_path);
|
||||
ds->root_path = str_dup(wa->root_dir);
|
||||
ds->seed_node = wa->ps->root_filter_node;
|
||||
ds->options.excluded_mutex = &wa->ps->result_mutex;
|
||||
Chunk* chunk;
|
||||
while ((chunk = directory_scanner_next(ds)) != NULL) {
|
||||
if (!queue_enqueue_multithreaded_cancel(wa->ps->result_queue, chunk, &wa->ps->result_mutex,
|
||||
&wa->ps->result_not_empty, &wa->ps->result_not_full,
|
||||
&wa->ps->cancelled)) {
|
||||
chunk_destroy(chunk);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (directory_scanner_failed(ds)) {
|
||||
mtx_lock(&wa->ps->result_mutex);
|
||||
wa->ps->failed = true;
|
||||
atomic_store(&wa->ps->cancelled, true);
|
||||
cnd_broadcast(&wa->ps->result_not_empty);
|
||||
cnd_broadcast(&wa->ps->result_not_full);
|
||||
mtx_unlock(&wa->ps->result_mutex);
|
||||
} else if (directory_scanner_had_io_error(ds)) {
|
||||
/* --ignore-errors path: an unreadable directory was skipped, not fatal. */
|
||||
mtx_lock(&wa->ps->result_mutex);
|
||||
wa->ps->io_error = true;
|
||||
mtx_unlock(&wa->ps->result_mutex);
|
||||
}
|
||||
directory_scanner_destroy(ds);
|
||||
free(wa->dirs[i]);
|
||||
}
|
||||
ParallelScanner* ps = wa->ps;
|
||||
free(wa->root_dir);
|
||||
free(wa->dirs);
|
||||
free(wa);
|
||||
mtx_lock(&ps->result_mutex);
|
||||
ps->completed++;
|
||||
if (ps->completed >= ps->expected_threads) {
|
||||
ps->done = true;
|
||||
cnd_signal(&ps->result_not_empty);
|
||||
}
|
||||
mtx_unlock(&ps->result_mutex);
|
||||
if (allocation_session)
|
||||
protocol_session_unbind();
|
||||
return thrd_success;
|
||||
}
|
||||
|
||||
static void parallel_scanner_creation_failed(ParallelScanner* ps) {
|
||||
mtx_lock(&ps->result_mutex);
|
||||
ps->failed = true;
|
||||
atomic_store(&ps->cancelled, true);
|
||||
ps->expected_threads = ps->created_threads;
|
||||
if (ps->completed >= ps->expected_threads)
|
||||
ps->done = true;
|
||||
cnd_broadcast(&ps->result_not_empty);
|
||||
cnd_broadcast(&ps->result_not_full);
|
||||
mtx_unlock(&ps->result_mutex);
|
||||
}
|
||||
|
||||
/* Initialize result queue and synchronization primitives. Returns true on success. */
|
||||
static bool parallel_scanner_init(ParallelScanner* ps) {
|
||||
ps->result_queue = queue_create(SCANNER_RESULT_QUEUE_CAP, chunk_destroy);
|
||||
if (!ps->result_queue)
|
||||
return false;
|
||||
atomic_init(&ps->cancelled, false);
|
||||
int init = 0;
|
||||
bool ok = true;
|
||||
if (mtx_init(&ps->result_mutex, mtx_plain) != thrd_success)
|
||||
ok = false;
|
||||
if (ok) {
|
||||
init++;
|
||||
if (cnd_init(&ps->result_not_empty) != thrd_success)
|
||||
ok = false;
|
||||
}
|
||||
if (ok) {
|
||||
// cppcheck-suppress unreadVariable
|
||||
init++;
|
||||
if (cnd_init(&ps->result_not_full) != thrd_success)
|
||||
ok = false;
|
||||
}
|
||||
if (!ok) {
|
||||
if (init >= 3)
|
||||
cnd_destroy(&ps->result_not_full);
|
||||
if (init >= 2)
|
||||
cnd_destroy(&ps->result_not_empty);
|
||||
if (init >= 1)
|
||||
mtx_destroy(&ps->result_mutex);
|
||||
queue_destroy(ps->result_queue);
|
||||
ps->result_queue = NULL;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Split files into chunks of roughly chunk_size bytes. Returns the first chunk (also stored
|
||||
* chunks beyond the first are enqueued on `queue`). Nulls out consumed entries in `files`.
|
||||
* Sets *failed on allocation/enqueue errors. */
|
||||
static Chunk* batch_files(ArrayList* files, unsigned long long chunk_size, Queue* queue,
|
||||
bool* failed) {
|
||||
Chunk* first = NULL;
|
||||
if (files->size <= 0)
|
||||
return NULL;
|
||||
ArrayList* batch = array_list_create(NULL);
|
||||
if (!batch) {
|
||||
*failed = true;
|
||||
return NULL;
|
||||
}
|
||||
unsigned long long batch_size = 0;
|
||||
for (int i = 0; i < files->size; i++) {
|
||||
File* f = (File*)files->items[i];
|
||||
if (!array_list_add(batch, f)) {
|
||||
*failed = true;
|
||||
break;
|
||||
}
|
||||
batch_size += f->data->size;
|
||||
if (batch_size >= chunk_size || i == files->size - 1) {
|
||||
void** items = array_list_to_array(batch);
|
||||
if (!items) {
|
||||
*failed = true;
|
||||
array_list_delete(batch);
|
||||
batch = NULL;
|
||||
break;
|
||||
}
|
||||
Chunk* c = chunk_create((File**)items, batch->size);
|
||||
free(items);
|
||||
if (!c) {
|
||||
*failed = true;
|
||||
array_list_delete(batch);
|
||||
batch = NULL;
|
||||
break;
|
||||
}
|
||||
int batch_start = i - batch->size + 1;
|
||||
for (int j = batch_start; j <= i; j++)
|
||||
files->items[j] = NULL;
|
||||
batch->item_destroyer = NULL;
|
||||
array_list_delete(batch);
|
||||
batch = NULL;
|
||||
if (!first) {
|
||||
first = c;
|
||||
} else {
|
||||
if (!queue_enqueue(queue, c)) {
|
||||
chunk_destroy(c);
|
||||
*failed = true;
|
||||
}
|
||||
}
|
||||
if (i < files->size - 1) {
|
||||
batch = array_list_create(NULL);
|
||||
if (!batch) {
|
||||
*failed = true;
|
||||
break;
|
||||
}
|
||||
batch_size = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (batch) {
|
||||
batch->item_destroyer = NULL;
|
||||
array_list_delete(batch);
|
||||
}
|
||||
return first;
|
||||
}
|
||||
|
||||
/* Record the delete-protection mirror of a root entry that
|
||||
* scanner_inspect_entry() skipped (inspection == 0): a dereferenced symlink
|
||||
* with no referent is a partial-transfer I/O error and a user-selection or size
|
||||
* prune protects the entry's destination mirror. */
|
||||
static void scan_root_record_skipped(const ScannerOptions* options, const char* root_directory,
|
||||
const char* name, const ScannerEntry* inspected,
|
||||
ParallelScanner* ps) {
|
||||
if (inspected->referent_error)
|
||||
ps->io_error = true;
|
||||
ArrayList* sink = NULL;
|
||||
if (inspected->excluded)
|
||||
sink = inspected->size_excluded ? options->size_skipped_paths : options->excluded_paths;
|
||||
if (!sink)
|
||||
return;
|
||||
/* A root-level prune protects the destination mirror of the entry's wire
|
||||
path: under -R + --files-from that is the bare relative name, otherwise it
|
||||
is the full source path with a leading '/' removed (matching the
|
||||
send_path/file_wire_path the scanner hands the sender). */
|
||||
if (options->relative && options->file_list != NULL) {
|
||||
if (!excluded_sink_append(sink, options->excluded_mutex, name))
|
||||
ps->failed = true;
|
||||
} else if (options->relative_prefix) {
|
||||
char* wrel = scanner_prefix_send_path(options->relative_prefix, name);
|
||||
if (!wrel) {
|
||||
ps->failed = true;
|
||||
} else {
|
||||
if (!excluded_sink_append(sink, options->excluded_mutex, wrel))
|
||||
ps->failed = true;
|
||||
free(wrel);
|
||||
}
|
||||
} else {
|
||||
char* abs_path = path_cat(root_directory, name);
|
||||
if (!abs_path) {
|
||||
ps->failed = true;
|
||||
} else {
|
||||
const char* rel = *abs_path == '/' ? abs_path + 1 : abs_path;
|
||||
if (!excluded_sink_append(sink, options->excluded_mutex, rel))
|
||||
ps->failed = true;
|
||||
free(abs_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Record the delete-protection mirror of a root entry dropped by the
|
||||
* --files-from allow-set or a filter rule. Returns false only when the -R
|
||||
* prefix could not be built (the caller must abandon the entry immediately);
|
||||
* other allocation failures mark the scan failed but let the caller continue to
|
||||
* the filter-notice step, matching the historical inlined flow. */
|
||||
static bool scan_root_record_protection(const ScannerOptions* options, const char* rel,
|
||||
const char* name, const char* cur_path, bool protect,
|
||||
bool passes, bool use_rel, ParallelScanner* ps) {
|
||||
if (passes && !protect)
|
||||
return true;
|
||||
/* --files-from subset pruning is not a filter exclusion; -R bare-wire-path
|
||||
exclusions are never recorded (see ScannerOptions.excluded_paths). */
|
||||
bool files_from_prune = options->file_list && !file_list_affects(options->file_list, rel);
|
||||
if ((!files_from_prune && !use_rel) || protect) {
|
||||
const char* rel_path;
|
||||
char* prefixed = NULL;
|
||||
if (use_rel) {
|
||||
/* -R + --files-from: the destination/wire path is the bare relative
|
||||
name, not the source path. */
|
||||
rel_path = rel;
|
||||
} else if (options->relative_prefix) {
|
||||
prefixed = scanner_prefix_send_path(options->relative_prefix, name);
|
||||
if (!prefixed)
|
||||
return false;
|
||||
rel_path = prefixed;
|
||||
} else {
|
||||
rel_path = *cur_path == '/' ? cur_path + 1 : cur_path;
|
||||
}
|
||||
if (options->excluded_paths &&
|
||||
!excluded_sink_append(options->excluded_paths, options->excluded_mutex, rel_path))
|
||||
ps->failed = true;
|
||||
free(prefixed);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Root-level directory node: apply -x/--one-file-system and either emit the
|
||||
* mount-point directory (plain -x) or queue the directory for a worker. */
|
||||
static void scan_root_dir(const ScannerOptions* options, const char* cur_path, const char* rel,
|
||||
const struct stat* st, ArrayList* root_files, ArrayList* subdirs,
|
||||
dev_t root_dev, ParallelScanner* ps) {
|
||||
if (!scanner_same_filesystem(options->one_file_system, root_dev, st->st_dev)) {
|
||||
if (options->one_file_system > 1) {
|
||||
/* -xx: drop the mount-point directory entirely (rsync) and print the
|
||||
--info=mount line when enabled. */
|
||||
scanner_note_mount(options, cur_path);
|
||||
return;
|
||||
}
|
||||
/* -x/--one-file-system: emit the mount-point directory entry (empty) but do
|
||||
not descend into it (see the sequential scanner for the same rule). */
|
||||
File* mount = scanner_build_dir_file(cur_path, st, options);
|
||||
if (!mount) {
|
||||
ps->failed = true;
|
||||
return;
|
||||
}
|
||||
if (options->relative_prefix) {
|
||||
mount->send_path = scanner_prefix_send_path(options->relative_prefix, rel);
|
||||
if (!mount->send_path) {
|
||||
file_destroy(mount);
|
||||
ps->failed = true;
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (!array_list_add(root_files, mount)) {
|
||||
file_destroy(mount);
|
||||
ps->failed = true;
|
||||
}
|
||||
return;
|
||||
}
|
||||
char* dir = str_dup(cur_path);
|
||||
if (!dir || !array_list_add(subdirs, dir)) {
|
||||
free(dir);
|
||||
ps->failed = true;
|
||||
}
|
||||
}
|
||||
|
||||
/* Build a non-directory root entry through the shared construction path and add
|
||||
* it to `root_files`. A non-regular entry the options do not preserve is
|
||||
* dropped by the builder (which prints rsync's nonreg line); an allocation
|
||||
* failure marks the scan failed. */
|
||||
static void scan_root_add_non_dir(const ScannerOptions* options, ScannerEntry* inspected,
|
||||
const char* rel, ArrayList* root_files, ParallelScanner* ps) {
|
||||
File* file = NULL;
|
||||
bool failed = false;
|
||||
ScannerBuildStatus status = scanner_build_file_entry(options, inspected, rel, &file, &failed);
|
||||
if (failed || status == SCANNER_BUILD_FAIL_CONTINUE || status == SCANNER_BUILD_FAIL_BREAK)
|
||||
ps->failed = true;
|
||||
if (status != SCANNER_BUILD_OK)
|
||||
return;
|
||||
if (!array_list_add(root_files, file)) {
|
||||
file_destroy(file);
|
||||
ps->failed = true;
|
||||
}
|
||||
}
|
||||
|
||||
/* Regular file or carried symlink at the transfer root. */
|
||||
static void scan_root_file(const ScannerOptions* options, ScannerEntry* inspected, const char* rel,
|
||||
ArrayList* root_files, ParallelScanner* ps) {
|
||||
scan_root_add_non_dir(options, inspected, rel, root_files, ps);
|
||||
}
|
||||
|
||||
/* Device/FIFO/socket at the transfer root: recreated under --devices/--specials,
|
||||
* otherwise dropped by the shared builder. */
|
||||
static void scan_root_special(const ScannerOptions* options, ScannerEntry* inspected,
|
||||
const char* rel, ArrayList* root_files, ParallelScanner* ps) {
|
||||
scan_root_add_non_dir(options, inspected, rel, root_files, ps);
|
||||
}
|
||||
|
||||
/* Scan one root-directory entry into either the subdirs or files list. */
|
||||
static void scan_root_entry(const ScannerOptions* options, const FilterNode* root_node,
|
||||
const char* root_directory, const struct dirent* entry,
|
||||
ArrayList* root_files, ArrayList* subdirs, dev_t root_dev,
|
||||
ParallelScanner* ps) {
|
||||
ScannerEntry inspected;
|
||||
int inspection =
|
||||
scanner_inspect_entry(options, root_directory, entry->d_name, entry->d_name, &inspected);
|
||||
if (inspection < 0) {
|
||||
ps->failed = true;
|
||||
return;
|
||||
}
|
||||
if (inspection == 0) {
|
||||
scan_root_record_skipped(options, root_directory, entry->d_name, &inspected, ps);
|
||||
return;
|
||||
}
|
||||
char* cur_path = inspected.path;
|
||||
char* rel = str_dup(entry->d_name);
|
||||
if (!rel) {
|
||||
ps->failed = true;
|
||||
goto done;
|
||||
}
|
||||
bool is_dir = inspected.is_directory;
|
||||
bool protect = false;
|
||||
bool passes = entry_passes_selection(options->file_list, options->base_filters, root_node, rel,
|
||||
entry->d_name, is_dir, options->per_dir_filters,
|
||||
options->exclude_per_dir_filter_files, &protect);
|
||||
/* -R + --files-from: root-level files keep their bare relative send path. */
|
||||
bool use_rel = options->relative && options->file_list != NULL;
|
||||
if (!passes || protect) {
|
||||
if (!scan_root_record_protection(options, rel, entry->d_name, cur_path, protect, passes,
|
||||
use_rel, ps)) {
|
||||
ps->failed = true;
|
||||
goto done;
|
||||
}
|
||||
if (!passes) {
|
||||
scanner_note_filter(options, entry->d_name);
|
||||
goto done;
|
||||
}
|
||||
}
|
||||
if (is_dir) {
|
||||
scan_root_dir(options, cur_path, rel, &inspected.stats, root_files, subdirs, root_dev, ps);
|
||||
} else if (S_ISCHR(inspected.stats.st_mode) || S_ISBLK(inspected.stats.st_mode) ||
|
||||
S_ISFIFO(inspected.stats.st_mode) || S_ISSOCK(inspected.stats.st_mode)) {
|
||||
scan_root_special(options, &inspected, rel, root_files, ps);
|
||||
} else {
|
||||
scan_root_file(options, &inspected, rel, root_files, ps);
|
||||
}
|
||||
done:
|
||||
free(rel);
|
||||
free(cur_path);
|
||||
free(inspected.link_target);
|
||||
}
|
||||
|
||||
/* Scan the root directory itself, collecting root files and subdirectories.
|
||||
* Returns false if the root directory could not be opened. */
|
||||
static bool scan_root_directory(ParallelScanner* ps, const char* root_directory,
|
||||
const ScannerOptions* options, const FilterNode* root_node,
|
||||
dev_t root_dev, ArrayList* root_files, ArrayList* subdirs) {
|
||||
DIR* dir = opendir(root_directory);
|
||||
if (!dir) {
|
||||
log_perror("Could not open root directory for parallel scan");
|
||||
return false;
|
||||
}
|
||||
/* The parallel scanner opens the transfer root directly (not through
|
||||
open_next_directory), so record it as synchronized here. */
|
||||
if (!scanner_record_synced_dir(options, root_directory, "",
|
||||
options->relative && options->file_list != NULL)) {
|
||||
closedir(dir);
|
||||
ps->failed = true;
|
||||
return false;
|
||||
}
|
||||
log_debug_message(LOG_DEBUG_FLIST, "flist: scanning %s", root_directory);
|
||||
const struct dirent* entry;
|
||||
while ((entry = readdir(dir)) != NULL) {
|
||||
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
|
||||
continue;
|
||||
scan_root_entry(options, root_node, root_directory, entry, root_files, subdirs, root_dev, ps);
|
||||
}
|
||||
closedir(dir);
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Spawn worker threads, one per group of subdirectories. */
|
||||
static void spawn_parallel_workers(ParallelScanner* ps, ArrayList* subdirs,
|
||||
const ScannerOptions* options, const char* root_directory,
|
||||
unsigned long long cs) {
|
||||
if (subdirs->size <= 0)
|
||||
return;
|
||||
int n = options->num_threads > 0 ? options->num_threads : 4;
|
||||
if (n > subdirs->size)
|
||||
n = subdirs->size;
|
||||
|
||||
ps->num_threads = n;
|
||||
ps->expected_threads = n;
|
||||
ps->threads = calloc(n, sizeof(thrd_t));
|
||||
if (!ps->threads) {
|
||||
ps->num_threads = 0;
|
||||
ps->expected_threads = 0;
|
||||
ps->failed = true;
|
||||
return;
|
||||
}
|
||||
int dirs_per_thread = subdirs->size / n;
|
||||
int remainder = subdirs->size % n;
|
||||
int start = 0;
|
||||
ps->num_threads = 0;
|
||||
for (int t = 0; t < n; t++) {
|
||||
int count = dirs_per_thread + (t < remainder ? 1 : 0);
|
||||
if (count == 0)
|
||||
break;
|
||||
ParallelWorkerArg* wa = calloc(1, sizeof(ParallelWorkerArg));
|
||||
if (!wa) {
|
||||
parallel_scanner_creation_failed(ps);
|
||||
break;
|
||||
}
|
||||
wa->ps = ps;
|
||||
wa->dirs = calloc(count, sizeof(char*));
|
||||
wa->root_dir = str_dup(root_directory);
|
||||
if (!wa->dirs || !wa->root_dir) {
|
||||
free(wa->root_dir);
|
||||
free(wa->dirs);
|
||||
free(wa);
|
||||
parallel_scanner_creation_failed(ps);
|
||||
break;
|
||||
}
|
||||
bool dup_ok = true;
|
||||
for (int j = 0; j < count; j++) {
|
||||
wa->dirs[j] = str_dup((char*)subdirs->items[start + j]);
|
||||
if (!wa->dirs[j])
|
||||
dup_ok = false;
|
||||
}
|
||||
if (!dup_ok) {
|
||||
for (int j = 0; j < count; j++)
|
||||
free(wa->dirs[j]);
|
||||
free(wa->root_dir);
|
||||
free(wa->dirs);
|
||||
free(wa);
|
||||
parallel_scanner_creation_failed(ps);
|
||||
break;
|
||||
}
|
||||
wa->dir_count = count;
|
||||
wa->options = *options;
|
||||
wa->options.chunk_size = cs;
|
||||
wa->allocation_session = ps->allocation_session;
|
||||
start += count;
|
||||
if (thrd_create(&ps->threads[t], parallel_worker_thread, wa) != thrd_success) {
|
||||
for (int j = 0; j < count; j++)
|
||||
free(wa->dirs[j]);
|
||||
free(wa->root_dir);
|
||||
free(wa->dirs);
|
||||
free(wa);
|
||||
parallel_scanner_creation_failed(ps);
|
||||
break;
|
||||
}
|
||||
ps->num_threads++;
|
||||
ps->created_threads++;
|
||||
}
|
||||
}
|
||||
|
||||
ParallelScanner* parallel_scanner_create_with_options(const char* root_directory,
|
||||
const ScannerOptions* options,
|
||||
ProtocolSession* allocation_session) {
|
||||
if (!root_directory || !options)
|
||||
return NULL;
|
||||
ParallelScanner* ps = calloc(1, sizeof(ParallelScanner));
|
||||
if (!ps)
|
||||
return NULL;
|
||||
if (!parallel_scanner_init(ps)) {
|
||||
free(ps);
|
||||
return NULL;
|
||||
}
|
||||
ps->allocation_session = allocation_session;
|
||||
ps->options = options;
|
||||
|
||||
ArrayList* root_files = array_list_create(file_destroy);
|
||||
ArrayList* subdirs = array_list_create(free);
|
||||
if (!root_files || !subdirs) {
|
||||
array_list_delete(root_files);
|
||||
array_list_delete(subdirs);
|
||||
parallel_scanner_destroy(ps);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
dev_t root_dev = 0;
|
||||
if (options->one_file_system) {
|
||||
struct stat root_stats;
|
||||
if (stat(root_directory, &root_stats) != 0) {
|
||||
log_perror("Could not stat source directory");
|
||||
array_list_delete(root_files);
|
||||
array_list_delete(subdirs);
|
||||
parallel_scanner_destroy(ps);
|
||||
return NULL;
|
||||
}
|
||||
root_dev = root_stats.st_dev;
|
||||
}
|
||||
|
||||
/* Build the root directory's per-directory filter context once; workers seed
|
||||
* their scanners with it so per-dir rules behave identically to the sequential
|
||||
* scanner. */
|
||||
FilterNode* root_node = NULL;
|
||||
{
|
||||
char err[256];
|
||||
bool any_exists = false;
|
||||
FilterRuleList* own = read_dir_filters(options, root_directory, "",
|
||||
options->relative && options->file_list != NULL,
|
||||
&any_exists, err, sizeof(err));
|
||||
if (!own) {
|
||||
/* A parse/allocation failure must fail the scan even when an earlier
|
||||
merge file in the same directory existed (see the sequential scanner). */
|
||||
if (err[0] != '\0') {
|
||||
log_message(LOG_LEVEL_ERROR, "invalid per-directory filter in %s: %s", root_directory, err);
|
||||
array_list_delete(root_files);
|
||||
array_list_delete(subdirs);
|
||||
parallel_scanner_destroy(ps);
|
||||
return NULL;
|
||||
}
|
||||
/* no files exist: leave root_node NULL */
|
||||
} else if (any_exists && (own->count > 0 || own->dir_merge_count > 0)) {
|
||||
root_node = filter_node_alloc(NULL, own);
|
||||
if (!root_node) {
|
||||
filter_rule_list_free(own);
|
||||
array_list_delete(root_files);
|
||||
array_list_delete(subdirs);
|
||||
parallel_scanner_destroy(ps);
|
||||
return NULL;
|
||||
}
|
||||
} else {
|
||||
filter_rule_list_free(own);
|
||||
}
|
||||
}
|
||||
ps->root_filter_node = root_node;
|
||||
|
||||
if (!scan_root_directory(ps, root_directory, options, root_node, root_dev, root_files, subdirs)) {
|
||||
array_list_delete(root_files);
|
||||
array_list_delete(subdirs);
|
||||
parallel_scanner_destroy(ps);
|
||||
return NULL;
|
||||
}
|
||||
/* The root itself is a traversed directory (rsync counts it in
|
||||
`Number of files`); the worker DirectoryScanners account for every
|
||||
subdirectory below it. */
|
||||
scanner_dir_count_count(options);
|
||||
/* P7 Wave D: the parallel scanner never runs a DirectoryScanner over the
|
||||
transfer root itself (it hands the root's immediate subdirectories to
|
||||
workers), so capture the root's directory time here. */
|
||||
if (options->capture_dir_times &&
|
||||
!scanner_capture_dir_time(
|
||||
options->dir_entries, options->dir_entries_mutex, root_directory, root_directory,
|
||||
options->relative && options->file_list != NULL, options->relative_prefix,
|
||||
options->preserve_atimes, options->preserve_crtimes, options->preserve_xattrs,
|
||||
options->preserve_acls, options->no_implied_dirs, options->file_list)) {
|
||||
array_list_delete(root_files);
|
||||
array_list_delete(subdirs);
|
||||
parallel_scanner_destroy(ps);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
unsigned long long cs = options->chunk_size > 0 ? options->chunk_size : DESIRED_CHUNK_SIZE;
|
||||
ps->initial_chunk = batch_files(root_files, cs, ps->result_queue, &ps->failed);
|
||||
array_list_delete(root_files);
|
||||
|
||||
spawn_parallel_workers(ps, subdirs, options, root_directory, cs);
|
||||
array_list_delete(subdirs);
|
||||
return ps;
|
||||
}
|
||||
|
||||
Chunk* parallel_scanner_next(ParallelScanner* ps) {
|
||||
if (ps->initial_chunk) {
|
||||
Chunk* c = ps->initial_chunk;
|
||||
ps->initial_chunk = NULL;
|
||||
return c;
|
||||
}
|
||||
if (ps->num_threads == 0) {
|
||||
mtx_lock(&ps->result_mutex);
|
||||
if (!queue_is_empty(ps->result_queue)) {
|
||||
Chunk* chunk = queue_dequeue(ps->result_queue);
|
||||
mtx_unlock(&ps->result_mutex);
|
||||
return chunk;
|
||||
}
|
||||
ps->done = true;
|
||||
mtx_unlock(&ps->result_mutex);
|
||||
return NULL;
|
||||
}
|
||||
Chunk* chunk = queue_dequeue_multithreaded(
|
||||
ps->result_queue, &ps->result_mutex, &ps->result_not_empty, &ps->result_not_full, &ps->done);
|
||||
return chunk;
|
||||
}
|
||||
|
||||
bool parallel_scanner_failed(const ParallelScanner* ps) {
|
||||
return ps == NULL || ps->failed;
|
||||
}
|
||||
|
||||
bool parallel_scanner_had_io_error(const ParallelScanner* ps) {
|
||||
return ps != NULL && ps->io_error;
|
||||
}
|
||||
|
||||
void parallel_scanner_destroy(ParallelScanner* ps) {
|
||||
if (!ps)
|
||||
return;
|
||||
mtx_lock(&ps->result_mutex);
|
||||
ps->done = true;
|
||||
atomic_store(&ps->cancelled, true);
|
||||
cnd_broadcast(&ps->result_not_empty);
|
||||
cnd_broadcast(&ps->result_not_full);
|
||||
mtx_unlock(&ps->result_mutex);
|
||||
for (int i = 0; i < ps->num_threads; i++)
|
||||
thrd_join(ps->threads[i], NULL);
|
||||
free(ps->threads);
|
||||
if (ps->root_filter_node)
|
||||
filter_node_destroy(ps->root_filter_node);
|
||||
if (ps->initial_chunk)
|
||||
chunk_destroy(ps->initial_chunk);
|
||||
queue_destroy(ps->result_queue);
|
||||
mtx_destroy(&ps->result_mutex);
|
||||
cnd_destroy(&ps->result_not_empty);
|
||||
cnd_destroy(&ps->result_not_full);
|
||||
free(ps);
|
||||
}
|
||||
+26
-52
@@ -19,16 +19,11 @@ void print_usage(void) {
|
||||
printf("\n");
|
||||
printf("Options:\n");
|
||||
printf(" -c, --checksum Verify content by checksum instead of size+mtime\n");
|
||||
printf(" -z, --compress [level] Enable compression. The default level is\n");
|
||||
printf(" per-codec: zstd 3 (range 1-22), zlib/zlibx 6, lz4\n");
|
||||
printf(" ignores the level\n");
|
||||
printf(" -z, --compress [level] Enable compression (level 1-22, default 5)\n");
|
||||
printf(" -a, --archive rsync archive mode (-rlptgoD): links, perms, times,\n");
|
||||
printf(" owner, group, devices and specials; not\n");
|
||||
printf(" compression/multithreading\n");
|
||||
printf(" -r, --recursive Recurse into directories (FastSync is always recursive)\n");
|
||||
printf(" --inc-recursive Accepted for rsync CLI compatibility; no effect (FastSync\n");
|
||||
printf(" always performs a full scan, so the destination is identical)\n");
|
||||
printf(" --no-inc-recursive Accepted for rsync CLI compatibility; no effect\n");
|
||||
printf(" -n, --dry-run Show what would be transferred\n");
|
||||
printf(" --remove-source-files Remove regular source files after successful transfer\n");
|
||||
printf(" -p, --perms Preserve permission bits\n");
|
||||
@@ -41,8 +36,8 @@ void print_usage(void) {
|
||||
printf(" arguments, e.g. -e \"ssh -p 2222\"\n");
|
||||
printf(" --rsync-path <path> Alias for --fastsync-server-path (path to the\n");
|
||||
printf(" fastsync server binary on the remote side)\n");
|
||||
printf(" --blocking-io SSH transport only: leave the socket without read/write\n");
|
||||
printf(" timeouts so it blocks naturally (no effect on TCP)\n");
|
||||
printf(" --blocking-io Leave the SSH transport socket without read/write\n");
|
||||
printf(" timeouts so it blocks naturally\n");
|
||||
printf(" --outbuf=MODE stdout/stderr buffering: N (none/unbuffered),\n");
|
||||
printf(" L (line-buffered), or B (block-buffered, default)\n");
|
||||
printf(" --progress Show transfer progress\n");
|
||||
@@ -54,7 +49,6 @@ void print_usage(void) {
|
||||
printf(" converted before transmission and back on receipt; a\n");
|
||||
printf(" name that cannot be represented in the target charset\n");
|
||||
printf(" fails that transfer cleanly (rsync-compatible)\n");
|
||||
printf(" --no-iconv Disable --iconv charset conversion (same as --iconv=-)\n");
|
||||
printf(" --protocol=NUM Force the wire protocol version (must equal the current\n");
|
||||
printf(" PROTOCOL_VERSION; FastSync cannot speak older/virtual\n");
|
||||
printf(" wire formats)\n");
|
||||
@@ -68,7 +62,8 @@ void print_usage(void) {
|
||||
printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n");
|
||||
printf(" files (different container format); do not mix the two tools.\n");
|
||||
printf(" --delete Delete files on receiver not in source\n");
|
||||
printf(" (default timing: delete-during, like rsync --del)\n");
|
||||
printf(" (default timing: delete only after the whole\n");
|
||||
printf(" transfer has succeeded)\n");
|
||||
printf(" --delete-before Delete extras before the transfer starts\n");
|
||||
printf(" (implies --delete)\n");
|
||||
printf(" --delete-during Delete a directory's extras as that directory is\n");
|
||||
@@ -77,9 +72,7 @@ void print_usage(void) {
|
||||
printf(" --delete-delay Record the extras during the scan but remove them\n");
|
||||
printf(" only after a successful transfer (implies --delete)\n");
|
||||
printf(" --delete-after Delete only after the whole transfer succeeded\n");
|
||||
printf(" (implies --delete)\n");
|
||||
printf(" --delete-commit FastSync-only: restore the late whole-tree commit\n");
|
||||
printf(" (identical to --delete-after; implies --delete)\n");
|
||||
printf(" (the default --delete timing; implies --delete)\n");
|
||||
printf(" --delete-excluded Also delete destination files that were excluded on\n");
|
||||
printf(" the source (default protects them, matching rsync)\n");
|
||||
printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n");
|
||||
@@ -97,11 +90,10 @@ void print_usage(void) {
|
||||
printf(" entry's destination mirror receiver-side. Independent of\n");
|
||||
printf(" --delete (it does not imply --delete; a non-empty directory\n");
|
||||
printf(" mirror is removed only with --force or --delete)\n");
|
||||
printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n");
|
||||
printf(" otherwise recreates them, like rsync)\n");
|
||||
printf(" -m, --prune-empty-dirs Do not transfer empty directory entries (--dirs mode);\n");
|
||||
printf(" recursive transfers never send empty dirs\n");
|
||||
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
|
||||
printf(" --no-delete (in either order) is rejected as a config error, as is more\n");
|
||||
printf(" than one timing flag.\n");
|
||||
printf(" --no-delete (in either order) is rejected as a config error.\n");
|
||||
printf(" --ignore-existing Skip files that already exist on receiver\n");
|
||||
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
|
||||
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
|
||||
@@ -111,9 +103,8 @@ void print_usage(void) {
|
||||
printf(" -R, --relative With --files-from, preserve each listed entry's relative path\n");
|
||||
printf(" below the destination root instead of mirroring the full\n");
|
||||
printf(" source path (no effect without --files-from)\n");
|
||||
printf(" --no-implied-dirs With -R, do not apply the source metadata of a listed file's\n");
|
||||
printf(" implied parent directories (they are still created with\n");
|
||||
printf(" default attributes)\n");
|
||||
printf(" --no-implied-dirs With -R --files-from, refuse to place a listed file whose\n");
|
||||
printf(" parent directory is not itself listed\n");
|
||||
printf(" --mkpath Create the destination root directory on the server when it\n");
|
||||
printf(" does not exist yet\n");
|
||||
printf(" --exclude <pattern>, --exclude=<pattern> Exclude files matching pattern\n");
|
||||
@@ -146,9 +137,6 @@ void print_usage(void) {
|
||||
printf(" into the destination instead of transferring its data\n");
|
||||
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
|
||||
printf(" into the destination (repeatable; earlier DIRs win)\n");
|
||||
printf(" --verify-basis FastSync-only: require a basis hit's content to match the\n");
|
||||
printf(" source by whole-file digest instead of trusting rsync's\n");
|
||||
printf(" size+mtime (or --size-only) quick-check\n");
|
||||
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
|
||||
printf(" --checksum compares. Accepted: xxh128 (default), xxh3, xxh64\n");
|
||||
printf(" (aka xxhash), md5, md4, sha1, or none. A two-name\n");
|
||||
@@ -168,7 +156,7 @@ void print_usage(void) {
|
||||
printf(" --no-delta, or --no-incremental)\n");
|
||||
printf(" --no-fuzzy Disable --fuzzy\n");
|
||||
printf(" -B <n>, --block-size <n>, --delta-block <n>\n");
|
||||
printf(" Delta block size in bytes (default: %u)\n", DELTA_BLOCK_SIZE_DEFAULT);
|
||||
printf(" Delta block size in bytes (default: %d)\n", DELTA_BLOCK_SIZE_DEFAULT);
|
||||
printf(" --delta-max <n> Max file size for delta transfer (default: %llu)\n",
|
||||
DELTA_MAX_FILE_SIZE);
|
||||
printf(" -j, --threads[=N] Enable the multithreaded scanner/loader/sender\n");
|
||||
@@ -198,21 +186,16 @@ void print_usage(void) {
|
||||
printf(" -U, --atimes Preserve access times\n");
|
||||
printf(" -N, --crtimes Capture birth time; cannot be applied (documented\n");
|
||||
printf(" divergence)\n");
|
||||
printf(" -O, --omit-dir-times Do not apply modification times to directories\n");
|
||||
printf(" -J, --omit-link-times Do not apply times to symlinks\n");
|
||||
printf(" --open-noatime Open source files with O_NOATIME so reading for a\n");
|
||||
printf(" transfer does not update their access time\n");
|
||||
printf(" -X, --xattrs Preserve user extended attributes (user.* only;\n");
|
||||
printf(" privileged security.*/trusted.* namespaces are\n");
|
||||
printf(" never captured or applied)\n");
|
||||
printf(" -A, --acls Preserve POSIX ACLs (the system.posix_acl_* xattrs;\n");
|
||||
printf(" setting an ACL the receiver is not permitted to\n");
|
||||
printf(" set is warned and skipped, never fatal)\n");
|
||||
printf(" --fake-super Store the source mode/rdev/uid/gid in rsync's\n");
|
||||
printf(" reserved user.rsync.%%stat xattr on each written\n");
|
||||
printf(" file (interoperable with rsync); it never performs a\n");
|
||||
printf(" real chown, so an unprivileged receiver records the\n");
|
||||
printf(" privileged stat for a later restore\n");
|
||||
printf(" --fake-super Store the source uid/gid/mode/mtime in a reserved\n");
|
||||
printf(" user.fastsync.stat xattr on each written file and\n");
|
||||
printf(" re-apply it (fd-relative) on a privileged run; the\n");
|
||||
printf(" recording format diverges from rsync's user.rsync.%%stat%%\n");
|
||||
printf(" --super Permit the receiver to attempt super-user activities\n");
|
||||
printf(" (char/block device-node creation, --write-devices)\n");
|
||||
printf(" within the confined receive root. Never elevates\n");
|
||||
@@ -260,9 +243,7 @@ void print_usage(void) {
|
||||
printf(" reusable digest is sent (keep the file mode 0600)\n");
|
||||
printf(" --no-motd Suppress display of the daemon's MOTD (the server\n");
|
||||
printf(" still sends it; the client just does not show it)\n");
|
||||
printf(" --bwlimit=RATE Limit socket I/O bandwidth (default unit KiB/s,\n");
|
||||
printf(" rsync-style: 0 = no limit; K/M/G/T/P suffixes are\n");
|
||||
printf(" binary, KB/MB decimal, KiB/MiB binary; decimals allowed)\n");
|
||||
printf(" --bwlimit <KB/s> Bandwidth limit in kilobytes per second\n");
|
||||
printf(" --tls Enable TLS encryption\n");
|
||||
printf(" --cert <path> TLS certificate file (PEM)\n");
|
||||
printf(" --key <path> TLS private key file (PEM)\n");
|
||||
@@ -297,15 +278,9 @@ void print_usage(void) {
|
||||
printf(" --max-depth <n> Maximum directory depth (0=unlimited)\n");
|
||||
printf(" -x, --one-file-system Do not cross filesystem boundaries\n");
|
||||
printf(" --log-file <path>, --log-file=<path> Write log messages to file\n");
|
||||
printf(" --stderr=MODE Route logging: errors (default), all, or client\n");
|
||||
printf(" (forward the client's diagnostics to the server's\n");
|
||||
printf(" stderr)\n");
|
||||
printf(" --msgs2stderr Route all messages to stderr (deprecated spelling of\n");
|
||||
printf(" --stderr=all)\n");
|
||||
printf(" --no-msgs2stderr Forward the client's diagnostics to the server\n");
|
||||
printf(" (deprecated spelling of --stderr=client)\n");
|
||||
printf(" --stderr=MODE Route logging to stderr: errors or all\n");
|
||||
printf(" --partial Keep partial files on interrupted transfer\n");
|
||||
printf(" --partial-dir <dir> Directory for partial files (implies --partial)\n");
|
||||
printf(" --partial-dir <dir> Directory for partial files\n");
|
||||
printf(" -T, --temp-dir <dir> Scratch dir for temp files before atomic install.\n");
|
||||
printf(" Confined to the receive root: a relative dir resolves below\n");
|
||||
printf(" it and an absolute/traversal dir is rejected. The dir must\n");
|
||||
@@ -354,8 +329,7 @@ void print_usage(void) {
|
||||
printf(" --append-verify Like --append, but verifies the retained prefix checksum\n");
|
||||
printf(" before appending (falls back to a full transfer on mismatch)\n");
|
||||
printf(" --fsync Fsync every written file before publication\n");
|
||||
printf(" --compress-level <n> Compression level (per-codec default: zstd 3,\n");
|
||||
printf(" zlib/zlibx 6, lz4 ignores it)\n");
|
||||
printf(" --compress-level <n> Compression level (default: 5)\n");
|
||||
printf(" --zl <n> Alias for --compress-level\n");
|
||||
printf(" --skip-compress=LIST Skip compression for suffixes in LIST (separated by\n");
|
||||
printf(" '/' as in rsync, or ','); a leading dot is optional. The\n");
|
||||
@@ -367,19 +341,19 @@ void print_usage(void) {
|
||||
}
|
||||
|
||||
void print_debug_usage(void) {
|
||||
printf("Emitting debug flags: IO,PROTO,PACK,UTIL,FLIST,DEL,HASH,DELTASUM,\n");
|
||||
printf("RECV,FILTER,SEND,ALL,NONE\n");
|
||||
printf("Emitting debug flags: IO,PROTO,PACK,UTIL,ALL,NONE\n");
|
||||
printf("Also accepted for rsync CLI parity (silent): ACL,BACKUP,BIND,CHDIR,\n");
|
||||
printf("CONNECT,CMD,DUP,EXIT,FUZZY,GENR,HLINK,ICONV,NSTR,OWN,TIME.\n");
|
||||
printf("CONNECT,CMD,DEL,DELTASUM,DUP,EXIT,FILTER,FLIST,FUZZY,GENR,HASH,HLINK,\n");
|
||||
printf("ICONV,NSTR,OWN,RECV,SEND,TIME.\n");
|
||||
printf("Flags may be comma-separated, for example: --debug=io,proto\n");
|
||||
printf("An optional level suffix is accepted (e.g. --debug=io2); level 0\n");
|
||||
printf("silences that item. Unknown names are rejected.\n");
|
||||
}
|
||||
|
||||
void print_info_usage(void) {
|
||||
printf("Emitting info flags: COPY,MISC,SKIP,STATS,DEL,REMOVE,NAME,FLIST,\n");
|
||||
printf("NONREG,PROGRESS,MOUNT,ALL,NONE\n");
|
||||
printf("Also accepted for rsync CLI parity (silent): BACKUP,SYMS,SYMSAFE.\n");
|
||||
printf("Emitting info flags: COPY,NAME,MISC,SKIP,STATS,ALL,NONE\n");
|
||||
printf("Also accepted for rsync CLI parity (silent): BACKUP,DEL,FLIST,MOUNT,\n");
|
||||
printf("NONREG,PROGRESS,REMOVE,SYMSAFE.\n");
|
||||
printf("Flags may be comma-separated, for example: --info=name,stats\n");
|
||||
printf("An optional level suffix is accepted (e.g. --info=stats2); level 0\n");
|
||||
printf("silences that item. Unknown names are rejected.\n");
|
||||
|
||||
+221
-593
@@ -11,13 +11,10 @@
|
||||
#include "metadata.h"
|
||||
#include "protocol.h"
|
||||
#include "utils.h"
|
||||
#include <fcntl.h>
|
||||
#include <limits.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code) {
|
||||
if (!outcomes)
|
||||
@@ -56,13 +53,7 @@ bool receiver_send_final_success(int fd, const Config* config, const ReceiverOut
|
||||
return send_status(fd, final_status);
|
||||
size_t count = outcomes ? outcomes->count : 0;
|
||||
for (size_t i = 0; i < count; i++) {
|
||||
Status per_file;
|
||||
if (outcomes->entries[i] == FILE_SAVE_WRITTEN)
|
||||
per_file = STATUS_NEXT;
|
||||
else if (outcomes->entries[i] == FILE_SAVE_FAILED)
|
||||
per_file = STATUS_ERROR;
|
||||
else
|
||||
per_file = STATUS_OK;
|
||||
Status per_file = outcomes->entries[i] == FILE_SAVE_WRITTEN ? STATUS_NEXT : STATUS_OK;
|
||||
if (!send_status(fd, per_file))
|
||||
return false;
|
||||
}
|
||||
@@ -70,18 +61,13 @@ bool receiver_send_final_success(int fd, const Config* config, const ReceiverOut
|
||||
}
|
||||
|
||||
bool receiver_send_stats_frame(int fd, const Config* config, const ReceiverStats* stats,
|
||||
const struct ArrayList* would_delete,
|
||||
const struct ArrayList* deleted_paths) {
|
||||
const struct ArrayList* would_delete) {
|
||||
if (!config->report_stats)
|
||||
return true;
|
||||
ReceiverStats local;
|
||||
memset(&local, 0, sizeof(local));
|
||||
const ReceiverStats* out = stats ? stats : &local;
|
||||
/* The path list carries the dry-run would-delete set for a -n run and the
|
||||
actually-removed set for a real --info=del run. */
|
||||
const struct ArrayList* paths =
|
||||
config->dry_run ? would_delete : (config->report_deletes ? deleted_paths : NULL);
|
||||
size_t count = paths ? (size_t)paths->size : 0;
|
||||
size_t count = would_delete ? (size_t)would_delete->size : 0;
|
||||
if (count > (size_t)MAX_MANIFEST_ENTRIES)
|
||||
count = MAX_MANIFEST_ENTRIES;
|
||||
ReceiverStats record = *out;
|
||||
@@ -90,7 +76,7 @@ bool receiver_send_stats_frame(int fd, const Config* config, const ReceiverStats
|
||||
!send_int(fd, (int)count))
|
||||
return false;
|
||||
for (size_t i = 0; i < count; i++) {
|
||||
const char* path = (const char*)paths->items[i];
|
||||
const char* path = (const char*)would_delete->items[i];
|
||||
if (!send_wire_str(fd, path ? path : ""))
|
||||
return false;
|
||||
}
|
||||
@@ -104,56 +90,6 @@ static void receiver_tally_deleted(const ReceiverSink* sink, size_t deleted) {
|
||||
sink->stats->deleted_files += deleted;
|
||||
}
|
||||
|
||||
/* Observer for --info=del/--stats: record each truly-removed destination-
|
||||
relative path (when the context carries a path list) and tally it by type
|
||||
(when it carries a stats record), so the terminal STATUS_STATS frame can list
|
||||
the paths and render rsync's per-type `Number of deleted files` breakdown. A
|
||||
failed append is best-effort (the deletion already happened; output is
|
||||
cosmetic). Shared by the single-threaded receiver and the -m pipeline's
|
||||
deferred commit. */
|
||||
void receiver_record_deleted_path(void* context, const char* rel_path, DeleteEntryType type) {
|
||||
ReceiverDeleteContext* del = context;
|
||||
if (!del || !rel_path)
|
||||
return;
|
||||
if (del->stats) {
|
||||
switch (type) {
|
||||
case DELETE_ENTRY_DIR:
|
||||
del->stats->deleted_dir++;
|
||||
break;
|
||||
case DELETE_ENTRY_LINK:
|
||||
del->stats->deleted_link++;
|
||||
break;
|
||||
case DELETE_ENTRY_SPECIAL:
|
||||
del->stats->deleted_special++;
|
||||
break;
|
||||
default:
|
||||
del->stats->deleted_reg++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
ArrayList* paths = del->deleted_paths;
|
||||
if (!paths)
|
||||
return;
|
||||
/* Bound the retained list like the keep-set manifest: only MAX_MANIFEST_ENTRIES
|
||||
paths are ever transmitted in the terminal STATUS_STATS frame, so recording
|
||||
more only grows memory. A hostile/huge deletion set is therefore capped. */
|
||||
if ((size_t)paths->size >= (size_t)MAX_MANIFEST_ENTRIES)
|
||||
return;
|
||||
char* copy = str_dup(rel_path);
|
||||
if (copy && !array_list_add(paths, copy))
|
||||
free(copy);
|
||||
}
|
||||
|
||||
/* Install the delete observer (and its context) for one commit when the sink
|
||||
carries a stats record or a path list. Returns NULL when neither is needed,
|
||||
so the delete engines skip the observer entirely. */
|
||||
static DeletePathObserver receiver_delete_observer(const ReceiverSink* sink,
|
||||
ReceiverDeleteContext* del) {
|
||||
del->stats = sink ? sink->stats : NULL;
|
||||
del->deleted_paths = sink ? sink->deleted_paths : NULL;
|
||||
return (del->stats || del->deleted_paths) ? receiver_record_deleted_path : NULL;
|
||||
}
|
||||
|
||||
static bool receiver_process_chunk(Chunk* chunk, const ReceiverSink* sink) {
|
||||
if (!chunk || !sink || !sink->store_file)
|
||||
return false;
|
||||
@@ -311,7 +247,6 @@ static bool status_counts_as_progress(Status status) {
|
||||
case STATUS_ABORT:
|
||||
case STATUS_CHECK_BATCH:
|
||||
case STATUS_DIR_TIMES:
|
||||
case STATUS_CLIENT_MSG:
|
||||
return false;
|
||||
default:
|
||||
return true;
|
||||
@@ -341,410 +276,21 @@ static bool receiver_note_status(const struct timespec* session_start,
|
||||
}
|
||||
|
||||
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink) {
|
||||
return receiver_process_pending_ctx(config, file_descriptor, sink, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
|
||||
return receiver_process_pending_ctx(config, file_descriptor, sink, pending_manifest,
|
||||
pending_plans, NULL);
|
||||
}
|
||||
|
||||
/* Per-connection state threaded through the status handlers below. The parked
|
||||
keep-set / per-directory session live here so one teardown helper can release
|
||||
them on every exit path. */
|
||||
typedef struct {
|
||||
Config* config;
|
||||
int fd;
|
||||
const ReceiverSink* sink;
|
||||
DeleteManifest** pending_manifest;
|
||||
DeletePlanSession** pending_plans;
|
||||
/* Parked keep-set for the late/commit timing. Every exit path frees it
|
||||
exactly once; the only exception is the successful FINISHED handoff, which
|
||||
transfers ownership to *pending_manifest (used by the -m receiver). */
|
||||
DeleteManifest* deferred_manifest;
|
||||
/* Per-directory delete session for --delete-during/--delete-delay. During the
|
||||
loop it applies plans inline (during) or snapshots their extras (delay); on
|
||||
a successful FINISHED it is either committed here or handed to
|
||||
*pending_plans so the -m caller commits after its disk writer drained. */
|
||||
DeletePlanSession* plan_session;
|
||||
/* Observer context for the per-directory delete session, which outlives the
|
||||
frame handler; must stay alive until the session commits. For a session
|
||||
handed to the caller (pending_plans) this points at a caller-owned
|
||||
long-lived context; otherwise it points at an internal stack context. */
|
||||
ReceiverDeleteContext* delete_ctx;
|
||||
bool early_delete;
|
||||
bool per_dir_delete;
|
||||
bool delete_limit_noted;
|
||||
} ReceiverPendingState;
|
||||
|
||||
/* Outcome of one frame handler. NEXT reads the following status frame; FAIL
|
||||
tears the connection down without a peer STATUS_ERROR; ERROR tears it down
|
||||
and (when the sink owns error reporting) emits STATUS_ERROR. */
|
||||
typedef enum {
|
||||
RECEIVER_STEP_NEXT,
|
||||
RECEIVER_STEP_FAIL,
|
||||
RECEIVER_STEP_ERROR,
|
||||
} ReceiverStep;
|
||||
|
||||
static ReceiverStep receiver_handle_keepalive(ReceiverPendingState* state) {
|
||||
if (!send_status(state->fd, STATUS_KEEPALIVE))
|
||||
return RECEIVER_STEP_FAIL;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
/* rsync --stderr=client: a client diagnostic forwarded over the wire. Read the
|
||||
* bounded string and log it through the normal destination/level gate. The
|
||||
* body is peer-controlled text: log_client_message() escapes every
|
||||
* non-printable byte (newlines, CR, ANSI ESC, ...) before writing, so a hostile
|
||||
* client cannot forge log lines or inject terminal control sequences. A
|
||||
* malformed string (over-long or embedded NUL) is a framing error and tears the
|
||||
* connection down. */
|
||||
static ReceiverStep receiver_handle_client_msg(ReceiverPendingState* state) {
|
||||
char* message = receive_str(state->fd);
|
||||
if (!message)
|
||||
return RECEIVER_STEP_FAIL;
|
||||
if (message[0] != '\0')
|
||||
log_client_message(message);
|
||||
free(message);
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_abort(ReceiverPendingState* state) {
|
||||
(void)state;
|
||||
log_message(LOG_LEVEL_INFO, "Received abort from client, cleaning up");
|
||||
return RECEIVER_STEP_FAIL;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_check(ReceiverPendingState* state) {
|
||||
bool skipped = false;
|
||||
bool would_transfer = false;
|
||||
File* file = receive_incremental_check_ex(state->fd, state->config, &skipped, &would_transfer);
|
||||
if (state->config->dry_run) {
|
||||
/* Server-contacting --dry-run: the reply has already been sent
|
||||
(STATUS_OK = up to date, STATUS_DRY_RUN_TRANSFER = would transfer) and
|
||||
nothing may be stored. Both flags false means a genuine protocol
|
||||
error (STATUS_ERROR already sent or sent by receive_error below). */
|
||||
if (!skipped && !would_transfer)
|
||||
return RECEIVER_STEP_ERROR;
|
||||
} else if (!skipped && (!file || !state->sink->store_file(file, state->sink->context))) {
|
||||
return RECEIVER_STEP_ERROR;
|
||||
}
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_chunk(ReceiverPendingState* state) {
|
||||
Chunk* chunk = receive_chunk_data(state->fd, state->config);
|
||||
if (!chunk || !receiver_process_chunk(chunk, state->sink))
|
||||
return RECEIVER_STEP_ERROR;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_check_batch(ReceiverPendingState* state) {
|
||||
if (!receiver_process_batch(state->config, state->fd))
|
||||
return RECEIVER_STEP_FAIL;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
/* Probe a destination entry's pre-transfer state for the output-parity
|
||||
dest-info report (protocol 2.30.0). `wire_path` is the destination-relative
|
||||
path; `incoming_target` is non-NULL only for a symlink probe, in which case
|
||||
the on-disk link target is compared with the target the receiver is about to
|
||||
store (after --munge-links). The final component is never followed and the
|
||||
parent walk is confined below the receive root. Returns false only on an
|
||||
allocation/secure-walk failure; a missing entry is reported as existed=false. */
|
||||
static bool receiver_probe_dest_state(const Config* config, const char* wire_path,
|
||||
const char* incoming_target, OutputDestState* out) {
|
||||
memset(out, 0, sizeof(*out));
|
||||
out->known = true;
|
||||
if (!config || !wire_path || wire_path[0] == '\0')
|
||||
return false;
|
||||
char* full = path_cat(config->receive_root_directory, wire_path);
|
||||
if (!full)
|
||||
return false;
|
||||
char* leaf = NULL;
|
||||
int parent_fd = file_open_secure_parent(full, &leaf, false);
|
||||
free(full);
|
||||
if (parent_fd < 0) {
|
||||
/* A missing/unreachable parent means the entry cannot exist yet. */
|
||||
free(leaf);
|
||||
return true;
|
||||
}
|
||||
struct stat st;
|
||||
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) == 0) {
|
||||
out->existed = true;
|
||||
out->size = (unsigned long long)st.st_size;
|
||||
out->mtime_sec = (long long)st.st_mtime;
|
||||
#ifdef __linux__
|
||||
out->mtime_nsec = st.st_mtim.tv_nsec;
|
||||
#endif
|
||||
out->mode = (uint32_t)st.st_mode;
|
||||
out->uid = (int32_t)st.st_uid;
|
||||
out->gid = (int32_t)st.st_gid;
|
||||
if (incoming_target && S_ISLNK(st.st_mode)) {
|
||||
char target_buf[PATH_MAX];
|
||||
ssize_t n = readlinkat(parent_fd, leaf, target_buf, sizeof(target_buf) - 1);
|
||||
if (n >= 0) {
|
||||
target_buf[n] = '\0';
|
||||
char* expected =
|
||||
config->munge_links ? file_symlink_munge(incoming_target) : str_dup(incoming_target);
|
||||
if (expected) {
|
||||
out->target_matches = strcmp(target_buf, expected) == 0;
|
||||
free(expected);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return true;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_mkdir(ReceiverPendingState* state) {
|
||||
const Config* config = state->config;
|
||||
int fd = state->fd;
|
||||
if (config->report_dest_info) {
|
||||
int probe = 0;
|
||||
if (!receive_int(fd, &probe) || (probe != 0 && probe != 1))
|
||||
return RECEIVER_STEP_FAIL;
|
||||
if (probe) {
|
||||
/* Probe-only frame: report the destination state and create nothing. */
|
||||
char* path = receive_wire_str(fd);
|
||||
if (!path || path[0] == '\0' || (!file_get_trust_sender() && has_path_traversal(path))) {
|
||||
free(path);
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return RECEIVER_STEP_FAIL;
|
||||
}
|
||||
OutputDestState info;
|
||||
bool ok = receiver_probe_dest_state(config, path, NULL, &info);
|
||||
free(path);
|
||||
if (!ok || !send_status(fd, STATUS_DEST_INFO) || !format_dest_state_send(fd, &info))
|
||||
return RECEIVER_STEP_FAIL;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
}
|
||||
File* dir = file_receive_directory(fd, config);
|
||||
if (!dir)
|
||||
return RECEIVER_STEP_ERROR;
|
||||
if (config->report_dest_info) {
|
||||
OutputDestState info;
|
||||
bool ok = receiver_probe_dest_state(config, file_wire_path(dir), NULL, &info);
|
||||
if (!ok || !send_status(fd, STATUS_DEST_INFO) || !format_dest_state_send(fd, &info)) {
|
||||
file_destroy(dir);
|
||||
return RECEIVER_STEP_FAIL;
|
||||
}
|
||||
}
|
||||
if (!state->sink->store_file(dir, state->sink->context))
|
||||
return RECEIVER_STEP_ERROR;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_dir_times(const ReceiverPendingState* state) {
|
||||
if (!receiver_process_dir_times(state->fd, state->config, state->sink))
|
||||
return RECEIVER_STEP_ERROR;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_hardlink(ReceiverPendingState* state) {
|
||||
File* file = file_receive_hardlink(state->fd);
|
||||
if (!file || !state->sink->store_file(file, state->sink->context))
|
||||
return RECEIVER_STEP_ERROR;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_symlink(ReceiverPendingState* state) {
|
||||
const Config* config = state->config;
|
||||
File* sym = file_receive_symlink(state->fd, config);
|
||||
if (!sym)
|
||||
return RECEIVER_STEP_ERROR;
|
||||
if (config->report_dest_info) {
|
||||
OutputDestState info;
|
||||
bool ok = receiver_probe_dest_state(config, file_wire_path(sym), sym->symlink_target, &info);
|
||||
if (!ok || !send_status(state->fd, STATUS_DEST_INFO) ||
|
||||
!format_dest_state_send(state->fd, &info)) {
|
||||
file_destroy(sym);
|
||||
return RECEIVER_STEP_FAIL;
|
||||
}
|
||||
}
|
||||
if (!state->sink->store_file(sym, state->sink->context))
|
||||
return RECEIVER_STEP_ERROR;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_special(ReceiverPendingState* state) {
|
||||
File* file = file_receive_special(state->fd);
|
||||
if (!file || !state->sink->store_file(file, state->sink->context))
|
||||
return RECEIVER_STEP_ERROR;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_manifest(ReceiverPendingState* state) {
|
||||
Config* config = state->config;
|
||||
int fd = state->fd;
|
||||
const ReceiverSink* sink = state->sink;
|
||||
DeleteManifest* manifest = receive_manifest_entries(fd);
|
||||
if (!manifest)
|
||||
return RECEIVER_STEP_FAIL; /* receive_manifest_entries already sent STATUS_ERROR */
|
||||
if (config->dry_run) {
|
||||
/* Server-contacting --dry-run mutates nothing, so a keep-set manifest
|
||||
is consumed and discarded. The early-delete mode still needs its ACK
|
||||
so a sender blocked on the delete handshake is not left hanging.
|
||||
When would-delete reporting is armed, enumerate (read-only) the
|
||||
destination extras so the terminal STATUS_STATS frame can list them. */
|
||||
if (config->use_delete && sink->would_delete) {
|
||||
size_t count = 0;
|
||||
if (!manifest_would_delete_list(config, manifest, sink->would_delete, &count))
|
||||
log_message(LOG_LEVEL_WARNING, "dry-run: could not enumerate would-delete paths");
|
||||
}
|
||||
delete_manifest_free(manifest);
|
||||
if (state->early_delete && !send_status(fd, STATUS_OK))
|
||||
return RECEIVER_STEP_FAIL;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
if (state->early_delete) {
|
||||
/* --delete-before: the whole-tree manifest is authoritative the moment
|
||||
it arrives, before any file data. Delete now and acknowledge so the
|
||||
sender only starts streaming once the deletion committed (or failed).
|
||||
A later transfer failure does not restore these deletions. A
|
||||
--max-delete-capped commit still succeeds and the transfer proceeds;
|
||||
the terminal success frame reports the cap. */
|
||||
size_t deleted = 0;
|
||||
ReceiverDeleteContext delctx;
|
||||
DeletePathObserver observer = receiver_delete_observer(sink, &delctx);
|
||||
DeleteCommitResult deletion =
|
||||
(config->use_delete || config->delete_missing_args)
|
||||
? manifest_delete_all_observed(config, manifest, &deleted, observer, &delctx)
|
||||
: DELETE_COMMIT_OK;
|
||||
receiver_tally_deleted(sink, deleted);
|
||||
delete_manifest_free(manifest);
|
||||
if (deletion == DELETE_COMMIT_ERROR) {
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return RECEIVER_STEP_FAIL;
|
||||
}
|
||||
if (deletion == DELETE_COMMIT_LIMIT_REACHED && sink->note_delete_limit)
|
||||
sink->note_delete_limit(sink->context);
|
||||
if (!send_status(fd, STATUS_OK))
|
||||
return RECEIVER_STEP_FAIL;
|
||||
} else if (config->use_delete || config->delete_missing_args) {
|
||||
/* Plain --delete / --delete-after and the --delete-missing-args
|
||||
exact-path deletions: hold the manifest and commit it only after
|
||||
STATUS_FINISHED. The per-directory modes never send this frame. */
|
||||
if (state->deferred_manifest) {
|
||||
log_message(LOG_LEVEL_ERROR, "Received a second delete manifest");
|
||||
delete_manifest_free(state->deferred_manifest);
|
||||
state->deferred_manifest = NULL;
|
||||
delete_manifest_free(manifest);
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return RECEIVER_STEP_FAIL;
|
||||
}
|
||||
state->deferred_manifest = manifest;
|
||||
} else {
|
||||
delete_manifest_free(manifest);
|
||||
}
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_delete_plan(ReceiverPendingState* state) {
|
||||
Config* config = state->config;
|
||||
int fd = state->fd;
|
||||
const ReceiverSink* sink = state->sink;
|
||||
if (!state->per_dir_delete) {
|
||||
log_message(LOG_LEVEL_ERROR, "Received a per-directory delete plan without a per-dir "
|
||||
"delete timing");
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return RECEIVER_STEP_FAIL;
|
||||
}
|
||||
if (!state->plan_session) {
|
||||
state->plan_session = delete_plan_session_create(config);
|
||||
if (state->plan_session && (sink->stats || sink->deleted_paths))
|
||||
delete_plan_session_set_delete_observer(state->plan_session, receiver_record_deleted_path,
|
||||
state->delete_ctx);
|
||||
}
|
||||
if (!state->plan_session || delete_plan_session_receive(state->plan_session, config, fd) != 0)
|
||||
return RECEIVER_STEP_FAIL;
|
||||
if (delete_plan_session_limit_reached(state->plan_session) && !state->delete_limit_noted &&
|
||||
sink->note_delete_limit) {
|
||||
sink->note_delete_limit(sink->context);
|
||||
state->delete_limit_noted = true;
|
||||
}
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
static ReceiverStep receiver_handle_file(ReceiverPendingState* state) {
|
||||
File* file = file_receive(state->config, state->fd);
|
||||
if (!file) {
|
||||
log_message(LOG_LEVEL_ERROR, "Failed to receive file");
|
||||
return RECEIVER_STEP_ERROR;
|
||||
}
|
||||
if (!state->sink->store_file(file, state->sink->context))
|
||||
return RECEIVER_STEP_ERROR;
|
||||
return RECEIVER_STEP_NEXT;
|
||||
}
|
||||
|
||||
/* One dispatch per admitted frame type; STATUS_NEXT (and any other
|
||||
data-bearing status) falls through to the regular file receiver. */
|
||||
static ReceiverStep receiver_dispatch_status(ReceiverPendingState* state, Status status) {
|
||||
switch (status) {
|
||||
case STATUS_KEEPALIVE:
|
||||
return receiver_handle_keepalive(state);
|
||||
case STATUS_CLIENT_MSG:
|
||||
return receiver_handle_client_msg(state);
|
||||
case STATUS_ABORT:
|
||||
return receiver_handle_abort(state);
|
||||
case STATUS_CHECK:
|
||||
return receiver_handle_check(state);
|
||||
case STATUS_CHUNK:
|
||||
return receiver_handle_chunk(state);
|
||||
case STATUS_CHECK_BATCH:
|
||||
return receiver_handle_check_batch(state);
|
||||
case STATUS_MKDIR:
|
||||
return receiver_handle_mkdir(state);
|
||||
case STATUS_DIR_TIMES:
|
||||
return receiver_handle_dir_times(state);
|
||||
case STATUS_HARDLINK:
|
||||
return receiver_handle_hardlink(state);
|
||||
case STATUS_SYMLINK:
|
||||
return receiver_handle_symlink(state);
|
||||
case STATUS_SPECIAL:
|
||||
return receiver_handle_special(state);
|
||||
case STATUS_MANIFEST:
|
||||
return receiver_handle_manifest(state);
|
||||
case STATUS_DELETE_PLAN:
|
||||
return receiver_handle_delete_plan(state);
|
||||
default:
|
||||
return receiver_handle_file(state);
|
||||
}
|
||||
}
|
||||
|
||||
/* Release the parked keep-set / per-directory session exactly once on every
|
||||
failure exit. Never commit a deletion for a failed stream. */
|
||||
static void receiver_drop_pending(ReceiverPendingState* state) {
|
||||
if (state->deferred_manifest) {
|
||||
delete_manifest_free(state->deferred_manifest);
|
||||
state->deferred_manifest = NULL;
|
||||
}
|
||||
if (state->plan_session) {
|
||||
delete_plan_session_destroy(state->plan_session);
|
||||
state->plan_session = NULL;
|
||||
}
|
||||
return receiver_process_pending(config, file_descriptor, sink, NULL, NULL);
|
||||
}
|
||||
|
||||
/* Runs the whole receive loop. The delete manifest may legitimately arrive
|
||||
either FIRST (--delete-before / --delete-during: the sender transmits the
|
||||
validated keep-set before any file data) or LAST (--delete-after /
|
||||
--delete-commit / --delete-delay: the manifest closes the data stream). In
|
||||
validated keep-set before any file data) or LAST (plain --delete /
|
||||
--delete-after / --delete-delay: the manifest closes the data stream). In
|
||||
the early modes the receiver deletes as soon as the manifest has been read
|
||||
and acknowledges with STATUS_OK so the sender only starts streaming once the
|
||||
deletion has committed (or failed); in the late modes the manifest is held
|
||||
and the deletion is committed only after the terminal STATUS_FINISHED proves
|
||||
the whole transfer succeeded. A plain --delete defaults to the per-directory
|
||||
delete-during plan mode (no manifest at all). See the per-frame handlers
|
||||
above for how the -m receiver defers that commit until its disk writer has
|
||||
drained. */
|
||||
int receiver_process_pending_ctx(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||
DeleteManifest** pending_manifest,
|
||||
DeletePlanSession** pending_plans,
|
||||
ReceiverDeleteContext* observer_ctx) {
|
||||
the whole transfer succeeded. See receiver_process_pending() for how the -m
|
||||
receiver defers that commit until its disk writer has drained. */
|
||||
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
|
||||
Status status;
|
||||
if (!receive_status(file_descriptor, &status))
|
||||
return -1;
|
||||
@@ -757,37 +303,158 @@ int receiver_process_pending_ctx(Config* config, int file_descriptor, const Rece
|
||||
last_progress = session_start;
|
||||
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
|
||||
return -1;
|
||||
/* A per-directory delete session handed to the caller outlives this stack
|
||||
frame, so its observer context must be caller-owned (observer_ctx); only
|
||||
the default inline-commit case may use the stack context. */
|
||||
ReceiverDeleteContext local_ctx;
|
||||
ReceiverDeleteContext* delete_ctx = observer_ctx ? observer_ctx : &local_ctx;
|
||||
delete_ctx->stats = sink ? sink->stats : NULL;
|
||||
delete_ctx->deleted_paths = sink ? sink->deleted_paths : NULL;
|
||||
ReceiverPendingState state = {
|
||||
.config = config,
|
||||
.fd = file_descriptor,
|
||||
.sink = sink,
|
||||
.pending_manifest = pending_manifest,
|
||||
.pending_plans = pending_plans,
|
||||
.deferred_manifest = NULL,
|
||||
.plan_session = NULL,
|
||||
.delete_ctx = delete_ctx,
|
||||
.early_delete = config_delete_timing_early(config),
|
||||
.per_dir_delete = config_delete_timing_per_dir(config),
|
||||
.delete_limit_noted = false,
|
||||
};
|
||||
bool notify_peer = false;
|
||||
bool early_delete = config_delete_timing_early(config);
|
||||
bool per_dir_delete = config_delete_timing_per_dir(config);
|
||||
/* Parked keep-set for the late/commit timing. Every exit path below frees it
|
||||
exactly once; the only exception is the successful FINISHED handoff, which
|
||||
transfers ownership to *pending_manifest (used by the -m receiver). */
|
||||
DeleteManifest* deferred_manifest = NULL;
|
||||
/* Per-directory delete session for --delete-during/--delete-delay. During the
|
||||
loop it applies plans inline (during) or snapshots their extras (delay); on
|
||||
a successful FINISHED it is either committed here or handed to
|
||||
*pending_plans so the -m caller commits after its disk writer drained. */
|
||||
DeletePlanSession* plan_session = NULL;
|
||||
bool delete_limit_noted = false;
|
||||
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK ||
|
||||
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH ||
|
||||
status == STATUS_MKDIR || status == STATUS_MANIFEST || status == STATUS_HARDLINK ||
|
||||
status == STATUS_SYMLINK || status == STATUS_SPECIAL || status == STATUS_DIR_TIMES ||
|
||||
status == STATUS_DELETE_PLAN || status == STATUS_CLIENT_MSG) {
|
||||
ReceiverStep step = receiver_dispatch_status(&state, status);
|
||||
if (step == RECEIVER_STEP_FAIL)
|
||||
status == STATUS_DELETE_PLAN) {
|
||||
if (status == STATUS_KEEPALIVE) {
|
||||
if (!send_status(file_descriptor, STATUS_KEEPALIVE))
|
||||
goto fail;
|
||||
goto next_status;
|
||||
}
|
||||
if (status == STATUS_ABORT) {
|
||||
log_message(LOG_LEVEL_INFO, "Received abort from client, cleaning up");
|
||||
goto fail;
|
||||
if (step == RECEIVER_STEP_ERROR)
|
||||
goto receive_error;
|
||||
}
|
||||
if (status == STATUS_CHECK) {
|
||||
bool skipped = false;
|
||||
bool would_transfer = false;
|
||||
File* file = receive_incremental_check_ex(file_descriptor, config, &skipped, &would_transfer);
|
||||
if (config->dry_run) {
|
||||
/* Server-contacting --dry-run: the reply has already been sent
|
||||
(STATUS_OK = up to date, STATUS_DRY_RUN_TRANSFER = would transfer) and
|
||||
nothing may be stored. Both flags false means a genuine protocol
|
||||
error (STATUS_ERROR already sent or sent by receive_error below). */
|
||||
if (!skipped && !would_transfer)
|
||||
goto receive_error;
|
||||
} else if (!skipped && (!file || !sink->store_file(file, sink->context))) {
|
||||
goto receive_error;
|
||||
}
|
||||
} else if (status == STATUS_CHUNK) {
|
||||
Chunk* chunk = receive_chunk_data(file_descriptor, config);
|
||||
if (!chunk || !receiver_process_chunk(chunk, sink))
|
||||
goto receive_error;
|
||||
} else if (status == STATUS_CHECK_BATCH) {
|
||||
if (!receiver_process_batch(config, file_descriptor))
|
||||
goto fail;
|
||||
goto next_status;
|
||||
} else if (status == STATUS_MKDIR) {
|
||||
File* dir = file_receive_directory(file_descriptor, config);
|
||||
if (!dir || !sink->store_file(dir, sink->context))
|
||||
goto receive_error;
|
||||
} else if (status == STATUS_DIR_TIMES) {
|
||||
if (!receiver_process_dir_times(file_descriptor, config, sink))
|
||||
goto receive_error;
|
||||
} else if (status == STATUS_HARDLINK) {
|
||||
File* file = file_receive_hardlink(file_descriptor);
|
||||
if (!file || !sink->store_file(file, sink->context))
|
||||
goto receive_error;
|
||||
} else if (status == STATUS_SYMLINK) {
|
||||
File* sym = file_receive_symlink(file_descriptor, config);
|
||||
if (!sym || !sink->store_file(sym, sink->context))
|
||||
goto receive_error;
|
||||
} else if (status == STATUS_SPECIAL) {
|
||||
File* file = file_receive_special(file_descriptor);
|
||||
if (!file || !sink->store_file(file, sink->context))
|
||||
goto receive_error;
|
||||
} else if (status == STATUS_MANIFEST) {
|
||||
DeleteManifest* manifest = receive_manifest_entries(file_descriptor);
|
||||
if (!manifest)
|
||||
goto fail; /* receive_manifest_entries already sent STATUS_ERROR */
|
||||
if (config->dry_run) {
|
||||
/* Server-contacting --dry-run mutates nothing, so a keep-set manifest
|
||||
is consumed and discarded. The early-delete mode still needs its ACK
|
||||
so a sender blocked on the delete handshake is not left hanging.
|
||||
When would-delete reporting is armed, enumerate (read-only) the
|
||||
destination extras so the terminal STATUS_STATS frame can list them. */
|
||||
if (config->use_delete && sink->would_delete) {
|
||||
size_t count = 0;
|
||||
if (!manifest_would_delete_list(config, manifest, sink->would_delete, &count))
|
||||
log_message(LOG_LEVEL_WARNING, "dry-run: could not enumerate would-delete paths");
|
||||
}
|
||||
delete_manifest_free(manifest);
|
||||
if (early_delete && !send_status(file_descriptor, STATUS_OK))
|
||||
goto fail;
|
||||
goto next_status;
|
||||
}
|
||||
if (early_delete) {
|
||||
/* --delete-before: the whole-tree manifest is authoritative the moment
|
||||
it arrives, before any file data. Delete now and acknowledge so the
|
||||
sender only starts streaming once the deletion committed (or failed).
|
||||
A later transfer failure does not restore these deletions. A
|
||||
--max-delete-capped commit still succeeds and the transfer proceeds;
|
||||
the terminal success frame reports the cap. */
|
||||
size_t deleted = 0;
|
||||
DeleteCommitResult deletion = (config->use_delete || config->delete_missing_args)
|
||||
? manifest_delete_all_counted(config, manifest, &deleted)
|
||||
: DELETE_COMMIT_OK;
|
||||
receiver_tally_deleted(sink, deleted);
|
||||
delete_manifest_free(manifest);
|
||||
if (deletion == DELETE_COMMIT_ERROR) {
|
||||
send_status(file_descriptor, STATUS_ERROR);
|
||||
goto fail;
|
||||
}
|
||||
if (deletion == DELETE_COMMIT_LIMIT_REACHED && sink->note_delete_limit)
|
||||
sink->note_delete_limit(sink->context);
|
||||
if (!send_status(file_descriptor, STATUS_OK))
|
||||
goto fail;
|
||||
} else if (config->use_delete || config->delete_missing_args) {
|
||||
/* Plain --delete / --delete-after and the --delete-missing-args
|
||||
exact-path deletions: hold the manifest and commit it only after
|
||||
STATUS_FINISHED. The per-directory modes never send this frame. */
|
||||
if (deferred_manifest) {
|
||||
log_message(LOG_LEVEL_ERROR, "Received a second delete manifest");
|
||||
delete_manifest_free(deferred_manifest);
|
||||
deferred_manifest = NULL;
|
||||
delete_manifest_free(manifest);
|
||||
send_status(file_descriptor, STATUS_ERROR);
|
||||
goto fail;
|
||||
}
|
||||
deferred_manifest = manifest;
|
||||
} else {
|
||||
delete_manifest_free(manifest);
|
||||
}
|
||||
goto next_status;
|
||||
} else if (status == STATUS_DELETE_PLAN) {
|
||||
if (!per_dir_delete) {
|
||||
log_message(LOG_LEVEL_ERROR, "Received a per-directory delete plan without a per-dir "
|
||||
"delete timing");
|
||||
send_status(file_descriptor, STATUS_ERROR);
|
||||
goto fail;
|
||||
}
|
||||
if (!plan_session)
|
||||
plan_session = delete_plan_session_create(config);
|
||||
if (!plan_session || delete_plan_session_receive(plan_session, config, file_descriptor) != 0)
|
||||
goto fail;
|
||||
if (delete_plan_session_limit_reached(plan_session) && !delete_limit_noted &&
|
||||
sink->note_delete_limit) {
|
||||
sink->note_delete_limit(sink->context);
|
||||
delete_limit_noted = true;
|
||||
}
|
||||
goto next_status;
|
||||
} else {
|
||||
File* file = file_receive(config, file_descriptor);
|
||||
if (!file) {
|
||||
log_message(LOG_LEVEL_ERROR, "Failed to receive file");
|
||||
goto receive_error;
|
||||
}
|
||||
if (!sink->store_file(file, sink->context))
|
||||
goto receive_error;
|
||||
}
|
||||
next_status:
|
||||
if (!receive_status(file_descriptor, &status))
|
||||
goto receive_error;
|
||||
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
|
||||
@@ -797,39 +464,26 @@ int receiver_process_pending_ctx(Config* config, int file_descriptor, const Rece
|
||||
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
|
||||
goto receive_error;
|
||||
}
|
||||
/* --delay-updates: publish every staged file BEFORE the deferred delete
|
||||
commit, matching rsync's --delete-after ordering (all updates land first,
|
||||
then extras are removed). The single-threaded receiver stores files
|
||||
synchronously, so every staged file is complete here. The -m receiver
|
||||
hands both publication and deletion to its caller via
|
||||
pending_manifest/pending_plans; that caller publishes first, after its disk
|
||||
writer has drained. */
|
||||
bool handoff = state.pending_manifest != NULL || state.pending_plans != NULL;
|
||||
if (!handoff && !config->dry_run && config->delay_updates && config->delay_context) {
|
||||
if (!delay_updates_publish(config->delay_context, config)) {
|
||||
send_status(file_descriptor, STATUS_ERROR);
|
||||
goto fail;
|
||||
}
|
||||
}
|
||||
/* Commit-style (late) deletion: every data frame has been received and the
|
||||
sender proved the whole tree with STATUS_FINISHED. The single-threaded
|
||||
receiver stores files synchronously, so everything is on disk here (and a
|
||||
--delay-updates run has already published above). The -m receiver passes
|
||||
`pending_manifest` because its disk writer may still be draining; the
|
||||
caller commits after the writer has joined so no extra file is removed
|
||||
unless the transfer is known to have succeeded. */
|
||||
if (state.deferred_manifest) {
|
||||
if (state.pending_manifest) {
|
||||
*state.pending_manifest = state.deferred_manifest;
|
||||
state.deferred_manifest = NULL;
|
||||
receiver stores files synchronously, so everything is on disk here and the
|
||||
deletion can be committed before the --delay-updates publication in
|
||||
send_success (the walker skips the staging dir, so staged files are never
|
||||
treated as extras). The -m receiver passes `pending_manifest` because its
|
||||
disk writer may still be draining; the caller commits after the writer has
|
||||
joined so no extra file is removed unless the transfer is known to have
|
||||
succeeded. */
|
||||
if (deferred_manifest) {
|
||||
if (pending_manifest) {
|
||||
*pending_manifest = deferred_manifest;
|
||||
deferred_manifest = NULL;
|
||||
} else {
|
||||
size_t deleted = 0;
|
||||
DeletePathObserver observer = receiver_delete_observer(sink, state.delete_ctx);
|
||||
DeleteCommitResult deletion = manifest_delete_all_observed(
|
||||
config, state.deferred_manifest, &deleted, observer, state.delete_ctx);
|
||||
DeleteCommitResult deletion =
|
||||
manifest_delete_all_counted(config, deferred_manifest, &deleted);
|
||||
receiver_tally_deleted(sink, deleted);
|
||||
delete_manifest_free(state.deferred_manifest);
|
||||
state.deferred_manifest = NULL;
|
||||
delete_manifest_free(deferred_manifest);
|
||||
deferred_manifest = NULL;
|
||||
if (deletion == DELETE_COMMIT_ERROR) {
|
||||
send_status(file_descriptor, STATUS_ERROR);
|
||||
goto fail;
|
||||
@@ -843,28 +497,25 @@ int receiver_process_pending_ctx(Config* config, int file_descriptor, const Rece
|
||||
nothing yet and applies its decompressed snapshot here. The -m receiver
|
||||
hands the session to its caller instead, which commits after the disk
|
||||
writer drained. */
|
||||
if (state.plan_session) {
|
||||
if (sink->stats || sink->deleted_paths)
|
||||
delete_plan_session_set_delete_observer(state.plan_session, receiver_record_deleted_path,
|
||||
state.delete_ctx);
|
||||
if (state.pending_plans) {
|
||||
*state.pending_plans = state.plan_session;
|
||||
state.plan_session = NULL;
|
||||
if (plan_session) {
|
||||
if (pending_plans) {
|
||||
*pending_plans = plan_session;
|
||||
plan_session = NULL;
|
||||
} else if (config->dry_run) {
|
||||
/* Central dry-run no-op: never commit a deletion for a -n run. */
|
||||
delete_plan_session_destroy(state.plan_session);
|
||||
state.plan_session = NULL;
|
||||
delete_plan_session_destroy(plan_session);
|
||||
plan_session = NULL;
|
||||
} else {
|
||||
DeleteCommitResult deletion = delete_plan_session_commit(state.plan_session, config);
|
||||
bool limit = delete_plan_session_limit_reached(state.plan_session);
|
||||
receiver_tally_deleted(sink, delete_plan_session_deleted(state.plan_session));
|
||||
delete_plan_session_destroy(state.plan_session);
|
||||
state.plan_session = NULL;
|
||||
DeleteCommitResult deletion = delete_plan_session_commit(plan_session, config);
|
||||
bool limit = delete_plan_session_limit_reached(plan_session);
|
||||
receiver_tally_deleted(sink, delete_plan_session_deleted(plan_session));
|
||||
delete_plan_session_destroy(plan_session);
|
||||
plan_session = NULL;
|
||||
if (deletion == DELETE_COMMIT_ERROR) {
|
||||
send_status(file_descriptor, STATUS_ERROR);
|
||||
goto fail;
|
||||
}
|
||||
if (limit && !state.delete_limit_noted && sink->note_delete_limit)
|
||||
if (limit && !delete_limit_noted && sink->note_delete_limit)
|
||||
sink->note_delete_limit(sink->context);
|
||||
}
|
||||
}
|
||||
@@ -878,14 +529,26 @@ int receiver_process_pending_ctx(Config* config, int file_descriptor, const Rece
|
||||
}
|
||||
return 0;
|
||||
|
||||
receive_error:
|
||||
notify_peer = true;
|
||||
fail:
|
||||
/* Failure exits that must not (or already did) report a STATUS_ERROR. The
|
||||
parked keep-set/session is dropped: never commit a deletion for a failed
|
||||
stream. */
|
||||
receiver_drop_pending(&state);
|
||||
if (notify_peer && sink->send_error)
|
||||
if (deferred_manifest) {
|
||||
delete_manifest_free(deferred_manifest);
|
||||
deferred_manifest = NULL;
|
||||
}
|
||||
if (plan_session)
|
||||
delete_plan_session_destroy(plan_session);
|
||||
return -1;
|
||||
|
||||
receive_error:
|
||||
if (deferred_manifest) {
|
||||
delete_manifest_free(deferred_manifest);
|
||||
deferred_manifest = NULL;
|
||||
}
|
||||
if (plan_session)
|
||||
delete_plan_session_destroy(plan_session);
|
||||
if (sink->send_error)
|
||||
send_status(file_descriptor, STATUS_ERROR);
|
||||
return -1;
|
||||
}
|
||||
@@ -906,20 +569,11 @@ typedef struct {
|
||||
--delete would-delete path list collected while processing the manifest. */
|
||||
ReceiverStats stats;
|
||||
ArrayList* would_delete;
|
||||
/* --info=del: actually-removed paths collected during the delete commit. */
|
||||
ArrayList* deleted_paths;
|
||||
/* Per-run count of entries that failed to materialize without aborting the
|
||||
stream (currently ONLY a --devices mknod EPERM/EACCES). A nonzero count
|
||||
makes the terminal frame carry a non-OK status so the client exits
|
||||
non-zero, matching rsync's continue-and-exit-partial behavior. */
|
||||
size_t failed_entries;
|
||||
} ReceiverSaveContext;
|
||||
|
||||
static bool receiver_save_file(File* file, void* context_pointer) {
|
||||
ReceiverSaveContext* context = context_pointer;
|
||||
FileSaveResult result = FILE_SAVE_ERROR;
|
||||
bool created = false;
|
||||
unsigned created_dirs = 0;
|
||||
if (context->config->dry_run) {
|
||||
/* Defense in depth: a dry-run receiver mutates nothing even if a data
|
||||
frame reaches the sink (the sender is not supposed to send one). */
|
||||
@@ -929,22 +583,12 @@ static bool receiver_save_file(File* file, void* context_pointer) {
|
||||
--remove-source-files sender keeps its source. */
|
||||
result = FILE_SAVE_SKIPPED;
|
||||
} else {
|
||||
result = file_save_to_disk_full_ex(context->config->receive_root_directory, file,
|
||||
context->config, &created, &created_dirs);
|
||||
result = file_save_to_disk_full(context->config->receive_root_directory, file, context->config);
|
||||
}
|
||||
/* Wire-stats tally: bytes reconstructed from the basis file (delta matches)
|
||||
count as matched data in the end-of-transfer report. */
|
||||
if (result != FILE_SAVE_ERROR && file->matched_bytes > 0)
|
||||
context->stats.matched_data += file->matched_bytes;
|
||||
/* --devices parity: a device node the receiver could not mknod (EPERM/EACCES)
|
||||
is counted per-run but does not abort the transfer. The terminal frame
|
||||
turns a nonzero count into a non-OK status so the client exits non-zero. */
|
||||
if (result == FILE_SAVE_FAILED)
|
||||
context->failed_entries++;
|
||||
/* Protocol 2.28.0: receiver-observed literal bytes and the created-entry
|
||||
breakdown (regular/dir/link/special) for the `--stats` report. */
|
||||
if (result == FILE_SAVE_WRITTEN)
|
||||
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
|
||||
/* A directory's metadata is deferred, never applied inline: collect it now
|
||||
and apply it at the end. -O/--omit-dir-times and --preserve_perms/-times
|
||||
are honored by dir_metadata_list_apply's caller (see
|
||||
@@ -974,37 +618,30 @@ static void receiver_note_delete_limit(void* context_pointer) {
|
||||
context->delete_limit_reached = true;
|
||||
}
|
||||
|
||||
/* Terminal status for a run. A capped --delete limit wins (rsync exit 25);
|
||||
otherwise any per-entry failure (for example an unprivileged --devices
|
||||
mknod) makes the terminal frame STATUS_PARTIAL so the client exits 23
|
||||
(rsync's "partial transfer due to error") while still removing the sources
|
||||
it successfully transferred under --remove-source-files. A fatal stream
|
||||
error keeps STATUS_ERROR (a non-23 exit). A clean run keeps STATUS_OK. */
|
||||
static Status receiver_final_status(bool delete_limit_reached, size_t failed_entries) {
|
||||
if (delete_limit_reached)
|
||||
return STATUS_DELETE_LIMIT;
|
||||
return failed_entries > 0 ? STATUS_PARTIAL : STATUS_OK;
|
||||
}
|
||||
|
||||
static bool receiver_send_success_frame(int fd, void* context_pointer) {
|
||||
ReceiverSaveContext* context = context_pointer;
|
||||
if (context->failed_entries > 0)
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"%zu entr%s failed to materialize; continuing (partial transfer)",
|
||||
context->failed_entries, context->failed_entries == 1 ? "y" : "ies");
|
||||
Status final_status =
|
||||
receiver_final_status(context->delete_limit_reached, context->failed_entries);
|
||||
if (!receiver_send_stats_frame(fd, context->config, &context->stats, context->would_delete,
|
||||
context->deleted_paths))
|
||||
Status final_status = context->delete_limit_reached ? STATUS_DELETE_LIMIT : STATUS_OK;
|
||||
if (!receiver_send_stats_frame(fd, context->config, &context->stats, context->would_delete))
|
||||
return false;
|
||||
/* Server-contacting --dry-run: nothing was staged or written, so there is
|
||||
nothing to publish and no directory times to stamp. */
|
||||
if (context->config->dry_run)
|
||||
return receiver_send_final_success(fd, context->config, &context->outcomes, final_status);
|
||||
/* P7 Wave D: every child is now written and the --delay-updates publication
|
||||
(done in receiver_process before the delete commit) plus the deferred
|
||||
deletion have both committed, so it is finally safe to stamp directory
|
||||
times. */
|
||||
/* --delay-updates: the whole protocol stream (including manifest/delete
|
||||
handling, which ran inside receiver_process) has succeeded and every
|
||||
staged file was fully written. Publish them atomically now, before the
|
||||
success/outcome frame tells a --remove-source-files sender it may delete
|
||||
its sources. */
|
||||
if (context->config->delay_updates && context->config->delay_context) {
|
||||
if (!delay_updates_publish(context->config->delay_context, context->config)) {
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
/* P7 Wave D: every child is now written and the delete / --delay-updates
|
||||
phases have committed, so it is finally safe to stamp directory times.
|
||||
This runs after the deferred deletion because receiver_process commits it
|
||||
before calling this success frame. */
|
||||
dir_metadata_list_apply(&context->dir_times, context->config->receive_root_directory,
|
||||
context->config);
|
||||
return receiver_send_final_success(fd, context->config, &context->outcomes, final_status);
|
||||
@@ -1014,15 +651,8 @@ int receiver_receive_files(Config* config, int file_descriptor) {
|
||||
ReceiverSaveContext context = {.config = config, .outcomes = {0}};
|
||||
dir_time_list_init(&context.dir_times);
|
||||
context.would_delete = array_list_create(free);
|
||||
/* report_deletes (--info=del / -i / --out-format under --delete) is the only
|
||||
reason to retain the actually-removed paths; a plain --delete must not
|
||||
str_dup every removal. NULL is handled by every consumer. */
|
||||
context.deleted_paths = config->report_deletes ? array_list_create(free) : NULL;
|
||||
if (!context.would_delete || (config->report_deletes && !context.deleted_paths)) {
|
||||
array_list_delete(context.would_delete);
|
||||
array_list_delete(context.deleted_paths);
|
||||
if (!context.would_delete)
|
||||
return -1;
|
||||
}
|
||||
ReceiverSink sink = {receiver_save_file,
|
||||
&context,
|
||||
true,
|
||||
@@ -1030,14 +660,12 @@ int receiver_receive_files(Config* config, int file_descriptor) {
|
||||
receiver_send_success_frame,
|
||||
receiver_note_delete_limit,
|
||||
&context.stats,
|
||||
context.would_delete,
|
||||
context.deleted_paths};
|
||||
context.would_delete};
|
||||
int ret = receiver_process(config, file_descriptor, &sink);
|
||||
if (ret != 0 && config->delay_updates && config->delay_context)
|
||||
delay_updates_cleanup(config->delay_context);
|
||||
receiver_outcomes_destroy(&context.outcomes);
|
||||
dir_time_list_free(&context.dir_times);
|
||||
array_list_delete(context.would_delete);
|
||||
array_list_delete(context.deleted_paths);
|
||||
return ret;
|
||||
}
|
||||
|
||||
+1
-32
@@ -46,30 +46,11 @@ typedef struct {
|
||||
carries the -n/--dry-run --delete path list. */
|
||||
ReceiverStats* stats;
|
||||
struct ArrayList* would_delete;
|
||||
/* When --info=del requested it, receiver-owned strings for every path the
|
||||
deletion commit ACTUALLY removed, sent in the terminal STATUS_STATS frame's
|
||||
path list so the sender can print rsync's `deleting PATH` lines. */
|
||||
struct ArrayList* deleted_paths;
|
||||
} ReceiverSink;
|
||||
|
||||
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code);
|
||||
void receiver_outcomes_destroy(ReceiverOutcomes* outcomes);
|
||||
|
||||
/* Delete observer context: `deleted_paths` (optional) receives owned copies of
|
||||
every truly-removed destination-relative path for --info=del; `stats`
|
||||
(optional) receives the per-type `Number of deleted files` tallies for
|
||||
--stats. Both may be NULL, in which case the observer is a no-op. */
|
||||
typedef struct {
|
||||
ReceiverStats* stats;
|
||||
struct ArrayList* deleted_paths;
|
||||
} ReceiverDeleteContext;
|
||||
|
||||
/* DeletePathObserver implementation: records each truly-removed path (when the
|
||||
context carries a path list) and tallies it by type (when it carries a stats
|
||||
record). Shared by the single-threaded receiver and the -m pipeline's
|
||||
deferred commit. */
|
||||
void receiver_record_deleted_path(void* context, const char* rel_path, DeleteEntryType type);
|
||||
|
||||
/* Send the terminal success frame. `final_status` is usually STATUS_OK, or
|
||||
STATUS_DELETE_LIMIT when a --max-delete commit was capped. */
|
||||
bool receiver_send_final_success(int fd, const Config* config, const ReceiverOutcomes* outcomes,
|
||||
@@ -79,8 +60,7 @@ bool receiver_send_final_success(int fd, const Config* config, const ReceiverOut
|
||||
non-NULL, a count and that many wire strings) when the wire config requested
|
||||
report_stats. A no-op otherwise. */
|
||||
bool receiver_send_stats_frame(int fd, const Config* config, const ReceiverStats* stats,
|
||||
const struct ArrayList* would_delete,
|
||||
const struct ArrayList* deleted_paths);
|
||||
const struct ArrayList* would_delete);
|
||||
|
||||
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink);
|
||||
/* receiver_process with an escape hatch for the commit-style (late) deletion:
|
||||
@@ -93,17 +73,6 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
|
||||
for either to keep the default behaviour (delete before the success frame). */
|
||||
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans);
|
||||
/* receiver_process_pending() with an explicit observer context for a
|
||||
per-directory delete session that is handed to the caller via
|
||||
`pending_plans`. The session outlives this call (the -m pipeline commits it
|
||||
after joining its disk writer), so its observer context must too: pass a
|
||||
long-lived object such as PipelineContextReceiver.delete_ctx. When
|
||||
`delete_ctx` is NULL an internal stack context is used, which is only safe
|
||||
when the session is committed before returning (the default behaviour). */
|
||||
int receiver_process_pending_ctx(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||
DeleteManifest** pending_manifest,
|
||||
DeletePlanSession** pending_plans,
|
||||
ReceiverDeleteContext* delete_ctx);
|
||||
int receiver_receive_files(Config* config, int file_descriptor);
|
||||
|
||||
/* ---- Connection time bounds (anti-slowloris) ----
|
||||
|
||||
@@ -28,13 +28,9 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
|
||||
context->max_queue_bytes = 0;
|
||||
context->deferred_manifest = NULL;
|
||||
context->deferred_plans = NULL;
|
||||
context->delete_ctx.stats = NULL;
|
||||
context->delete_ctx.deleted_paths = NULL;
|
||||
context->delete_limit_reached = false;
|
||||
context->failed_entries = 0;
|
||||
memset(&context->stats, 0, sizeof(context->stats));
|
||||
context->would_delete = NULL;
|
||||
context->deleted_paths = NULL;
|
||||
atomic_init(&context->cancelled, false);
|
||||
int init = 0;
|
||||
if (mtx_init(&context->mutex, mtx_plain) != thrd_success)
|
||||
@@ -50,15 +46,6 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
|
||||
context->would_delete = array_list_create(free);
|
||||
if (!context->would_delete)
|
||||
goto fail;
|
||||
/* The actually-removed path list is only needed to render rsync's
|
||||
`deleting PATH` lines, which the client requests via report_deletes
|
||||
(--info=del / -i / --out-format under --delete). A plain --delete run must
|
||||
not allocate it or observe every removal. */
|
||||
if (config->report_deletes) {
|
||||
context->deleted_paths = array_list_create(free);
|
||||
if (!context->deleted_paths)
|
||||
goto fail;
|
||||
}
|
||||
return context;
|
||||
|
||||
fail:
|
||||
@@ -69,12 +56,6 @@ fail:
|
||||
cnd_destroy(&context->condition_not_full);
|
||||
if (init >= 1)
|
||||
mtx_destroy(&context->mutex);
|
||||
/* Free every list that was already created before the failing allocation:
|
||||
`context` itself is freed below, so they would otherwise leak. */
|
||||
if (context->would_delete)
|
||||
array_list_delete(context->would_delete);
|
||||
if (context->deleted_paths)
|
||||
array_list_delete(context->deleted_paths);
|
||||
free(context);
|
||||
return NULL;
|
||||
}
|
||||
@@ -90,8 +71,6 @@ void pipeline_context_receiver_destroy(PipelineContextReceiver* context) {
|
||||
dir_time_list_free(&context->dir_times);
|
||||
if (context->would_delete)
|
||||
array_list_delete(context->would_delete);
|
||||
if (context->deleted_paths)
|
||||
array_list_delete(context->deleted_paths);
|
||||
mtx_destroy(&context->mutex);
|
||||
cnd_destroy(&context->condition_not_full);
|
||||
cnd_destroy(&context->condition_not_empty);
|
||||
@@ -205,11 +184,9 @@ int receive_thread(void* pipeline_context) {
|
||||
NULL,
|
||||
receiver_pipeline_note_delete_limit,
|
||||
&context->stats,
|
||||
context->would_delete,
|
||||
context->deleted_paths};
|
||||
if (receiver_process_pending_ctx((Config*)config, file_descriptor, &sink,
|
||||
&context->deferred_manifest, &context->deferred_plans,
|
||||
&context->delete_ctx) != 0) {
|
||||
context->would_delete};
|
||||
if (receiver_process_pending((Config*)config, file_descriptor, &sink, &context->deferred_manifest,
|
||||
&context->deferred_plans) != 0) {
|
||||
receiver_thread_fail(context);
|
||||
protocol_session_unbind();
|
||||
return thrd_error;
|
||||
@@ -251,30 +228,12 @@ int write_thread(void* pipeline_context) {
|
||||
}
|
||||
size_t file_bytes = file->data ? file->data->size : 0;
|
||||
FileSaveResult result = FILE_SAVE_SKIPPED;
|
||||
bool created = false;
|
||||
unsigned created_dirs = 0;
|
||||
/* Server-contacting --dry-run: never write. The receiver thread does not
|
||||
enqueue anything on the dry-run path, but this keeps the writer thread
|
||||
provably mutation-free if a data frame ever reached it. */
|
||||
bool dry_run = context->config->dry_run;
|
||||
if (save_to_disk && !dry_run) {
|
||||
result =
|
||||
file_save_to_disk_full_ex(root_directory, file, context->config, &created, &created_dirs);
|
||||
if (result == FILE_SAVE_WRITTEN) {
|
||||
/* Protocol 2.28.0: fold the receiver-observed literal bytes and the
|
||||
created-entry type into the shared stats block under its mutex (the
|
||||
receive thread also writes stats.matched_data). */
|
||||
mtx_lock(&context->mutex);
|
||||
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
|
||||
mtx_unlock(&context->mutex);
|
||||
}
|
||||
/* --devices parity: a device node that could not be mknod'ed is counted
|
||||
per-run but does NOT abort the transfer. */
|
||||
if (result == FILE_SAVE_FAILED) {
|
||||
mtx_lock(&context->mutex);
|
||||
context->failed_entries++;
|
||||
mtx_unlock(&context->mutex);
|
||||
}
|
||||
result = file_save_to_disk_full(root_directory, file, context->config);
|
||||
if (result == FILE_SAVE_ERROR) {
|
||||
file_destroy(file);
|
||||
pipeline_context_receiver_note_bytes_released(context, file_bytes);
|
||||
|
||||
@@ -46,11 +46,6 @@ typedef struct PipelineContextReceiver {
|
||||
committing while the disk writer may still be draining; server.c commits it
|
||||
after both threads joined. NULL for every other timing. */
|
||||
DeletePlanSession* deferred_plans;
|
||||
/* Observer context for `deferred_plans`. It must outlive the receive thread
|
||||
(the session is committed by server.c after both threads join), so it lives
|
||||
here rather than on receiver_process_pending()'s stack; receive_thread
|
||||
installs it on the session. */
|
||||
ReceiverDeleteContext delete_ctx;
|
||||
/* Set by server.c when the deferred delete commit hit the --max-delete
|
||||
budget; the terminal success frame then carries STATUS_DELETE_LIMIT
|
||||
(rsync exit 25) while the transfer itself still succeeds. */
|
||||
@@ -66,14 +61,6 @@ typedef struct PipelineContextReceiver {
|
||||
/* -n/--dry-run --delete would-delete path list, collected by receive_thread
|
||||
and reported in the STATUS_STATS frame. */
|
||||
struct ArrayList* would_delete;
|
||||
/* --info=del actually-removed path list, collected by the deferred delete
|
||||
commit in server.c and reported in the STATUS_STATS frame. */
|
||||
struct ArrayList* deleted_paths;
|
||||
/* Per-run count of entries that failed to materialize without aborting the
|
||||
stream (currently ONLY a --devices mknod EPERM/EACCES). write_thread
|
||||
increments it under `mutex`; server.c turns a nonzero count into a non-OK
|
||||
terminal status so the client exits non-zero. */
|
||||
size_t failed_entries;
|
||||
} PipelineContextReceiver;
|
||||
|
||||
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
|
||||
|
||||
+436
-596
File diff suppressed because it is too large
Load Diff
@@ -3,12 +3,20 @@
|
||||
#include "credentials.h"
|
||||
#include "utils.h"
|
||||
#include <limits.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/socket.h>
|
||||
|
||||
#define set_error utils_set_error
|
||||
static void set_error(char* err, size_t err_size, const char* fmt, ...) {
|
||||
if (!err || err_size == 0)
|
||||
return;
|
||||
va_list args;
|
||||
va_start(args, fmt);
|
||||
vsnprintf(err, err_size, fmt, args);
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
void server_cli_options_default(ServerCliOptions* opts) {
|
||||
if (!opts)
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
ArrayList* array_list_create(void (*item_destroyer)(void* item)) {
|
||||
ArrayList* list = (ArrayList*)protocol_alloc(sizeof(ArrayList));
|
||||
if (list == NULL) {
|
||||
log_message(LOG_LEVEL_ERROR, "%s", "ERROR: Could not allocate memory for array list struct");
|
||||
log_perror("ERROR: Could not allocate memory for array list struct");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
@@ -47,7 +47,7 @@ static bool array_list_extend(ArrayList* array_list) {
|
||||
new_capacity = INITIAL_ARRAY_SIZE;
|
||||
void* new_items = protocol_realloc(array_list->items, new_capacity * sizeof(void*));
|
||||
if (new_items == NULL) {
|
||||
log_message(LOG_LEVEL_ERROR, "%s", "ERROR: Could not reallocate memory for array list items");
|
||||
log_perror("ERROR: Could not reallocate memory for array list items");
|
||||
return false;
|
||||
}
|
||||
array_list->items = new_items;
|
||||
@@ -73,7 +73,7 @@ void** array_list_to_array(const ArrayList* array_list) {
|
||||
}
|
||||
void** array = protocol_alloc(array_list->size * sizeof(void*));
|
||||
if (array == NULL) {
|
||||
log_message(LOG_LEVEL_ERROR, "%s", "Could not malloc space for array from array list!");
|
||||
log_perror("Could not malloc space for array from array list!");
|
||||
return NULL;
|
||||
}
|
||||
memcpy(array, array_list->items, array_list->size * sizeof(void*));
|
||||
|
||||
+10
-15
@@ -168,14 +168,9 @@ bool charset_spec_valid_direction(const char* from_charset, const char* to_chars
|
||||
return direction_probe_valid(from_charset, to_charset);
|
||||
}
|
||||
|
||||
/* The receiver's conversion is wire charset -> destination charset. rsync's
|
||||
* CONVERT_SPEC is LOCAL,REMOTE and "stays the same whether you're pushing or
|
||||
* pulling", so for a PUSH (FastSync's only direction) the destination end's
|
||||
* charset is the spec's REMOTE half: the client converts LOCAL -> REMOTE on the
|
||||
* sender and the receiver writes the wire bytes verbatim. Only a server that
|
||||
* declares its OWN --iconv (the daemon "charset" analog) has a different local
|
||||
* charset, and then it is that spec's LOCAL half and the receiver converts
|
||||
* wire -> server-local. A dedicated pre-ack check so an impossible direction is
|
||||
/* The receiver's real conversion is wire(client REMOTE) -> server-local (the
|
||||
* server's own --iconv LOCAL half, or the client's LOCAL half when the server
|
||||
* has no --iconv). A dedicated pre-ack check so an impossible direction is
|
||||
* rejected before the connection instead of refusing mid-transfer. */
|
||||
bool charset_wire_receiver_spec_valid(const char* spec, const char* server_spec) {
|
||||
if (!spec)
|
||||
@@ -185,7 +180,7 @@ bool charset_wire_receiver_spec_valid(const char* spec, const char* server_spec)
|
||||
if (charset_spec_parse(spec, &local, &remote) != 0)
|
||||
return false;
|
||||
const char* wire = remote;
|
||||
const char* target_local = remote;
|
||||
const char* target_local = local;
|
||||
char* server_local = NULL;
|
||||
char* server_remote = NULL;
|
||||
if (server_spec) {
|
||||
@@ -307,13 +302,13 @@ bool charset_wire_init_receiver(const char* spec, const char* server_spec) {
|
||||
char* remote;
|
||||
if (charset_spec_parse(spec, &local, &remote) != 0)
|
||||
return false;
|
||||
/* The wire charset is the client spec's REMOTE half (rsync's LOCAL,REMOTE
|
||||
* spec stays the same push or pull, so on a push the destination end's
|
||||
* charset is REMOTE and the receiver writes the wire bytes verbatim). Only a
|
||||
* server started with its own --iconv declares a different local charset (the
|
||||
* server halves above never travel), and then it is that spec's LOCAL half. */
|
||||
/* The wire charset is the client spec's REMOTE half; the local charset is
|
||||
* the client spec's LOCAL half unless the server was itself started with
|
||||
* --iconv naming a different local charset (the server halves above never
|
||||
* travel, so the server's own flag is the only way its local charset can
|
||||
* differ from what the client assumed). */
|
||||
const char* wire = remote;
|
||||
const char* target_local = remote;
|
||||
const char* target_local = local;
|
||||
char* server_local = NULL;
|
||||
char* server_remote = NULL;
|
||||
if (server_spec) {
|
||||
|
||||
@@ -57,9 +57,8 @@ void charset_conversion_close(void* conversion);
|
||||
/* Process-wide wire conversion. charset_wire_init_sender (client side) opens
|
||||
* LOCAL->REMOTE; charset_wire_init_receiver (server side) opens
|
||||
* wire(REMOTE)->server-local. server_spec is the server's own --iconv, whose
|
||||
* LOCAL half overrides the destination charset; NULL means the destination
|
||||
* charset is the client spec's REMOTE half (rsync's push semantics: the wire
|
||||
* bytes are written verbatim). Both return false on an unsupported spec.
|
||||
* LOCAL half may override the local charset the client assumed; NULL reuses
|
||||
* the client spec's LOCAL half. Both return false on an unsupported spec.
|
||||
* The state is freed with charset_wire_free. */
|
||||
bool charset_wire_init_sender(const char* spec);
|
||||
bool charset_wire_init_receiver(const char* spec, const char* server_spec);
|
||||
@@ -67,9 +66,9 @@ void charset_wire_free(void);
|
||||
bool charset_wire_active(void);
|
||||
|
||||
/* Pre-ack receiver-direction sanity (see charset_wire_init_receiver): true
|
||||
* when the exact wire->destination conversion the receiver will use (client
|
||||
* spec's REMOTE half into the server's own LOCAL half, or REMOTE->REMOTE when
|
||||
* the server has no --iconv) opens and produces NUL-free output. */
|
||||
* when the exact wire->server-local conversion the receiver will use (client
|
||||
* spec's REMOTE half into the server's own LOCAL half, or the client's LOCAL
|
||||
* half when the server has no --iconv) opens and produces NUL-free output. */
|
||||
bool charset_wire_receiver_spec_valid(const char* spec, const char* server_spec);
|
||||
|
||||
/* Convert a path across the wire in the process direction. Returns a malloc'd
|
||||
|
||||
+12
-53
@@ -1,5 +1,4 @@
|
||||
#include "checksum.h"
|
||||
#include "utils.h"
|
||||
#include <fcntl.h>
|
||||
#include <openssl/evp.h>
|
||||
#include <string.h>
|
||||
@@ -224,33 +223,17 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
||||
if (fd < 0)
|
||||
return false;
|
||||
|
||||
bool ok = checksum_digest_fd(algo, seed, fd, out, out_capacity, out_len);
|
||||
close(fd);
|
||||
return ok;
|
||||
}
|
||||
|
||||
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
|
||||
size_t* out_len) {
|
||||
if (fd < 0 || !out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
|
||||
return false;
|
||||
|
||||
if (algo == CHECKSUM_ALGO_NONE) {
|
||||
/* No checksum requested: nothing to read; an empty digest succeeds. */
|
||||
*out_len = 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
uint8_t buffer[64 * 1024];
|
||||
bool ok = false;
|
||||
lseek(fd, 0, SEEK_SET);
|
||||
|
||||
if (algo == CHECKSUM_ALGO_MD5 || algo == CHECKSUM_ALGO_SHA1) {
|
||||
const EVP_MD* md = algo == CHECKSUM_ALGO_MD5 ? EVP_md5() : EVP_sha1();
|
||||
if (algo == CHECKSUM_ALGO_MD5) {
|
||||
EVP_MD_CTX* ctx = EVP_MD_CTX_new();
|
||||
if (!ctx)
|
||||
if (!ctx) {
|
||||
close(fd);
|
||||
return false;
|
||||
}
|
||||
unsigned int digest_len = 0;
|
||||
if (EVP_DigestInit_ex(ctx, md, NULL) == 1) {
|
||||
if (EVP_DigestInit_ex(ctx, EVP_md5(), NULL) == 1) {
|
||||
ok = true;
|
||||
ssize_t got;
|
||||
while ((got = read(fd, buffer, sizeof(buffer))) > 0) {
|
||||
@@ -267,22 +250,7 @@ bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out,
|
||||
ok = false;
|
||||
}
|
||||
EVP_MD_CTX_free(ctx);
|
||||
return ok;
|
||||
}
|
||||
|
||||
if (algo == CHECKSUM_ALGO_MD4) {
|
||||
Md4Ctx ctx;
|
||||
md4_init(&ctx);
|
||||
ok = true;
|
||||
ssize_t got;
|
||||
while ((got = read(fd, buffer, sizeof(buffer))) > 0)
|
||||
md4_update(&ctx, buffer, (size_t)got);
|
||||
if (got < 0)
|
||||
ok = false;
|
||||
if (ok) {
|
||||
md4_final(&ctx, out);
|
||||
*out_len = 16;
|
||||
}
|
||||
close(fd);
|
||||
return ok;
|
||||
}
|
||||
|
||||
@@ -292,13 +260,16 @@ bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out,
|
||||
XXH64_reset(&xxh64, seed);
|
||||
} else if (algo == CHECKSUM_ALGO_XXH3 || algo == CHECKSUM_ALGO_XXH128) {
|
||||
xxh3 = XXH3_createState();
|
||||
if (!xxh3)
|
||||
if (!xxh3) {
|
||||
close(fd);
|
||||
return false;
|
||||
}
|
||||
if (algo == CHECKSUM_ALGO_XXH3)
|
||||
XXH3_64bits_reset_withSeed(xxh3, seed);
|
||||
else
|
||||
XXH3_128bits_reset_withSeed(xxh3, seed);
|
||||
} else {
|
||||
close(fd);
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -332,6 +303,7 @@ bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out,
|
||||
}
|
||||
if (xxh3)
|
||||
XXH3_freeState(xxh3);
|
||||
close(fd);
|
||||
return ok;
|
||||
}
|
||||
|
||||
@@ -399,7 +371,7 @@ uint8_t checksum_digest_len(ChecksumAlgo algo) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
static ChecksumAlgo compiled_checksum_preference_first(void) {
|
||||
ChecksumAlgo checksum_negotiate_default(void) {
|
||||
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
||||
* resolves to xxh128. */
|
||||
static const ChecksumAlgo preference[] = {
|
||||
@@ -412,16 +384,3 @@ static ChecksumAlgo compiled_checksum_preference_first(void) {
|
||||
}
|
||||
return CHECKSUM_ALGO_XXH64;
|
||||
}
|
||||
|
||||
int checksum_choice_resolve(void) {
|
||||
bool specified = false;
|
||||
int env = env_choice_first("RSYNC_CHECKSUM_LIST", checksum_algo_from_name, &specified);
|
||||
if (specified)
|
||||
return env; /* -1 = the list named no supported checksum */
|
||||
return (int)compiled_checksum_preference_first();
|
||||
}
|
||||
|
||||
ChecksumAlgo checksum_negotiate_default(void) {
|
||||
int resolved = checksum_choice_resolve();
|
||||
return resolved >= 0 ? (ChecksumAlgo)resolved : compiled_checksum_preference_first();
|
||||
}
|
||||
|
||||
@@ -51,13 +51,6 @@ bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t
|
||||
bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, uint8_t* out,
|
||||
size_t out_capacity, size_t* out_len);
|
||||
|
||||
/* Descriptor form of the streaming digest: rewinds `fd` to the start and hashes
|
||||
* to EOF without closing it. Used by the --verify-basis path to hash an
|
||||
* already-open, root-confined basis descriptor. Same contract as
|
||||
* checksum_digest_file. */
|
||||
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
|
||||
size_t* out_len);
|
||||
|
||||
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
|
||||
* Accepts "xxh64"/"xxhash", "xxh3", "xxh128", "md5", "md4", "sha1", "none".
|
||||
* "auto" is not an algorithm here; the caller resolves it to the negotiated
|
||||
@@ -79,11 +72,4 @@ uint8_t checksum_digest_len(ChecksumAlgo algo);
|
||||
* xxh128 xxh3 xxh64 md5 md4 sha1 none). Used to resolve "auto". */
|
||||
ChecksumAlgo checksum_negotiate_default(void);
|
||||
|
||||
/* Resolve "auto" the way rsync does: the first supported name in
|
||||
* RSYNC_CHECKSUM_LIST (whitespace-separated, client half ends at '&'), then the
|
||||
* compiled-in preference order when the variable is unset/blank. Returns -1
|
||||
* when the variable is set but names no supported checksum (rsync's failed
|
||||
* negotiation), otherwise a valid ChecksumAlgo id. */
|
||||
int checksum_choice_resolve(void);
|
||||
|
||||
#endif /* CHECKSUM_H */
|
||||
|
||||
+4
-5
@@ -17,9 +17,8 @@
|
||||
#include "protocol.h"
|
||||
#include "utils.h"
|
||||
|
||||
/* Maximum individual file data size within a chunk (64 MB). Distinct from the
|
||||
* receiver's whole-file MAX_FILE_DATA_SIZE (256 MB) in file_receive.c. */
|
||||
#define MAX_CHUNK_FILE_DATA_SIZE (64ULL * 1024 * 1024)
|
||||
/* Maximum individual file data size within a chunk (64 MB) */
|
||||
#define MAX_FILE_DATA_SIZE (64ULL * 1024 * 1024)
|
||||
#define MAX_FILES_PER_CHUNK 65536U
|
||||
|
||||
/* Reserve `charge` against `session`'s connection budget. This mirrors the
|
||||
@@ -383,9 +382,9 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
|
||||
}
|
||||
|
||||
// Reject individual file data larger than the maximum allowed size.
|
||||
if (file_data_size > MAX_CHUNK_FILE_DATA_SIZE) {
|
||||
if (file_data_size > MAX_FILE_DATA_SIZE) {
|
||||
log_message(LOG_LEVEL_ERROR, "File data size %zu exceeds maximum %llu", file_data_size,
|
||||
(unsigned long long)MAX_CHUNK_FILE_DATA_SIZE);
|
||||
(unsigned long long)MAX_FILE_DATA_SIZE);
|
||||
goto error;
|
||||
}
|
||||
|
||||
|
||||
+4
-439
@@ -2,8 +2,6 @@
|
||||
#include "data.h"
|
||||
#include "log.h"
|
||||
#include "protocol.h"
|
||||
#include "utils.h"
|
||||
#include <errno.h>
|
||||
#include <limits.h>
|
||||
#include <lz4.h>
|
||||
#include <stdatomic.h>
|
||||
@@ -17,14 +15,7 @@
|
||||
#include <zstd.h>
|
||||
|
||||
#define INITIAL_DECOMPRESS_BUF_SIZE (1024 * 1024)
|
||||
|
||||
/* Hard ceiling for a single decompression. The sender compresses whole files
|
||||
* up to the protocol's whole-file receive bound, so the decompressor must
|
||||
* accept payloads that large; referencing the protocol constant keeps the two
|
||||
* bounds from drifting apart (they previously did: a 100 MB ceiling rejected
|
||||
* 100-256 MB files). This remains a real bomb guard -- every allocation in the
|
||||
* paths below is clamped to it -- so it must not exceed the protocol bound. */
|
||||
#define MAX_DECOMPRESSED_SIZE MAX_RECEIVE_WHOLE_FILE_SIZE
|
||||
#define MAX_DECOMPRESSED_SIZE (100ULL * 1024 * 1024) /* 100 MB hard ceiling */
|
||||
|
||||
/* rsync 3.4.1's built-in skip-compress suffix list (the `--skip-compress`
|
||||
* defaults, in the man page's order). rsync stores it as space-separated
|
||||
@@ -128,7 +119,7 @@ bool compression_algo_enabled(CompressionAlgo algo) {
|
||||
return algo != COMPRESSION_ALGO_NONE;
|
||||
}
|
||||
|
||||
static CompressionAlgo compiled_preference_first(void) {
|
||||
CompressionAlgo compression_negotiate_default(void) {
|
||||
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
|
||||
* resolves to zstd. */
|
||||
static const CompressionAlgo preference[] = {
|
||||
@@ -142,57 +133,6 @@ static CompressionAlgo compiled_preference_first(void) {
|
||||
return COMPRESSION_ALGO_ZSTD;
|
||||
}
|
||||
|
||||
int compression_choice_resolve(void) {
|
||||
bool specified = false;
|
||||
int env = env_choice_first("RSYNC_COMPRESS_LIST", compression_algo_from_name, &specified);
|
||||
if (specified)
|
||||
return env; /* -1 = the list named no supported codec */
|
||||
return (int)compiled_preference_first();
|
||||
}
|
||||
|
||||
CompressionAlgo compression_negotiate_default(void) {
|
||||
int resolved = compression_choice_resolve();
|
||||
return resolved >= 0 ? (CompressionAlgo)resolved : compiled_preference_first();
|
||||
}
|
||||
|
||||
int compression_default_level(CompressionAlgo algo) {
|
||||
switch (algo) {
|
||||
case COMPRESSION_ALGO_ZSTD:
|
||||
return ZSTD_CLEVEL_DEFAULT;
|
||||
case COMPRESSION_ALGO_ZLIB:
|
||||
case COMPRESSION_ALGO_ZLIBX:
|
||||
return 6; /* rsync resolves zlib's Z_DEFAULT_COMPRESSION (-1) to 6 */
|
||||
case COMPRESSION_ALGO_LZ4:
|
||||
return 1; /* rsync lz4 level is 0/ignored; positive keeps the gate on */
|
||||
case COMPRESSION_ALGO_NONE:
|
||||
return 0;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int compression_clamp_level(CompressionAlgo algo, int level) {
|
||||
switch (algo) {
|
||||
case COMPRESSION_ALGO_ZSTD:
|
||||
if (level < 1)
|
||||
return 1;
|
||||
if (level > 22)
|
||||
return 22;
|
||||
return level;
|
||||
case COMPRESSION_ALGO_ZLIB:
|
||||
case COMPRESSION_ALGO_ZLIBX:
|
||||
if (level < 1)
|
||||
return 1;
|
||||
if (level > 9)
|
||||
return 9;
|
||||
return level;
|
||||
case COMPRESSION_ALGO_LZ4:
|
||||
return 1; /* ignored by lz4_compress; keeps the "compress" gate on */
|
||||
case COMPRESSION_ALGO_NONE:
|
||||
return 0;
|
||||
}
|
||||
return level;
|
||||
}
|
||||
|
||||
void compression_set_algo(CompressionAlgo algo) {
|
||||
if (compression_algo_valid((int)algo))
|
||||
atomic_store(&g_compression_algo, (int)algo);
|
||||
@@ -645,7 +585,8 @@ static Data* zstd_decompress(Data* compressed_data, size_t maximum_size) {
|
||||
}
|
||||
if (ret > 0 && output.pos == output.size) {
|
||||
if (buf_size >= hard_limit || buf_size > SIZE_MAX / 2) {
|
||||
log_message(LOG_LEVEL_ERROR, "Decompressed data exceeds %llu bytes", hard_limit);
|
||||
log_message(LOG_LEVEL_ERROR, "Decompressed data exceeds %llu bytes",
|
||||
(unsigned long long)MAX_DECOMPRESSED_SIZE);
|
||||
data_destroy(uncompressed_data);
|
||||
uncompressed_data = NULL;
|
||||
goto cleanup;
|
||||
@@ -717,379 +658,3 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
|
||||
Data* data_decompress(Data* compressed_data) {
|
||||
return data_decompress_limited(compressed_data, MAX_DECOMPRESSED_SIZE);
|
||||
}
|
||||
|
||||
/* ---- streaming decompression ---- */
|
||||
|
||||
#define STREAM_DECOMPRESS_OUT_CHUNK (256 * 1024)
|
||||
|
||||
struct CompressionStreamDecompressor {
|
||||
CompressionAlgo algo;
|
||||
unsigned long long expected_out;
|
||||
unsigned long long total;
|
||||
int out_fd;
|
||||
unsigned char* out_buf;
|
||||
ZSTD_DCtx* dctx;
|
||||
z_stream zs;
|
||||
bool zs_initialized;
|
||||
bool failed;
|
||||
};
|
||||
|
||||
static bool stream_write_all(int fd, const void* data, size_t size) {
|
||||
const unsigned char* p = data;
|
||||
size_t done = 0;
|
||||
while (done < size) {
|
||||
ssize_t n = write(fd, p + done, size - done);
|
||||
if (n < 0 && errno == EINTR)
|
||||
continue;
|
||||
if (n <= 0)
|
||||
return false;
|
||||
done += (size_t)n;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
CompressionStreamDecompressor*
|
||||
compression_stream_decompressor_create(CompressionAlgo algo, unsigned long long expected_out) {
|
||||
CompressionStreamDecompressor* d = calloc(1, sizeof(*d));
|
||||
if (!d)
|
||||
return NULL;
|
||||
d->algo = algo;
|
||||
d->expected_out = expected_out;
|
||||
d->out_fd = -1;
|
||||
d->out_buf = malloc(STREAM_DECOMPRESS_OUT_CHUNK);
|
||||
if (!d->out_buf) {
|
||||
free(d);
|
||||
return NULL;
|
||||
}
|
||||
if (algo == COMPRESSION_ALGO_ZSTD) {
|
||||
d->dctx = ZSTD_createDCtx();
|
||||
if (!d->dctx) {
|
||||
free(d->out_buf);
|
||||
free(d);
|
||||
return NULL;
|
||||
}
|
||||
} else if (algo == COMPRESSION_ALGO_ZLIB || algo == COMPRESSION_ALGO_ZLIBX) {
|
||||
if (inflateInit(&d->zs) != Z_OK) {
|
||||
free(d->out_buf);
|
||||
free(d);
|
||||
return NULL;
|
||||
}
|
||||
d->zs_initialized = true;
|
||||
} else if (algo != COMPRESSION_ALGO_NONE) {
|
||||
/* lz4's block format cannot be decompressed incrementally. */
|
||||
free(d->out_buf);
|
||||
free(d);
|
||||
return NULL;
|
||||
}
|
||||
return d;
|
||||
}
|
||||
|
||||
static bool stream_emit(CompressionStreamDecompressor* d, const void* buf, size_t len) {
|
||||
if (len == 0)
|
||||
return true;
|
||||
if (d->expected_out != 0 && (d->total > d->expected_out || len > d->expected_out - d->total)) {
|
||||
d->failed = true;
|
||||
return false;
|
||||
}
|
||||
if (!stream_write_all(d->out_fd, buf, len)) {
|
||||
d->failed = true;
|
||||
return false;
|
||||
}
|
||||
d->total += len;
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool stream_feed_none(CompressionStreamDecompressor* d, const void* in, size_t in_len,
|
||||
bool* done) {
|
||||
if (!stream_emit(d, in, in_len))
|
||||
return false;
|
||||
/* NONE has no end marker; the caller knows the frame length. */
|
||||
*done = true;
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool stream_feed_zstd(CompressionStreamDecompressor* d, const void* in, size_t in_len,
|
||||
bool* done) {
|
||||
ZSTD_inBuffer input = {in, in_len, 0};
|
||||
while (input.pos < input.size) {
|
||||
ZSTD_outBuffer output = {d->out_buf, STREAM_DECOMPRESS_OUT_CHUNK, 0};
|
||||
size_t ret = ZSTD_decompressStream(d->dctx, &output, &input);
|
||||
if (ZSTD_isError(ret)) {
|
||||
d->failed = true;
|
||||
return false;
|
||||
}
|
||||
if (!stream_emit(d, d->out_buf, output.pos))
|
||||
return false;
|
||||
if (ret == 0) {
|
||||
*done = true;
|
||||
/* Trailing bytes after a complete frame are malformed; stop consuming. */
|
||||
if (input.pos < input.size) {
|
||||
d->failed = true;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool stream_feed_zlib(CompressionStreamDecompressor* d, const void* in, size_t in_len,
|
||||
bool* done) {
|
||||
d->zs.next_in = (Bytef*)in;
|
||||
d->zs.avail_in = (uInt)in_len;
|
||||
while (d->zs.avail_in > 0) {
|
||||
d->zs.next_out = d->out_buf;
|
||||
d->zs.avail_out = STREAM_DECOMPRESS_OUT_CHUNK;
|
||||
int rc = inflate(&d->zs, Z_NO_FLUSH);
|
||||
if (rc != Z_OK && rc != Z_STREAM_END && rc != Z_BUF_ERROR) {
|
||||
d->failed = true;
|
||||
return false;
|
||||
}
|
||||
size_t produced = STREAM_DECOMPRESS_OUT_CHUNK - d->zs.avail_out;
|
||||
if (!stream_emit(d, d->out_buf, produced))
|
||||
return false;
|
||||
if (rc == Z_STREAM_END) {
|
||||
*done = true;
|
||||
return d->zs.avail_in == 0;
|
||||
}
|
||||
if (rc == Z_BUF_ERROR && produced == 0) {
|
||||
/* Need more input. */
|
||||
break;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool compression_stream_decompressor_feed(CompressionStreamDecompressor* d, const void* in,
|
||||
size_t in_len, int out_fd, bool* done) {
|
||||
if (!d || d->failed)
|
||||
return false;
|
||||
d->out_fd = out_fd;
|
||||
if (done)
|
||||
*done = false;
|
||||
switch (d->algo) {
|
||||
case COMPRESSION_ALGO_NONE:
|
||||
return stream_feed_none(d, in, in_len, done);
|
||||
case COMPRESSION_ALGO_ZSTD:
|
||||
return stream_feed_zstd(d, in, in_len, done);
|
||||
case COMPRESSION_ALGO_ZLIB:
|
||||
case COMPRESSION_ALGO_ZLIBX:
|
||||
return stream_feed_zlib(d, in, in_len, done);
|
||||
case COMPRESSION_ALGO_LZ4:
|
||||
break;
|
||||
}
|
||||
d->failed = true;
|
||||
return false;
|
||||
}
|
||||
|
||||
unsigned long long compression_stream_decompressor_total(const CompressionStreamDecompressor* d) {
|
||||
return d ? d->total : 0;
|
||||
}
|
||||
|
||||
void compression_stream_decompressor_destroy(CompressionStreamDecompressor* d) {
|
||||
if (!d)
|
||||
return;
|
||||
if (d->dctx)
|
||||
ZSTD_freeDCtx(d->dctx);
|
||||
if (d->zs_initialized)
|
||||
inflateEnd(&d->zs);
|
||||
free(d->out_buf);
|
||||
free(d);
|
||||
}
|
||||
|
||||
/* ---- streaming compression ---- */
|
||||
|
||||
struct CompressionStreamCompressor {
|
||||
CompressionAlgo algo;
|
||||
int level;
|
||||
ZSTD_CCtx* cctx;
|
||||
z_stream zs;
|
||||
bool zs_initialized;
|
||||
unsigned char* out_buf;
|
||||
bool failed;
|
||||
};
|
||||
|
||||
bool compression_stream_compress_supported(CompressionAlgo algo) {
|
||||
return algo == COMPRESSION_ALGO_ZSTD || algo == COMPRESSION_ALGO_ZLIB ||
|
||||
algo == COMPRESSION_ALGO_ZLIBX;
|
||||
}
|
||||
|
||||
CompressionStreamCompressor* compression_stream_compressor_create(CompressionAlgo algo, int level,
|
||||
int threads) {
|
||||
(void)threads;
|
||||
if (!compression_algo_valid((int)algo) || algo == COMPRESSION_ALGO_LZ4)
|
||||
return NULL;
|
||||
CompressionStreamCompressor* c = calloc(1, sizeof(*c));
|
||||
if (!c)
|
||||
return NULL;
|
||||
c->algo = algo;
|
||||
c->level = level;
|
||||
c->out_buf = malloc(STREAM_DECOMPRESS_OUT_CHUNK);
|
||||
if (!c->out_buf) {
|
||||
free(c);
|
||||
return NULL;
|
||||
}
|
||||
if (algo == COMPRESSION_ALGO_ZSTD) {
|
||||
c->cctx = ZSTD_createCCtx();
|
||||
if (!c->cctx) {
|
||||
free(c->out_buf);
|
||||
free(c);
|
||||
return NULL;
|
||||
}
|
||||
} else if (algo == COMPRESSION_ALGO_ZLIB || algo == COMPRESSION_ALGO_ZLIBX) {
|
||||
if (deflateInit(&c->zs, level < 1 ? Z_DEFAULT_COMPRESSION : level) != Z_OK) {
|
||||
free(c->out_buf);
|
||||
free(c);
|
||||
return NULL;
|
||||
}
|
||||
c->zs_initialized = true;
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
bool compression_stream_compressor_begin(CompressionStreamCompressor* c,
|
||||
unsigned long long raw_size, int out_fd) {
|
||||
if (!c || c->failed)
|
||||
return false;
|
||||
unsigned char hdr[1 + 4];
|
||||
size_t hdr_len = 1;
|
||||
hdr[0] = (unsigned char)c->algo;
|
||||
if (c->algo == COMPRESSION_ALGO_ZLIB || c->algo == COMPRESSION_ALGO_ZLIBX) {
|
||||
uint32_t size32 = raw_size > UINT32_MAX ? UINT32_MAX : (uint32_t)raw_size;
|
||||
for (int i = 0; i < 4; i++)
|
||||
hdr[1 + i] = (uint8_t)((size32 >> (8 * i)) & 0xff);
|
||||
hdr_len = 5;
|
||||
}
|
||||
if (c->algo == COMPRESSION_ALGO_ZSTD) {
|
||||
/* Pledge the source size and force the frame content-size field so the
|
||||
receiver can decide whether to stream from the frame header alone. */
|
||||
if (ZSTD_isError(ZSTD_CCtx_setPledgedSrcSize(c->cctx, raw_size)) ||
|
||||
ZSTD_isError(ZSTD_CCtx_setParameter(c->cctx, ZSTD_c_compressionLevel, c->level)) ||
|
||||
ZSTD_isError(ZSTD_CCtx_setParameter(c->cctx, ZSTD_c_contentSizeFlag, 1))) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
if (ZSTD_isError(ZSTD_CCtx_setParameter(c->cctx, ZSTD_c_checksumFlag, 0))) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
if (!stream_write_all(out_fd, hdr, hdr_len)) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool stream_compress_zlib(CompressionStreamCompressor* c, const void* in, size_t in_len,
|
||||
int out_fd, int flush) {
|
||||
c->zs.next_in = (Bytef*)in;
|
||||
c->zs.avail_in = (uInt)in_len;
|
||||
do {
|
||||
c->zs.next_out = c->out_buf;
|
||||
c->zs.avail_out = STREAM_DECOMPRESS_OUT_CHUNK;
|
||||
int rc = deflate(&c->zs, flush);
|
||||
if (rc != Z_OK && rc != Z_STREAM_END && rc != Z_BUF_ERROR) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
size_t produced = STREAM_DECOMPRESS_OUT_CHUNK - c->zs.avail_out;
|
||||
if (!stream_write_all(out_fd, c->out_buf, produced)) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
if (rc == Z_STREAM_END)
|
||||
return true;
|
||||
if (rc == Z_BUF_ERROR && produced == 0)
|
||||
break;
|
||||
} while (c->zs.avail_in > 0 || flush == Z_FINISH);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool compression_stream_compressor_feed(CompressionStreamCompressor* c, const void* in,
|
||||
size_t in_len, int out_fd) {
|
||||
if (!c || c->failed)
|
||||
return false;
|
||||
if (c->algo == COMPRESSION_ALGO_NONE)
|
||||
return stream_write_all(out_fd, in, in_len);
|
||||
if (c->algo == COMPRESSION_ALGO_ZSTD) {
|
||||
ZSTD_inBuffer input = {in, in_len, 0};
|
||||
while (input.pos < input.size) {
|
||||
ZSTD_outBuffer output = {c->out_buf, STREAM_DECOMPRESS_OUT_CHUNK, 0};
|
||||
size_t ret = ZSTD_compressStream2(c->cctx, &output, &input, ZSTD_e_continue);
|
||||
if (ZSTD_isError(ret)) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
if (!stream_write_all(out_fd, c->out_buf, output.pos)) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
if (output.pos == 0 && input.pos < input.size)
|
||||
break; /* avoid spinning; zstd buffers the rest internally */
|
||||
}
|
||||
return true;
|
||||
}
|
||||
return stream_compress_zlib(c, in, in_len, out_fd, Z_NO_FLUSH);
|
||||
}
|
||||
|
||||
bool compression_stream_compressor_finish(CompressionStreamCompressor* c, int out_fd) {
|
||||
if (!c || c->failed)
|
||||
return false;
|
||||
if (c->algo == COMPRESSION_ALGO_NONE)
|
||||
return true;
|
||||
if (c->algo == COMPRESSION_ALGO_ZSTD) {
|
||||
size_t ret;
|
||||
do {
|
||||
ZSTD_inBuffer input = {NULL, 0, 0};
|
||||
ZSTD_outBuffer output = {c->out_buf, STREAM_DECOMPRESS_OUT_CHUNK, 0};
|
||||
ret = ZSTD_compressStream2(c->cctx, &output, &input, ZSTD_e_end);
|
||||
if (ZSTD_isError(ret)) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
if (!stream_write_all(out_fd, c->out_buf, output.pos)) {
|
||||
c->failed = true;
|
||||
return false;
|
||||
}
|
||||
} while (ret > 0);
|
||||
return true;
|
||||
}
|
||||
return stream_compress_zlib(c, NULL, 0, out_fd, Z_FINISH);
|
||||
}
|
||||
|
||||
void compression_stream_compressor_destroy(CompressionStreamCompressor* c) {
|
||||
if (!c)
|
||||
return;
|
||||
if (c->cctx)
|
||||
ZSTD_freeCCtx(c->cctx);
|
||||
if (c->zs_initialized)
|
||||
deflateEnd(&c->zs);
|
||||
free(c->out_buf);
|
||||
free(c);
|
||||
}
|
||||
|
||||
unsigned long long compression_peek_frame_content_size(const void* buf, size_t len) {
|
||||
if (!buf || len < 1)
|
||||
return 0;
|
||||
const uint8_t* p = buf;
|
||||
uint8_t codec = p[0];
|
||||
if (!compression_algo_valid(codec))
|
||||
return 0;
|
||||
if (codec == (uint8_t)COMPRESSION_ALGO_NONE)
|
||||
return len - 1;
|
||||
if (codec == (uint8_t)COMPRESSION_ALGO_ZSTD) {
|
||||
if (len < 2)
|
||||
return 0;
|
||||
unsigned long long size = ZSTD_getFrameContentSize(p + 1, len - 1);
|
||||
if (size == ZSTD_CONTENTSIZE_ERROR || size == ZSTD_CONTENTSIZE_UNKNOWN)
|
||||
return 0;
|
||||
return size;
|
||||
}
|
||||
if (len < 1 + LZ4_SIZE_PREFIX_LEN)
|
||||
return 0;
|
||||
uint32_t raw = 0;
|
||||
for (int i = 0; i < LZ4_SIZE_PREFIX_LEN; i++)
|
||||
raw |= (uint32_t)p[1 + i] << (8 * i);
|
||||
return raw;
|
||||
}
|
||||
|
||||
@@ -35,26 +35,6 @@ bool compression_algo_valid(int algo);
|
||||
* "auto". */
|
||||
CompressionAlgo compression_negotiate_default(void);
|
||||
|
||||
/* Resolve "auto" the way rsync does: the first supported name in
|
||||
* RSYNC_COMPRESS_LIST (whitespace-separated, client half ends at '&'), then the
|
||||
* compiled-in preference order when the variable is unset/blank. Returns -1
|
||||
* when the variable is set but names no supported codec (rsync's failed
|
||||
* negotiation), otherwise a valid CompressionAlgo id. */
|
||||
int compression_choice_resolve(void);
|
||||
|
||||
/* rsync 3.4.1's per-codec default level, applied when the user did not pass
|
||||
* --compress-level/--zl. zstd uses ZSTD_CLEVEL_DEFAULT (3) and zlib/zlibx the
|
||||
* resolved Z_DEFAULT_COMPRESSION (6). lz4 has no tunable level in rsync
|
||||
* (always the default acceleration); FastSync returns a positive placeholder so
|
||||
* its "level > 0" compression gate stays engaged, and lz4_compress ignores the
|
||||
* value, so the output is identical to rsync's. none is 0. */
|
||||
int compression_default_level(CompressionAlgo algo);
|
||||
|
||||
/* Clamp an explicit --compress-level to the codec's accepted range the way
|
||||
* rsync's init_compression_level() does: zstd 1..22, zlib/zlibx 1..9, lz4
|
||||
* ignored (fixed positive placeholder), none 0. */
|
||||
int compression_clamp_level(CompressionAlgo algo, int level);
|
||||
|
||||
/* True when the algorithm actually compresses (i.e. is not NONE). */
|
||||
bool compression_algo_enabled(CompressionAlgo algo);
|
||||
|
||||
@@ -81,54 +61,6 @@ Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
|
||||
Data* data_decompress(Data* compressed_data);
|
||||
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count);
|
||||
|
||||
/* Streaming decompression for a payload too large to hold in memory. The
|
||||
* caller consumes the frame's leading codec byte (and, for lz4/zlib/zlibx, the
|
||||
* 4-byte little-endian raw-size prefix) and then feeds the remaining frame
|
||||
* bytes in bounded chunks; decompressed output is written straight to `out_fd`
|
||||
* so neither the compressed nor the decompressed image is ever materialized.
|
||||
* Only zstd (the default), zlib/zlibx and none support streaming; lz4's block
|
||||
* format is one-shot, so its stream decompressor reports failure and the caller
|
||||
* falls back (the whole-buffer path keeps its existing bound). */
|
||||
typedef struct CompressionStreamDecompressor CompressionStreamDecompressor;
|
||||
|
||||
CompressionStreamDecompressor*
|
||||
compression_stream_decompressor_create(CompressionAlgo algo, unsigned long long expected_out);
|
||||
/* Feed one chunk. Returns false on a malformed frame, an I/O error, or when the
|
||||
* total output would exceed `expected_out` (when non-zero). *done is set once
|
||||
* the frame end has been reached. */
|
||||
bool compression_stream_decompressor_feed(CompressionStreamDecompressor* d, const void* in,
|
||||
size_t in_len, int out_fd, bool* done);
|
||||
unsigned long long compression_stream_decompressor_total(const CompressionStreamDecompressor* d);
|
||||
void compression_stream_decompressor_destroy(CompressionStreamDecompressor* d);
|
||||
|
||||
/* Peek the logical (decompressed) size from the leading bytes of a compressed
|
||||
* frame (codec byte + header), returning 0 when it cannot be determined from
|
||||
* the supplied prefix. Used to decide whether a frame must take the streaming
|
||||
* path before its body is read. */
|
||||
unsigned long long compression_peek_frame_content_size(const void* buf, size_t len);
|
||||
|
||||
/* Streaming compression (sender side). Compresses a source in bounded chunks
|
||||
* into `out_fd` as one self-describing frame (codec byte, the lz4/zlib raw-size
|
||||
* prefix, then the codec stream), so a whole file can be compressed without
|
||||
* materializing it in memory. zstd/zlib/zlibx/none are supported; lz4's block
|
||||
* format is one-shot, so its create() returns NULL and the caller keeps the
|
||||
* buffered path. `raw_size` is the known source length (used for the zlib
|
||||
* prefix and, for zstd, the frame content-size field). */
|
||||
typedef struct CompressionStreamCompressor CompressionStreamCompressor;
|
||||
|
||||
/* True when `algo` can be stream-compressed (zstd/zlib/zlibx; lz4's block format
|
||||
* is one-shot). Used by the sender to decide whether an over-threshold source
|
||||
* may stay unloaded. */
|
||||
bool compression_stream_compress_supported(CompressionAlgo algo);
|
||||
CompressionStreamCompressor* compression_stream_compressor_create(CompressionAlgo algo, int level,
|
||||
int threads);
|
||||
bool compression_stream_compressor_begin(CompressionStreamCompressor* c,
|
||||
unsigned long long raw_size, int out_fd);
|
||||
bool compression_stream_compressor_feed(CompressionStreamCompressor* c, const void* in,
|
||||
size_t in_len, int out_fd);
|
||||
bool compression_stream_compressor_finish(CompressionStreamCompressor* c, int out_fd);
|
||||
void compression_stream_compressor_destroy(CompressionStreamCompressor* c);
|
||||
|
||||
/* Release the calling thread's cached zstd contexts (compressor, decompressor
|
||||
* and scratch buffer). The cache is thread-local and is also released
|
||||
* automatically when a worker thread exits (via a C11 tss destructor) and for
|
||||
|
||||
+16
-169
@@ -20,9 +20,9 @@
|
||||
|
||||
static void config_set_defaults(Config* config) {
|
||||
config->scanner_threads = 0;
|
||||
config->cli.preserve_perms_explicit_off = false;
|
||||
config->cli.preserve_times_explicit_off = false;
|
||||
config->cli.metadata_explicitly_disabled = false;
|
||||
config->metadata_explicitly_disabled = false;
|
||||
config->preserve_perms_explicit_off = false;
|
||||
config->preserve_times_explicit_off = false;
|
||||
config->show_progress = false;
|
||||
config->compression_threads = 0;
|
||||
config->ssh_port = 22;
|
||||
@@ -44,8 +44,8 @@ static void config_set_defaults(Config* config) {
|
||||
config->tls_ca = NULL;
|
||||
config->server_host = str_dup("127.0.0.1");
|
||||
config->server_port = 8080;
|
||||
config->cli.server_port_set = false;
|
||||
config->cli.server_host_set = false;
|
||||
config->server_port_set = false;
|
||||
config->server_host_set = false;
|
||||
/* rsync defaults: --timeout=0 (I/O timeouts disabled) and --contimeout=60.
|
||||
* A value of 0 disables the client's own deadline on both the socket layer
|
||||
* (tcp_set_timeouts) and the protocol layer
|
||||
@@ -68,10 +68,8 @@ static void config_set_defaults(Config* config) {
|
||||
config->human_readable = false;
|
||||
config->ignore_errors = false;
|
||||
config->ignore_missing_args = false;
|
||||
config->cli.checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
|
||||
config->cli.cli_exit_code = 0;
|
||||
config->cli.compression_level_set = false;
|
||||
config->cli.checksum_choice_set = false;
|
||||
config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
|
||||
config->cli_exit_code = 0;
|
||||
config->filters = NULL;
|
||||
config->files_from = NULL;
|
||||
config->files_from_set = NULL;
|
||||
@@ -79,12 +77,13 @@ static void config_set_defaults(Config* config) {
|
||||
config->cvs_exclude = false;
|
||||
config->per_dir_filter = false;
|
||||
config->per_dir_filter_count = 0;
|
||||
config->one_file_system = 0;
|
||||
config->one_file_system = false;
|
||||
config->no_implied_dirs = false;
|
||||
config->dirs = false;
|
||||
config->rsh_command = NULL;
|
||||
config->blocking_io = false;
|
||||
config->outbuf = OUTBUF_BLOCK;
|
||||
config->old_args = false;
|
||||
config->remote_options = NULL;
|
||||
config->remote_option_count = 0;
|
||||
config->address = NULL;
|
||||
@@ -100,7 +99,7 @@ static void config_set_defaults(Config* config) {
|
||||
config->trust_sender = false;
|
||||
config->stop_after_mins = 0;
|
||||
config->stop_at = 0;
|
||||
config->cli.stop_at_set = false;
|
||||
config->stop_at_set = false;
|
||||
config->write_batch = NULL;
|
||||
config->only_write_batch = NULL;
|
||||
config->read_batch = NULL;
|
||||
@@ -207,8 +206,7 @@ static bool validate_received_config(const Config* config) {
|
||||
valid_wire_bool(config->preserve_perms) && valid_wire_bool(config->preserve_times) &&
|
||||
valid_wire_bool(config->preserve_owner) && valid_wire_bool(config->preserve_group) &&
|
||||
valid_wire_bool(config->munge_links) && valid_wire_bool(config->keep_dirlinks) &&
|
||||
valid_wire_bool(config->fake_super) && valid_wire_bool(config->report_dest_info) &&
|
||||
valid_wire_bool(config->report_stats) && valid_wire_bool(config->report_deletes) &&
|
||||
valid_wire_bool(config->fake_super) &&
|
||||
(!config->copy_as_set || (config->copy_as_uid >= 0 && config->copy_as_gid >= 0)) &&
|
||||
(!config->use_compression ||
|
||||
(config->compression_level >= 1 && config->compression_level <= 22)) &&
|
||||
@@ -229,13 +227,6 @@ Config* config_create(void) {
|
||||
if (!config)
|
||||
return NULL;
|
||||
config_set_defaults(config);
|
||||
/* config_set_defaults() dups the default server host; a failure there leaves
|
||||
* server_host NULL and would crash later consumers, so fail the whole create
|
||||
* (every caller already handles a NULL return). */
|
||||
if (!config->server_host) {
|
||||
config_delete(config);
|
||||
return NULL;
|
||||
}
|
||||
return config;
|
||||
}
|
||||
|
||||
@@ -341,8 +332,7 @@ bool config_derived_use_metadata(const Config* config) {
|
||||
config->chown_uid_set || config->chown_gid_set || config->usermap_count > 0 ||
|
||||
config->groupmap_count > 0 || config->update)
|
||||
return true;
|
||||
return (config->use_incremental || config->use_delta) &&
|
||||
!config->cli.metadata_explicitly_disabled;
|
||||
return (config->use_incremental || config->use_delta) && !config->metadata_explicitly_disabled;
|
||||
}
|
||||
|
||||
bool config_has_basis(const Config* config) {
|
||||
@@ -693,15 +683,9 @@ int config_parse_ssh_dest(Config* config) {
|
||||
return daemon_dest_parse_error("invalid remote destination user@host (must not be empty or "
|
||||
"start with '-')",
|
||||
dest);
|
||||
char* ssh_destination = str_dup(dest);
|
||||
char* path = str_dup(colon + 1);
|
||||
if (!ssh_destination || !path) {
|
||||
free(ssh_destination);
|
||||
free(path);
|
||||
return daemon_dest_parse_error("out of memory parsing remote destination", dest);
|
||||
}
|
||||
config->transport = TRANSPORT_SSH;
|
||||
config->ssh_destination = ssh_destination;
|
||||
config->ssh_destination = str_dup(dest);
|
||||
char* path = str_dup(colon + 1);
|
||||
free(config->receive_root_directory);
|
||||
config->receive_root_directory = path;
|
||||
return 0;
|
||||
@@ -804,8 +788,6 @@ void config_delete(Config* config) {
|
||||
if (config->filters) {
|
||||
array_list_delete(config->filters);
|
||||
}
|
||||
filter_rule_list_free(config->protect_rules);
|
||||
config->protect_rules = NULL;
|
||||
/* A --delay-updates staging tree is transient receiver state: remove any
|
||||
leftovers on every exit path (success already emptied it). */
|
||||
if (config->delay_context)
|
||||
@@ -1041,134 +1023,6 @@ static bool receive_basis_entries(int fd, Config* c, ConfigStringBudget* budget)
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Receiver-side delete-protection rules (protocol 2.28.0). The sender compiles
|
||||
* its command-line selection rules exactly as the scanner does and streams the
|
||||
* result as one bounded, self-describing block (count + per-rule records); the
|
||||
* receiver reconstructs a FilterRuleList for the --delete extras walk. owner
|
||||
* and pattern are charged through the shared ConfigStringBudget and the block
|
||||
* additionally enforces MAX_FILTER_RULES / MAX_FILTER_BYTES. */
|
||||
static bool send_protect_entries(int fd, const Config* c) {
|
||||
int count = c->filters ? c->filters->size : 0;
|
||||
const char** texts = NULL;
|
||||
if (count > 0) {
|
||||
texts = malloc((size_t)count * sizeof(char*));
|
||||
if (!texts)
|
||||
return false;
|
||||
for (int i = 0; i < count; i++)
|
||||
texts[i] = (const char*)c->filters->items[i];
|
||||
}
|
||||
char err[160];
|
||||
FilterRuleList* rules =
|
||||
filter_base_build(texts, count, c->cvs_exclude, c->delete_excluded, err, sizeof(err));
|
||||
free(texts);
|
||||
if (!rules) {
|
||||
log_message(LOG_LEVEL_ERROR, "invalid filter rule: %s", err);
|
||||
return false;
|
||||
}
|
||||
/* The receiver rejects any block with more than MAX_FILTER_RULES entries as a
|
||||
* protocol error; refuse to emit such a frame at all. filter_base_build()
|
||||
* can expand the client rule set (cvs-exclude, merge files), so this is the
|
||||
* authoritative bound, not config->filters->size. */
|
||||
if (rules->count < 0 || rules->count > MAX_FILTER_RULES) {
|
||||
log_message(LOG_LEVEL_ERROR, "too many filter rules: %d (maximum %d)", rules->count,
|
||||
MAX_FILTER_RULES);
|
||||
filter_rule_list_free(rules);
|
||||
return false;
|
||||
}
|
||||
bool ok = send_int(fd, rules->count);
|
||||
for (int i = 0; ok && i < rules->count; i++) {
|
||||
const FilterRule* r = rules->items[i];
|
||||
/* Mirror the receiver's limit so the peer never receives a rule it will
|
||||
reject as a protocol error. */
|
||||
if (r->pattern && strlen(r->pattern) > MAX_PROTECT_PATTERN_LEN) {
|
||||
log_message(LOG_LEVEL_ERROR, "filter pattern exceeds %d bytes", MAX_PROTECT_PATTERN_LEN);
|
||||
filter_rule_list_free(rules);
|
||||
return false;
|
||||
}
|
||||
ok = send_int(fd, (int)r->action) && send_int(fd, (int)r->sides) &&
|
||||
send_int(fd, r->anchored ? 1 : 0) && send_int(fd, r->dir_only ? 1 : 0) &&
|
||||
send_int(fd, r->negate ? 1 : 0) && send_str(fd, r->owner ? r->owner : "") &&
|
||||
send_str(fd, r->pattern ? r->pattern : "");
|
||||
}
|
||||
filter_rule_list_free(rules);
|
||||
return ok;
|
||||
}
|
||||
|
||||
static bool receive_protect_entries(int fd, Config* c, ConfigStringBudget* budget) {
|
||||
int count;
|
||||
if (!receive_int(fd, &count))
|
||||
return false;
|
||||
if (count < 0 || count > MAX_FILTER_RULES)
|
||||
return false;
|
||||
if (count == 0)
|
||||
return true;
|
||||
FilterRuleList* list = filter_rule_list_create();
|
||||
if (!list)
|
||||
return false;
|
||||
size_t pattern_bytes = 0;
|
||||
for (int i = 0; i < count; i++) {
|
||||
int action;
|
||||
int sides;
|
||||
bool anchored;
|
||||
bool dir_only;
|
||||
bool negate;
|
||||
if (!receive_int(fd, &action) ||
|
||||
(action != FILTER_ACTION_EXCLUDE && action != FILTER_ACTION_INCLUDE) ||
|
||||
!receive_int(fd, &sides) || sides < (int)FILTER_SIDE_SENDER ||
|
||||
sides > (int)(FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER) ||
|
||||
!receive_wire_bool(fd, &anchored) || !receive_wire_bool(fd, &dir_only) ||
|
||||
!receive_wire_bool(fd, &negate))
|
||||
goto fail;
|
||||
char* owner = config_receive_str(fd, budget);
|
||||
if (!owner)
|
||||
goto fail;
|
||||
char* pattern = config_receive_str(fd, budget);
|
||||
if (!pattern || pattern[0] == '\0') {
|
||||
free(owner);
|
||||
free(pattern);
|
||||
goto fail;
|
||||
}
|
||||
/* A pattern too long to be evaluated by glob_match against a PATH_MAX path
|
||||
would silently fail to match and leave a protect rule inert (fail-open:
|
||||
the entry is then deleted). Reject it up front as a protocol error
|
||||
rather than accept a rule that can never shield anything. */
|
||||
if (strlen(pattern) > MAX_PROTECT_PATTERN_LEN) {
|
||||
free(owner);
|
||||
free(pattern);
|
||||
goto fail;
|
||||
}
|
||||
size_t bytes = strlen(owner) + strlen(pattern);
|
||||
if (bytes > MAX_FILTER_BYTES - pattern_bytes) {
|
||||
free(owner);
|
||||
free(pattern);
|
||||
goto fail;
|
||||
}
|
||||
pattern_bytes += bytes;
|
||||
FilterRule* rule = calloc(1, sizeof(FilterRule));
|
||||
if (!rule) {
|
||||
free(owner);
|
||||
free(pattern);
|
||||
goto fail;
|
||||
}
|
||||
rule->action = (FilterAction)action;
|
||||
rule->sides = (unsigned)sides;
|
||||
rule->anchored = anchored;
|
||||
rule->dir_only = dir_only;
|
||||
rule->negate = negate;
|
||||
rule->owner = owner;
|
||||
rule->pattern = pattern;
|
||||
if (!filter_rule_list_add(list, rule)) {
|
||||
filter_rule_free(rule);
|
||||
goto fail;
|
||||
}
|
||||
}
|
||||
c->protect_rules = list;
|
||||
return true;
|
||||
fail:
|
||||
filter_rule_list_free(list);
|
||||
return false;
|
||||
}
|
||||
|
||||
static bool send_identity_entries(int fd, const IdentityMap* map, int count) {
|
||||
for (int i = 0; i < count; i++) {
|
||||
if (!send_int(fd, map[i].from) || !send_int(fd, map[i].from_hi) || !send_int(fd, map[i].to) ||
|
||||
@@ -1296,9 +1150,6 @@ fail:
|
||||
#define CONFIG_RECV_BLOCK_IDMAP(name) \
|
||||
receive_identity_entries(fd, budget, c->name##_count, &c->name)
|
||||
|
||||
#define CONFIG_SEND_BLOCK_PROTECT_RULES(name) send_protect_entries(fd, c)
|
||||
#define CONFIG_RECV_BLOCK_PROTECT_RULES(name) receive_protect_entries(fd, c, budget)
|
||||
|
||||
/* One table entry, applied in sequence. XSEND/XRECV are statement macros so
|
||||
* consecutive entries read as a plain sequence of assignments. */
|
||||
#define XSEND(name, ctype, def, kind) ok = ok && (CONFIG_SEND_##kind(name));
|
||||
@@ -1336,7 +1187,6 @@ CONFIG_DEFINE_SEND(send_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
|
||||
CONFIG_DEFINE_SEND(send_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
||||
CONFIG_DEFINE_SEND(send_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
||||
CONFIG_DEFINE_SEND(send_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
||||
CONFIG_DEFINE_SEND(send_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
|
||||
|
||||
CONFIG_DEFINE_RECV(receive_core_fields, CONFIG_WIRE_CORE_FIELDS)
|
||||
CONFIG_DEFINE_RECV(receive_delta_fields, CONFIG_WIRE_DELTA_FIELDS)
|
||||
@@ -1357,7 +1207,6 @@ CONFIG_DEFINE_RECV(receive_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
|
||||
CONFIG_DEFINE_RECV(receive_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
||||
CONFIG_DEFINE_RECV(receive_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
||||
CONFIG_DEFINE_RECV(receive_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
||||
CONFIG_DEFINE_RECV(receive_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
|
||||
|
||||
#undef XSEND
|
||||
#undef XRECV
|
||||
@@ -1476,8 +1325,7 @@ bool config_send_wire_block(int file_descriptor, const Config* config) {
|
||||
send_privilege_options(file_descriptor, config) &&
|
||||
send_copy_as_options(file_descriptor, config) &&
|
||||
send_output_options(file_descriptor, config) &&
|
||||
send_codec_options(file_descriptor, config) &&
|
||||
send_protect_options(file_descriptor, config);
|
||||
send_codec_options(file_descriptor, config);
|
||||
}
|
||||
|
||||
bool config_send(int file_descriptor, const Config* config) {
|
||||
@@ -1549,8 +1397,7 @@ Config* config_receive_with_validate(int file_descriptor, ConfigValidateFunc val
|
||||
!receive_privilege_options(file_descriptor, config, &budget) ||
|
||||
!receive_copy_as_options(file_descriptor, config, &budget) ||
|
||||
!receive_output_options(file_descriptor, config, &budget) ||
|
||||
!receive_codec_options(file_descriptor, config, &budget) ||
|
||||
!receive_protect_options(file_descriptor, config, &budget))
|
||||
!receive_codec_options(file_descriptor, config, &budget))
|
||||
goto error;
|
||||
/* Validate/normalize the negotiated codec. compress_choice is the human
|
||||
* spelling (NULL or "" when -z was not given); compression_algo is the
|
||||
|
||||
+53
-180
@@ -4,7 +4,6 @@
|
||||
#include "array_list.h"
|
||||
#include "checksum.h"
|
||||
#include "compression.h"
|
||||
#include "filter.h"
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
@@ -83,7 +82,7 @@ typedef struct {
|
||||
typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF = 2 } SuperMode;
|
||||
|
||||
/* ===========================================================================
|
||||
* Config wire-field table (single source of truth for protocol 2.30.0).
|
||||
* Config wire-field table (single source of truth for protocol 2.26.0).
|
||||
*
|
||||
* Every field below crosses the wire. The table is the ONLY place a
|
||||
* serialized field is named: config.h expands CONFIG_WIRE_FIELDS() to declare
|
||||
@@ -198,18 +197,9 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
||||
X(skip_compress_count, int, 0, INT_SKIPCOUNT) \
|
||||
X(skip_compress_suffixes, char**, NULL, BLOCK_SKIP_SUFFIXES)
|
||||
|
||||
/* FastSync-only --verify-basis (protocol 2.28.0, no version bump by project
|
||||
* decision): restores the stricter content equality on a basis hit. By
|
||||
* default a basis hit is accepted on rsync's metadata quick-check alone (equal
|
||||
* size plus equal mtime, or size alone under --size-only); with this flag the
|
||||
* receiver ALSO requires the basis bytes' whole-file digest (the negotiated
|
||||
* --checksum-choice algorithm) to equal the sender's, exactly FastSync's
|
||||
* historical behavior. It is a receiver policy and crosses the wire so the
|
||||
* receiver knows whether to read and hash the basis content. */
|
||||
#define CONFIG_WIRE_BASIS_FIELDS(X) \
|
||||
X(basis_count, int, 0, INT_BASISCOUNT) \
|
||||
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS) \
|
||||
X(verify_basis, bool, false, BOOL)
|
||||
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS)
|
||||
|
||||
#define CONFIG_WIRE_FUZZY_FIELDS(X) X(fuzzy, bool, false, BOOL)
|
||||
|
||||
@@ -269,18 +259,9 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
||||
* for -n/--dry-run --delete, the destination-relative paths it WOULD have
|
||||
* deleted. It is set by the client only when --stats, --progress/-P, an
|
||||
* --out-format token needs a wire counter (%b/%c), or a dry-run carries
|
||||
* --delete; the transfer decision itself is unchanged.
|
||||
*
|
||||
* --info wave (protocol 2.27.0). report_deletes tells the receiver to include
|
||||
* the destination-relative paths it ACTUALLY removed in its terminal
|
||||
* STATUS_STATS record (the same path-list field the dry-run would-delete report
|
||||
* uses), so the sender can print rsync's `deleting PATH`/`*deleting` lines for a
|
||||
* real (non-dry-run) deletion. It is set when --delete is active and any of
|
||||
* --info=del, -i/--itemize-changes or --out-format requests per-file change
|
||||
* output; the transfer decision itself is unchanged. */
|
||||
* --delete; the transfer decision itself is unchanged. */
|
||||
#define CONFIG_WIRE_OUTPUT_FIELDS(X) \
|
||||
X(report_dest_info, bool, false, BOOL) \
|
||||
X(report_stats, bool, false, BOOL) X(report_deletes, bool, false, BOOL)
|
||||
X(report_dest_info, bool, false, BOOL) X(report_stats, bool, false, BOOL)
|
||||
|
||||
/* Codec-negotiation wave (protocol 2.26.0). compression_algo is the concrete
|
||||
* codec the client selected for this transfer (a CompressionAlgo id) and is the
|
||||
@@ -303,20 +284,6 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
||||
#define CONFIG_WIRE_CODEC_FIELDS(X) \
|
||||
X(compression_algo, int, COMPRESSION_ALGO_ZSTD, INT_COMPRESSION_ALGO)
|
||||
|
||||
/* Receiver-side delete-protection filter rules (protocol 2.28.0). The sender
|
||||
* compiles its root-level selection rules exactly as the scanner does
|
||||
* (filter_base_build over --filter/-f/--exclude/--include/-C) and streams them
|
||||
* as one self-describing, bounded block (count followed by per-rule records).
|
||||
* The receiver reconstructs `protect_rules` and evaluates them against
|
||||
* DESTINATION-ONLY entries during the --delete extras walk, so a
|
||||
* `protect`/`P` rule protects an extra that never appeared on the sender
|
||||
* (rsync re-derives deletion protection from the filter list; FastSync
|
||||
* historically derived it only from the source scan). `protect_rules` is NULL
|
||||
* on the sender and is owned/freed by the receiver Config. Bounded by
|
||||
* MAX_FILTER_RULES and MAX_FILTER_BYTES; an unknown action/sides is a protocol
|
||||
* error. */
|
||||
#define CONFIG_WIRE_PROTECT_FIELDS(X) X(protect_rules, FilterRuleList*, NULL, BLOCK_PROTECT_RULES)
|
||||
|
||||
/* All serialized fields, in exact wire order. Concatenating the per-segment
|
||||
* lists here is what keeps the declaration order = the wire order. */
|
||||
#define CONFIG_WIRE_FIELDS(X) \
|
||||
@@ -339,54 +306,7 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
||||
CONFIG_WIRE_PRIVILEGE_FIELDS(X) \
|
||||
CONFIG_WIRE_COPY_AS_FIELDS(X) \
|
||||
CONFIG_WIRE_OUTPUT_FIELDS(X) \
|
||||
CONFIG_WIRE_CODEC_FIELDS(X) \
|
||||
CONFIG_WIRE_PROTECT_FIELDS(X)
|
||||
|
||||
/* Client-only, CLI-parse bookkeeping (never serialized). These members exist
|
||||
* only so the client command-line parser can record HOW an option was
|
||||
* specified (explicitly set, explicitly negated, or a parser-requested exit
|
||||
* code); no other module and no wire peer ever needs them. Grouping them in
|
||||
* one nested member keeps the public Config free of client-CLI-only state. */
|
||||
typedef struct {
|
||||
/* Set when the user explicitly turned an attribute off with --no-perms /
|
||||
* --no-times (long or short form). --incremental/--delta historically
|
||||
* auto-enabled mode and mtime preservation; these flags let
|
||||
* cli_finalize_config restore that behavior while still honoring the
|
||||
* explicit per-attribute negation. A later -p/-t re-enables the attribute
|
||||
* directly, so the flag only prevents the incremental/delta implication,
|
||||
* never a POSITIVE request. */
|
||||
bool preserve_perms_explicit_off;
|
||||
bool preserve_times_explicit_off;
|
||||
/* Set by --no-preserve, the explicit opt-out of the whole preservation
|
||||
* bundle, so the --incremental/--delta auto-preserve implication stays off. */
|
||||
bool metadata_explicitly_disabled;
|
||||
/* True when --server-port/--port was explicitly given. --dry-run uses it to
|
||||
* decide whether a real server handshake was requested, so a plain local
|
||||
* destination (no explicit port) keeps the existing client-side dry-run
|
||||
* behavior instead of dialing the default 127.0.0.1:8080. */
|
||||
bool server_port_set;
|
||||
/* True when --server-host was explicitly given, and distinct from the
|
||||
* "127.0.0.1" default: --dry-run uses it to route an explicit remote target
|
||||
* to the server so it reports receiver state exactly like a real run,
|
||||
* instead of silently running the client-side manifest. */
|
||||
bool server_host_set;
|
||||
/* Codec-negotiation CLI state. The effective pre-transfer checksum is
|
||||
* Config->checksum_algo (serialized); checksum_transfer_algo is the rsync
|
||||
* "transfer" half of a two-name --checksum-choice form (validated and used
|
||||
* only to mirror rsync's whole-file forcing, since FastSync's per-block
|
||||
* strong hash is fixed). cli_exit_code carries a parser-requested process
|
||||
* exit status (rsync uses 4 for an unsupported checksum/compress algorithm)
|
||||
* so main() can mirror it. */
|
||||
int checksum_transfer_algo;
|
||||
int cli_exit_code;
|
||||
/* "The user explicitly chose" bits. They let the per-codec default level /
|
||||
* checksum list be applied only when the corresponding rsync option was
|
||||
* omitted (an explicit --compress-level / --checksum-choice always wins). */
|
||||
bool compression_level_set;
|
||||
bool checksum_choice_set;
|
||||
/* True when --stop-at was given. */
|
||||
bool stop_at_set;
|
||||
} ConfigCliParse;
|
||||
CONFIG_WIRE_CODEC_FIELDS(X)
|
||||
|
||||
typedef struct Config {
|
||||
/* -j/--threads=N: number of parallel scanner worker threads for the -m
|
||||
@@ -394,8 +314,16 @@ typedef struct Config {
|
||||
* scanner's built-in default" (4). CLIENT-ONLY: it is a local scheduling
|
||||
* concern and is NEVER serialized into the wire config frame. */
|
||||
int scanner_threads;
|
||||
/* Client-only CLI-parse bookkeeping (never serialized). See ConfigCliParse. */
|
||||
ConfigCliParse cli;
|
||||
bool metadata_explicitly_disabled;
|
||||
/* CLIENT-ONLY (never serialized; not in CONFIG_WIRE_FIELDS). Set when the
|
||||
* user explicitly turned an attribute off with --no-perms / --no-times (long
|
||||
* or short form). --incremental/--delta historically auto-enabled mode and
|
||||
* mtime preservation; these flags let cli_finalize_config restore that
|
||||
* behavior while still honoring the explicit per-attribute negation. A
|
||||
* later -p/-t re-enables the attribute directly, so the flag only prevents
|
||||
* the incremental/delta implication, never a POSITIVE request. */
|
||||
bool preserve_perms_explicit_off;
|
||||
bool preserve_times_explicit_off;
|
||||
bool show_progress;
|
||||
int compression_threads;
|
||||
int ssh_port;
|
||||
@@ -416,6 +344,18 @@ typedef struct Config {
|
||||
bool use_tls;
|
||||
char* server_host;
|
||||
int server_port;
|
||||
/* True when --server-port/--port was explicitly given. CLIENT-ONLY (never
|
||||
* serialized): --dry-run uses it to decide whether a real server handshake
|
||||
* was requested, so a plain local destination (no explicit port) keeps the
|
||||
* existing client-side dry-run behavior instead of dialing the default
|
||||
* 127.0.0.1:8080. */
|
||||
bool server_port_set;
|
||||
/* True when --server-host was explicitly given. CLIENT-ONLY (never
|
||||
* serialized), and distinct from the "127.0.0.1" default: --dry-run uses it
|
||||
* to route an explicit remote target to the server so it reports receiver
|
||||
* state exactly like a real run, instead of silently running the client-side
|
||||
* manifest. */
|
||||
bool server_host_set;
|
||||
char* tls_cert;
|
||||
char* tls_key;
|
||||
char* tls_ca;
|
||||
@@ -461,6 +401,16 @@ typedef struct Config {
|
||||
* enters the keep-set. Implied by --delete-missing-args. */
|
||||
bool ignore_missing_args;
|
||||
|
||||
/* Codec-negotiation CLI state (all client-only, never serialized). The
|
||||
* effective pre-transfer checksum is Config->checksum_algo (serialized);
|
||||
* checksum_transfer_algo is the rsync "transfer" half of a two-name
|
||||
* --checksum-choice form (validated and used only to mirror rsync's
|
||||
* whole-file forcing, since FastSync's per-block strong hash is fixed).
|
||||
* cli_exit_code carries a parser-requested process exit status (rsync uses 4
|
||||
* for an unsupported checksum/compress algorithm) so main() can mirror it. */
|
||||
int checksum_transfer_algo;
|
||||
int cli_exit_code;
|
||||
|
||||
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
|
||||
// never serialized to the wire (the receiver must not learn them).
|
||||
ArrayList* filters; /* --filter=RULE rule strings, in order */
|
||||
@@ -473,13 +423,9 @@ typedef struct Config {
|
||||
* /.rsync-filter' (the .rsync-filter files themselves are transferred); a
|
||||
* repeated -F adds --filter='- .rsync-filter' so they are excluded too. */
|
||||
int per_dir_filter_count;
|
||||
int one_file_system; /* -x/--one-file-system: do not cross filesystem boundaries.
|
||||
Repeated -x (rsync's -xx) drops the mount-point
|
||||
directory entirely instead of recreating it empty. */
|
||||
/* --no-implied-dirs: client-only. With -R, do not transfer the source
|
||||
* metadata of the parent directories implied by a listed path; an unlisted
|
||||
* implied parent is still created (with default attributes) so the listed
|
||||
* file can be placed, matching rsync. */
|
||||
bool one_file_system; /* -x/--one-file-system: do not cross filesystem boundaries */
|
||||
/* --no-implied-dirs: client-only. With -R + --files-from, refuse to place a
|
||||
* listed file whose ancestor directory is not itself explicitly listed. */
|
||||
bool no_implied_dirs;
|
||||
/* -d/--dirs: client-only. Transfer the directory entries named by the
|
||||
* source argument / --files-from list without recursing into contents. */
|
||||
@@ -496,6 +442,7 @@ typedef struct Config {
|
||||
/* --outbuf mode (OutbufMode): stdout/stderr buffering. Client-only launch
|
||||
* concern: NEVER crosses the wire. */
|
||||
int outbuf;
|
||||
bool old_args;
|
||||
/* --remote-option=OPT (Phase 5, long form only): one or more extra command-line
|
||||
* options to append to the REMOTE server invocation over SSH. CLIENT-ONLY:
|
||||
* they are composed into the remote command line by ssh_build_remote_command()
|
||||
@@ -565,6 +512,7 @@ typedef struct Config {
|
||||
* process and are NEVER serialized into the config frame. */
|
||||
int stop_after_mins; /* --stop-after=MINS minutes; 0 when unset */
|
||||
time_t stop_at; /* --stop-at=... absolute wall-clock deadline */
|
||||
bool stop_at_set; /* true when --stop-at was given */
|
||||
|
||||
/* Client-only residual-batch paths. A residual batch is a self-contained
|
||||
* single-file record of the whole source tree (full file images using the
|
||||
@@ -652,14 +600,10 @@ typedef struct Config {
|
||||
source directory is streamed in directory order, and the receiver removes
|
||||
each directory's extras when its plan arrives (during) or snapshots them
|
||||
and removes them only after a successful transfer (delay). delete_after
|
||||
keeps the whole-tree commit mode: extras are removed from a fresh
|
||||
end-of-transfer destination scan only after the whole transfer succeeded.
|
||||
A plain --delete with no explicit timing flag defaults to delete_during on
|
||||
the client (cli_finalize_config), matching rsync's --del default; the old
|
||||
late-commit behavior is selected explicitly by --delete-after or the
|
||||
FastSync-only long spelling --delete-commit (an exact alias for
|
||||
--delete-after, mapped onto the same wire field). See
|
||||
config_delete_timing_early()/config_delete_timing_per_dir() below. */
|
||||
(and plain --delete) keep the whole-tree commit mode: extras are removed
|
||||
from a fresh end-of-transfer destination scan only after the whole transfer
|
||||
succeeded. See config_delete_timing_early()/config_delete_timing_per_dir()
|
||||
below. */
|
||||
/* partial_dir */
|
||||
// PR #174: Partial transfer resumption
|
||||
/* suffix */
|
||||
@@ -735,13 +679,11 @@ typedef struct Config {
|
||||
* --copy-as) imply it. */
|
||||
/* fake_super */
|
||||
/* --fake-super: receiver-only. When set, each written file additionally gets
|
||||
* rsync's reserved user.rsync.%stat xattr recording the RESOLVED uid/gid (the
|
||||
* a reserved user.fastsync.stat xattr recording the RESOLVED uid/gid (the
|
||||
* source's own when no ownership request is active, else the --chown/--usermap
|
||||
* result) plus the full mode and rdev, in rsync 3.4.1's grammar, so the tree is
|
||||
* interoperable and a later privileged restore could re-apply them. mtime is
|
||||
* carried by the file's own timestamp, exactly as rsync does it. It NEVER
|
||||
* real-chowns: the point is to record the source ownership on an unprivileged
|
||||
* receiver. Crosses the wire. */
|
||||
* result) plus mode/mtime so a later privileged restore could re-apply them.
|
||||
* It NEVER real-chowns: the point is to record the source ownership on an
|
||||
* unprivileged receiver. Crosses the wire. */
|
||||
/* module */
|
||||
/* Daemon module selection (Wave A, protocol 2.15.0). Client-composed from a
|
||||
* host::module/path destination; NULL or "" means "no module" (the ordinary
|
||||
@@ -1046,77 +988,11 @@ typedef struct Config {
|
||||
* boundary, and the strict same-version handshake (config_receive rejects a
|
||||
* mismatched version before parsing anything else) keeps mixed deployments from
|
||||
* ever reaching that state. */
|
||||
/* (7) --info=del report (protocol 2.27.0): the config frame gains one trailing
|
||||
* bool, report_deletes, appended after report_stats. When set, the receiver
|
||||
* lists the paths it actually removed in the terminal STATUS_STATS path list
|
||||
* (the same count-delimited list the -n/--dry-run would-delete report uses), so
|
||||
* the sender can print rsync's `deleting PATH` lines for a real deletion. No
|
||||
* change to the fixed STATUS_STATS record itself; only a new trailing config
|
||||
* bool, which still requires the version bump for the strict lockstep. */
|
||||
/* (8) --stats receiver-observed counters (protocol 2.28.0): the fixed
|
||||
* STATUS_STATS record grows from three counters to eight. The receiver now
|
||||
* reports the bytes it literally stored (`literal_data`) and the count of
|
||||
* destination entries it newly CREATED, split by type
|
||||
* (reg/dir/link/special), so the sender can print rsync's exact
|
||||
* `Number of created files: N (reg: X, dir: Y, link: Z, special: W)` line and
|
||||
* an exact `Literal data` total even for delta transfers. The config-frame
|
||||
* LAYOUT is unchanged (no new config field), but the STATUS_STATS body grows,
|
||||
* so a 2.27 peer that does not consume the five new fixed-width counters would
|
||||
* desynchronize on the trailing would-delete path list; the strict
|
||||
* same-version handshake (config_receive rejects a mismatched version before
|
||||
* parsing anything else) keeps mixed deployments from ever reaching that
|
||||
* state. */
|
||||
/* (9) Receiver-side delete protection (still protocol 2.28.0): the config frame
|
||||
* gains one trailing self-describing block carrying the sender's compiled base
|
||||
* filter rules so the receiver can protect DESTINATION-ONLY entries from
|
||||
* --delete with `protect`/`risk` rules (rsync parity). The block appends after
|
||||
* compression_algo; see CONFIG_WIRE_PROTECT_FIELDS. */
|
||||
/* (10) Symlink xattrs/ACLs (protocol 2.29.0): the config-frame LAYOUT is
|
||||
* unchanged (the derived use_xattrs bit already crosses the wire), but the
|
||||
* STATUS_SYMLINK frame BODY grows a trailing bounded xattr block when -X/-A is
|
||||
* negotiated -- exactly the block STATUS_MKDIR, STATUS_DIR_TIMES and regular
|
||||
* files already carry. The sender captures the symlink's OWN xattrs with
|
||||
* llistxattr/lgetxattr (so it can never attach the REFERENT's attributes to the
|
||||
* link) and the receiver re-applies them to the link itself with lsetxattr on a
|
||||
* confined /proc/self/fd/<parent>/<leaf> path (there is no *at xattr syscall and
|
||||
* fsetxattr cannot target a symlink). A 2.28 peer that does not consume the new
|
||||
* trailing block would desynchronize after every symlink, so the protocol
|
||||
* version must bump; the strict same-version handshake (config_receive rejects a
|
||||
* mismatched version before parsing anything else) keeps a 2.29 client and a
|
||||
* 2.28 server from ever reaching that state. */
|
||||
/* (11) Client-message channel + partial exit (protocol 2.30.0): the
|
||||
* config-frame LAYOUT is unchanged (no new config field), but the frame stream
|
||||
* gains two statuses. STATUS_CLIENT_MSG (client->server) carries a bounded,
|
||||
* length-prefixed diagnostic string so a client running with --stderr=client
|
||||
* (rsync's --no-msgs2stderr spelling) can forward its own diagnostics to the
|
||||
* server's stderr. STATUS_PARTIAL (receiver->client) is the terminal status
|
||||
* sent instead of STATUS_OK when a per-entry receiver failure (e.g. an
|
||||
* unprivileged --devices mknod) did not abort the stream; the sender exits 23
|
||||
* (rsync's partial transfer) and still removes successfully transferred
|
||||
* --remove-source-files sources. A 2.29 peer that does not know these status
|
||||
* values would reject them as an unknown status and tear the connection down,
|
||||
* so the protocol version must bump; the strict same-version handshake keeps a
|
||||
* 2.30 client and a 2.29 server from ever reaching that state. */
|
||||
#define PROTOCOL_VERSION "2.30.0"
|
||||
#define PROTOCOL_VERSION "2.26.0"
|
||||
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
|
||||
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
|
||||
#define MAX_BASIS_DIRS 64
|
||||
|
||||
/* Bounds on the received receiver-side delete-protection rule block. The rule
|
||||
* count and the aggregate pattern+owner bytes are each capped so a hostile
|
||||
* peer cannot pin unbounded pre-auth memory; both are validated strictly on
|
||||
* receive (alongside the per-string ConfigStringBudget). */
|
||||
/* A peer may supply protect rules; cap the list so a crafted config cannot make
|
||||
* the receiver's delete walk evaluate an unbounded number of glob patterns per
|
||||
* destination entry (glob_match is O(pattern x path)). 1024 is far above any
|
||||
* legitimate selection. */
|
||||
#define MAX_FILTER_RULES 1024
|
||||
#define MAX_FILTER_BYTES (256 * 1024)
|
||||
/* glob_match's DP is capped at 64 Mi work units; a pattern longer than this
|
||||
* could exceed the cap against a PATH_MAX path and silently stop matching,
|
||||
* leaving a protect rule inert. Reject such a rule at receive time. */
|
||||
#define MAX_PROTECT_PATTERN_LEN 8192
|
||||
|
||||
/* Upper bound on the number of --skip-compress suffixes accepted from the wire.
|
||||
* Each suffix is an independent wire string (up to MAX_STRING_SIZE = 64 KiB), so
|
||||
* without this a hostile pre-auth client could otherwise retain
|
||||
@@ -1218,11 +1094,8 @@ bool config_delete_timing_early(const Config* config);
|
||||
* commits them only after a fully-successful transfer (delay). */
|
||||
bool config_delete_timing_per_dir(const Config* config);
|
||||
/* Delete-timing sanity: with deletion enabled at most one timing flag may be
|
||||
* set; without deletion no timing flag may be set (each timing flag implies
|
||||
* --delete). A plain --delete is normalized to delete_during by
|
||||
* cli_finalize_config on the client, so a transmitted use_delete config always
|
||||
* carries exactly one timing; the zero-timing case remains valid only for a
|
||||
* config that has not been through the CLI. */
|
||||
* set (none = the default delete-after commit timing); without deletion no
|
||||
* timing flag may be set (each timing flag implies --delete). */
|
||||
bool config_has_valid_delete_timing(const Config* config);
|
||||
|
||||
/* Single source of truth for the cross-field ("combination") invariants a
|
||||
|
||||
+18
-204
@@ -8,13 +8,12 @@
|
||||
#include <openssl/evp.h>
|
||||
#include <openssl/params.h>
|
||||
#include <openssl/rand.h>
|
||||
#include <poll.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
/* One store entry: a username and its salted PBKDF2 verifier. The plaintext
|
||||
@@ -59,37 +58,19 @@ struct CredentialStore {
|
||||
static const uint8_t k_dummy_stored_key[CREDENTIAL_KEY_LEN] = {0};
|
||||
static const uint8_t k_dummy_server_key[CREDENTIAL_KEY_LEN] = {0};
|
||||
|
||||
#define set_error utils_set_error
|
||||
static void set_error(char* err, size_t err_size, const char* fmt, ...) {
|
||||
if (!err || err_size == 0)
|
||||
return;
|
||||
va_list args;
|
||||
va_start(args, fmt);
|
||||
vsnprintf(err, err_size, fmt, args);
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
static bool is_comment_char(char c) {
|
||||
return c == '#' || c == ';';
|
||||
}
|
||||
|
||||
/* True for a literal fd-backed store path: exactly "/dev/fd/<digits>" or
|
||||
* "/proc/self/fd/<digits>", with no trailing component and no "..". These name
|
||||
* the calling process's own open descriptors (e.g. a bash process substitution
|
||||
* `<(...)`, which passes /dev/fd/N), and both prefixes are symlinks by
|
||||
* construction. */
|
||||
static bool is_fd_backed_path(const char* path) {
|
||||
static const char* const prefixes[] = {"/dev/fd/", "/proc/self/fd/"};
|
||||
if (!path)
|
||||
return false;
|
||||
for (size_t i = 0; i < sizeof(prefixes) / sizeof(prefixes[0]); i++) {
|
||||
const char* prefix = prefixes[i];
|
||||
size_t prefix_len = strlen(prefix);
|
||||
if (strncmp(path, prefix, prefix_len) != 0)
|
||||
continue;
|
||||
const char* digits = path + prefix_len;
|
||||
if (*digits < '0' || *digits > '9')
|
||||
return false;
|
||||
const char* p = digits;
|
||||
while (*p >= '0' && *p <= '9')
|
||||
p++;
|
||||
return *p == '\0';
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/* Open a --password-file / --early-input after verifying the EXACT inode we
|
||||
* will read: it must be owned by the effective user and grant no group/other
|
||||
* permission bit (so 0600 and stricter modes such as 0400 are accepted),
|
||||
@@ -99,31 +80,12 @@ static bool is_fd_backed_path(const char* path) {
|
||||
* path and then fstat the resulting fd (rather than stat()ing the path first
|
||||
* and reopening it), so the permission decision is made on the same inode that
|
||||
* is read and cannot be raced by swapping the path between check and open.
|
||||
* O_NOFOLLOW refuses a symlinked path outright (ELOOP fails closed) instead of
|
||||
* following it before the owner/mode gate can run. The one exception is a
|
||||
* literal fd-backed path (/dev/fd/N or /proc/self/fd/N, see
|
||||
* is_fd_backed_path): those entries are symlinks to the CALLING process's own
|
||||
* descriptors, so following them is not the untrusted-symlink hazard
|
||||
* O_NOFOLLOW guards against, and requiring O_NOFOLLOW would break the
|
||||
* documented process-substitution/FIFO usage. For them only, O_NOFOLLOW is
|
||||
* omitted; the same fstat owner/mode gate still applies to the resolved inode.
|
||||
* O_NONBLOCK keeps the OPEN itself from
|
||||
* blocking forever on a writer-less FIFO (a blocking O_RDONLY open would wait
|
||||
* for a writer). The fd is left nonblocking for FIFOs so a read never blocks
|
||||
* either; the read loop (secret_read_line) absorbs the resulting EAGAIN by
|
||||
* waiting, under a bounded deadline, for the writer -- this is what makes a
|
||||
* slow process substitution (`--password-file <(sleep 1; ...)`) work while a
|
||||
* writer-less FIFO still fails after the deadline instead of hanging. Only
|
||||
* regular files and FIFOs pass the ownership/mode checks; O_NONBLOCK is
|
||||
* cleared for regular files, where it is a no-op anyway and no EAGAIN can
|
||||
* occur, so their stdio read path is byte-for-byte unchanged.
|
||||
* The path may be a process-substitution pipe (`<(...)` -> /dev/fd/N), so
|
||||
* regular files and FIFOs are accepted when the ownership/mode checks pass.
|
||||
*
|
||||
* Returns a FILE* the caller must fclose, or NULL with `err` filled. */
|
||||
static FILE* secret_file_open(const char* path, char* err, size_t err_size) {
|
||||
int flags = O_RDONLY | O_NONBLOCK | O_CLOEXEC;
|
||||
if (!is_fd_backed_path(path))
|
||||
flags |= O_NOFOLLOW;
|
||||
int fd = open(path, flags);
|
||||
int fd = open(path, O_RDONLY | O_CLOEXEC);
|
||||
if (fd < 0) {
|
||||
set_error(err, err_size, "cannot open secret file '%s': %s", path, strerror(errno));
|
||||
return NULL;
|
||||
@@ -143,15 +105,6 @@ static FILE* secret_file_open(const char* path, char* err, size_t err_size) {
|
||||
close(fd);
|
||||
return NULL;
|
||||
}
|
||||
/* O_NONBLOCK is only meaningful for the FIFO allowance. Restore blocking
|
||||
* mode on a regular file so its read path is exactly as before; a no-op on
|
||||
* most systems, but explicit. Failures here are ignored: O_NONBLOCK on a
|
||||
* regular file does not affect reads either way. */
|
||||
if (S_ISREG(st.st_mode)) {
|
||||
int status_flags = fcntl(fd, F_GETFL);
|
||||
if (status_flags >= 0)
|
||||
(void)fcntl(fd, F_SETFL, status_flags & ~O_NONBLOCK);
|
||||
}
|
||||
FILE* fp = fdopen(fd, "r");
|
||||
if (!fp) {
|
||||
set_error(err, err_size, "cannot read secret file '%s': %s", path, strerror(errno));
|
||||
@@ -161,123 +114,6 @@ static FILE* secret_file_open(const char* path, char* err, size_t err_size) {
|
||||
return fp;
|
||||
}
|
||||
|
||||
/* Overall bound on how long the reader waits for a process-substitution/FIFO
|
||||
* writer to produce data before giving up. It must comfortably exceed a
|
||||
* producer's startup delay (e.g. `--password-file <(sleep 1; ...)`) while still
|
||||
* bounding a writer-less FIFO, so a stray or hostile FIFO cannot stall the
|
||||
* daemon or client indefinitely. */
|
||||
#define CREDENTIAL_FIFO_READ_TIMEOUT_MS 3000
|
||||
|
||||
/* Monotonic milliseconds, used only for the read deadline (wall-clock changes
|
||||
* must not extend or shorten the wait). */
|
||||
static int64_t credential_monotonic_ms(void) {
|
||||
struct timespec ts;
|
||||
if (clock_gettime(CLOCK_MONOTONIC, &ts) != 0)
|
||||
return 0;
|
||||
return (int64_t)ts.tv_sec * 1000 + (int64_t)(ts.tv_nsec / 1000000);
|
||||
}
|
||||
|
||||
/* Wait until `fd` is readable or the deadline passes. Returns true when it is
|
||||
* readable, false on timeout or a poll error (err filled). EINTR is retried
|
||||
* against the same deadline, so signals cannot extend the wait. */
|
||||
static bool credential_wait_readable(int fd, int64_t deadline, const char* label, const char* path,
|
||||
char* err, size_t err_size) {
|
||||
for (;;) {
|
||||
int64_t remaining = deadline - credential_monotonic_ms();
|
||||
if (remaining <= 0)
|
||||
break;
|
||||
if (remaining > INT_MAX)
|
||||
remaining = INT_MAX;
|
||||
struct pollfd pfd = {.fd = fd, .events = POLLIN, .revents = 0};
|
||||
int rc = poll(&pfd, 1, (int)remaining);
|
||||
if (rc > 0)
|
||||
return true;
|
||||
if (rc == 0)
|
||||
break;
|
||||
if (errno != EINTR) {
|
||||
set_error(err, err_size, "error waiting for %s '%s': %s", label, path, strerror(errno));
|
||||
return false;
|
||||
}
|
||||
}
|
||||
set_error(err, err_size, "timed out after %d ms waiting for %s '%s'",
|
||||
CREDENTIAL_FIFO_READ_TIMEOUT_MS, label, path);
|
||||
return false;
|
||||
}
|
||||
|
||||
typedef enum {
|
||||
SECRET_READ_LINE,
|
||||
SECRET_READ_EOF,
|
||||
SECRET_READ_ERROR,
|
||||
} SecretReadResult;
|
||||
|
||||
/* Read one complete line from `fp` into `line` (capacity `cap`), including the
|
||||
* trailing newline when present and always NUL-terminating. `*out_len`
|
||||
* receives strlen(line).
|
||||
*
|
||||
* A regular file is read exactly as before: secret_file_open leaves it
|
||||
* blocking, so fgets never sees EAGAIN. A FIFO stays nonblocking, so fgets
|
||||
* returns NULL (or a partial line) with EAGAIN while the writer is still
|
||||
* starting up; instead of treating that as a fatal error the loop clearerr()s
|
||||
* and polls for readability against one overall deadline. The `used`
|
||||
* accumulator reassembles a line that arrived in several write()s into a single
|
||||
* line, so a split write is not misparsed as two entries.
|
||||
*
|
||||
* Returns SECRET_READ_LINE, SECRET_READ_EOF, or SECRET_READ_ERROR (err filled)
|
||||
* on timeout or a genuine read error. */
|
||||
static SecretReadResult secret_read_line(char* line, size_t cap, FILE* fp, const char* label,
|
||||
const char* path, size_t* out_len, char* err,
|
||||
size_t err_size) {
|
||||
int fd = fileno(fp);
|
||||
int64_t deadline = credential_monotonic_ms() + CREDENTIAL_FIFO_READ_TIMEOUT_MS;
|
||||
size_t used = 0;
|
||||
line[0] = '\0';
|
||||
for (;;) {
|
||||
errno = 0;
|
||||
if (fgets(line + used, (int)(cap - used), fp)) {
|
||||
used += strlen(line + used);
|
||||
if (used > 0 && line[used - 1] == '\n') {
|
||||
*out_len = used;
|
||||
return SECRET_READ_LINE;
|
||||
}
|
||||
if (feof(fp)) {
|
||||
*out_len = used; /* final unterminated line */
|
||||
return SECRET_READ_LINE;
|
||||
}
|
||||
/* No newline and not EOF. A full buffer is the caller's over-long-line
|
||||
* case; otherwise the line is only partially available (a nonblocking
|
||||
* FIFO under a slow writer), so any genuine read error fails and anything
|
||||
* else waits for the rest. */
|
||||
if (used >= cap - 1) {
|
||||
*out_len = used;
|
||||
return SECRET_READ_LINE;
|
||||
}
|
||||
int e = ferror(fp) ? errno : 0;
|
||||
if (e != 0 && e != EAGAIN && e != EWOULDBLOCK) {
|
||||
set_error(err, err_size, "error reading %s '%s': %s", label, path, strerror(e));
|
||||
return SECRET_READ_ERROR;
|
||||
}
|
||||
clearerr(fp);
|
||||
if (!credential_wait_readable(fd, deadline, label, path, err, err_size))
|
||||
return SECRET_READ_ERROR;
|
||||
continue;
|
||||
}
|
||||
/* fgets returned NULL: EOF, a not-yet-readable FIFO, or a real error. */
|
||||
if (feof(fp)) {
|
||||
*out_len = used;
|
||||
return used > 0 ? SECRET_READ_LINE : SECRET_READ_EOF;
|
||||
}
|
||||
if (errno == EAGAIN || errno == EWOULDBLOCK) {
|
||||
clearerr(fp);
|
||||
if (!credential_wait_readable(fd, deadline, label, path, err, err_size))
|
||||
return SECRET_READ_ERROR;
|
||||
continue;
|
||||
}
|
||||
set_error(err, err_size, "error reading %s '%s': %s", label, path,
|
||||
errno != 0 ? strerror(errno) : "read failed");
|
||||
return SECRET_READ_ERROR;
|
||||
}
|
||||
}
|
||||
|
||||
/* Trim leading/trailing ASCII space and tab in place; returns the new start. */
|
||||
static char* trim_space(char* s) {
|
||||
while (*s == ' ' || *s == '\t')
|
||||
@@ -673,17 +509,9 @@ static CredentialStore* load_store_file(const char* path, char* err, size_t err_
|
||||
char line[CREDENTIAL_MAX_LINE + 2];
|
||||
bool ok = true;
|
||||
|
||||
for (;;) {
|
||||
size_t len = 0;
|
||||
SecretReadResult rr =
|
||||
secret_read_line(line, sizeof(line), fp, "credential file", path, &len, err, err_size);
|
||||
if (rr == SECRET_READ_EOF)
|
||||
break;
|
||||
if (rr == SECRET_READ_ERROR) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
while (fgets(line, sizeof(line), fp)) {
|
||||
line_no++;
|
||||
size_t len = strlen(line);
|
||||
if (len == CREDENTIAL_MAX_LINE + 1 && line[len - 1] != '\n' && !feof(fp)) {
|
||||
set_error(err, err_size, "credential file '%s' line %d exceeds the %d-byte limit", path,
|
||||
line_no, CREDENTIAL_MAX_LINE);
|
||||
@@ -1320,17 +1148,9 @@ int credentials_hash_file(const char* path, uint32_t iters, FILE* out, char* err
|
||||
int line_no = 0;
|
||||
int result = 0;
|
||||
char line[CREDENTIAL_MAX_LINE + 2];
|
||||
for (;;) {
|
||||
size_t len = 0;
|
||||
SecretReadResult rr =
|
||||
secret_read_line(line, sizeof(line), fp, "plaintext file", path, &len, err, err_size);
|
||||
if (rr == SECRET_READ_EOF)
|
||||
break;
|
||||
if (rr == SECRET_READ_ERROR) {
|
||||
result = -1;
|
||||
break;
|
||||
}
|
||||
while (fgets(line, sizeof(line), fp)) {
|
||||
line_no++;
|
||||
size_t len = strlen(line);
|
||||
if (len == CREDENTIAL_MAX_LINE + 1 && line[len - 1] != '\n' && !feof(fp)) {
|
||||
set_error(err, err_size, "plaintext file '%s' line %d exceeds the %d-byte limit", path,
|
||||
line_no, CREDENTIAL_MAX_LINE);
|
||||
@@ -1408,15 +1228,9 @@ int credentials_read_secret_file(const char* path, char** user_out, char** passw
|
||||
char line[CREDENTIAL_MAX_LINE + 2];
|
||||
int result = -1;
|
||||
|
||||
for (;;) {
|
||||
size_t len = 0;
|
||||
SecretReadResult rr =
|
||||
secret_read_line(line, sizeof(line), fp, "password file", path, &len, err, err_size);
|
||||
if (rr == SECRET_READ_EOF)
|
||||
break;
|
||||
if (rr == SECRET_READ_ERROR)
|
||||
goto done;
|
||||
while (fgets(line, sizeof(line), fp)) {
|
||||
line_no++;
|
||||
size_t len = strlen(line);
|
||||
if (len == CREDENTIAL_MAX_LINE + 1 && line[len - 1] != '\n' && !feof(fp)) {
|
||||
set_error(err, err_size, "password file '%s' line %d exceeds the %d-byte limit", path,
|
||||
line_no, CREDENTIAL_MAX_LINE);
|
||||
|
||||
+10
-217
@@ -1,12 +1,12 @@
|
||||
#include "daemon_conf.h"
|
||||
#include "credentials.h"
|
||||
#include "log.h"
|
||||
#include "utils.h"
|
||||
#include <arpa/inet.h>
|
||||
#include <ctype.h>
|
||||
#include <errno.h>
|
||||
#include <limits.h>
|
||||
#include <netinet/in.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
@@ -17,7 +17,14 @@
|
||||
/* helpers */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
#define set_error utils_set_error
|
||||
static void set_error(char* err, size_t err_size, const char* fmt, ...) {
|
||||
if (!err || err_size == 0)
|
||||
return;
|
||||
va_list args;
|
||||
va_start(args, fmt);
|
||||
vsnprintf(err, err_size, fmt, args);
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
/* Trim leading and trailing ASCII space/tab in place; returns the new start. */
|
||||
static char* trim_ws(char* s) {
|
||||
@@ -34,158 +41,6 @@ static bool key_equals(const char* key, const char* canonical) {
|
||||
return strcasecmp(key, canonical) == 0;
|
||||
}
|
||||
|
||||
/* True when `key` matches one of the NUL-terminated names in `list`. */
|
||||
static bool key_in_list(const char* key, const char* const* list, size_t count) {
|
||||
for (size_t i = 0; i < count; i++) {
|
||||
if (strcasecmp(key, list[i]) == 0)
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/* rsync 3.4.1 rsyncd.conf GLOBAL keys accepted in the pre-module section that
|
||||
* have no FastSync equivalent. They are recognized and documented as inert:
|
||||
* accepting a real rsync config must not fail on a logging/process key, but a
|
||||
* silently-reinterpreted key is never invented. `pidfile`/`logfile` are the
|
||||
* compact --dparam spellings rsync documents. The same list is used by the
|
||||
* `--dparam` dispatch (apply_global_key), so there is a single impl. */
|
||||
static const char* const kRsyncInertGlobalKeys[] = {
|
||||
"pid file",
|
||||
"pidfile",
|
||||
"log file",
|
||||
"logfile",
|
||||
"socket options",
|
||||
"sockopts",
|
||||
"listen backlog",
|
||||
"syslog facility",
|
||||
"syslog tag",
|
||||
"log format",
|
||||
"use chroot",
|
||||
"uid",
|
||||
"gid",
|
||||
"timeout",
|
||||
"max verbosity",
|
||||
"min verbosity",
|
||||
"lock file",
|
||||
"transfer logging",
|
||||
"strict modes",
|
||||
"reverse lookup",
|
||||
"forward lookup",
|
||||
"ignore errors",
|
||||
"ignore nonreadable",
|
||||
"dont compress",
|
||||
};
|
||||
|
||||
/* rsync 3.4.1 rsyncd.conf MODULE keys accepted in a [module] section that have
|
||||
* no FastSync equivalent (accepted-and-documented inert). Keys with a FastSync
|
||||
* meaning (`path`, `read only`, `write only`, `auth users`, `max connections`,
|
||||
* `hosts allow`/`hosts deny`, `client owner`) are handled by apply_module_key
|
||||
* before this list is consulted. Security-relevant keys (`exclude`, `filter`,
|
||||
* `secrets file`, `refuse options`, ...) are inert, so a daemon-side filter or
|
||||
* rsync secrets file is NOT enforced: each is loudly warned about at load time
|
||||
* (see kRsyncUnenforcedModuleSecurityKeys) and documented as a residual in
|
||||
* RSYNC_COMPAT.md. */
|
||||
static const char* const kRsyncInertModuleKeys[] = {
|
||||
"comment",
|
||||
"use chroot",
|
||||
"daemon chroot",
|
||||
"uid",
|
||||
"gid",
|
||||
"daemon uid",
|
||||
"daemon gid",
|
||||
"exclude",
|
||||
"include",
|
||||
"exclude from",
|
||||
"include from",
|
||||
"filter",
|
||||
"max verbosity",
|
||||
"min verbosity",
|
||||
"lock file",
|
||||
"transfer logging",
|
||||
"log file",
|
||||
"log format",
|
||||
"syslog facility",
|
||||
"syslog tag",
|
||||
"timeout",
|
||||
"secrets file",
|
||||
"auth digest",
|
||||
"strict modes",
|
||||
"numeric ids",
|
||||
"fake super",
|
||||
"munge symlinks",
|
||||
"list",
|
||||
"dont compress",
|
||||
"charset",
|
||||
"refuse options",
|
||||
"incoming chmod",
|
||||
"outgoing chmod",
|
||||
"open noatime",
|
||||
"max size",
|
||||
"min size",
|
||||
"temp dir",
|
||||
"pre-xfer exec",
|
||||
"post-xfer exec",
|
||||
"name converter",
|
||||
"proxy protocol",
|
||||
"proxy protocol hosts",
|
||||
"reverse lookup",
|
||||
"forward lookup",
|
||||
"ignore errors",
|
||||
"ignore nonreadable",
|
||||
};
|
||||
|
||||
/* Subset of the inert rsync keys whose intent is access control (data
|
||||
* visibility, credential source, transfer hooks, daemon privilege), plus the
|
||||
* global keys that shape the daemon's privilege/identity. These load for
|
||||
* rsync-config compatibility, but because FastSync ignores them an operator
|
||||
* migrating a hardened rsyncd.conf must not believe the restriction applies.
|
||||
* The loader emits one LOG_LEVEL_WARNING per occurrence naming the key (and the
|
||||
* module, for a module key). `write only` is deliberately absent: it is mapped
|
||||
* onto writability instead (FastSync is push-only, so a write-only module is
|
||||
* simply writable). */
|
||||
static const char* const kRsyncUnenforcedModuleSecurityKeys[] = {
|
||||
"secrets file",
|
||||
"auth digest",
|
||||
"refuse options",
|
||||
"exclude",
|
||||
"include",
|
||||
"exclude from",
|
||||
"include from",
|
||||
"filter",
|
||||
"max size",
|
||||
"min size",
|
||||
"pre-xfer exec",
|
||||
"post-xfer exec",
|
||||
"incoming chmod",
|
||||
"outgoing chmod",
|
||||
"name converter",
|
||||
"use chroot",
|
||||
"daemon chroot",
|
||||
"uid",
|
||||
"gid",
|
||||
"daemon uid",
|
||||
"daemon gid",
|
||||
"munge symlinks",
|
||||
"fake super",
|
||||
"strict modes",
|
||||
"proxy protocol",
|
||||
"proxy protocol hosts",
|
||||
};
|
||||
|
||||
static const char* const kRsyncUnenforcedGlobalSecurityKeys[] = {
|
||||
"use chroot",
|
||||
"uid",
|
||||
"gid",
|
||||
"strict modes",
|
||||
};
|
||||
|
||||
#define kRsyncInertGlobalCount (sizeof(kRsyncInertGlobalKeys) / sizeof(kRsyncInertGlobalKeys[0]))
|
||||
#define kRsyncInertModuleCount (sizeof(kRsyncInertModuleKeys) / sizeof(kRsyncInertModuleKeys[0]))
|
||||
#define kRsyncUnenforcedModuleSecurityCount \
|
||||
(sizeof(kRsyncUnenforcedModuleSecurityKeys) / sizeof(kRsyncUnenforcedModuleSecurityKeys[0]))
|
||||
#define kRsyncUnenforcedGlobalSecurityCount \
|
||||
(sizeof(kRsyncUnenforcedGlobalSecurityKeys) / sizeof(kRsyncUnenforcedGlobalSecurityKeys[0]))
|
||||
|
||||
static bool parse_bool_value(const char* value, bool* out) {
|
||||
if (strcasecmp(value, "yes") == 0 || strcasecmp(value, "true") == 0 || strcmp(value, "1") == 0) {
|
||||
*out = true;
|
||||
@@ -403,10 +258,6 @@ DaemonConf* daemon_conf_create(void) {
|
||||
if (!conf)
|
||||
return NULL;
|
||||
conf->global.port = DAEMON_CONF_DEFAULT_PORT;
|
||||
/* rsync modules are READ-ONLY unless `read only = no` (or `write only = yes`)
|
||||
* is set, so FastSync must default the same way: a migrated rsyncd.conf that
|
||||
* omits `read only` is served read-only, never writable. */
|
||||
conf->global.read_only_default = true;
|
||||
conf->global.max_connections = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS;
|
||||
conf->global.auth_failure_delay_ms = DAEMON_CONF_DEFAULT_AUTH_FAILURE_DELAY_MS;
|
||||
conf->global.max_connections_per_host = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS_PER_HOST;
|
||||
@@ -481,7 +332,7 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
|
||||
char* err, size_t err_size) {
|
||||
if (key_equals(key, "port"))
|
||||
return store_port(&conf->global.port, value, err, err_size);
|
||||
if (key_equals(key, "motd file") || key_equals(key, "motdfile")) {
|
||||
if (key_equals(key, "motd file")) {
|
||||
if (!store_string(&conf->global.motd_file, value)) {
|
||||
set_error(err, err_size, "out of memory parsing 'motd file'");
|
||||
return false;
|
||||
@@ -495,25 +346,6 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
|
||||
}
|
||||
return true;
|
||||
}
|
||||
/* rsync allows the `read only` module key in the global section as the
|
||||
* default for modules defined after it. Map it to that default (a later
|
||||
* --dparam re-applies it to modules that did not set their own value) so a
|
||||
* global `read only = yes` cannot be silently dropped into a writable
|
||||
* default. */
|
||||
if (key_equals(key, "read only")) {
|
||||
bool parsed;
|
||||
if (!parse_bool_value(value, &parsed)) {
|
||||
set_error(err, err_size, "global 'read only' must be yes/no (or true/false/1/0), got '%s'",
|
||||
value);
|
||||
return false;
|
||||
}
|
||||
conf->global.read_only_default = parsed;
|
||||
for (int i = 0; i < conf->module_count; i++) {
|
||||
if (!conf->modules[i].read_only_explicit)
|
||||
conf->modules[i].read_only = parsed;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
if (key_equals(key, "max connections"))
|
||||
return store_max_connections(&conf->global.max_connections, value, NULL, err, err_size);
|
||||
if (key_equals(key, "max connections per host"))
|
||||
@@ -536,15 +368,6 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
|
||||
if (key_equals(key, "hosts deny"))
|
||||
return store_host_list(&conf->global.hosts_deny, &conf->global.hosts_deny_count, value,
|
||||
"hosts deny", NULL, replace_hosts, err, err_size);
|
||||
/* A recognized rsync global key with no FastSync equivalent loads inert. */
|
||||
if (key_in_list(key, kRsyncInertGlobalKeys, kRsyncInertGlobalCount)) {
|
||||
if (key_in_list(key, kRsyncUnenforcedGlobalSecurityKeys, kRsyncUnenforcedGlobalSecurityCount))
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"daemon config: global key '%s' is accepted for rsync compatibility but is NOT "
|
||||
"enforced by FastSync; the restriction it expresses will not be applied",
|
||||
key);
|
||||
return true;
|
||||
}
|
||||
set_error(err, err_size, "unknown global key '%s'", key);
|
||||
return false;
|
||||
}
|
||||
@@ -573,26 +396,6 @@ static bool apply_module_key(DaemonModule* module, char* key, char* value, char*
|
||||
return false;
|
||||
}
|
||||
module->read_only = parsed;
|
||||
module->read_only_explicit = true;
|
||||
return true;
|
||||
}
|
||||
/* rsync's `write only = yes` makes the module client-writable. FastSync has
|
||||
* no read/pull path, so mapping it to writability is the exact
|
||||
* security-relevant effect; set `read_only_explicit` so a global default
|
||||
* cannot override the module's explicit choice. `write only = no` is the
|
||||
* rsync default and leaves the module's read-only state untouched. */
|
||||
if (key_equals(key, "write only")) {
|
||||
bool parsed;
|
||||
if (!parse_bool_value(value, &parsed)) {
|
||||
set_error(err, err_size,
|
||||
"module '%s': 'write only' must be yes/no (or true/false/1/0), got '%s'",
|
||||
module->name, value);
|
||||
return false;
|
||||
}
|
||||
if (parsed) {
|
||||
module->read_only = false;
|
||||
module->read_only_explicit = true;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
if (key_equals(key, "client owner")) {
|
||||
@@ -662,15 +465,6 @@ static bool apply_module_key(DaemonModule* module, char* key, char* value, char*
|
||||
if (key_equals(key, "hosts deny"))
|
||||
return store_host_list(&module->hosts_deny, &module->hosts_deny_count, value, "hosts deny",
|
||||
module->name, false, err, err_size);
|
||||
/* A recognized rsync module key with no FastSync equivalent loads inert. */
|
||||
if (key_in_list(key, kRsyncInertModuleKeys, kRsyncInertModuleCount)) {
|
||||
if (key_in_list(key, kRsyncUnenforcedModuleSecurityKeys, kRsyncUnenforcedModuleSecurityCount))
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"daemon config: module '%s' key '%s' is accepted for rsync compatibility but is "
|
||||
"NOT enforced by FastSync; the restriction it expresses will not be applied",
|
||||
module->name, key);
|
||||
return true;
|
||||
}
|
||||
set_error(err, err_size, "unknown key '%s' in module '%s'", key, module->name);
|
||||
return false;
|
||||
}
|
||||
@@ -721,7 +515,6 @@ static int open_module(DaemonConf* conf, int* current_module, const char* name,
|
||||
}
|
||||
conf->modules = grown;
|
||||
memset(&conf->modules[conf->module_count], 0, sizeof(DaemonModule));
|
||||
conf->modules[conf->module_count].read_only = conf->global.read_only_default;
|
||||
conf->modules[conf->module_count].name = str_dup(name);
|
||||
if (!conf->modules[conf->module_count].name) {
|
||||
set_error(err, err_size, "out of memory adding module '%s'", name);
|
||||
|
||||
+13
-39
@@ -17,20 +17,7 @@
|
||||
* DAEMON_CONF_MAX_LINE all fail the whole load with a clear, line-numbered
|
||||
* error instead of being silently ignored. This keeps a typo from silently
|
||||
* changing what a module serves.
|
||||
*
|
||||
* rsync compatibility: to reduce the divergence from rsync 3.4.1's rsyncd.conf
|
||||
* grammar, the parser also ACCEPTS the common rsync GLOBAL and MODULE keys.
|
||||
* Keys with a FastSync equivalent are mapped onto it (the native spellings are
|
||||
* unchanged; `read only` defaults to yes like rsync, and `write only = yes`
|
||||
* opts a module into writability). Keys with no FastSync equivalent are
|
||||
* accepted and documented as inert (they load successfully but have no effect)
|
||||
* rather than failing the whole config; the accepted inert set is listed in
|
||||
* kRsyncInertGlobalKeys / kRsyncInertModuleKeys in daemon_conf.c and in
|
||||
* RSYNC_COMPAT.md. Every inert key whose intent is access control is loudly
|
||||
* warned about at load time (kRsyncUnenforced*SecurityKeys) so an operator
|
||||
* migrating a hardened rsyncd.conf is never misled into believing the
|
||||
* restriction is enforced. A key outside both the FastSync-native grammar and
|
||||
* the recognized rsync subset is still rejected as unknown. */
|
||||
*/
|
||||
|
||||
/* A daemon module's configured root is used exactly like the standalone
|
||||
* server's --destination-root: the daemon confines every connection that
|
||||
@@ -55,21 +42,15 @@
|
||||
* store refuses (fail closed) rather than falling open; see server.c. Auth is
|
||||
* never bypassed by ignoring the list. */
|
||||
typedef struct DaemonModule {
|
||||
char* name; /* module name, as the client requests it */
|
||||
char* path; /* module root (daemon-side authorized root) */
|
||||
bool read_only; /* `read only = yes/no`; defaults to the global `read only`
|
||||
default (rsync allows it in the global section), which is
|
||||
itself default YES (rsync modules are read-only unless
|
||||
`read only = no` / `write only = yes` opts in) */
|
||||
bool read_only_explicit; /* set when this module set its own `read only` or
|
||||
`write only = yes`, so a later global default (from a
|
||||
`--dparam read only=`) does not override it */
|
||||
bool client_owner; /* `client owner = yes/no`; default no. Per-module opt-in
|
||||
that lets this module's clients choose ownership
|
||||
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/
|
||||
--copy-as) and request explicit --super super-user
|
||||
activities. Without it the daemon refuses all of them. */
|
||||
char** auth_users; /* `auth users = a,b`; Wave B credential list */
|
||||
char* name; /* module name, as the client requests it */
|
||||
char* path; /* module root (daemon-side authorized root) */
|
||||
bool read_only; /* `read only = yes/no`; default no */
|
||||
bool client_owner; /* `client owner = yes/no`; default no. Per-module opt-in
|
||||
that lets this module's clients choose ownership
|
||||
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/
|
||||
--copy-as) and request explicit --super super-user
|
||||
activities. Without it the daemon refuses all of them. */
|
||||
char** auth_users; /* `auth users = a,b`; Wave B credential list */
|
||||
int auth_user_count;
|
||||
/* `max connections = N` (optional per-module cap). 0 means unlimited. The
|
||||
* per-connection child records the selected module in the shared registry
|
||||
@@ -88,10 +69,6 @@ typedef struct DaemonConfGlobals {
|
||||
int port; /* `port`, default DAEMON_CONF_DEFAULT_PORT (873) */
|
||||
char* motd_file; /* `motd file`, may be NULL */
|
||||
char* address; /* `address` (optional bind address), may be NULL */
|
||||
bool read_only_default; /* global `read only` default for modules defined
|
||||
after it (rsync allows the module key in the
|
||||
global section); default YES to match rsync's
|
||||
read-only modules */
|
||||
int max_connections; /* `max connections`, default
|
||||
DAEMON_CONF_DEFAULT_MAX_CONNECTIONS (100) */
|
||||
int auth_failure_delay_ms; /* `auth failure delay`, milliseconds; default
|
||||
@@ -175,13 +152,10 @@ const DaemonModule* daemon_conf_find_module(const DaemonConf* conf, const char*
|
||||
bool daemon_module_name_valid(const char* name);
|
||||
|
||||
/* Parse one --dparam=KEY=VALUE (or "--dparam KEY=VALUE") override string and
|
||||
* apply it to the global keys only. Keys are case-insensitive and cover the
|
||||
* global keys defined by the grammar (port, motd file, address, read only,
|
||||
* apply it to the global keys only. Keys are case-insensitive and limited to
|
||||
* the global keys defined by the grammar (port, motd file, address,
|
||||
* max connections, max connections per host, auth failure delay,
|
||||
* auth lockout threshold, auth lockout duration, hosts allow, hosts deny) plus
|
||||
* the recognized inert rsync global keys and the compact rsync spellings
|
||||
* (`motdfile`, `pidfile`, `logfile`). Applying `read only` sets the global
|
||||
* default and re-applies it to every module that did not set its own value.
|
||||
* auth lockout threshold, auth lockout duration, hosts allow, hosts deny).
|
||||
* Returns 0 on success, -1 on error (err filled). */
|
||||
int daemon_conf_apply_dparam(DaemonConf* conf, const char* assignment, char* err, size_t err_size);
|
||||
|
||||
|
||||
+36
-118
@@ -8,49 +8,13 @@
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <libgen.h>
|
||||
#include <stdatomic.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/file.h>
|
||||
#include <sys/stat.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
/* Process-wide counter so two staging contexts created in the same process (or
|
||||
within the same clock tick) can never pick the same name. */
|
||||
static unsigned long long delay_updates_next_sequence(void) {
|
||||
static atomic_ullong sequence;
|
||||
return atomic_fetch_add_explicit(&sequence, 1, memory_order_relaxed);
|
||||
}
|
||||
|
||||
/* Build the per-run staging directory basename: the reserved prefix plus the
|
||||
pid and an entropy token. A fixed name could collide with a genuine
|
||||
destination entry; the token makes such a collision vanishingly unlikely and,
|
||||
if it ever happens, prepare() refuses to touch the existing directory. */
|
||||
static char* delay_updates_make_staging_name(void) {
|
||||
unsigned long long entropy = 0;
|
||||
int fd = open("/dev/urandom", O_RDONLY | O_CLOEXEC);
|
||||
if (fd >= 0) {
|
||||
ssize_t got = read(fd, &entropy, sizeof(entropy));
|
||||
close(fd);
|
||||
if (got != (ssize_t)sizeof(entropy))
|
||||
entropy = 0;
|
||||
}
|
||||
if (entropy == 0)
|
||||
entropy = ((unsigned long long)time(NULL) << 20) ^ ((unsigned long long)getpid() << 8) ^
|
||||
delay_updates_next_sequence();
|
||||
int length = snprintf(NULL, 0, DELAY_UPDATES_STAGING_DIR ".%ld.%llx", (long)getpid(), entropy);
|
||||
if (length < 0)
|
||||
return NULL;
|
||||
char* name = malloc((size_t)length + 1);
|
||||
if (!name)
|
||||
return NULL;
|
||||
snprintf(name, (size_t)length + 1, DELAY_UPDATES_STAGING_DIR ".%ld.%llx", (long)getpid(),
|
||||
entropy);
|
||||
return name;
|
||||
}
|
||||
|
||||
DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
|
||||
if (!root_directory)
|
||||
return NULL;
|
||||
@@ -62,15 +26,8 @@ DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
|
||||
free(context);
|
||||
return NULL;
|
||||
}
|
||||
context->staging_name = delay_updates_make_staging_name();
|
||||
if (!context->staging_name) {
|
||||
free(context->root_directory);
|
||||
free(context);
|
||||
return NULL;
|
||||
}
|
||||
context->staging_root = path_cat(root_directory, context->staging_name);
|
||||
context->staging_root = path_cat(root_directory, DELAY_UPDATES_STAGING_DIR);
|
||||
if (!context->staging_root) {
|
||||
free(context->staging_name);
|
||||
free(context->root_directory);
|
||||
free(context);
|
||||
return NULL;
|
||||
@@ -82,7 +39,6 @@ DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
|
||||
context->lock_fd = -1;
|
||||
if (mtx_init(&context->mutex, mtx_plain) != thrd_success) {
|
||||
free(context->staging_root);
|
||||
free(context->staging_name);
|
||||
free(context->root_directory);
|
||||
free(context);
|
||||
return NULL;
|
||||
@@ -98,7 +54,6 @@ void delay_updates_context_destroy(DelayUpdatesContext* context) {
|
||||
close(context->lock_fd);
|
||||
context->lock_fd = -1;
|
||||
free(context->staging_root);
|
||||
free(context->staging_name);
|
||||
free(context->root_directory);
|
||||
for (size_t i = 0; i < context->count; i++) {
|
||||
free(context->entries[i].staged_path);
|
||||
@@ -170,81 +125,48 @@ bool delay_updates_prepare(DelayUpdatesContext* context) {
|
||||
return false;
|
||||
if (context->prepared)
|
||||
return true;
|
||||
/* Create the per-run staging directory with O_EXCL semantics. The name is
|
||||
unique to this transfer, so if the path already exists it is NOT ours:
|
||||
either a genuine destination entry that happens to share the name or a
|
||||
leftover from another session. Refuse rather than wipe it -- the old
|
||||
fixed-name design could destroy a real destination entry. A crash
|
||||
leftover is never reused (the next run picks a fresh name). */
|
||||
char* leaf = NULL;
|
||||
int parent_fd = file_open_secure_parent(context->staging_root, &leaf, true);
|
||||
if (parent_fd < 0) {
|
||||
int saved_errno = errno;
|
||||
char* escaped = output_escape(context->staging_root, false);
|
||||
log_message(LOG_LEVEL_ERROR, "could not create --delay-updates staging directory '%s': %s",
|
||||
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||
free(escaped);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
int fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (fd >= 0) {
|
||||
close(fd);
|
||||
close(parent_fd);
|
||||
char* escaped = output_escape(context->staging_root, false);
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"--delay-updates staging directory '%s' already exists and is not owned by this "
|
||||
"transfer; refusing to overwrite it",
|
||||
escaped ? escaped : "<allocation failed>");
|
||||
free(escaped);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
if (errno != ENOENT) {
|
||||
int saved_errno = errno;
|
||||
close(parent_fd);
|
||||
char* escaped = output_escape(context->staging_root, false);
|
||||
log_message(LOG_LEVEL_ERROR, "could not open --delay-updates staging directory '%s': %s",
|
||||
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||
free(escaped);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
if (mkdirat(parent_fd, leaf, 0700) != 0) {
|
||||
int saved_errno = errno;
|
||||
close(parent_fd);
|
||||
char* escaped = output_escape(context->staging_root, false);
|
||||
log_message(LOG_LEVEL_ERROR, "could not create --delay-updates staging directory '%s': %s",
|
||||
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||
free(escaped);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
int fd = file_open_private_dir(context->staging_root);
|
||||
if (fd < 0) {
|
||||
int saved_errno = errno;
|
||||
char* escaped = output_escape(context->staging_root, false);
|
||||
log_message(LOG_LEVEL_ERROR, "could not open --delay-updates staging directory '%s': %s",
|
||||
log_message(LOG_LEVEL_ERROR, "could not create --delay-updates staging directory '%s': %s",
|
||||
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||
free(escaped);
|
||||
return false;
|
||||
}
|
||||
/* Keep the exclusive advisory lock as defense in depth: the unique name
|
||||
already prevents two sessions from sharing a staging directory, but the
|
||||
lock also catches an improbable same-name collision that raced between the
|
||||
existence check above and the open. */
|
||||
/* Hold an exclusive advisory lock on the staging directory for the whole
|
||||
transfer. The staging directory name is fixed, so two simultaneous
|
||||
delayed transfers to the same destination root would otherwise share it
|
||||
and destroy each other's staged files. The lock makes the second session
|
||||
fail cleanly instead of corrupting the first. The lock is released when
|
||||
the context (and its file descriptor) is destroyed. */
|
||||
if (flock(fd, LOCK_EX | LOCK_NB) != 0) {
|
||||
int saved_errno = errno;
|
||||
close(fd);
|
||||
char* escaped = output_escape(context->staging_root, false);
|
||||
log_message(LOG_LEVEL_ERROR, "could not lock --delay-updates staging directory '%s': %s",
|
||||
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
|
||||
free(escaped);
|
||||
if (saved_errno == EWOULDBLOCK || saved_errno == EAGAIN) {
|
||||
char* escaped = output_escape(context->staging_root, false);
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"another --delay-updates transfer to '%s' is already in progress; refusing to "
|
||||
"share the staging directory",
|
||||
escaped ? escaped : "<allocation failed>");
|
||||
free(escaped);
|
||||
} else {
|
||||
log_message(LOG_LEVEL_ERROR, "could not lock --delay-updates staging directory '%s': %s",
|
||||
context->staging_root, strerror(saved_errno));
|
||||
}
|
||||
return false;
|
||||
}
|
||||
context->lock_fd = fd;
|
||||
/* Only now, with exclusive ownership, wipe leftovers from an interrupted
|
||||
earlier transfer; this can never race with a live session. */
|
||||
bool ok = delay_wipe_dir_fd(fd);
|
||||
if (!ok) {
|
||||
log_message(LOG_LEVEL_ERROR, "could not clear stale --delay-updates staging files under '%s'",
|
||||
context->staging_root);
|
||||
close(context->lock_fd);
|
||||
context->lock_fd = -1;
|
||||
return false;
|
||||
}
|
||||
context->prepared = true;
|
||||
return true;
|
||||
}
|
||||
@@ -342,16 +264,12 @@ static bool delay_publish_entry(DelayUpdatesContext* context, const Config* conf
|
||||
const StagedFileEntry* entry) {
|
||||
if (!delay_publish_backup(context, config, entry))
|
||||
return false;
|
||||
/* An incoming regular file/symlink may replace a destination DIRECTORY that
|
||||
blocks it. rsync removes the blocker recursively when --delete or --force
|
||||
is active (its generator's "make way" deletion), and a --delay-updates run
|
||||
stages elsewhere so it only discovers the blocker here. FastSync's
|
||||
immediate-install path clears it too; without --delete/--force a non-empty
|
||||
blocker fails the run (rsync's "could not make way for new regular file").
|
||||
use_delete is gated by the server --allow-delete policy, so a client can
|
||||
never use this to bypass deletion authorization. */
|
||||
if (config && (config->force_delete || config->use_delete) &&
|
||||
file_directory_exists_secure(entry->final_path)) {
|
||||
/* --force: an incoming regular file/symlink may replace a destination
|
||||
DIRECTORY (possibly non-empty). The immediate-install path handles this in
|
||||
file_receive; a --delay-updates run stages elsewhere and only discovers the
|
||||
blocking directory here, so clear it before the rename (rsync's
|
||||
"could not make way for new regular file" without --force). */
|
||||
if (config && config->force_delete && file_directory_exists_secure(entry->final_path)) {
|
||||
if (!file_remove_tree_secure(entry->final_path)) {
|
||||
char* escaped = output_escape(entry->final_path, false);
|
||||
log_message(LOG_LEVEL_ERROR, "could not remove destination directory blocking '%s': %s",
|
||||
|
||||
@@ -24,7 +24,6 @@ typedef struct {
|
||||
shared with the publish/cleanup phase that runs after the threads join. */
|
||||
typedef struct DelayUpdatesContext {
|
||||
char* root_directory; /* receive root the staging dir lives under */
|
||||
char* staging_name; /* per-run unique staging dir basename */
|
||||
char* staging_root; /* root_directory/<staging dir name> */
|
||||
mtx_t mutex;
|
||||
StagedFileEntry* entries;
|
||||
@@ -34,11 +33,7 @@ typedef struct DelayUpdatesContext {
|
||||
int lock_fd; /* advisory exclusive flock held on the staging dir, or -1 */
|
||||
} DelayUpdatesContext;
|
||||
|
||||
/* Reserved prefix for the private staging subdirectory created under the
|
||||
receive root. The actual directory name is per-run unique (the prefix plus a
|
||||
pid/entropy token) so it can never clobber a genuine destination entry that
|
||||
happens to share the name; the bare prefix is still what a --backup-dir must
|
||||
not collide with. */
|
||||
/* Name of the private staging subdirectory created under the receive root. */
|
||||
#define DELAY_UPDATES_STAGING_DIR ".fastsync-stage"
|
||||
|
||||
/* True when `dir` (ignoring a trailing "/") is the reserved staging directory
|
||||
|
||||
@@ -1,777 +0,0 @@
|
||||
#include "delete.h"
|
||||
|
||||
#include "delay_updates.h"
|
||||
#include "filter.h"
|
||||
#include "log.h"
|
||||
#include "utils.h"
|
||||
#include <dirent.h>
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
|
||||
/* Build the keep-set index from the exact manifest entries only. A lookup of
|
||||
`rel` succeeds iff `rel` is a kept entry, a kept directory, or an ancestor
|
||||
directory of kept content (the old is_dir_in_manifest predicate); the sorted
|
||||
view answers "is an ancestor of kept content" without materializing any
|
||||
per-component prefix copy, so the index is O(manifest size) memory. */
|
||||
static bool build_keep_index(const ArrayList* manifest, PathIndex* index) {
|
||||
if (!manifest || manifest->size <= 0)
|
||||
return path_index_build(index, NULL, 0);
|
||||
return path_index_build(index, (const char* const*)manifest->items, (size_t)manifest->size);
|
||||
}
|
||||
|
||||
static bool keep_is_dir(const PathIndex* index, const char* rel_path) {
|
||||
return path_index_contains(index, rel_path) || path_index_has_descendant(index, rel_path);
|
||||
}
|
||||
|
||||
static bool keep_is_file(const PathIndex* index, const char* rel_path) {
|
||||
return path_index_contains(index, rel_path);
|
||||
}
|
||||
|
||||
/* rsync's receiver-side verdict for one candidate extra: the per-directory
|
||||
* chain first (deepest directory before ancestors), then the command-line base
|
||||
* rules. Either rule set may be absent. */
|
||||
FilterAction delete_protect_verdict(const DeleteProtectRules* protect, const char* rel_path,
|
||||
const char* leaf, bool is_dir) {
|
||||
if (!protect)
|
||||
return FILTER_ACTION_NONE;
|
||||
/* rsync protects its own --backup files from the delete pass: a name ending
|
||||
in the backup suffix is never an extra. Checked before the filter rules so
|
||||
an explicit exclude cannot be bypassed (the suffix is always a shield). */
|
||||
if (protect->backup_suffix && protect->backup_suffix[0] != '\0') {
|
||||
size_t name_len = strlen(leaf);
|
||||
size_t suffix_len = strlen(protect->backup_suffix);
|
||||
if (name_len > suffix_len &&
|
||||
strcmp(leaf + (name_len - suffix_len), protect->backup_suffix) == 0)
|
||||
return FILTER_ACTION_PROTECT;
|
||||
}
|
||||
FilterAction action = filter_dir_rules_apply_side(protect->dir_rules, rel_path, leaf, is_dir);
|
||||
if (action != FILTER_ACTION_NONE)
|
||||
return action;
|
||||
return filter_rules_apply_side(protect->base_rules, rel_path, leaf, is_dir, FILTER_SIDE_RECEIVER);
|
||||
}
|
||||
|
||||
const char* delete_backup_suffix(const Config* config) {
|
||||
if (!config || !config->backup || config->ignore_existing)
|
||||
return NULL;
|
||||
const char* suffix = config->suffix ? config->suffix : "~";
|
||||
if (!suffix[0] || strchr(suffix, '/'))
|
||||
return NULL;
|
||||
return suffix;
|
||||
}
|
||||
|
||||
/* Classify a removed entry from its st_mode for the per-type delete counters. */
|
||||
DeleteEntryType delete_entry_type_of_mode(mode_t mode) {
|
||||
if (S_ISDIR(mode))
|
||||
return DELETE_ENTRY_DIR;
|
||||
if (S_ISLNK(mode))
|
||||
return DELETE_ENTRY_LINK;
|
||||
if (S_ISREG(mode))
|
||||
return DELETE_ENTRY_REG;
|
||||
return DELETE_ENTRY_SPECIAL;
|
||||
}
|
||||
|
||||
/* True when child_rel is, or lies below, a protected entry. A prefix "a"
|
||||
therefore protects "a" and "a/b/c" but not "ab". Entries with top_level_only
|
||||
set only protect DIRECT children of the receive root (at_root); nested
|
||||
directories that share such a name stay ordinary destination content. */
|
||||
bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSkipEntry* skips,
|
||||
int skip_count) {
|
||||
for (int i = 0; i < skip_count; i++) {
|
||||
if (skips[i].top_level_only && !at_root)
|
||||
continue;
|
||||
size_t prefix_len = strlen(skips[i].prefix);
|
||||
if (strncmp(child_rel, skips[i].prefix, prefix_len) == 0 &&
|
||||
(child_rel[prefix_len] == '\0' || child_rel[prefix_len] == '/'))
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/* Per-run deletion budget and tallies. `max_delete` is the cap on the number
|
||||
of entries the walker may remove (SIZE_MAX = unlimited); once it is reached
|
||||
the remaining extras are counted in `skipped` and left in place, matching
|
||||
rsync's partial --max-delete behavior. */
|
||||
typedef struct {
|
||||
size_t max_delete;
|
||||
size_t deleted;
|
||||
size_t skipped;
|
||||
bool limit_hit;
|
||||
} DeleteBudget;
|
||||
|
||||
/* True when direct children of the directory named by `rel` may be removed.
|
||||
With no synchronization info (dirs == NULL) the whole tree is deletable; when
|
||||
a dirs index is supplied only its exact entries are (the receive root is the
|
||||
"." sentinel). */
|
||||
static bool is_synced_dir(const PathIndex* dirs, const char* rel) {
|
||||
if (!dirs)
|
||||
return true;
|
||||
return path_index_contains(dirs, rel[0] == '\0' ? "." : rel);
|
||||
}
|
||||
|
||||
/* Unsigned byte-wise string compare, matching rsync's u_strcmp (a signed
|
||||
strcmp would order bytes >= 0x80 differently). */
|
||||
static int delete_name_cmp(const char* a, const char* b) {
|
||||
const unsigned char* pa = (const unsigned char*)a;
|
||||
const unsigned char* pb = (const unsigned char*)b;
|
||||
while (*pa != '\0' && *pa == *pb) {
|
||||
pa++;
|
||||
pb++;
|
||||
}
|
||||
return (int)*pa - (int)*pb;
|
||||
}
|
||||
|
||||
bool delete_dir_entries_collect(int dirfd, DeleteDirEntry** out, size_t* count,
|
||||
bool* operation_ok) {
|
||||
*out = NULL;
|
||||
*count = 0;
|
||||
if (operation_ok)
|
||||
*operation_ok = true;
|
||||
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (scanfd < 0)
|
||||
return false;
|
||||
DIR* dir = fdopendir(scanfd);
|
||||
if (!dir) {
|
||||
close(scanfd);
|
||||
return false;
|
||||
}
|
||||
DeleteDirEntry* entries = NULL;
|
||||
size_t used = 0;
|
||||
size_t capacity = 0;
|
||||
bool ok = true;
|
||||
const struct dirent* entry;
|
||||
while ((entry = readdir(dir)) != NULL) {
|
||||
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
|
||||
continue;
|
||||
struct stat st;
|
||||
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
||||
if (errno != ENOENT && operation_ok)
|
||||
*operation_ok = false;
|
||||
continue;
|
||||
}
|
||||
if (used == capacity) {
|
||||
size_t next = capacity == 0 ? 16 : capacity * 2;
|
||||
DeleteDirEntry* grown = realloc(entries, next * sizeof(*grown));
|
||||
if (!grown) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
entries = grown;
|
||||
capacity = next;
|
||||
}
|
||||
entries[used].name = str_dup(entry->d_name);
|
||||
if (!entries[used].name) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
entries[used].is_dir = S_ISDIR(st.st_mode);
|
||||
entries[used].mode = st.st_mode;
|
||||
used++;
|
||||
}
|
||||
closedir(dir);
|
||||
if (!ok) {
|
||||
delete_dir_entries_free(entries, used);
|
||||
return false;
|
||||
}
|
||||
*out = entries;
|
||||
*count = used;
|
||||
return true;
|
||||
}
|
||||
|
||||
void delete_dir_entries_free(DeleteDirEntry* entries, size_t count) {
|
||||
if (!entries)
|
||||
return;
|
||||
for (size_t i = 0; i < count; i++)
|
||||
free(entries[i].name);
|
||||
free(entries);
|
||||
}
|
||||
|
||||
/* rsync's extraneous-entry order: subdirectories before files, each group in
|
||||
descending name order. */
|
||||
int delete_dir_entry_cmp_desc(const void* a, const void* b) {
|
||||
const DeleteDirEntry* ea = a;
|
||||
const DeleteDirEntry* eb = b;
|
||||
if (ea->is_dir != eb->is_dir)
|
||||
return ea->is_dir ? -1 : 1;
|
||||
return -delete_name_cmp(ea->name, eb->name);
|
||||
}
|
||||
|
||||
/* rsync's kept-subdirectory order: plain ascending name. */
|
||||
int delete_dir_entry_cmp_asc(const void* a, const void* b) {
|
||||
const DeleteDirEntry* ea = a;
|
||||
const DeleteDirEntry* eb = b;
|
||||
return delete_name_cmp(ea->name, eb->name);
|
||||
}
|
||||
|
||||
/* How the shared classification/descent walk disposes of an extra it has
|
||||
identified. LIST records the destination-relative path without touching disk
|
||||
(the -n/--dry-run would-delete enumeration); DELETE unlinks/rmdirs it, charges
|
||||
the shared --max-delete budget and notifies the observer. Both modes classify
|
||||
and traverse identically, so the dry-run enumeration and the real deletion
|
||||
cannot drift. */
|
||||
typedef enum { DELETE_WALK_MODE_DELETE, DELETE_WALK_MODE_LIST } DeleteWalkMode;
|
||||
|
||||
typedef struct {
|
||||
DeleteWalkMode mode;
|
||||
DeleteBudget* budget; /* DELETE mode */
|
||||
ArrayList* out; /* LIST mode: receives strdup'd relative paths */
|
||||
size_t* recorded; /* LIST mode */
|
||||
DeletePathObserver observer; /* DELETE mode */
|
||||
void* observer_context; /* DELETE mode */
|
||||
} DeleteWalkState;
|
||||
|
||||
/* The per-walk invariants threaded unchanged through every recursive descent:
|
||||
the keep/synchronized-dir indexes, the destination mode and the protection
|
||||
rules. Bundling them keeps the recursive helpers below to a handful of
|
||||
positional arguments. */
|
||||
typedef struct {
|
||||
const PathIndex* keep;
|
||||
const PathIndex* dirs;
|
||||
DeleteWalkState* state;
|
||||
const DeleteSkipEntry* skips;
|
||||
int skip_count;
|
||||
const DeleteProtectRules* protect;
|
||||
} DeleteWalkContext;
|
||||
|
||||
/* Duplicate `path` with rsync's trailing-slash convention, used to report a
|
||||
removed (or would-be-removed) directory. Returns NULL on allocation
|
||||
failure. */
|
||||
static char* with_trailing_slash(const char* path) {
|
||||
size_t len = strlen(path);
|
||||
char* copy = malloc(len + 2);
|
||||
if (!copy)
|
||||
return NULL;
|
||||
memcpy(copy, path, len);
|
||||
copy[len] = '/';
|
||||
copy[len + 1] = '\0';
|
||||
return copy;
|
||||
}
|
||||
|
||||
/* Forward declaration: the ordered passes below recurse through the driver. */
|
||||
static bool delete_walk_fd(int dirfd, const char* rel_path, const DeleteWalkContext* ctx,
|
||||
bool parent_deletable, bool* all_removed);
|
||||
|
||||
/* Descend into the child directory `name` of `dirfd`, walking it as part of the
|
||||
current operation. Returns false on a genuine open/walk failure; on success
|
||||
*child_all_removed reports whether the child removed everything it held (so
|
||||
the caller may rmdir it). */
|
||||
static bool delete_walk_child(int dirfd, const char* name, const char* child_rel,
|
||||
const DeleteWalkContext* ctx, bool deletable,
|
||||
bool* child_all_removed) {
|
||||
*child_all_removed = false;
|
||||
int childfd = openat(dirfd, name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (childfd < 0)
|
||||
return errno == ENOENT;
|
||||
bool ok = delete_walk_fd(childfd, child_rel, ctx, deletable, child_all_removed);
|
||||
close(childfd);
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Classify every entry up front (the verdict does not depend on processing
|
||||
order) so the ordered passes below can act on it. Sets shielded[]/is_extra[]
|
||||
and reports through *local_survives whether anything in this directory stays
|
||||
in place. Returns false on a path-construction failure. */
|
||||
static bool delete_walk_classify(const char* rel_path, const DeleteDirEntry* entries, size_t count,
|
||||
const DeleteWalkContext* ctx, bool deletable, bool at_root,
|
||||
bool* shielded, bool* is_extra, bool* local_survives) {
|
||||
bool ok = true;
|
||||
for (size_t i = 0; i < count; i++) {
|
||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||
if (!child_rel) {
|
||||
ok = false;
|
||||
continue;
|
||||
}
|
||||
/* A --delay-updates run keeps its staging directory as a direct child of
|
||||
the receive root, and basis-dir snapshots live below it too. Their
|
||||
contents are not manifest entries, so descending into them would delete
|
||||
every staged / basis file as an "extra". Only the staging name (a
|
||||
top-level-only prefix) and the basis prefixes are protected: a nested
|
||||
destination directory that happens to be called .fastsync-stage is
|
||||
ordinary content. */
|
||||
if (path_under_skip_prefix(child_rel, at_root, ctx->skips, ctx->skip_count)) {
|
||||
shielded[i] = true;
|
||||
*local_survives = true;
|
||||
} else if (delete_protect_verdict(ctx->protect, child_rel, entries[i].name,
|
||||
entries[i].is_dir) == FILTER_ACTION_PROTECT) {
|
||||
/* A first-match protect rule shields the extra; for a directory the whole
|
||||
subtree is shielded (rsync prunes an excluded directory), so do not
|
||||
descend. */
|
||||
shielded[i] = true;
|
||||
*local_survives = true;
|
||||
} else if (entries[i].is_dir) {
|
||||
bool child_synced = ctx->dirs && path_index_contains(ctx->dirs, child_rel);
|
||||
is_extra[i] = deletable && !child_synced && !keep_is_dir(ctx->keep, child_rel);
|
||||
if (!is_extra[i])
|
||||
*local_survives = true;
|
||||
} else {
|
||||
is_extra[i] = deletable && !keep_is_file(ctx->keep, child_rel);
|
||||
if (!is_extra[i])
|
||||
*local_survives = true;
|
||||
}
|
||||
free(child_rel);
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Pass 1: extraneous subdirectories, descending. Recurses into each and, when
|
||||
the child removed everything it held, records or removes it and charges the
|
||||
budget. */
|
||||
static bool delete_walk_extra_dirs(int dirfd, const char* rel_path, const DeleteDirEntry* entries,
|
||||
size_t dir_count, const DeleteWalkContext* ctx, bool deletable,
|
||||
const bool* is_extra, bool* local_survives) {
|
||||
bool ok = true;
|
||||
for (size_t i = 0; i < dir_count; i++) {
|
||||
if (!is_extra[i])
|
||||
continue;
|
||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||
if (!child_rel) {
|
||||
ok = false;
|
||||
continue;
|
||||
}
|
||||
bool child_all_removed = false;
|
||||
if (!delete_walk_child(dirfd, entries[i].name, child_rel, ctx, deletable, &child_all_removed))
|
||||
ok = false;
|
||||
if (child_all_removed && deletable) {
|
||||
if (ctx->state->mode == DELETE_WALK_MODE_LIST) {
|
||||
/* Record the directory with rsync's trailing slash. */
|
||||
char* copy = with_trailing_slash(child_rel);
|
||||
if (!copy) {
|
||||
ok = false;
|
||||
} else if (!array_list_add(ctx->state->out, copy)) {
|
||||
free(copy);
|
||||
ok = false;
|
||||
} else {
|
||||
(*ctx->state->recorded)++;
|
||||
}
|
||||
} else if (ctx->state->budget->deleted >= ctx->state->budget->max_delete) {
|
||||
ctx->state->budget->limit_hit = true;
|
||||
ctx->state->budget->skipped++;
|
||||
*local_survives = true;
|
||||
} else if (unlinkat(dirfd, entries[i].name, AT_REMOVEDIR) != 0) {
|
||||
/* ENOENT: already gone (fine). ENOTEMPTY/EEXIST: the directory still
|
||||
holds entries the walker leaves in place (a protected excluded
|
||||
prefix, a kept file the manifest protects, a symlink); rsync leaves
|
||||
such a directory behind, so this is not an error. Only genuine I/O
|
||||
failures abort the deletion. */
|
||||
if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST)
|
||||
ok = false;
|
||||
*local_survives = true;
|
||||
} else {
|
||||
ctx->state->budget->deleted++;
|
||||
/* rsync reports a removed directory with a trailing slash. */
|
||||
if (ctx->state->observer) {
|
||||
char* with_slash = with_trailing_slash(child_rel);
|
||||
if (with_slash) {
|
||||
ctx->state->observer(ctx->state->observer_context, with_slash, DELETE_ENTRY_DIR);
|
||||
free(with_slash);
|
||||
} else {
|
||||
ctx->state->observer(ctx->state->observer_context, child_rel, DELETE_ENTRY_DIR);
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
*local_survives = true;
|
||||
}
|
||||
free(child_rel);
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Pass 2: extraneous files, descending. */
|
||||
static bool delete_walk_extra_files(int dirfd, const char* rel_path, const DeleteDirEntry* entries,
|
||||
size_t dir_count, size_t count, const DeleteWalkContext* ctx,
|
||||
const bool* is_extra, bool* local_survives) {
|
||||
bool ok = true;
|
||||
for (size_t i = dir_count; i < count; i++) {
|
||||
if (!is_extra[i])
|
||||
continue;
|
||||
if (ctx->state->mode == DELETE_WALK_MODE_LIST) {
|
||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||
if (!child_rel) {
|
||||
ok = false;
|
||||
continue;
|
||||
}
|
||||
char* copy = str_dup(child_rel);
|
||||
if (!copy || !array_list_add(ctx->state->out, copy)) {
|
||||
free(copy);
|
||||
ok = false;
|
||||
} else {
|
||||
(*ctx->state->recorded)++;
|
||||
}
|
||||
free(child_rel);
|
||||
} else if (ctx->state->budget->deleted >= ctx->state->budget->max_delete) {
|
||||
ctx->state->budget->limit_hit = true;
|
||||
ctx->state->budget->skipped++;
|
||||
*local_survives = true;
|
||||
} else if (unlinkat(dirfd, entries[i].name, 0) != 0) {
|
||||
if (errno != ENOENT)
|
||||
ok = false;
|
||||
*local_survives = true;
|
||||
} else {
|
||||
ctx->state->budget->deleted++;
|
||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||
if (child_rel) {
|
||||
if (ctx->state->observer)
|
||||
ctx->state->observer(ctx->state->observer_context, child_rel,
|
||||
delete_entry_type_of_mode(entries[i].mode));
|
||||
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
|
||||
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
|
||||
free(escaped_path);
|
||||
}
|
||||
free(child_rel);
|
||||
}
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Pass 3: kept subdirectories, ascending (rsync descends into these only after
|
||||
the parent's own extras have been handled). */
|
||||
static bool delete_walk_kept_dirs(int dirfd, const char* rel_path, const DeleteDirEntry* entries,
|
||||
size_t dir_count, const DeleteWalkContext* ctx, bool deletable,
|
||||
const bool* is_extra, const bool* shielded,
|
||||
bool* local_survives) {
|
||||
bool ok = true;
|
||||
for (size_t i = dir_count; i-- > 0;) {
|
||||
if (is_extra[i] || shielded[i])
|
||||
continue;
|
||||
char* child_rel = path_cat((char*)rel_path, entries[i].name);
|
||||
if (!child_rel) {
|
||||
ok = false;
|
||||
continue;
|
||||
}
|
||||
bool child_all_removed = false;
|
||||
if (!delete_walk_child(dirfd, entries[i].name, child_rel, ctx, deletable, &child_all_removed))
|
||||
ok = false;
|
||||
/* A kept/synchronized directory is never removed. */
|
||||
*local_survives = true;
|
||||
free(child_rel);
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Remove the extras directly inside the directory open on `dirfd` (DELETE mode)
|
||||
or record the paths that WOULD be removed (LIST mode), recursing into every
|
||||
child directory so kept content below a synchronized prefix is reached.
|
||||
`all_removed` reports whether every child entry was removed (so the caller may
|
||||
rmdir this directory). A child directory is never removed when it is itself a
|
||||
synchronized directory or holds kept content; with a dirs index supplied,
|
||||
direct children of a non-synchronized directory are never extras at all (they
|
||||
are left in place but still descended into). Symlinks are unlinked like any
|
||||
other non-directory extra (never followed).
|
||||
|
||||
Entries are processed in rsync's order (extraneous subdirectories in
|
||||
descending name order, then extraneous files, then kept subdirectories in
|
||||
ascending order) rather than readdir() order, so `--max-delete` leaves the
|
||||
same survivors and the `--info=del`/dry-run line order matches rsync. */
|
||||
static bool delete_walk_fd(int dirfd, const char* rel_path, const DeleteWalkContext* ctx,
|
||||
bool parent_deletable, bool* all_removed) {
|
||||
DeleteDirEntry* entries = NULL;
|
||||
size_t count = 0;
|
||||
bool collect_ok = true;
|
||||
if (!delete_dir_entries_collect(dirfd, &entries, &count, &collect_ok))
|
||||
return false;
|
||||
bool operation_ok = collect_ok;
|
||||
bool local_survives = false;
|
||||
bool* shielded = calloc(count ? count : 1, sizeof(bool));
|
||||
bool* is_extra = calloc(count ? count : 1, sizeof(bool));
|
||||
if (!shielded || !is_extra) {
|
||||
free(shielded);
|
||||
free(is_extra);
|
||||
delete_dir_entries_free(entries, count);
|
||||
return false;
|
||||
}
|
||||
/* A directory is deletable when it or ANY ancestor is synchronized; the
|
||||
`parent_deletable` flag carries that down the recursion so dest-only
|
||||
directories below a synchronized root are removed wholesale. */
|
||||
bool deletable = parent_deletable || is_synced_dir(ctx->dirs, rel_path);
|
||||
bool at_root = rel_path[0] == '\0';
|
||||
|
||||
/* Reproduce rsync's traversal order: extraneous subdirectories in descending
|
||||
name order, then extraneous files in descending name order, and kept
|
||||
subdirectories only afterwards (ascending). Sorting up front also fixes the
|
||||
identity of the survivors under a partial --max-delete. */
|
||||
if (count > 1)
|
||||
qsort(entries, count, sizeof(*entries), delete_dir_entry_cmp_desc);
|
||||
size_t dir_count = 0;
|
||||
while (dir_count < count && entries[dir_count].is_dir)
|
||||
dir_count++;
|
||||
|
||||
if (!delete_walk_classify(rel_path, entries, count, ctx, deletable, at_root, shielded, is_extra,
|
||||
&local_survives))
|
||||
operation_ok = false;
|
||||
if (!delete_walk_extra_dirs(dirfd, rel_path, entries, dir_count, ctx, deletable, is_extra,
|
||||
&local_survives))
|
||||
operation_ok = false;
|
||||
if (!delete_walk_extra_files(dirfd, rel_path, entries, dir_count, count, ctx, is_extra,
|
||||
&local_survives))
|
||||
operation_ok = false;
|
||||
if (!delete_walk_kept_dirs(dirfd, rel_path, entries, dir_count, ctx, deletable, is_extra,
|
||||
shielded, &local_survives))
|
||||
operation_ok = false;
|
||||
|
||||
free(shielded);
|
||||
free(is_extra);
|
||||
delete_dir_entries_free(entries, count);
|
||||
*all_removed = !local_survives;
|
||||
return operation_ok;
|
||||
}
|
||||
|
||||
/* Open the receive root following the same authorized-root confinement the
|
||||
walker uses, or dest_root directly when no authorized root is installed. */
|
||||
static int open_destination_root(const char* dest_root) {
|
||||
int root_fd = utils_get_authorized_root_fd();
|
||||
if (root_fd >= 0) {
|
||||
if (utils_get_authorized_root_path())
|
||||
return utils_open_authorized_destination(dest_root);
|
||||
if (dest_root == NULL)
|
||||
return dup(root_fd);
|
||||
return -1;
|
||||
}
|
||||
return open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
}
|
||||
|
||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||
const DeleteProtectRules* protect, ArrayList* out, size_t* count_out) {
|
||||
if (count_out)
|
||||
*count_out = 0;
|
||||
if (!manifest || !out)
|
||||
return false;
|
||||
PathIndex keep;
|
||||
if (!build_keep_index(manifest, &keep))
|
||||
return false;
|
||||
PathIndex dirs;
|
||||
bool have_dirs = synced_dirs != NULL;
|
||||
if (have_dirs &&
|
||||
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
|
||||
path_index_free(&keep);
|
||||
return false;
|
||||
}
|
||||
int rootfd = open_destination_root(dest_root);
|
||||
if (rootfd < 0) {
|
||||
path_index_free(&keep);
|
||||
if (have_dirs)
|
||||
path_index_free(&dirs);
|
||||
return false;
|
||||
}
|
||||
bool all_removed = false;
|
||||
size_t recorded = 0;
|
||||
DeleteWalkState state = {.mode = DELETE_WALK_MODE_LIST,
|
||||
.budget = NULL,
|
||||
.out = out,
|
||||
.recorded = &recorded,
|
||||
.observer = NULL,
|
||||
.observer_context = NULL};
|
||||
DeleteWalkContext ctx = {.keep = &keep,
|
||||
.dirs = have_dirs ? &dirs : NULL,
|
||||
.state = &state,
|
||||
.skips = skips,
|
||||
.skip_count = skip_count,
|
||||
.protect = protect};
|
||||
bool ok = delete_walk_fd(rootfd, "", &ctx, false, &all_removed);
|
||||
if (close(rootfd) != 0)
|
||||
ok = false;
|
||||
path_index_free(&keep);
|
||||
if (have_dirs)
|
||||
path_index_free(&dirs);
|
||||
if (count_out)
|
||||
*count_out = recorded;
|
||||
return ok;
|
||||
}
|
||||
|
||||
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, size_t max_delete,
|
||||
const DeleteSkipEntry* skips, int skip_count,
|
||||
const DeleteProtectRules* protect,
|
||||
size_t* deleted_out, size_t* skipped_out,
|
||||
DeletePathObserver observer,
|
||||
void* observer_context) {
|
||||
if (deleted_out)
|
||||
*deleted_out = 0;
|
||||
if (skipped_out)
|
||||
*skipped_out = 0;
|
||||
if (!manifest)
|
||||
return DELETE_WALK_ERROR;
|
||||
/* Index the keep-set (and the synchronized-dir set, when supplied) once so
|
||||
membership is answered in O(path length) instead of scanning every entry
|
||||
for every destination entry. */
|
||||
PathIndex keep;
|
||||
if (!build_keep_index(manifest, &keep))
|
||||
return DELETE_WALK_ERROR;
|
||||
PathIndex dirs;
|
||||
bool have_dirs = synced_dirs != NULL;
|
||||
if (have_dirs &&
|
||||
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
|
||||
path_index_free(&keep);
|
||||
return DELETE_WALK_ERROR;
|
||||
}
|
||||
int rootfd = open_destination_root(dest_root);
|
||||
if (rootfd < 0) {
|
||||
path_index_free(&keep);
|
||||
if (have_dirs)
|
||||
path_index_free(&dirs);
|
||||
return DELETE_WALK_ERROR;
|
||||
}
|
||||
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
|
||||
bool all_removed = false;
|
||||
DeleteWalkState state = {.mode = DELETE_WALK_MODE_DELETE,
|
||||
.budget = &budget,
|
||||
.out = NULL,
|
||||
.recorded = NULL,
|
||||
.observer = observer,
|
||||
.observer_context = observer_context};
|
||||
DeleteWalkContext ctx = {.keep = &keep,
|
||||
.dirs = have_dirs ? &dirs : NULL,
|
||||
.state = &state,
|
||||
.skips = skips,
|
||||
.skip_count = skip_count,
|
||||
.protect = protect};
|
||||
bool ok = delete_walk_fd(rootfd, "", &ctx, false, &all_removed);
|
||||
if (close(rootfd) != 0)
|
||||
ok = false;
|
||||
path_index_free(&keep);
|
||||
if (have_dirs)
|
||||
path_index_free(&dirs);
|
||||
if (deleted_out)
|
||||
*deleted_out = budget.deleted;
|
||||
if (skipped_out)
|
||||
*skipped_out = budget.skipped;
|
||||
if (!ok)
|
||||
return DELETE_WALK_ERROR;
|
||||
return budget.limit_hit ? DELETE_WALK_LIMIT_REACHED : DELETE_WALK_OK;
|
||||
}
|
||||
|
||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, size_t max_delete,
|
||||
const DeleteSkipEntry* skips, int skip_count,
|
||||
const DeleteProtectRules* protect, size_t* deleted_out,
|
||||
size_t* skipped_out) {
|
||||
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
|
||||
skip_count, protect, deleted_out, skipped_out, NULL, NULL);
|
||||
}
|
||||
|
||||
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
|
||||
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL, NULL) ==
|
||||
DELETE_WALK_OK;
|
||||
}
|
||||
|
||||
/* Build the delete-walk protection prefix for one basis directory. The walker
|
||||
compares paths relative to the receive root, so a relative entry is already
|
||||
in the right form; an absolute entry that lies below the root is converted to
|
||||
its root-relative form, and one outside the root returns NULL (the walk
|
||||
cannot reach it, and it is not protected data beneath the root). Exposed so
|
||||
tests can exercise the root-of-"/" child mapping directly. */
|
||||
char* delete_basis_relative(const Config* config, const char* path) {
|
||||
if (!path)
|
||||
return NULL;
|
||||
if (path[0] != '/')
|
||||
return str_dup(path);
|
||||
const char* root = config->receive_root_directory;
|
||||
if (!root || root[0] != '/')
|
||||
return NULL;
|
||||
size_t root_len = strlen(root);
|
||||
while (root_len > 1 && root[root_len - 1] == '/')
|
||||
root_len--;
|
||||
if (strncmp(path, root, root_len) != 0)
|
||||
return NULL;
|
||||
if (root_len == 1) {
|
||||
/* `root` is "/" (the only single-character absolute root): every absolute
|
||||
path is below it, and the child relative form is everything after the
|
||||
leading '/'. */
|
||||
if (path[1] == '\0')
|
||||
return NULL; /* identical to the root, not a child */
|
||||
return str_dup(path + 1);
|
||||
}
|
||||
if (path[root_len] != '/')
|
||||
return NULL; /* identical or a sibling sharing a name prefix */
|
||||
return str_dup(path + root_len + 1);
|
||||
}
|
||||
|
||||
bool delete_skips_build(const Config* config, const ArrayList* protected_paths,
|
||||
const ArrayList* size_skipped, bool basis_root_relative,
|
||||
DeleteSkipSet* out) {
|
||||
if (!out)
|
||||
return false;
|
||||
out->entries = NULL;
|
||||
out->owned_prefixes = NULL;
|
||||
out->count = 0;
|
||||
out->owned_count = 0;
|
||||
if (!config)
|
||||
return false;
|
||||
int protected_count = protected_paths ? protected_paths->size : 0;
|
||||
int size_skipped_count = size_skipped ? size_skipped->size : 0;
|
||||
int count =
|
||||
(config->delay_updates ? 1 : 0) + config->basis_count + protected_count + size_skipped_count;
|
||||
if (count == 0)
|
||||
return true;
|
||||
out->entries = calloc((size_t)count, sizeof(DeleteSkipEntry));
|
||||
if (!out->entries)
|
||||
return false;
|
||||
if (basis_root_relative && config->basis_count > 0) {
|
||||
out->owned_prefixes = calloc((size_t)config->basis_count, sizeof(char*));
|
||||
if (!out->owned_prefixes) {
|
||||
free(out->entries);
|
||||
out->entries = NULL;
|
||||
return false;
|
||||
}
|
||||
out->owned_count = config->basis_count;
|
||||
}
|
||||
int idx = 0;
|
||||
if (config->delay_updates) {
|
||||
/* Protect this transfer's actual (per-run unique) staging directory. The
|
||||
runtime name is only known to the receiver-side context; fall back to the
|
||||
reserved prefix for a context that was never created (e.g. a dry run). */
|
||||
const char* staging_name = (config->delay_context && config->delay_context->staging_name)
|
||||
? config->delay_context->staging_name
|
||||
: DELAY_UPDATES_STAGING_DIR;
|
||||
out->entries[idx].prefix = staging_name;
|
||||
out->entries[idx].top_level_only = true;
|
||||
idx++;
|
||||
}
|
||||
for (int i = 0; i < config->basis_count; i++) {
|
||||
const char* prefix = config->basis_dirs[i].path;
|
||||
if (basis_root_relative) {
|
||||
/* An absolute basis outside the receive root is unreachable by this walk,
|
||||
so it contributes no protection prefix (and no slot). */
|
||||
char* relative = delete_basis_relative(config, config->basis_dirs[i].path);
|
||||
if (!relative)
|
||||
continue;
|
||||
out->owned_prefixes[i] = relative;
|
||||
prefix = relative;
|
||||
}
|
||||
out->entries[idx].prefix = prefix;
|
||||
out->entries[idx].top_level_only = false;
|
||||
idx++;
|
||||
}
|
||||
for (int i = 0; i < protected_count; i++) {
|
||||
out->entries[idx].prefix = (const char*)protected_paths->items[i];
|
||||
out->entries[idx].top_level_only = false;
|
||||
idx++;
|
||||
}
|
||||
for (int i = 0; i < size_skipped_count; i++) {
|
||||
out->entries[idx].prefix = (const char*)size_skipped->items[i];
|
||||
out->entries[idx].top_level_only = false;
|
||||
idx++;
|
||||
}
|
||||
out->count = idx;
|
||||
return true;
|
||||
}
|
||||
|
||||
void delete_skips_free(DeleteSkipSet* set) {
|
||||
if (!set)
|
||||
return;
|
||||
if (set->owned_prefixes) {
|
||||
for (int i = 0; i < set->owned_count; i++)
|
||||
free(set->owned_prefixes[i]);
|
||||
}
|
||||
free(set->owned_prefixes);
|
||||
free(set->entries);
|
||||
set->entries = NULL;
|
||||
set->owned_prefixes = NULL;
|
||||
set->count = 0;
|
||||
set->owned_count = 0;
|
||||
}
|
||||
@@ -1,198 +0,0 @@
|
||||
#ifndef DELETE_H
|
||||
#define DELETE_H
|
||||
|
||||
#include "array_list.h"
|
||||
#include "config.h"
|
||||
#include "filter.h"
|
||||
#include <stdbool.h>
|
||||
#include <stddef.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
/* Delete engine.
|
||||
*
|
||||
* This module owns destination-relative delete traversal: the ordered directory
|
||||
* walker that reproduces rsync's extraneous-entry order, the skip-prefix
|
||||
* protection set shared by every delete pass, and the read-only enumeration
|
||||
* that mirrors the walker for -n/--dry-run. The budgeted manifest commit
|
||||
* (delete_commit.c) and the per-directory delete plans (delete_plan.c) are
|
||||
* built on the primitives exported here. */
|
||||
|
||||
/* Result of a bounded extra-file deletion run. */
|
||||
typedef enum {
|
||||
/* Every extra entry was removed (or there were none). */
|
||||
DELETE_WALK_OK = 0,
|
||||
/* The numeric cap for this run was reached before every extra was removed.
|
||||
The walker removed exactly the entries the cap allowed and skipped (without
|
||||
removing) the rest, matching rsync's partial --max-delete behavior. */
|
||||
DELETE_WALK_LIMIT_REACHED,
|
||||
/* A traversal or unlink failure aborted the deletion (partial removal is
|
||||
possible, mirroring the delete pass). */
|
||||
DELETE_WALK_ERROR
|
||||
} DeleteWalkResult;
|
||||
|
||||
/* Receiver-side delete-protection rules for one walk. `base_rules` is the
|
||||
* command-line rule set the config frame carried (owner "" rules); `dir_rules`
|
||||
* is the received per-directory rule set (rules carrying their owner directory
|
||||
* and no-inherit flag). Either may be NULL. */
|
||||
typedef struct {
|
||||
const FilterRuleList* base_rules;
|
||||
const FilterRuleList* dir_rules;
|
||||
/* When non-NULL and non-empty, a destination entry whose name ends with this
|
||||
suffix is protected from deletion. rsync never treats a --backup file as
|
||||
an extra, so a backup created at --delay-updates publication (or a
|
||||
pre-existing one) survives the delete-after pass. */
|
||||
const char* backup_suffix;
|
||||
} DeleteProtectRules;
|
||||
|
||||
/* rsync's first-match-wins receiver verdict for one candidate extra: the
|
||||
* per-directory chain is evaluated first (the containing directory's rules,
|
||||
* then each ancestor's, then the receive root's), then the base rules. Returns
|
||||
* FILTER_ACTION_PROTECT when the entry is shielded by a receiver-side exclude,
|
||||
* FILTER_ACTION_RISK when an include explicitly leaves it at risk, or
|
||||
* FILTER_ACTION_NONE when no rule matched. */
|
||||
FilterAction delete_protect_verdict(const DeleteProtectRules* protect, const char* rel_path,
|
||||
const char* leaf, bool is_dir);
|
||||
|
||||
/* The backup suffix the delete walker must shield from deletion, or NULL when
|
||||
--backup is inactive or the configured suffix is unusable (empty, or holding
|
||||
a path separator). Matches the suffix file_save uses for backups. */
|
||||
const char* delete_backup_suffix(const Config* config);
|
||||
|
||||
/* One protected entry for the delete walker. When top_level_only is true the
|
||||
prefix is skipped only as a DIRECT child of dest_root (the --delay-updates
|
||||
staging directory, which must not hide genuine extras inside a nested
|
||||
destination directory that happens to share the staging name); otherwise the
|
||||
prefix is skipped at any depth (the --compare-dest/--copy-dest/--link-dest
|
||||
basis trees, and the sender-side protected filter-excluded prefixes, which
|
||||
are never destination content). */
|
||||
typedef struct {
|
||||
const char* prefix;
|
||||
bool top_level_only;
|
||||
} DeleteSkipEntry;
|
||||
|
||||
/* A built skip-prefix set. `entries`/`count` are what path_under_skip_prefix()
|
||||
consumes. `owned_prefixes` holds any prefix strings the builder had to
|
||||
allocate (root-relative basis-dir conversions); it is NULL when every prefix
|
||||
is borrowed from the config or the caller's lists. Release with
|
||||
delete_skips_free(). */
|
||||
typedef struct {
|
||||
DeleteSkipEntry* entries;
|
||||
char** owned_prefixes;
|
||||
int count;
|
||||
int owned_count;
|
||||
} DeleteSkipSet;
|
||||
|
||||
/* True when child_rel is, or lies below, one of the protected entries (a prefix
|
||||
"a" protects "a" and "a/b/c" but not "ab"; top_level_only entries protect
|
||||
only DIRECT children of the destination root, i.e. child_rel has no '/'). */
|
||||
bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSkipEntry* skips,
|
||||
int skip_count);
|
||||
|
||||
/* One destination-directory entry collected up front so the delete walkers can
|
||||
reproduce rsync's traversal order instead of readdir() order. rsync processes
|
||||
a directory's extraneous subdirectories first (descending name, depth-first),
|
||||
then its extraneous files (descending name), and only afterwards descends into
|
||||
its kept subdirectories (ascending name). */
|
||||
typedef struct {
|
||||
char* name;
|
||||
bool is_dir;
|
||||
/* The entry's full st_mode from the AT_SYMLINK_NOFOLLOW stat, so a delete
|
||||
observer can classify a removed non-directory as reg/link/special. */
|
||||
mode_t mode;
|
||||
} DeleteDirEntry;
|
||||
/* Collect the entries of the directory open on `dirfd` (excluding "." and ".."),
|
||||
stat'ing each with AT_SYMLINK_NOFOLLOW. On success *out is a malloc'd array of
|
||||
*count entries whose names the caller frees with delete_dir_entries_free().
|
||||
Returns false on an allocation/readdir failure; a vanished entry (ENOENT) is
|
||||
skipped, any other stat failure is reported through *operation_ok while the
|
||||
walk continues. */
|
||||
bool delete_dir_entries_collect(int dirfd, DeleteDirEntry** out, size_t* count, bool* operation_ok);
|
||||
void delete_dir_entries_free(DeleteDirEntry* entries, size_t count);
|
||||
/* Sort comparators: `_desc` orders subdirectories before files and each group by
|
||||
descending name (rsync's extraneous-entry order); `_asc` orders plain ascending
|
||||
name (rsync's kept-subdirectory order). */
|
||||
int delete_dir_entry_cmp_desc(const void* a, const void* b);
|
||||
int delete_dir_entry_cmp_asc(const void* a, const void* b);
|
||||
|
||||
/* Remove files/dirs/symlinks under dest_root that are not listed in manifest
|
||||
without ever descending into a protected prefix (see DeleteSkipEntry). When
|
||||
`synced_dirs` is non-NULL, extras are only removed directly inside a directory
|
||||
whose destination-relative path is an exact entry in that list (the receive
|
||||
root is the "." sentinel); directories outside the synchronized set are still
|
||||
descended into so kept content below a listed directory is preserved, but
|
||||
nothing in them is removed. A NULL `synced_dirs` keeps the legacy behavior of
|
||||
treating the whole destination tree as deletable. `max_delete` caps the
|
||||
number of removed entries (SIZE_MAX = unlimited): the walker removes up to the
|
||||
cap and returns DELETE_WALK_LIMIT_REACHED when more extras remained.
|
||||
`deleted_out`/`skipped_out` optionally receive the number of entries removed
|
||||
and the number skipped because of the cap. */
|
||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, size_t max_delete,
|
||||
const DeleteSkipEntry* skips, int skip_count,
|
||||
const DeleteProtectRules* protect, size_t* deleted_out,
|
||||
size_t* skipped_out);
|
||||
|
||||
/* Entry kind of a removed path, reported to the delete observer so the receiver
|
||||
can build rsync's `--stats` `Number of deleted files` per-type breakdown. The
|
||||
four categories are a strict partition of every removed entry. */
|
||||
typedef enum {
|
||||
DELETE_ENTRY_REG = 0,
|
||||
DELETE_ENTRY_DIR,
|
||||
DELETE_ENTRY_LINK,
|
||||
DELETE_ENTRY_SPECIAL
|
||||
} DeleteEntryType;
|
||||
|
||||
/* Optional per-deletion observer: called for each destination-relative path
|
||||
actually removed (a file, symlink, or directory) with its entry kind, in
|
||||
removal order, so the receiver can stream rsync's `--info=del`/`--info=remove`
|
||||
lines and tally the per-type `--stats` counters. */
|
||||
typedef void (*DeletePathObserver)(void* context, const char* rel_path, DeleteEntryType type);
|
||||
|
||||
/* Classify a removed entry from its st_mode for the per-type delete counters. */
|
||||
DeleteEntryType delete_entry_type_of_mode(mode_t mode);
|
||||
|
||||
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
|
||||
* observer; the observer is invoked only for entries truly removed. When
|
||||
* `protect` is non-NULL its receiver-side verdict is evaluated for every
|
||||
* candidate extra: a first-match PROTECT leaves the entry (and, for a
|
||||
* directory, its whole subtree) in place, while RISK/NONE fall through to the
|
||||
* ordinary skip-prefix/keep-set logic. */
|
||||
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, size_t max_delete,
|
||||
const DeleteSkipEntry* skips, int skip_count,
|
||||
const DeleteProtectRules* protect,
|
||||
size_t* deleted_out, size_t* skipped_out,
|
||||
DeletePathObserver observer,
|
||||
void* observer_context);
|
||||
/* Read-only companion to delete_extras_limited: walk the destination exactly as
|
||||
the delete pass would and APPEND (strdup'd) destination-relative paths that
|
||||
WOULD be removed, without touching disk. Used for -n/--dry-run --delete
|
||||
would-delete reporting. Returns true on a clean walk; the caller owns the
|
||||
strings appended to `out` and receives their count in *count_out. */
|
||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||
const DeleteProtectRules* protect, ArrayList* out, size_t* count_out);
|
||||
bool delete_extras(const char* dest_root, const ArrayList* manifest);
|
||||
|
||||
/* Build the delete walk's skip-prefix set from the config's --delay-updates
|
||||
staging directory, its --compare-dest/--copy-dest/--link-dest basis dirs, and
|
||||
the caller-supplied protection lists, in that order. `protected_paths` and
|
||||
`size_skipped` are borrowed (may be NULL); every entry in them is protected at
|
||||
any depth. The staging directory is protected only as a DIRECT child of the
|
||||
receive root. `basis_root_relative` selects how a basis path becomes a
|
||||
prefix: true converts an absolute path under the receive root to its
|
||||
root-relative form (the whole-tree commit walk; an unreachable path
|
||||
contributes no slot), false keeps the configured path verbatim (the
|
||||
per-directory plan walk). On success the caller releases `*out` with
|
||||
delete_skips_free(); returns false on allocation failure. */
|
||||
bool delete_skips_build(const Config* config, const ArrayList* protected_paths,
|
||||
const ArrayList* size_skipped, bool basis_root_relative,
|
||||
DeleteSkipSet* out);
|
||||
void delete_skips_free(DeleteSkipSet* set);
|
||||
|
||||
/* Convert one basis-directory path to the receive-root-relative protection
|
||||
prefix the delete walker uses (NULL when it lies outside the root). Exposed
|
||||
for unit tests of the root-of-"/" and normalization edge cases. */
|
||||
char* delete_basis_relative(const Config* config, const char* path);
|
||||
|
||||
#endif
|
||||
@@ -1,530 +0,0 @@
|
||||
#include <errno.h>
|
||||
#include <ctype.h>
|
||||
#include <dirent.h>
|
||||
#include <fcntl.h>
|
||||
#include <libgen.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <sys/sysmacros.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include "array_list.h"
|
||||
#include "charset.h"
|
||||
#include "chmod.h"
|
||||
#include "chunk.h"
|
||||
#include "compression.h"
|
||||
#include "config.h"
|
||||
#include "data.h"
|
||||
#include "delay_updates.h"
|
||||
#include "delete_commit.h"
|
||||
#include "delete_plan.h"
|
||||
#include "delta.h"
|
||||
#include "file.h"
|
||||
#include "format.h"
|
||||
#include "identity.h"
|
||||
#include "log.h"
|
||||
#include "metadata.h"
|
||||
#include "protocol.h"
|
||||
#include "utils.h"
|
||||
#include "xattr.h"
|
||||
|
||||
#define MAX_SERVER_DELETE_COUNT 100000U
|
||||
/* Retained cost of one delete-manifest entry beyond its path bytes: the
|
||||
ArrayList pointer slot plus an approximate malloc header/rounding for the
|
||||
heap copy. Charged against MAX_MANIFEST_BYTES so a frame full of tiny paths
|
||||
cannot retain far more than the byte budget (B5). */
|
||||
#define MANIFEST_ENTRY_OVERHEAD (sizeof(char*) + 16)
|
||||
|
||||
/* Read a delete-manifest frame (the STATUS_MANIFEST leading code has already
|
||||
been consumed): a keep-set entry count followed by that many
|
||||
destination-relative paths, then a protected-prefix count followed by that
|
||||
many destination-relative prefixes, then a missing-args count followed by that
|
||||
many destination-relative delete paths, then (protocol 2.23.0) a
|
||||
synchronized-directory count followed by that many destination-relative
|
||||
directory paths (the receive root is the "." sentinel). The frame is
|
||||
self-delimiting (the counts are authoritative), so the caller decides what to
|
||||
do next and continues reading the following STATUS_* frame. Every section is
|
||||
validated identically: an entry must be non-empty, relative and traversal-free
|
||||
and the aggregate length across ALL sections is capped by MAX_MANIFEST_BYTES
|
||||
(so the missing-args deletion requests are confined like the rest of the
|
||||
manifest). Returns an owned DeleteManifest, or NULL after sending STATUS_ERROR
|
||||
when the frame is malformed (bad count, empty/absolute path, path traversal,
|
||||
or an aggregate size beyond MAX_MANIFEST_BYTES). */
|
||||
static bool receive_manifest_section(int fd, ArrayList* list, size_t* manifest_bytes,
|
||||
size_t* manifest_entries) {
|
||||
int count;
|
||||
if (!receive_int(fd, &count)) {
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return false;
|
||||
}
|
||||
if (count < 0 || count > MAX_MANIFEST_ENTRIES ||
|
||||
(size_t)count > MAX_MANIFEST_ENTRIES - *manifest_entries) {
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return false;
|
||||
}
|
||||
for (int i = 0; i < count; i++) {
|
||||
char* s = receive_wire_str(fd);
|
||||
size_t entry_size = s ? strlen(s) + MANIFEST_ENTRY_OVERHEAD : 0;
|
||||
if (!s || s[0] == '\0' || s[0] == '/' || has_path_traversal(s) ||
|
||||
entry_size > MAX_MANIFEST_BYTES - *manifest_bytes ||
|
||||
(*manifest_bytes += entry_size) > MAX_MANIFEST_BYTES || !array_list_add(list, s)) {
|
||||
free(s);
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
*manifest_entries += (size_t)count;
|
||||
return true;
|
||||
}
|
||||
|
||||
DeleteManifest* receive_manifest_entries(int fd) {
|
||||
DeleteManifest* manifest = calloc(1, sizeof(DeleteManifest));
|
||||
if (!manifest) {
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return NULL;
|
||||
}
|
||||
manifest->keeps = array_list_create(free);
|
||||
manifest->protected = array_list_create(free);
|
||||
manifest->missing = array_list_create(free);
|
||||
manifest->dirs = array_list_create(free);
|
||||
if (!manifest->keeps || !manifest->protected || !manifest->missing || !manifest->dirs) {
|
||||
delete_manifest_free(manifest);
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return NULL;
|
||||
}
|
||||
size_t manifest_bytes = 0;
|
||||
size_t manifest_entries = 0;
|
||||
if (!receive_manifest_section(fd, manifest->keeps, &manifest_bytes, &manifest_entries) ||
|
||||
!receive_manifest_section(fd, manifest->protected, &manifest_bytes, &manifest_entries) ||
|
||||
!receive_manifest_section(fd, manifest->missing, &manifest_bytes, &manifest_entries) ||
|
||||
!receive_manifest_section(fd, manifest->dirs, &manifest_bytes, &manifest_entries)) {
|
||||
delete_manifest_free(manifest);
|
||||
return NULL;
|
||||
}
|
||||
/* Per-directory filter rules (protocol 2.30.0) follow the manifest sections
|
||||
* with their own bounded self-describing format. */
|
||||
if (!delete_filter_dir_rules_receive(fd, &manifest->per_dir_rules)) {
|
||||
delete_manifest_free(manifest);
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return NULL;
|
||||
}
|
||||
return manifest;
|
||||
}
|
||||
|
||||
void delete_manifest_free(DeleteManifest* manifest) {
|
||||
if (!manifest)
|
||||
return;
|
||||
array_list_delete(manifest->keeps);
|
||||
array_list_delete(manifest->protected);
|
||||
array_list_delete(manifest->missing);
|
||||
array_list_delete(manifest->dirs);
|
||||
filter_rule_list_free(manifest->per_dir_rules);
|
||||
free(manifest);
|
||||
}
|
||||
|
||||
/* Shared --max-delete budget for one receiver-side deletion commit. Both the
|
||||
--delete-missing-args exact-path removals and the ordinary extras walk draw
|
||||
from the same tally, matching rsync (whose --max-delete counts every deleted
|
||||
file or directory). `max_delete` is SIZE_MAX for an unlimited budget. */
|
||||
typedef struct {
|
||||
size_t max_delete;
|
||||
size_t deleted;
|
||||
size_t skipped;
|
||||
bool limit_hit;
|
||||
} DeleteBudgetState;
|
||||
|
||||
/* Remove every destination entry under the receive root that is not in the
|
||||
keep-set, bounded by the shared budget (a smaller client --max-delete=NUM
|
||||
replaces the server hard bound; rsync deletes up to the bound and skips the
|
||||
rest). With --delay-updates the not-yet-published staging directory is a
|
||||
direct child of the receive root and must not be treated as a set of extras;
|
||||
the manifest's protected prefixes (paths excluded on the source), the
|
||||
size-pruned prefixes (--max-size/--min-size, always protected) and the
|
||||
alternate basis directories are never destination content and are skipped at
|
||||
any depth. Returns true unless a traversal/unlink error aborted the walk;
|
||||
the budget's limit_hit/skipped fields report a cap-stopped run. */
|
||||
static bool delete_extras_budgeted_observed(const Config* config, const DeleteManifest* manifest,
|
||||
DeleteBudgetState* budget, DeletePathObserver observer,
|
||||
void* observer_context) {
|
||||
if (!config || !manifest || !manifest->keeps)
|
||||
return false;
|
||||
fprintf(stderr, "Deleting files not in manifest...\n");
|
||||
/* Protected entries: the --delay-updates staging name (only as a DIRECT child
|
||||
of the receive root), the alternate basis directories and the sender-side
|
||||
protected prefixes (filter-excluded and size-pruned source mirrors), all at
|
||||
any depth. See delete_skips_build(). */
|
||||
DeleteSkipSet skips;
|
||||
if (!delete_skips_build(config, manifest->protected, NULL, true, &skips))
|
||||
return false;
|
||||
/* Clamp rather than subtract: an accounting bug where deleted already exceeds
|
||||
max_delete must never underflow into an effectively unlimited budget. */
|
||||
size_t remaining;
|
||||
if (budget->max_delete == SIZE_MAX)
|
||||
remaining = SIZE_MAX;
|
||||
else if (budget->deleted >= budget->max_delete)
|
||||
remaining = 0;
|
||||
else
|
||||
remaining = budget->max_delete - budget->deleted;
|
||||
size_t deleted = 0;
|
||||
size_t skipped = 0;
|
||||
DeleteProtectRules protect = {.base_rules = config->protect_rules,
|
||||
.dir_rules = manifest->per_dir_rules,
|
||||
.backup_suffix = delete_backup_suffix(config)};
|
||||
DeleteWalkResult result = delete_extras_limited_observed(
|
||||
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips.entries,
|
||||
skips.count, &protect, &deleted, &skipped, observer, observer_context);
|
||||
delete_skips_free(&skips);
|
||||
budget->deleted += deleted;
|
||||
budget->skipped += skipped;
|
||||
if (result == DELETE_WALK_LIMIT_REACHED) {
|
||||
budget->limit_hit = true;
|
||||
return true;
|
||||
}
|
||||
if (result != DELETE_WALK_OK) {
|
||||
log_message(LOG_LEVEL_ERROR, "deletion failed while removing extraneous files");
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool delete_extras_budgeted(const Config* config, const DeleteManifest* manifest,
|
||||
DeleteBudgetState* budget) {
|
||||
return delete_extras_budgeted_observed(config, manifest, budget, NULL, NULL);
|
||||
}
|
||||
|
||||
/* Prefixes every observed path with a fixed subtree root, so a nested walk
|
||||
(a recursively removed missing-arg directory) reports receive-root-relative
|
||||
names like the rest of the delete output. */
|
||||
typedef struct {
|
||||
DeletePathObserver inner;
|
||||
void* inner_context;
|
||||
const char* prefix;
|
||||
} PrefixedDeleteObserver;
|
||||
|
||||
static void prefixed_delete_observer(void* context, const char* rel, DeleteEntryType type) {
|
||||
PrefixedDeleteObserver* prefixed = context;
|
||||
if (!prefixed->inner || !rel)
|
||||
return;
|
||||
char* joined = path_cat((char*)prefixed->prefix, rel);
|
||||
if (joined) {
|
||||
prefixed->inner(prefixed->inner_context, joined, type);
|
||||
free(joined);
|
||||
}
|
||||
}
|
||||
|
||||
/* --delete-missing-args exact-path deletions: each destination mirror in
|
||||
manifest->missing is an explicit user request, so it is removed even when the
|
||||
ordinary extras walk (with its protected prefixes) would leave it alone. The
|
||||
--delay-updates staging directory and basis snapshots are receiver artifacts
|
||||
and stay protected exactly as in the extras walker. A regular file or
|
||||
symlink is unlinked, an empty directory removed, and a NON-empty directory is
|
||||
removed recursively only when --delete or --force is in effect (rsync parity:
|
||||
the man page says a non-empty directory mirror is only deleted with --force
|
||||
or --delete); otherwise it is left with a warning and the run continues. A
|
||||
mirror that does not exist is a no-op. Each removal draws from the shared
|
||||
--max-delete budget: once it is exhausted the remaining requests are skipped
|
||||
and counted. Returns false only on a genuine error (a confinement failure on
|
||||
a validated path or an I/O error), which fails the run. */
|
||||
|
||||
/* How one missing-args request leaves the driver loop. The original walker
|
||||
`continue`s past an invalid/protected/absent/budget-skipped request (without
|
||||
breaking) but stops after a request that ran to completion while an error is
|
||||
pending; NEXT/STOP preserve that control flow exactly. */
|
||||
typedef enum { MISSING_ARG_NEXT, MISSING_ARG_STOP } MissingArgStep;
|
||||
|
||||
/* Remove a NON-empty missing-args directory recursively (--delete/--force in
|
||||
effect): walk its contents through the budgeted extras walker so every removed
|
||||
file/dir counts toward --max-delete (rsync parity), then remove the now-empty
|
||||
directory itself, which costs one more budget unit. A run that hits the cap
|
||||
leaves the remaining entries in place. The observer is wrapped so the nested
|
||||
walk reports receive-root-relative paths. Sets the *removed and *ok outputs. */
|
||||
static void delete_nonempty_missing_dir(const char* full, const char* rel,
|
||||
DeleteBudgetState* budget, DeletePathObserver observer,
|
||||
void* observer_context, bool* removed, bool* ok) {
|
||||
ArrayList* no_keeps = array_list_create(free);
|
||||
/* Never let an accounting slip (deleted > max_delete) underflow the remaining
|
||||
budget into SIZE_MAX, which would grant unlimited deletions. */
|
||||
size_t remaining =
|
||||
budget->deleted >= budget->max_delete ? 0 : budget->max_delete - budget->deleted;
|
||||
size_t contents_deleted = 0;
|
||||
size_t contents_skipped = 0;
|
||||
PrefixedDeleteObserver nested = {observer, observer_context, rel};
|
||||
DeleteWalkResult walk =
|
||||
no_keeps ? delete_extras_limited_observed(full, no_keeps, NULL, remaining, NULL, 0, NULL,
|
||||
&contents_deleted, &contents_skipped,
|
||||
observer ? prefixed_delete_observer : NULL,
|
||||
observer ? &nested : NULL)
|
||||
: DELETE_WALK_ERROR;
|
||||
if (no_keeps)
|
||||
array_list_delete(no_keeps);
|
||||
budget->deleted += contents_deleted;
|
||||
budget->skipped += contents_skipped;
|
||||
if (walk == DELETE_WALK_LIMIT_REACHED) {
|
||||
budget->limit_hit = true;
|
||||
} else if (walk != DELETE_WALK_OK) {
|
||||
*ok = false;
|
||||
} else if (budget->deleted >= budget->max_delete) {
|
||||
budget->limit_hit = true;
|
||||
budget->skipped++;
|
||||
} else if (file_remove_tree_secure(full)) {
|
||||
/* The shared `if (removed)` tail charges this directory exactly once;
|
||||
counting it here too would consume two budget units. */
|
||||
*removed = true;
|
||||
} else {
|
||||
*ok = false;
|
||||
}
|
||||
}
|
||||
|
||||
/* Remove one missing-args destination mirror. `skips` holds the receiver
|
||||
artifacts (staging directory, basis snapshots) that stay protected. Returns
|
||||
MISSING_ARG_STOP when the driver loop must stop (a completed removal left a
|
||||
genuine error pending) and MISSING_ARG_NEXT otherwise; *ok accumulates the
|
||||
overall success across the whole run. */
|
||||
static MissingArgStep delete_one_missing_arg(const Config* config, const char* rel,
|
||||
const DeleteSkipSet* skips, DeleteBudgetState* budget,
|
||||
DeletePathObserver observer, void* observer_context,
|
||||
bool* ok) {
|
||||
if (!rel || *rel == '\0' || *rel == '/' || has_path_traversal(rel)) {
|
||||
/* Defensive only: receive_manifest_entries already validated every
|
||||
section identically, so a controlled peer never reaches this branch. */
|
||||
log_message(LOG_LEVEL_ERROR, "invalid missing-args delete path");
|
||||
*ok = false;
|
||||
return MISSING_ARG_NEXT;
|
||||
}
|
||||
bool at_root = strchr(rel, '/') == NULL;
|
||||
if (path_under_skip_prefix(rel, at_root, skips->entries, skips->count)) {
|
||||
char* escaped = output_escape(rel, log_get_8_bit_output());
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"missing-args path '%s' is protected (staging directory or basis snapshot); "
|
||||
"not deleting",
|
||||
escaped ? escaped : "<allocation failed>");
|
||||
free(escaped);
|
||||
return MISSING_ARG_NEXT;
|
||||
}
|
||||
char* full = path_cat(config->receive_root_directory, rel);
|
||||
if (!full) {
|
||||
*ok = false;
|
||||
return MISSING_ARG_NEXT;
|
||||
}
|
||||
char* leaf = NULL;
|
||||
int parent_fd = file_open_secure_parent(full, &leaf, false);
|
||||
if (parent_fd < 0) {
|
||||
/* The mirror's parent directory may itself not exist on the destination
|
||||
(a deeper missing entry whose leading directories were never created).
|
||||
That is a no-op -- there is nothing to delete -- matching
|
||||
file_remove_tree_secure's absent-path handling; only a genuine I/O
|
||||
error (EACCES, a symlink loop, ...) fails the run. */
|
||||
bool absent = errno == ENOENT || errno == ENOTDIR;
|
||||
free(full);
|
||||
free(leaf);
|
||||
if (!absent)
|
||||
*ok = false;
|
||||
return MISSING_ARG_NEXT;
|
||||
}
|
||||
struct stat st;
|
||||
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
||||
/* Already absent: nothing to delete (a no-op, not a deletion). */
|
||||
if (errno != ENOENT)
|
||||
*ok = false;
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
free(full);
|
||||
return MISSING_ARG_NEXT;
|
||||
}
|
||||
/* An entry that exists is one deletion: skip it (and count it) when the
|
||||
shared --max-delete budget is already exhausted. */
|
||||
if (budget->deleted >= budget->max_delete) {
|
||||
budget->limit_hit = true;
|
||||
budget->skipped++;
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
free(full);
|
||||
return MISSING_ARG_NEXT;
|
||||
}
|
||||
bool removed = false;
|
||||
if (S_ISDIR(st.st_mode)) {
|
||||
if (unlinkat(parent_fd, leaf, AT_REMOVEDIR) == 0) {
|
||||
removed = true;
|
||||
} else if (errno == ENOTEMPTY || errno == EEXIST) {
|
||||
close(parent_fd);
|
||||
parent_fd = -1;
|
||||
free(leaf);
|
||||
leaf = NULL;
|
||||
if (config->use_delete || config->force_delete) {
|
||||
delete_nonempty_missing_dir(full, rel, budget, observer, observer_context, &removed, ok);
|
||||
} else {
|
||||
char* escaped = output_escape(rel, log_get_8_bit_output());
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"missing-args destination '%s' is a non-empty directory; use --force or "
|
||||
"--delete to remove it",
|
||||
escaped ? escaped : "<allocation failed>");
|
||||
free(escaped);
|
||||
}
|
||||
} else if (errno != ENOENT) {
|
||||
*ok = false;
|
||||
}
|
||||
} else {
|
||||
if (unlinkat(parent_fd, leaf, 0) == 0) {
|
||||
removed = true;
|
||||
} else if (errno != ENOENT) {
|
||||
*ok = false;
|
||||
}
|
||||
}
|
||||
if (removed) {
|
||||
budget->deleted++;
|
||||
if (observer)
|
||||
observer(observer_context, rel, delete_entry_type_of_mode(st.st_mode));
|
||||
char* escaped = output_escape(rel, log_get_8_bit_output());
|
||||
fprintf(stderr, " Deleted: %s\n", escaped ? escaped : "<allocation failed>");
|
||||
free(escaped);
|
||||
}
|
||||
if (parent_fd >= 0)
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
free(full);
|
||||
return *ok ? MISSING_ARG_NEXT : MISSING_ARG_STOP;
|
||||
}
|
||||
|
||||
static bool delete_missing_args_budgeted_observed(const Config* config,
|
||||
const DeleteManifest* manifest,
|
||||
DeleteBudgetState* budget,
|
||||
DeletePathObserver observer,
|
||||
void* observer_context) {
|
||||
if (!config || !manifest)
|
||||
return false;
|
||||
if (!manifest->missing || manifest->missing->size == 0)
|
||||
return true;
|
||||
fprintf(stderr, "Deleting destination mirrors of missing source arguments...\n");
|
||||
/* The staging directory and basis snapshots stay protected exactly as in the
|
||||
extras walker (the missing-args path overrides the ordinary protected
|
||||
prefixes, so those are not passed here). */
|
||||
DeleteSkipSet skips;
|
||||
if (!delete_skips_build(config, NULL, NULL, true, &skips))
|
||||
return false;
|
||||
bool ok = true;
|
||||
for (int i = 0; i < manifest->missing->size; i++) {
|
||||
const char* rel = (const char*)manifest->missing->items[i];
|
||||
if (delete_one_missing_arg(config, rel, &skips, budget, observer, observer_context, &ok) ==
|
||||
MISSING_ARG_STOP)
|
||||
break;
|
||||
}
|
||||
delete_skips_free(&skips);
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Public wrappers used outside the commit path (and by unit tests): no
|
||||
--max-delete budget. */
|
||||
bool manifest_would_delete_list(const Config* config, const DeleteManifest* manifest,
|
||||
ArrayList* out, size_t* count_out) {
|
||||
if (count_out)
|
||||
*count_out = 0;
|
||||
if (!config || !manifest || !manifest->keeps || !out)
|
||||
return false;
|
||||
DeleteSkipSet skips;
|
||||
if (!delete_skips_build(config, manifest->protected, NULL, true, &skips))
|
||||
return false;
|
||||
DeleteProtectRules protect = {.base_rules = config->protect_rules,
|
||||
.dir_rules = manifest->per_dir_rules,
|
||||
.backup_suffix = delete_backup_suffix(config)};
|
||||
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
|
||||
skips.entries, skips.count, &protect, out, count_out);
|
||||
delete_skips_free(&skips);
|
||||
return ok;
|
||||
}
|
||||
|
||||
bool manifest_delete_extras(const Config* config, const DeleteManifest* manifest) {
|
||||
DeleteBudgetState budget = {
|
||||
.max_delete = SIZE_MAX, .deleted = 0, .skipped = 0, .limit_hit = false};
|
||||
return delete_extras_budgeted(config, manifest, &budget);
|
||||
}
|
||||
|
||||
bool manifest_delete_missing_args(const Config* config, const DeleteManifest* manifest) {
|
||||
DeleteBudgetState budget = {
|
||||
.max_delete = SIZE_MAX, .deleted = 0, .skipped = 0, .limit_hit = false};
|
||||
return delete_missing_args_budgeted_observed(config, manifest, &budget, NULL, NULL);
|
||||
}
|
||||
|
||||
bool manifest_delete_missing_args_limited(const Config* config, const DeleteManifest* manifest,
|
||||
size_t max_delete, size_t* deleted, size_t* skipped,
|
||||
bool* limit_hit) {
|
||||
return manifest_delete_missing_args_limited_observed(config, manifest, max_delete, deleted,
|
||||
skipped, limit_hit, NULL, NULL);
|
||||
}
|
||||
|
||||
bool manifest_delete_missing_args_limited_observed(
|
||||
const Config* config, const DeleteManifest* manifest, size_t max_delete, size_t* deleted,
|
||||
size_t* skipped, bool* limit_hit, DeletePathObserver observer, void* observer_context) {
|
||||
DeleteBudgetState budget = {
|
||||
.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
|
||||
bool ok =
|
||||
delete_missing_args_budgeted_observed(config, manifest, &budget, observer, observer_context);
|
||||
if (deleted)
|
||||
*deleted = budget.deleted;
|
||||
if (skipped)
|
||||
*skipped = budget.skipped;
|
||||
if (limit_hit)
|
||||
*limit_hit = budget.limit_hit;
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* Commit every deletion family the manifest carries. The --delete-missing-args
|
||||
exact-path deletions run FIRST: they are explicit user requests and must not
|
||||
be blocked by the extras walker's filter-exclusion protection (a protected
|
||||
leftover inside a missing-argument directory must not make that user-requested
|
||||
removal fail). The ordinary extras walk then runs when --delete is active.
|
||||
Both draw from one --max-delete budget; the result reports a cap-stopped
|
||||
(partial) commit distinctly so the client can exit 25 like rsync. */
|
||||
DeleteCommitResult manifest_delete_all(const Config* config, const DeleteManifest* manifest) {
|
||||
return manifest_delete_all_counted(config, manifest, NULL);
|
||||
}
|
||||
|
||||
DeleteCommitResult manifest_delete_all_counted(const Config* config, const DeleteManifest* manifest,
|
||||
size_t* deleted) {
|
||||
return manifest_delete_all_observed(config, manifest, deleted, NULL, NULL);
|
||||
}
|
||||
|
||||
DeleteCommitResult manifest_delete_all_observed(const Config* config,
|
||||
const DeleteManifest* manifest, size_t* deleted,
|
||||
DeletePathObserver observer,
|
||||
void* observer_context) {
|
||||
if (deleted)
|
||||
*deleted = 0;
|
||||
if (!config || !manifest)
|
||||
return DELETE_COMMIT_ERROR;
|
||||
/* Central no-mutation guard: a dry-run never deletes. No manifest is sent on
|
||||
the dry-run path, but a hostile/buggy peer could; treat it as a no-op so
|
||||
the receiver can never remove anything. */
|
||||
if (config->dry_run)
|
||||
return DELETE_COMMIT_OK;
|
||||
/* A client --max-delete=NUM smaller than the server's hard bound replaces it
|
||||
for this run; both still bound the commit. */
|
||||
bool user_limited =
|
||||
config->max_delete >= 0 && (size_t)config->max_delete < MAX_SERVER_DELETE_COUNT;
|
||||
DeleteBudgetState budget = {.max_delete = user_limited ? (size_t)config->max_delete
|
||||
: MAX_SERVER_DELETE_COUNT,
|
||||
.deleted = 0,
|
||||
.skipped = 0,
|
||||
.limit_hit = false};
|
||||
if (config->delete_missing_args &&
|
||||
!delete_missing_args_budgeted_observed(config, manifest, &budget, observer, observer_context))
|
||||
return DELETE_COMMIT_ERROR;
|
||||
if (config->use_delete &&
|
||||
!delete_extras_budgeted_observed(config, manifest, &budget, observer, observer_context))
|
||||
return DELETE_COMMIT_ERROR;
|
||||
if (deleted)
|
||||
*deleted = budget.deleted;
|
||||
if (budget.limit_hit) {
|
||||
if (user_limited) {
|
||||
log_message(LOG_LEVEL_ERROR, "Deletions stopped due to --max-delete limit (%zu skipped)",
|
||||
budget.skipped);
|
||||
} else {
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"Deletions stopped due to the server deletion limit of %u (%zu skipped)",
|
||||
(unsigned)MAX_SERVER_DELETE_COUNT, budget.skipped);
|
||||
}
|
||||
return DELETE_COMMIT_LIMIT_REACHED;
|
||||
}
|
||||
return DELETE_COMMIT_OK;
|
||||
}
|
||||
@@ -1,113 +0,0 @@
|
||||
#ifndef DELETE_COMMIT_H
|
||||
#define DELETE_COMMIT_H
|
||||
|
||||
#include "array_list.h"
|
||||
#include "config.h"
|
||||
#include "delete.h"
|
||||
#include "filter.h"
|
||||
#include <stdbool.h>
|
||||
|
||||
/* Delete-commit module: delete-manifest receive plus the budgeted extras and
|
||||
* --delete-missing-args walkers. These declarations are re-exported by the
|
||||
* file_receive.h facade. */
|
||||
|
||||
/* A received delete-manifest frame: the keep-set (`keeps`, destination-relative
|
||||
paths the sender transferred/keeps) plus `protected`, destination-relative
|
||||
prefixes the sender asks the receiver never to delete (paths excluded on the
|
||||
source, protected at any depth). When --delete-excluded is given the sender
|
||||
transmits an empty protected list so excluded destination mirrors are treated
|
||||
as ordinary extras. With --delete-missing-args a third section (`missing`)
|
||||
carries the destination mirrors of explicitly-listed source entries that do
|
||||
not exist: each is an exact deletion request, independent of the ordinary
|
||||
extras walk (never blocked by the protected prefixes) and processed when the
|
||||
manifest is committed. */
|
||||
typedef struct DeleteManifest {
|
||||
ArrayList* keeps;
|
||||
ArrayList* protected;
|
||||
ArrayList* missing;
|
||||
/* Destination-relative paths of the directories the sender synchronized for
|
||||
this run. The extras walker only removes entries directly inside one of
|
||||
these (the receive root is the "." sentinel); `--files-from` runs therefore
|
||||
leave untransmitted directories and the unlisted parts of listed ones
|
||||
alone, matching rsync's "delete only in synchronized directories". */
|
||||
ArrayList* dirs;
|
||||
/* Per-directory filter rules the sender compiled while scanning (protocol
|
||||
2.30.0), each carrying its owner directory and no-inherit flag. The
|
||||
receiver evaluates them (deepest before ancestors, then the command-line
|
||||
base rules) against every candidate extra so a destination-only entry that
|
||||
matches ONLY a per-directory `.rsync-filter`/dir-merge rule is shielded.
|
||||
NULL when the sender transmitted none. */
|
||||
FilterRuleList* per_dir_rules;
|
||||
} DeleteManifest;
|
||||
|
||||
void delete_manifest_free(DeleteManifest* manifest);
|
||||
/* Read a delete-manifest frame (protocol 2.23.0): keep count + keeps, then
|
||||
protected count + protected prefixes, then missing count + missing paths,
|
||||
then synchronized-directory count + directory paths (self-delimiting; the
|
||||
leading STATUS_MANIFEST code has been consumed). Returns an owned
|
||||
DeleteManifest, or NULL after signalling STATUS_ERROR on a malformed frame. */
|
||||
DeleteManifest* receive_manifest_entries(int fd);
|
||||
/* Remove destination entries under config->receive_root_directory that are not
|
||||
in `manifest` (bounded, all-or-nothing walk; staging-dir, basis-dir and
|
||||
protected-prefix skips). `--max-delete` and `--force` are honored here. The
|
||||
caller decides WHEN to run it based on the negotiated delete timing. Returns
|
||||
false (and the transfer fails) when the deletion cannot be committed. */
|
||||
bool manifest_delete_extras(const Config* config, const DeleteManifest* manifest);
|
||||
/* --delete-missing-args exact-path deletions: remove each destination mirror
|
||||
in `manifest->missing` (never blocked by the protected prefixes, staging dir
|
||||
and basis dirs excluded). A regular file/symlink is unlinked; an empty
|
||||
directory is removed; a NON-empty directory is removed recursively only when
|
||||
--delete or --force is in effect, otherwise it is left with a warning (rsync
|
||||
parity). A missing path is a no-op. Returns false only on a genuine
|
||||
confinement or I/O error (the run then fails); tolerated per-path cases are
|
||||
reported and skipped. */
|
||||
bool manifest_delete_missing_args(const Config* config, const DeleteManifest* manifest);
|
||||
/* Budgeted form of manifest_delete_missing_args for the per-directory delete
|
||||
session: each removed mirror draws from `max_delete` (SIZE_MAX = unlimited)
|
||||
and the tallies are accumulated into `*deleted`/`*skipped`. `*limit_hit` is set
|
||||
when the budget stopped the pass with entries left over. Returns false only
|
||||
on a genuine deletion error. */
|
||||
bool manifest_delete_missing_args_limited(const Config* config, const DeleteManifest* manifest,
|
||||
size_t max_delete, size_t* deleted, size_t* skipped,
|
||||
bool* limit_hit);
|
||||
/* Observer-aware form of manifest_delete_missing_args_limited: `observer` (may
|
||||
be NULL) is invoked for every destination-relative path truly removed. */
|
||||
bool manifest_delete_missing_args_limited_observed(
|
||||
const Config* config, const DeleteManifest* manifest, size_t max_delete, size_t* deleted,
|
||||
size_t* skipped, bool* limit_hit, DeletePathObserver observer, void* observer_context);
|
||||
/* Outcome of committing a delete manifest. LIMIT_REACHED reports rsync's
|
||||
partial --max-delete result: the budget allowed some deletions and the rest
|
||||
were skipped (the run still stores all file data but the client exits 25). */
|
||||
typedef enum {
|
||||
DELETE_COMMIT_OK = 0,
|
||||
DELETE_COMMIT_LIMIT_REACHED,
|
||||
DELETE_COMMIT_ERROR
|
||||
} DeleteCommitResult;
|
||||
|
||||
/* Run every deletion family the manifest carries: the --delete-missing-args
|
||||
exact-path deletions first (user requests are not blocked by exclusion
|
||||
protection), then the ordinary extras walk when --delete is active. Both
|
||||
share one --max-delete budget. Returns DELETE_COMMIT_OK when nothing was to
|
||||
do or everything committed, DELETE_COMMIT_LIMIT_REACHED when the budget
|
||||
stopped part of the work, or DELETE_COMMIT_ERROR on a genuine failure. */
|
||||
DeleteCommitResult manifest_delete_all(const Config* config, const DeleteManifest* manifest);
|
||||
/* Like manifest_delete_all, but reports how many destination entries the commit
|
||||
removed (for the end-of-transfer wire stats). `deleted` may be NULL. */
|
||||
DeleteCommitResult manifest_delete_all_counted(const Config* config, const DeleteManifest* manifest,
|
||||
size_t* deleted);
|
||||
/* Observer-aware form of manifest_delete_all_counted: `observer` (may be NULL)
|
||||
is invoked for every destination-relative path truly removed. */
|
||||
DeleteCommitResult manifest_delete_all_observed(const Config* config,
|
||||
const DeleteManifest* manifest, size_t* deleted,
|
||||
DeletePathObserver observer,
|
||||
void* observer_context);
|
||||
|
||||
/* -n/--dry-run --delete would-delete reporting: walk the destination exactly as
|
||||
the delete pass would and append (strdup'd) destination-relative paths that
|
||||
WOULD be removed to `out`, without touching disk. Uses the same staging-dir,
|
||||
basis-dir and protected-prefix skips as the real commit. Returns true on a
|
||||
clean walk; `*count_out` receives the number of paths appended. */
|
||||
bool manifest_would_delete_list(const Config* config, const DeleteManifest* manifest,
|
||||
ArrayList* out, size_t* count_out);
|
||||
|
||||
#endif
|
||||
+113
-482
@@ -2,7 +2,6 @@
|
||||
|
||||
#include "charset.h"
|
||||
#include "delay_updates.h"
|
||||
#include "delete.h"
|
||||
#include "file.h"
|
||||
#include "log.h"
|
||||
#include "utils.h"
|
||||
@@ -48,7 +47,6 @@ struct DeletePlanSender {
|
||||
const ArrayList* protected_prefixes;
|
||||
const ArrayList* size_skipped;
|
||||
const ArrayList* missing_args;
|
||||
const FilterRuleList* per_dir_rules;
|
||||
size_t entries;
|
||||
/* Transmitted FILE entries only. The caller's "empty scan" safety guard keys
|
||||
off this (an I/O error that hid every file must refuse to delete even when
|
||||
@@ -285,14 +283,12 @@ bool delete_plan_sender_empty(const DeletePlanSender* sender) {
|
||||
}
|
||||
|
||||
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
||||
const ArrayList* size_skipped, const ArrayList* missing_args,
|
||||
const FilterRuleList* per_dir_rules) {
|
||||
const ArrayList* size_skipped, const ArrayList* missing_args) {
|
||||
if (!sender)
|
||||
return;
|
||||
sender->protected_prefixes = protected_prefixes;
|
||||
sender->size_skipped = size_skipped;
|
||||
sender->missing_args = missing_args;
|
||||
sender->per_dir_rules = per_dir_rules;
|
||||
}
|
||||
|
||||
/* True when `dir` is `root` itself or a descendant of it (path-component
|
||||
@@ -323,181 +319,6 @@ static int send_str_section(int fd, const ArrayList* list) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* The directory a rule belongs to (its owner, or the transfer root for ""). */
|
||||
static const char* filter_dir_rule_owner(const FilterRule* rule) {
|
||||
return (rule && rule->owner) ? rule->owner : "";
|
||||
}
|
||||
|
||||
/* Transmit the received-side per-directory filter rules (protocol 2.30.0) as a
|
||||
* self-describing list of directory groups: a group count, then for each group
|
||||
* the relative owner directory followed by that directory's rule records (run
|
||||
* order = the sender's traversal/rule order). Rules of one directory are
|
||||
* appended to the sink contiguously, so runs reproduce the compilation order.
|
||||
* Bounded by MAX_FILTER_RULES / MAX_FILTER_BYTES and MAX_PROTECT_PATTERN_LEN so
|
||||
* the peer never sees a frame it would reject. The sender enforces exactly the
|
||||
* receiver's limits (including the cumulative owner+pattern byte budget) and
|
||||
* fails with a clear local error instead of emitting a frame that would abort
|
||||
* the transfer with STATUS_ERROR. */
|
||||
bool delete_filter_dir_rules_send(int fd, const FilterRuleList* rules) {
|
||||
int count = rules ? rules->count : 0;
|
||||
if (count < 0 || count > MAX_FILTER_RULES) {
|
||||
log_message(LOG_LEVEL_ERROR, "too many per-directory filter rules: %d (maximum %d)", count,
|
||||
MAX_FILTER_RULES);
|
||||
return false;
|
||||
}
|
||||
size_t bytes = 0;
|
||||
for (int i = 0; i < count; i++) {
|
||||
const FilterRule* rule = rules->items[i];
|
||||
const char* owner = filter_dir_rule_owner(rule);
|
||||
size_t owner_len = strlen(owner);
|
||||
size_t pattern_len = rule && rule->pattern ? strlen(rule->pattern) : 0;
|
||||
if (!rule || !rule->pattern || pattern_len == 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "invalid per-directory filter pattern");
|
||||
return false;
|
||||
}
|
||||
if (pattern_len > MAX_PROTECT_PATTERN_LEN) {
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"per-directory filter pattern exceeds %d bytes (use a shorter pattern)",
|
||||
MAX_PROTECT_PATTERN_LEN);
|
||||
return false;
|
||||
}
|
||||
if (!(owner_len == 0 || (owner[0] != '/' && !has_path_traversal(owner)))) {
|
||||
log_message(LOG_LEVEL_ERROR, "invalid per-directory filter owner directory");
|
||||
return false;
|
||||
}
|
||||
if (owner_len + pattern_len > MAX_FILTER_BYTES - bytes) {
|
||||
log_message(LOG_LEVEL_ERROR, "per-directory filter rules exceed %d bytes", MAX_FILTER_BYTES);
|
||||
return false;
|
||||
}
|
||||
bytes += owner_len + pattern_len;
|
||||
}
|
||||
int groups = 0;
|
||||
for (int i = 0; i < count;) {
|
||||
const char* owner = filter_dir_rule_owner(rules->items[i]);
|
||||
groups++;
|
||||
i++;
|
||||
while (i < count && strcmp(filter_dir_rule_owner(rules->items[i]), owner) == 0)
|
||||
i++;
|
||||
}
|
||||
if (!send_int(fd, groups))
|
||||
return false;
|
||||
for (int i = 0; i < count;) {
|
||||
const char* owner = filter_dir_rule_owner(rules->items[i]);
|
||||
int start = i;
|
||||
i++;
|
||||
while (i < count && strcmp(filter_dir_rule_owner(rules->items[i]), owner) == 0)
|
||||
i++;
|
||||
if (!send_wire_str(fd, owner) || !send_int(fd, i - start))
|
||||
return false;
|
||||
for (int j = start; j < i; j++) {
|
||||
const FilterRule* rule = rules->items[j];
|
||||
if (!send_int(fd, (int)rule->action) || !send_int(fd, (int)rule->sides) ||
|
||||
!send_int(fd, rule->anchored ? 1 : 0) || !send_int(fd, rule->dir_only ? 1 : 0) ||
|
||||
!send_int(fd, rule->negate ? 1 : 0) || !send_int(fd, rule->no_inherit ? 1 : 0) ||
|
||||
!send_wire_str(fd, rule->pattern))
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Read one wire flag (an int restricted to 0/1). */
|
||||
static bool receive_flag(int fd, bool* value) {
|
||||
int raw;
|
||||
if (!receive_int(fd, &raw) || (raw != 0 && raw != 1))
|
||||
return false;
|
||||
*value = raw != 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Read the per-directory filter block emitted by delete_filter_dir_rules_send.
|
||||
* Reconstructs a flat FilterRuleList whose rules carry their owner directory;
|
||||
* `*out` is NULL when the sender transmitted no rules. Every bound is enforced
|
||||
* (group/rule counts, owner/pattern bytes, pattern length, action/sides domain)
|
||||
* so a malicious peer can neither overread nor allocate unboundedly. Returns
|
||||
* false on a malformed frame (the caller signals STATUS_ERROR). */
|
||||
bool delete_filter_dir_rules_receive(int fd, FilterRuleList** out) {
|
||||
if (!out)
|
||||
return false;
|
||||
*out = NULL;
|
||||
int groups;
|
||||
if (!receive_int(fd, &groups) || groups < 0 || groups > MAX_FILTER_RULES)
|
||||
return false;
|
||||
if (groups == 0)
|
||||
return true;
|
||||
FilterRuleList* list = filter_rule_list_create();
|
||||
if (!list)
|
||||
return false;
|
||||
int total_rules = 0;
|
||||
size_t bytes = 0;
|
||||
for (int g = 0; g < groups; g++) {
|
||||
char* dir = receive_wire_str(fd);
|
||||
if (!dir)
|
||||
goto fail;
|
||||
size_t dir_bytes = strlen(dir);
|
||||
if (!(dir[0] == '\0' || (dir[0] != '/' && !has_path_traversal(dir))) ||
|
||||
dir_bytes > MAX_FILTER_BYTES - bytes) {
|
||||
free(dir);
|
||||
goto fail;
|
||||
}
|
||||
bytes += dir_bytes;
|
||||
int rule_count;
|
||||
if (!receive_int(fd, &rule_count) || rule_count < 0 || rule_count > MAX_FILTER_RULES ||
|
||||
rule_count > MAX_FILTER_RULES - total_rules) {
|
||||
free(dir);
|
||||
goto fail;
|
||||
}
|
||||
for (int r = 0; r < rule_count; r++) {
|
||||
int action, sides;
|
||||
bool anchored, dir_only, negate, no_inherit;
|
||||
if (!receive_int(fd, &action) ||
|
||||
(action != FILTER_ACTION_EXCLUDE && action != FILTER_ACTION_INCLUDE) ||
|
||||
!receive_int(fd, &sides) || sides < (int)FILTER_SIDE_SENDER ||
|
||||
sides > (int)(FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER) ||
|
||||
!receive_flag(fd, &anchored) || !receive_flag(fd, &dir_only) ||
|
||||
!receive_flag(fd, &negate) || !receive_flag(fd, &no_inherit)) {
|
||||
free(dir);
|
||||
goto fail;
|
||||
}
|
||||
char* pattern = receive_wire_str(fd);
|
||||
size_t pattern_bytes = pattern ? strlen(pattern) : 0;
|
||||
if (!pattern || pattern_bytes == 0 || pattern_bytes > MAX_PROTECT_PATTERN_LEN ||
|
||||
pattern_bytes > MAX_FILTER_BYTES - bytes) {
|
||||
free(pattern);
|
||||
free(dir);
|
||||
goto fail;
|
||||
}
|
||||
bytes += pattern_bytes;
|
||||
FilterRule* rule = calloc(1, sizeof(FilterRule));
|
||||
if (!rule) {
|
||||
free(pattern);
|
||||
free(dir);
|
||||
goto fail;
|
||||
}
|
||||
rule->action = (FilterAction)action;
|
||||
rule->sides = (unsigned)sides;
|
||||
rule->anchored = anchored;
|
||||
rule->dir_only = dir_only;
|
||||
rule->negate = negate;
|
||||
rule->no_inherit = no_inherit;
|
||||
rule->owner = str_dup(dir);
|
||||
rule->pattern = pattern;
|
||||
if (!rule->owner || !filter_rule_list_add(list, rule)) {
|
||||
filter_rule_free(rule);
|
||||
free(dir);
|
||||
goto fail;
|
||||
}
|
||||
total_rules++;
|
||||
}
|
||||
free(dir);
|
||||
}
|
||||
*out = list;
|
||||
return true;
|
||||
fail:
|
||||
filter_rule_list_free(list);
|
||||
return false;
|
||||
}
|
||||
|
||||
static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
||||
if (!send_status(fd, STATUS_DELETE_PLAN))
|
||||
return -1;
|
||||
@@ -506,13 +327,10 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
||||
if (!sender->config_sent) {
|
||||
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
|
||||
send_str_section(fd, sender->size_skipped) != 0 ||
|
||||
send_str_section(fd, sender->missing_args) != 0 ||
|
||||
!delete_filter_dir_rules_send(fd, sender->per_dir_rules))
|
||||
send_str_section(fd, sender->missing_args) != 0)
|
||||
return -1;
|
||||
sender->config_sent = true;
|
||||
}
|
||||
if (!send_int(fd, 1)) /* apply = true */
|
||||
return -1;
|
||||
if (!send_wire_str(fd, node->dir))
|
||||
return -1;
|
||||
if (send_str_section(fd, node->dirs) != 0 || send_str_section(fd, node->files) != 0)
|
||||
@@ -521,30 +339,6 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Transmit the one-shot per-run config block (protected prefixes, size-pruned
|
||||
* mirrors, --delete-missing-args exact paths) on its own carrier frame, with
|
||||
* apply=false so the receiver consumes the config but walks nothing. This is
|
||||
* how the config still reaches the receiver when the scope allows no directory
|
||||
* plan at all (a --files-from list of bare files synchronizes no directory):
|
||||
* without it, the missing-args exact deletions would be lost. Idempotent. */
|
||||
static int send_config_only(int fd, DeletePlanSender* sender) {
|
||||
if (!sender || sender->config_sent)
|
||||
return 0;
|
||||
if (!send_status(fd, STATUS_DELETE_PLAN) || !send_int(fd, 1))
|
||||
return -1;
|
||||
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
|
||||
send_str_section(fd, sender->size_skipped) != 0 ||
|
||||
send_str_section(fd, sender->missing_args) != 0 ||
|
||||
!delete_filter_dir_rules_send(fd, sender->per_dir_rules))
|
||||
return -1;
|
||||
sender->config_sent = true;
|
||||
if (!send_int(fd, 0)) /* apply = false */
|
||||
return -1;
|
||||
if (!send_wire_str(fd, ".") || !send_int(fd, 0) || !send_int(fd, 0))
|
||||
return -1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int send_prefix_plan(int fd, DeletePlanSender* sender, const char* dir) {
|
||||
PlanNode* node = plan_find(sender, dir);
|
||||
if (!node || node->sent)
|
||||
@@ -560,10 +354,6 @@ int delete_plan_send_root(int fd, DeletePlanSender* sender) {
|
||||
const char* root = sender->walk_root ? sender->walk_root : ".";
|
||||
if (!plan_ensure(sender, root))
|
||||
return -1;
|
||||
/* Put the config block on the wire first, on its own carrier frame, so the
|
||||
receiver always sees it even when the scope permits no directory plan. */
|
||||
if (send_config_only(fd, sender) != 0)
|
||||
return -1;
|
||||
return send_prefix_plan(fd, sender, root);
|
||||
}
|
||||
|
||||
@@ -613,17 +403,6 @@ int delete_plan_send_remaining(int fd, DeletePlanSender* sender, const ArrayList
|
||||
return 0;
|
||||
}
|
||||
|
||||
int delete_plan_send_all(int fd, DeletePlanSender* sender, const ArrayList* dirs) {
|
||||
if (!sender)
|
||||
return -1;
|
||||
/* Root first: this also transmits the one-shot per-run config block on its
|
||||
own carrier frame (see send_config_only), so it reaches the receiver even
|
||||
when the scope permits no directory plan at all. */
|
||||
if (delete_plan_send_root(fd, sender) != 0)
|
||||
return -1;
|
||||
return delete_plan_send_remaining(fd, sender, dirs);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Receiver: delete session */
|
||||
/* ------------------------------------------------------------------ */
|
||||
@@ -633,17 +412,6 @@ struct DeletePlanSession {
|
||||
bool dry_run;
|
||||
size_t max_delete;
|
||||
size_t deleted;
|
||||
/* Removals charged against --max-delete. The budget is charged on ACTUAL
|
||||
removals (an unlink/rmdir that succeeded), matching rsync: a snapshotted
|
||||
entry that fails removal consumes nothing, so a later extra is still
|
||||
deleted. `planned` and `deleted` advance together for the inline paths and
|
||||
`apply_missing`; `deleted` is the reported count. */
|
||||
size_t planned;
|
||||
/* Hard bound on the deferred snapshot list. Because the budget is no longer
|
||||
charged at snapshot time, this independent cap keeps a huge destination
|
||||
from growing the list without limit (it matches the receiver's overall
|
||||
deletion bound). */
|
||||
size_t defer_cap;
|
||||
size_t skipped;
|
||||
bool limit_hit;
|
||||
bool limit_logged;
|
||||
@@ -652,39 +420,9 @@ struct DeletePlanSession {
|
||||
ArrayList* protected_prefixes;
|
||||
ArrayList* size_skipped;
|
||||
ArrayList* missing;
|
||||
/* Received per-directory filter rules (protocol 2.30.0), or NULL. Evaluated
|
||||
deepest-directory-first for every candidate extra so a destination-only
|
||||
entry matching only a per-directory rule is protected. */
|
||||
FilterRuleList* per_dir_rules;
|
||||
ArrayList* deferred;
|
||||
DeletePathObserver observer;
|
||||
void* observer_context;
|
||||
};
|
||||
|
||||
/* Report one path the session truly removed (no-op without an observer). */
|
||||
static void notify_deleted(DeletePlanSession* session, const char* rel, DeleteEntryType type) {
|
||||
if (session && session->observer && rel)
|
||||
session->observer(session->observer_context, rel, type);
|
||||
}
|
||||
|
||||
/* A removed directory is reported with rsync's trailing slash (`deleting dir/`)
|
||||
while files keep their bare path. */
|
||||
static void notify_deleted_dir(DeletePlanSession* session, const char* rel) {
|
||||
if (!session || !session->observer || !rel)
|
||||
return;
|
||||
size_t len = strlen(rel);
|
||||
char* with_slash = malloc(len + 2);
|
||||
if (!with_slash) {
|
||||
session->observer(session->observer_context, rel, DELETE_ENTRY_DIR);
|
||||
return;
|
||||
}
|
||||
memcpy(with_slash, rel, len);
|
||||
with_slash[len] = '/';
|
||||
with_slash[len + 1] = '\0';
|
||||
session->observer(session->observer_context, with_slash, DELETE_ENTRY_DIR);
|
||||
free(with_slash);
|
||||
}
|
||||
|
||||
DeletePlanSession* delete_plan_session_create(const Config* config) {
|
||||
if (!config)
|
||||
return NULL;
|
||||
@@ -697,7 +435,6 @@ DeletePlanSession* delete_plan_session_create(const Config* config) {
|
||||
config->max_delete >= 0 && (size_t)config->max_delete < DELETE_PLAN_SERVER_LIMIT;
|
||||
session->max_delete =
|
||||
user_limited ? (size_t)config->max_delete : (size_t)DELETE_PLAN_SERVER_LIMIT;
|
||||
session->defer_cap = DELETE_PLAN_SERVER_LIMIT;
|
||||
session->protected_prefixes = array_list_create(free);
|
||||
session->size_skipped = array_list_create(free);
|
||||
session->missing = array_list_create(free);
|
||||
@@ -716,7 +453,6 @@ void delete_plan_session_destroy(DeletePlanSession* session) {
|
||||
array_list_delete(session->protected_prefixes);
|
||||
array_list_delete(session->size_skipped);
|
||||
array_list_delete(session->missing);
|
||||
filter_rule_list_free(session->per_dir_rules);
|
||||
array_list_delete(session->deferred);
|
||||
free(session);
|
||||
}
|
||||
@@ -787,30 +523,49 @@ static int open_plan_dir(const Config* config, const char* dir) {
|
||||
return fd;
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
DeleteSkipSet set;
|
||||
/* Receiver-side delete-protection rules. `base` is the config-frame
|
||||
command-line set and `dir` the received per-directory set (both NULL when
|
||||
the sender sent none). Evaluated per extra so a protect/risk rule is
|
||||
honored under --delete-during/--delete-delay exactly like the whole-tree
|
||||
commit walker. */
|
||||
DeleteProtectRules protect;
|
||||
typedef struct PlanSkips {
|
||||
DeleteSkipEntry* entries;
|
||||
int count;
|
||||
} PlanSkips;
|
||||
|
||||
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
|
||||
PlanSkips* out) {
|
||||
out->protect.base_rules = config->protect_rules;
|
||||
out->protect.dir_rules = session->per_dir_rules;
|
||||
out->protect.backup_suffix = delete_backup_suffix(config);
|
||||
/* The per-directory plan walk keeps each basis path verbatim (it does not
|
||||
convert an absolute under-root path to its root-relative form, unlike the
|
||||
whole-tree commit walk). */
|
||||
return delete_skips_build(config, session->protected_prefixes, session->size_skipped, false,
|
||||
&out->set);
|
||||
out->entries = NULL;
|
||||
out->count = 0;
|
||||
int count = (config->delay_updates ? 1 : 0) + config->basis_count +
|
||||
session->protected_prefixes->size + session->size_skipped->size;
|
||||
if (count == 0)
|
||||
return true;
|
||||
out->entries = calloc((size_t)count, sizeof(DeleteSkipEntry));
|
||||
if (!out->entries)
|
||||
return false;
|
||||
int idx = 0;
|
||||
if (config->delay_updates) {
|
||||
out->entries[idx].prefix = DELAY_UPDATES_STAGING_DIR;
|
||||
out->entries[idx].top_level_only = true;
|
||||
idx++;
|
||||
}
|
||||
for (int i = 0; i < config->basis_count; i++) {
|
||||
out->entries[idx].prefix = config->basis_dirs[i].path;
|
||||
out->entries[idx].top_level_only = false;
|
||||
idx++;
|
||||
}
|
||||
for (int i = 0; i < session->protected_prefixes->size; i++) {
|
||||
out->entries[idx].prefix = (const char*)session->protected_prefixes->items[i];
|
||||
out->entries[idx].top_level_only = false;
|
||||
idx++;
|
||||
}
|
||||
for (int i = 0; i < session->size_skipped->size; i++) {
|
||||
out->entries[idx].prefix = (const char*)session->size_skipped->items[i];
|
||||
out->entries[idx].top_level_only = false;
|
||||
idx++;
|
||||
}
|
||||
out->count = idx;
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool budget_available(const DeletePlanSession* session) {
|
||||
return session->planned < session->max_delete;
|
||||
return session->deleted < session->max_delete;
|
||||
}
|
||||
|
||||
static void note_skipped(DeletePlanSession* session) {
|
||||
@@ -824,15 +579,8 @@ static void log_deleted(const char* rel) {
|
||||
free(escaped);
|
||||
}
|
||||
|
||||
/* Append a snapshot path for --delete-delay. The budget is NOT charged here:
|
||||
* the remover charges --max-delete only when a path is actually unlinked (see
|
||||
* apply_deferred_path), so a snapshotted entry that survives ENOTEMPTY cannot
|
||||
* deny budget to a later extra. The independent `defer_cap` bounds the list. */
|
||||
/* Append a snapshot path for --delete-delay. */
|
||||
static bool defer_add(DeletePlanSession* session, const char* rel) {
|
||||
if ((size_t)session->deferred->size >= session->defer_cap) {
|
||||
note_skipped(session);
|
||||
return true;
|
||||
}
|
||||
char* copy = str_dup(rel);
|
||||
if (!copy)
|
||||
return false;
|
||||
@@ -840,6 +588,7 @@ static bool defer_add(DeletePlanSession* session, const char* rel) {
|
||||
free(copy);
|
||||
return false;
|
||||
}
|
||||
session->deleted++;
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -872,21 +621,19 @@ static bool process_extra_dir(int dirfd, const char* name, const char* child_rel
|
||||
return false;
|
||||
if (survives)
|
||||
return true;
|
||||
if (!budget_available(session)) {
|
||||
note_skipped(session);
|
||||
return true;
|
||||
}
|
||||
if (session->defer && !force_now) {
|
||||
if (!defer_add(session, child_rel))
|
||||
return false;
|
||||
*removed = true;
|
||||
return true;
|
||||
}
|
||||
if (!budget_available(session)) {
|
||||
note_skipped(session);
|
||||
return true;
|
||||
}
|
||||
if (unlinkat(dirfd, name, AT_REMOVEDIR) == 0) {
|
||||
session->deleted++;
|
||||
session->planned++;
|
||||
log_deleted(child_rel);
|
||||
notify_deleted_dir(session, child_rel);
|
||||
*removed = true;
|
||||
return true;
|
||||
}
|
||||
@@ -899,20 +646,18 @@ static bool process_extra_dir(int dirfd, const char* name, const char* child_rel
|
||||
return errno == ENOTEMPTY || errno == EEXIST;
|
||||
}
|
||||
|
||||
static bool process_extra_file(int dirfd, const char* name, const char* child_rel, mode_t mode,
|
||||
bool force_now, DeletePlanSession* session) {
|
||||
if (session->defer && !force_now) {
|
||||
return defer_add(session, child_rel);
|
||||
}
|
||||
static bool process_extra_file(int dirfd, const char* name, const char* child_rel, bool force_now,
|
||||
DeletePlanSession* session) {
|
||||
if (!budget_available(session)) {
|
||||
note_skipped(session);
|
||||
return true;
|
||||
}
|
||||
if (session->defer && !force_now) {
|
||||
return defer_add(session, child_rel);
|
||||
}
|
||||
if (unlinkat(dirfd, name, 0) == 0) {
|
||||
session->deleted++;
|
||||
session->planned++;
|
||||
log_deleted(child_rel);
|
||||
notify_deleted(session, child_rel, delete_entry_type_of_mode(mode));
|
||||
} else if (errno != ENOENT) {
|
||||
return false;
|
||||
}
|
||||
@@ -923,107 +668,70 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
|
||||
const ArrayList* keep_files, bool at_root, bool force_now,
|
||||
const PlanSkips* skips, DeletePlanSession* session, bool* survives) {
|
||||
*survives = false;
|
||||
DeleteDirEntry* entries = NULL;
|
||||
size_t count = 0;
|
||||
bool collect_ok = true;
|
||||
if (!delete_dir_entries_collect(dirfd, &entries, &count, &collect_ok))
|
||||
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (scanfd < 0)
|
||||
return false;
|
||||
bool operation_ok = collect_ok;
|
||||
bool local_survives = false;
|
||||
bool* shielded = calloc(count ? count : 1, sizeof(bool));
|
||||
bool* is_extra = calloc(count ? count : 1, sizeof(bool));
|
||||
bool* force = calloc(count ? count : 1, sizeof(bool));
|
||||
if (!shielded || !is_extra || !force) {
|
||||
free(shielded);
|
||||
free(is_extra);
|
||||
free(force);
|
||||
delete_dir_entries_free(entries, count);
|
||||
DIR* dir = fdopendir(scanfd);
|
||||
if (!dir) {
|
||||
close(scanfd);
|
||||
return false;
|
||||
}
|
||||
|
||||
/* rsync's order: extraneous subdirectories in descending name order, then
|
||||
extraneous files in descending name order (kept entries survive and are not
|
||||
touched here — a kept subdirectory gets its own per-directory plan). */
|
||||
if (count > 1)
|
||||
qsort(entries, count, sizeof(*entries), delete_dir_entry_cmp_desc);
|
||||
size_t dir_count = 0;
|
||||
while (dir_count < count && entries[dir_count].is_dir)
|
||||
dir_count++;
|
||||
|
||||
for (size_t i = 0; i < count; i++) {
|
||||
bool operation_ok = true;
|
||||
bool local_survives = false;
|
||||
const struct dirent* entry;
|
||||
while ((entry = readdir(dir)) != NULL) {
|
||||
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
|
||||
continue;
|
||||
char* child_rel =
|
||||
(strcmp(dir_rel, ".") == 0) ? str_dup(entries[i].name) : path_cat(dir_rel, entries[i].name);
|
||||
(strcmp(dir_rel, ".") == 0) ? str_dup(entry->d_name) : path_cat(dir_rel, entry->d_name);
|
||||
if (!child_rel) {
|
||||
operation_ok = false;
|
||||
continue;
|
||||
}
|
||||
if (path_under_skip_prefix(child_rel, at_root, skips->set.entries, skips->set.count)) {
|
||||
shielded[i] = true;
|
||||
if (path_under_skip_prefix(child_rel, at_root, skips->entries, skips->count)) {
|
||||
local_survives = true;
|
||||
free(child_rel);
|
||||
continue;
|
||||
}
|
||||
bool is_dir = entries[i].is_dir;
|
||||
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entries[i].name);
|
||||
bool in_keep_files = !is_dir && list_contains_str(keep_files, entries[i].name);
|
||||
bool rule_protected = delete_protect_verdict(&skips->protect, child_rel, entries[i].name,
|
||||
is_dir) == FILTER_ACTION_PROTECT;
|
||||
if (in_keep_dirs || in_keep_files || rule_protected) {
|
||||
shielded[i] = true;
|
||||
struct stat st;
|
||||
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
||||
if (errno != ENOENT)
|
||||
operation_ok = false;
|
||||
free(child_rel);
|
||||
continue;
|
||||
}
|
||||
bool is_dir = S_ISDIR(st.st_mode);
|
||||
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entry->d_name);
|
||||
bool in_keep_files = !is_dir && list_contains_str(keep_files, entry->d_name);
|
||||
if (in_keep_dirs) {
|
||||
local_survives = true;
|
||||
} else if (keep_dirs && !is_dir && list_contains_str(keep_dirs, entry->d_name)) {
|
||||
/* Destination file blocks a source directory: clear it now, whatever the
|
||||
delete timing, so the directory can be created. */
|
||||
if (!process_extra_file(dirfd, entry->d_name, child_rel, true, session))
|
||||
operation_ok = false;
|
||||
} else if (in_keep_files) {
|
||||
local_survives = true;
|
||||
} else if (keep_files && is_dir && list_contains_str(keep_files, entry->d_name)) {
|
||||
/* Destination directory blocks a source file: remove it now. */
|
||||
bool removed = false;
|
||||
if (!process_extra_dir(dirfd, entry->d_name, child_rel, true, skips, session, &removed))
|
||||
operation_ok = false;
|
||||
else if (!removed)
|
||||
local_survives = true;
|
||||
} else if (is_dir) {
|
||||
/* A destination directory blocks a source file of the same name: remove
|
||||
it now, whatever the delete timing, so the file can be created. */
|
||||
is_extra[i] = true;
|
||||
force[i] = keep_files && list_contains_str(keep_files, entries[i].name);
|
||||
bool removed = false;
|
||||
if (!process_extra_dir(dirfd, entry->d_name, child_rel, force_now, skips, session, &removed))
|
||||
operation_ok = false;
|
||||
else if (!removed)
|
||||
local_survives = true;
|
||||
} else {
|
||||
/* A destination file blocks a source directory of the same name: clear it
|
||||
now so the directory can be created. */
|
||||
is_extra[i] = true;
|
||||
force[i] = keep_dirs && list_contains_str(keep_dirs, entries[i].name);
|
||||
if (!process_extra_file(dirfd, entry->d_name, child_rel, force_now, session))
|
||||
operation_ok = false;
|
||||
}
|
||||
free(child_rel);
|
||||
}
|
||||
|
||||
/* Pass 1: extraneous subdirectories, descending. */
|
||||
for (size_t i = 0; i < dir_count; i++) {
|
||||
if (!is_extra[i])
|
||||
continue;
|
||||
char* child_rel =
|
||||
(strcmp(dir_rel, ".") == 0) ? str_dup(entries[i].name) : path_cat(dir_rel, entries[i].name);
|
||||
if (!child_rel) {
|
||||
operation_ok = false;
|
||||
continue;
|
||||
}
|
||||
bool removed = false;
|
||||
if (!process_extra_dir(dirfd, entries[i].name, child_rel, force[i] || force_now, skips, session,
|
||||
&removed))
|
||||
operation_ok = false;
|
||||
else if (!removed)
|
||||
local_survives = true;
|
||||
free(child_rel);
|
||||
}
|
||||
|
||||
/* Pass 2: extraneous files, descending. */
|
||||
for (size_t i = dir_count; i < count; i++) {
|
||||
if (!is_extra[i])
|
||||
continue;
|
||||
char* child_rel =
|
||||
(strcmp(dir_rel, ".") == 0) ? str_dup(entries[i].name) : path_cat(dir_rel, entries[i].name);
|
||||
if (!child_rel) {
|
||||
operation_ok = false;
|
||||
continue;
|
||||
}
|
||||
if (!process_extra_file(dirfd, entries[i].name, child_rel, entries[i].mode,
|
||||
force[i] || force_now, session))
|
||||
operation_ok = false;
|
||||
free(child_rel);
|
||||
}
|
||||
|
||||
free(shielded);
|
||||
free(is_extra);
|
||||
free(force);
|
||||
delete_dir_entries_free(entries, count);
|
||||
closedir(dir);
|
||||
*survives = local_survives;
|
||||
return operation_ok;
|
||||
}
|
||||
@@ -1043,7 +751,7 @@ static bool apply_plan_dir(DeletePlanSession* session, const Config* config, con
|
||||
bool survives = false;
|
||||
bool ok = process_children(dirfd, dir, dirs, files, strcmp(dir, ".") == 0, false, &skips, session,
|
||||
&survives);
|
||||
delete_skips_free(&skips.set);
|
||||
free(skips.entries);
|
||||
close(dirfd);
|
||||
if (!ok)
|
||||
log_message(LOG_LEVEL_ERROR, "deletion failed while removing extraneous files");
|
||||
@@ -1060,15 +768,13 @@ static bool apply_missing(DeletePlanSession* session, const Config* config) {
|
||||
return true;
|
||||
DeleteManifest manifest = {
|
||||
.keeps = NULL, .protected = NULL, .missing = session->missing, .dirs = NULL};
|
||||
size_t remaining = budget_available(session) ? session->max_delete - session->planned : 0;
|
||||
size_t remaining = budget_available(session) ? session->max_delete - session->deleted : 0;
|
||||
size_t deleted = 0;
|
||||
size_t skipped = 0;
|
||||
bool limit = false;
|
||||
bool ok = manifest_delete_missing_args_limited_observed(config, &manifest, remaining, &deleted,
|
||||
&skipped, &limit, session->observer,
|
||||
session->observer_context);
|
||||
bool ok = manifest_delete_missing_args_limited(config, &manifest, remaining, &deleted, &skipped,
|
||||
&limit);
|
||||
session->deleted += deleted;
|
||||
session->planned += deleted;
|
||||
session->skipped += skipped;
|
||||
if (limit)
|
||||
session->limit_hit = true;
|
||||
@@ -1089,21 +795,12 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
||||
if (has_config) {
|
||||
if (session->config_seen || !read_section(fd, session->protected_prefixes, true, &bytes) ||
|
||||
!read_section(fd, session->size_skipped, true, &bytes) ||
|
||||
!read_section(fd, session->missing, true, &bytes) ||
|
||||
!delete_filter_dir_rules_receive(fd, &session->per_dir_rules)) {
|
||||
!read_section(fd, session->missing, true, &bytes)) {
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return -1;
|
||||
}
|
||||
session->config_seen = true;
|
||||
}
|
||||
/* apply=false is the config-only carrier frame: the receiver consumes the
|
||||
config (and the missing-args exact deletions) but must not walk any
|
||||
directory. Every real plan carries apply=true. */
|
||||
int apply;
|
||||
if (!receive_int(fd, &apply) || (apply != 0 && apply != 1)) {
|
||||
send_status(fd, STATUS_ERROR);
|
||||
return -1;
|
||||
}
|
||||
char* dir = receive_wire_str(fd);
|
||||
ArrayList* dirs = array_list_create(free);
|
||||
ArrayList* files = array_list_create(free);
|
||||
@@ -1121,7 +818,7 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
||||
if (!session->dry_run && enabled) {
|
||||
if (!session->defer && !apply_missing(session, config))
|
||||
ok = false;
|
||||
if (ok && apply && !apply_plan_dir(session, config, dir, dirs, files))
|
||||
if (ok && !apply_plan_dir(session, config, dir, dirs, files))
|
||||
ok = false;
|
||||
}
|
||||
free(dir);
|
||||
@@ -1140,103 +837,37 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
||||
}
|
||||
|
||||
/* Apply one snapshotted --delete-delay path (post-order: children precede their
|
||||
* parent directory). A directory that is still present is re-scanned so content
|
||||
* created after the plan is removed too; every actual removal charges
|
||||
* --max-delete. */
|
||||
* parent directory). */
|
||||
static bool apply_deferred_path(DeletePlanSession* session, const Config* config, const char* rel) {
|
||||
(void)session;
|
||||
char* full = path_cat(config->receive_root_directory, rel);
|
||||
if (!full)
|
||||
return false;
|
||||
char* leaf = NULL;
|
||||
int parent_fd = file_open_secure_parent(full, &leaf, false);
|
||||
int open_errno = errno;
|
||||
free(full);
|
||||
if (parent_fd < 0) {
|
||||
free(leaf);
|
||||
return open_errno == ENOENT || open_errno == ENOTDIR;
|
||||
return errno == ENOENT || errno == ENOTDIR;
|
||||
}
|
||||
struct stat st;
|
||||
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
||||
bool absent = errno == ENOENT || errno == ENOTDIR;
|
||||
bool absent = errno == ENOENT;
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return absent;
|
||||
}
|
||||
if (S_ISDIR(st.st_mode)) {
|
||||
if (!budget_available(session)) {
|
||||
note_skipped(session);
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return true;
|
||||
}
|
||||
int dirfd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (dirfd < 0) {
|
||||
bool absent = errno == ENOENT || errno == ENOTDIR;
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return absent;
|
||||
}
|
||||
PlanSkips skips;
|
||||
if (!build_plan_skips(config, session, &skips)) {
|
||||
close(dirfd);
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
bool survives = false;
|
||||
bool ok = process_children(dirfd, rel, NULL, NULL, false, true, &skips, session, &survives);
|
||||
delete_skips_free(&skips.set);
|
||||
close(dirfd);
|
||||
if (!ok) {
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
if (!survives) {
|
||||
if (!budget_available(session)) {
|
||||
note_skipped(session);
|
||||
} else if (unlinkat(parent_fd, leaf, AT_REMOVEDIR) == 0) {
|
||||
session->deleted++;
|
||||
session->planned++;
|
||||
log_deleted(rel);
|
||||
notify_deleted_dir(session, rel);
|
||||
} else if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST) {
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return true;
|
||||
}
|
||||
if (!budget_available(session)) {
|
||||
note_skipped(session);
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return true;
|
||||
}
|
||||
if (unlinkat(parent_fd, leaf, 0) == 0) {
|
||||
session->deleted++;
|
||||
session->planned++;
|
||||
int rc;
|
||||
if (S_ISDIR(st.st_mode))
|
||||
rc = unlinkat(parent_fd, leaf, AT_REMOVEDIR);
|
||||
else
|
||||
rc = unlinkat(parent_fd, leaf, 0);
|
||||
bool ok = rc == 0 || errno == ENOENT || errno == ENOTEMPTY || errno == EEXIST;
|
||||
if (rc == 0)
|
||||
log_deleted(rel);
|
||||
notify_deleted(session, rel, delete_entry_type_of_mode(st.st_mode));
|
||||
} else if (errno != ENOENT) {
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
close(parent_fd);
|
||||
free(leaf);
|
||||
return true;
|
||||
}
|
||||
|
||||
void delete_plan_session_set_delete_observer(DeletePlanSession* session,
|
||||
DeletePathObserver observer, void* context) {
|
||||
if (!session)
|
||||
return;
|
||||
session->observer = observer;
|
||||
session->observer_context = context;
|
||||
return ok;
|
||||
}
|
||||
|
||||
DeleteCommitResult delete_plan_session_commit(DeletePlanSession* session, const Config* config) {
|
||||
|
||||
@@ -3,10 +3,8 @@
|
||||
|
||||
#include "array_list.h"
|
||||
#include "config.h"
|
||||
#include "delete.h"
|
||||
#include "file_receive.h"
|
||||
#include "protocol.h"
|
||||
#include "utils.h"
|
||||
#include <stdbool.h>
|
||||
|
||||
/* Per-directory delete plans (protocol 2.24.0).
|
||||
@@ -25,19 +23,6 @@
|
||||
* --delete-missing-args exact deletions, the shared --max-delete budget and,
|
||||
* for --delete-delay, the snapshotted extras. */
|
||||
|
||||
/* Per-directory filter-rule block (protocol 2.30.0). The sender compiles the
|
||||
* source's per-directory merge rules as it scans and streams them so the
|
||||
* receiver can re-derive the receiver-side protect/risk verdicts for
|
||||
* destination-only entries. The wire format is a group count, then for each
|
||||
* directory group its relative owner path followed by that directory's rule
|
||||
* records (action, sides, anchored, dir-only, negate, no-inherit, pattern).
|
||||
* All bounds (MAX_FILTER_RULES, MAX_FILTER_BYTES, MAX_PROTECT_PATTERN_LEN) are
|
||||
* enforced on both sides; a malformed receive frame signals STATUS_ERROR and
|
||||
* returns false. Receive yields a flat FilterRuleList whose rules carry their
|
||||
* owner, or NULL when no rules were sent. */
|
||||
bool delete_filter_dir_rules_send(int fd, const FilterRuleList* rules);
|
||||
bool delete_filter_dir_rules_receive(int fd, FilterRuleList** out);
|
||||
|
||||
/* ---- Sender: plan builder ---- */
|
||||
|
||||
typedef struct DeletePlanSender DeletePlanSender;
|
||||
@@ -62,33 +47,19 @@ void delete_plan_sender_finalize(DeletePlanSender* sender, const ArrayList* sync
|
||||
Directory keep entries do not count, so an I/O error that hid every file
|
||||
still refuses to delete. */
|
||||
bool delete_plan_sender_empty(const DeletePlanSender* sender);
|
||||
/* Attach the global config sections advertised on the first plan frame. The
|
||||
* block is always transmitted by delete_plan_send_root(), on a config-only
|
||||
* carrier frame when the scope allows no directory plan. */
|
||||
/* Attach the global config sections advertised on the first plan frame. */
|
||||
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
||||
const ArrayList* size_skipped, const ArrayList* missing_args,
|
||||
const FilterRuleList* per_dir_rules);
|
||||
const ArrayList* size_skipped, const ArrayList* missing_args);
|
||||
/* Send the root plan (even before any data, so root extras are handled like
|
||||
* rsync's first generator directory), after transmitting the per-run config
|
||||
* block on its own carrier frame. Returns -1 on I/O error. */
|
||||
* rsync's first generator directory). Returns -1 on I/O error. */
|
||||
int delete_plan_send_root(int fd, DeletePlanSender* sender);
|
||||
/* Send the plans for every ancestor of `path` (root-first) and, when is_dir,
|
||||
* for `path` itself; already-sent plans are skipped. */
|
||||
int delete_plan_send_for_path(int fd, DeletePlanSender* sender, const char* path, bool is_dir);
|
||||
/* Send the plan for every directory in `dirs` that has not been transmitted
|
||||
* yet. */
|
||||
* yet. Called after the data stream so an empty source directory's plan still
|
||||
* clears its destination extras even though no file frame triggered it. */
|
||||
int delete_plan_send_remaining(int fd, DeletePlanSender* sender, const ArrayList* dirs);
|
||||
/* Transmit the COMPLETE per-directory plan set in one pass, before any data
|
||||
* frame: the root plan (with the one-shot per-run config block on its carrier
|
||||
* frame) followed by every directory in `dirs`. Because the whole plan set is
|
||||
* known from the path-only pre-scan, sending it all up front means a
|
||||
* mid-transfer abort has already applied every planned removal, matching
|
||||
* rsync's generator (which runs ahead of its throttled sender). A completed
|
||||
* run is unaffected. `dirs` is the set of directories whose direct children
|
||||
* were enumerated (the scanner's plan_dirs sink), so a merely listed but
|
||||
* untraversed directory never gets a plan and its mirror is left intact.
|
||||
* Returns -1 on I/O error. */
|
||||
int delete_plan_send_all(int fd, DeletePlanSender* sender, const ArrayList* dirs);
|
||||
|
||||
/* ---- Receiver: delete session ---- */
|
||||
|
||||
@@ -105,16 +76,8 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
||||
DeleteCommitResult delete_plan_session_commit(DeletePlanSession* session, const Config* config);
|
||||
/* True once the shared --max-delete budget stopped part of a deletion. */
|
||||
bool delete_plan_session_limit_reached(const DeletePlanSession* session);
|
||||
/* Number of destination entries the session actually removed, for the
|
||||
end-of-transfer stats. For --delete-delay this excludes a snapshotted entry
|
||||
that survived (e.g. a refilled directory that failed ENOTEMPTY), even though
|
||||
that entry already consumed --max-delete budget at snapshot time. */
|
||||
/* Number of destination entries the session's plans removed (or, for
|
||||
--delete-delay, snapshotted for removal), for the end-of-transfer stats. */
|
||||
size_t delete_plan_session_deleted(const DeletePlanSession* session);
|
||||
/* Install an observer invoked for every destination-relative path the session
|
||||
truly removes (including the deferred --delete-delay commit), so the receiver
|
||||
can report rsync's `deleting PATH` lines through the terminal STATUS_STATS
|
||||
record. Pass NULL/0 to clear. */
|
||||
void delete_plan_session_set_delete_observer(DeletePlanSession* session,
|
||||
DeletePathObserver observer, void* context);
|
||||
|
||||
#endif
|
||||
|
||||
@@ -1,12 +1,10 @@
|
||||
#include "delta.h"
|
||||
#include "log.h"
|
||||
#include "protocol.h"
|
||||
#include <errno.h>
|
||||
#include <stdint.h>
|
||||
#include <limits.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#define XXH_STATIC_LINKING_ONLY
|
||||
#define XXH_IMPLEMENTATION
|
||||
@@ -84,64 +82,6 @@ DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_
|
||||
return sig;
|
||||
}
|
||||
|
||||
/* Bounded read of exactly `len` bytes at `off`; retries on EINTR. */
|
||||
static bool pread_all(int fd, void* buf, size_t len, uint64_t off) {
|
||||
uint8_t* p = buf;
|
||||
size_t done = 0;
|
||||
while (done < len) {
|
||||
ssize_t n = pread(fd, p + done, len - done, (off_t)(off + done));
|
||||
if (n < 0 && errno == EINTR)
|
||||
continue;
|
||||
if (n <= 0)
|
||||
return false;
|
||||
done += (size_t)n;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
DeltaSignature* delta_signature_create_fd_seeded(int fd, uint64_t old_file_size,
|
||||
uint32_t block_size, uint32_t seed) {
|
||||
if (fd < 0 || old_file_size == 0 || block_size == 0 || block_size > DELTA_BLOCK_SIZE_MAX ||
|
||||
old_file_size > UINT32_MAX * (uint64_t)block_size)
|
||||
return NULL;
|
||||
uint32_t block_count = (uint32_t)((old_file_size + block_size - 1) / block_size);
|
||||
/* Bound the signature's own memory (block_count * sizeof(DeltaBlockSig)). */
|
||||
if (block_count == 0 || block_count > MAX_DELTA_BLOCKS)
|
||||
return NULL;
|
||||
DeltaSignature* sig = protocol_alloc(sizeof(DeltaSignature));
|
||||
if (!sig)
|
||||
return NULL;
|
||||
sig->file_size = old_file_size;
|
||||
sig->block_size = block_size;
|
||||
sig->block_count = block_count;
|
||||
sig->blocks = protocol_alloc((size_t)block_count * sizeof(DeltaBlockSig));
|
||||
if (!sig->blocks) {
|
||||
free(sig);
|
||||
return NULL;
|
||||
}
|
||||
uint8_t* block = malloc(block_size);
|
||||
if (!block) {
|
||||
free(sig->blocks);
|
||||
free(sig);
|
||||
return NULL;
|
||||
}
|
||||
for (uint32_t i = 0; i < block_count; i++) {
|
||||
uint64_t offset = (uint64_t)i * block_size;
|
||||
uint32_t len =
|
||||
(uint32_t)((old_file_size - offset < block_size) ? (old_file_size - offset) : block_size);
|
||||
if (!pread_all(fd, block, len, offset)) {
|
||||
free(block);
|
||||
free(sig->blocks);
|
||||
free(sig);
|
||||
return NULL;
|
||||
}
|
||||
sig->blocks[i].adler32 = delta_adler32(block, len);
|
||||
sig->blocks[i].xxhash = delta_xxhash32_seeded(block, len, seed);
|
||||
}
|
||||
free(block);
|
||||
return sig;
|
||||
}
|
||||
|
||||
Data* delta_signature_serialize(const DeltaSignature* sig) {
|
||||
if (!sig)
|
||||
return NULL;
|
||||
@@ -747,130 +687,6 @@ void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta,
|
||||
return output;
|
||||
}
|
||||
|
||||
void* delta_apply_fd(int src_fd, uint64_t old_size, const Delta* delta, uint32_t block_size) {
|
||||
if (!delta || block_size == 0 || block_size > DELTA_BLOCK_SIZE_MAX ||
|
||||
(delta->instruction_count > 0 && !delta->instructions) || delta->new_file_size == 0 ||
|
||||
delta->new_file_size > SIZE_MAX)
|
||||
return NULL;
|
||||
|
||||
void* output = protocol_alloc((size_t)delta->new_file_size);
|
||||
if (!output)
|
||||
return NULL;
|
||||
|
||||
uint8_t* out = (uint8_t*)output;
|
||||
uint64_t out_pos = 0;
|
||||
|
||||
for (uint32_t i = 0; i < delta->instruction_count; i++) {
|
||||
if (delta->instructions[i].type == DELTA_INSTR_BLOCK_MATCH) {
|
||||
uint64_t src_offset = (uint64_t)delta->instructions[i].match.block_index * block_size;
|
||||
if (src_offset > UINT64_MAX - delta->instructions[i].match.block_offset) {
|
||||
free(output);
|
||||
return NULL;
|
||||
}
|
||||
src_offset += delta->instructions[i].match.block_offset;
|
||||
uint32_t len = delta->instructions[i].match.length;
|
||||
|
||||
if (src_offset > old_size || (uint64_t)len > old_size - src_offset ||
|
||||
out_pos > delta->new_file_size || (uint64_t)len > delta->new_file_size - out_pos) {
|
||||
free(output);
|
||||
return NULL;
|
||||
}
|
||||
if (!pread_all(src_fd, out + out_pos, len, src_offset)) {
|
||||
free(output);
|
||||
return NULL;
|
||||
}
|
||||
out_pos += len;
|
||||
} else if (delta->instructions[i].type == DELTA_INSTR_LITERAL) {
|
||||
uint32_t len = delta->instructions[i].literal.length;
|
||||
if (out_pos > delta->new_file_size || (uint64_t)len > delta->new_file_size - out_pos) {
|
||||
free(output);
|
||||
return NULL;
|
||||
}
|
||||
memcpy(out + out_pos, delta->instructions[i].literal.data, len);
|
||||
out_pos += len;
|
||||
} else {
|
||||
free(output);
|
||||
return NULL;
|
||||
}
|
||||
}
|
||||
|
||||
if (out_pos != delta->new_file_size) {
|
||||
free(output);
|
||||
return NULL;
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
bool delta_apply_to_fd(const void* old_data, int src_fd, uint64_t old_size, const Delta* delta,
|
||||
uint32_t block_size, int dst_fd) {
|
||||
if (!delta || (old_data == NULL && src_fd < 0) || block_size == 0 ||
|
||||
block_size > DELTA_BLOCK_SIZE_MAX || (delta->instruction_count > 0 && !delta->instructions))
|
||||
return false;
|
||||
const int chunk = 1 << 20;
|
||||
uint8_t* buf = malloc((size_t)chunk);
|
||||
if (!buf)
|
||||
return false;
|
||||
uint64_t out_pos = 0;
|
||||
bool ok = true;
|
||||
for (uint32_t i = 0; i < delta->instruction_count && ok; i++) {
|
||||
uint64_t src_offset = 0;
|
||||
uint64_t len = 0;
|
||||
const uint8_t* lit = NULL;
|
||||
if (delta->instructions[i].type == DELTA_INSTR_BLOCK_MATCH) {
|
||||
src_offset = (uint64_t)delta->instructions[i].match.block_index * block_size;
|
||||
if (src_offset > UINT64_MAX - delta->instructions[i].match.block_offset ||
|
||||
src_offset + delta->instructions[i].match.block_offset > old_size) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
src_offset += delta->instructions[i].match.block_offset;
|
||||
len = delta->instructions[i].match.length;
|
||||
if (len > old_size - src_offset) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
} else if (delta->instructions[i].type == DELTA_INSTR_LITERAL) {
|
||||
lit = delta->instructions[i].literal.data;
|
||||
len = delta->instructions[i].literal.length;
|
||||
} else {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
if (out_pos > delta->new_file_size || len > delta->new_file_size - out_pos) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
uint64_t done = 0;
|
||||
while (ok && done < len) {
|
||||
size_t want = (len - done) < (uint64_t)chunk ? (size_t)(len - done) : (size_t)chunk;
|
||||
if (lit) {
|
||||
memcpy(buf, lit + done, want);
|
||||
} else if (old_data) {
|
||||
memcpy(buf, (const uint8_t*)old_data + src_offset + done, want);
|
||||
} else if (!pread_all(src_fd, buf, want, src_offset + done)) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
const uint8_t* p = buf;
|
||||
size_t written = 0;
|
||||
while (written < want) {
|
||||
ssize_t n = write(dst_fd, p + written, want - written);
|
||||
if (n < 0 && errno == EINTR)
|
||||
continue;
|
||||
if (n <= 0) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
written += (size_t)n;
|
||||
}
|
||||
done += want;
|
||||
}
|
||||
out_pos += len;
|
||||
}
|
||||
free(buf);
|
||||
return ok && out_pos == delta->new_file_size;
|
||||
}
|
||||
|
||||
void delta_destroy(Delta* delta) {
|
||||
if (!delta)
|
||||
return;
|
||||
|
||||
@@ -61,13 +61,6 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
|
||||
* identical to the unseeded function. */
|
||||
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
|
||||
uint32_t block_size, uint32_t seed);
|
||||
/* Streaming equivalent of delta_signature_create_seeded: reads the basis blocks
|
||||
* from `fd` in bounded chunks, so an arbitrarily large basis can be signed
|
||||
* without materializing it. The signature itself is bounded (MAX_DELTA_BLOCKS
|
||||
* entries); an over-large basis returns NULL and the caller falls back to a
|
||||
* whole-file transfer. */
|
||||
DeltaSignature* delta_signature_create_fd_seeded(int fd, uint64_t old_file_size,
|
||||
uint32_t block_size, uint32_t seed);
|
||||
Data* delta_signature_serialize(const DeltaSignature* sig);
|
||||
DeltaSignature* delta_signature_deserialize(const Data* data);
|
||||
void delta_signature_destroy(DeltaSignature* sig);
|
||||
@@ -82,15 +75,6 @@ Delta* delta_compute_seeded(const void* new_file_data, uint64_t new_file_size,
|
||||
Data* delta_serialize(const Delta* delta);
|
||||
Delta* delta_deserialize(const Data* data);
|
||||
void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta, uint32_t block_size);
|
||||
/* Streaming equivalent of delta_apply: matched blocks are read from `src_fd` as
|
||||
* they are emitted, so the basis never has to be resident. */
|
||||
void* delta_apply_fd(int src_fd, uint64_t old_size, const Delta* delta, uint32_t block_size);
|
||||
/* Fully streamed reconstruction: matched blocks come from `old_data` (when
|
||||
* non-NULL) or are read from `src_fd`, and the reconstructed bytes are written
|
||||
* straight to `dst_fd` in bounded chunks, so a reconstructed file larger than
|
||||
* memory is never materialized. */
|
||||
bool delta_apply_to_fd(const void* old_data, int src_fd, uint64_t old_size, const Delta* delta,
|
||||
uint32_t block_size, int dst_fd);
|
||||
void delta_destroy(Delta* delta);
|
||||
|
||||
bool delta_should_attempt(uint64_t old_size, uint64_t new_size, uint64_t max_file_size);
|
||||
|
||||
+38
-695
@@ -15,9 +15,6 @@
|
||||
#include <unistd.h>
|
||||
|
||||
#include "data.h"
|
||||
#include "checksum.h"
|
||||
#include "chunk.h"
|
||||
#include "compression.h"
|
||||
#include "delta.h"
|
||||
#include "file.h"
|
||||
#include "file_store.h"
|
||||
@@ -42,31 +39,6 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Streaming copy of an open source descriptor into the just-created destination
|
||||
`fd` (already at offset 0). Used by the --copy-dest basis install so a basis
|
||||
larger than any in-memory whole-file bound still materializes without
|
||||
buffering the entire file. `expected_size` is the caller-verified basis
|
||||
size; the copy must produce exactly that many bytes (a short source is a hard
|
||||
error, never a silently truncated destination). The final ftruncate drops
|
||||
any residual tail a raced-in longer source might have left. */
|
||||
static bool copy_fd_all(int dst_fd, int src_fd, unsigned long long expected_size) {
|
||||
unsigned char buf[1 << 20];
|
||||
unsigned long long done = 0;
|
||||
while (done < expected_size) {
|
||||
unsigned long long remaining = expected_size - done;
|
||||
size_t want = remaining < sizeof(buf) ? (size_t)remaining : sizeof(buf);
|
||||
ssize_t n = read(src_fd, buf, want);
|
||||
if (n < 0 && errno == EINTR)
|
||||
continue;
|
||||
if (n <= 0)
|
||||
return false;
|
||||
if (!write_all(dst_fd, buf, (unsigned long long)n))
|
||||
return false;
|
||||
done += (unsigned long long)n;
|
||||
}
|
||||
return ftruncate(dst_fd, (off_t)expected_size) == 0;
|
||||
}
|
||||
|
||||
/* Preallocate `size` bytes on `fd` before any data is written (--preallocate).
|
||||
* fallocate(2) reserves real disk blocks, so an out-of-space condition
|
||||
* (ENOSPC/EDQUOT) surfaces up front instead of partway through a transfer;
|
||||
@@ -168,12 +140,6 @@ bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, s
|
||||
if (file->data->size == 0) {
|
||||
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
|
||||
}
|
||||
/* A streamed source (data not loaded) may exceed any in-memory whole-file
|
||||
bound; hash it from the file path in bounded buffers instead of forcing a
|
||||
full load. This is the same digest the receiver recomputes on the basis. */
|
||||
if (!file->data->data && file->path && file->data->size > STREAM_THRESHOLD &&
|
||||
checksum_digest_file(algo, seed, file->path, out, out_capacity, out_len))
|
||||
return true;
|
||||
if (!file->data->data && !file_load_data(file))
|
||||
return false;
|
||||
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
|
||||
@@ -210,8 +176,6 @@ File* file_create(const char* path) {
|
||||
file->is_dir = false;
|
||||
file->dir_time_only = false;
|
||||
file->basis_link = NULL;
|
||||
file->basis_copy = NULL;
|
||||
file->data_spool = false;
|
||||
file->link_group = 0;
|
||||
file->link_first = false;
|
||||
file->hardlink_target = NULL;
|
||||
@@ -223,7 +187,6 @@ File* file_create(const char* path) {
|
||||
file->xattrs = NULL;
|
||||
file->dest_state = (OutputDestState){0};
|
||||
file->matched_bytes = 0;
|
||||
file->literal_bytes = 0;
|
||||
return file;
|
||||
}
|
||||
|
||||
@@ -241,11 +204,6 @@ void file_destroy(void* item) {
|
||||
file->send_path = NULL;
|
||||
free(file->basis_link);
|
||||
file->basis_link = NULL;
|
||||
if (file->data_spool && file->basis_copy)
|
||||
unlink(file->basis_copy);
|
||||
free(file->basis_copy);
|
||||
file->basis_copy = NULL;
|
||||
file->data_spool = false;
|
||||
free(file->hardlink_target);
|
||||
file->hardlink_target = NULL;
|
||||
free(file->symlink_target);
|
||||
@@ -387,261 +345,6 @@ size_t file_content_to_buffer(File* file) {
|
||||
return bytes_read;
|
||||
}
|
||||
|
||||
/* ---- Streamed whole-file payload receive ----
|
||||
*
|
||||
* A whole-file data frame is a uint64 length followed by that many bytes. When
|
||||
* the logical payload is at or below the receiver's streaming bound the
|
||||
* historical charged whole-buffer path is kept (decompressing in one shot for
|
||||
* `-z`). Above the bound the frame is read in bounded chunks and written into
|
||||
* a spool temp file next to the destination, decompressing incrementally for
|
||||
* zstd/zlib (lz4's block format cannot be streamed and keeps the buffered path).
|
||||
* The spool is then installed by the existing bounded-buffer basis-copy path
|
||||
* (file_copy_basis_stream_attrs), so the atomic temp+rename store, --partial/
|
||||
* --partial-dir, --delay-updates, --preallocate and metadata/xattr application
|
||||
* are all reused unchanged. Only bounded buffers (64 KiB read chunk + the
|
||||
* decompressor's 256 KiB output window) are ever live. */
|
||||
|
||||
#define FILE_PAYLOAD_READ_CHUNK (64 * 1024)
|
||||
#define FILE_PAYLOAD_PEEK_MAX 32
|
||||
|
||||
/* Create a confined spool temp file in the destination's directory. Returns an
|
||||
* open write fd and an owned absolute path, or -1 (errno set). The parent walk
|
||||
* creates missing directories exactly as a normal store would. */
|
||||
static int file_spool_create(const char* dest_path, char** out_spool_path) {
|
||||
*out_spool_path = NULL;
|
||||
char* leaf = NULL;
|
||||
int dirfd = file_open_secure_parent(dest_path, &leaf, true);
|
||||
free(leaf);
|
||||
if (dirfd < 0)
|
||||
return -1;
|
||||
char name[64];
|
||||
for (unsigned int i = 0; i < 100; i++) {
|
||||
snprintf(name, sizeof(name), ".fastsync-spool.%ld.%llu", (long)getpid(), next_temp_sequence());
|
||||
int fd = openat(dirfd, name, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
|
||||
if (fd < 0) {
|
||||
if (errno != EEXIST)
|
||||
break;
|
||||
continue;
|
||||
}
|
||||
char* copy = str_dup(dest_path);
|
||||
const char* dir = copy ? dirname(copy) : NULL;
|
||||
size_t need = dir ? strlen(dir) + 1 + strlen(name) + 1 : 0;
|
||||
char* full = need ? malloc(need) : NULL;
|
||||
if (!full) {
|
||||
free(copy);
|
||||
close(fd);
|
||||
unlinkat(dirfd, name, 0);
|
||||
close(dirfd);
|
||||
return -1;
|
||||
}
|
||||
snprintf(full, need, "%s/%s", dir, name);
|
||||
free(copy);
|
||||
close(dirfd);
|
||||
*out_spool_path = full;
|
||||
return fd;
|
||||
}
|
||||
close(dirfd);
|
||||
return -1;
|
||||
}
|
||||
|
||||
int file_spool_for_payload(const char* dest_path, char** out_spool_path) {
|
||||
return file_spool_create(dest_path, out_spool_path);
|
||||
}
|
||||
|
||||
bool file_receive_payload(int fd, bool compress, unsigned long long expected_size,
|
||||
const char* dest_path, unsigned long long stream_limit, Data** out_buffer,
|
||||
char** out_spool, unsigned long long* out_size) {
|
||||
if (out_buffer)
|
||||
*out_buffer = NULL;
|
||||
if (out_spool)
|
||||
*out_spool = NULL;
|
||||
if (out_size)
|
||||
*out_size = 0;
|
||||
if (!out_buffer || !out_spool || !out_size || !dest_path)
|
||||
return false;
|
||||
|
||||
unsigned long long frame_size = 0;
|
||||
if (!receive_n_data(fd, &frame_size, sizeof(frame_size)))
|
||||
return false;
|
||||
/* A frame larger than MAX_DATA_PAYLOAD_SIZE is allowed only on the streamed
|
||||
path, which never materializes it; the buffered path's
|
||||
receive_data_alloc/body enforces the historical bound itself. Only a size
|
||||
unrepresentable on this platform is rejected here. */
|
||||
if (frame_size > SIZE_MAX) {
|
||||
log_message(LOG_LEVEL_ERROR, "Data size %llu is not representable", frame_size);
|
||||
return false;
|
||||
}
|
||||
|
||||
unsigned char prefix[FILE_PAYLOAD_PEEK_MAX];
|
||||
size_t prefix_len = 0;
|
||||
bool stream;
|
||||
if (expected_size != 0) {
|
||||
/* The check frame already told us the logical size: no need to peek. */
|
||||
stream = expected_size > stream_limit;
|
||||
} else if (!compress) {
|
||||
stream = frame_size > stream_limit;
|
||||
} else {
|
||||
/* Non-incremental compressed frame with unknown logical size: peek the
|
||||
header to decide, so a small compressed frame that expands past the bound
|
||||
still streams instead of allocating the whole logical image. */
|
||||
size_t want = frame_size < sizeof(prefix) ? (size_t)frame_size : sizeof(prefix);
|
||||
if (want > 0 && !receive_n_data(fd, prefix, want))
|
||||
return false;
|
||||
prefix_len = want;
|
||||
unsigned long long logical = compression_peek_frame_content_size(prefix, prefix_len);
|
||||
stream = logical != 0 ? logical > stream_limit : frame_size > stream_limit;
|
||||
}
|
||||
|
||||
if (!stream) {
|
||||
Data* frame;
|
||||
if (prefix_len > 0) {
|
||||
frame = receive_data_alloc(fd, frame_size);
|
||||
if (!frame)
|
||||
return false;
|
||||
memcpy(frame->data, prefix, prefix_len);
|
||||
if (frame_size > prefix_len &&
|
||||
!receive_n_data(fd, (char*)frame->data + prefix_len, (size_t)(frame_size - prefix_len))) {
|
||||
data_destroy(frame);
|
||||
return false;
|
||||
}
|
||||
} else {
|
||||
frame = receive_data_body(fd, frame_size);
|
||||
if (!frame)
|
||||
return false;
|
||||
}
|
||||
if (compress) {
|
||||
unsigned long long bound =
|
||||
expected_size != 0 ? expected_size : (unsigned long long)MAX_RECEIVE_WHOLE_FILE_SIZE;
|
||||
Data* uncompressed = data_decompress_limited(frame, (size_t)bound);
|
||||
ProtocolSession* owner = frame->owner;
|
||||
data_destroy(frame);
|
||||
if (!uncompressed)
|
||||
return false;
|
||||
if (expected_size != 0 && uncompressed->size != expected_size) {
|
||||
data_destroy(uncompressed);
|
||||
return false;
|
||||
}
|
||||
if (!data_charge_session(uncompressed, owner, uncompressed->size)) {
|
||||
data_destroy(uncompressed);
|
||||
return false;
|
||||
}
|
||||
*out_buffer = uncompressed;
|
||||
*out_size = uncompressed->size;
|
||||
return true;
|
||||
}
|
||||
*out_buffer = frame;
|
||||
*out_size = frame->size;
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Streamed path. */
|
||||
int spool_fd = file_spool_create(dest_path, out_spool);
|
||||
if (spool_fd < 0)
|
||||
return false;
|
||||
|
||||
CompressionStreamDecompressor* dec = NULL;
|
||||
size_t header_len = 0;
|
||||
unsigned long long learned_size = expected_size;
|
||||
if (compress) {
|
||||
unsigned char hdr[1 + 4];
|
||||
size_t hdr_have = 0;
|
||||
if (prefix_len >= 1) {
|
||||
hdr[0] = prefix[0];
|
||||
hdr_have = 1;
|
||||
} else {
|
||||
if (!receive_n_data(fd, hdr, 1))
|
||||
goto stream_fail;
|
||||
hdr_have = 1;
|
||||
}
|
||||
CompressionAlgo algo = (CompressionAlgo)hdr[0];
|
||||
if (!compression_algo_valid((int)algo) || algo == COMPRESSION_ALGO_LZ4)
|
||||
goto stream_fail;
|
||||
header_len = (algo == COMPRESSION_ALGO_ZLIB || algo == COMPRESSION_ALGO_ZLIBX) ? 5 : 1;
|
||||
while (hdr_have < header_len) {
|
||||
size_t need = header_len - hdr_have;
|
||||
if (prefix_len > hdr_have) {
|
||||
size_t avail = prefix_len - hdr_have;
|
||||
size_t take = avail < need ? avail : need;
|
||||
memcpy(hdr + hdr_have, prefix + hdr_have, take);
|
||||
hdr_have += take;
|
||||
} else {
|
||||
if (!receive_n_data(fd, hdr + hdr_have, need))
|
||||
goto stream_fail;
|
||||
hdr_have = header_len;
|
||||
}
|
||||
}
|
||||
if (header_len == 5) {
|
||||
uint32_t raw = 0;
|
||||
for (int i = 0; i < 4; i++)
|
||||
raw |= (uint32_t)hdr[1 + i] << (8 * i);
|
||||
if (expected_size != 0 && raw != expected_size)
|
||||
goto stream_fail;
|
||||
if (learned_size == 0)
|
||||
learned_size = raw;
|
||||
}
|
||||
dec = compression_stream_decompressor_create(algo, learned_size);
|
||||
if (!dec)
|
||||
goto stream_fail;
|
||||
}
|
||||
|
||||
if (frame_size < header_len)
|
||||
goto stream_fail;
|
||||
{
|
||||
unsigned long long body_remaining = frame_size - header_len;
|
||||
if (prefix_len > header_len) {
|
||||
size_t avail = prefix_len - header_len;
|
||||
if (compress) {
|
||||
bool done = false;
|
||||
if (!compression_stream_decompressor_feed(dec, prefix + header_len, avail, spool_fd, &done))
|
||||
goto stream_fail;
|
||||
} else if (!write_all(spool_fd, prefix + header_len, avail)) {
|
||||
goto stream_fail;
|
||||
}
|
||||
body_remaining -= avail;
|
||||
}
|
||||
unsigned char buf[FILE_PAYLOAD_READ_CHUNK];
|
||||
while (body_remaining > 0) {
|
||||
size_t want = body_remaining < sizeof(buf) ? (size_t)body_remaining : (size_t)sizeof(buf);
|
||||
if (!receive_n_data(fd, buf, want))
|
||||
goto stream_fail;
|
||||
if (compress) {
|
||||
bool done = false;
|
||||
if (!compression_stream_decompressor_feed(dec, buf, want, spool_fd, &done))
|
||||
goto stream_fail;
|
||||
} else if (!write_all(spool_fd, buf, want)) {
|
||||
goto stream_fail;
|
||||
}
|
||||
body_remaining -= want;
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
unsigned long long total = compress ? compression_stream_decompressor_total(dec) : frame_size;
|
||||
if (learned_size != 0 && total != learned_size)
|
||||
goto stream_fail;
|
||||
*out_size = total;
|
||||
}
|
||||
if (dec)
|
||||
compression_stream_decompressor_destroy(dec);
|
||||
if (close(spool_fd) != 0) {
|
||||
spool_fd = -1;
|
||||
goto stream_fail;
|
||||
}
|
||||
return true;
|
||||
|
||||
stream_fail:
|
||||
if (dec)
|
||||
compression_stream_decompressor_destroy(dec);
|
||||
if (spool_fd >= 0)
|
||||
close(spool_fd);
|
||||
if (*out_spool) {
|
||||
unlink(*out_spool);
|
||||
free(*out_spool);
|
||||
*out_spool = NULL;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/* ---- Secure filesystem primitives ---- */
|
||||
|
||||
bool file_path_exists_secure(const char* path) {
|
||||
@@ -911,103 +614,6 @@ static int open_dir_beneath_root(const char* resolved, const char* root) {
|
||||
}
|
||||
|
||||
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs) {
|
||||
return file_open_secure_parent_counted(path, leaf_out, create_dirs, NULL, NULL);
|
||||
}
|
||||
|
||||
/* The logical transfer root expressed in the same coordinate as the secure
|
||||
* parent walk's `rel_buf` (relative to the authorized root, with a leading
|
||||
* '/'), used as the floor at or below which a created directory is a real
|
||||
* file-list entry. The on-disk transfer root is the receive root joined to the
|
||||
* wire path; the mirror scaffolding above it (the absolute source path below
|
||||
* the destination root) is not an rsync entry. Returns an allocated string or
|
||||
* NULL (count every created component). */
|
||||
static char* transfer_root_floor(const Config* config) {
|
||||
if (!config || !config->send_directory || config->send_directory[0] == '\0')
|
||||
return NULL;
|
||||
const char* spec = config->send_directory;
|
||||
const char* after = spec;
|
||||
if (spec[0] == '.' && spec[1] == '/') {
|
||||
after = spec + 2;
|
||||
} else {
|
||||
const char* cut = strstr(spec, "/./");
|
||||
if (cut)
|
||||
after = cut + 3;
|
||||
}
|
||||
while (*after == '/')
|
||||
after++;
|
||||
char* wire_root = str_dup(after);
|
||||
if (!wire_root)
|
||||
return NULL;
|
||||
size_t wlen = strlen(wire_root);
|
||||
while (wlen > 0 && wire_root[wlen - 1] == '/')
|
||||
wire_root[--wlen] = '\0';
|
||||
if (wlen == 0) {
|
||||
free(wire_root);
|
||||
return NULL;
|
||||
}
|
||||
char* disk_root = config->receive_root_directory
|
||||
? path_cat(config->receive_root_directory, wire_root)
|
||||
: str_dup(wire_root);
|
||||
free(wire_root);
|
||||
if (!disk_root)
|
||||
return NULL;
|
||||
const char* root_path = utils_get_authorized_root_path();
|
||||
const char* floor = disk_root;
|
||||
if (root_path && root_path[0] == '/') {
|
||||
size_t rl = strlen(root_path);
|
||||
while (rl > 0 && root_path[rl - 1] == '/')
|
||||
rl--;
|
||||
if (strncmp(disk_root, root_path, rl) == 0 && (disk_root[rl] == '/' || disk_root[rl] == '\0'))
|
||||
floor = disk_root + rl;
|
||||
}
|
||||
while (*floor == '/')
|
||||
floor++;
|
||||
char* out = str_dup(floor);
|
||||
free(disk_root);
|
||||
if (!out)
|
||||
return NULL;
|
||||
if (out[0] == '\0') {
|
||||
free(out);
|
||||
return NULL;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/* A created parent component counts toward `Number of created files` only when
|
||||
* its receive-root-relative path is at or below the logical transfer root
|
||||
* (`count_floor`). The transfer root itself corresponds to rsync's `.` entry
|
||||
* (created on a fresh destination, pre-existing otherwise); the mirror
|
||||
* scaffolding above it is FastSync's absolute-path layout, not an rsync entry. */
|
||||
static bool created_dir_counts(const char* count_floor, const char* rel_buf,
|
||||
const char* component) {
|
||||
if (!count_floor)
|
||||
return true;
|
||||
char candidate[PATH_MAX];
|
||||
int n = snprintf(candidate, sizeof(candidate), "%s/%s", rel_buf, component);
|
||||
if (n < 0 || (size_t)n >= sizeof(candidate))
|
||||
return false;
|
||||
const char* cand = candidate;
|
||||
while (*cand == '/')
|
||||
cand++;
|
||||
size_t fl = strlen(count_floor);
|
||||
if (strncmp(cand, count_floor, fl) != 0)
|
||||
return false;
|
||||
return cand[fl] == '\0' || cand[fl] == '/';
|
||||
}
|
||||
|
||||
/* Public wrapper for the receiver's created-directory accounting: the logical
|
||||
* transfer root expressed receive-root-relative, or NULL when the wire paths
|
||||
* carry no mirror scaffolding above it (--relative and --files-from, whose
|
||||
* paths are already relative to the transfer root). The caller frees a
|
||||
* non-NULL result. */
|
||||
char* file_transfer_root_floor(const Config* config) {
|
||||
if (!config || config->relative || config->files_from_set != NULL)
|
||||
return NULL;
|
||||
return transfer_root_floor(config);
|
||||
}
|
||||
|
||||
int file_open_secure_parent_counted(const char* path, char** leaf_out, bool create_dirs,
|
||||
unsigned* dirs_created, const char* count_floor) {
|
||||
char* copy = str_dup(path);
|
||||
if (!copy)
|
||||
return -1;
|
||||
@@ -1068,14 +674,6 @@ int file_open_secure_parent_counted(const char* path, char** leaf_out, bool crea
|
||||
if (next < 0 && create_dirs && errno == ENOENT) {
|
||||
bool created = mkdirat(fd, component, (mode_t)(0777 & ~(mode_t)file_process_umask())) == 0;
|
||||
if (created || errno == EEXIST) {
|
||||
/* Protocol 2.28.0: only directories the logical file list would
|
||||
create count toward `Number of created files`; the mirror
|
||||
scaffolding above the transfer root (e.g. the absolute source path
|
||||
under the destination root) is not an rsync entry. `count_floor`
|
||||
is a receive-root-relative prefix that must be reached before a
|
||||
created component is counted. */
|
||||
if (created && dirs_created && created_dir_counts(count_floor, rel_buf, component))
|
||||
(*dirs_created)++;
|
||||
/* P7 Wave E: --copy-as owns EVERY entry, including the intermediate
|
||||
directories this walk creates implicitly. Its target ids are a
|
||||
global policy, so they are available here without per-entry source
|
||||
@@ -1209,25 +807,6 @@ bool file_ensure_directory_secure(const char* path) {
|
||||
} else if (errno == EEXIST) {
|
||||
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
}
|
||||
} else if (dir_fd < 0 && errno == ENOTDIR) {
|
||||
/* rsync replaces a destination non-directory (regular file) with an
|
||||
incoming directory. Confined to the already-opened secure parent fd:
|
||||
the leaf is unlinked by name (never followed) and only a non-directory
|
||||
is ever removed, so this cannot escape the authorized root or remove a
|
||||
pre-existing directory tree. A symlink is left alone (openat with
|
||||
O_NOFOLLOW reports ELOOP, which takes no branch here), since replacing
|
||||
it is not required for FastSync's transferred directories and keeps
|
||||
--keep-dirlinks semantics untouched. */
|
||||
struct stat leaf_st;
|
||||
if (fstatat(parent_fd, leaf, &leaf_st, AT_SYMLINK_NOFOLLOW) == 0 && !S_ISDIR(leaf_st.st_mode) &&
|
||||
!S_ISLNK(leaf_st.st_mode)) {
|
||||
if (unlinkat(parent_fd, leaf, 0) == 0) {
|
||||
if (mkdirat(parent_fd, leaf, (mode_t)(0777 & ~(mode_t)file_process_umask())) == 0)
|
||||
created = true;
|
||||
/* On failure dir_fd stays < 0 below, so the caller still sees it. */
|
||||
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
}
|
||||
}
|
||||
}
|
||||
bool ok = dir_fd >= 0;
|
||||
/* --copy-as owns a directory this call just created (the final component;
|
||||
@@ -1390,51 +969,17 @@ int file_open_private_dir(const char* dir_path) {
|
||||
return fd;
|
||||
}
|
||||
|
||||
/* Open a --temp-dir scratch directory. The directory must already exist (rsync
|
||||
* never creates it); a relative path was already resolved against the
|
||||
* destination root by the caller. Unlike file_open_private_dir this neither
|
||||
* creates it nor requires it to be a direct child of the receive root, because
|
||||
* rsync permits a scratch dir that (via a symlink) lands on another filesystem
|
||||
* -- but it MUST resolve inside the authorized receive root. The directory is
|
||||
* opened following symlinks and then judged by the REAL path of the opened fd
|
||||
* (through /proc/self/fd), so a client-planted symlink under the receive root
|
||||
* can never redirect receiver scratch files outside the sandbox while an
|
||||
* in-root link to another filesystem (the EXDEV fallback case) still works.
|
||||
* Returns an O_DIRECTORY|O_CLOEXEC fd, or -1 on error (errno set; an escaping
|
||||
* target is reported as EACCES with a logged reason). */
|
||||
/* Open a --temp-dir scratch directory exactly as rsync does: the directory must
|
||||
* already exist and is used as given (an absolute path is used verbatim, a
|
||||
* relative one was already resolved against the destination root by the
|
||||
* caller). Unlike file_open_private_dir this neither creates it nor confines
|
||||
* it below the receive root, because rsync accepts any temp dir -- including
|
||||
* one outside the destination tree or on another filesystem. Returns an
|
||||
* O_DIRECTORY|O_CLOEXEC fd, or -1 on error. */
|
||||
int file_open_temp_dir(const char* dir_path) {
|
||||
if (!dir_path)
|
||||
return -1;
|
||||
int fd = open(dir_path, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
|
||||
if (fd < 0)
|
||||
return -1;
|
||||
const char* root = utils_get_authorized_root_path();
|
||||
if (!root) {
|
||||
/* No authorized root (e.g. a local batch apply): nothing to confine
|
||||
against, so preserve the historical open-as-given behavior. */
|
||||
return fd;
|
||||
}
|
||||
char fd_path[64];
|
||||
int fd_path_length = snprintf(fd_path, sizeof(fd_path), "/proc/self/fd/%d", fd);
|
||||
char resolved[PATH_MAX];
|
||||
if (fd_path_length < 0 || (size_t)fd_path_length >= sizeof(fd_path) ||
|
||||
!realpath(fd_path, resolved)) {
|
||||
int saved_errno = errno;
|
||||
close(fd);
|
||||
errno = saved_errno;
|
||||
return -1;
|
||||
}
|
||||
if (!path_is_within_root(root, resolved)) {
|
||||
char* escaped = output_escape(dir_path, log_get_8_bit_output());
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"--temp-dir '%s' resolves outside the authorized receive root; refusing",
|
||||
escaped ? escaped : "<allocation failed>");
|
||||
free(escaped);
|
||||
close(fd);
|
||||
errno = EACCES;
|
||||
return -1;
|
||||
}
|
||||
return fd;
|
||||
return open(dir_path, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
|
||||
}
|
||||
|
||||
/* After the content and mode/times are restored on the just-written file, apply
|
||||
@@ -1443,37 +988,34 @@ int file_open_temp_dir(const char* dir_path) {
|
||||
* destination file) and best-effort: a per-attribute or privilege failure is
|
||||
* logged and skipped, never fatal. */
|
||||
static void restore_extra_fd(int fd, const FileMetadata* metadata, const FileXattrList* xattrs,
|
||||
bool fake_super, FileAttrPolicy policy, uint32_t fake_super_rdev_major,
|
||||
uint32_t fake_super_rdev_minor) {
|
||||
bool fake_super, FileAttrPolicy policy) {
|
||||
xattr_apply_fd(fd, xattrs);
|
||||
if (fake_super && metadata) {
|
||||
/* Record the ownership that WOULD have been applied: when an explicit
|
||||
ownership request (--chown/--usermap/--groupmap/--copy-as or -o/-g) is
|
||||
active, the resolved mapping; otherwise the source's own id. The real
|
||||
chown is suppressed (identity_apply_ownership early-returns under
|
||||
--fake-super) so recording never defeats the flag. The recorded stat is
|
||||
rsync's format; the permission bits are replayed (policy-gated) so
|
||||
unprivileged --fake-super keeps working while mtime comes from the
|
||||
normal metadata path above. */
|
||||
--fake-super) so recording never defeats the flag. Mode/mtime are still
|
||||
replayed (policy-gated) so unprivileged --fake-super keeps working. */
|
||||
uint32_t store_uid;
|
||||
uint32_t store_gid;
|
||||
identity_resolve_storage_ids((int32_t)metadata->uid, (int32_t)metadata->gid, &store_uid,
|
||||
&store_gid);
|
||||
fake_super_store_fd(fd, store_uid, store_gid, (uint32_t)metadata->mode, fake_super_rdev_major,
|
||||
fake_super_rdev_minor);
|
||||
fake_super_store_fd(fd, store_uid, store_gid, (uint32_t)metadata->mode, metadata->mtime_sec,
|
||||
metadata->mtime_nsec);
|
||||
fake_super_restore_fd(fd, policy);
|
||||
}
|
||||
}
|
||||
|
||||
static bool
|
||||
file_to_disk_secure_impl(const char* path, const void* data, unsigned long long data_size,
|
||||
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
|
||||
FileAttrPolicy policy, bool update, bool no_replace, bool use_fsync,
|
||||
const char* temp_dir, const FileXattrList* xattrs, bool fake_super,
|
||||
bool keep_partial, unsigned* dirs_created, const char* count_floor,
|
||||
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor) {
|
||||
static bool file_to_disk_secure_impl(const char* path, const void* data,
|
||||
unsigned long long data_size, bool inplace, bool sparse,
|
||||
bool preallocate, const FileMetadata* metadata,
|
||||
FileAttrPolicy policy, bool update, bool no_replace,
|
||||
bool use_fsync, const char* temp_dir,
|
||||
const FileXattrList* xattrs, bool fake_super,
|
||||
bool keep_partial) {
|
||||
char* leaf = NULL;
|
||||
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||
int dirfd = file_open_secure_parent(path, &leaf, true);
|
||||
if (dirfd < 0)
|
||||
return false;
|
||||
int fd = -1;
|
||||
@@ -1578,8 +1120,7 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
|
||||
}
|
||||
}
|
||||
if (ok)
|
||||
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, fake_super_rdev_major,
|
||||
fake_super_rdev_minor);
|
||||
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
|
||||
if (ok && use_fsync)
|
||||
ok = fsync(fd) == 0;
|
||||
}
|
||||
@@ -1697,8 +1238,7 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
|
||||
}
|
||||
}
|
||||
if (ok)
|
||||
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, fake_super_rdev_major,
|
||||
fake_super_rdev_minor);
|
||||
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
|
||||
if (ok && use_fsync)
|
||||
ok = fsync(fd) == 0;
|
||||
}
|
||||
@@ -1766,8 +1306,7 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
|
||||
"non-atomic copy into the destination directory");
|
||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||
policy, update, no_replace, use_fsync, NULL, xattrs, fake_super,
|
||||
keep_partial, dirs_created, count_floor, fake_super_rdev_major,
|
||||
fake_super_rdev_minor);
|
||||
keep_partial);
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
@@ -1776,8 +1315,7 @@ bool file_to_disk_secure(const char* path, const void* data, unsigned long long
|
||||
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
|
||||
FileAttrPolicy policy, const char* temp_dir) {
|
||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||
policy, false, false, false, temp_dir, NULL, false, false, NULL,
|
||||
NULL, 0, 0);
|
||||
policy, false, false, false, temp_dir, NULL, false, false);
|
||||
}
|
||||
|
||||
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
|
||||
@@ -1785,8 +1323,7 @@ bool file_to_disk_secure_update(const char* path, const void* data, unsigned lon
|
||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||
const char* temp_dir) {
|
||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||
policy, true, false, false, temp_dir, NULL, false, false, NULL,
|
||||
NULL, 0, 0);
|
||||
policy, true, false, false, temp_dir, NULL, false, false);
|
||||
}
|
||||
|
||||
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
|
||||
@@ -1794,8 +1331,7 @@ bool file_to_disk_secure_with_fsync(const char* path, const void* data,
|
||||
bool preallocate, const FileMetadata* metadata,
|
||||
FileAttrPolicy policy, bool use_fsync, const char* temp_dir) {
|
||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||
policy, false, false, use_fsync, temp_dir, NULL, false, false,
|
||||
NULL, NULL, 0, 0);
|
||||
policy, false, false, use_fsync, temp_dir, NULL, false, false);
|
||||
}
|
||||
|
||||
bool file_to_disk_secure_no_replace(const char* path, const void* data,
|
||||
@@ -1803,8 +1339,7 @@ bool file_to_disk_secure_no_replace(const char* path, const void* data,
|
||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||
const char* temp_dir) {
|
||||
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
|
||||
policy, false, true, false, temp_dir, NULL, false, false, NULL,
|
||||
NULL, 0, 0);
|
||||
policy, false, true, false, temp_dir, NULL, false, false);
|
||||
}
|
||||
|
||||
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
|
||||
@@ -1817,21 +1352,9 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
|
||||
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||
bool no_replace, bool use_fsync, const FileXattrList* xattrs,
|
||||
bool fake_super, bool keep_partial, const char* temp_dir) {
|
||||
return file_to_disk_secure_attrs_counted(path, data, data_size, inplace, sparse, preallocate,
|
||||
metadata, policy, update, no_replace, use_fsync, xattrs,
|
||||
fake_super, keep_partial, temp_dir, NULL, NULL, 0, 0);
|
||||
}
|
||||
|
||||
bool file_to_disk_secure_attrs_counted(
|
||||
const char* path, const void* data, unsigned long long data_size, bool inplace, bool sparse,
|
||||
bool preallocate, const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||
bool no_replace, bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||
bool keep_partial, const char* temp_dir, unsigned* dirs_created, const char* count_floor,
|
||||
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor) {
|
||||
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
|
||||
policy, update, no_replace, use_fsync, temp_dir, xattrs,
|
||||
fake_super, keep_partial, dirs_created, count_floor,
|
||||
fake_super_rdev_major, fake_super_rdev_minor);
|
||||
fake_super, keep_partial);
|
||||
}
|
||||
|
||||
/* Atomic --link-dest install. The destination is replaced (via a temporary
|
||||
@@ -1849,176 +1372,16 @@ bool file_to_disk_secure_attrs_counted(
|
||||
* basis). Likewise `xattrs`/`fake_super` are applied only on the copy
|
||||
* fallback, so a fallback copy preserves the per-file attributes instead of
|
||||
* silently dropping them. */
|
||||
/* Streaming --copy-dest basis install: atomically materialize `path` from the
|
||||
* bytes of `basis_path` without holding the file in memory, so a basis larger
|
||||
* than any whole-file bound still works. Mirrors the ordinary secure store
|
||||
* path (confined parent walk, temp + rename, --update/--ignore-existing/
|
||||
* --preallocate/--temp-dir) but sources the data from the basis descriptor
|
||||
* rather than a caller buffer, and applies the SOURCE metadata (rsync copies
|
||||
* then fixes attributes). A hard-link install that falls back to a byte copy
|
||||
* also routes through here when the caller supplies the basis path. */
|
||||
static bool file_copy_basis_stream_impl(const char* path, const char* basis_path,
|
||||
unsigned long long expected_size, bool preallocate,
|
||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||
bool update, bool no_replace, bool use_fsync,
|
||||
const FileXattrList* xattrs, bool fake_super,
|
||||
const char* temp_dir, unsigned* dirs_created,
|
||||
const char* count_floor) {
|
||||
if (!path || !basis_path)
|
||||
return false;
|
||||
char* leaf = NULL;
|
||||
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||
if (dirfd < 0)
|
||||
return false;
|
||||
|
||||
char* basis_leaf = NULL;
|
||||
int basis_dirfd = file_open_secure_parent(basis_path, &basis_leaf, false);
|
||||
int src_fd = -1;
|
||||
if (basis_dirfd >= 0 && basis_leaf != NULL) {
|
||||
/* O_NONBLOCK rejects a raced-in FIFO without blocking; the S_ISREG gate
|
||||
below is the real type check. */
|
||||
src_fd = openat(basis_dirfd, basis_leaf, O_RDONLY | O_CLOEXEC | O_NOFOLLOW | O_NONBLOCK);
|
||||
struct stat src_st;
|
||||
if (src_fd >= 0 && (fstat(src_fd, &src_st) != 0 || !S_ISREG(src_st.st_mode))) {
|
||||
close(src_fd);
|
||||
src_fd = -1;
|
||||
}
|
||||
}
|
||||
if (basis_dirfd >= 0)
|
||||
close(basis_dirfd);
|
||||
free(basis_leaf);
|
||||
if (src_fd < 0) {
|
||||
close(dirfd);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
|
||||
struct stat destination_stat;
|
||||
bool destination_is_regular = fstatat(dirfd, leaf, &destination_stat, AT_SYMLINK_NOFOLLOW) == 0 &&
|
||||
S_ISREG(destination_stat.st_mode);
|
||||
if (update && metadata && destination_is_regular && stat_is_newer(&destination_stat, metadata)) {
|
||||
close(src_fd);
|
||||
close(dirfd);
|
||||
free(leaf);
|
||||
return true;
|
||||
}
|
||||
if (no_replace && file_path_exists_secure(path)) {
|
||||
close(src_fd);
|
||||
close(dirfd);
|
||||
free(leaf);
|
||||
return true;
|
||||
}
|
||||
|
||||
int scratch_dirfd = -1;
|
||||
if (temp_dir) {
|
||||
scratch_dirfd = file_open_temp_dir(temp_dir);
|
||||
if (scratch_dirfd < 0) {
|
||||
int saved_errno = errno;
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"--temp-dir '%s' could not be opened (rsync requires it to already exist): %s",
|
||||
temp_dir, strerror(saved_errno));
|
||||
close(src_fd);
|
||||
close(dirfd);
|
||||
free(leaf);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
int target_dirfd = scratch_dirfd >= 0 ? scratch_dirfd : dirfd;
|
||||
int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%llu", leaf, (long)getpid(), ~0ULL);
|
||||
char* tmp = NULL;
|
||||
bool ok = false;
|
||||
if (tmp_size >= 0)
|
||||
tmp = malloc((size_t)tmp_size + 1);
|
||||
if (tmp) {
|
||||
for (unsigned int i = 0; i < 100 && !ok; ++i) {
|
||||
if (scratch_dirfd >= 0)
|
||||
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%llu", leaf, (long)getpid(),
|
||||
next_temp_sequence());
|
||||
else
|
||||
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
|
||||
int fd =
|
||||
openat(target_dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
|
||||
if (fd < 0) {
|
||||
if (errno != EEXIST)
|
||||
break;
|
||||
continue;
|
||||
}
|
||||
bool wrote = true;
|
||||
if (preallocate && expected_size > 0 && preallocate_fd(fd, expected_size) != 0)
|
||||
wrote = false;
|
||||
if (wrote)
|
||||
wrote = copy_fd_all(fd, src_fd, expected_size);
|
||||
if (wrote && metadata) {
|
||||
if (!policy.perms &&
|
||||
fchmod(fd, file_mode_base(metadata, destination_is_regular,
|
||||
destination_is_regular ? destination_stat.st_mode & 0777
|
||||
: 0)) != 0)
|
||||
wrote = false;
|
||||
if (wrote)
|
||||
wrote = file_restore_metadata_fd(fd, metadata, policy);
|
||||
} else if (wrote && fchmod(fd, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH) != 0) {
|
||||
wrote = false;
|
||||
}
|
||||
if (wrote)
|
||||
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, 0, 0);
|
||||
if (wrote && use_fsync)
|
||||
wrote = fsync(fd) == 0;
|
||||
if (close(fd) != 0)
|
||||
wrote = false;
|
||||
if (wrote && renameat(target_dirfd, tmp, dirfd, leaf) != 0)
|
||||
wrote = false;
|
||||
if (!wrote)
|
||||
unlinkat(target_dirfd, tmp, 0);
|
||||
ok = wrote;
|
||||
}
|
||||
free(tmp);
|
||||
}
|
||||
if (!ok && scratch_dirfd >= 0) {
|
||||
/* Retry once with no scratch dir (rsync's EXDEV fallback). */
|
||||
close(scratch_dirfd);
|
||||
close(src_fd);
|
||||
close(dirfd);
|
||||
free(leaf);
|
||||
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata,
|
||||
policy, update, no_replace, use_fsync, xattrs, fake_super,
|
||||
NULL, dirs_created, count_floor);
|
||||
}
|
||||
if (scratch_dirfd >= 0)
|
||||
close(scratch_dirfd);
|
||||
close(src_fd);
|
||||
close(dirfd);
|
||||
free(leaf);
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* --copy-dest basis install (streaming). Applies the source metadata and the
|
||||
per-file xattrs / --fake-super record. */
|
||||
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
|
||||
unsigned long long expected_size, bool preallocate,
|
||||
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||
const char* temp_dir) {
|
||||
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata, policy,
|
||||
update, false, use_fsync, xattrs, fake_super, temp_dir, NULL,
|
||||
NULL);
|
||||
}
|
||||
|
||||
static bool file_to_disk_secure_link_impl(const char* path, const char* basis_path,
|
||||
const void* data, unsigned long long data_size,
|
||||
bool preallocate, const FileMetadata* metadata,
|
||||
FileAttrPolicy policy, bool use_fsync,
|
||||
const FileXattrList* xattrs, bool fake_super,
|
||||
const char* temp_dir, unsigned* dirs_created,
|
||||
const char* count_floor) {
|
||||
const char* temp_dir) {
|
||||
if (!path || !basis_path)
|
||||
return false;
|
||||
/* The caller-supplied buffer is no longer used: the copy fallback streams
|
||||
from the basis path (which may hold an over-limit file). Kept in the
|
||||
signature for the existing API. */
|
||||
(void)data;
|
||||
char* leaf = NULL;
|
||||
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||
int dirfd = file_open_secure_parent(path, &leaf, true);
|
||||
if (dirfd < 0)
|
||||
return false;
|
||||
|
||||
@@ -2100,17 +1463,10 @@ static bool file_to_disk_secure_link_impl(const char* path, const char* basis_pa
|
||||
close(dirfd);
|
||||
free(leaf);
|
||||
/* The basis file could not be linked in (missing, cross-device, refused
|
||||
by the filesystem). Stream a byte-identical local copy from the basis
|
||||
itself (never the possibly-absent caller buffer) so an over-limit basis
|
||||
still materializes. When the basis path is not a readable regular file
|
||||
(e.g. a directory raced in), fall back to the caller-supplied bytes. */
|
||||
if (file_copy_basis_stream_impl(path, basis_path, data_size, preallocate, metadata, policy,
|
||||
false, false, use_fsync, xattrs, fake_super, temp_dir,
|
||||
dirs_created, count_floor))
|
||||
return true;
|
||||
return file_to_disk_secure_attrs_counted(
|
||||
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
|
||||
xattrs, fake_super, false, temp_dir, dirs_created, count_floor, 0, 0);
|
||||
by the filesystem). Write a byte-identical local copy instead. */
|
||||
return file_to_disk_secure_attrs(path, data, data_size, false, false, preallocate, metadata,
|
||||
policy, false, false, use_fsync, xattrs, fake_super, false,
|
||||
temp_dir);
|
||||
}
|
||||
|
||||
if (scratch_dirfd >= 0)
|
||||
@@ -2125,7 +1481,7 @@ bool file_to_disk_secure_link(const char* path, const char* basis_path, const vo
|
||||
const FileMetadata* metadata, FileAttrPolicy policy, bool use_fsync,
|
||||
const char* temp_dir) {
|
||||
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
|
||||
policy, use_fsync, NULL, false, temp_dir, NULL, NULL);
|
||||
policy, use_fsync, NULL, false, temp_dir);
|
||||
}
|
||||
|
||||
bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, const void* data,
|
||||
@@ -2133,27 +1489,14 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
|
||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||
const char* temp_dir) {
|
||||
return file_to_disk_secure_link_attrs_counted(path, basis_path, data, data_size, preallocate,
|
||||
metadata, policy, use_fsync, xattrs, fake_super,
|
||||
temp_dir, NULL, NULL);
|
||||
}
|
||||
|
||||
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
|
||||
const void* data, unsigned long long data_size,
|
||||
bool preallocate, const FileMetadata* metadata,
|
||||
FileAttrPolicy policy, bool use_fsync,
|
||||
const FileXattrList* xattrs, bool fake_super,
|
||||
const char* temp_dir, unsigned* dirs_created,
|
||||
const char* count_floor) {
|
||||
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
|
||||
policy, use_fsync, xattrs, fake_super, temp_dir,
|
||||
dirs_created, count_floor);
|
||||
policy, use_fsync, xattrs, fake_super, temp_dir);
|
||||
}
|
||||
|
||||
bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
|
||||
bool inplace, bool sparse) {
|
||||
if (!path || (!data && data_size != 0) || has_path_traversal(path))
|
||||
return false;
|
||||
FileAttrPolicy policy = {0};
|
||||
FileAttrPolicy policy = {false, false, false, false};
|
||||
return file_to_disk_secure(path, data, data_size, inplace, sparse, false, NULL, policy, NULL);
|
||||
}
|
||||
|
||||
+2
-60
@@ -87,11 +87,6 @@ bool file_path_exists_secure(const char* path);
|
||||
bool file_stat_secure(const char* path, struct stat* st);
|
||||
bool file_destination_is_newer_secure(const char* path, const FileMetadata* metadata);
|
||||
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
|
||||
/* Protocol 2.28.0 variant: also increments *dirs_created for every missing
|
||||
* parent directory this walk creates that lies strictly below `count_floor`
|
||||
* (a receive-root-relative path, or NULL to count all of them). */
|
||||
int file_open_secure_parent_counted(const char* path, char** leaf_out, bool create_dirs,
|
||||
unsigned* dirs_created, const char* count_floor);
|
||||
bool file_ensure_directory_secure(const char* path);
|
||||
bool file_directory_exists_secure(const char* path);
|
||||
bool file_rename_secure(const char* old_path, const char* new_path);
|
||||
@@ -103,12 +98,8 @@ bool file_remove_tree_secure(const char* path);
|
||||
the authorized root. Used for the --delay-updates staging directory. */
|
||||
int file_open_private_dir(const char* dir_path);
|
||||
|
||||
/* Open an existing --temp-dir scratch directory (relative or absolute; no
|
||||
creation). When an authorized receive root is configured the directory's
|
||||
REAL path (symlinks resolved) must lie within it, so a client-planted
|
||||
symlink cannot redirect receiver scratch files outside the sandbox; an
|
||||
in-root symlink to another filesystem is still allowed for rsync's EXDEV
|
||||
fallback. */
|
||||
/* Open an existing --temp-dir scratch directory as-is (absolute or relative;
|
||||
no creation, no root confinement), matching rsync's --temp-dir handling. */
|
||||
int file_open_temp_dir(const char* dir_path);
|
||||
|
||||
/* The file_to_disk_secure* variants write a temporary copy in the destination
|
||||
@@ -167,54 +158,5 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
|
||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||
const char* temp_dir);
|
||||
/* Streaming --copy-dest install: atomically materialize `path` by copying the
|
||||
* bytes of `basis_path` through a bounded buffer (no whole-file buffering, so
|
||||
* an arbitrarily large basis works), applying the SOURCE metadata and the
|
||||
* per-file xattrs / --fake-super record. `update` honors a newer destination;
|
||||
* a --temp-dir scratch location falls back to a direct write on EXDEV. */
|
||||
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
|
||||
unsigned long long expected_size, bool preallocate,
|
||||
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||
const char* temp_dir);
|
||||
/* Receive one length-prefixed whole-file data frame, streaming the payload
|
||||
* through a bounded buffer when it (or its known logical size) exceeds
|
||||
* `stream_limit`. On success exactly one of *out_buffer / *out_spool is set:
|
||||
* - *out_buffer: the historical charged whole-buffer Data (caller destroys);
|
||||
* - *out_spool: an owned temp path holding the payload, installed through the
|
||||
* File's basis_copy field with File.data_spool set so file_destroy unlinks
|
||||
* it. The destination policy/metadata/atomic-store handling is then the
|
||||
* existing bounded-buffer basis install (file_copy_basis_stream_attrs).
|
||||
* `expected_size` (0 = unknown) is the logical size from the check frame;
|
||||
* `dest_path` locates the spool next to the destination; `compress` selects
|
||||
* incremental decompression. Returns false on any framing/I/O/size error. */
|
||||
bool file_receive_payload(int fd, bool compress, unsigned long long expected_size,
|
||||
const char* dest_path, unsigned long long stream_limit, Data** out_buffer,
|
||||
char** out_spool, unsigned long long* out_size);
|
||||
/* Create a confined spool temp file next to `dest_path`; returns an open write
|
||||
* fd and an owned absolute path (to be installed via File.basis_copy with
|
||||
* File.data_spool set, and unlinked by file_destroy). */
|
||||
int file_spool_for_payload(const char* dest_path, char** out_spool_path);
|
||||
/* Protocol 2.28.0 receiver-stat variants: like the two above but additionally
|
||||
* report through `dirs_created` (when non-NULL) how many parent directories the
|
||||
* confined secure walk had to create that lie strictly below `count_floor` (a
|
||||
* receive-root-relative prefix, or NULL for all). Used to reproduce rsync's
|
||||
* `Number of created files` directory count on a fresh destination. */
|
||||
bool file_to_disk_secure_attrs_counted(
|
||||
const char* path, const void* data, unsigned long long data_size, bool inplace, bool sparse,
|
||||
bool preallocate, const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||
bool no_replace, bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||
bool keep_partial, const char* temp_dir, unsigned* dirs_created, const char* count_floor,
|
||||
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor);
|
||||
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
|
||||
const void* data, unsigned long long data_size,
|
||||
bool preallocate, const FileMetadata* metadata,
|
||||
FileAttrPolicy policy, bool use_fsync,
|
||||
const FileXattrList* xattrs, bool fake_super,
|
||||
const char* temp_dir, unsigned* dirs_created,
|
||||
const char* count_floor);
|
||||
/* The logical transfer root expressed receive-root-relative, or NULL when the
|
||||
* wire paths carry no mirror scaffolding above it. Caller frees non-NULL. */
|
||||
char* file_transfer_root_floor(const Config* config);
|
||||
|
||||
#endif
|
||||
|
||||
@@ -29,12 +29,6 @@ typedef struct FileAttrPolicy {
|
||||
bool times; /* config->preserve_times: apply the source mtime */
|
||||
bool atimes; /* config->preserve_atimes (-U): apply the source atime */
|
||||
bool executability; /* config->use_executability (-E): exec-bits-only mode */
|
||||
/* privilege_super_mode_permitted(): when false (SUPER_MODE_OFF / --no-super,
|
||||
or a daemon that did not grant `client owner = yes`), the setuid/setgid/
|
||||
sticky bits are stripped from every applied mode (source mode and any
|
||||
--chmod result) even under --perms. When true, rsync's exact semantics are
|
||||
preserved: -p copies the special bits and the kernel decides. */
|
||||
bool super_permitted;
|
||||
} FileAttrPolicy;
|
||||
|
||||
/* Build the per-attribute policy from a connection's Config. A NULL config
|
||||
|
||||
+3118
-102
File diff suppressed because it is too large
Load Diff
+104
-10
@@ -2,19 +2,10 @@
|
||||
#define FILE_RECEIVE_H
|
||||
|
||||
#include "config.h"
|
||||
#include "delete_commit.h"
|
||||
#include "file_save.h"
|
||||
#include "file_types.h"
|
||||
#include "incremental_check.h"
|
||||
#include "utils.h"
|
||||
#include <stdbool.h>
|
||||
|
||||
/* Server-side file receive/save path.
|
||||
*
|
||||
* This header is the public facade for the file_receive module family: the
|
||||
* wire receive dispatch (this file) plus the save-to-disk (file_save.h), the
|
||||
* incremental check (incremental_check.h) and the delete-commit
|
||||
* (delete_commit.h) modules. */
|
||||
/* Server-side file receive/save path. */
|
||||
|
||||
/* Cumulative caps for the deferred directory-time accumulator. The sender may
|
||||
* legitimately split a large tree across repeated STATUS_DIR_TIMES frames, so a
|
||||
@@ -31,6 +22,15 @@ File* file_receive_dir_time(int file_descriptor, const Config* config);
|
||||
File* file_receive_hardlink(int file_descriptor);
|
||||
File* file_receive_symlink(int file_descriptor, const Config* config);
|
||||
File* file_receive_special(int file_descriptor);
|
||||
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
|
||||
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
|
||||
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
|
||||
* true only on the server-contacting --dry-run path when the file is not up to
|
||||
* date: the receiver has already sent STATUS_DRY_RUN_TRANSFER and returns NULL
|
||||
* without storing anything. On that path `*skipped` is true for an up-to-date
|
||||
* (STATUS_OK) file and both flags are false for a genuine error. */
|
||||
File* receive_incremental_check_ex(int fd, const Config* config, bool* skipped,
|
||||
bool* would_transfer);
|
||||
|
||||
/* P7 Wave D directory-time accumulator. The receiver collects the metadata of
|
||||
* every directory it creates/receives (STATUS_MKDIR with metadata and/or the
|
||||
@@ -74,4 +74,98 @@ bool dir_time_list_add(DirTimeList* list, const char* wire_path, const FileMetad
|
||||
void dir_metadata_list_apply(const DirTimeList* list, const char* root_directory,
|
||||
const Config* config);
|
||||
|
||||
/* A received delete-manifest frame: the keep-set (`keeps`, destination-relative
|
||||
paths the sender transferred/keeps) plus `protected`, destination-relative
|
||||
prefixes the sender asks the receiver never to delete (paths excluded on the
|
||||
source, protected at any depth). When --delete-excluded is given the sender
|
||||
transmits an empty protected list so excluded destination mirrors are treated
|
||||
as ordinary extras. With --delete-missing-args a third section (`missing`)
|
||||
carries the destination mirrors of explicitly-listed source entries that do
|
||||
not exist: each is an exact deletion request, independent of the ordinary
|
||||
extras walk (never blocked by the protected prefixes) and processed when the
|
||||
manifest is committed. */
|
||||
typedef struct DeleteManifest {
|
||||
ArrayList* keeps;
|
||||
ArrayList* protected;
|
||||
ArrayList* missing;
|
||||
/* Destination-relative paths of the directories the sender synchronized for
|
||||
this run. The extras walker only removes entries directly inside one of
|
||||
these (the receive root is the "." sentinel); `--files-from` runs therefore
|
||||
leave untransmitted directories and the unlisted parts of listed ones
|
||||
alone, matching rsync's "delete only in synchronized directories". */
|
||||
ArrayList* dirs;
|
||||
} DeleteManifest;
|
||||
|
||||
void delete_manifest_free(DeleteManifest* manifest);
|
||||
/* Read a delete-manifest frame (protocol 2.23.0): keep count + keeps, then
|
||||
protected count + protected prefixes, then missing count + missing paths,
|
||||
then synchronized-directory count + directory paths (self-delimiting; the
|
||||
leading STATUS_MANIFEST code has been consumed). Returns an owned
|
||||
DeleteManifest, or NULL after signalling STATUS_ERROR on a malformed frame. */
|
||||
DeleteManifest* receive_manifest_entries(int fd);
|
||||
/* Remove destination entries under config->receive_root_directory that are not
|
||||
in `manifest` (bounded, all-or-nothing walk; staging-dir, basis-dir and
|
||||
protected-prefix skips). `--max-delete` and `--force` are honored here. The
|
||||
caller decides WHEN to run it based on the negotiated delete timing. Returns
|
||||
false (and the transfer fails) when the deletion cannot be committed. */
|
||||
bool manifest_delete_extras(const Config* config, DeleteManifest* manifest);
|
||||
/* --delete-missing-args exact-path deletions: remove each destination mirror
|
||||
in `manifest->missing` (never blocked by the protected prefixes, staging dir
|
||||
and basis dirs excluded). A regular file/symlink is unlinked; an empty
|
||||
directory is removed; a NON-empty directory is removed recursively only when
|
||||
--delete or --force is in effect, otherwise it is left with a warning (rsync
|
||||
parity). A missing path is a no-op. Returns false only on a genuine
|
||||
confinement or I/O error (the run then fails); tolerated per-path cases are
|
||||
reported and skipped. */
|
||||
bool manifest_delete_missing_args(const Config* config, DeleteManifest* manifest);
|
||||
/* Budgeted form of manifest_delete_missing_args for the per-directory delete
|
||||
session: each removed mirror draws from `max_delete` (SIZE_MAX = unlimited)
|
||||
and the tallies are accumulated into `*deleted`/`*skipped`. `*limit_hit` is set
|
||||
when the budget stopped the pass with entries left over. Returns false only
|
||||
on a genuine deletion error. */
|
||||
bool manifest_delete_missing_args_limited(const Config* config, DeleteManifest* manifest,
|
||||
size_t max_delete, size_t* deleted, size_t* skipped,
|
||||
bool* limit_hit);
|
||||
/* Outcome of committing a delete manifest. LIMIT_REACHED reports rsync's
|
||||
partial --max-delete result: the budget allowed some deletions and the rest
|
||||
were skipped (the run still stores all file data but the client exits 25). */
|
||||
typedef enum {
|
||||
DELETE_COMMIT_OK = 0,
|
||||
DELETE_COMMIT_LIMIT_REACHED,
|
||||
DELETE_COMMIT_ERROR
|
||||
} DeleteCommitResult;
|
||||
|
||||
/* Run every deletion family the manifest carries: the --delete-missing-args
|
||||
exact-path deletions first (user requests are not blocked by exclusion
|
||||
protection), then the ordinary extras walk when --delete is active. Both
|
||||
share one --max-delete budget. Returns DELETE_COMMIT_OK when nothing was to
|
||||
do or everything committed, DELETE_COMMIT_LIMIT_REACHED when the budget
|
||||
stopped part of the work, or DELETE_COMMIT_ERROR on a genuine failure. */
|
||||
DeleteCommitResult manifest_delete_all(const Config* config, DeleteManifest* manifest);
|
||||
/* Like manifest_delete_all, but reports how many destination entries the commit
|
||||
removed (for the end-of-transfer wire stats). `deleted` may be NULL. */
|
||||
DeleteCommitResult manifest_delete_all_counted(const Config* config, DeleteManifest* manifest,
|
||||
size_t* deleted);
|
||||
|
||||
/* -n/--dry-run --delete would-delete reporting: walk the destination exactly as
|
||||
the delete pass would and append (strdup'd) destination-relative paths that
|
||||
WOULD be removed to `out`, without touching disk. Uses the same staging-dir,
|
||||
basis-dir and protected-prefix skips as the real commit. Returns true on a
|
||||
clean walk; `*count_out` receives the number of paths appended. */
|
||||
bool manifest_would_delete_list(const Config* config, DeleteManifest* manifest, ArrayList* out,
|
||||
size_t* count_out);
|
||||
/* Convert one basis-directory path to the receive-root-relative protection
|
||||
prefix the delete walker uses (NULL when it lies outside the root). Exposed
|
||||
for unit tests of the root-of-"/" and normalization edge cases. */
|
||||
char* file_receive_basis_delete_relative(const Config* config, const char* path);
|
||||
|
||||
/* Outcome of a single file_save_to_disk operation. The receiver needs to
|
||||
distinguish "written" from "skipped" so --remove-source-files can be told
|
||||
which sources were actually stored. */
|
||||
typedef enum { FILE_SAVE_ERROR = 0, FILE_SAVE_WRITTEN = 1, FILE_SAVE_SKIPPED = 2 } FileSaveResult;
|
||||
|
||||
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
|
||||
const Config* config);
|
||||
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
|
||||
|
||||
#endif
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,48 +0,0 @@
|
||||
#ifndef FILE_SAVE_H
|
||||
#define FILE_SAVE_H
|
||||
|
||||
#include "config.h"
|
||||
#include "file_types.h"
|
||||
#include "format.h"
|
||||
#include <stdbool.h>
|
||||
|
||||
/* Save-to-disk module: regular-file/symlink/hardlink/special install, xattr
|
||||
* application, --fake-super and the --delay-updates staging path. These
|
||||
* declarations are re-exported by the file_receive.h facade. */
|
||||
|
||||
/* Outcome of a single file_save_to_disk operation. The receiver needs to
|
||||
distinguish "written" from "skipped" so --remove-source-files can be told
|
||||
which sources were actually stored. FILE_SAVE_FAILED is a per-entry failure
|
||||
(for example a device node that mknodat() refused with EPERM/EACCES): it is
|
||||
logged and counted by the receiver but does NOT abort the transfer, matching
|
||||
rsync's continue-and-exit-partial behavior. */
|
||||
typedef enum {
|
||||
FILE_SAVE_ERROR = 0,
|
||||
FILE_SAVE_WRITTEN = 1,
|
||||
FILE_SAVE_SKIPPED = 2,
|
||||
FILE_SAVE_FAILED = 3
|
||||
} FileSaveResult;
|
||||
|
||||
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
|
||||
|
||||
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
|
||||
const Config* config);
|
||||
/* Protocol 2.28.0 variant: also reports through `created` (when non-NULL)
|
||||
* whether the destination entry did not exist before this save, and through
|
||||
* `created_dirs` how many parent directories the confined walk created, so the
|
||||
* receiver can build rsync's `Number of created files` breakdown. The plain
|
||||
* file_save_to_disk_full() is this with both out-params NULL. */
|
||||
FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File* file,
|
||||
const Config* config, bool* created,
|
||||
unsigned* created_dirs);
|
||||
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
|
||||
|
||||
/* Protocol 2.28.0 receiver counter accumulator: fold one successfully saved
|
||||
* entry into `stats`, adding its receiver-observed literal bytes and, when
|
||||
* `created`, the matching created-by-type counter (regular file / symlink /
|
||||
* special) plus `created_dirs` implicitly-created parent directories.
|
||||
* Non-first hardlink siblings contribute no literal bytes. */
|
||||
void receiver_stats_note_saved(ReceiverStats* stats, const File* file, bool created,
|
||||
unsigned created_dirs);
|
||||
|
||||
#endif
|
||||
+3
-136
@@ -44,138 +44,12 @@ bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
|
||||
send_path, NULL, -1, 0, false);
|
||||
}
|
||||
|
||||
/* Stream-compress a whole source file into a temp file, then transmit it as the
|
||||
* normal length-prefixed data frame. Used when the source was too large to
|
||||
* load (File.data.data == NULL): the source is read in bounded chunks through a
|
||||
* streaming codec, so neither the raw nor the compressed image is held in
|
||||
* memory. The compressed length must be known before the frame is sent (the
|
||||
* wire is length-prefixed), so the stream lands in a private temp file first. */
|
||||
bool file_send_compressed_stream_with_skip(File* file, int file_descriptor, bool use_metadata,
|
||||
int compression_level, bool send_path,
|
||||
char* const* skip_suffixes, int skip_count,
|
||||
int compression_threads, bool send_xattrs) {
|
||||
/* The caller has already decided this file compresses; the skip list is part
|
||||
of the public signature for symmetry with the buffered path. */
|
||||
(void)skip_suffixes;
|
||||
(void)skip_count;
|
||||
if (!file || !file->path || !file->data || file->data->size == 0 || file->data->data != NULL)
|
||||
return false;
|
||||
CompressionAlgo algo = compression_get_algo();
|
||||
CompressionStreamCompressor* compressor =
|
||||
compression_stream_compressor_create(algo, compression_level, compression_threads);
|
||||
if (!compressor) {
|
||||
log_message(LOG_LEVEL_ERROR, "streaming compression is not available for this codec; "
|
||||
"the source was not loaded for the buffered path");
|
||||
return false;
|
||||
}
|
||||
|
||||
int src = file_open_for_read(file->path);
|
||||
if (src < 0) {
|
||||
compression_stream_compressor_destroy(compressor);
|
||||
return false;
|
||||
}
|
||||
struct stat src_st;
|
||||
if (fstat(src, &src_st) != 0 || !S_ISREG(src_st.st_mode) ||
|
||||
(unsigned long long)src_st.st_size < file->data->size) {
|
||||
close(src);
|
||||
compression_stream_compressor_destroy(compressor);
|
||||
return false;
|
||||
}
|
||||
|
||||
const char* tmpdir = getenv("TMPDIR");
|
||||
if (!tmpdir || tmpdir[0] == '\0')
|
||||
tmpdir = "/tmp";
|
||||
size_t tmplen = strlen(tmpdir) + strlen("/fastsync-z-XXXXXX") + 1;
|
||||
char* tmpl = malloc(tmplen);
|
||||
if (!tmpl) {
|
||||
close(src);
|
||||
compression_stream_compressor_destroy(compressor);
|
||||
return false;
|
||||
}
|
||||
snprintf(tmpl, tmplen, "%s/fastsync-z-XXXXXX", tmpdir);
|
||||
int tmp_fd = mkstemp(tmpl);
|
||||
if (tmp_fd < 0) {
|
||||
log_perror("Could not create compression temp file");
|
||||
free(tmpl);
|
||||
close(src);
|
||||
compression_stream_compressor_destroy(compressor);
|
||||
return false;
|
||||
}
|
||||
unlink(tmpl);
|
||||
free(tmpl);
|
||||
|
||||
bool ok = compression_stream_compressor_begin(compressor, file->data->size, tmp_fd);
|
||||
unsigned char buf[64 * 1024];
|
||||
unsigned long long remaining = file->data->size;
|
||||
while (ok && remaining > 0) {
|
||||
size_t want = remaining < sizeof(buf) ? (size_t)remaining : sizeof(buf);
|
||||
ssize_t got = read(src, buf, want);
|
||||
if (got <= 0) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
if (!compression_stream_compressor_feed(compressor, buf, (size_t)got, tmp_fd))
|
||||
ok = false;
|
||||
remaining -= (unsigned long long)got;
|
||||
}
|
||||
if (ok)
|
||||
ok = compression_stream_compressor_finish(compressor, tmp_fd);
|
||||
close(src);
|
||||
compression_stream_compressor_destroy(compressor);
|
||||
if (!ok) {
|
||||
close(tmp_fd);
|
||||
return false;
|
||||
}
|
||||
struct stat tmp_st;
|
||||
if (fstat(tmp_fd, &tmp_st) != 0 || tmp_st.st_size < 0) {
|
||||
close(tmp_fd);
|
||||
return false;
|
||||
}
|
||||
unsigned long long compressed_size = (unsigned long long)tmp_st.st_size;
|
||||
if (lseek(tmp_fd, 0, SEEK_SET) == (off_t)-1) {
|
||||
close(tmp_fd);
|
||||
return false;
|
||||
}
|
||||
|
||||
ok = true;
|
||||
if (send_path && !send_wire_str(file_descriptor, file_wire_path(file)))
|
||||
ok = false;
|
||||
if (ok && use_metadata && !metadata_send(file_descriptor, file->metadata))
|
||||
ok = false;
|
||||
if (ok && send_xattrs && !xattr_send(file_descriptor, file ? file->xattrs : NULL))
|
||||
ok = false;
|
||||
if (ok && !send_n_data(file_descriptor, &compressed_size, sizeof(compressed_size)))
|
||||
ok = false;
|
||||
unsigned long long left = compressed_size;
|
||||
while (ok && left > 0) {
|
||||
size_t want = left < sizeof(buf) ? (size_t)left : sizeof(buf);
|
||||
ssize_t got = read(tmp_fd, buf, want);
|
||||
if (got <= 0 || !send_n_data(file_descriptor, buf, (size_t)got)) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
left -= (unsigned long long)got;
|
||||
}
|
||||
close(tmp_fd);
|
||||
return ok;
|
||||
}
|
||||
|
||||
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
|
||||
int compression_level, bool send_path,
|
||||
char* const* skip_suffixes, int skip_count,
|
||||
int compression_threads, bool send_xattrs) {
|
||||
if (!file || !file->path || !file->data)
|
||||
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data))
|
||||
return false;
|
||||
/* A source too large to load is streamed: compression streams through a
|
||||
* temp file, no compression must have taken the sendfile path instead. */
|
||||
if (file->data->size != 0 && file->data->data == NULL) {
|
||||
if (compression_level <= 0 ||
|
||||
compression_should_skip_with_suffixes(file->path, skip_suffixes, skip_count))
|
||||
return false;
|
||||
return file_send_compressed_stream_with_skip(file, file_descriptor, use_metadata,
|
||||
compression_level, send_path, skip_suffixes,
|
||||
skip_count, compression_threads, send_xattrs);
|
||||
}
|
||||
const Data* data_to_send = file->data;
|
||||
Data* compressed_data = NULL;
|
||||
if (compression_level > 0 &&
|
||||
@@ -250,12 +124,8 @@ bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_meta
|
||||
}
|
||||
|
||||
/* sendfile cannot encrypt TLS records. Keep the framing identical but
|
||||
route encrypted transfers through the deadline-aware IO layer. Resolve
|
||||
the transport from the bound session, not the thread-local io_ssl: a
|
||||
worker thread running a TLS transfer has its SSL only on the session it
|
||||
bound, so io_get_ssl() would be NULL there and the raw sendfile() path
|
||||
would be taken on an encrypted socket. */
|
||||
if (protocol_current_ssl() != NULL) {
|
||||
route encrypted transfers through the deadline-aware IO layer. */
|
||||
if (io_get_ssl() != NULL) {
|
||||
unsigned char buffer[64 * 1024];
|
||||
unsigned long long remaining = file_size;
|
||||
bool ok = true;
|
||||
@@ -296,8 +166,6 @@ bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_meta
|
||||
}
|
||||
struct pollfd pfd = {.fd = file_descriptor, .events = POLLOUT};
|
||||
int polled = poll(&pfd, 1, timeout);
|
||||
if (polled < 0 && errno == EINTR)
|
||||
continue;
|
||||
if (polled <= 0 || (pfd.revents & (POLLERR | POLLHUP | POLLNVAL))) {
|
||||
close(fd);
|
||||
return false;
|
||||
@@ -315,7 +183,6 @@ bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_meta
|
||||
return false;
|
||||
}
|
||||
protocol_note_bytes_written((unsigned long long)sent);
|
||||
protocol_throttle_bytes(file_descriptor, (size_t)sent);
|
||||
}
|
||||
|
||||
close(fd);
|
||||
|
||||
@@ -13,12 +13,6 @@ bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_
|
||||
int compression_level, bool send_path,
|
||||
char* const* skip_suffixes, int skip_count,
|
||||
int compression_threads, bool send_xattrs);
|
||||
/* Stream-compress an unloaded whole source file into a temp file and send it as
|
||||
* the usual data frame (see file_send.c). */
|
||||
bool file_send_compressed_stream_with_skip(File* file, int file_descriptor, bool use_metadata,
|
||||
int compression_level, bool send_path,
|
||||
char* const* skip_suffixes, int skip_count,
|
||||
int compression_threads, bool send_xattrs);
|
||||
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
|
||||
bool send_path);
|
||||
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
|
||||
|
||||
@@ -57,16 +57,6 @@ typedef struct {
|
||||
* equals the incoming file, and `data` is kept as the cross-filesystem
|
||||
* fallback (a local copy) if the hard link cannot be created. */
|
||||
char* basis_link;
|
||||
/* Receiver-only, --copy-dest: when set (and basis_link is NULL), stream the
|
||||
* basis file's bytes into the destination instead of `data`/`data->size`.
|
||||
* This lets a basis larger than any whole-file bound materialize without
|
||||
* buffering it; the source metadata on `metadata` is applied afterwards. */
|
||||
char* basis_copy;
|
||||
/* Receiver-only. When set, `basis_copy` points at a receiver-created spool
|
||||
* temp file holding a STREAMED whole-file payload (rather than a --copy-dest
|
||||
* basis). file_destroy unlinks it after the install consumes it, so an
|
||||
* over-limit file leaves no scratch behind. */
|
||||
bool data_spool;
|
||||
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
|
||||
* id shared by every member of one source inode (0 = not part of a group).
|
||||
* The FIRST member (link_first == true) carries its data on the wire and is
|
||||
@@ -107,11 +97,6 @@ typedef struct {
|
||||
* basis file (matched delta blocks) for this entry. 0 when the file was sent
|
||||
* whole. Accumulated into ReceiverStats.matched_data by the receiver sink. */
|
||||
unsigned long long matched_bytes;
|
||||
/* Receiver-only (protocol 2.28.0) wire-stats tally: the literal delta fragment
|
||||
* bytes this entry carried (DELTA_INSTR_LITERAL). 0 when the file was sent
|
||||
* whole; the sink then falls back to the whole payload size. Accumulated
|
||||
* into ReceiverStats.literal_bytes. */
|
||||
unsigned long long literal_bytes;
|
||||
} File;
|
||||
|
||||
/* The path that should be sent on the wire and used for the receiver-side
|
||||
|
||||
+113
-424
@@ -4,12 +4,22 @@
|
||||
#include <ctype.h>
|
||||
#include <errno.h>
|
||||
#include <limits.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
/* Write a diagnostic message into the caller's optional buffer. */
|
||||
#define filter_set_error utils_set_error
|
||||
/* Write a diagnostic message into the caller's optional buffer. A NULL `err`
|
||||
* (or a zero size) is a no-op, so a caller that only needs the boolean status
|
||||
* may pass NULL without the snprintf-on-NULL undefined behaviour. */
|
||||
static void filter_set_error(char* err, size_t err_size, const char* fmt, ...) {
|
||||
if (!err || err_size == 0)
|
||||
return;
|
||||
va_list ap;
|
||||
va_start(ap, fmt);
|
||||
vsnprintf(err, err_size, fmt, ap);
|
||||
va_end(ap);
|
||||
}
|
||||
|
||||
/* ---- Ordered rule lists ---- */
|
||||
|
||||
@@ -21,28 +31,6 @@ void filter_rule_free(FilterRule* rule) {
|
||||
free(rule);
|
||||
}
|
||||
|
||||
FilterRule* filter_rule_clone(const FilterRule* rule) {
|
||||
if (!rule)
|
||||
return NULL;
|
||||
FilterRule* copy = calloc(1, sizeof(FilterRule));
|
||||
if (!copy)
|
||||
return NULL;
|
||||
copy->action = rule->action;
|
||||
copy->sides = rule->sides;
|
||||
copy->anchored = rule->anchored;
|
||||
copy->dir_only = rule->dir_only;
|
||||
copy->negate = rule->negate;
|
||||
copy->perishable = rule->perishable;
|
||||
copy->no_inherit = rule->no_inherit;
|
||||
copy->owner = str_dup(rule->owner ? rule->owner : "");
|
||||
copy->pattern = str_dup(rule->pattern ? rule->pattern : "");
|
||||
if (!copy->owner || !copy->pattern) {
|
||||
filter_rule_free(copy);
|
||||
return NULL;
|
||||
}
|
||||
return copy;
|
||||
}
|
||||
|
||||
FilterRuleList* filter_rule_list_create(void) {
|
||||
return calloc(1, sizeof(FilterRuleList));
|
||||
}
|
||||
@@ -70,94 +58,37 @@ void filter_rule_list_free(FilterRuleList* list) {
|
||||
for (int i = 0; i < list->count; i++)
|
||||
filter_rule_free(list->items[i]);
|
||||
for (int i = 0; i < list->dir_merge_count; i++)
|
||||
free(list->dir_merges[i].name);
|
||||
free(list->dir_merges);
|
||||
free(list->dir_merge_names[i]);
|
||||
free(list->dir_merge_names);
|
||||
free(list->items);
|
||||
free(list);
|
||||
}
|
||||
|
||||
/* Append an implicit exclude rule for the merge file itself (rsync's 'e'
|
||||
* modifier). The rule is owned by the transfer root and matches the basename
|
||||
* anywhere, exactly like rsync's EXCLUDE_SELF (a dual-sided exclude: it hides
|
||||
* the file and protects its destination mirror from --delete). */
|
||||
static bool filter_list_add_exclude_self(FilterRuleList* list, const char* name) {
|
||||
const char* base = strrchr(name, '/');
|
||||
base = base ? base + 1 : name;
|
||||
if (base[0] == '\0')
|
||||
return true;
|
||||
FilterRule* rule = calloc(1, sizeof(FilterRule));
|
||||
if (!rule)
|
||||
return false;
|
||||
rule->action = FILTER_ACTION_EXCLUDE;
|
||||
rule->sides = FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER;
|
||||
rule->pattern = str_dup(base);
|
||||
if (!rule->pattern || !filter_rule_set_owner(rule, "")) {
|
||||
filter_rule_free(rule);
|
||||
return false;
|
||||
}
|
||||
if (!filter_rule_list_add(list, rule)) {
|
||||
filter_rule_free(rule);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Register a per-directory merge-file basename (for "dir-merge NAME"/": NAME"
|
||||
* and -F's .rsync-filter). Duplicate names are ignored. */
|
||||
bool filter_rule_list_add_dir_merge(FilterRuleList* list, const char* name) {
|
||||
return filter_rule_list_add_dir_merge_ex(list, name, false, false, false, false, false);
|
||||
}
|
||||
|
||||
bool filter_rule_list_add_dir_merge_ex(FilterRuleList* list, const char* name, bool no_prefixes,
|
||||
bool include, bool word_split, bool no_inherit,
|
||||
bool exclude_self) {
|
||||
if (!list || !name || name[0] == '\0')
|
||||
return false;
|
||||
for (int i = 0; i < list->dir_merge_count; i++) {
|
||||
if (strcmp(list->dir_merges[i].name, name) == 0) {
|
||||
/* rsync keeps the first registration (first-wins), but the 'e' modifier
|
||||
is a list side effect, not a registration field: honor it on the
|
||||
duplicate path too, adding the implicit exclude-self rule at most
|
||||
once. */
|
||||
if (exclude_self && !list->dir_merges[i].exclude_self) {
|
||||
if (!filter_list_add_exclude_self(list, name))
|
||||
return false;
|
||||
list->dir_merges[i].exclude_self = true;
|
||||
}
|
||||
if (strcmp(list->dir_merge_names[i], name) == 0)
|
||||
return true;
|
||||
}
|
||||
}
|
||||
if (list->dir_merge_count == list->dir_merge_capacity) {
|
||||
if (list->dir_merge_capacity > INT_MAX / 2)
|
||||
return false;
|
||||
int new_cap = list->dir_merge_capacity > 0 ? list->dir_merge_capacity * 2 : 4;
|
||||
FilterDirMerge* grown = realloc(list->dir_merges, (size_t)new_cap * sizeof(*grown));
|
||||
char** grown = realloc(list->dir_merge_names, (size_t)new_cap * sizeof(char*));
|
||||
if (!grown)
|
||||
return false;
|
||||
list->dir_merges = grown;
|
||||
list->dir_merge_names = grown;
|
||||
list->dir_merge_capacity = new_cap;
|
||||
}
|
||||
char* dup = str_dup(name);
|
||||
if (!dup)
|
||||
return false;
|
||||
if (exclude_self && !filter_list_add_exclude_self(list, name)) {
|
||||
free(dup);
|
||||
return false;
|
||||
}
|
||||
FilterDirMerge* entry = &list->dir_merges[list->dir_merge_count];
|
||||
entry->name = dup;
|
||||
entry->no_prefixes = no_prefixes;
|
||||
entry->include = include;
|
||||
entry->word_split = word_split;
|
||||
entry->no_inherit = no_inherit;
|
||||
entry->exclude_self = exclude_self;
|
||||
list->dir_merge_count++;
|
||||
list->dir_merge_names[list->dir_merge_count++] = dup;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool filter_rule_set_owner(FilterRule* rule, const char* owner) {
|
||||
if (!rule)
|
||||
return false;
|
||||
static bool set_rule_owner(FilterRule* rule, const char* owner) {
|
||||
char* dup = str_dup(owner ? owner : "");
|
||||
if (!dup)
|
||||
return false;
|
||||
@@ -241,77 +172,14 @@ static bool is_modifier_char(char c) {
|
||||
return c == 's' || c == 'r' || c == 'p' || c == 'x' || c == '/' || c == '!' || c == 'C';
|
||||
}
|
||||
|
||||
/* merge/dir-merge rules are the only rules rsync accepts the merge-file
|
||||
* modifiers on. */
|
||||
static bool is_merge_rule(RuleKind kind) {
|
||||
return kind == RULE_KIND_MERGE || kind == RULE_KIND_DIR_MERGE;
|
||||
}
|
||||
|
||||
/* Merge-file modifiers rsync defines but FastSync does not implement:
|
||||
* 'e' exclude the merge file itself, 'n' do not inherit the merge file, 'w'
|
||||
* word-split the merge file. They are recognized as part of a modifier run on
|
||||
* every rule (so a pure e/n/w token is rejected rather than folded into the
|
||||
* pattern), but are accepted (and ignored) only on merge/dir-merge rules. */
|
||||
static bool is_unsupported_modifier_char(char c) {
|
||||
return c == 'e' || c == 'n' || c == 'w';
|
||||
}
|
||||
|
||||
/* Merge-file modifiers rsync accepts on merge/dir-merge rules: 'e' (exclude
|
||||
* self), 'n' (no inherit), 'w' (word split), '-' (bare excludes) and '+'
|
||||
* (bare includes). */
|
||||
static bool is_merge_modifier_char(char c) {
|
||||
return c == 'e' || c == 'n' || c == 'w' || c == '-' || c == '+';
|
||||
}
|
||||
|
||||
/* Characters that count as part of a modifier run for `kind` when deciding
|
||||
* whether a token is a pure modifier run. e/n/w count on every rule so that a
|
||||
* pure e/n/w token is rejected on non-merge rules; '-' only on merge rules. */
|
||||
static bool is_modifier_scan_char(char c, RuleKind kind) {
|
||||
return is_modifier_char(c) || is_unsupported_modifier_char(c) ||
|
||||
(is_merge_rule(kind) && is_merge_modifier_char(c));
|
||||
}
|
||||
|
||||
/* Characters actually consumed as modifiers for `kind`. The merge-file
|
||||
* modifiers are consumed only on merge/dir-merge rules; elsewhere e/n/w fall
|
||||
* through to the pattern (so mixed tokens such as "H,!secret" keep their
|
||||
* historical "ecret" pattern). */
|
||||
static bool is_consumed_modifier_char(char c, RuleKind kind) {
|
||||
return is_modifier_char(c) || (is_merge_rule(kind) && is_merge_modifier_char(c));
|
||||
}
|
||||
|
||||
/* Inspect the token that follows a rule name (up to the first space/underscore
|
||||
* or the end). If the token is composed *solely* of modifier characters and
|
||||
* includes one that is invalid for `kind`, it is unambiguously a modifier run:
|
||||
* return that character so the caller can reject it. A token that contains any
|
||||
* non-modifier character is a pattern (e.g. "-newfile") and returns '\0', which
|
||||
* keeps the historical parsing of mixed tokens such as "H,!secret" intact. */
|
||||
static char unsupported_modifier_in_token(const char* tok, RuleKind kind) {
|
||||
if (*tok == '\0' || *tok == ' ' || *tok == '_')
|
||||
return '\0';
|
||||
char bad = '\0';
|
||||
for (const char* q = tok; *q != '\0' && *q != ' ' && *q != '_'; q++) {
|
||||
if (!is_modifier_scan_char(*q, kind))
|
||||
return '\0';
|
||||
if (!is_merge_rule(kind) && is_unsupported_modifier_char(*q))
|
||||
bad = *q;
|
||||
}
|
||||
return bad;
|
||||
}
|
||||
|
||||
/* Parse "RULE[,MODIFIERS] [PATTERN]". On success `kind`, `sides`,
|
||||
* `sides_explicit`, `negate`, `anchored_mod`, `perishable`, `xattr`,
|
||||
* `cvs_inject` and the pattern span (`pat_start`/`pat_len`, possibly 0 for
|
||||
* merge/clear) are filled. Returns true on success.
|
||||
*
|
||||
* On failure `*bad_mod` is set to the offending modifier character when the
|
||||
* rule carried a modifier FastSync does not implement, and left '\0' for a
|
||||
* generic syntax error so callers can emit a precise diagnostic. */
|
||||
* merge/clear) are filled. Returns true on success. */
|
||||
static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
|
||||
bool* sides_explicit, bool* negate, bool* anchored_mod,
|
||||
bool* perishable, bool* xattr, bool* cvs_inject, bool* no_prefixes,
|
||||
bool* include_defaults, bool* word_split, bool* no_inherit,
|
||||
bool* exclude_self, const char** pat_start, size_t* pat_len,
|
||||
char* bad_mod) {
|
||||
bool* perishable, bool* xattr, bool* cvs_inject,
|
||||
const char** pat_start, size_t* pat_len) {
|
||||
const char* p = text;
|
||||
*sides = FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER;
|
||||
*sides_explicit = false;
|
||||
@@ -320,14 +188,8 @@ static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
|
||||
*perishable = false;
|
||||
*xattr = false;
|
||||
*cvs_inject = false;
|
||||
*no_prefixes = false;
|
||||
*include_defaults = false;
|
||||
*word_split = false;
|
||||
*no_inherit = false;
|
||||
*exclude_self = false;
|
||||
*pat_start = NULL;
|
||||
*pat_len = 0;
|
||||
*bad_mod = '\0';
|
||||
|
||||
bool is_short = false;
|
||||
if (short_rule_char(*p, kind)) {
|
||||
@@ -348,25 +210,17 @@ static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
|
||||
/* Modifiers: long names require a comma; short names may attach directly.
|
||||
Only commit a modifier run that terminates at a separator or the end, so a
|
||||
pattern such as "*.tmp" written as "-*.tmp" is not mistaken for modifiers. */
|
||||
if (*p == ',') {
|
||||
*bad_mod = unsupported_modifier_in_token(p + 1, *kind);
|
||||
} else if (is_short) {
|
||||
*bad_mod = unsupported_modifier_in_token(p, *kind);
|
||||
}
|
||||
if (*bad_mod != '\0')
|
||||
return false;
|
||||
|
||||
const char* mod_start = p;
|
||||
const char* mod_end = p;
|
||||
if (*p == ',') {
|
||||
p++;
|
||||
mod_start = p;
|
||||
while (is_consumed_modifier_char(*p, *kind))
|
||||
while (is_modifier_char(*p))
|
||||
p++;
|
||||
mod_end = p;
|
||||
} else if (is_short) {
|
||||
const char* scan = p;
|
||||
while (is_consumed_modifier_char(*scan, *kind))
|
||||
while (is_modifier_char(*scan))
|
||||
scan++;
|
||||
if (*scan == '\0' || *scan == ' ' || *scan == '_') {
|
||||
mod_start = p;
|
||||
@@ -399,21 +253,6 @@ static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
|
||||
case 'C':
|
||||
*cvs_inject = true;
|
||||
break;
|
||||
case '-':
|
||||
*no_prefixes = true;
|
||||
break;
|
||||
case '+':
|
||||
*include_defaults = true;
|
||||
break;
|
||||
case 'e':
|
||||
*exclude_self = true;
|
||||
break;
|
||||
case 'n':
|
||||
*no_inherit = true;
|
||||
break;
|
||||
case 'w':
|
||||
*word_split = true;
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
@@ -449,17 +288,11 @@ FilterRule* filter_rule_parse(const char* line, const FilterParseOptions* opts,
|
||||
RuleKind kind = RULE_KIND_UNKNOWN;
|
||||
unsigned sides;
|
||||
bool sides_explicit, negate, anchored_mod, perishable, xattr, cvs_inject;
|
||||
bool no_prefixes, include_defaults, word_split, no_inherit, exclude_self;
|
||||
const char* pat;
|
||||
size_t pat_len;
|
||||
char bad_mod;
|
||||
if (!parse_rule_syntax(p, &kind, &sides, &sides_explicit, &negate, &anchored_mod, &perishable,
|
||||
&xattr, &cvs_inject, &no_prefixes, &include_defaults, &word_split,
|
||||
&no_inherit, &exclude_self, &pat, &pat_len, &bad_mod)) {
|
||||
if (bad_mod != '\0')
|
||||
filter_set_error(err, err_size, "unsupported filter modifier '%c'", bad_mod);
|
||||
else
|
||||
filter_set_error(err, err_size, "unrecognized filter rule syntax");
|
||||
&xattr, &cvs_inject, &pat, &pat_len)) {
|
||||
filter_set_error(err, err_size, "unrecognized filter rule syntax");
|
||||
return NULL;
|
||||
}
|
||||
if (cvs_inject) {
|
||||
@@ -568,6 +401,7 @@ FilterRule* filter_rule_parse(const char* line, const FilterParseOptions* opts,
|
||||
rule->dir_only = dir_only;
|
||||
rule->negate = negate;
|
||||
rule->perishable = perishable;
|
||||
(void)xattr; /* xattr-name rules never match file/dir names; accepted/ignored */
|
||||
return rule;
|
||||
}
|
||||
|
||||
@@ -608,7 +442,7 @@ static bool filter_list_append_cvs(FilterRuleList* list, unsigned sides) {
|
||||
}
|
||||
memcpy(rule->pattern, CVS_DEFAULTS[i].pattern, plen);
|
||||
rule->pattern[plen] = '\0';
|
||||
if (!filter_rule_set_owner(rule, "")) {
|
||||
if (!set_rule_owner(rule, "")) {
|
||||
filter_rule_free(rule);
|
||||
return false;
|
||||
}
|
||||
@@ -626,138 +460,16 @@ static bool filter_list_parse_append_depth(FilterRuleList* list, const char* lin
|
||||
const FilterParseOptions* opts, const char* base_dir,
|
||||
int depth, char* err, size_t err_size);
|
||||
|
||||
/* Append one merge-file token/line to `list`, honoring the merge rule's
|
||||
* no-prefix/include mode. In no-prefix mode the token is a bare pattern whose
|
||||
* include/exclude default comes from the merge rule (rsync's "-"/"+" merge
|
||||
* modifiers); otherwise the token is parsed as a full filter rule. */
|
||||
static bool filter_merge_append_token(FilterRuleList* list, const char* token,
|
||||
const FilterDirMerge* spec, const FilterParseOptions* opts,
|
||||
const char* base_dir, int depth, char* err, size_t err_size) {
|
||||
if (spec->no_prefixes || spec->include) {
|
||||
size_t tlen = strlen(token);
|
||||
char* text = malloc(tlen + 3);
|
||||
if (!text) {
|
||||
filter_set_error(err, err_size, "memory allocation failed");
|
||||
return false;
|
||||
}
|
||||
text[0] = spec->include ? '+' : '-';
|
||||
text[1] = ' ';
|
||||
memcpy(text + 2, token, tlen + 1);
|
||||
bool ok = filter_list_parse_append_depth(list, text, opts, base_dir, depth + 1, err, err_size);
|
||||
free(text);
|
||||
return ok;
|
||||
}
|
||||
return filter_list_parse_append_depth(list, token, opts, base_dir, depth + 1, err, err_size);
|
||||
}
|
||||
|
||||
/* Read a merge file's tokens/lines into `list` for `spec`. A `w` merge rule
|
||||
* word-splits on whitespace (turning comments off); otherwise lines are parsed
|
||||
* and whole-line `#` comments skipped. When `owner_rel` is non-NULL the newly
|
||||
* added rules are owned by that directory; a no-inherit spec marks them so they
|
||||
* apply only there. Returns false on parse/allocation failure. */
|
||||
static bool filter_merge_read(FilterRuleList* list, FILE* fp, const char* display_path,
|
||||
const FilterDirMerge* spec, const FilterParseOptions* opts,
|
||||
const char* base_dir, const char* owner_rel, int depth, char* err,
|
||||
size_t err_size) {
|
||||
/* Lowest list index this read is responsible for. A "clear"/"!" inside the
|
||||
* file resets list->count to 0 (freeing the caller's earlier rules too), so
|
||||
* the base must follow it down: otherwise post-clear rules sit below the
|
||||
* original count and never receive an owner (nor no-inherit) and are missed
|
||||
* by the rollback. */
|
||||
int floor = list->count;
|
||||
char* line = NULL;
|
||||
size_t cap = 0;
|
||||
bool ok = true;
|
||||
while (ok) {
|
||||
ssize_t n = utils_getdelim_bounded(fp, &line, &cap, '\n', UTILS_MAX_LINE_LEN);
|
||||
if (n < 0) {
|
||||
if (errno == EFBIG)
|
||||
filter_set_error(err, err_size, "line in %s exceeds %d bytes", display_path,
|
||||
(int)UTILS_MAX_LINE_LEN);
|
||||
else
|
||||
filter_set_error(err, err_size, "error reading %s: %s", display_path, strerror(errno));
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
if (n == 0)
|
||||
break;
|
||||
if (spec->word_split) {
|
||||
/* Whitespace-separated tokens; newlines are ordinary separators and
|
||||
* comments are disabled. */
|
||||
const char* s = line;
|
||||
while (*s) {
|
||||
while (*s == ' ' || *s == '\t' || *s == '\n' || *s == '\r')
|
||||
s++;
|
||||
if (*s == '\0')
|
||||
break;
|
||||
const char* start = s;
|
||||
while (*s != '\0' && *s != ' ' && *s != '\t' && *s != '\n' && *s != '\r')
|
||||
s++;
|
||||
size_t tlen = (size_t)(s - start);
|
||||
char* token = malloc(tlen + 1);
|
||||
if (!token) {
|
||||
filter_set_error(err, err_size, "memory allocation failed");
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
memcpy(token, start, tlen);
|
||||
token[tlen] = '\0';
|
||||
if (!filter_merge_append_token(list, token, spec, opts, base_dir, depth, err, err_size))
|
||||
ok = false;
|
||||
if (list->count < floor)
|
||||
floor = list->count; /* a "clear" reset the list below this read's base */
|
||||
free(token);
|
||||
}
|
||||
} else {
|
||||
const char* lp = line;
|
||||
while (*lp == ' ' || *lp == '\t')
|
||||
lp++;
|
||||
if (*lp == '\0' || *lp == '\n' || *lp == '\r' || *lp == '#')
|
||||
continue;
|
||||
if (!filter_merge_append_token(list, lp, spec, opts, base_dir, depth, err, err_size))
|
||||
ok = false;
|
||||
if (list->count < floor)
|
||||
floor = list->count; /* a "clear" reset the list below this read's base */
|
||||
}
|
||||
}
|
||||
free(line);
|
||||
if (!ok) {
|
||||
/* Drop every live rule this read is responsible for. After a "clear" that
|
||||
* base is 0, so the post-clear rules are freed too instead of leaking. */
|
||||
for (int i = floor; i < list->count; i++)
|
||||
filter_rule_free(list->items[i]);
|
||||
list->count = floor;
|
||||
return false;
|
||||
}
|
||||
for (int i = floor; i < list->count; i++) {
|
||||
FilterRule* rule = list->items[i];
|
||||
if (spec->no_inherit)
|
||||
rule->no_inherit = true;
|
||||
if (owner_rel && !filter_rule_set_owner(rule, owner_rel)) {
|
||||
filter_set_error(err, err_size, "memory allocation failed");
|
||||
for (int j = floor; j < list->count; j++)
|
||||
filter_rule_free(list->items[j]);
|
||||
list->count = floor;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Read a single-instance merge file and splice its rules into `list`. A
|
||||
* relative path is resolved below `base_dir` when given, else used as-is (rsync
|
||||
* resolves a command-line merge file relative to the current directory). */
|
||||
/* Read a merge file and splice its rules into `list`. A relative path is
|
||||
* resolved below `base_dir` when given, else used as-is (rsync resolves a
|
||||
* command-line merge file relative to the current directory). */
|
||||
static bool filter_list_merge_file(FilterRuleList* list, const char* name,
|
||||
const FilterDirMerge* spec, const FilterParseOptions* opts,
|
||||
const char* base_dir, int depth, char* err, size_t err_size) {
|
||||
const FilterParseOptions* opts, const char* base_dir, int depth,
|
||||
char* err, size_t err_size) {
|
||||
if (name[0] == '\0') {
|
||||
filter_set_error(err, err_size, "merge requires a filename");
|
||||
return false;
|
||||
}
|
||||
if (spec->exclude_self && !filter_list_add_exclude_self(list, name)) {
|
||||
filter_set_error(err, err_size, "memory allocation failed");
|
||||
return false;
|
||||
}
|
||||
char* path =
|
||||
(base_dir && base_dir[0] && name[0] != '/') ? path_cat(base_dir, name) : str_dup(name);
|
||||
if (!path) {
|
||||
@@ -770,7 +482,29 @@ static bool filter_list_merge_file(FilterRuleList* list, const char* name,
|
||||
free(path);
|
||||
return false;
|
||||
}
|
||||
bool ok = filter_merge_read(list, fp, path, spec, opts, base_dir, NULL, depth, err, err_size);
|
||||
char* line = NULL;
|
||||
size_t cap = 0;
|
||||
bool ok = true;
|
||||
while (true) {
|
||||
ssize_t n = utils_getdelim_bounded(fp, &line, &cap, '\n', UTILS_MAX_LINE_LEN);
|
||||
if (n < 0) {
|
||||
filter_set_error(err, err_size, "error reading merge file '%s'", path);
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
if (n == 0)
|
||||
break;
|
||||
const char* lp = line;
|
||||
while (*lp == ' ' || *lp == '\t')
|
||||
lp++;
|
||||
if (*lp == '\0' || *lp == '\n' || *lp == '\r' || *lp == '#')
|
||||
continue;
|
||||
if (!filter_list_parse_append_depth(list, lp, opts, base_dir, depth + 1, err, err_size)) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
free(line);
|
||||
fclose(fp);
|
||||
free(path);
|
||||
return ok;
|
||||
@@ -794,31 +528,18 @@ static bool filter_list_parse_append_depth(FilterRuleList* list, const char* lin
|
||||
RuleKind kind = RULE_KIND_UNKNOWN;
|
||||
unsigned sides;
|
||||
bool sides_explicit, negate, anchored_mod, perishable, xattr, cvs_inject;
|
||||
bool no_prefixes, include_defaults, word_split, no_inherit, exclude_self;
|
||||
const char* pat;
|
||||
size_t pat_len;
|
||||
char bad_mod;
|
||||
if (!parse_rule_syntax(p, &kind, &sides, &sides_explicit, &negate, &anchored_mod, &perishable,
|
||||
&xattr, &cvs_inject, &no_prefixes, &include_defaults, &word_split,
|
||||
&no_inherit, &exclude_self, &pat, &pat_len, &bad_mod)) {
|
||||
if (bad_mod != '\0')
|
||||
filter_set_error(err, err_size, "unsupported filter modifier '%c': %s", bad_mod, p);
|
||||
else
|
||||
filter_set_error(err, err_size, "unrecognized filter rule syntax: %s", p);
|
||||
&xattr, &cvs_inject, &pat, &pat_len)) {
|
||||
filter_set_error(err, err_size, "unrecognized filter rule syntax: %s", p);
|
||||
return false;
|
||||
}
|
||||
(void)sides_explicit;
|
||||
(void)negate;
|
||||
(void)anchored_mod;
|
||||
(void)perishable;
|
||||
|
||||
/* xattr-name rules are not implemented; reject them everywhere (including on
|
||||
* merge/dir-merge, where the flag would otherwise be silently dropped) with
|
||||
* the same diagnostic the standalone parser gives. */
|
||||
if (xattr) {
|
||||
filter_set_error(err, err_size, "xattr-name filter rules (the x modifier) are not supported");
|
||||
return false;
|
||||
}
|
||||
(void)xattr;
|
||||
|
||||
if (cvs_inject) {
|
||||
/* "C" injects the CVS defaults in place; no pattern is expected. */
|
||||
@@ -846,15 +567,7 @@ static bool filter_list_parse_append_depth(FilterRuleList* list, const char* lin
|
||||
}
|
||||
memcpy(name, pat, pat_len);
|
||||
name[pat_len] = '\0';
|
||||
/* A single-instance merge has no inheritance, so 'n' is meaningless; the
|
||||
* other merge modifiers still shape how the file is read. */
|
||||
FilterDirMerge spec = {.name = name,
|
||||
.no_prefixes = no_prefixes,
|
||||
.include = include_defaults,
|
||||
.word_split = word_split,
|
||||
.no_inherit = false,
|
||||
.exclude_self = exclude_self};
|
||||
bool ok = filter_list_merge_file(list, name, &spec, opts, base_dir, depth, err, err_size);
|
||||
bool ok = filter_list_merge_file(list, name, opts, base_dir, depth, err, err_size);
|
||||
free(name);
|
||||
return ok;
|
||||
}
|
||||
@@ -870,8 +583,7 @@ static bool filter_list_parse_append_depth(FilterRuleList* list, const char* lin
|
||||
}
|
||||
memcpy(name, pat, pat_len);
|
||||
name[pat_len] = '\0';
|
||||
bool ok = filter_rule_list_add_dir_merge_ex(list, name, no_prefixes, include_defaults,
|
||||
word_split, no_inherit, exclude_self);
|
||||
bool ok = filter_rule_list_add_dir_merge(list, name);
|
||||
free(name);
|
||||
if (!ok) {
|
||||
filter_set_error(err, err_size, "memory allocation failed");
|
||||
@@ -932,30 +644,27 @@ FilterRuleList* filter_base_build(const char* const* rule_texts, int rule_count,
|
||||
/* Undo the rules and dir-merge registrations that one merge file appended,
|
||||
* leaving the caller's earlier content intact. A "clear" rule inside the file
|
||||
* frees every rule, including the caller's; clamp to the surviving count so
|
||||
* those already-freed rules are never resurrected and freed a second time.
|
||||
* (filter_merge_read() has already rolled its own range back by the time this
|
||||
* runs, so on a post-clear failure `list->count` is below `rules_before` and
|
||||
* this is a no-op for the rules.) */
|
||||
* those already-freed rules are never resurrected and freed a second time. */
|
||||
static void filter_file_rollback(FilterRuleList* list, int rules_before, int dir_merges_before) {
|
||||
int first = rules_before < list->count ? rules_before : list->count;
|
||||
for (int i = first; i < list->count; i++)
|
||||
filter_rule_free(list->items[i]);
|
||||
list->count = first;
|
||||
for (int i = dir_merges_before; i < list->dir_merge_count; i++)
|
||||
free(list->dir_merges[i].name);
|
||||
free(list->dir_merge_names[i]);
|
||||
list->dir_merge_count = dir_merges_before;
|
||||
}
|
||||
|
||||
bool filter_dir_merge_append(FilterRuleList* list, const char* dir_path, const FilterDirMerge* spec,
|
||||
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||
char* err, size_t err_size) {
|
||||
bool filter_file_append(FilterRuleList* list, const char* dir_path, const char* name,
|
||||
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||
char* err, size_t err_size) {
|
||||
if (err && err_size > 0)
|
||||
err[0] = '\0';
|
||||
if (exists)
|
||||
*exists = false;
|
||||
if (!list || !spec || !spec->name)
|
||||
if (!list)
|
||||
return false;
|
||||
char* filter_path = path_cat(dir_path, spec->name);
|
||||
char* filter_path = path_cat(dir_path, name);
|
||||
if (!filter_path) {
|
||||
filter_set_error(err, err_size, "memory allocation failed");
|
||||
return false;
|
||||
@@ -966,7 +675,7 @@ bool filter_dir_merge_append(FilterRuleList* list, const char* dir_path, const F
|
||||
if (errno == ENOENT || errno == ENOTDIR)
|
||||
return true;
|
||||
char* escaped_dir = output_escape(dir_path, log_get_8_bit_output());
|
||||
log_message(LOG_LEVEL_WARNING, "Could not read %s in %s: %s", spec->name,
|
||||
log_message(LOG_LEVEL_WARNING, "Could not read %s in %s: %s", name,
|
||||
escaped_dir ? escaped_dir : "<allocation failed>", strerror(errno));
|
||||
free(escaped_dir);
|
||||
return true;
|
||||
@@ -975,25 +684,51 @@ bool filter_dir_merge_append(FilterRuleList* list, const char* dir_path, const F
|
||||
*exists = true;
|
||||
int rules_before = list->count;
|
||||
int dir_merges_before = list->dir_merge_count;
|
||||
/* Merge files inside a per-directory file resolve relative to that
|
||||
directory. */
|
||||
bool ok =
|
||||
filter_merge_read(list, fp, spec->name, spec, opts, dir_path, owner_rel, 0, err, err_size);
|
||||
char* line = NULL;
|
||||
size_t line_cap = 0;
|
||||
bool ok = true;
|
||||
while (true) {
|
||||
ssize_t n = utils_getdelim_bounded(fp, &line, &line_cap, '\n', UTILS_MAX_LINE_LEN);
|
||||
if (n < 0) {
|
||||
if (errno == EFBIG) {
|
||||
filter_set_error(err, err_size, "line in %s exceeds %d bytes", name,
|
||||
(int)UTILS_MAX_LINE_LEN);
|
||||
} else {
|
||||
filter_set_error(err, err_size, "error reading %s: %s", name, strerror(errno));
|
||||
}
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
if (n == 0)
|
||||
break;
|
||||
const char* p = line;
|
||||
while (*p == ' ' || *p == '\t')
|
||||
p++;
|
||||
if (*p == '\0' || *p == '\n' || *p == '\r' || *p == '#')
|
||||
continue;
|
||||
/* Merge files inside a per-directory file resolve relative to that
|
||||
directory. */
|
||||
if (!filter_list_parse_append_depth(list, p, opts, dir_path, 0, err, err_size)) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
free(line);
|
||||
fclose(fp);
|
||||
if (!ok) {
|
||||
filter_file_rollback(list, rules_before, dir_merges_before);
|
||||
return false;
|
||||
}
|
||||
for (int i = rules_before; i < list->count; i++) {
|
||||
if (!set_rule_owner(list->items[i], owner_rel)) {
|
||||
filter_set_error(err, err_size, "memory allocation failed");
|
||||
filter_file_rollback(list, rules_before, dir_merges_before);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool filter_file_append(FilterRuleList* list, const char* dir_path, const char* name,
|
||||
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||
char* err, size_t err_size) {
|
||||
FilterDirMerge spec = {.name = (char*)name};
|
||||
return filter_dir_merge_append(list, dir_path, &spec, owner_rel, opts, exists, err, err_size);
|
||||
}
|
||||
|
||||
FilterRuleList* filter_file_read_named(const char* dir_path, const char* name,
|
||||
const char* owner_rel, const FilterParseOptions* opts,
|
||||
bool* exists, char* err, size_t err_size) {
|
||||
@@ -1035,16 +770,11 @@ static FilterAction rule_matches(const FilterRule* rule, const char* rel_path, c
|
||||
return FILTER_ACTION_NONE;
|
||||
if (!(rule->sides & side))
|
||||
return FILTER_ACTION_NONE;
|
||||
/* A rule applies only to entries below its owner directory. The receive
|
||||
* root's destination-relative coordinate may be written as "." (the
|
||||
* synced-directory sentinel), which is the same scope as the empty owner. */
|
||||
const char* owner = rule->owner;
|
||||
if (owner && strcmp(owner, ".") == 0)
|
||||
owner = "";
|
||||
/* A rule applies only to entries below its owner directory. */
|
||||
const char* rel2 = rel_path;
|
||||
if (owner && owner[0] != '\0') {
|
||||
size_t owner_len = strlen(owner);
|
||||
if (strncmp(owner, rel_path, owner_len) != 0)
|
||||
if (rule->owner && rule->owner[0] != '\0') {
|
||||
size_t owner_len = strlen(rule->owner);
|
||||
if (strncmp(rule->owner, rel_path, owner_len) != 0)
|
||||
return FILTER_ACTION_NONE;
|
||||
if (rel_path[owner_len] != '/')
|
||||
return FILTER_ACTION_NONE;
|
||||
@@ -1052,10 +782,6 @@ static FilterAction rule_matches(const FilterRule* rule, const char* rel_path, c
|
||||
}
|
||||
if (rel2[0] == '\0')
|
||||
return FILTER_ACTION_NONE;
|
||||
/* A no-inherit rule ('n' on its dir-merge) applies only to direct children of
|
||||
* its owner directory, never to deeper entries. */
|
||||
if (rule->no_inherit && strchr(rel2, '/') != NULL)
|
||||
return FILTER_ACTION_NONE;
|
||||
bool matched;
|
||||
if (rule->dir_only && !is_dir)
|
||||
matched = false;
|
||||
@@ -1086,44 +812,7 @@ FilterAction filter_rules_apply_side(const FilterRuleList* list, const char* rel
|
||||
return FILTER_ACTION_NONE;
|
||||
}
|
||||
|
||||
FilterAction filter_dir_rules_apply_side(const FilterRuleList* dir_rules, const char* rel_path,
|
||||
const char* leaf, bool is_dir) {
|
||||
if (!dir_rules || !rel_path)
|
||||
return FILTER_ACTION_NONE;
|
||||
size_t len = strlen(rel_path);
|
||||
if (len == 0)
|
||||
return FILTER_ACTION_NONE;
|
||||
/* The containing directory of rel_path is the prefix before its final '/'. */
|
||||
size_t owner_len = 0;
|
||||
for (size_t i = 0; i < len; i++) {
|
||||
if (rel_path[i] == '/')
|
||||
owner_len = i;
|
||||
}
|
||||
for (;;) {
|
||||
for (int i = 0; i < dir_rules->count; i++) {
|
||||
const FilterRule* rule = dir_rules->items[i];
|
||||
const char* rule_owner = rule && rule->owner ? rule->owner : "";
|
||||
/* "." is the receive root's coordinate (see rule_matches). */
|
||||
if (strcmp(rule_owner, ".") == 0)
|
||||
rule_owner = "";
|
||||
size_t rule_owner_len = strlen(rule_owner);
|
||||
if (rule_owner_len != owner_len)
|
||||
continue;
|
||||
if (owner_len != 0 && memcmp(rule_owner, rel_path, owner_len) != 0)
|
||||
continue;
|
||||
FilterAction action = rule_matches(rule, rel_path, leaf, is_dir, FILTER_SIDE_RECEIVER);
|
||||
if (action != FILTER_ACTION_NONE)
|
||||
return action;
|
||||
}
|
||||
if (owner_len == 0)
|
||||
break;
|
||||
/* Move to the parent directory: the last '/' before owner_len. */
|
||||
size_t parent = 0;
|
||||
for (size_t j = 0; j < owner_len; j++) {
|
||||
if (rel_path[j] == '/')
|
||||
parent = j;
|
||||
}
|
||||
owner_len = parent;
|
||||
}
|
||||
return FILTER_ACTION_NONE;
|
||||
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
|
||||
bool is_dir) {
|
||||
return filter_rules_apply_side(list, rel_path, leaf, is_dir, FILTER_SIDE_SENDER);
|
||||
}
|
||||
|
||||
+11
-62
@@ -23,14 +23,7 @@
|
||||
* dir-merge/: per-directory merge file (registered for the scanner)
|
||||
* clear/! clear the current rule list (takes no argument)
|
||||
* Modifiers: '/' absolute anchor, '!' negate match, 'C' inject CVS defaults,
|
||||
* 's' sender side, 'r' receiver side, 'p' perishable. The rsync 'x'
|
||||
* (xattr-name) modifier is not implemented and is rejected explicitly
|
||||
* everywhere. The merge-only modifiers are accepted only on merge/dir-merge
|
||||
* rules (rejected on every other rule, matching rsync): 'e' excludes the merge
|
||||
* file itself, 'n' makes the merged rules non-inheriting (they apply only to
|
||||
* the directory that holds the merge file), 'w' word-splits the merge file on
|
||||
* whitespace instead of lines, and '-' reads the merge file as a list of bare
|
||||
* exclude patterns with no rule prefixes.
|
||||
* 's' sender side, 'r' receiver side, 'p' perishable, 'x' xattr name rule.
|
||||
* A trailing '/' makes a pattern match directories only. A leading '/' anchors
|
||||
* the pattern to its owner directory.
|
||||
*/
|
||||
@@ -56,32 +49,18 @@ typedef struct {
|
||||
bool dir_only; /* pattern had a trailing '/': matches directories only */
|
||||
bool negate; /* '!' modifier: match succeeds when the pattern does not */
|
||||
bool perishable; /* 'p' modifier (ignored in deleted directories) */
|
||||
bool no_inherit; /* 'n' on the owning dir-merge: applies only in `owner` */
|
||||
char* owner; /* owning directory rel path ("" == transfer root) */
|
||||
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
|
||||
} FilterRule;
|
||||
|
||||
/* One per-directory merge-file registration ("dir-merge NAME"/": NAME", "merge
|
||||
* NAME"/". NAME" and -F's .rsync-filter) together with the merge-only modifiers
|
||||
* parsed from the rule. The scanner reads `name` in every directory it
|
||||
* traverses and applies `no_prefixes`/`include`/`word_split`/`no_inherit`/
|
||||
* `exclude_self` while merging the file's rules. */
|
||||
typedef struct {
|
||||
char* name;
|
||||
bool no_prefixes; /* '-' : file holds only bare exclude patterns */
|
||||
bool include; /* '+' : file holds only bare include patterns */
|
||||
bool word_split; /* 'w' : split the file on whitespace, not lines */
|
||||
bool no_inherit; /* 'n' : the merged rules do not inherit below their dir */
|
||||
bool exclude_self; /* 'e' : exclude the merge file itself from the transfer */
|
||||
} FilterDirMerge;
|
||||
|
||||
typedef struct FilterRuleList {
|
||||
FilterRule** items; /* owned array of rule pointers */
|
||||
int count;
|
||||
int capacity;
|
||||
/* Per-directory merge-file registrations. Owned; the scanner reads each
|
||||
* name in every directory it traverses. */
|
||||
FilterDirMerge* dir_merges;
|
||||
/* Per-directory merge-file basenames registered by "dir-merge NAME"/": NAME"
|
||||
* or by -F (.rsync-filter). Owned strings; the scanner reads each name in
|
||||
* every directory it traverses. */
|
||||
char** dir_merge_names;
|
||||
int dir_merge_count;
|
||||
int dir_merge_capacity;
|
||||
} FilterRuleList;
|
||||
@@ -98,28 +77,13 @@ typedef struct {
|
||||
FilterRule* filter_rule_parse(const char* line, const FilterParseOptions* opts, char* err,
|
||||
size_t err_size);
|
||||
void filter_rule_free(FilterRule* rule);
|
||||
/* Deep-copy a rule (owned pattern/owner). Returns NULL on allocation failure. */
|
||||
FilterRule* filter_rule_clone(const FilterRule* rule);
|
||||
/* Replace a rule's owner directory (owned copy of `owner`, "" for the transfer
|
||||
* root). Returns false on allocation failure, leaving the rule unchanged.
|
||||
* Used to re-express a mirrored per-directory rule in the receiver's
|
||||
* destination-relative coordinate system. */
|
||||
bool filter_rule_set_owner(FilterRule* rule, const char* owner);
|
||||
|
||||
FilterRuleList* filter_rule_list_create(void);
|
||||
/* Append a fully-parsed rule (takes ownership). Returns false on OOM. */
|
||||
bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule);
|
||||
/* Register a per-directory merge-file basename (idempotent, no modifiers).
|
||||
* Returns false on OOM. Used by the scanner to read custom "dir-merge" files. */
|
||||
/* Register a per-directory merge-file basename (idempotent). Returns false on
|
||||
* OOM. Used by the scanner to read custom "dir-merge" files. */
|
||||
bool filter_rule_list_add_dir_merge(FilterRuleList* list, const char* name);
|
||||
/* Register a per-directory merge file with its merge-only modifiers. On a
|
||||
* duplicate name the existing registration is kept (rsync's first wins) and true
|
||||
* is returned. When `exclude_self` is set an implicit exclude rule for the
|
||||
* merge file's basename is appended to the list at this position, matching
|
||||
* rsync's `e` modifier. Returns false on OOM. */
|
||||
bool filter_rule_list_add_dir_merge_ex(FilterRuleList* list, const char* name, bool no_prefixes,
|
||||
bool include, bool word_split, bool no_inherit,
|
||||
bool exclude_self);
|
||||
/* Parse `line` and append it. Handles "clear"/"!" (resets the list), "merge
|
||||
* FILE"/". FILE" (splices the file's rules) and "dir-merge NAME"/": NAME"
|
||||
* (registers a per-directory filename). Returns false and fills `err` on bad
|
||||
@@ -152,15 +116,6 @@ bool filter_file_append(FilterRuleList* list, const char* dir_path, const char*
|
||||
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||
char* err, size_t err_size);
|
||||
|
||||
/* Append a per-directory merge file honoring its merge-only modifiers: `-`
|
||||
* reads every (word-split, when `w`) token as a bare exclude, `+` as a bare
|
||||
* include, and `n` marks each read rule non-inheriting. A plain name behaves
|
||||
* like filter_file_append. A missing file yields *exists=false with no error;
|
||||
* returns false only on a parse/allocation failure (message in `err`). */
|
||||
bool filter_dir_merge_append(FilterRuleList* list, const char* dir_path, const FilterDirMerge* spec,
|
||||
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
|
||||
char* err, size_t err_size);
|
||||
|
||||
/* filter_file_read_named with the default ".rsync-filter" name. */
|
||||
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
|
||||
char* err, size_t err_size);
|
||||
@@ -174,15 +129,9 @@ FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bo
|
||||
FilterAction filter_rules_apply_side(const FilterRuleList* list, const char* rel_path,
|
||||
const char* leaf, bool is_dir, unsigned side);
|
||||
|
||||
/* Evaluate a received per-directory rule set for the receiver side, using
|
||||
* rsync's per-directory-before-ancestors order: the entry's containing
|
||||
* directory's rules are tried first, then each ancestor's, then the receive
|
||||
* root's. `dir_rules` is a flat list whose rules carry `owner`; a rule applies
|
||||
* only when `owner` is exactly the directory being examined (a no-inherit rule
|
||||
* therefore applies only to that directory's direct children). Returns the
|
||||
* first matching rule's receiver verdict (PROTECT/RISK) or FILTER_ACTION_NONE.
|
||||
* The caller evaluates the command-line base rules after this chain. */
|
||||
FilterAction filter_dir_rules_apply_side(const FilterRuleList* dir_rules, const char* rel_path,
|
||||
const char* leaf, bool is_dir);
|
||||
/* Sender-side convenience wrapper (kept for callers/tests that only need the
|
||||
* transfer decision). */
|
||||
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
|
||||
bool is_dir);
|
||||
|
||||
#endif
|
||||
|
||||
+17
-29
@@ -62,16 +62,14 @@ bool format_dest_state_send(int fd, const OutputDestState* state) {
|
||||
if (!state)
|
||||
return false;
|
||||
int32_t has_old = state->existed ? 1 : 0;
|
||||
int32_t target_matches = state->target_matches ? 1 : 0;
|
||||
uint64_t size = (uint64_t)state->size;
|
||||
int64_t mtime = (int64_t)state->mtime_sec;
|
||||
int64_t mtime_nsec = state->mtime_nsec;
|
||||
uint32_t mode = state->mode;
|
||||
int32_t uid = state->uid;
|
||||
int32_t gid = state->gid;
|
||||
return send_n_data(fd, &has_old, sizeof(has_old)) &&
|
||||
send_n_data(fd, &target_matches, sizeof(target_matches)) &&
|
||||
send_n_data(fd, &size, sizeof(size)) && send_n_data(fd, &mtime, sizeof(mtime)) &&
|
||||
return send_n_data(fd, &has_old, sizeof(has_old)) && send_n_data(fd, &size, sizeof(size)) &&
|
||||
send_n_data(fd, &mtime, sizeof(mtime)) &&
|
||||
send_n_data(fd, &mtime_nsec, sizeof(mtime_nsec)) && send_n_data(fd, &mode, sizeof(mode)) &&
|
||||
send_n_data(fd, &uid, sizeof(uid)) && send_n_data(fd, &gid, sizeof(gid));
|
||||
}
|
||||
@@ -80,16 +78,14 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
|
||||
if (!state)
|
||||
return false;
|
||||
int32_t has_old = 0;
|
||||
int32_t target_matches = 0;
|
||||
uint64_t size = 0;
|
||||
int64_t mtime = 0;
|
||||
int64_t mtime_nsec = 0;
|
||||
uint32_t mode = 0;
|
||||
int32_t uid = 0;
|
||||
int32_t gid = 0;
|
||||
if (!receive_n_data(fd, &has_old, sizeof(has_old)) ||
|
||||
!receive_n_data(fd, &target_matches, sizeof(target_matches)) ||
|
||||
!receive_n_data(fd, &size, sizeof(size)) || !receive_n_data(fd, &mtime, sizeof(mtime)) ||
|
||||
if (!receive_n_data(fd, &has_old, sizeof(has_old)) || !receive_n_data(fd, &size, sizeof(size)) ||
|
||||
!receive_n_data(fd, &mtime, sizeof(mtime)) ||
|
||||
!receive_n_data(fd, &mtime_nsec, sizeof(mtime_nsec)) ||
|
||||
!receive_n_data(fd, &mode, sizeof(mode)) || !receive_n_data(fd, &uid, sizeof(uid)) ||
|
||||
!receive_n_data(fd, &gid, sizeof(gid)))
|
||||
@@ -97,7 +93,6 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
|
||||
memset(state, 0, sizeof(*state));
|
||||
state->known = true;
|
||||
state->existed = has_old != 0;
|
||||
state->target_matches = target_matches != 0;
|
||||
state->size = size;
|
||||
state->mtime_sec = mtime;
|
||||
state->mtime_nsec = mtime_nsec;
|
||||
@@ -110,32 +105,25 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
|
||||
bool format_stats_send(int fd, const ReceiverStats* stats) {
|
||||
if (!stats)
|
||||
return false;
|
||||
unsigned long long fields[12] = {
|
||||
stats->matched_data, stats->deleted_files, stats->would_delete_count, stats->literal_bytes,
|
||||
stats->created_reg, stats->created_dir, stats->created_link, stats->created_special,
|
||||
stats->deleted_reg, stats->deleted_dir, stats->deleted_link, stats->deleted_special,
|
||||
};
|
||||
return send_n_data(fd, fields, sizeof(fields));
|
||||
unsigned long long matched = stats->matched_data;
|
||||
unsigned long long deleted = stats->deleted_files;
|
||||
unsigned long long would = stats->would_delete_count;
|
||||
return send_n_data(fd, &matched, sizeof(matched)) && send_n_data(fd, &deleted, sizeof(deleted)) &&
|
||||
send_n_data(fd, &would, sizeof(would));
|
||||
}
|
||||
|
||||
bool format_stats_receive(int fd, ReceiverStats* stats) {
|
||||
if (!stats)
|
||||
return false;
|
||||
unsigned long long fields[12] = {0};
|
||||
if (!receive_n_data(fd, fields, sizeof(fields)))
|
||||
unsigned long long matched = 0;
|
||||
unsigned long long deleted = 0;
|
||||
unsigned long long would = 0;
|
||||
if (!receive_n_data(fd, &matched, sizeof(matched)) ||
|
||||
!receive_n_data(fd, &deleted, sizeof(deleted)) || !receive_n_data(fd, &would, sizeof(would)))
|
||||
return false;
|
||||
memset(stats, 0, sizeof(*stats));
|
||||
stats->matched_data = fields[0];
|
||||
stats->deleted_files = fields[1];
|
||||
stats->would_delete_count = fields[2];
|
||||
stats->literal_bytes = fields[3];
|
||||
stats->created_reg = fields[4];
|
||||
stats->created_dir = fields[5];
|
||||
stats->created_link = fields[6];
|
||||
stats->created_special = fields[7];
|
||||
stats->deleted_reg = fields[8];
|
||||
stats->deleted_dir = fields[9];
|
||||
stats->deleted_link = fields[10];
|
||||
stats->deleted_special = fields[11];
|
||||
stats->matched_data = matched;
|
||||
stats->deleted_files = deleted;
|
||||
stats->would_delete_count = would;
|
||||
return true;
|
||||
}
|
||||
|
||||
+5
-59
@@ -17,18 +17,10 @@
|
||||
/* Pre-transfer destination snapshot, reported by the receiver when the wire
|
||||
* config carries report_dest_info. `known` distinguishes "no report was
|
||||
* requested/received" from "the destination did not exist" (`existed == false`
|
||||
* with `known == true`).
|
||||
*
|
||||
* `target_matches` is meaningful only for a symlink destination (protocol
|
||||
* 2.30.0): the receiver compares its on-disk link target with the incoming
|
||||
* target and reports whether they are equal, so the sender can render rsync's
|
||||
* `cLc........` (target changed) versus `.L..t......` (attributes only) and
|
||||
* suppress an unchanged symlink's line entirely. It is always false for every
|
||||
* other entry kind. */
|
||||
* with `known == true`). */
|
||||
typedef struct {
|
||||
bool known;
|
||||
bool existed;
|
||||
bool target_matches;
|
||||
unsigned long long size;
|
||||
long long mtime_sec;
|
||||
long long mtime_nsec;
|
||||
@@ -65,37 +57,14 @@ bool format_dest_state_send(int fd, const OutputDestState* state);
|
||||
bool format_dest_state_receive(int fd, OutputDestState* state);
|
||||
|
||||
/* End-of-transfer receiver counters reported through STATUS_STATS (protocol
|
||||
* 2.25.0, extended in 2.28.0 and 2.30.0) when the wire config carries
|
||||
* report_stats. `would_delete_count` is the number of destination-relative
|
||||
* paths the receiver would have deleted in a -n/--dry-run --delete run; that
|
||||
* many wire strings immediately follow the fixed record (sent/read by the
|
||||
* caller).
|
||||
*
|
||||
* Protocol 2.28.0 adds the receiver-observed counters the sender cannot see:
|
||||
* `literal_bytes` is the file data the receiver actually stored literally
|
||||
* (whole files plus the literal fragments of a delta) and the four `created_*`
|
||||
* counters split the destination entries the receiver newly created by type,
|
||||
* reproducing rsync's `Number of created files` breakdown and an exact
|
||||
* `Literal data` for a delta run.
|
||||
*
|
||||
* Protocol 2.30.0 appends the four `deleted_*` counters: the same reg/dir/link/
|
||||
* special split for the entries the receiver ACTUALLY removed, so `--stats` can
|
||||
* render rsync's `Number of deleted files: X (reg: A, dir: B, link: C,
|
||||
* special: D)` parenthetical. The scalar `deleted_files` stays the authoritative
|
||||
* total (the breakdown is a strict partition of it). */
|
||||
* 2.25.0) when the wire config carries report_stats. `would_delete_count` is
|
||||
* the number of destination-relative paths the receiver would have deleted in a
|
||||
* -n/--dry-run --delete run; that many wire strings immediately follow the
|
||||
* fixed record (sent/read by the caller). */
|
||||
typedef struct {
|
||||
unsigned long long matched_data;
|
||||
unsigned long long deleted_files;
|
||||
unsigned long long would_delete_count;
|
||||
unsigned long long literal_bytes;
|
||||
unsigned long long created_reg;
|
||||
unsigned long long created_dir;
|
||||
unsigned long long created_link;
|
||||
unsigned long long created_special;
|
||||
unsigned long long deleted_reg;
|
||||
unsigned long long deleted_dir;
|
||||
unsigned long long deleted_link;
|
||||
unsigned long long deleted_special;
|
||||
} ReceiverStats;
|
||||
|
||||
/* Fixed-width STATUS_STATS counter record. The status frame and the optional
|
||||
@@ -104,27 +73,4 @@ typedef struct {
|
||||
bool format_stats_send(int fd, const ReceiverStats* stats);
|
||||
bool format_stats_receive(int fd, ReceiverStats* stats);
|
||||
|
||||
/* Sender-side file-list accounting for rsync's `--stats` block. Filled while
|
||||
* the scan/send loops walk each entry: the flist counters describe every
|
||||
* scanned source entry (transferred or skipped), while the transferred/literal
|
||||
* counters describe only the regular files the receiver actually stored. The
|
||||
* type split lets the client print rsync's `Number of files` breakdown; the
|
||||
* receiver-only counters (matched data, deleted, created) come from
|
||||
* STATUS_STATS. */
|
||||
typedef struct {
|
||||
unsigned long long flist_reg;
|
||||
unsigned long long flist_dir;
|
||||
unsigned long long flist_link;
|
||||
unsigned long long flist_special;
|
||||
unsigned long long total_file_size; /* sum of entry sizes (link target len) */
|
||||
unsigned long long transferred_regular; /* regular files actually stored */
|
||||
unsigned long long transferred_file_size; /* source size of those files */
|
||||
/* Whole-file accuracy: the `--stats` "Literal data" row. The sender counts
|
||||
* the source size of every stored file, so a whole-file transfer matches
|
||||
* rsync. A delta run actually ships only the literal fragments of the diff
|
||||
* (the rest is matched/copied), so here the value is an upper bound, not
|
||||
* rsync's literal-byte total; see RSYNC_COMPAT.md's `--stats` row. */
|
||||
unsigned long long literal_data;
|
||||
} TransferStats;
|
||||
|
||||
#endif
|
||||
|
||||
+12
-164
@@ -3,7 +3,6 @@
|
||||
#include "utils.h"
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <fnmatch.h>
|
||||
#include <grp.h>
|
||||
#include <limits.h>
|
||||
#include <pwd.h>
|
||||
@@ -13,8 +12,6 @@
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
|
||||
static bool identity_id_fits_int32(unsigned long id);
|
||||
|
||||
/* The active identity snapshot lives in a per-process global. The TCP server
|
||||
* forks one child process per connection, so a connection never shares this
|
||||
* with another; within a connection the multithreaded receiver reads it without
|
||||
@@ -276,7 +273,7 @@ static bool identity_wire_map_valid(const IdentityMap* map) {
|
||||
}
|
||||
if (map->to < IDENTITY_CURRENT)
|
||||
return false;
|
||||
if (map->to_name && strlen(map->to_name) > IDENTITY_MAX_NAME_LEN)
|
||||
if (map->to_name && strlen(map->to_name) > 255)
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
@@ -422,10 +419,17 @@ static int identity_parse_from(const char* token, bool is_group, int32_t* out_fr
|
||||
/* Not a numeric LOW-HIGH range: fall through and treat as a name (a
|
||||
* hyphenated account name like "wayne-smith" must still resolve). */
|
||||
}
|
||||
/* A sender-side name. A FROM name wildcard other than the bare '*' is handled
|
||||
* by identity_expand_from_glob() in the caller (it expands against the
|
||||
* sender's account database at CLI-parse time), so this function only sees the
|
||||
* bare '*' or a literal name here. */
|
||||
/* A sender-side name. A wildcard other than the bare '*' is matched by rsync
|
||||
* against the sender's names; because FastSync transmits numeric ids only, the
|
||||
* receiver cannot evaluate it, so reject rather than silently mis-match. */
|
||||
if (identity_token_has_glob(token)) {
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"%smap FROM '%s': name wildcards other than '*' are not supported "
|
||||
"(FastSync transmits numeric ids, so sender names are unavailable on the "
|
||||
"receiver)",
|
||||
is_group ? "--group" : "--user", token);
|
||||
return -1;
|
||||
}
|
||||
int32_t id;
|
||||
if (identity_resolve_token(token, is_group, &id) != 0)
|
||||
return -1;
|
||||
@@ -482,138 +486,6 @@ static int identity_append_rule(IdentityMap** map, int* count, const IdentityMap
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* True when `lo` and `hi` are adjacent ids (no overflow at INT32_MAX). */
|
||||
static bool identity_ids_adjacent(int32_t lo, int32_t hi) {
|
||||
return lo < INT32_MAX && hi == lo + 1;
|
||||
}
|
||||
|
||||
static int identity_id_cmp(const void* a, const void* b) {
|
||||
int32_t x = *(const int32_t*)a;
|
||||
int32_t y = *(const int32_t*)b;
|
||||
return (x > y) - (x < y);
|
||||
}
|
||||
|
||||
static bool identity_ids_push(int32_t** ids, size_t* count, size_t* cap, int32_t id) {
|
||||
if (*count == *cap) {
|
||||
size_t grown_cap = *cap ? *cap * 2 : 16;
|
||||
int32_t* grown = realloc(*ids, grown_cap * sizeof(int32_t));
|
||||
if (!grown)
|
||||
return false;
|
||||
*ids = grown;
|
||||
*cap = grown_cap;
|
||||
}
|
||||
(*ids)[(*count)++] = id;
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Expand a FROM name wildcard (rsync's match against sender-side account names)
|
||||
* into one rule per contiguous run of matching numeric ids, all sharing the same
|
||||
* TO side. FastSync transmits numeric ids only, so the wildcard must be
|
||||
* resolved here -- at CLI-parse time -- against the SENDER's passwd/group
|
||||
* database; the receiver has no sender names to match. Contiguous matched ids
|
||||
* are collapsed into a single LOW-HIGH range (a range of adjacent ids contains
|
||||
* exactly the ids it spans, so this is semantically exact). Returns 0 on
|
||||
* success, -1 on an allocation failure, a wildcard that matches no sender
|
||||
* account, or an expansion that would push the map past MAX_IDENTITY_MAP. */
|
||||
static int identity_expand_from_glob(Config* config, const char* glob, bool is_group,
|
||||
const IdentityMap* to_rule) {
|
||||
const char* optname = is_group ? "--groupmap" : "--usermap";
|
||||
size_t cap = 0;
|
||||
size_t n = 0;
|
||||
int32_t* ids = NULL;
|
||||
bool alloc_failed = false;
|
||||
|
||||
if (is_group) {
|
||||
setgrent();
|
||||
struct group* gr;
|
||||
while ((gr = getgrent()) != NULL) {
|
||||
if (fnmatch(glob, gr->gr_name, 0) != 0)
|
||||
continue;
|
||||
if (!identity_id_fits_int32((unsigned long)gr->gr_gid))
|
||||
continue;
|
||||
if (!identity_ids_push(&ids, &n, &cap, (int32_t)gr->gr_gid)) {
|
||||
alloc_failed = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
endgrent();
|
||||
} else {
|
||||
setpwent();
|
||||
struct passwd* pw;
|
||||
while ((pw = getpwent()) != NULL) {
|
||||
if (fnmatch(glob, pw->pw_name, 0) != 0)
|
||||
continue;
|
||||
if (!identity_id_fits_int32((unsigned long)pw->pw_uid))
|
||||
continue;
|
||||
if (!identity_ids_push(&ids, &n, &cap, (int32_t)pw->pw_uid)) {
|
||||
alloc_failed = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
endpwent();
|
||||
}
|
||||
|
||||
if (alloc_failed) {
|
||||
free(ids);
|
||||
log_message(LOG_LEVEL_ERROR, "%s: memory allocation failed expanding FROM '%s'", optname, glob);
|
||||
return -1;
|
||||
}
|
||||
if (n == 0) {
|
||||
free(ids);
|
||||
log_message(LOG_LEVEL_ERROR, "%s FROM '%s': no source account name matches the wildcard",
|
||||
optname, glob);
|
||||
return -1;
|
||||
}
|
||||
|
||||
qsort(ids, n, sizeof(int32_t), identity_id_cmp);
|
||||
size_t unique = 0;
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
if (unique == 0 || ids[unique - 1] != ids[i])
|
||||
ids[unique++] = ids[i];
|
||||
}
|
||||
n = unique;
|
||||
|
||||
int runs = 0;
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
if (i == 0 || !identity_ids_adjacent(ids[i - 1], ids[i]))
|
||||
runs++;
|
||||
}
|
||||
|
||||
IdentityMap** map = is_group ? &config->groupmap : &config->usermap;
|
||||
int* count = is_group ? &config->groupmap_count : &config->usermap_count;
|
||||
if (*count > MAX_IDENTITY_MAP - runs) {
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"%s FROM '%s': the name wildcard expands to %d rule(s), which would exceed "
|
||||
"the maximum of %d map rules",
|
||||
optname, glob, runs, MAX_IDENTITY_MAP);
|
||||
free(ids);
|
||||
return -1;
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < n;) {
|
||||
size_t j = i;
|
||||
while (j + 1 < n && identity_ids_adjacent(ids[j], ids[j + 1]))
|
||||
j++;
|
||||
IdentityMap rule;
|
||||
rule.from = ids[i];
|
||||
rule.from_hi = ids[j];
|
||||
rule.to = to_rule->to;
|
||||
rule.to_name = to_rule->to_name ? str_dup(to_rule->to_name) : NULL;
|
||||
if (to_rule->to_name && !rule.to_name) {
|
||||
free(ids);
|
||||
return -1;
|
||||
}
|
||||
if (identity_append_rule(map, count, &rule) != 0) {
|
||||
free(rule.to_name);
|
||||
free(ids);
|
||||
return -1;
|
||||
}
|
||||
i = j + 1;
|
||||
}
|
||||
free(ids);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int identity_parse_map(Config* config, const char* value, bool is_group) {
|
||||
if (!config || !value || *value == '\0') {
|
||||
log_message(LOG_LEVEL_ERROR, "%smap requires a value", is_group ? "--group" : "--user");
|
||||
@@ -637,30 +509,6 @@ int identity_parse_map(Config* config, const char* value, bool is_group) {
|
||||
char* to_token = colon + 1;
|
||||
IdentityMap parsed;
|
||||
memset(&parsed, 0, sizeof(parsed));
|
||||
/* A FROM name wildcard (anything with a glob metacharacter other than the
|
||||
* bare '*') is expanded against the sender's account database here, while
|
||||
* the sender's passwd/group DB is still available; the resulting numeric
|
||||
* rules travel on the wire like an explicit list. The TO side is parsed
|
||||
* first so every expanded rule shares it. */
|
||||
if (strcmp(from_token, "*") != 0 && identity_token_has_glob(from_token)) {
|
||||
if (identity_parse_to(to_token, is_group, &parsed.to, &parsed.to_name) != 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "%s could not parse TO '%s' in '%s'", optname, to_token,
|
||||
value);
|
||||
free(list);
|
||||
return -1;
|
||||
}
|
||||
if (identity_expand_from_glob(config, from_token, is_group, &parsed) != 0) {
|
||||
free(parsed.to_name);
|
||||
free(list);
|
||||
return -1;
|
||||
}
|
||||
/* Every rule emitted by the expansion took its own str_dup of the name,
|
||||
* so the parse-time copy is unreachable on success: release it here (the
|
||||
* failure path above already does). `parsed.to_name` is NULL for a
|
||||
* numeric TO. */
|
||||
free(parsed.to_name);
|
||||
continue;
|
||||
}
|
||||
if (identity_parse_from(from_token, is_group, &parsed.from, &parsed.from_hi) != 0) {
|
||||
log_message(LOG_LEVEL_ERROR,
|
||||
"%s could not resolve FROM '%s' in '%s' (a name must exist on the "
|
||||
|
||||
+2
-13
@@ -6,11 +6,6 @@
|
||||
#include <stdint.h>
|
||||
#include <sys/types.h>
|
||||
|
||||
/* Maximum length of a receiver-resolved identity name in a FROM:TO map's TO
|
||||
* field. Bounded so a malicious/huge name can never cross the wire (see
|
||||
* identity_wire_map_valid). */
|
||||
#define IDENTITY_MAX_NAME_LEN 255
|
||||
|
||||
/*
|
||||
* Identity mapping: --numeric-ids / --usermap / --groupmap / --chown / --copy-as.
|
||||
*
|
||||
@@ -30,14 +25,8 @@
|
||||
|
||||
/* Parse one --usermap= / --groupmap= value (comma-separated FROM:TO rules,
|
||||
* first match wins) into config->usermap / config->groupmap. is_group selects
|
||||
* the group tables and name databases. A FROM name wildcard (containing `*`,
|
||||
* `?` or `[...]`, but not the bare `*`) is expanded against the SENDER's
|
||||
* account database at parse time into one or more numeric id/range rules
|
||||
* (contiguous ids collapse to a range) sharing the same TO, because only
|
||||
* numeric ids cross the wire; the expansion is capped at MAX_IDENTITY_MAP and a
|
||||
* wildcard matching no account is an error. Returns 0 on success, -1 on a
|
||||
* malformed spec, an unresolvable name, an unmatched wildcard, or a map that
|
||||
* would exceed MAX_IDENTITY_MAP (never a silent no-op). */
|
||||
* the group tables and name databases. Returns 0 on success, -1 on a
|
||||
* malformed spec or an unresolvable name (never a silent no-op). */
|
||||
int identity_parse_map(Config* config, const char* value, bool is_group);
|
||||
|
||||
/* Parse --chown=USER:GROUP. Supports USER:GROUP, USER (owner only), :GROUP
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,41 +0,0 @@
|
||||
#ifndef INCREMENTAL_CHECK_H
|
||||
#define INCREMENTAL_CHECK_H
|
||||
|
||||
#include "config.h"
|
||||
#include "file_types.h"
|
||||
#include "protocol.h"
|
||||
#include <stdbool.h>
|
||||
|
||||
/* Incremental-check module: the per-file STATUS_CHECK state machine, the
|
||||
* incremental delta / alternate-basis / fuzzy matching helpers and the shared
|
||||
* xattr receive helper. These declarations are re-exported by the
|
||||
* file_receive.h facade. */
|
||||
|
||||
/* Whole-file payload bound shared by the plain receive path and the
|
||||
* incremental check paths. */
|
||||
#define MAX_FILE_DATA_SIZE MAX_RECEIVE_WHOLE_FILE_SIZE
|
||||
|
||||
/* Receive a file's xattr block (when the config enables xattr transport) and
|
||||
* attach it to `file`. Returns false on a malformed/oversized frame. */
|
||||
bool receive_file_xattrs(File* file, int fd, const Config* config);
|
||||
|
||||
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
|
||||
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
|
||||
* true only on the server-contacting --dry-run path when the file is not up to
|
||||
* date: the receiver has already sent STATUS_DRY_RUN_TRANSFER and returns NULL
|
||||
* without storing anything. On that path `*skipped` is true for an up-to-date
|
||||
* (STATUS_OK) file and both flags are false for a genuine error. */
|
||||
File* receive_incremental_check_ex(int fd, const Config* config, bool* skipped,
|
||||
bool* would_transfer);
|
||||
|
||||
/* Testable basis quick-check / verification policy. file_basis_quick_match is
|
||||
* rsync's metadata quick-check for a basis candidate (equal size is required
|
||||
* separately by the caller; this adds the --size-only / mtime / --modify-window
|
||||
* leg). file_basis_content_required reports whether a hit must ALSO be
|
||||
* confirmed by a whole-file content digest (--verify-basis; false is the
|
||||
* default rsync-parity behavior). */
|
||||
bool file_basis_quick_match(const Config* config, const struct stat* st, time_t check_mtime,
|
||||
long check_mtime_nsec);
|
||||
bool file_basis_content_required(const Config* config);
|
||||
|
||||
#endif
|
||||
+20
-87
@@ -1,5 +1,4 @@
|
||||
#include "log.h"
|
||||
#include "utils.h"
|
||||
#include <errno.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdarg.h>
|
||||
@@ -17,7 +16,6 @@ static bool info_flags_explicit = false;
|
||||
static FILE* log_fp = NULL;
|
||||
static _Thread_local bool eight_bit_output;
|
||||
static LogStderrMode stderr_mode = LOG_STDERR_ERRORS;
|
||||
static LogClientMsgSink client_msg_sink = NULL;
|
||||
|
||||
/* Serializes access to log_fp and makes each emitted line atomic: the
|
||||
* timestamp prefix, formatted body, and trailing newline are written as one
|
||||
@@ -79,51 +77,6 @@ LogStderrMode log_get_stderr_mode(void) {
|
||||
return stderr_mode;
|
||||
}
|
||||
|
||||
void log_set_client_msg_sink(LogClientMsgSink sink) {
|
||||
client_msg_sink = sink;
|
||||
}
|
||||
|
||||
LogClientMsgSink log_get_client_msg_sink(void) {
|
||||
return client_msg_sink;
|
||||
}
|
||||
|
||||
/* Format just the message body (no prefix/newline) into a freshly allocated
|
||||
* buffer. Shared by log_message (which may hand the body to a client-message
|
||||
* sink) and log_client_message. Returns NULL on allocation/format failure. */
|
||||
static char* format_log_body(const char* format, va_list args) {
|
||||
va_list copy;
|
||||
va_copy(copy, args);
|
||||
int body_len = vsnprintf(NULL, 0, format, copy);
|
||||
va_end(copy);
|
||||
if (body_len < 0)
|
||||
return NULL;
|
||||
char* body = malloc((size_t)body_len + 1);
|
||||
if (!body)
|
||||
return NULL;
|
||||
vsnprintf(body, (size_t)body_len + 1, format, args);
|
||||
return body;
|
||||
}
|
||||
|
||||
/* Assemble a complete log line (prefix + body + newline) from an already
|
||||
* formatted body. Returns NULL on allocation failure. */
|
||||
static char* format_log_line_from_body(LogLevel log_level, const struct tm* t, const char* body) {
|
||||
char prefix[64];
|
||||
int prefix_len = snprintf(
|
||||
prefix, sizeof(prefix), "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t->tm_year + 1900,
|
||||
t->tm_mon + 1, t->tm_mday, t->tm_hour, t->tm_min, t->tm_sec, log_level_strings[log_level]);
|
||||
if (prefix_len < 0 || prefix_len >= (int)sizeof(prefix))
|
||||
return NULL;
|
||||
size_t body_len = strlen(body);
|
||||
char* line = malloc((size_t)prefix_len + body_len + 2); /* body + '\n' + NUL */
|
||||
if (!line)
|
||||
return NULL;
|
||||
memcpy(line, prefix, (size_t)prefix_len);
|
||||
memcpy(line + prefix_len, body, body_len);
|
||||
line[(size_t)prefix_len + body_len] = '\n';
|
||||
line[(size_t)prefix_len + body_len + 1] = '\0';
|
||||
return line;
|
||||
}
|
||||
|
||||
/* Format one complete log line (timestamp prefix + body + newline) into a
|
||||
* freshly allocated buffer. This is pure CPU/malloc work and must happen
|
||||
* OUTSIDE the log mutex: the mutex only guards the log_fp pointer, so a
|
||||
@@ -131,11 +84,26 @@ static char* format_log_line_from_body(LogLevel log_level, const struct tm* t, c
|
||||
* on allocation/formatting failure. */
|
||||
static char* format_log_line(LogLevel log_level, const struct tm* t, const char* format,
|
||||
va_list args) {
|
||||
char* body = format_log_body(format, args);
|
||||
if (!body)
|
||||
char prefix[64];
|
||||
int prefix_len = snprintf(
|
||||
prefix, sizeof(prefix), "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t->tm_year + 1900,
|
||||
t->tm_mon + 1, t->tm_mday, t->tm_hour, t->tm_min, t->tm_sec, log_level_strings[log_level]);
|
||||
if (prefix_len < 0 || prefix_len >= (int)sizeof(prefix))
|
||||
return NULL;
|
||||
char* line = format_log_line_from_body(log_level, t, body);
|
||||
free(body);
|
||||
va_list copy;
|
||||
va_copy(copy, args);
|
||||
int body_len = vsnprintf(NULL, 0, format, copy);
|
||||
va_end(copy);
|
||||
if (body_len < 0)
|
||||
return NULL;
|
||||
size_t total = (size_t)prefix_len + (size_t)body_len;
|
||||
char* line = malloc(total + 2); /* body bytes + '\n' + NUL */
|
||||
if (!line)
|
||||
return NULL;
|
||||
memcpy(line, prefix, (size_t)prefix_len);
|
||||
vsnprintf(line + prefix_len, (size_t)body_len + 1, format, args);
|
||||
line[total] = '\n';
|
||||
line[total + 1] = '\0';
|
||||
return line;
|
||||
}
|
||||
|
||||
@@ -153,26 +121,6 @@ static void emit_log_line(FILE* console, const char* line) {
|
||||
mtx_unlock(&log_mutex);
|
||||
}
|
||||
|
||||
void log_client_message(const char* message) {
|
||||
if (!message)
|
||||
return;
|
||||
/* The body is peer-controlled: escape every non-printable byte (newlines,
|
||||
CR, ANSI ESC, ...) so a hostile client cannot forge log lines or inject
|
||||
terminal control sequences. output_escape() is the codebase's canonical
|
||||
escaper and leaves printable text untouched. */
|
||||
char* escaped = output_escape(message, log_get_8_bit_output());
|
||||
if (!escaped)
|
||||
return;
|
||||
/* Route through the ordinary log level / destination gate (log_message):
|
||||
this respects --log-file, the configured stderr mode and the level
|
||||
threshold instead of always writing to stderr. The wire body carries no
|
||||
severity, so the forwarded diagnostic is emitted as a warning -- the
|
||||
lowest level the default gate admits, which keeps the peer's messages
|
||||
visible without bypassing --quiet. */
|
||||
log_message(LOG_LEVEL_WARNING, "%s", escaped);
|
||||
free(escaped);
|
||||
}
|
||||
|
||||
void log_message(LogLevel log_level, const char* format, ...) {
|
||||
if (log_level < current_log_level)
|
||||
return;
|
||||
@@ -190,23 +138,8 @@ void log_message(LogLevel log_level, const char* format, ...) {
|
||||
|
||||
va_list args;
|
||||
va_start(args, format);
|
||||
char* body = format_log_body(format, args);
|
||||
char* line = format_log_line(log_level, &t, format, args);
|
||||
va_end(args);
|
||||
if (!body)
|
||||
return;
|
||||
/* LOG_STDERR_CLIENT: hand the diagnostic to the client-message channel. A
|
||||
sink that takes ownership suppresses the local write; otherwise (no sink
|
||||
yet, or the peer connection is not up) fall through to local output so the
|
||||
diagnostic is never lost. */
|
||||
if (stderr_mode == LOG_STDERR_CLIENT) {
|
||||
LogClientMsgSink sink = client_msg_sink;
|
||||
if (sink && sink(body)) {
|
||||
free(body);
|
||||
return;
|
||||
}
|
||||
}
|
||||
char* line = format_log_line_from_body(log_level, &t, body);
|
||||
free(body);
|
||||
if (!line)
|
||||
return;
|
||||
emit_log_line(dest_io, line);
|
||||
|
||||
+6
-69
@@ -5,40 +5,15 @@
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
|
||||
/* Ask the compiler to type-check the printf-style arguments of the variadic
|
||||
* logging helpers. Only enabled for GNU-compatible compilers (gcc/clang). */
|
||||
#if defined(__GNUC__)
|
||||
#define LOG_PRINTF_ATTR(fmt_idx, first_vararg_idx) \
|
||||
__attribute__((format(printf, fmt_idx, first_vararg_idx)))
|
||||
#else
|
||||
#define LOG_PRINTF_ATTR(fmt_idx, first_vararg_idx)
|
||||
#endif
|
||||
|
||||
typedef enum { LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_WARNING, LOG_LEVEL_ERROR } LogLevel;
|
||||
/* --stderr=MODE destinations. ERRORS keeps errors on stderr and everything
|
||||
* else on stdout; ALL sends every message to stderr; CLIENT routes the client's
|
||||
* own diagnostics over the protocol stream to the peer's stderr (rsync's
|
||||
* --stderr=client / --no-msgs2stderr). */
|
||||
typedef enum { LOG_STDERR_ERRORS, LOG_STDERR_ALL, LOG_STDERR_CLIENT } LogStderrMode;
|
||||
typedef enum { LOG_STDERR_ERRORS, LOG_STDERR_ALL } LogStderrMode;
|
||||
|
||||
typedef enum {
|
||||
LOG_DEBUG_IO = 1u << 0,
|
||||
LOG_DEBUG_PROTO = 1u << 1,
|
||||
LOG_DEBUG_PACK = 1u << 2,
|
||||
LOG_DEBUG_UTIL = 1u << 3,
|
||||
/* rsync --debug categories that now map to a natural FastSync event:
|
||||
* flist (file-list scan progress), del (deletions), hash/deltasum
|
||||
* (whole-file hashing and delta-sum generation), recv (receiver
|
||||
* responses/signatures), filter (selection/exclusion decisions) and send
|
||||
* (files handed to the sender). Only emitted when the category is
|
||||
* explicitly enabled; a normal run stays silent. */
|
||||
LOG_DEBUG_FLIST = 1u << 4,
|
||||
LOG_DEBUG_DEL = 1u << 5,
|
||||
LOG_DEBUG_HASH = 1u << 6,
|
||||
LOG_DEBUG_RECV = 1u << 7,
|
||||
LOG_DEBUG_FILTER = 1u << 8,
|
||||
LOG_DEBUG_SEND = 1u << 9,
|
||||
LOG_DEBUG_ALL = (1u << 10) - 1,
|
||||
LOG_DEBUG_ALL = (1u << 4) - 1,
|
||||
} LogDebugFlag;
|
||||
|
||||
typedef enum {
|
||||
@@ -46,33 +21,10 @@ typedef enum {
|
||||
LOG_INFO_MISC = 1u << 1,
|
||||
LOG_INFO_SKIP = 1u << 2,
|
||||
LOG_INFO_STATS = 1u << 3,
|
||||
/* rsync categories that map to a FastSync event (emitted in rsync's line
|
||||
* format): del (deletions), remove (sender-side source removal), name
|
||||
* (transferred entry names), flist (file-list header), nonreg (skipped
|
||||
* non-regular files), progress (per-file progress). rsync's `backup`
|
||||
* category is accepted for CLI parity but stays silent: the receiver does the
|
||||
* backing-up and FastSync has no backup event to report from the sender. */
|
||||
LOG_INFO_DEL = 1u << 4,
|
||||
LOG_INFO_REMOVE = 1u << 5,
|
||||
LOG_INFO_NAME = 1u << 6,
|
||||
LOG_INFO_FLIST = 1u << 7,
|
||||
LOG_INFO_NONREG = 1u << 8,
|
||||
LOG_INFO_PROGRESS = 1u << 9,
|
||||
/* Marker for `--info=name2` and higher: also print rsync's
|
||||
"NAME is uptodate" line for entries the receiver already has. It rides in
|
||||
the info_level bitset (there is no separate Config field) and is never set
|
||||
by --info=all (which selects level 1). */
|
||||
LOG_INFO_NAME_UPTODATE = 1u << 10,
|
||||
/* --info=mount: print rsync's `[sender] skipping mount-point dir NAME` when
|
||||
* -xx/--one-file-system drops a mount-point directory (FastSync's client is
|
||||
* the sender). */
|
||||
LOG_INFO_MOUNT = 1u << 11,
|
||||
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS | LOG_INFO_DEL |
|
||||
LOG_INFO_REMOVE | LOG_INFO_NAME | LOG_INFO_FLIST | LOG_INFO_NONREG |
|
||||
LOG_INFO_PROGRESS | LOG_INFO_MOUNT,
|
||||
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS,
|
||||
} LogInfoFlag;
|
||||
|
||||
void log_message(LogLevel log_level, const char* message, ...) LOG_PRINTF_ATTR(2, 3);
|
||||
void log_message(LogLevel log_level, const char* message, ...);
|
||||
void log_perror(const char* context);
|
||||
void set_log_level(LogLevel level);
|
||||
void set_log_debug_flags(uint32_t flags);
|
||||
@@ -81,29 +33,14 @@ uint32_t get_log_debug_flags(void);
|
||||
* the debug log level is enabled AND the flag is selected. Hot paths use this
|
||||
* to skip expensive message formatting/escaping when the line is filtered. */
|
||||
bool log_debug_enabled(LogDebugFlag flag);
|
||||
void log_debug_message(LogDebugFlag flag, const char* message, ...) LOG_PRINTF_ATTR(2, 3);
|
||||
void log_debug_message(LogDebugFlag flag, const char* message, ...);
|
||||
void set_log_info_flags(uint32_t flags);
|
||||
uint32_t get_log_info_flags(void);
|
||||
void log_info_message(LogInfoFlag flag, const char* message, ...) LOG_PRINTF_ATTR(2, 3);
|
||||
void log_info_message(LogInfoFlag flag, const char* message, ...);
|
||||
void log_set_file(FILE* fp);
|
||||
void log_set_8_bit_output(bool enabled);
|
||||
bool log_get_8_bit_output(void);
|
||||
void log_set_stderr_mode(LogStderrMode mode);
|
||||
LogStderrMode log_get_stderr_mode(void);
|
||||
/* Write a message a peer forwarded over the client-message channel to this
|
||||
* process's stderr (and log file), with the standard log prefix. Used by the
|
||||
* server side of rsync's --stderr=client. */
|
||||
void log_client_message(const char* message);
|
||||
|
||||
/* Sink for LOG_STDERR_CLIENT. log_message() passes the un-prefixed message
|
||||
* body to the installed sink; a `true` return means the sink took ownership
|
||||
* (e.g. queued it for protocol transmission) and the message must NOT also be
|
||||
* written locally. A `false` return (or a NULL sink) makes log_message fall
|
||||
* back to the normal local destination, so a diagnostic emitted before the peer
|
||||
* connection exists is never lost (rsync's documented fallback). The sink may
|
||||
* be called from any thread and must be tolerant of that. */
|
||||
typedef bool (*LogClientMsgSink)(const char* message);
|
||||
void log_set_client_msg_sink(LogClientMsgSink sink);
|
||||
LogClientMsgSink log_get_client_msg_sink(void);
|
||||
|
||||
#endif
|
||||
|
||||
+5
-20
@@ -211,22 +211,13 @@ FileMetadata* metadata_receive(int file_descriptor, int* ok) {
|
||||
|
||||
bool metadata_mode_for_policy(mode_t source_mode, mode_t current_mode, FileAttrPolicy policy,
|
||||
mode_t* out_mode) {
|
||||
const mode_t special_bits = (mode_t)(S_ISUID | S_ISGID | S_ISVTX);
|
||||
const mode_t execute_bits = S_IXUSR | S_IXGRP | S_IXOTH;
|
||||
if (policy.perms) {
|
||||
/* rsync --perms copies the source's permission and special bits exactly,
|
||||
* including group/other write and setuid/setgid/sticky. The kernel may
|
||||
* still clear setgid when the receiver is not in the file's group; the
|
||||
* caller logs a failed chmod rather than silently masking the bits here.
|
||||
* Setuid/setgid/sticky are super-user activities: when the connection did
|
||||
* not permit them (SUPER_MODE_OFF / --no-super) they are stripped, so a
|
||||
* client can never install a privileged bit on a receiver that forbade
|
||||
* super-user activities. This also covers bits introduced by --chmod,
|
||||
* whose result is fed in as source_mode. */
|
||||
mode_t bits = source_mode & (mode_t)(special_bits | 0777);
|
||||
if (!policy.super_permitted)
|
||||
bits &= ~special_bits;
|
||||
*out_mode = bits;
|
||||
* caller logs a failed chmod rather than silently masking the bits here. */
|
||||
*out_mode = source_mode & (mode_t)(S_ISUID | S_ISGID | S_ISVTX | 0777);
|
||||
return true;
|
||||
}
|
||||
if (policy.executability) {
|
||||
@@ -236,11 +227,8 @@ bool metadata_mode_for_policy(mode_t source_mode, mode_t current_mode, FileAttrP
|
||||
* execute); otherwise clear every execute bit. This runs on the
|
||||
* destination-derived base (pre-existing dest mode, or source&~umask for a
|
||||
* new file), and leaves the special bits untouched. --perms wins when both
|
||||
* are set (handled above). The destination's own special bits survive
|
||||
* unless super-user activities are forbidden. */
|
||||
mode_t base = current_mode & (mode_t)(special_bits | 0777);
|
||||
if (!policy.super_permitted)
|
||||
base &= ~special_bits;
|
||||
* are set (handled above). */
|
||||
mode_t base = current_mode & (mode_t)(S_ISUID | S_ISGID | S_ISVTX | 0777);
|
||||
if (source_mode & 0111)
|
||||
*out_mode = base | ((base & 0444) >> 2);
|
||||
else
|
||||
@@ -252,13 +240,12 @@ bool metadata_mode_for_policy(mode_t source_mode, mode_t current_mode, FileAttrP
|
||||
}
|
||||
|
||||
FileAttrPolicy file_attr_policy_from_config(const Config* config) {
|
||||
FileAttrPolicy policy = {0};
|
||||
FileAttrPolicy policy = {false, false, false, false};
|
||||
if (config) {
|
||||
policy.perms = config->preserve_perms;
|
||||
policy.times = config->preserve_times;
|
||||
policy.atimes = config->preserve_atimes;
|
||||
policy.executability = config->use_executability;
|
||||
policy.super_permitted = privilege_super_mode_permitted(config->super_mode);
|
||||
}
|
||||
return policy;
|
||||
}
|
||||
@@ -327,8 +314,6 @@ bool file_restore_symlink_metadata(const char* path, const FileMetadata* metadat
|
||||
transfer never fails over it. */
|
||||
if (policy.perms) {
|
||||
mode_t link_mode = metadata->mode & (mode_t)(S_ISUID | S_ISGID | S_ISVTX | 0777);
|
||||
if (!policy.super_permitted)
|
||||
link_mode &= ~(mode_t)(S_ISUID | S_ISGID | S_ISVTX);
|
||||
if (fchmodat(parent_fd, leaf, link_mode, AT_SYMLINK_NOFOLLOW) != 0 && errno != EOPNOTSUPP &&
|
||||
errno != ENOTSUP && errno != ENOSYS) {
|
||||
log_message(LOG_LEVEL_DEBUG, "Could not set symlink mode on %s: %s", path, strerror(errno));
|
||||
|
||||
@@ -34,27 +34,21 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
|
||||
context->synced_dirs = NULL;
|
||||
context->plan_dirs = NULL;
|
||||
context->missing_args = NULL;
|
||||
context->per_dir_rules = NULL;
|
||||
context->scan_had_io_error = false;
|
||||
context->remove_source_files = NULL;
|
||||
context->early_delete = false;
|
||||
context->prescan_chunks = NULL;
|
||||
context->delete_plans = NULL;
|
||||
context->delete_suppressed = false;
|
||||
context->scan_stopped_early = false;
|
||||
context->total_files = 0;
|
||||
context->progress_bytes = 0;
|
||||
context->total_bytes = 0;
|
||||
memset(&context->stats, 0, sizeof(context->stats));
|
||||
context->sender_done = false;
|
||||
atomic_init(&context->cancelled, false);
|
||||
protocol_session_init(&context->allocation_session, -1, -1);
|
||||
protocol_session_set_max_alloc(&context->allocation_session, config->max_alloc);
|
||||
context->dir_entries = NULL;
|
||||
context->dir_entries_mutex_init = false;
|
||||
atomic_init(&context->dir_count, 0);
|
||||
context->delete_limit = false;
|
||||
context->partial = false;
|
||||
int init = 0;
|
||||
if (config->use_metadata) {
|
||||
context->dir_entries = array_list_create(file_destroy);
|
||||
@@ -89,13 +83,11 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
|
||||
return context;
|
||||
|
||||
fail:
|
||||
log_message(LOG_LEVEL_ERROR, "%s", "Error initializing synchronization objects");
|
||||
log_perror("Error initializing synchronization objects");
|
||||
if (context->dir_entries_mutex_init)
|
||||
mtx_destroy(&context->dir_entries_mutex);
|
||||
if (context->dir_entries)
|
||||
array_list_delete(context->dir_entries);
|
||||
if (init >= 7)
|
||||
mtx_destroy(&context->mutex_progress);
|
||||
if (init >= 6)
|
||||
cnd_destroy(&context->condition_not_empty_loader);
|
||||
if (init >= 5)
|
||||
@@ -197,8 +189,6 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
|
||||
if (context->manifest) {
|
||||
array_list_delete(context->manifest);
|
||||
}
|
||||
if (context->prescan_chunks)
|
||||
array_list_delete(context->prescan_chunks);
|
||||
if (context->delete_plans)
|
||||
delete_plan_sender_destroy(context->delete_plans);
|
||||
if (context->excluded_paths)
|
||||
@@ -211,8 +201,6 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
|
||||
array_list_delete(context->plan_dirs);
|
||||
if (context->missing_args)
|
||||
array_list_delete(context->missing_args);
|
||||
if (context->per_dir_rules)
|
||||
filter_rule_list_free(context->per_dir_rules);
|
||||
if (context->remove_source_files)
|
||||
array_list_delete(context->remove_source_files);
|
||||
if (context->dir_entries)
|
||||
|
||||
@@ -9,7 +9,6 @@
|
||||
#include "config.h"
|
||||
#include "delete_plan.h"
|
||||
#include "file.h"
|
||||
#include "format.h"
|
||||
#include "protocol.h"
|
||||
#include "queue.h"
|
||||
#include "stop_condition.h"
|
||||
@@ -67,12 +66,6 @@ typedef struct {
|
||||
them in the manifest frame's third section and the receiver deletes each as
|
||||
an explicit request. */
|
||||
ArrayList* missing_args;
|
||||
/* Per-directory filter rules the source scan compiled (protocol 2.30.0),
|
||||
sent with the delete manifest/plan config so the receiver can re-derive the
|
||||
per-directory protect/risk set. NULL when --delete is off. Populated by
|
||||
the scanner (parallel workers append under mutex_scanner) or the early
|
||||
pre-scan. */
|
||||
FilterRuleList* per_dir_rules;
|
||||
/* A source I/O error (unreadable directory) was recorded during the scan.
|
||||
Set by the pre-scan (before the threads start) or by the scanner thread
|
||||
under mutex_scanner; the caller turns it into a non-zero exit when
|
||||
@@ -84,34 +77,15 @@ typedef struct {
|
||||
path-only pre-scan on the calling thread and the pipeline scanner must not
|
||||
append to it. Set once before the worker threads start. */
|
||||
bool early_delete;
|
||||
/* --delete-before: the path-only pre-scan that built the early keep-set,
|
||||
retained as the pipeline's file list (owning Chunk*; consumed and NULLed by
|
||||
the scanner thread) so the data pass replays rsync's single file list
|
||||
instead of re-reading the source. NULL in every other mode, where the
|
||||
scanner thread scans normally. Set once before the worker threads start
|
||||
and freed with the context. */
|
||||
ArrayList* prescan_chunks;
|
||||
/* Non-NULL for --delete-during/--delete-delay: the per-directory plan set
|
||||
prebuilt by the path-only pre-scan on the calling thread. The sender
|
||||
thread transmits the root plan before any data and the remaining plans
|
||||
alongside the chunks. Set once before the worker threads start. */
|
||||
DeletePlanSender* delete_plans;
|
||||
/* A scan I/O error without --ignore-errors suppressed deletion: the prebuilt
|
||||
keep-set/plans were dropped, and the streaming scanner must not build a
|
||||
fresh manifest or re-send the per-directory plans. Set once before the
|
||||
worker threads start. */
|
||||
bool delete_suppressed;
|
||||
mtx_t mutex_progress;
|
||||
int total_files;
|
||||
unsigned long long progress_bytes;
|
||||
unsigned long long total_bytes;
|
||||
/* Per-type flist / transferred accounting for the rsync --stats breakdown and
|
||||
the progress `to-chk` denominator. Owned by the sender thread: it is the
|
||||
only writer (the entry/transfer notes in send_chunks_multithreaded) and it
|
||||
reads the totals in its completion tail, so no lock is needed. This is NOT
|
||||
guarded by mutex_progress (which covers total_files/progress_bytes/
|
||||
total_bytes/sender_done). */
|
||||
TransferStats stats;
|
||||
bool sender_done;
|
||||
atomic_bool cancelled;
|
||||
ProtocolSession allocation_session;
|
||||
@@ -132,19 +106,10 @@ typedef struct {
|
||||
ArrayList* dir_entries;
|
||||
mtx_t dir_entries_mutex;
|
||||
bool dir_entries_mutex_init;
|
||||
/* --stats directory accounting for a `-r` scan (no directory metadata):
|
||||
shared by the parallel scanner workers, read by the sender thread once the
|
||||
scanner is done. See ScannerOptions.dir_count. */
|
||||
atomic_ullong dir_count;
|
||||
/* Set by the sender thread when the receiver reported a --max-delete-capped
|
||||
deletion (STATUS_DELETE_LIMIT): the transfer succeeded and the process must
|
||||
exit 25 like rsync. Read by the caller after the sender thread is joined. */
|
||||
bool delete_limit;
|
||||
/* Set by the sender thread when the receiver reported STATUS_PARTIAL (a
|
||||
per-entry receiver failure that did not abort the stream): the transfer
|
||||
otherwise succeeded, successfully stored --remove-source-files sources were
|
||||
removed, and the process must exit 23 like rsync. Read after join. */
|
||||
bool partial;
|
||||
} PipelineContextSender;
|
||||
|
||||
/* `config` is borrowed and must outlive the context: destroy does NOT free it,
|
||||
|
||||
+91
-303
@@ -38,150 +38,6 @@ static atomic_ullong io_bytes_read = 0;
|
||||
|
||||
static unsigned long long global_bwlimit(void);
|
||||
|
||||
/* Runtime whole-file receive bound (see protocol.h). Resolved once; an
|
||||
* override can only LOWER the ceiling, never raise it above the protocol
|
||||
* constant, so the wire/security bound is unchanged. A parse failure or a
|
||||
* non-positive value leaves the default in place. */
|
||||
unsigned long long protocol_whole_file_receive_limit(void) {
|
||||
static atomic_ullong cached = 0;
|
||||
unsigned long long value = atomic_load_explicit(&cached, memory_order_relaxed);
|
||||
if (value != 0)
|
||||
return value;
|
||||
value = MAX_RECEIVE_WHOLE_FILE_SIZE;
|
||||
const char* env = getenv("FASTSYNC_MAX_WHOLE_FILE_SIZE");
|
||||
if (env && env[0] != '\0') {
|
||||
char* end = NULL;
|
||||
unsigned long long parsed = strtoull(env, &end, 10);
|
||||
if (end && *end == '\0' && parsed > 0 && parsed < value)
|
||||
value = parsed;
|
||||
}
|
||||
atomic_store_explicit(&cached, value, memory_order_relaxed);
|
||||
return value;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------------- *
|
||||
* Transport vtable implementations.
|
||||
*
|
||||
* Each op performs exactly one transfer attempt. WANT_READ/WANT_WRITE and an
|
||||
* EINTR-interrupted syscall are reported as PROTOCOL_IO_RETRY (with
|
||||
* *wait_events set to the poll event the caller must wait on); a clean peer
|
||||
* close is PROTOCOL_IO_CLOSED and anything else is PROTOCOL_IO_ERROR. This
|
||||
* keeps every WANT_READ/WANT_WRITE and EINTR retry exactly where it was before
|
||||
* the vtable was introduced, just moved behind the function pointer.
|
||||
* ------------------------------------------------------------------------- */
|
||||
|
||||
static ssize_t plain_io_send(ProtocolSession* session, const void* data, size_t size,
|
||||
short* wait_events) {
|
||||
ssize_t written = write(session->write_fd, data, size);
|
||||
if (written < 0) {
|
||||
if (errno == EINTR)
|
||||
return PROTOCOL_IO_RETRY;
|
||||
return PROTOCOL_IO_ERROR;
|
||||
}
|
||||
if (written == 0)
|
||||
return PROTOCOL_IO_ERROR;
|
||||
*wait_events = POLLOUT;
|
||||
return written;
|
||||
}
|
||||
|
||||
static ssize_t plain_io_recv(ProtocolSession* session, void* data, size_t size,
|
||||
short* wait_events) {
|
||||
ssize_t received = read(session->read_fd, data, size);
|
||||
if (received < 0) {
|
||||
if (errno == EINTR)
|
||||
return PROTOCOL_IO_RETRY;
|
||||
return PROTOCOL_IO_ERROR;
|
||||
}
|
||||
if (received == 0)
|
||||
return PROTOCOL_IO_CLOSED;
|
||||
*wait_events = POLLIN;
|
||||
return received;
|
||||
}
|
||||
|
||||
static bool plain_io_has_pending(const ProtocolSession* session) {
|
||||
(void)session;
|
||||
return false;
|
||||
}
|
||||
|
||||
static ssize_t tls_io_send(ProtocolSession* session, const void* data, size_t size,
|
||||
short* wait_events) {
|
||||
/* SSL_write takes an int length; clamp a >INT_MAX request into chunks so the
|
||||
* size_t downcast can never truncate into a negative/partial write. */
|
||||
size_t chunk = size > (size_t)INT_MAX ? (size_t)INT_MAX : size;
|
||||
ssize_t written = SSL_write(session->ssl, data, (int)chunk);
|
||||
if (written <= 0) {
|
||||
int ssl_err = SSL_get_error(session->ssl, (int)written);
|
||||
if (ssl_err == SSL_ERROR_WANT_WRITE) {
|
||||
*wait_events = POLLOUT;
|
||||
return PROTOCOL_IO_RETRY;
|
||||
}
|
||||
if (ssl_err == SSL_ERROR_WANT_READ) {
|
||||
*wait_events = POLLIN;
|
||||
return PROTOCOL_IO_RETRY;
|
||||
}
|
||||
/* A signal (e.g. Ctrl-C) interrupts the blocking TLS write: retry so the
|
||||
* send loop can observe the abort flag at the next checkpoint. Only an
|
||||
* actual negative return is an interrupted syscall; a 0-byte SSL_write is
|
||||
* not a valid EINTR retry. */
|
||||
if (written < 0 && ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
|
||||
return PROTOCOL_IO_RETRY;
|
||||
return PROTOCOL_IO_ERROR;
|
||||
}
|
||||
*wait_events = POLLOUT;
|
||||
return written;
|
||||
}
|
||||
|
||||
static ssize_t tls_io_recv(ProtocolSession* session, void* data, size_t size, short* wait_events) {
|
||||
/* SSL_read takes an int length; clamp a >INT_MAX request into chunks
|
||||
* (mirrors the send path) so the size_t downcast can never truncate into a
|
||||
* negative/partial read. */
|
||||
size_t chunk = size > (size_t)INT_MAX ? (size_t)INT_MAX : size;
|
||||
ssize_t received = SSL_read(session->ssl, data, (int)chunk);
|
||||
if (received <= 0) {
|
||||
int ssl_err = SSL_get_error(session->ssl, (int)received);
|
||||
if (ssl_err == SSL_ERROR_WANT_WRITE) {
|
||||
*wait_events = POLLOUT;
|
||||
return PROTOCOL_IO_RETRY;
|
||||
}
|
||||
if (ssl_err == SSL_ERROR_WANT_READ) {
|
||||
*wait_events = POLLIN;
|
||||
return PROTOCOL_IO_RETRY;
|
||||
}
|
||||
/* A signal interrupts the blocking TLS read: retry (mirrors the send path)
|
||||
* so the loop reaches its next abort/deadline checkpoint. Only an actual
|
||||
* negative return is an interrupted syscall: a 0-byte SSL_read is an
|
||||
* unexpected EOF (the peer closed without close_notify), which OpenSSL also
|
||||
* reports as SSL_ERROR_SYSCALL with errno possibly still EINTR from an
|
||||
* earlier interrupted poll/read. Retrying that would busy-spin the
|
||||
* status-read loop until its deadline, so classify it as closed instead. */
|
||||
if (received < 0 && ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
|
||||
return PROTOCOL_IO_RETRY;
|
||||
/* A zero-length SSL_read is the peer's clean close_notify (or EOF without
|
||||
* one); report it distinctly so the caller can log it as a close. */
|
||||
if (received == 0)
|
||||
return PROTOCOL_IO_CLOSED;
|
||||
return PROTOCOL_IO_ERROR;
|
||||
}
|
||||
*wait_events = POLLIN;
|
||||
return received;
|
||||
}
|
||||
|
||||
static bool tls_io_has_pending(const ProtocolSession* session) {
|
||||
return session->ssl != NULL && SSL_pending(session->ssl) > 0;
|
||||
}
|
||||
|
||||
static const ProtocolIoOps plain_io_ops = {
|
||||
.send = plain_io_send,
|
||||
.recv = plain_io_recv,
|
||||
.has_pending = plain_io_has_pending,
|
||||
};
|
||||
|
||||
static const ProtocolIoOps tls_io_ops = {
|
||||
.send = tls_io_send,
|
||||
.recv = tls_io_recv,
|
||||
.has_pending = tls_io_has_pending,
|
||||
};
|
||||
|
||||
static bool protocol_reserve_memory(ProtocolSession* session, size_t charge) {
|
||||
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
|
||||
while (true) {
|
||||
@@ -223,7 +79,6 @@ void io_set_fds(int read_fd, int write_fd) {
|
||||
legacy_io_session.read_fd = read_fd;
|
||||
legacy_io_session.write_fd = write_fd;
|
||||
legacy_io_session.ssl = NULL;
|
||||
legacy_io_session.ops = &plain_io_ops;
|
||||
legacy_io_session.eight_bit_output = false;
|
||||
atomic_store(&legacy_io_session.total_allocated_bytes, 0);
|
||||
legacy_io_session.max_alloc = DEFAULT_MAX_ALLOC;
|
||||
@@ -236,7 +91,6 @@ void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd)
|
||||
memset(session, 0, sizeof(*session));
|
||||
session->read_fd = read_fd;
|
||||
session->write_fd = write_fd;
|
||||
session->ops = &plain_io_ops;
|
||||
session->max_alloc = DEFAULT_MAX_ALLOC;
|
||||
session->io_timeout_sec = RECEIVE_TIMEOUT_SEC;
|
||||
atomic_init(&session->total_allocated_bytes, 0);
|
||||
@@ -304,12 +158,8 @@ void protocol_session_unbind(void) {
|
||||
}
|
||||
|
||||
void protocol_session_set_ssl(ProtocolSession* session, SSL* ssl) {
|
||||
if (!session)
|
||||
return;
|
||||
session->ssl = ssl;
|
||||
/* Select the transport dispatch once, here, instead of branching on the SSL
|
||||
* pointer inside every I/O loop. */
|
||||
session->ops = ssl ? &tls_io_ops : &plain_io_ops;
|
||||
if (session)
|
||||
session->ssl = ssl;
|
||||
}
|
||||
|
||||
static void bw_mutex_init(void) {
|
||||
@@ -333,28 +183,12 @@ void io_set_bwlimit(unsigned long long bytes_per_sec) {
|
||||
mtx_unlock(&bw_mutex);
|
||||
}
|
||||
|
||||
unsigned long long io_get_bwlimit(void) {
|
||||
return global_bwlimit();
|
||||
}
|
||||
|
||||
/* rsync's throttle (io.c sleep_for_bwlimit) sleeps once its unslept debt
|
||||
* reaches ~100 ms of bandwidth, so its effective initial burst is about 0.1 s
|
||||
* worth of bytes, not a full second. FastSync models the same with a token
|
||||
* bucket whose capacity is bwlimit/10, so a throttled run paces like rsync
|
||||
* instead of sending a full second's worth up front. */
|
||||
static long long bw_burst_capacity(unsigned long long bwlimit) {
|
||||
if (bwlimit == 0)
|
||||
return 0;
|
||||
long long burst = (long long)(bwlimit / 10);
|
||||
return burst > 0 ? burst : 1;
|
||||
}
|
||||
|
||||
void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long bytes_per_sec) {
|
||||
if (!session)
|
||||
return;
|
||||
session->bwlimit =
|
||||
bytes_per_sec > (unsigned long long)LLONG_MAX ? (unsigned long long)LLONG_MAX : bytes_per_sec;
|
||||
session->bw_tokens = bw_burst_capacity(session->bwlimit);
|
||||
session->bw_tokens = (long long)session->bwlimit;
|
||||
struct timespec now;
|
||||
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||
session->bw_last_refill_sec = now.tv_sec;
|
||||
@@ -388,9 +222,8 @@ static void bw_throttle_session(ProtocolSession* session, size_t bytes_written)
|
||||
|
||||
long long tokens_to_add = (long long)((double)session->bwlimit * elapsed_ns / 1000000000.0);
|
||||
session->bw_tokens += tokens_to_add;
|
||||
long long burst = bw_burst_capacity(session->bwlimit);
|
||||
if (session->bw_tokens > burst)
|
||||
session->bw_tokens = burst;
|
||||
if (session->bw_tokens > (long long)session->bwlimit)
|
||||
session->bw_tokens = (long long)session->bwlimit;
|
||||
|
||||
session->bw_tokens -= bytes_written;
|
||||
|
||||
@@ -401,11 +234,7 @@ static void bw_throttle_session(ProtocolSession* session, size_t bytes_written)
|
||||
poll(NULL, 0, (int)(deficit_us / 1000));
|
||||
else
|
||||
usleep((useconds_t)deficit_us);
|
||||
/* Reset the bucket AFTER the sleep: crediting the sleep duration as elapsed
|
||||
refill time would cancel half the throttle (the next call would see the
|
||||
whole sleep as refill and immediately grant a fresh burst). */
|
||||
session->bw_tokens = 0;
|
||||
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||
session->bw_last_refill_sec = now.tv_sec;
|
||||
session->bw_last_refill_nsec = now.tv_nsec;
|
||||
}
|
||||
@@ -420,20 +249,6 @@ SSL* io_get_ssl(void) {
|
||||
return io_ssl;
|
||||
}
|
||||
|
||||
SSL* protocol_current_ssl(void) {
|
||||
/* The bound session is the authoritative transport for a worker thread: it
|
||||
* was explicitly handed to protocol_session_bind() and carries its own SSL,
|
||||
* whereas io_ssl is thread-local and NULL in a thread that never performed
|
||||
* the handshake. Only a session whose selected dispatch is TLS may supply
|
||||
* the SSL: a bound plaintext session has ssl == NULL and must not shadow a
|
||||
* live thread-local io_ssl, or file_send.c would take the raw sendfile(2)
|
||||
* path on a socket this thread is driving with TLS. With no TLS session
|
||||
* bound (plaintext session, or the fd-shim path), fall back to io_ssl. */
|
||||
if (bound_session && bound_session->ops == &tls_io_ops && bound_session->ssl)
|
||||
return bound_session->ssl;
|
||||
return io_ssl;
|
||||
}
|
||||
|
||||
unsigned long long protocol_bytes_written(void) {
|
||||
return atomic_load(&io_bytes_written);
|
||||
}
|
||||
@@ -462,21 +277,9 @@ static ProtocolSession* legacy_session(int read_fd, int write_fd) {
|
||||
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
|
||||
}
|
||||
legacy_io_session.ssl = io_ssl;
|
||||
legacy_io_session.ops = io_ssl ? &tls_io_ops : &plain_io_ops;
|
||||
return &legacy_io_session;
|
||||
}
|
||||
|
||||
/* Pace an out-of-band write that bypassed protocol_send_n_data (the plaintext
|
||||
* sendfile fast path). The bound/legacy session is resolved exactly as the
|
||||
* preceding send_n_data(fd, ...) resolved it, so the same token-bucket state is
|
||||
* throttled and the TLS and plaintext transports share identical --bwlimit
|
||||
* semantics. Passing the wire fd (rather than -1) is essential: the sendfile
|
||||
* send left legacy_io_session.write_fd bound to it, so resolving with -1 would
|
||||
* mismatch, re-initialize the session and hand out a second first-call burst. */
|
||||
void protocol_throttle_bytes(int file_descriptor, size_t bytes) {
|
||||
bw_throttle_session(legacy_session(-1, file_descriptor), bytes);
|
||||
}
|
||||
|
||||
bool send_n_data(int file_descriptor, const void* data, size_t data_size) {
|
||||
return protocol_send_n_data(legacy_session(-1, file_descriptor), data, data_size);
|
||||
}
|
||||
@@ -500,9 +303,8 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
|
||||
if (!data && data_size != 0)
|
||||
return false;
|
||||
log_debug_message(LOG_DEBUG_IO, " Sending n Data: %zu", data_size);
|
||||
if (!session || !session->ops)
|
||||
if (!session)
|
||||
return false;
|
||||
const ProtocolIoOps* ops = session->ops;
|
||||
/* A non-positive session timeout disables the deadline entirely (rsync's
|
||||
* --timeout=0 default); poll then blocks until the socket becomes writable. */
|
||||
int timeout_sec = session->io_timeout_sec > 0 ? session->io_timeout_sec : 0;
|
||||
@@ -528,18 +330,38 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
|
||||
continue;
|
||||
if (pfd.revents & (POLLERR | POLLNVAL))
|
||||
return false;
|
||||
ssize_t bytes_send =
|
||||
ops->send(session, (const char*)data + total_bytes_send, chunk, &wait_events);
|
||||
if (bytes_send == PROTOCOL_IO_RETRY)
|
||||
continue;
|
||||
ssize_t bytes_send;
|
||||
if (session->ssl) {
|
||||
/* SSL_write takes an int length; clamp a >INT_MAX request into chunks so
|
||||
* the size_t downcast can never truncate into a negative/partial write. */
|
||||
size_t ssl_chunk = chunk > (size_t)INT_MAX ? (size_t)INT_MAX : chunk;
|
||||
bytes_send = SSL_write(session->ssl, (const char*)data + total_bytes_send, (int)ssl_chunk);
|
||||
} else {
|
||||
bytes_send = write(fd, (const char*)data + total_bytes_send, chunk);
|
||||
}
|
||||
if (bytes_send <= 0) {
|
||||
if (session->ssl) {
|
||||
int ssl_err = SSL_get_error(session->ssl, (int)bytes_send);
|
||||
if (ssl_err == SSL_ERROR_WANT_WRITE || ssl_err == SSL_ERROR_WANT_READ) {
|
||||
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
|
||||
continue;
|
||||
}
|
||||
/* A signal (e.g. Ctrl-C) interrupts the blocking TLS write: retry so
|
||||
the send loop can observe the abort flag at the next checkpoint. */
|
||||
if (ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
|
||||
continue;
|
||||
} else if (errno == EINTR) {
|
||||
continue;
|
||||
}
|
||||
log_message(LOG_LEVEL_ERROR, "Could not send data");
|
||||
return false;
|
||||
}
|
||||
bw_throttle_session(session, (size_t)bytes_send);
|
||||
total_bytes_send += bytes_send;
|
||||
if (session->ssl)
|
||||
wait_events = POLLOUT;
|
||||
}
|
||||
log_debug_message(LOG_DEBUG_IO, " Send n Data: %zd", total_bytes_send);
|
||||
log_debug_message(LOG_DEBUG_IO, " Send n Data: %zu", total_bytes_send);
|
||||
atomic_fetch_add(&io_bytes_written, (unsigned long long)total_bytes_send);
|
||||
return true;
|
||||
}
|
||||
@@ -570,15 +392,14 @@ bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_s
|
||||
static bool protocol_receive_n_data_until(ProtocolSession* session, void* data, size_t data_size,
|
||||
const struct timespec* deadline) {
|
||||
log_debug_message(LOG_DEBUG_IO, " Receiving n Data: %zu", data_size);
|
||||
if (!session || !session->ops)
|
||||
if (!session)
|
||||
return false;
|
||||
const ProtocolIoOps* ops = session->ops;
|
||||
int fd = session->read_fd;
|
||||
|
||||
size_t total_bytes_received = 0;
|
||||
short wait_events = POLLIN;
|
||||
while (total_bytes_received < data_size) {
|
||||
if (!ops->has_pending(session)) {
|
||||
if (!session->ssl || SSL_pending(session->ssl) == 0) {
|
||||
struct pollfd pfd = {.fd = fd, .events = wait_events};
|
||||
/* A NULL deadline means "wait indefinitely" (timeout disabled). */
|
||||
int poll_result = poll(&pfd, 1, deadline ? deadline_remaining_ms(deadline) : -1);
|
||||
@@ -596,19 +417,37 @@ static bool protocol_receive_n_data_until(ProtocolSession* session, void* data,
|
||||
return false;
|
||||
}
|
||||
|
||||
ssize_t bytes_received = ops->recv(session, (char*)data + total_bytes_received,
|
||||
data_size - total_bytes_received, &wait_events);
|
||||
if (bytes_received == PROTOCOL_IO_RETRY)
|
||||
continue;
|
||||
if (bytes_received == PROTOCOL_IO_CLOSED) {
|
||||
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
|
||||
return false;
|
||||
}
|
||||
ssize_t bytes_received;
|
||||
if (session->ssl)
|
||||
bytes_received = SSL_read(session->ssl, (char*)data + total_bytes_received,
|
||||
data_size - total_bytes_received);
|
||||
else
|
||||
bytes_received =
|
||||
read(fd, (char*)data + total_bytes_received, data_size - total_bytes_received);
|
||||
if (bytes_received <= 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "Could not receive bytes");
|
||||
if (session->ssl) {
|
||||
int ssl_err = SSL_get_error(session->ssl, (int)bytes_received);
|
||||
if (ssl_err == SSL_ERROR_WANT_WRITE || ssl_err == SSL_ERROR_WANT_READ) {
|
||||
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
|
||||
continue;
|
||||
}
|
||||
/* A signal interrupts the blocking TLS read: retry (mirrors the send
|
||||
path and protocol_read_status_until) so the loop reaches its next
|
||||
abort/deadline checkpoint instead of failing spuriously. */
|
||||
if (ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
|
||||
continue;
|
||||
} else if (errno == EINTR) {
|
||||
continue;
|
||||
}
|
||||
if (bytes_received == 0)
|
||||
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
|
||||
else
|
||||
log_message(LOG_LEVEL_ERROR, "Could not receive bytes");
|
||||
return false;
|
||||
}
|
||||
total_bytes_received += (size_t)bytes_received;
|
||||
if (session->ssl)
|
||||
wait_events = POLLIN;
|
||||
}
|
||||
log_debug_message(LOG_DEBUG_IO, " Received n Data: %zu", total_bytes_received);
|
||||
atomic_fetch_add(&io_bytes_read, (unsigned long long)total_bytes_received);
|
||||
@@ -685,24 +524,11 @@ static const char* status_to_string(Status status) {
|
||||
return "DELETE_LIMIT";
|
||||
case STATUS_DEST_INFO:
|
||||
return "DEST_INFO";
|
||||
case STATUS_CLIENT_MSG:
|
||||
return "CLIENT_MSG";
|
||||
case STATUS_PARTIAL:
|
||||
return "PARTIAL";
|
||||
default:
|
||||
return "UNKNOWN";
|
||||
}
|
||||
}
|
||||
|
||||
/* Reject a raw wire status outside the known enum range before it is handed to
|
||||
* callers, so an unknown/corrupt frame fails as a protocol error instead of
|
||||
* being silently interpreted as an unexpected-but-valid verdict. STATUS_OK is
|
||||
* the first enumerator and STATUS_PARTIAL the last, so the range check accepts
|
||||
* every status the protocol defines. */
|
||||
static bool status_is_valid(Status status) {
|
||||
return status >= STATUS_OK && status <= STATUS_PARTIAL;
|
||||
}
|
||||
|
||||
/* Shared string send/receive implementation. `redact` selects whether the
|
||||
* payload body is written to the LOG_DEBUG_PROTO debug log: daemon auth material
|
||||
* (the username and the proof/signature fields) sets it so a --verbose log never
|
||||
@@ -786,15 +612,21 @@ bool protocol_send_data(ProtocolSession* session, const Data* data) {
|
||||
return false;
|
||||
if (!protocol_send_n_data(session, data->data, data_size))
|
||||
return false;
|
||||
log_debug_message(LOG_DEBUG_PROTO, "Send %llu data", data_size);
|
||||
log_debug_message(LOG_DEBUG_PROTO, "Send %lld data", data_size);
|
||||
return true;
|
||||
}
|
||||
|
||||
Data* protocol_receive_data_alloc(ProtocolSession* session, unsigned long long size) {
|
||||
Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long maximum_size) {
|
||||
if (!session)
|
||||
return NULL;
|
||||
if (size > MAX_DATA_PAYLOAD_SIZE)
|
||||
unsigned long long size = 0;
|
||||
if (!protocol_receive_n_data(session, &size, sizeof(unsigned long long)))
|
||||
return NULL;
|
||||
if (size > MAX_DATA_PAYLOAD_SIZE || size > maximum_size) {
|
||||
log_message(LOG_LEVEL_ERROR, "Data size %llu exceeds maximum %llu", size,
|
||||
(unsigned long long)MAX_DATA_PAYLOAD_SIZE);
|
||||
return NULL;
|
||||
}
|
||||
if (size > SIZE_MAX)
|
||||
return NULL;
|
||||
size_t allocation_size = size == 0 ? 1 : (size_t)size;
|
||||
@@ -809,6 +641,12 @@ Data* protocol_receive_data_alloc(ProtocolSession* session, unsigned long long s
|
||||
protocol_release_memory_for_session(session, allocation_size);
|
||||
return NULL;
|
||||
}
|
||||
if (!protocol_receive_n_data(session, data, (size_t)size)) {
|
||||
free(data);
|
||||
protocol_release_memory_for_session(session, allocation_size);
|
||||
return NULL;
|
||||
}
|
||||
log_debug_message(LOG_DEBUG_PROTO, "Received %lld data", size);
|
||||
Data* result = data_create(data, (size_t)size);
|
||||
if (!result) {
|
||||
protocol_release_memory_for_session(session, allocation_size);
|
||||
@@ -819,32 +657,6 @@ Data* protocol_receive_data_alloc(ProtocolSession* session, unsigned long long s
|
||||
return result;
|
||||
}
|
||||
|
||||
Data* protocol_receive_data_body(ProtocolSession* session, unsigned long long size) {
|
||||
Data* result = protocol_receive_data_alloc(session, size);
|
||||
if (!result)
|
||||
return NULL;
|
||||
if (!protocol_receive_n_data(session, result->data, (size_t)size)) {
|
||||
data_destroy(result);
|
||||
return NULL;
|
||||
}
|
||||
log_debug_message(LOG_DEBUG_PROTO, "Received %llu data", size);
|
||||
return result;
|
||||
}
|
||||
|
||||
Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long maximum_size) {
|
||||
if (!session)
|
||||
return NULL;
|
||||
unsigned long long size = 0;
|
||||
if (!protocol_receive_n_data(session, &size, sizeof(unsigned long long)))
|
||||
return NULL;
|
||||
if (size > MAX_DATA_PAYLOAD_SIZE || size > maximum_size) {
|
||||
log_message(LOG_LEVEL_ERROR, "Data size %llu exceeds maximum %llu", size,
|
||||
(unsigned long long)MAX_DATA_PAYLOAD_SIZE);
|
||||
return NULL;
|
||||
}
|
||||
return protocol_receive_data_body(session, size);
|
||||
}
|
||||
|
||||
bool protocol_send_int(ProtocolSession* session, int data) {
|
||||
if (!protocol_send_n_data(session, &data, sizeof(int)))
|
||||
return false;
|
||||
@@ -944,10 +756,6 @@ bool protocol_receive_status(ProtocolSession* session, Status* status) {
|
||||
}
|
||||
if (!protocol_receive_n_data_until(session, status, sizeof(Status), deadline_ptr))
|
||||
return false;
|
||||
if (!status_is_valid(*status)) {
|
||||
log_message(LOG_LEVEL_ERROR, "Received unknown protocol status %d", *status);
|
||||
return false;
|
||||
}
|
||||
if (!protocol_capture_error_detail(session, status, deadline_ptr, NULL))
|
||||
return false;
|
||||
log_debug_message(LOG_DEBUG_PROTO, "Received Status: %s", status_to_string(*status));
|
||||
@@ -969,10 +777,6 @@ bool protocol_receive_status_timed(ProtocolSession* session, Status* status, int
|
||||
deadline.tv_sec += timeout_sec;
|
||||
if (!protocol_receive_n_data_until(session, status, sizeof(Status), &deadline))
|
||||
return false;
|
||||
if (!status_is_valid(*status)) {
|
||||
log_message(LOG_LEVEL_ERROR, "Received unknown protocol status %d", *status);
|
||||
return false;
|
||||
}
|
||||
if (!protocol_capture_error_detail(session, status, &deadline, NULL))
|
||||
return false;
|
||||
log_debug_message(LOG_DEBUG_PROTO, "Received Status: %s", status_to_string(*status));
|
||||
@@ -986,14 +790,11 @@ bool protocol_receive_status_timed(ProtocolSession* session, Status* status, int
|
||||
* reply across a frame boundary. Returns false on timeout/EOF/error. */
|
||||
static bool protocol_read_status_until(ProtocolSession* session, Status* status,
|
||||
const struct timespec* deadline) {
|
||||
if (!session || !session->ops)
|
||||
return false;
|
||||
const ProtocolIoOps* ops = session->ops;
|
||||
Status received = STATUS_ERROR;
|
||||
size_t got = 0;
|
||||
short wait_events = POLLIN;
|
||||
while (got < sizeof(Status)) {
|
||||
if (!ops->has_pending(session)) {
|
||||
if (!session->ssl || SSL_pending(session->ssl) == 0) {
|
||||
int remaining_ms = deadline ? deadline_remaining_ms(deadline) : -1;
|
||||
if (remaining_ms == 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "Receive timeout while reading status");
|
||||
@@ -1013,11 +814,21 @@ static bool protocol_read_status_until(ProtocolSession* session, Status* status,
|
||||
if (pfd.revents & (POLLERR | POLLNVAL))
|
||||
return false;
|
||||
}
|
||||
ssize_t bytes_received =
|
||||
ops->recv(session, (char*)&received + got, sizeof(Status) - got, &wait_events);
|
||||
if (bytes_received == PROTOCOL_IO_RETRY)
|
||||
continue;
|
||||
ssize_t bytes_received;
|
||||
if (session->ssl)
|
||||
bytes_received = SSL_read(session->ssl, (char*)&received + got, sizeof(Status) - got);
|
||||
else
|
||||
bytes_received = read(session->read_fd, (char*)&received + got, sizeof(Status) - got);
|
||||
if (bytes_received <= 0) {
|
||||
if (session->ssl) {
|
||||
int ssl_err = SSL_get_error(session->ssl, (int)bytes_received);
|
||||
if (ssl_err == SSL_ERROR_WANT_READ || ssl_err == SSL_ERROR_WANT_WRITE) {
|
||||
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (bytes_received < 0 && errno == EINTR)
|
||||
continue;
|
||||
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving status");
|
||||
return false;
|
||||
}
|
||||
@@ -1046,7 +857,7 @@ bool protocol_receive_status_keepalive(ProtocolSession* session, Status* status,
|
||||
while (true) {
|
||||
if (abort_check && abort_check())
|
||||
return false;
|
||||
if (!session->ops || !session->ops->has_pending(session)) {
|
||||
if (!session->ssl || SSL_pending(session->ssl) == 0) {
|
||||
int remaining_ms = deadline_remaining_ms(&deadline);
|
||||
if (remaining_ms <= 0) {
|
||||
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", timeout_sec);
|
||||
@@ -1079,10 +890,6 @@ bool protocol_receive_status_keepalive(ProtocolSession* session, Status* status,
|
||||
Status received;
|
||||
if (!protocol_read_status_until(session, &received, &deadline))
|
||||
return false;
|
||||
if (!status_is_valid(received)) {
|
||||
log_message(LOG_LEVEL_ERROR, "Received unknown protocol status %d", received);
|
||||
return false;
|
||||
}
|
||||
if (!protocol_capture_error_detail(session, &received, &deadline, abort_check))
|
||||
return false;
|
||||
if (received == STATUS_KEEPALIVE) {
|
||||
@@ -1149,12 +956,6 @@ Data* receive_data(int fd) {
|
||||
Data* receive_data_limited(int fd, unsigned long long maximum_size) {
|
||||
return protocol_receive_data_limited(legacy_session(fd, -1), maximum_size);
|
||||
}
|
||||
Data* receive_data_body(int fd, unsigned long long size) {
|
||||
return protocol_receive_data_body(legacy_session(fd, -1), size);
|
||||
}
|
||||
Data* receive_data_alloc(int fd, unsigned long long size) {
|
||||
return protocol_receive_data_alloc(legacy_session(fd, -1), size);
|
||||
}
|
||||
bool send_int(int fd, int data) {
|
||||
return protocol_send_int(legacy_session(-1, fd), data);
|
||||
}
|
||||
@@ -1189,19 +990,6 @@ bool send_error_detail(int fd, const char* message) {
|
||||
return send_status(fd, STATUS_ERROR_DETAIL) && send_str(fd, message);
|
||||
}
|
||||
|
||||
bool send_client_message(int fd, const char* message) {
|
||||
if (!message)
|
||||
message = "";
|
||||
char bounded[MAX_CLIENT_MSG_BYTES + 1];
|
||||
size_t len = strlen(message);
|
||||
if (len > MAX_CLIENT_MSG_BYTES) {
|
||||
memcpy(bounded, message, MAX_CLIENT_MSG_BYTES);
|
||||
bounded[MAX_CLIENT_MSG_BYTES] = '\0';
|
||||
message = bounded;
|
||||
}
|
||||
return send_status(fd, STATUS_CLIENT_MSG) && send_str(fd, message);
|
||||
}
|
||||
|
||||
const char* protocol_last_error(void) {
|
||||
return io_error_detail;
|
||||
}
|
||||
|
||||
+18
-141
@@ -15,12 +15,6 @@
|
||||
* this for a rejection and the detail frame stays a small, fixed bound. */
|
||||
#define MAX_ERROR_DETAIL_BYTES 4096
|
||||
|
||||
/* Hard cap on a client diagnostic forwarded over the STATUS_CLIENT_MSG channel
|
||||
* (protocol 2.30.0, rsync's --stderr=client). The body is reused from the
|
||||
* bounded-string wire helper and sliced to this many bytes before it is sent,
|
||||
* so a peer can never be made to retain more than this per message. */
|
||||
#define MAX_CLIENT_MSG_BYTES 4096
|
||||
|
||||
/* Maximum uncompressed file payload accepted by the receiver's whole-file
|
||||
* paths. A single whole file is charged against the per-connection memory
|
||||
* reservation (MAX_CONNECTION_MEMORY) and against the server allocation
|
||||
@@ -32,22 +26,8 @@
|
||||
/* Maximum allowed data payload size for receive_data (whole-file bound) */
|
||||
#define MAX_DATA_PAYLOAD_SIZE MAX_RECEIVE_WHOLE_FILE_SIZE
|
||||
|
||||
/* Runtime whole-file receive bound. It defaults to MAX_RECEIVE_WHOLE_FILE_SIZE
|
||||
* and exists so the test suite can lower the ceiling (via the
|
||||
* FASTSYNC_MAX_WHOLE_FILE_SIZE environment variable, a byte count) and exercise
|
||||
* the streaming path with a small, fast transfer. A payload at or below the
|
||||
* bound keeps the historical whole-buffer path; a larger one is streamed
|
||||
* through a bounded buffer. The value is resolved once per process and never
|
||||
* exceeds the compile-time ceiling, so a malicious environment cannot raise it
|
||||
* beyond the protocol limit. */
|
||||
unsigned long long protocol_whole_file_receive_limit(void);
|
||||
|
||||
/* Maximum chunk size (64 MB) — prevents unbounded allocation from the wire */
|
||||
#define MAX_CHUNK_SIZE (64ULL * 1024 * 1024)
|
||||
/* Files larger than this are not kept fully in memory while loading: the
|
||||
* loader skips them so the sender streams from the path, and file_checksum
|
||||
* hashes them from disk in bounded buffers instead of forcing a full load. */
|
||||
#define STREAM_THRESHOLD (64ULL * 1024 * 1024)
|
||||
#define MAX_MANIFEST_ENTRIES (1024 * 1024)
|
||||
/* Aggregate bytes retained by one received deletion manifest. */
|
||||
#define MAX_MANIFEST_BYTES (16ULL * 1024 * 1024)
|
||||
@@ -66,48 +46,16 @@ unsigned long long protocol_whole_file_receive_limit(void);
|
||||
|
||||
typedef struct ssl_st SSL;
|
||||
|
||||
typedef struct ProtocolSession ProtocolSession;
|
||||
|
||||
/*
|
||||
* Transport vtable: the per-session set of I/O primitives the three protocol
|
||||
* loops (send, receive, status-read) dispatch through. The ops are selected
|
||||
* once, when the session is initialized or its SSL is installed, so the loops
|
||||
* never branch on the transport at runtime. A plaintext session uses the
|
||||
* read()/write() ops; a TLS session uses the SSL_read()/SSL_write() ops.
|
||||
*
|
||||
* `send`/`recv` attempt exactly one transfer and return:
|
||||
* > 0 bytes transferred,
|
||||
* PROTOCOL_IO_RETRY no progress; poll on *wait_events and retry,
|
||||
* PROTOCOL_IO_CLOSED peer closed the stream,
|
||||
* PROTOCOL_IO_ERROR fatal transport error.
|
||||
* `has_pending` reports bytes already buffered by the transport (a TLS record
|
||||
* residue); the receive loops skip the poll() gate when it is true.
|
||||
*/
|
||||
typedef struct ProtocolIoOps {
|
||||
ssize_t (*send)(ProtocolSession* session, const void* data, size_t size, short* wait_events);
|
||||
ssize_t (*recv)(ProtocolSession* session, void* data, size_t size, short* wait_events);
|
||||
bool (*has_pending)(const ProtocolSession* session);
|
||||
} ProtocolIoOps;
|
||||
|
||||
/* Negative sentinels returned by ProtocolIoOps.send/recv (see above). */
|
||||
enum {
|
||||
PROTOCOL_IO_RETRY = -1,
|
||||
PROTOCOL_IO_CLOSED = -2,
|
||||
PROTOCOL_IO_ERROR = -3,
|
||||
};
|
||||
|
||||
/*
|
||||
* Explicit owner of protocol I/O. A session does not own the descriptors or
|
||||
* SSL object; it only describes the transport used by a transfer. This makes
|
||||
* it safe to pass the transport to a worker without relying on inherited
|
||||
* thread-local state.
|
||||
*/
|
||||
struct ProtocolSession {
|
||||
typedef struct ProtocolSession {
|
||||
int read_fd;
|
||||
int write_fd;
|
||||
SSL* ssl;
|
||||
/* Transport dispatch selected by protocol_session_init()/set_ssl(). */
|
||||
const ProtocolIoOps* ops;
|
||||
unsigned long long bwlimit;
|
||||
long long bw_tokens;
|
||||
long long bw_last_refill_sec;
|
||||
@@ -123,7 +71,7 @@ struct ProtocolSession {
|
||||
* SO_RCVTIMEO/SO_SNDTIMEO. The server does not propagate a client 0 here: it
|
||||
* installs protocol_server_io_timeout_sec() so its sessions keep a floor. */
|
||||
int io_timeout_sec;
|
||||
};
|
||||
} ProtocolSession;
|
||||
|
||||
typedef int Status;
|
||||
enum NET_STATUS {
|
||||
@@ -139,13 +87,8 @@ enum NET_STATUS {
|
||||
STATUS_KEEPALIVE,
|
||||
STATUS_ABORT,
|
||||
STATUS_CHECK_BATCH,
|
||||
/* An explicit directory entry (--dirs / an empty source directory): the sender
|
||||
* transmits the path and, when metadata/xattrs are negotiated, their blocks;
|
||||
* the receiver creates the directory below the receive root. Protocol 2.30.0
|
||||
* inserts an int32 probe flag right after the status when report_dest_info is
|
||||
* negotiated: probe=1 is a report-only frame (path only; the receiver answers
|
||||
* STATUS_DEST_INFO and creates nothing), probe=0 is a real create that is
|
||||
* answered with the directory's pre-transfer state before it is created. */
|
||||
/* An explicit directory entry (--dirs): the sender transmits only the path;
|
||||
* the receiver creates the directory below the receive root. */
|
||||
STATUS_MKDIR,
|
||||
/* --append / --append-verify tail resume. STATUS_APPEND is sent by the
|
||||
* receiver after a per-file STATUS_CHECK when the existing destination file
|
||||
@@ -228,22 +171,16 @@ enum NET_STATUS {
|
||||
* limit stopped deletions"). Appended after STATUS_DRY_RUN_TRANSFER so no
|
||||
* existing status is renumbered. */
|
||||
STATUS_DELETE_LIMIT,
|
||||
/* Destination-state report for output parity (protocol 2.23.0; extended to
|
||||
* directories/symlinks in 2.30.0). When the wire config carries
|
||||
* report_dest_info=true, the receiver answers every per-file STATUS_CHECK
|
||||
* request with STATUS_DEST_INFO FIRST, followed by a fixed record describing
|
||||
* the pre-transfer destination entry (int32 has_old; int32 target_matches;
|
||||
* uint64 size; int64 mtime; int64 mtime_nsec; uint32 mode; int32 uid;
|
||||
* int32 gid). The ordinary STATUS_OK/STATUS_NEXT/... verdict follows, so the
|
||||
* sender can render rsync-accurate -i/--out-format columns (new vs modified,
|
||||
* and which of size/time/perms/owner/group differ) without changing the
|
||||
* transfer decision itself. Protocol 2.30.0 also uses this record for
|
||||
* STATUS_MKDIR and STATUS_SYMLINK: the sender consumes it into the entry's
|
||||
* dest_state before emitting its change line, and target_matches reports
|
||||
* whether an existing symlink's on-disk target already equals the incoming
|
||||
* one (so the sender can render `cLc........` vs `.L..t......` and suppress
|
||||
* an unchanged symlink). Appended after STATUS_DELETE_LIMIT so no existing
|
||||
* status is renumbered. */
|
||||
/* Destination-state report for output parity (protocol 2.23.0). When the
|
||||
* wire config carries report_dest_info=true, the receiver answers every
|
||||
* per-file STATUS_CHECK request with STATUS_DEST_INFO FIRST, followed by a
|
||||
* fixed record describing the pre-transfer destination entry
|
||||
* (int32 has_old; uint64 size; int64 mtime; int64 mtime_nsec; uint32 mode;
|
||||
* int32 uid; int32 gid). The ordinary STATUS_OK/STATUS_NEXT/... verdict
|
||||
* follows, so the sender can render rsync-accurate -i/--out-format columns
|
||||
* (new vs modified, and which of size/time/perms/owner/group differ) without
|
||||
* changing the transfer decision itself. Appended after
|
||||
* STATUS_DELETE_LIMIT so no existing status is renumbered. */
|
||||
STATUS_DEST_INFO,
|
||||
/* Per-directory delete plan (protocol 2.24.0). The sender of a
|
||||
* --delete-during/--delete-delay transfer streams one frame per source
|
||||
@@ -254,9 +191,7 @@ enum NET_STATUS {
|
||||
* (--delete-delay). Payload: an int32 has_config flag (1 on the first plan
|
||||
* of the run, 0 afterwards); when set, the three global config sections
|
||||
* (protected-prefix count+paths, size-skipped count+paths, missing-args
|
||||
* count+paths); then an int32 apply flag (1 for a real plan, 0 for a
|
||||
* config-only carrier frame that must not walk a directory); then the
|
||||
* destination-relative directory path wire string
|
||||
* count+paths); then the destination-relative directory path wire string
|
||||
* ("." for the receive root); then the child-directory count + names and the
|
||||
* child-file count + names that must be kept. Appended after
|
||||
* STATUS_DEST_INFO so no existing status is renumbered. */
|
||||
@@ -265,47 +200,15 @@ enum NET_STATUS {
|
||||
* config carries report_stats=true, the receiver sends this status once,
|
||||
* immediately before its terminal success status, followed by a fixed stats
|
||||
* record (see format_stats_send/receive in format.h) and, when the run is a
|
||||
* --dry-run with --delete, the would-delete path list. Protocol 2.30.0
|
||||
* appends the four deleted_reg/dir/link/special counters to that record, so
|
||||
* --stats can render rsync's `Number of deleted files` per-type breakdown.
|
||||
* Appended after STATUS_DELETE_PLAN so no existing status is renumbered. */
|
||||
STATUS_STATS,
|
||||
/* Client diagnostic channel (protocol 2.30.0, rsync's --stderr=client /
|
||||
* --no-msgs2stderr). When the client's --stderr mode is `client`, the
|
||||
* client forwards its own diagnostics over this client->server frame
|
||||
* (STATUS_CLIENT_MSG followed by a bounded length-prefixed string, capped at
|
||||
* MAX_CLIENT_MSG_BYTES) instead of writing them to its local stderr. The
|
||||
* receiver reads the string and writes it to the server's stderr (respecting
|
||||
* the server log destination). Appended after STATUS_STATS so no existing
|
||||
* status is renumbered. */
|
||||
STATUS_CLIENT_MSG,
|
||||
/* Receiver-side partial transfer (protocol 2.30.0). Sent by the receiver as
|
||||
* the terminal status INSTEAD of STATUS_OK when one or more entries failed
|
||||
* per-entry without aborting the stream (currently a --devices mknod
|
||||
* EPERM/EACCES). The transfer otherwise succeeded and every successfully
|
||||
* stored file was acknowledged, so the sender may still remove
|
||||
* --remove-source-files sources; the sender maps this to rsync's exit code
|
||||
* 23 ("partial transfer due to error"), distinct from a fatal STATUS_ERROR.
|
||||
* Appended after STATUS_CLIENT_MSG so no existing status is renumbered. */
|
||||
STATUS_PARTIAL
|
||||
* --dry-run with --delete, the would-delete path list. Appended after
|
||||
* STATUS_DELETE_PLAN so no existing status is renumbered. */
|
||||
STATUS_STATS
|
||||
};
|
||||
|
||||
void io_set_fds(int read_fd, int write_fd);
|
||||
void io_set_bwlimit(unsigned long long bytes_per_sec);
|
||||
unsigned long long io_get_bwlimit(void);
|
||||
void io_set_ssl(SSL* ssl);
|
||||
SSL* io_get_ssl(void);
|
||||
/* SSL object of the transport in effect on this thread: the currently bound
|
||||
* session's SSL when a TLS session is bound, otherwise the legacy thread-local
|
||||
* io_ssl. NULL for a plaintext transport. Unlike io_get_ssl(), this resolves
|
||||
* worker threads that bound a TLS session via protocol_session_set_ssl()/
|
||||
* protocol_session_bind() but never called io_set_ssl() themselves (C11
|
||||
* _Thread_local state is not inherited by a new thread). A bound session only
|
||||
* wins when its selected dispatch is TLS; a bound plaintext session (ssl ==
|
||||
* NULL) falls back to io_ssl so it can never mask a live encrypted transport.
|
||||
* Callers that must choose a TLS-only code path (e.g. file_send.c's sendfile
|
||||
* fallback) must use this instead of io_get_ssl(). */
|
||||
SSL* protocol_current_ssl(void);
|
||||
|
||||
/* Process-wide wire byte counters. protocol_send_n_data/protocol_receive_n_data
|
||||
* update them; the zero-copy sendfile path reports through
|
||||
@@ -314,15 +217,6 @@ SSL* protocol_current_ssl(void);
|
||||
unsigned long long protocol_bytes_written(void);
|
||||
unsigned long long protocol_bytes_read(void);
|
||||
void protocol_note_bytes_written(unsigned long long bytes);
|
||||
/* Apply --bwlimit pacing to bytes written outside protocol_send_n_data (the
|
||||
* plaintext zero-copy sendfile fast path). `file_descriptor` is the wire fd
|
||||
* the bytes were written to, so the legacy session is resolved exactly as the
|
||||
* preceding send_n_data call resolved it (the bound TLS session still wins when
|
||||
* set); resolving with the same fd avoids re-initializing the legacy session
|
||||
* and granting a second first-call burst. Runs the same token-bucket throttle,
|
||||
* so the sendfile transport is paced identically to the buffered/TLS paths. A
|
||||
* no-op when the effective session has no bandwidth limit. */
|
||||
void protocol_throttle_bytes(int file_descriptor, size_t bytes);
|
||||
|
||||
void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd);
|
||||
/* Transitional bridge for helpers whose signatures still carry only an fd. */
|
||||
@@ -366,9 +260,6 @@ bool protocol_send_int(ProtocolSession* session, int data);
|
||||
bool protocol_receive_int(ProtocolSession* session, int* data);
|
||||
bool protocol_send_status(ProtocolSession* session, Status status);
|
||||
bool protocol_receive_status(ProtocolSession* session, Status* status);
|
||||
/* As protocol_receive_status, but with an explicit per-message deadline
|
||||
* (seconds) instead of the session's configured io_timeout_sec. */
|
||||
bool protocol_receive_status_timed(ProtocolSession* session, Status* status, int timeout_sec);
|
||||
bool send_n_data(int file_descriptor, const void* data, size_t data_size);
|
||||
bool receive_n_data(int file_descriptor, void* data, size_t data_size);
|
||||
|
||||
@@ -380,15 +271,6 @@ char* receive_str_redacted(int file_descriptor);
|
||||
bool send_data(int file_descriptor, const Data* data);
|
||||
Data* receive_data(int file_descriptor);
|
||||
Data* receive_data_limited(int file_descriptor, unsigned long long maximum_size);
|
||||
/* Read exactly `size` bytes as a charged Data body. The length-prefixed
|
||||
* receive_data_limited() reads the header itself; this variant is for callers
|
||||
* that must inspect the declared size (and possibly stream the body instead)
|
||||
* before allocating. `size` must already be within MAX_DATA_PAYLOAD_SIZE. */
|
||||
Data* receive_data_body(int file_descriptor, unsigned long long size);
|
||||
/* Allocate (and charge) a `size`-byte Data body without reading it; the caller
|
||||
* fills `result->data` itself. Used when a frame's leading bytes must be
|
||||
* inspected before the rest of the body is read. */
|
||||
Data* receive_data_alloc(int file_descriptor, unsigned long long size);
|
||||
bool send_int(int file_descriptor, int data);
|
||||
bool receive_int(int file_descriptor, int* data);
|
||||
bool send_status(int file_descriptor, Status status);
|
||||
@@ -397,11 +279,6 @@ bool receive_status(int file_descriptor, Status* status);
|
||||
* length-prefixed string. Over-long messages are sliced and NULL is treated
|
||||
* as "". Returns false if the status or the string could not be sent. */
|
||||
bool send_error_detail(int file_descriptor, const char* message);
|
||||
/* Send STATUS_CLIENT_MSG followed by a bounded (<= MAX_CLIENT_MSG_BYTES)
|
||||
* length-prefixed string carrying a client diagnostic. Over-long messages are
|
||||
* sliced and NULL is treated as "". Returns false if the status or the string
|
||||
* could not be sent. */
|
||||
bool send_client_message(int file_descriptor, const char* message);
|
||||
/* Human-readable reason captured from the most recent STATUS_ERROR_DETAIL
|
||||
* received on this thread, or "" when the last status was a bare STATUS_ERROR
|
||||
* (or no detail was seen). Thread-local, and valid until the next non-keepalive
|
||||
|
||||
+1
-18
@@ -128,7 +128,7 @@ bool queue_enqueue_multithreaded_cancel(Queue* queue, void* item, mtx_t* mutex,
|
||||
|
||||
void* queue_dequeue(Queue* queue) {
|
||||
if (queue == NULL || queue_is_empty(queue)) {
|
||||
log_message(LOG_LEVEL_ERROR, "%s", "ERROR: Could not dequeue from null or empty queue.");
|
||||
log_perror("ERROR: Could not dequeue from null or empty queue.");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
@@ -139,23 +139,6 @@ void* queue_dequeue(Queue* queue) {
|
||||
return item;
|
||||
}
|
||||
|
||||
bool queue_push(Queue* queue, void* item) {
|
||||
return queue_enqueue(queue, item);
|
||||
}
|
||||
|
||||
void* queue_pop(Queue* queue) {
|
||||
if (queue == NULL || queue_is_empty(queue)) {
|
||||
log_message(LOG_LEVEL_ERROR, "%s", "ERROR: Could not pop from null or empty queue.");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
queue->rear = (queue->rear - 1 + queue->capacity) % queue->capacity;
|
||||
void* item = queue->items[queue->rear];
|
||||
queue->items[queue->rear] = NULL;
|
||||
queue->size--;
|
||||
return item;
|
||||
}
|
||||
|
||||
void* queue_dequeue_multithreaded(Queue* queue, mtx_t* mutex, cnd_t* condition_not_empty,
|
||||
cnd_t* condition_not_full, const bool* other_thread_done) {
|
||||
mtx_lock(mutex);
|
||||
|
||||
@@ -28,11 +28,4 @@ void* queue_dequeue(Queue* queue);
|
||||
void* queue_dequeue_multithreaded(Queue* queue, mtx_t* mutex, cnd_t* condition_not_empty,
|
||||
cnd_t* condition_not_full, const bool* other_thread_done);
|
||||
|
||||
/* LIFO stack operations over the same ring buffer. queue_push() is the enqueue
|
||||
primitive; queue_pop() removes from the rear, so a sequence of pushes is
|
||||
returned in reverse order. Used by the sequential scanner's depth-first
|
||||
traversal. */
|
||||
bool queue_push(Queue* queue, void* item);
|
||||
void* queue_pop(Queue* queue);
|
||||
|
||||
#endif
|
||||
|
||||
@@ -87,7 +87,7 @@ static int parse_remote_dest(const char* dest, RemoteDest* r) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
char* ssh_build_remote_command(const char* server_path, char* const* remote_options,
|
||||
char* ssh_build_remote_command(const char* server_path, bool old_args, char* const* remote_options,
|
||||
int remote_option_count) {
|
||||
const char* path = server_path ? server_path : "fastsync-server";
|
||||
const char* suffix = " --stdio";
|
||||
@@ -105,7 +105,9 @@ char* ssh_build_remote_command(const char* server_path, char* const* remote_opti
|
||||
shell word (remote options below reuse the same escaping), then
|
||||
" --stdio". Quoting the path is the only injection-safe construction: an
|
||||
unquoted path would carry shell metacharacters straight into the remote
|
||||
shell command. (rsync's --old-args no longer disables that protection.) */
|
||||
shell command. --old-args is kept for CLI/ABI compatibility but no longer
|
||||
disables that protection. */
|
||||
(void)old_args;
|
||||
size_t quote_count = 0;
|
||||
for (const char* p = path; *p; p++)
|
||||
if (*p == '\'')
|
||||
@@ -294,8 +296,8 @@ void ssh_free_client_argv(char** argv) {
|
||||
}
|
||||
|
||||
Client* client_connect_ssh(const char* destination, int port, const char* server_path,
|
||||
const char* rsh_command, bool blocking_io, char* const* remote_options,
|
||||
int remote_option_count) {
|
||||
bool old_args, const char* rsh_command, bool blocking_io,
|
||||
char* const* remote_options, int remote_option_count) {
|
||||
RemoteDest r;
|
||||
if (parse_remote_dest(destination, &r) != 0) {
|
||||
char* escaped = output_escape(destination, false);
|
||||
@@ -374,7 +376,7 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
|
||||
snprintf(ssh_user, ssh_user_len, "%s", r.host);
|
||||
|
||||
char* remote_command =
|
||||
ssh_build_remote_command(server_path, remote_options, remote_option_count);
|
||||
ssh_build_remote_command(server_path, old_args, remote_options, remote_option_count);
|
||||
if (!remote_command)
|
||||
ssh_child_setup_failed(exec_pipe[1]);
|
||||
char** ssh_argv = ssh_build_client_argv(rsh_command, port, ssh_user, remote_command);
|
||||
|
||||
@@ -4,17 +4,17 @@
|
||||
#include "transport_tcp.h"
|
||||
|
||||
Client* client_connect_ssh(const char* destination, int port, const char* server_path,
|
||||
const char* rsh_command, bool blocking_io, char* const* remote_options,
|
||||
int remote_option_count);
|
||||
bool old_args, const char* rsh_command, bool blocking_io,
|
||||
char* const* remote_options, int remote_option_count);
|
||||
/* Build the escaped remote-shell command string (the server program path always
|
||||
* quoted as one remote-shell word, followed by ` --stdio` and each
|
||||
* --remote-option value appended as an individually single-quoted shell word).
|
||||
* The path is always escaped so a metacharacter-bearing --rsync-path can never
|
||||
* be interpreted by the remote shell (the --old-args no-op does not disable
|
||||
* quoting). Every --remote-option value is individually escaped with the '\''
|
||||
* sequence and values with empty/control characters are rejected at the CLI
|
||||
* parse layer. */
|
||||
char* ssh_build_remote_command(const char* server_path, char* const* remote_options,
|
||||
* `old_args` is accepted for CLI/ABI compatibility but no longer disables
|
||||
* quoting: the path is always escaped so a metacharacter-bearing
|
||||
* --rsync-path can never be interpreted by the remote shell. Every
|
||||
* --remote-option value is individually escaped with the '\'' sequence and
|
||||
* values with empty/control characters are rejected at the CLI parse layer. */
|
||||
char* ssh_build_remote_command(const char* server_path, bool old_args, char* const* remote_options,
|
||||
int remote_option_count);
|
||||
/* Build the NULL-terminated child argv for the remote-shell client (argv[0] is
|
||||
* the exec/execvp program). rsh_command is whitespace-split into leading argv
|
||||
|
||||
@@ -134,7 +134,7 @@ Server* server_create_ex(int port, const ServerBindOptions* bind_opts) {
|
||||
|
||||
server->file_descriptor = file_descriptor;
|
||||
server->ssl_ctx = NULL;
|
||||
server->max_connections = SERVER_DEFAULT_MAX_CONNECTIONS;
|
||||
server->max_connections = 100;
|
||||
server->active_connections = 0;
|
||||
server->limit_registry = NULL;
|
||||
|
||||
|
||||
@@ -11,10 +11,6 @@
|
||||
* stored here so the transport layer does not depend on daemon config. */
|
||||
struct DaemonLimitRegistry;
|
||||
|
||||
/* Connection cap applied by server_create_ex() until the daemon's configured
|
||||
* `max connections` overrides it via server_set_max_connections(). */
|
||||
#define SERVER_DEFAULT_MAX_CONNECTIONS 100
|
||||
|
||||
typedef struct Server {
|
||||
struct sockaddr_storage address;
|
||||
unsigned int address_length;
|
||||
|
||||
+383
-77
@@ -2,12 +2,10 @@
|
||||
#include "array_list.h"
|
||||
#include "log.h"
|
||||
#include <arpa/inet.h>
|
||||
#include <ctype.h>
|
||||
#include <dirent.h>
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <netinet/in.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
@@ -49,45 +47,11 @@ const char* utils_get_authorized_root_path(void) {
|
||||
return authorized_root_path;
|
||||
}
|
||||
|
||||
void utils_set_error(char* err, size_t err_size, const char* fmt, ...) {
|
||||
if (!err || err_size == 0)
|
||||
return;
|
||||
va_list args;
|
||||
va_start(args, fmt);
|
||||
vsnprintf(err, err_size, fmt, args);
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
bool path_is_within_root(const char* root, const char* path) {
|
||||
size_t root_len = strlen(root);
|
||||
return strncmp(root, path, root_len) == 0 && (path[root_len] == '\0' || path[root_len] == '/');
|
||||
}
|
||||
|
||||
/* Borrowed transfer-relative view of `path`: strip any leading '/' and then a
|
||||
* `root` prefix (its own leading/trailing slashes tolerated), returning a
|
||||
* pointer into `path`. Non-allocating, so it is safe on the hot scan/print
|
||||
* paths. A NULL/empty root, or a path not under `root`, leaves only the
|
||||
* leading-slash strip. `path` must be NUL-terminated and live in the caller. */
|
||||
const char* utils_strip_transfer_root(const char* path, const char* root) {
|
||||
if (path == NULL)
|
||||
return NULL;
|
||||
const char* rel = path;
|
||||
while (*rel == '/')
|
||||
rel++;
|
||||
if (root == NULL)
|
||||
return rel;
|
||||
while (*root == '/')
|
||||
root++;
|
||||
size_t root_len = strlen(root);
|
||||
while (root_len > 0 && root[root_len - 1] == '/')
|
||||
root_len--;
|
||||
if (root_len == 0)
|
||||
return rel;
|
||||
if (strncmp(rel, root, root_len) == 0 && (rel[root_len] == '/' || rel[root_len] == '\0'))
|
||||
return rel + root_len + (rel[root_len] == '/' ? 1 : 0);
|
||||
return rel;
|
||||
}
|
||||
|
||||
/* Open the destination root directory itself, confined to the authorized root.
|
||||
* NOTE (do not merge with file_open_secure_parent): this walk opens dest_root
|
||||
* (a directory that must already exist) and returns its fd, whereas
|
||||
@@ -153,47 +117,6 @@ char* str_dup(const char* string) {
|
||||
return new_string;
|
||||
}
|
||||
|
||||
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified) {
|
||||
if (specified)
|
||||
*specified = false;
|
||||
if (!env_name || !resolve)
|
||||
return -1;
|
||||
const char* env = getenv(env_name);
|
||||
if (!env)
|
||||
return -1;
|
||||
|
||||
bool saw_nonblank = false;
|
||||
const char* p = env;
|
||||
while (*p) {
|
||||
if (*p == '&')
|
||||
break;
|
||||
if (isspace((unsigned char)*p)) {
|
||||
p++;
|
||||
continue;
|
||||
}
|
||||
saw_nonblank = true;
|
||||
char token[64];
|
||||
size_t len = 0;
|
||||
while (*p && *p != '&' && !isspace((unsigned char)*p)) {
|
||||
if (len < sizeof(token) - 1)
|
||||
token[len++] = *p;
|
||||
p++;
|
||||
}
|
||||
token[len] = '\0';
|
||||
if (len > 0) {
|
||||
int id = resolve(token);
|
||||
if (id >= 0) {
|
||||
if (specified)
|
||||
*specified = true;
|
||||
return id;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (specified)
|
||||
*specified = saw_nonblank;
|
||||
return -1;
|
||||
}
|
||||
|
||||
#define STR_HASH_SET_MIN_CAPACITY 16
|
||||
|
||||
static size_t str_hash_set_hash(const char* key, size_t len) {
|
||||
@@ -625,6 +548,389 @@ bool format_human_bytes(unsigned long long bytes, char* buffer, size_t buffer_si
|
||||
return written >= 0 && (size_t)written < buffer_size;
|
||||
}
|
||||
|
||||
/* Build the keep-set index from the exact manifest entries only. A lookup of
|
||||
`rel` succeeds iff `rel` is a kept entry, a kept directory, or an ancestor
|
||||
directory of kept content (the old is_dir_in_manifest predicate); the sorted
|
||||
view answers "is an ancestor of kept content" without materializing any
|
||||
per-component prefix copy, so the index is O(manifest size) memory. */
|
||||
static bool build_keep_index(const ArrayList* manifest, PathIndex* index) {
|
||||
if (!manifest || manifest->size <= 0)
|
||||
return path_index_build(index, NULL, 0);
|
||||
return path_index_build(index, (const char* const*)manifest->items, (size_t)manifest->size);
|
||||
}
|
||||
|
||||
static bool keep_is_dir(const PathIndex* index, const char* rel_path) {
|
||||
return path_index_contains(index, rel_path) || path_index_has_descendant(index, rel_path);
|
||||
}
|
||||
|
||||
static bool keep_is_file(const PathIndex* index, const char* rel_path) {
|
||||
return path_index_contains(index, rel_path);
|
||||
}
|
||||
|
||||
/* True when child_rel is, or lies below, a protected entry. A prefix "a"
|
||||
therefore protects "a" and "a/b/c" but not "ab". Entries with top_level_only
|
||||
set only protect DIRECT children of the receive root (at_root); nested
|
||||
directories that share such a name stay ordinary destination content. */
|
||||
bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSkipEntry* skips,
|
||||
int skip_count) {
|
||||
for (int i = 0; i < skip_count; i++) {
|
||||
if (skips[i].top_level_only && !at_root)
|
||||
continue;
|
||||
size_t prefix_len = strlen(skips[i].prefix);
|
||||
if (strncmp(child_rel, skips[i].prefix, prefix_len) == 0 &&
|
||||
(child_rel[prefix_len] == '\0' || child_rel[prefix_len] == '/'))
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/* Per-run deletion budget and tallies. `max_delete` is the cap on the number
|
||||
of entries the walker may remove (SIZE_MAX = unlimited); once it is reached
|
||||
the remaining extras are counted in `skipped` and left in place, matching
|
||||
rsync's partial --max-delete behavior. */
|
||||
typedef struct {
|
||||
size_t max_delete;
|
||||
size_t deleted;
|
||||
size_t skipped;
|
||||
bool limit_hit;
|
||||
} DeleteBudget;
|
||||
|
||||
/* True when direct children of the directory named by `rel` may be removed.
|
||||
With no synchronization info (dirs == NULL) the whole tree is deletable; when
|
||||
a dirs index is supplied only its exact entries are (the receive root is the
|
||||
"." sentinel). */
|
||||
static bool is_synced_dir(const PathIndex* dirs, const char* rel) {
|
||||
if (!dirs)
|
||||
return true;
|
||||
return path_index_contains(dirs, rel[0] == '\0' ? "." : rel);
|
||||
}
|
||||
|
||||
/* Remove the extras directly inside the directory open on `dirfd`, recursing
|
||||
into every child directory so kept content below a synchronized prefix is
|
||||
reached. `all_removed` reports whether every child entry was removed (so the
|
||||
caller may rmdir this directory). A child directory is never removed when it
|
||||
is itself a synchronized directory or holds kept content; with a dirs index
|
||||
supplied, direct children of a non-synchronized directory are never extras at
|
||||
all (they are left in place but still descended into). Symlinks are unlinked
|
||||
like any other non-directory extra (never followed). */
|
||||
static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
||||
const PathIndex* dirs, DeleteBudget* budget,
|
||||
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
|
||||
bool* all_removed) {
|
||||
/* openat(dirfd, ".") opens an independent file description: a dup() would
|
||||
share dirfd's file offset and a prior pass could leave the stream drained. */
|
||||
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (scanfd < 0)
|
||||
return false;
|
||||
DIR* dir = fdopendir(scanfd);
|
||||
if (!dir) {
|
||||
close(scanfd);
|
||||
return false;
|
||||
}
|
||||
bool operation_ok = true;
|
||||
bool local_survives = false;
|
||||
/* A directory is deletable when it or ANY ancestor is synchronized; the
|
||||
`parent_deletable` flag carries that down the recursion so dest-only
|
||||
directories below a synchronized root are removed wholesale. */
|
||||
bool deletable = parent_deletable || is_synced_dir(dirs, rel_path);
|
||||
const struct dirent* entry;
|
||||
while ((entry = readdir(dir)) != NULL) {
|
||||
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
|
||||
continue;
|
||||
char* child_rel = path_cat((char*)rel_path, entry->d_name);
|
||||
if (!child_rel) {
|
||||
operation_ok = false;
|
||||
continue;
|
||||
}
|
||||
/* A --delay-updates run keeps its staging directory as a direct child of
|
||||
the receive root, and basis-dir snapshots live below it too. Their
|
||||
contents are not manifest entries, so descending into them would delete
|
||||
every staged / basis file as an "extra". Only the staging name (a
|
||||
top-level-only prefix) and the basis prefixes are protected: a nested
|
||||
destination directory that happens to be called .fastsync-stage is
|
||||
ordinary content. */
|
||||
if (path_under_skip_prefix(child_rel, rel_path[0] == '\0', skips, skip_count)) {
|
||||
local_survives = true;
|
||||
free(child_rel);
|
||||
continue;
|
||||
}
|
||||
struct stat st;
|
||||
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
||||
if (errno != ENOENT)
|
||||
operation_ok = false;
|
||||
free(child_rel);
|
||||
continue;
|
||||
}
|
||||
if (S_ISDIR(st.st_mode)) {
|
||||
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
bool child_all_removed = false;
|
||||
if (childfd >= 0) {
|
||||
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count, deletable,
|
||||
&child_all_removed))
|
||||
operation_ok = false;
|
||||
close(childfd);
|
||||
} else if (errno != ENOENT) {
|
||||
operation_ok = false;
|
||||
}
|
||||
bool child_synced = dirs && path_index_contains(dirs, child_rel);
|
||||
if (child_synced || keep_is_dir(keep, child_rel)) {
|
||||
/* A synchronized directory and a directory holding kept content are
|
||||
never removed. */
|
||||
local_survives = true;
|
||||
} else if (child_all_removed && deletable) {
|
||||
if (budget->deleted >= budget->max_delete) {
|
||||
budget->limit_hit = true;
|
||||
budget->skipped++;
|
||||
local_survives = true;
|
||||
} else if (unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0) {
|
||||
/* ENOENT: already gone (fine). ENOTEMPTY/EEXIST: the directory
|
||||
still holds entries the walker leaves in place (a protected
|
||||
excluded prefix, a kept file the manifest protects, a symlink);
|
||||
rsync leaves such a directory behind, so this is not an error.
|
||||
Only genuine I/O failures abort the deletion. */
|
||||
if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST)
|
||||
operation_ok = false;
|
||||
local_survives = true;
|
||||
} else {
|
||||
budget->deleted++;
|
||||
}
|
||||
} else {
|
||||
local_survives = true;
|
||||
}
|
||||
} else {
|
||||
bool found = keep_is_file(keep, child_rel);
|
||||
if (found || !deletable) {
|
||||
/* Kept file, or a child of a directory that is not synchronized: never
|
||||
an extra for this run. */
|
||||
local_survives = true;
|
||||
} else if (budget->deleted >= budget->max_delete) {
|
||||
budget->limit_hit = true;
|
||||
budget->skipped++;
|
||||
local_survives = true;
|
||||
} else if (unlinkat(dirfd, entry->d_name, 0) != 0) {
|
||||
if (errno != ENOENT)
|
||||
operation_ok = false;
|
||||
local_survives = true;
|
||||
} else {
|
||||
budget->deleted++;
|
||||
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
|
||||
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
|
||||
free(escaped_path);
|
||||
}
|
||||
}
|
||||
free(child_rel);
|
||||
}
|
||||
closedir(dir);
|
||||
*all_removed = !local_survives;
|
||||
return operation_ok;
|
||||
}
|
||||
|
||||
/* Read-only mirror of delete_extras_fd: records the paths that WOULD be removed
|
||||
without unlinking anything. A child directory is reported after its own
|
||||
reportable children (depth-first), matching the delete pass's ordering. */
|
||||
static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
||||
const PathIndex* dirs, ArrayList* out, size_t* recorded,
|
||||
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
|
||||
bool* all_removed) {
|
||||
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (scanfd < 0)
|
||||
return false;
|
||||
DIR* dir = fdopendir(scanfd);
|
||||
if (!dir) {
|
||||
close(scanfd);
|
||||
return false;
|
||||
}
|
||||
bool operation_ok = true;
|
||||
bool local_survives = false;
|
||||
bool deletable = parent_deletable || is_synced_dir(dirs, rel_path);
|
||||
const struct dirent* entry;
|
||||
while ((entry = readdir(dir)) != NULL) {
|
||||
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
|
||||
continue;
|
||||
char* child_rel = path_cat((char*)rel_path, entry->d_name);
|
||||
if (!child_rel) {
|
||||
operation_ok = false;
|
||||
continue;
|
||||
}
|
||||
if (path_under_skip_prefix(child_rel, rel_path[0] == '\0', skips, skip_count)) {
|
||||
local_survives = true;
|
||||
free(child_rel);
|
||||
continue;
|
||||
}
|
||||
struct stat st;
|
||||
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
|
||||
if (errno != ENOENT)
|
||||
operation_ok = false;
|
||||
free(child_rel);
|
||||
continue;
|
||||
}
|
||||
if (S_ISDIR(st.st_mode)) {
|
||||
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
bool child_all_removed = false;
|
||||
if (childfd >= 0) {
|
||||
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
|
||||
deletable, &child_all_removed))
|
||||
operation_ok = false;
|
||||
close(childfd);
|
||||
} else if (errno != ENOENT) {
|
||||
operation_ok = false;
|
||||
}
|
||||
bool child_synced = dirs && path_index_contains(dirs, child_rel);
|
||||
if (child_synced || keep_is_dir(keep, child_rel)) {
|
||||
local_survives = true;
|
||||
} else if (child_all_removed && deletable) {
|
||||
size_t len = strlen(child_rel);
|
||||
char* copy = malloc(len + 2);
|
||||
if (!copy) {
|
||||
operation_ok = false;
|
||||
} else {
|
||||
memcpy(copy, child_rel, len);
|
||||
copy[len] = '/';
|
||||
copy[len + 1] = '\0';
|
||||
if (!array_list_add(out, copy)) {
|
||||
free(copy);
|
||||
operation_ok = false;
|
||||
} else {
|
||||
(*recorded)++;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
local_survives = true;
|
||||
}
|
||||
} else {
|
||||
bool found = keep_is_file(keep, child_rel);
|
||||
if (found || !deletable) {
|
||||
local_survives = true;
|
||||
} else {
|
||||
char* copy = str_dup(child_rel);
|
||||
if (!copy || !array_list_add(out, copy)) {
|
||||
free(copy);
|
||||
operation_ok = false;
|
||||
} else {
|
||||
(*recorded)++;
|
||||
}
|
||||
}
|
||||
}
|
||||
free(child_rel);
|
||||
}
|
||||
closedir(dir);
|
||||
*all_removed = !local_survives;
|
||||
return operation_ok;
|
||||
}
|
||||
|
||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||
ArrayList* out, size_t* count_out) {
|
||||
if (count_out)
|
||||
*count_out = 0;
|
||||
if (!manifest || !out)
|
||||
return false;
|
||||
PathIndex keep;
|
||||
if (!build_keep_index(manifest, &keep))
|
||||
return false;
|
||||
PathIndex dirs;
|
||||
bool have_dirs = synced_dirs != NULL;
|
||||
if (have_dirs &&
|
||||
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
|
||||
path_index_free(&keep);
|
||||
return false;
|
||||
}
|
||||
int rootfd;
|
||||
int root_fd = utils_get_authorized_root_fd();
|
||||
if (root_fd >= 0) {
|
||||
if (utils_get_authorized_root_path())
|
||||
rootfd = utils_open_authorized_destination(dest_root);
|
||||
else if (dest_root == NULL)
|
||||
rootfd = dup(root_fd);
|
||||
else
|
||||
rootfd = -1;
|
||||
} else {
|
||||
rootfd = open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
}
|
||||
if (rootfd < 0) {
|
||||
path_index_free(&keep);
|
||||
if (have_dirs)
|
||||
path_index_free(&dirs);
|
||||
return false;
|
||||
}
|
||||
bool all_removed = false;
|
||||
size_t recorded = 0;
|
||||
bool ok = list_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, out, &recorded, skips,
|
||||
skip_count, false, &all_removed);
|
||||
if (close(rootfd) != 0)
|
||||
ok = false;
|
||||
path_index_free(&keep);
|
||||
if (have_dirs)
|
||||
path_index_free(&dirs);
|
||||
if (count_out)
|
||||
*count_out = recorded;
|
||||
return ok;
|
||||
}
|
||||
|
||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, size_t max_delete,
|
||||
const DeleteSkipEntry* skips, int skip_count,
|
||||
size_t* deleted_out, size_t* skipped_out) {
|
||||
if (deleted_out)
|
||||
*deleted_out = 0;
|
||||
if (skipped_out)
|
||||
*skipped_out = 0;
|
||||
if (!manifest)
|
||||
return DELETE_WALK_ERROR;
|
||||
/* Index the keep-set (and the synchronized-dir set, when supplied) once so
|
||||
membership is answered in O(path length) instead of scanning every entry
|
||||
for every destination entry. */
|
||||
PathIndex keep;
|
||||
if (!build_keep_index(manifest, &keep))
|
||||
return DELETE_WALK_ERROR;
|
||||
PathIndex dirs;
|
||||
bool have_dirs = synced_dirs != NULL;
|
||||
if (have_dirs &&
|
||||
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
|
||||
path_index_free(&keep);
|
||||
return DELETE_WALK_ERROR;
|
||||
}
|
||||
int rootfd;
|
||||
int root_fd = utils_get_authorized_root_fd();
|
||||
if (root_fd >= 0) {
|
||||
if (utils_get_authorized_root_path())
|
||||
rootfd = utils_open_authorized_destination(dest_root);
|
||||
else if (dest_root == NULL)
|
||||
rootfd = dup(root_fd);
|
||||
else
|
||||
rootfd = -1;
|
||||
} else {
|
||||
rootfd = open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
}
|
||||
if (rootfd < 0) {
|
||||
path_index_free(&keep);
|
||||
if (have_dirs)
|
||||
path_index_free(&dirs);
|
||||
return DELETE_WALK_ERROR;
|
||||
}
|
||||
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
|
||||
bool all_removed = false;
|
||||
bool ok = delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips,
|
||||
skip_count, false, &all_removed);
|
||||
if (close(rootfd) != 0)
|
||||
ok = false;
|
||||
path_index_free(&keep);
|
||||
if (have_dirs)
|
||||
path_index_free(&dirs);
|
||||
if (deleted_out)
|
||||
*deleted_out = budget.deleted;
|
||||
if (skipped_out)
|
||||
*skipped_out = budget.skipped;
|
||||
if (!ok)
|
||||
return DELETE_WALK_ERROR;
|
||||
return budget.limit_hit ? DELETE_WALK_LIMIT_REACHED : DELETE_WALK_OK;
|
||||
}
|
||||
|
||||
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
|
||||
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL) ==
|
||||
DELETE_WALK_OK;
|
||||
}
|
||||
|
||||
bool has_path_traversal(const char* path) {
|
||||
if (!path)
|
||||
return true;
|
||||
|
||||
+53
-23
@@ -2,7 +2,6 @@
|
||||
#define UTILS_H
|
||||
|
||||
#include "array_list.h"
|
||||
#include "filter.h"
|
||||
#include <stddef.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
@@ -82,17 +81,6 @@ bool path_index_has_descendant(const PathIndex* index, const char* path);
|
||||
|
||||
char* str_dup(const char* string);
|
||||
char* output_escape(const char* string, bool eight_bit_output);
|
||||
|
||||
/* Resolve the first supported name from a rsync algorithm-preference
|
||||
* environment variable (RSYNC_COMPRESS_LIST / RSYNC_CHECKSUM_LIST). `resolve`
|
||||
* maps a case-insensitive name to an algorithm id (>= 0) or -1 for an unknown
|
||||
* name. rsync's syntax is a whitespace-separated list (comma/colon are NOT
|
||||
* separators); the client-side half ends at '&'. Unknown entries are skipped
|
||||
* and the first resolvable one wins. *specified is set true when the variable
|
||||
* holds at least one non-blank character. Returns the first resolvable id, or
|
||||
* -1 when the variable is unset/blank or names no supported algorithm. */
|
||||
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified);
|
||||
|
||||
/* Upper bound on one line/token read from a local list file (--files-from,
|
||||
* --exclude-from/--include-from, .rsync-filter). Mirrors MAX_STRING_SIZE and
|
||||
* stops a hostile multi-gigabyte line from forcing unbounded allocation. */
|
||||
@@ -106,7 +94,59 @@ int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* sp
|
||||
ssize_t utils_getdelim_bounded(FILE* stream, char** line, size_t* cap, int delim, size_t max_len);
|
||||
char* path_cat(const char* path1, const char* path2);
|
||||
bool glob_match(const char* pattern, const char* str);
|
||||
|
||||
/* Result of a bounded extra-file deletion run. */
|
||||
typedef enum {
|
||||
/* Every extra entry was removed (or there were none). */
|
||||
DELETE_WALK_OK = 0,
|
||||
/* The numeric cap for this run was reached before every extra was removed.
|
||||
The walker removed exactly the entries the cap allowed and skipped (without
|
||||
removing) the rest, matching rsync's partial --max-delete behavior. */
|
||||
DELETE_WALK_LIMIT_REACHED,
|
||||
/* A traversal or unlink failure aborted the deletion (partial removal is
|
||||
possible, mirroring the delete pass). */
|
||||
DELETE_WALK_ERROR
|
||||
} DeleteWalkResult;
|
||||
/* One protected entry for the delete walker. When top_level_only is true the
|
||||
prefix is skipped only as a DIRECT child of dest_root (the --delay-updates
|
||||
staging directory, which must not hide genuine extras inside a nested
|
||||
destination directory that happens to share the staging name); otherwise the
|
||||
prefix is skipped at any depth (the --compare-dest/--copy-dest/--link-dest
|
||||
basis trees, and the sender-side protected filter-excluded prefixes, which
|
||||
are never destination content). */
|
||||
typedef struct {
|
||||
const char* prefix;
|
||||
bool top_level_only;
|
||||
} DeleteSkipEntry;
|
||||
/* True when child_rel is, or lies below, one of the protected entries (a prefix
|
||||
"a" protects "a" and "a/b/c" but not "ab"; top_level_only entries protect
|
||||
only DIRECT children of the destination root, i.e. child_rel has no '/'). */
|
||||
bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSkipEntry* skips,
|
||||
int skip_count);
|
||||
/* Remove files/dirs/symlinks under dest_root that are not listed in manifest
|
||||
without ever descending into a protected prefix (see DeleteSkipEntry). When
|
||||
`synced_dirs` is non-NULL, extras are only removed directly inside a directory
|
||||
whose destination-relative path is an exact entry in that list (the receive
|
||||
root is the "." sentinel); directories outside the synchronized set are still
|
||||
descended into so kept content below a listed directory is preserved, but
|
||||
nothing in them is removed. A NULL `synced_dirs` keeps the legacy behavior of
|
||||
treating the whole destination tree as deletable. `max_delete` caps the
|
||||
number of removed entries (SIZE_MAX = unlimited): the walker removes up to the
|
||||
cap and returns DELETE_WALK_LIMIT_REACHED when more extras remained.
|
||||
`deleted_out`/`skipped_out` optionally receive the number of entries removed
|
||||
and the number skipped because of the cap. */
|
||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, size_t max_delete,
|
||||
const DeleteSkipEntry* skips, int skip_count,
|
||||
size_t* deleted_out, size_t* skipped_out);
|
||||
/* Read-only companion to delete_extras_limited: walk the destination exactly as
|
||||
the delete pass would and APPEND (strdup'd) destination-relative paths that
|
||||
WOULD be removed, without touching disk. Used for -n/--dry-run --delete
|
||||
would-delete reporting. Returns true on a clean walk; the caller owns the
|
||||
strings appended to `out` and receives their count in *count_out. */
|
||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||
ArrayList* out, size_t* count_out);
|
||||
bool delete_extras(const char* dest_root, const ArrayList* manifest);
|
||||
/* Open the existing destination directory at `dest_root`, confined to the
|
||||
authorized root with an O_NOFOLLOW component walk (the same confinement the
|
||||
deletion walker uses for its root). Returns a new fd the caller owns, or -1
|
||||
@@ -131,22 +171,12 @@ void utils_set_authorized_root_fd(int fd);
|
||||
* threads spawn; see utils.c). */
|
||||
int utils_get_authorized_root_fd(void);
|
||||
const char* utils_get_authorized_root_path(void);
|
||||
/* Write a diagnostic message into a caller-supplied buffer, mirroring
|
||||
* vsnprintf. A NULL `err` or a zero `err_size` is a no-op, so a caller that
|
||||
* only needs the boolean status may safely pass NULL. Returns nothing; the
|
||||
* buffer is always NUL-terminated by vsnprintf when err_size > 0. */
|
||||
void utils_set_error(char* err, size_t err_size, const char* fmt, ...);
|
||||
/* True when `path` is `root` itself or lies directly beneath it: a lexical
|
||||
* prefix test requiring the byte after `root` to be '\0' or '/'. Both `root`
|
||||
* and `path` must be absolute canonical paths free of "."/".." components (the
|
||||
* callers guarantee this); this is containment by string, not by resolved
|
||||
* symlinks. Shared by the utils and file secure-walk root confinement. */
|
||||
bool path_is_within_root(const char* root, const char* path);
|
||||
/* Non-allocating transfer-relative view of `path`: strip any leading '/' and
|
||||
* then a `root` prefix (leading/trailing slashes tolerated), returning a
|
||||
* borrowed pointer into `path`. A NULL/empty root, or a path not under
|
||||
* `root`, yields just the leading-slash strip. `path`/`root` must stay alive. */
|
||||
const char* utils_strip_transfer_root(const char* path, const char* root);
|
||||
/* True when `path` contains a ".." component. This is a purely lexical
|
||||
* dot-dot check: an absolute path is NOT rejected here, because default
|
||||
* (non-relative) transfers legitimately put the sender's absolute source path
|
||||
|
||||
+40
-161
@@ -7,7 +7,6 @@
|
||||
#include "utils.h"
|
||||
#include "file_types.h"
|
||||
#include <errno.h>
|
||||
#include <limits.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
@@ -134,23 +133,16 @@ static bool xattr_name_is_posix_acl(const char* name) {
|
||||
|
||||
/* ---- SENDER: capture ---- */
|
||||
|
||||
/* The two syscall families differ only in whether the FINAL component is
|
||||
* followed (`listxattr`/`getxattr` follow; `llistxattr`/`lgetxattr` do not), so
|
||||
* one common implementation backs both public entry points. */
|
||||
typedef ssize_t (*XattrListFn)(const char* path, char* list, size_t size);
|
||||
typedef ssize_t (*XattrGetFn)(const char* path, const char* name, void* value, size_t size);
|
||||
|
||||
static FileXattrList* xattr_capture_common(const char* path, bool preserve_acls,
|
||||
XattrListFn list_fn, XattrGetFn get_fn) {
|
||||
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
|
||||
if (!path)
|
||||
return NULL;
|
||||
ssize_t list_size = list_fn(path, NULL, 0);
|
||||
ssize_t list_size = listxattr(path, NULL, 0);
|
||||
if (list_size <= 0)
|
||||
return NULL; /* no xattrs, ENOTSUP, or error: nothing appliable */
|
||||
char* names = malloc((size_t)list_size);
|
||||
if (!names)
|
||||
return NULL;
|
||||
ssize_t got = list_fn(path, names, (size_t)list_size);
|
||||
ssize_t got = listxattr(path, names, (size_t)list_size);
|
||||
if (got < 0) {
|
||||
free(names);
|
||||
return NULL;
|
||||
@@ -173,7 +165,7 @@ static FileXattrList* xattr_capture_common(const char* path, bool preserve_acls,
|
||||
negotiated. Without it a plain -X capture never carries an ACL. */
|
||||
if (!xattr_name_appliable(name, preserve_acls))
|
||||
continue;
|
||||
ssize_t value_size = get_fn(path, name, NULL, 0);
|
||||
ssize_t value_size = getxattr(path, name, NULL, 0);
|
||||
if (value_size < 0)
|
||||
continue;
|
||||
if (value_size > XATTR_VALUE_MAX)
|
||||
@@ -186,7 +178,7 @@ static FileXattrList* xattr_capture_common(const char* path, bool preserve_acls,
|
||||
free(names);
|
||||
return NULL;
|
||||
}
|
||||
ssize_t read_len = get_fn(path, name, buffer, (size_t)value_size);
|
||||
ssize_t read_len = getxattr(path, name, buffer, (size_t)value_size);
|
||||
if (read_len < 0 || read_len != value_size) {
|
||||
free(buffer);
|
||||
continue;
|
||||
@@ -208,14 +200,6 @@ static FileXattrList* xattr_capture_common(const char* path, bool preserve_acls,
|
||||
return list;
|
||||
}
|
||||
|
||||
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
|
||||
return xattr_capture_common(path, preserve_acls, listxattr, getxattr);
|
||||
}
|
||||
|
||||
FileXattrList* xattr_capture_path_nofollow(const char* path, bool preserve_acls) {
|
||||
return xattr_capture_common(path, preserve_acls, llistxattr, lgetxattr);
|
||||
}
|
||||
|
||||
/* ---- WIRE ---- */
|
||||
|
||||
bool xattr_send(int fd, const FileXattrList* list) {
|
||||
@@ -379,140 +363,29 @@ bool xattr_apply_fd(int fd, const FileXattrList* list) {
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Symlink counterpart of xattr_apply_fd(): target the link ITSELF, never its
|
||||
* referent. fsetxattr cannot be used (no *at xattr syscall exists, and the
|
||||
* kernel rejects xattr syscalls on an O_PATH descriptor), so the already-open,
|
||||
* confinement-checked parent directory is addressed through /proc/self/fd and
|
||||
* the final component is applied with lsetxattr, which does not follow it.
|
||||
*
|
||||
* The list is trusted to come from xattr_receive() (already whitelisted), but
|
||||
* every name is re-validated here so this path-based primitive is confined on
|
||||
* its own -- this is the only apply primitive that addresses a path, and the
|
||||
* header promises a whitelisted apply. The apply is best-effort: if /proc is
|
||||
* not mounted (the anchor cannot be formed) or the kernel refuses the set, the
|
||||
* failure is skipped and never fails the transfer. See xattr.h for the bounded
|
||||
* residual TOCTOU between link creation and lsetxattr. */
|
||||
bool xattr_apply_path_nofollow(int parent_fd, const char* leaf, const FileXattrList* list,
|
||||
bool preserve_acls) {
|
||||
if (parent_fd < 0 || !leaf || leaf[0] == '\0' || strchr(leaf, '/') != NULL || !list)
|
||||
return false;
|
||||
if (list->count == 0)
|
||||
return true;
|
||||
char prefix[64];
|
||||
int prefix_len = snprintf(prefix, sizeof(prefix), "/proc/self/fd/%d/", parent_fd);
|
||||
if (prefix_len < 0 || (size_t)prefix_len >= sizeof(prefix))
|
||||
return false;
|
||||
size_t leaf_len = strlen(leaf);
|
||||
char* path = malloc((size_t)prefix_len + leaf_len + 1);
|
||||
if (!path)
|
||||
return false;
|
||||
memcpy(path, prefix, (size_t)prefix_len);
|
||||
memcpy(path + prefix_len, leaf, leaf_len + 1);
|
||||
bool warned = false;
|
||||
int first_errno = 0;
|
||||
for (int i = 0; i < list->count; i++) {
|
||||
const FileXattr* xa = &list->items[i];
|
||||
/* Defense in depth: re-validate against the receiver's full whitelist, so a
|
||||
hand-crafted list can never apply a privileged namespace or the reserved
|
||||
--fake-super key through this path-based primitive. */
|
||||
if (!xattr_name_appliable(xa->name, preserve_acls))
|
||||
continue;
|
||||
if (lsetxattr(path, xa->name, xa->value, xa->value_len, 0) != 0) {
|
||||
if (!warned) {
|
||||
warned = true;
|
||||
first_errno = errno;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (warned)
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"could not set one or more xattrs on the destination symlink: %s",
|
||||
strerror(first_errno));
|
||||
free(path);
|
||||
return true;
|
||||
}
|
||||
/* ---- --fake-super: park ownership/mode/mtime in a reserved xattr ---- */
|
||||
|
||||
/* ---- --fake-super: park ownership/mode/rdev in a reserved xattr ---- */
|
||||
|
||||
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, uint32_t rdev_major,
|
||||
uint32_t rdev_minor) {
|
||||
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, int64_t mtime_sec,
|
||||
int64_t mtime_nsec) {
|
||||
if (fd < 0)
|
||||
return;
|
||||
/* rsync 3.4.1's exact grammar: "<octal full st_mode> <rdev_major>,<rdev_minor>
|
||||
* <uid>:<gid>". The octal mode carries the S_IFMT bits (e.g. 0104711 for a
|
||||
* setuid regular file, 020644 for a char device); the rdev pair is 0,0 for a
|
||||
* non-device. No mtime field: rsync leaves the file's own timestamp in
|
||||
* charge of mtime. This value is what rsync reads back to restore a
|
||||
* fake-super tree, so the field order and separators must not change. */
|
||||
char record[96];
|
||||
int len = snprintf(record, sizeof(record), "%o %u,%u %u:%u", (unsigned)mode, (unsigned)rdev_major,
|
||||
(unsigned)rdev_minor, (unsigned)uid, (unsigned)gid);
|
||||
char record[128];
|
||||
int len =
|
||||
snprintf(record, sizeof(record), "%lu:%lu:%03o:%lld:%ld", (unsigned long)uid,
|
||||
(unsigned long)gid, (unsigned)mode & 0777U, (long long)mtime_sec, (long)mtime_nsec);
|
||||
if (len <= 0 || (size_t)len >= sizeof(record))
|
||||
return;
|
||||
if (fsetxattr(fd, FAKESUPER_XATTR, record, (size_t)len, 0) != 0) {
|
||||
log_message(LOG_LEVEL_WARNING, "--fake-super: could not store %s on destination entry: %s",
|
||||
log_message(LOG_LEVEL_WARNING, "--fake-super: could not store %s on destination file: %s",
|
||||
FAKESUPER_XATTR, strerror(errno));
|
||||
}
|
||||
}
|
||||
|
||||
/* Parse rsync's `user.rsync.%stat` grammar strictly:
|
||||
* "<octal st_mode> <rdev_major>,<rdev_minor> <uid>:<gid>"
|
||||
* Every field is parsed with strtoul() so an out-of-range value is a clean
|
||||
* rejection rather than the undefined behavior sscanf("%u") exhibited, each
|
||||
* field is range-checked against the same bounds the wire validator uses, and
|
||||
* the whole record must be consumed (only trailing whitespace is tolerated) so
|
||||
* trailing garbage is refused. Returns false on any malformed input. */
|
||||
static bool fake_super_parse_stat(const char* record, unsigned* mode_out, unsigned* rdev_major_out,
|
||||
unsigned* rdev_minor_out, unsigned* uid_out, unsigned* gid_out) {
|
||||
if (!record)
|
||||
return false;
|
||||
char* end = NULL;
|
||||
const char* p = record;
|
||||
errno = 0;
|
||||
unsigned long mode = strtoul(p, &end, 8);
|
||||
if (errno != 0 || end == p || mode > (unsigned long)UINT_MAX || *end != ' ')
|
||||
return false;
|
||||
p = end + 1;
|
||||
errno = 0;
|
||||
unsigned long rdev_major = strtoul(p, &end, 10);
|
||||
if (errno != 0 || end == p || rdev_major > 0xffffUL || *end != ',')
|
||||
return false;
|
||||
p = end + 1;
|
||||
errno = 0;
|
||||
unsigned long rdev_minor = strtoul(p, &end, 10);
|
||||
if (errno != 0 || end == p || rdev_minor > 0x00ffffffUL || *end != ' ')
|
||||
return false;
|
||||
p = end + 1;
|
||||
errno = 0;
|
||||
unsigned long uid = strtoul(p, &end, 10);
|
||||
if (errno != 0 || end == p || uid > (unsigned long)UINT_MAX || *end != ':')
|
||||
return false;
|
||||
p = end + 1;
|
||||
errno = 0;
|
||||
unsigned long gid = strtoul(p, &end, 10);
|
||||
if (errno != 0 || end == p || gid > (unsigned long)UINT_MAX)
|
||||
return false;
|
||||
p = end;
|
||||
while (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r')
|
||||
p++;
|
||||
if (*p != '\0')
|
||||
return false;
|
||||
*mode_out = (unsigned)mode;
|
||||
*rdev_major_out = (unsigned)rdev_major;
|
||||
*rdev_minor_out = (unsigned)rdev_minor;
|
||||
*uid_out = (unsigned)uid;
|
||||
*gid_out = (unsigned)gid;
|
||||
return true;
|
||||
}
|
||||
|
||||
/* --fake-super replay: read the freshly-stored record and re-apply its
|
||||
* permission bits fd-relative. The recorded uid/gid are retained for a later
|
||||
* privileged restore but are NEVER chowned here: --fake-super only RECORDS
|
||||
* ownership, it must not real-chown the recorded (resolved) owner. The
|
||||
* recorded rdev is likewise parsed for grammar compatibility but is not acted
|
||||
* on (device recreation is a separate, privilege-gated path). mtime is not in
|
||||
* the record: the normal metadata path applies it (policy.times), exactly as
|
||||
* rsync relies on the file's own timestamp. */
|
||||
/* --fake-super replay: read the freshly-stored record and re-apply mode/mtime
|
||||
* fd-relative. The recorded uid/gid are retained for a later privileged
|
||||
* restore but are NEVER chowned here: --fake-super only RECORDS ownership, it
|
||||
* must not real-chown the recorded (resolved) owner. Mode/mtime still apply so
|
||||
* unprivileged --fake-super keeps working. */
|
||||
bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
|
||||
if (fd < 0)
|
||||
return false;
|
||||
@@ -521,25 +394,24 @@ bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
|
||||
if (len < 0)
|
||||
return false; /* absent or filesystem without xattrs: silent no-op */
|
||||
record[len] = '\0';
|
||||
unsigned ul_mode, rdev_major, rdev_minor, ul_uid, ul_gid;
|
||||
if (!fake_super_parse_stat(record, &ul_mode, &rdev_major, &rdev_minor, &ul_uid, &ul_gid))
|
||||
unsigned long ul_uid, ul_gid, ul_mode;
|
||||
long long mtime_sec;
|
||||
long mtime_nsec;
|
||||
if (sscanf(record, "%lu:%lu:%lo:%lld:%ld", &ul_uid, &ul_gid, &ul_mode, &mtime_sec, &mtime_nsec) !=
|
||||
5)
|
||||
return false; /* malformed record: skip, never fatal */
|
||||
|
||||
/* --fake-super NEVER performs a real chown: that would defeat the whole point
|
||||
of the flag (record privileged ownership on an unprivileged receiver for a
|
||||
later privileged restore). The uid/gid parsed above are retained in the
|
||||
record for that later restore, but no ownership change happens here. The
|
||||
rdev is retained for the same reason. */
|
||||
(void)rdev_major;
|
||||
(void)rdev_minor;
|
||||
/* --fake-super NEVER performs a real chown: that would defeat the whole
|
||||
point of the flag (record privileged ownership on an unprivileged receiver
|
||||
for a later privileged restore). The uid/gid parsed above are retained in
|
||||
the record for that later restore, but no ownership change happens here. */
|
||||
(void)ul_uid;
|
||||
(void)ul_gid;
|
||||
/* Mode is applied only when the per-attribute policy asks for it, through the
|
||||
SAME shared helper the normal metadata path uses (metadata_mode_for_policy).
|
||||
The recorded special bits are stripped first: rsync's fake-super receiver
|
||||
stores the full mode in the xattr but never installs setuid/setgid/sticky on
|
||||
the real file, so only the 0777 permission bits may be replayed. The -E
|
||||
rule then derives exec bits from the destination's read bits exactly like
|
||||
SAME shared helper the normal metadata path uses (metadata_mode_for_policy):
|
||||
under --perms the recorded source mode is copied exactly, including
|
||||
group/other write and setuid/setgid/sticky bits (rsync parity), and the -E
|
||||
rule derives exec bits from the destination's read bits exactly like
|
||||
file_restore_metadata_fd. */
|
||||
if (policy.perms || policy.executability) {
|
||||
struct stat cur;
|
||||
@@ -547,12 +419,19 @@ bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
|
||||
if (fstat(fd, &cur) != 0) {
|
||||
log_message(LOG_LEVEL_WARNING, "--fake-super: could not read destination mode: %s",
|
||||
strerror(errno));
|
||||
} else if (metadata_mode_for_policy((mode_t)(ul_mode & 0777U), cur.st_mode, policy, &want)) {
|
||||
} else if (metadata_mode_for_policy((mode_t)ul_mode, cur.st_mode, policy, &want)) {
|
||||
if (fchmod(fd, want) != 0)
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"--fake-super: could not restore mode on destination entry: %s",
|
||||
"--fake-super: could not restore mode on destination file: %s",
|
||||
strerror(errno));
|
||||
}
|
||||
}
|
||||
if (policy.times) {
|
||||
struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
|
||||
{.tv_sec = (time_t)mtime_sec, .tv_nsec = mtime_nsec}};
|
||||
if (futimens(fd, times) != 0)
|
||||
log_message(LOG_LEVEL_WARNING,
|
||||
"--fake-super: could not restore mtime on destination file: %s", strerror(errno));
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
+23
-77
@@ -26,22 +26,16 @@
|
||||
* and total bytes) on BOTH ends to prevent OOM/memory abuse; an oversized
|
||||
* or malformed frame is a clean protocol rejection, never an allocation
|
||||
* blowup.
|
||||
* * Application is confined to the exact destination entry: fsetxattr on the
|
||||
* just-written fd for regular files/directories, and for a symlink an
|
||||
* lsetxattr on "/proc/self/fd/<parent_fd>/<leaf>" reached through the
|
||||
* already-opened, confinement-checked parent directory -- never a
|
||||
* caller-controlled path, and never following the link.
|
||||
* * Application is confined to the exact destination file descriptor
|
||||
* (fsetxattr on the just-written fd), never a caller-controlled path.
|
||||
*/
|
||||
|
||||
/* Reserved key used by --fake-super to park the source's privileged ownership
|
||||
* / mode / rdev on the destination file as an unprivileged user.* xattr, so the
|
||||
* tree is interoperable with rsync 3.4.1 and a later privileged restore can
|
||||
* re-apply them. This is rsync's own key and value grammar exactly:
|
||||
* <octal st_mode with S_IFMT> <rdev_major>,<rdev_minor> <uid>:<gid>
|
||||
* e.g. "104711 0,0 1234:5678" for a setuid regular file owned by 1234:5678,
|
||||
* or "20644 1,3 111:222" for a char device. mtime is deliberately NOT part of
|
||||
* the record: exactly like rsync, the file's own timestamp carries it. */
|
||||
#define FAKESUPER_XATTR "user.rsync.%stat"
|
||||
* / mode / mtime on the destination file as an unprivileged user.* xattr, so a
|
||||
* later privileged restore could re-apply them. Exact documented format:
|
||||
* uid:gid:mode:mtime_sec:mtime_nsec (decimal, decimal, octal, dec, dec)
|
||||
* e.g. "1000:1000:644:1765238400:0". */
|
||||
#define FAKESUPER_XATTR "user.fastsync.stat"
|
||||
|
||||
/* --- bounds --- */
|
||||
#define XATTR_NAME_MAX 255 /* xattr names are limited to 255 bytes */
|
||||
@@ -82,17 +76,6 @@ bool xattr_name_appliable(const char* name, bool preserve_acls);
|
||||
* distinct from NULL. */
|
||||
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls);
|
||||
|
||||
/* Sender: like xattr_capture_path() but reads the xattrs of `path` ITSELF,
|
||||
* never following a final symlink (llistxattr/lgetxattr). A symlink entry must
|
||||
* use this so the scanner never captures the REFERENT's attributes onto the
|
||||
* link (the path-following variant would). On Linux the VFS refuses to
|
||||
* associate xattrs with symlinks at all, so this normally returns NULL; it is
|
||||
* still correct and portable for a filesystem/platform that supports them.
|
||||
* The same whitelist/bounds as xattr_capture_path() apply. Returns NULL when
|
||||
* the link has no appliable xattrs (or the filesystem does not support them);
|
||||
* an empty-but-valid list is never returned distinct from NULL. */
|
||||
FileXattrList* xattr_capture_path_nofollow(const char* path, bool preserve_acls);
|
||||
|
||||
/* Wire: bounded serialization. xattr_send returns false on write failure; an
|
||||
* empty/NULL list transmits a zero-count block. xattr_receive returns NULL and
|
||||
* sets *ok = 0 on any malformed / oversized / non-whitelisted entry. When
|
||||
@@ -108,61 +91,24 @@ FileXattrList* xattr_receive(int fd, int* ok, bool preserve_acls);
|
||||
* true when apply was attempted (allowing callers to treat it as best-effort). */
|
||||
bool xattr_apply_fd(int fd, const FileXattrList* list);
|
||||
|
||||
/* Receiver: apply every entry to the symlink named by (parent_fd, leaf) WITHOUT
|
||||
* following it, via lsetxattr() on the confined path
|
||||
* "/proc/self/fd/<parent_fd>/<leaf>". Every incoming name is independently
|
||||
* re-validated against xattr_name_appliable() with `preserve_acls`, exactly like
|
||||
* xattr_apply_fd(): a non-whitelisted namespace (including the reserved
|
||||
* --fake-super key) is skipped, so this primitive stays confined even if handed
|
||||
* a hand-crafted list. A symlink cannot be targeted by the fd-relative
|
||||
* fsetxattr() path: there is no *at() xattr syscall and the kernel rejects
|
||||
* xattr syscalls on an O_PATH descriptor, so the already-opened,
|
||||
* confinement-checked parent directory is the anchor and only the final
|
||||
* component is the (no-follow) link. `leaf` must be a single path component.
|
||||
*
|
||||
* Portability: the "/proc/self/fd/<parent_fd>" anchor requires a mounted /proc.
|
||||
* Where /proc is unavailable (or the fd cannot be addressed that way) the
|
||||
* lsetxattr simply fails and is skipped -- the apply is best-effort exactly like
|
||||
* xattr_apply_fd(), so no error is propagated and the transfer continues. A
|
||||
* per-attribute failure (on Linux every set on a symlink fails with EPERM) is
|
||||
* logged once and skipped, never fatal. Returns false only for an invalid
|
||||
* anchor/list; true when an apply was attempted.
|
||||
*
|
||||
* Residual TOCTOU: `leaf` is a caller-supplied name resolved by path in the
|
||||
* parent, so a local writer could replace the just-created symlink between its
|
||||
* creation and lsetxattr(). This is bounded: it requires write access to the
|
||||
* confinement-checked destination directory (already trusted), can only install
|
||||
* a whitelisted user namespace or POSIX-ACL name, and never follows the link (a
|
||||
* replacement symlink is still applied to as the final, no-follow component). */
|
||||
bool xattr_apply_path_nofollow(int parent_fd, const char* leaf, const FileXattrList* list,
|
||||
bool preserve_acls);
|
||||
|
||||
/* --fake-super: write the source uid/gid/mode/rdev record into the reserved
|
||||
* FAKESUPER_XATTR on `fd`, using rsync 3.4.1's exact grammar (see the key
|
||||
* comment above). `mode` is the full st_mode including its S_IFMT bits.
|
||||
* `fd` may be a regular file, a faked char/block device (written as a regular
|
||||
* file), or a DIRECTORY: rsync stores a directory's faked mode/uid/gid in the
|
||||
* reserved xattr on the directory itself. Best-effort (logged, never fatal).
|
||||
* Only meaningful when metadata was transmitted so the values exist. */
|
||||
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, uint32_t rdev_major,
|
||||
uint32_t rdev_minor);
|
||||
/* --fake-super: write the source uid/gid/mode/mtime record into the reserved
|
||||
* FAKESUPER_XATTR on `fd`. Best-effort (logged, never fatal). Only meaningful
|
||||
* when metadata was transmitted so the values exist. */
|
||||
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, int64_t mtime_sec,
|
||||
int64_t mtime_nsec);
|
||||
|
||||
/* --fake-super replay: parse the FAKESUPER_XATTR record previously written on
|
||||
* `fd` by fake_super_store_fd and re-apply the recorded permission bits
|
||||
* fd-relative. `fd` may be a regular file, a faked device, or a DIRECTORY;
|
||||
* fgetxattr/fchmod work identically on a directory descriptor. The recorded
|
||||
* uid/gid are deliberately NOT chowned for real: --fake-super only RECORDS
|
||||
* ownership (the caller stores the resolved mapping via
|
||||
* identity_resolve_storage_ids), it never performs a real chown. The
|
||||
* recorded rdev is retained for a later privileged restore but is not acted on
|
||||
* here. Best-effort: absence of the xattr or a malformed record is a silent
|
||||
* no-op that never fails the transfer. The MODE leg is applied only when
|
||||
* policy.perms||policy.executability, and the recorded special bits
|
||||
* (setuid/setgid/sticky) are NOT applied to the real entry -- exactly like
|
||||
* rsync's fake-super receiver, which stores the full mode in the xattr but
|
||||
* strips the special bits on disk. mtime is not part of the record; the normal
|
||||
* metadata path carries it (policy.times) exactly as rsync sets the file's own
|
||||
* timestamp. Returns true when the xattr was present and parsed. */
|
||||
* `fd` by fake_super_store_fd and re-apply mode/mtime fd-relative. The
|
||||
* recorded uid/gid are deliberately NOT chowned for real: --fake-super only
|
||||
* RECORDS ownership (the caller stores the resolved mapping via
|
||||
* identity_resolve_storage_ids), it never performs a real chown. Best-effort:
|
||||
* absence of the xattr or a malformed record is a silent no-op that never fails
|
||||
* the transfer. The MODE leg is applied only when policy.perms||policy.
|
||||
* executability and the MTIME leg only when policy.times, so the fake-super
|
||||
* replay cannot bypass the per-attribute split; the mode follows the normal
|
||||
* metadata path exactly (under --perms the source mode is copied verbatim,
|
||||
* special and group/other write bits included).
|
||||
* Returns true when the xattr was present and parsed. */
|
||||
bool fake_super_restore_fd(int fd, FileAttrPolicy policy);
|
||||
|
||||
#endif
|
||||
@@ -1 +0,0 @@
|
||||
hello
|
||||
@@ -1 +0,0 @@
|
||||
x
|
||||
@@ -1 +0,0 @@
|
||||
y
|
||||
@@ -1 +0,0 @@
|
||||
a.txt
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user