64 Commits
Author SHA1 Message Date
TapTap 7585f46eb4 Release v2.29.0 (#312)
CI / lint (push) Successful in 1m50s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 21s
CI / sanitizers (address) (push) Successful in 55s
CI / sanitizers (undefined) (push) Successful in 43s
CI / build-and-test (push) Successful in 1m25s
CI / fuzz-build (push) Successful in 51s
CI / coverage (push) Successful in 46s
CI / valgrind (push) Successful in 2m41s
2026-09-23 02:05:14 +02:00
TapTap 00197102bf Release v2.29.0
CI / lint (push) Successful in 1m48s
CI / parity-fast (push) Skipped
CI / lint (pull_request) Successful in 1m48s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-full (push) Successful in 19s
CI / sanitizers (address) (push) Successful in 50s
CI / sanitizers (undefined) (push) Successful in 44s
CI / build-and-test (push) Successful in 1m19s
CI / coverage (push) Successful in 45s
CI / fuzz-build (push) Successful in 54s
CI / parity-fast (pull_request) Successful in 20s
CI / build-and-test (pull_request) Successful in 56s
CI / valgrind (push) Successful in 2m43s
- Transport I/O vtable over TCP/TLS; TLS multithreaded sendfile fixed
- Symlink-xattr wire block (protocol 2.29.0; config frame unchanged)
- No-wire parity burn-down: --inc-recursive, in-root --temp-dir,
  --delete-before phase-0, rsync-interoperable --fake-super, --devices
  per-entry failure, rsyncd.conf key subset (read-only default)
- Protocol version: 2.29.0
- Tested: unit, integration, ASan, UBSan, valgrind, differential parity
2026-09-23 01:59:54 +02:00
TapTap 13b257d1dd Merge PR #311: transport I/O vtable + symlink-xattr wire block (2.29.0)
CI / lint (push) Successful in 1m48s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 20s
CI / sanitizers (address) (push) Successful in 54s
CI / sanitizers (undefined) (push) Successful in 45s
CI / build-and-test (push) Successful in 1m20s
CI / fuzz-build (push) Successful in 48s
CI / coverage (push) Successful in 46s
CI / valgrind (push) Successful in 2m35s
2026-09-23 01:50:51 +02:00
TapTap f3d7672694 docs: bump release version to 2.29.0 and reconcile protocol docs
CI / lint (pull_request) Successful in 1m48s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 20s
CI / build-and-test (pull_request) Successful in 57s
2026-09-23 01:45:30 +02:00
TapTap 18b821d32c Merge branch 'feat/transport-g2' into feat/transport-xattr 2026-09-23 01:36:25 +02:00
TapTap 6ad065925f Merge branch 'feat/transport-g1' into feat/transport-xattr 2026-09-23 01:36:25 +02:00
TapTap 46dcefe218 fix(protocol): classify TLS EOF before EINTR retry; harden current-ssl resolver; real TLS regression test 2026-09-23 01:36:08 +02:00
TapTap b88acdbd3c fix(xattr): whitelist path-based symlink apply; strengthen symlink xattr tests 2026-09-23 01:01:18 +02:00
TapTap c29bce54fc Merge branch 'feat/transport-c2' into feat/transport-xattr 2026-09-23 00:35:05 +02:00
TapTap d98e971fcc Merge branch 'feat/transport-c1' into feat/transport-xattr 2026-09-23 00:35:05 +02:00
TapTap 492ce0ce89 feat(xattr): carry and apply symlink xattrs (protocol 2.29.0) 2026-09-23 00:34:39 +02:00
TapTap ec6692ac41 protocol: add transport I/O vtable over TCP/TLS primitives
Introduce ProtocolIoOps (send/recv/has_pending), selected once by
protocol_session_init() and protocol_session_set_ssl(), and dispatch the
send, receive and status-read loops through session->ops instead of
branching on session->ssl at runtime.

Each op performs one transfer attempt and classifies the result
(PROTOCOL_IO_RETRY/CLOSED/ERROR), preserving the WANT_READ/WANT_WRITE
wait_events switching, the SSL_ERROR_SYSCALL/EINTR retry, the
SSL_pending poll gating and the deadline handling. The raw read()/write()
fallback lives in the plaintext ops.

Add unit tests: a socketpair session with a counting ops wrapper proving
the loops dispatch through the vtable, and a worker-thread test that
protocol_current_ssl() resolves the bound session's SSL when io_ssl is NULL.
2026-09-23 00:18:36 +02:00
TapTap 1f8d60e30d protocol: resolve TLS transport from the bound session, not thread-local io_ssl
file_send.c chose between sendfile() and the TLS-aware buffered path by
calling io_get_ssl(), which reads the thread-local io_ssl. A worker thread
that bound a TLS ProtocolSession via protocol_session_bind() never ran the
handshake in that thread, so io_ssl is NULL there and a TLS + --threads
transfer took the raw sendfile() path on an encrypted socket.

Add protocol_current_ssl(), which prefers the bound session's SSL and falls
back to io_ssl on the fd-shim path, and use it in file_send.c. Un-xfail
test_tls_with_multithreading.
2026-09-23 00:18:29 +02:00
TapTap 6db9827b87 Merge PR #310: restore dumpable flag for ASan/LSan
CI / lint (push) Successful in 1m46s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 21s
CI / sanitizers (address) (push) Successful in 54s
CI / sanitizers (undefined) (push) Successful in 43s
CI / build-and-test (push) Successful in 1m22s
CI / fuzz-build (push) Successful in 48s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Successful in 2m17s
2026-09-22 23:50:54 +02:00
TapTap a07cc00bb0 test: restore dumpable flag after setuid drop so LSan can run under ASan
CI / lint (pull_request) Successful in 1m48s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 18s
CI / build-and-test (pull_request) Successful in 55s
2026-09-22 23:45:41 +02:00
TapTap 3ae7685655 Merge PR #309: parity burn-down
CI / lint (push) Successful in 1m46s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 22s
CI / sanitizers (address) (push) Failing after 54s
CI / sanitizers (undefined) (push) Successful in 43s
CI / build-and-test (push) Successful in 1m22s
CI / fuzz-build (push) Successful in 49s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Successful in 2m19s
2026-09-22 23:33:21 +02:00
TapTap 4f945a8e39 docs: reconcile RSYNC_COMPAT for parity-next review follow-ups
CI / lint (pull_request) Successful in 1m46s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 17s
CI / build-and-test (pull_request) Successful in 54s
2026-09-22 23:27:54 +02:00
TapTap 15f38f5b76 Merge branch 'feat/parity-f4' into feat/parity-next 2026-09-22 23:20:21 +02:00
TapTap 787967d3ce Merge branch 'feat/parity-f2' into feat/parity-next 2026-09-22 23:20:21 +02:00
TapTap 06e7aef5c9 Merge branch 'feat/parity-f1' into feat/parity-next 2026-09-22 23:20:21 +02:00
TapTap ee265d78ba fix(delete-before): replay pre-scan list in --threads path 2026-09-22 23:19:55 +02:00
TapTap 47b1b9b915 fix(xattr): fake-super device round-trip, --devices continue-on-error, harden stat parse 2026-09-22 23:05:40 +02:00
TapTap bb6c788cf9 fix(daemon): rsync read-only module default; warn on unenforced security keys 2026-09-22 22:53:23 +02:00
TapTap 0b40c47d6a Merge branch 'feat/parity-b4' into feat/parity-next
# Conflicts:
#	RSYNC_COMPAT.md
2026-09-22 22:08:16 +02:00
TapTap 6f81005094 Merge branch 'feat/parity-b3' into feat/parity-next
# Conflicts:
#	RSYNC_COMPAT.md
2026-09-22 22:07:45 +02:00
TapTap 193d358c64 Merge branch 'feat/parity-b2' into feat/parity-next 2026-09-22 22:06:25 +02:00
TapTap 8792e3265a Merge branch 'feat/parity-b1' into feat/parity-next 2026-09-22 22:06:25 +02:00
TapTap 3ec0ffb644 fix(delete-before): reuse pre-scan file list in single-threaded data pass 2026-09-22 22:05:50 +02:00
TapTap 80e8d7f450 feat(xattr): rsync-interoperable --fake-super stat; fix --devices error parity 2026-09-22 22:03:41 +02:00
TapTap 86741725fd feat(daemon): accept rsync rsyncd.conf key subset and --dparam mapping 2026-09-22 22:02:09 +02:00
TapTap f96f1764af feat(cli): accept --inc-recursive no-op and in-root absolute --temp-dir 2026-09-22 21:58:54 +02:00
TapTap d780a4625e Merge PR #308: close valid residuals of issues #286-#297
CI / lint (push) Successful in 1m45s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 21s
CI / sanitizers (address) (push) Successful in 52s
CI / sanitizers (undefined) (push) Successful in 42s
CI / build-and-test (push) Successful in 1m16s
CI / fuzz-build (push) Successful in 47s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Successful in 2m18s
2026-09-22 17:30:36 +02:00
TapTap 5d1303ffdf Merge branch 'fix/issues-f4' into fix/issues-triage
CI / lint (pull_request) Successful in 1m45s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 17s
CI / build-and-test (pull_request) Successful in 54s
2026-09-22 17:18:59 +02:00
TapTap 06b53c5b3d Merge branch 'fix/issues-f3' into fix/issues-triage 2026-09-22 17:18:59 +02:00
TapTap bf09e8893e Merge branch 'fix/issues-f2' into fix/issues-triage 2026-09-22 17:18:59 +02:00
TapTap 034c27926f Merge branch 'fix/issues-f1' into fix/issues-triage 2026-09-22 17:18:59 +02:00
TapTap aa15099d22 fix(output): restore --info=flist header; dir metadata on plan path; root line for -d 2026-09-22 17:17:04 +02:00
TapTap ff4db23831 fix(identity): free TO name on glob success path; harden test oracle 2026-09-22 17:08:03 +02:00
TapTap 79e45441c0 fix(receiver): defer --dirs directory mode to avoid EACCES on children
file_save_directory_to_disk() applied the exact source mode (fchmod)
inline for explicit --dirs/STATUS_MKDIR entries.  A restrictive source
mode (e.g. 0555) then made the directory read-only before its children
were written, so a non-root receiver failed each child with EACCES.  The
recursive -a path never hit this because it defers directory metadata.

Remove the inline fchmod and let the existing deferred
dir_metadata_list_apply() stamp the exact mode at end of transfer, as the
recursive path does.  Keep the inline ownership and xattrs (a direct
file_save_to_disk_full() caller has no deferred pass) and document the
resulting intentional ordering.  Capture errno before output_escape() in
the inline timestamp diagnostic so strerror() reports the real error, and
add the missing trailing newline to tests/test_xattr.c.
2026-09-22 17:06:54 +02:00
TapTap 6537227467 test(transport): de-race fallback/fd tests and gate for valgrind 2026-09-22 17:05:54 +02:00
TapTap 309c9aed98 refactor: const-correct dir/root locals in progress itemize 2026-09-22 16:46:23 +02:00
TapTap 84594197ee Merge branch 'fix/issues-b1' into fix/issues-triage 2026-09-22 16:37:56 +02:00
TapTap 7bf25048f6 docs: correct parity claims for issues #286-#297 2026-09-22 16:37:21 +02:00
TapTap b6b5eee20e Merge branch 'fix/issues-a3' into fix/issues-triage 2026-09-22 16:20:55 +02:00
TapTap 8dcd87609d Merge branch 'fix/issues-a2' into fix/issues-triage 2026-09-22 16:20:55 +02:00
TapTap 19fe63bd59 Merge branch 'fix/issues-a1' into fix/issues-triage 2026-09-22 16:20:55 +02:00
TapTap 1f5f8dc5a7 fix(output): emit directory/root lines for -i and --out-format (#292) 2026-09-22 16:20:03 +02:00
TapTap 3787695ba6 fix(xattr): apply --dirs directory xattrs fd-relative (#286) 2026-09-22 16:05:18 +02:00
TapTap b7cb213c8c fix: FROM name globs for identity maps; transport fallback tests (#294, #219) 2026-09-22 16:03:18 +02:00
TapTap 8cc3dd993b Merge PR #307: structural refactor cycle (no behavior change)
CI / lint (push) Successful in 1m42s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 21s
CI / sanitizers (address) (push) Successful in 53s
CI / sanitizers (undefined) (push) Successful in 43s
CI / build-and-test (push) Successful in 1m16s
CI / fuzz-build (push) Successful in 48s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Successful in 2m18s
2026-09-22 14:58:26 +02:00
TapTap 1167e7970b refactor(delete): rename basis helper to delete_basis_relative
CI / lint (pull_request) Successful in 1m43s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 19s
CI / build-and-test (pull_request) Successful in 57s
2026-09-22 14:52:36 +02:00
TapTap e98729f00e refactor: const-correct delete-manifest API; apply clang-format 2026-09-22 14:24:39 +02:00
TapTap c0020364b2 Merge branch 'refactor/minor' into refactor/structural 2026-09-22 14:06:55 +02:00
TapTap d7ac940a6b Merge branch 'refactor/cfg' into refactor/structural 2026-09-22 14:06:55 +02:00
TapTap be20e836de fix: correct throttle legacy resolution; add EXDEV temp-dir coverage 2026-09-22 14:06:26 +02:00
TapTap b549887138 refactor(config): group CLI-parse state; drop old_args field 2026-09-22 14:03:10 +02:00
TapTap 1ffd4744c6 Merge branch 'refactor/delete' into refactor/structural 2026-09-22 13:53:20 +02:00
TapTap 494cef2a0b Merge branch 'refactor/pending' into refactor/structural 2026-09-22 13:53:20 +02:00
TapTap ed7527cc2c Merge branch 'refactor/handler' into refactor/structural 2026-09-22 13:53:20 +02:00
TapTap 934defa965 refactor(delete): consolidate delete engine into delete.c 2026-09-22 13:52:57 +02:00
TapTap ade9be8600 refactor(receiver): per-status dispatch and shared pending teardown 2026-09-22 13:49:28 +02:00
TapTap 707bb659e8 refactor(server): decompose handler into phases 2026-09-22 13:46:17 +02:00
TapTap b4d54504f9 Release v2.28.0 (#304)
CI / lint (push) Successful in 2m1s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 19s
CI / sanitizers (address) (push) Successful in 50s
CI / sanitizers (undefined) (push) Successful in 43s
CI / build-and-test (push) Successful in 1m8s
CI / fuzz-build (push) Successful in 45s
CI / coverage (push) Successful in 43s
CI / valgrind (push) Successful in 2m13s
2026-09-20 01:17:25 +02:00
TapTap ef76c9034d Merge pull request 'Release v2.26.0' (#284) from dev into main
CI / lint (push) Successful in 1m40s
CI / sanitizers (undefined) (push) Successful in 44s
CI / sanitizers (address) (push) Successful in 50s
CI / build-and-test (push) Successful in 58s
CI / coverage (push) Successful in 40s
CI / fuzz-build (push) Successful in 46s
CI / valgrind (push) Successful in 2m12s
Reviewed-on: #284
2026-09-18 19:05:51 +02:00
65 changed files with 6292 additions and 2008 deletions
+1 -1
View File
@@ -16,7 +16,7 @@ Ask the user or determine from context:
- **Minor** (x.Y.0) — new features, backward compatible
- **Patch** (x.y.Z) — bug fixes, no protocol changes
Current version: `PROTOCOL_VERSION "2.26.0"` in `src/shared/config.h`
Current version: `PROTOCOL_VERSION "2.29.0"` in `src/shared/config.h`
### Step 2: Check Protocol Version
+71 -1
View File
@@ -6,6 +6,8 @@ run the same version because the handshake is strict.
## [Unreleased]
## [2.29.0] - 2026-09-23
The rsync-parity cycle 2.29 (no wire change; `PROTOCOL_VERSION` stays 2.28.0).
`RSYNC_COMPAT.md` moves from **116 ✅ / 14 ⚠️ / 27 ❌** to
**120 ✅ / 10 ⚠️ / 27 ❌** of 157 rows.
@@ -21,7 +23,31 @@ reclassification is `--filter=RULE` moving ✅ → ⚠️, because its merge-onl
`e`/`n`/`w`/`-` modifiers are now accepted and consumed but their semantics
remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ /
11 ⚠️ / 27 ❌** of 157 rows. The affected rows' notes and the summary tally in
`RSYNC_COMPAT.md` were updated.
`RSYNC_COMPAT.md` were updated. A following triage-fix cycle (see **Triage
fixes** below) moves `-F` and `-i` to ⚠️, for a final **117 ✅ / 13 ⚠️ / 27 ❌**
of 157 rows.
A no-wire parity burn-down cycle follows on 2.28.0: it accepts
`--inc-recursive`/`--no-inc-recursive` as inert no-ops, accepts an absolute
`--temp-dir` that canonicalizes inside the receive root, closes the
`--delete-before` phase-0 divergence (both the single-threaded and `--threads`
data passes replay the pre-scan list), makes `--fake-super` interoperable with
rsync's `user.rsync.%stat` key/grammar (regular files and char/block devices
faked as regular files), turns a failed device `mknod` into a continuing
per-entry failure, and accepts a practical subset of rsync's `rsyncd.conf`
grammar (modules are read-only by default, and accepted-but-unenforced
access-control keys emit a startup warning). The matrix moves to **119 ✅ /
14 ⚠️ / 24 ❌** of 157 rows.
A structural cycle then lands a transport I/O vtable over TCP/TLS (fixing the
TLS-multithreaded sendfile path and making the per-thread SSL resolution
explicit) and bumps the wire to **2.29.0**: the `STATUS_SYMLINK` frame grows an
optional symlink-xattr block (captured no-follow with `llistxattr`/`lgetxattr`,
applied no-follow with `lsetxattr`). Because the handshake is strict, 2.28.0 and
2.29.0 peers are incompatible. Note: Linux refuses to associate xattrs with a
symlink at all, so the symlink-xattr block is a no-op on Linux and is carried
for correctness on platforms/filesystems that do support it; the config-frame
layout is unchanged (golden length still 886).
### Changed
@@ -77,6 +103,24 @@ remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ /
- **Daemon umask no longer forced to `0`.** `daemonize()` now sets the
conventional `022`, so implied parent directories created without `-p` are no
longer world-writable `0777`.
- **Daemon modules are read-only by default.** A `--daemon` module is now
served read-only unless it sets `read only = no` (or rsync's `write only =
yes`), matching rsync: a real `rsyncd.conf` that omits `read only` is no
longer silently writable. A global `read only` still sets the default for
later modules, and an explicit module value wins. This is a behavior change
for existing FastSync-native configs that relied on the old writable default;
add `read only = no` to keep them writable. An rsync `write only = yes` is
mapped to writability (FastSync is push-only, so a module can never be read
from the network).
- **Accepted-but-unenforced rsync security keys now warn at startup.** The
rsync keys FastSync recognizes but does not implement — `secrets file`,
`refuse options`, `exclude`/`include`/`filter`, `max size`/`min size`,
`pre-xfer exec`/`post-xfer exec`, `incoming chmod`/`outgoing chmod`,
`name converter`, `use chroot`, `uid`/`gid`, and the rest of the
access-control set — load for migration compatibility but now emit a
`WARN` naming the key (and module) so an operator does not believe the
restriction is enforced. `auth users`/`secrets file` stay fail-closed: a
module declaring `auth users` still requires a FastSync credential store.
- **Credentials and signal handling hardened.** Secret files are opened with
`O_NOFOLLOW|O_NONBLOCK` (while allowing fd-backed store paths and bound-waiting
a FIFO read for ~3 s so a slow process substitution works but a connected-but-
@@ -135,6 +179,32 @@ remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ /
`CHANGELOG.md` and `HANDOFF.md` were updated for the audit cycle; the
`RSYNC_COMPAT.md` summary tally was corrected to match the rows.
### Triage fixes
- **`--dirs` directory xattrs applied inline.** A `-d/--dirs` transfer now
applies captured directory `-X`/`-A` xattrs fd-relative on the directory entry
instead of dropping them, so directory xattrs survive the non-recursive path
(`src/shared/file_save.c`, `tests/test_xattr.c`).
- **Directory/root itemize and `--out-format` lines.** `-i`/`--itemize-changes`
and `--out-format` now emit the transfer-root `./` line and per-directory
`cd...`/`.d..t...` lines, rendered by the shared itemize code. This matches
rsync's fresh-transfer output; because the root line is unconditional and an
incremental re-run may itemize directories/symlinks that rsync's quick-check
leaves silent, `-i` is now a ⚠️ Caveat row.
- **FROM name globs for identity maps.** `--usermap`/`--groupmap` `FROM` tokens
now accept `*`/`?`/`[...]` globs, expanded sender-side against the passwd/group
database and collapsed into bounded numeric ranges (`MAX_IDENTITY_MAP`),
matching rsync.
- **Transport fallback unit tests.** Added unit coverage for the TCP/TLS
transport fallback paths (`tests/test_transport_tcp.c`,
`tests/test_transport_tls.c`).
- **Docs corrections.** `RSYNC_COMPAT.md`/`README.md` corrected stale parity
claims for issues #286–#297: the `-F` and `-i` reclassifications, the
`--munge-links` direction, the accepted checksum/compression name sets,
`--bwlimit` parsing, `--stop-at` grammar, `--trust-sender`, symlink xattrs, and
the native/non-interoperable batch and credential notes. The summary tally is
now **117 ✅ / 13 ⚠️ / 27 ❌** of 157 rows.
## [2.28.0] - 2026-09-20
The rsync-parity cycle. `PROTOCOL_VERSION` moves `2.26.0 → 2.27.0 → 2.28.0`;
+2 -1
View File
@@ -1,6 +1,6 @@
cmake_minimum_required(VERSION 3.22)
project(FastFileTransfer VERSION 2.28.0)
project(FastFileTransfer VERSION 2.29.0)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_C_STANDARD 11)
@@ -99,6 +99,7 @@ set(SHARED_SRCS
src/shared/daemon_limits.c
src/shared/data.c
src/shared/delay_updates.c
src/shared/delete.c
src/shared/delete_commit.c
src/shared/delete_plan.c
src/shared/delta.c
+36 -26
View File
@@ -187,7 +187,7 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `--groupmap=MAP` | Map group names when applying ownership |
| `--numeric-ids` | Apply source numeric uid/gid directly instead of mapping by name |
| `--copy-as=USER[:GROUP]` | Force every written entry to USER[:GROUP] (requires a privileged receiver) |
| `--fake-super` | Record the resolved owner plus mode/time in a reserved `user.fastsync.stat` xattr and replay mode/time; never performs a real chown |
| `--fake-super` | Record the resolved owner plus full mode/rdev in rsync's reserved `user.rsync.%stat` xattr (rsync 3.4.1 grammar) and replay the permission bits; never performs a real chown |
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes) |
| `-D` | Preserve device and special files (implies `--devices --specials`) |
| `--devices` | Recreate device nodes on the destination (privileged; skipped without `CAP_MKNOD`) |
@@ -219,7 +219,7 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication; the fixed `.fastsync-stage` staging name diverges from rsync — see [`RSYNC_COMPAT.md`](RSYNC_COMPAT.md)) |
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install; confined to the receive root (relative only), with an `EXDEV` non-atomic copy fallback |
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install; confined to the receive root (a relative path resolves below it; an absolute path is accepted only when it canonicalizes inside it), with an `EXDEV` non-atomic copy fallback |
| `-n, --dry-run` | Report what would be transferred without mutating the destination. Since protocol 2.21.0 a server-routed target contacts the receiver and reports would-transfer based on receiver state; a plain local destination keeps the client-side scan. Never mutates or deletes. |
| `-v, --verbose` | Enable debug logging |
| `-q, --quiet` | Suppress non-error output |
@@ -233,9 +233,9 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `-h, --human-readable` | Format transfer byte/rate counts with rsync's decimal (base-1000) units |
| `--max-depth <n>` | Maximum directory depth to recurse (0 = unlimited, default: 0) |
| `--log-file <path>` | Write log messages to file instead of stderr |
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree |
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server) |
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server) |
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree (FastSync-native format, not rsync-interoperable) |
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server); FastSync-native format, not rsync-interoperable |
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server); FastSync-native format, not rsync-interoperable |
| `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
| `--dest-dir <path>` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) |
| `--save-to-disk` | Write received files to disk |
@@ -248,12 +248,12 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `-4, --ipv4` | Force IPv4 for destination resolution |
| `-6, --ipv6` | Force IPv6 for destination resolution |
| `--sockopts=OPTS` | Comma-separated OPT=VAL socket options applied before connect (`TCP_NODELAY`, `SO_KEEPALIVE`, `SO_RCVBUF`, `SO_SNDBUF`, `SO_REUSEADDR`) |
| `--bwlimit <KB/s>` | Bandwidth limit in kilobytes per second; also paces `--sendfile` transfers |
| `--bwlimit <RATE>` | Bandwidth limit, using rsync's exact `parse_size_arg` grammar: a bare value is KiB/s; `K`/`M`/`G`/`T`/`P` are binary suffixes; `KB`/`MB` are decimal and `KiB`/`MiB` binary; decimals are accepted and quantized to whole KiB; `0` (or empty) means no limit. Also paces `--sendfile` transfers |
| `--chunk-size <n>` | Chunk size in bytes (default: 10485760) |
| `--timeout <sec>` | I/O timeout in seconds, applied to both the socket (`SO_RCVTIMEO`/`SO_SNDTIMEO`) and the per-message protocol poll deadline. Default `0` = disabled (matching rsync); `0` disables it. `--no-timeout` is the negation. The value is not sent on the wire; the server side keeps its own safe floor. |
| `--contimeout <sec>` | Connection timeout in seconds (default: 60, matching rsync); `0` disables it (`--no-contimeout` is the negation) |
| `--stop-after=MINS` | Stop the transfer after MINS minutes (a positive integer); whatever was already transferred is kept |
| `--stop-at=TIME` | Stop at an absolute time (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`); an early stop skips the late `--delete` keep-set |
| `--stop-at=TIME` | Stop at an absolute time. Accepts rsync's `parse_time` forms (`Y-M-DTh:m`, `Y/M/DTh:m`, `Y-M-D`, `M-D`, `D`, `h:m`, `:m`, `T h:m`; omitted fields resolve to the next matching point in the local timezone), plus `now+N[smhd]` and FastSync's `HH:MM`/`HH:MM:SS` clock-time spelling. An early stop skips the late `--delete` keep-set |
| `-b, --backup` | Backup existing destination files before overwriting |
| `--backup-dir <dir>` | Target directory for backups (requires `--backup`) |
| `--tls` | Enable TLS encryption |
@@ -535,7 +535,7 @@ features without changing the meaning of ordinary compatibility options.
| `--server-host <host>` | Select the TCP server host. |
| `--server-port <port>` | Select the TCP server port (`--port <port>` and `--port=<port>` are rsync-friendly aliases). |
| `--tls` | Enable TLS for TCP transport. |
| `--bwlimit <KB/s>` | Apply token-bucket bandwidth limiting (also paces `--sendfile` transfers). |
| `--bwlimit <RATE>` | Apply token-bucket bandwidth limiting with rsync's exact `parse_size_arg` grammar (bare = KiB/s, `K`/`M`/`G`/`T`/`P` binary, `KB`/`MB` decimal, `KiB`/`MiB` binary, decimals quantized to whole KiB, `0`/empty = no limit; also paces `--sendfile` transfers). |
| `--progress` | Show rsync-style per-file progress blocks from the receiver's wire counters; the root `./` line is printed whenever progress is active (rsync prints it only when the transfer root is created). |
| `--stats` | Print transfer statistics, including the receiver-only counters reported over the wire; `Number of files`/`Number of created files` carry rsync's per-type breakdown (deleted files are a single total). |
| `--timeout <seconds>` | Set the socket **and** per-message protocol I/O timeout. Default `0` = disabled (matching rsync); `0` disables it. |
@@ -606,18 +606,18 @@ remote SSH argv is already built injection-safe.
| `--max-alloc <SIZE>` | Maximum single allocation (binary units; default 1G; `0` = no local limit). |
| `--max-depth <n>` | Limit recursive scanning depth; zero means unlimited. |
| `-b, --backup` | Back up overwritten files. |
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install (confined to the receive root; `EXDEV` falls back to a non-atomic copy). |
| `-T, --temp-dir <dir>` | Scratch directory for temp files before the atomic install (confined to the receive root: relative resolves below it, absolute must canonicalize inside it; `EXDEV` falls back to a non-atomic copy). |
| `--backup-dir <dir>` | Store backups under a separate directory (requires `--backup`). |
| `--suffix <suffix>` | Set the backup filename suffix (default: `~`). |
| `--partial` | Select partial-transfer handling. On failed/interrupted writes the already-written temp file is retained (best-effort) for resumption. With `--partial --partial-dir <dir>`, completed files are written under the partial directory and installed atomically. |
| `--partial-dir <dir>` | Set a relative partial-transfer directory below the server destination root. Implies `--partial`. Rejected together with `--inplace` (`--inplace cannot be used with --partial-dir`, matching rsync), because the inplace path bypasses partial/temp staging. |
| `--inplace` | Write directly to the destination instead of using a temporary file. Cannot be combined with `--partial-dir`. |
| `--fsync` | Fsync every written file before publication. |
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree. |
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server). |
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server). |
| `--write-batch=FILE` | Run the normal live transfer and also emit a self-contained batch file of the source tree (FastSync-native format, not rsync-interoperable). |
| `--only-write-batch=FILE` | Emit the batch file only (no destination, no server); FastSync-native format, not rsync-interoperable. |
| `--read-batch=FILE` | Apply a batch file to the destination (no source, no server); FastSync-native format, not rsync-interoperable. |
| `--stop-after=MINS` | Stop the transfer after MINS minutes; whatever was already transferred is kept. |
| `--stop-at=TIME` | Stop at an absolute time (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`). An early stop skips the late `--delete` keep-set. |
| `--stop-at=TIME` | Stop at an absolute time. Accepts rsync's `parse_time` forms (`Y-M-DTh:m`, `Y/M/DTh:m`, `Y-M-D`, `M-D`, `D`, `h:m`, `:m`, `T h:m`; omitted fields resolve to the next matching point in the local timezone), plus `now+N[smhd]` and FastSync's `HH:MM`/`HH:MM:SS` clock-time spelling. An early stop skips the late `--delete` keep-set. |
### Metadata and links
@@ -643,7 +643,7 @@ remote SSH argv is already built injection-safe.
| `--groupmap=MAP` | Map group names when applying ownership (same syntax as `--usermap`). |
| `--numeric-ids` | Mapping modifier: apply the source numeric uid/gid directly instead of mapping by name (combine with `-o`/`-g`, `-a`, or a map). |
| `--copy-as=USER[:GROUP]` | Force every written entry to USER[:GROUP]; requires a privileged receiver. |
| `--fake-super` | Record the resolved owner plus mode/time in a reserved `user.fastsync.stat` xattr and replay mode/time; never performs a real chown. |
| `--fake-super` | Record the resolved owner plus full mode/rdev in rsync's reserved `user.rsync.%stat` xattr (rsync 3.4.1 grammar) and replay the permission bits; never performs a real chown. |
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes). |
| `--no-super` | Forbid those super-user activities even when the receiver is root. |
| `-l`, `--links` | Copy symlinks as symlinks; the target is stored verbatim (absolute and `..`-bearing targets included), matching rsync. |
@@ -689,7 +689,7 @@ remote SSH argv is already built injection-safe.
| `--fastsync-server-path <path>` | Remote FastSync server path for SSH mode (client-only; never crosses the wire). |
| `--rsync-path <path>` | Alias for `--fastsync-server-path`. |
| `-M`, `--remote-option=OPT` | Append OPT to the remote server invocation over SSH (repeatable; rejected for daemon/TCP destinations). |
| `--trust-sender` | Receiver-local: trust the remote sender's file list and skip path re-validation (does not affect symlink targets). |
| `--trust-sender` | Receiver-local: trust the remote sender's file list and skip path re-validation (does not affect symlink targets). **On the client this flag alone is inert** — it is never sent on the wire; the server must be started with its own `--trust-sender`, or the client must forward it with `-M--trust-sender` (SSH only). |
| `--timeout <sec>` | Socket + per-message I/O timeout; default `0` = disabled. |
| `--contimeout <sec>` | Connection timeout; default 60; `0` disables. |
| `--source-dir <path>` | Set the source directory explicitly. |
@@ -732,14 +732,14 @@ remote SSH argv is already built injection-safe.
| `-6`, `--ipv6` | Bind an IPv6 socket. |
| `--allow-delete` | Permit client delete manifests. Deletion is refused by default. This also gates `--force` (which can recursively replace/remove a destination directory tree). |
| `--allow-super` | Standalone TCP listener only: keep super-user activities enabled for a **root** receiver. Without it a root standalone server forces `SUPER_MODE_OFF`, so client `--devices`/`--write-devices`/`--super` and client-chosen ownership requests are skipped/refused. Rejected with `--stdio` (the SSH remote argv is client-composed; use a forced command if the default must hold). No effect when not root. Daemon modules opt in per module with `client owner = yes`. |
| `--trust-sender` | Trust the remote sender's file list: skip the receiver's up-front path-traversal re-validation (fewer checks, faster, potentially unsafe; off by default). It does not affect symlink targets, which are stored verbatim either way. |
| `--trust-sender` | Trust the remote sender's file list: skip the receiver's up-front path-traversal re-validation (fewer checks, faster, potentially unsafe; off by default). It does not affect symlink targets, which are stored verbatim either way. A client `--trust-sender` is never sent over the wire — the server must set this flag itself, or the client must forward it via `-M--trust-sender`. |
| `--no-super` | Operator veto: never attempt super-user activities (ownership, device nodes) even as root, and refuse any client `--copy-as`/`--super` request. |
| `--allow-unauthenticated` | Permit plaintext/anonymous network clients; an auth-required module still accepts only opted-in loopback plaintext. |
| `--iconv=LOCAL[,REMOTE]` | Declare this server's LOCAL charset for file-name conversion. |
| `--password-file=FILE` | Credential store for modules that declare `auth users`. Requires `--daemon`. |
| `--early-input=FILE` | Second credential store layered over `--password-file`. Requires `--daemon`. |
| `--hash-credentials <file>` | Read `<file>`'s `user:password` lines and print PBKDF2 credential-store lines to stdout, then exit. Cannot be combined with `--daemon` or `--stdio`. |
| `--iterations N` | PBKDF2 iteration count for `--hash-credentials` (default 600000, range 100000–10000000). Requires `--hash-credentials`. |
| `--password-file=FILE` | Credential store for modules that declare `auth users`. Requires `--daemon`. FastSync-native SCRAM/PBKDF2 format, not rsync-interoperable. |
| `--early-input=FILE` | Second credential store layered over `--password-file`. Requires `--daemon`. FastSync-native format, not rsync-interoperable. |
| `--hash-credentials <file>` | Read `<file>`'s `user:password` lines and print PBKDF2 credential-store lines to stdout, then exit. Cannot be combined with `--daemon` or `--stdio`. FastSync-native, not rsync-interoperable. |
| `--iterations N` | PBKDF2 iteration count for `--hash-credentials` (default 600000, range 100000–10000000). Requires `--hash-credentials`. FastSync-native, not rsync-interoperable. |
| `-v`, `--verbose` | Enable debug logging. |
| `--help` | Print server usage. |
@@ -768,9 +768,17 @@ and `address`, the global section accepts:
- `hosts allow` / `hosts deny` — comma- and/or whitespace-separated host access
patterns.
A `[module]` requires `path`, and may also set `read only`, `client owner`,
`auth users`, `max connections` (0 = unlimited; enforced per module across all
connection children), and its own `hosts allow`/`hosts deny`.
A `[module]` requires `path`, and may also set `read only`, `write only`,
`client owner`, `auth users`, `max connections` (0 = unlimited; enforced per
module across all connection children), and its own `hosts allow`/`hosts deny`.
Like rsync, a module is **read-only by default**: a bare `[module]` with only a
`path` refuses a write transfer. Opt a module into writability explicitly with
`read only = no` or `write only = yes`; a global `read only` value in the
section before the first `[module]` sets the default for later modules, and a
module's own `read only`/`write only = yes` always wins over it. An
rsync-style `write only = yes` is mapped to writability because FastSync is
push-only (a module can never be read from the network).
The per-host cap and the shared auth lockout identify a source by its numeric
peer IP. **Loopback peers (127.0.0.0/8, IPv6 `::1`) are exempt**: every local
@@ -824,7 +832,7 @@ before the module list, before authentication, and the connecting peer address
## Protocol and Security
FastSync protocol version `2.28.0` is shared by the client and server. The
FastSync protocol version `2.29.0` is shared by the client and server. The
current protocol is sender-driven and includes configuration negotiation,
including the maximum allocation limit, incremental checks, checksums,
manifests, keep-alives, abort handling, per-file remove-source results, and
@@ -897,8 +905,10 @@ The project will reach the drop-in replacement goal in stages:
completion wave's scope; the tests live in `tests/integration/` and skip
cleanly when rsync is unavailable.
3. `-a` implements full rsync `-rlptgoD`; under `-p` the source mode is copied
exactly (no masking). Ownership application stays privilege-gated, as in
rsync.
exactly, including group/other-write bits, with setuid/setgid/sticky copied
only when super-user activities are permitted (masked under
`SUPER_MODE_OFF`/`--no-super`). Ownership application stays privilege-gated,
as in rsync.
4. Symlink (verbatim storage), sparse-file, metadata, delete-policy (including
`--max-delete` partial + exit 25, per-directory `--delete-during`/
`--delete-delay`), codecs, and resumable-write semantics are implemented;
+123 -87
View File
@@ -7,17 +7,17 @@ This document maps rsync's full feature set to FastSync's current implementation
| Status | Count | Description |
|--------|-------|-------------|
| ✅ Parity | 119 | Reproduces rsync's semantics for this option's scope |
| ⚠️ Caveat | 11 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
| ❌ Divergent | 27 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
| ⚠️ Caveat | 14 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
| ❌ Divergent | 24 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
| **Total** | **157** | One row per rsync option/feature group; a row may name several spellings |
This matrix reports honest rsync parity, not "implemented" as a synonym for
"parsed". A ✅ row matches rsync for the option's scope. A ⚠️ row is real and
tested but diverges in at least one documented way. An ❌ row is either
rejected (`--protocol` with any value but the current one, `--inc-recursive`),
rejected (`--protocol` with any value but the current one),
an accepted no-op (`-s`/`--secluded-args`, `--protect-args`, `--old-args`),
deliberately non-rsync and non-interoperable (the FastSync daemon config/auth,
the batch container, `--fake-super`'s xattr format, `--copy-as` credential
the batch container, `--copy-as` credential
switching), or impossible (`-N`/`--crtimes`). The counts are derived from the
rows below; update them together with the table.
@@ -62,7 +62,7 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
- **Fuzzy eligibility.** The `-y/--fuzzy` candidate search no longer inherits the ordinary delta engine's 16 KiB minimum or 10× ratio bound, so an oversized or sub-16-KiB sibling is reused as rsync reuses it (`test_parity_basis_fuzzy.py`).
- **Output partials.** `--info=mount`/`--info=stats`, the `--stats` `dir:` breakdown under `-r`, and real `--debug` output for `flist`/`del`/`hash`/`deltasum`/`recv`/`filter`/`send` were added (`test_parity_info_mount_stats.py`, `test_output_parity.py`, `test_parity_debug.py`); those rows stay ⚠️ for their remaining documented residuals. `--delete-before`'s phase-0 late-file divergence and the `--progress` root/ancestor/symlink feedback remain open (they need a receiver→sender event channel), and the >256 MiB single-file streaming limit (B4) was not addressed. The matrix is now **120 ✅ / 10 ⚠️ / 27 ❌ = 157**.
**Audit cycle (no wire change; `PROTOCOL_VERSION` stays 2.28.0).** A security-and-correctness audit pass ran against the parity-2.29 baseline, followed by a set of audit follow-ups (filter merge modifiers, the `--inplace`/`--partial-dir` conflict, credential-file hardening, and small leak/log/test fixes). The only classification change is `--filter=RULE` moving ✅ → ⚠️, because its merge-only `e`/`n`/`w`/`-` modifiers are now accepted and consumed but their semantics remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ / 11 ⚠️ / 27 ❌ = 157**. The affected rows (`-z`/`--compress`, `--bwlimit`, `-T`/`--temp-dir`, `-p`/`--chmod`, `--partial-dir`, `--filter`) had their notes updated in place:
**Audit cycle (no wire change; `PROTOCOL_VERSION` stays 2.28.0).** A security-and-correctness audit pass ran against the parity-2.29 baseline, followed by a set of audit follow-ups (filter merge modifiers, the `--inplace`/`--partial-dir` conflict, credential-file hardening, and small leak/log/test fixes). The only classification change is `--filter=RULE` moving ✅ → ⚠️, because its merge-only `e`/`n`/`w`/`-` modifiers are now accepted and consumed but their semantics remain unimplemented (accepted-but-ignored); the matrix is therefore **119 ✅ / 11 ⚠️ / 27 ❌ = 157**. The affected rows (`-z`/`--compress`, `--bwlimit`, `-T`/`--temp-dir`, `-p`/`--chmod`, `--partial-dir`, `--filter`) had their notes updated in place. A later triage cycle moved `-F` and `-i` ✅ → ⚠️ (see the triage-cycle note below), giving **117 ✅ / 13 ⚠️ / 27 ❌ = 157**:
- **Decompression ceiling.** `MAX_DECOMPRESSED_SIZE` was 100 MiB while the receiver advertises and the sender compresses whole files up to `MAX_RECEIVE_WHOLE_FILE_SIZE` (256 MiB), so `-z` on a 100–256 MiB regular file failed with `Declared decompressed size exceeds 104857600 bytes`. The ceiling is now defined in terms of the protocol whole-file bound (still a real allocation-clamped bomb guard), so the two cannot drift; `-z` on 100–256 MiB files now works.
- **`--bwlimit` with `--sendfile`.** The plaintext-TCP `--sendfile` fast path wrote through `sendfile(2)` without passing through the protocol's token bucket, so `--bwlimit` was ignored on that path. It is now paced through the same per-session leaky bucket, so TLS and plaintext transports share identical `--bwlimit` semantics.
@@ -73,6 +73,8 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
- **Bounds and wire validation.** `--filter` rule count is now checked client-side against `MAX_FILTER_RULES` (with an actionable message before any network I/O) rather than surfacing as an opaque receiver protocol error; `send_protect_entries()` still re-checks the expanded count. Unknown wire `Status` values are rejected as protocol errors (`status_is_valid()`), and the audit also fixed a mutex leak on an init-failure path, an `errno`-after-`free()` in deferred delete application, `log_perror` misuse for non-`errno` conditions, `SSL_read` length clamping, `sendfile` `poll` `EINTR` retry, and printf-format/attribute issues.
- **Credential-file hardening follow-up.** `secret_file_open()` now opens `--password-file`/`--early-input`/`--hash-credentials` inputs with `O_NOFOLLOW`, so a symlinked credential path fails closed (`ELOOP`) instead of being followed before the owner/mode gate; literal fd-backed paths (`/dev/fd/<digits>`, `/proc/self/fd/<digits>`, which is what a bash process substitution passes) are exempt, so process substitution still works. A FIFO/process-substitution read now waits under a bounded ~3 s deadline for its writer, so a slow producer works while a connected-but-silent FIFO fails instead of hanging. The follow-up also fixed a `config_create` allocation leak on its `server_host` failure path (`config_delete` now releases it), corrected the decompression-limit log message to print the effective bound rather than the compile-time ceiling, and hardened the daemon umask/root test fixtures.
**Triage cycle (no wire change; `PROTOCOL_VERSION` stays 2.28.0).** A documentation-and-correctness triage pass over the parity baseline corrected stale prose and reclassified two rows that carried a real behavioral residual: `-F` moves ✅ → ⚠️ (its own note already documented that per-directory merge rules are not carried to the receiver filter engine, so a destination-only entry matching ONLY a `.rsync-filter` rule is not shielded from `--delete`), and `-i`/`--itemize-changes` moves ✅ → ⚠️ (directory and transfer-root lines are now emitted, but the root `./` line is emitted unconditionally and an incremental re-run itemizes directories/symlinks that rsync's quick-check leaves silent). The matrix is **117 ✅ / 13 ⚠️ / 27 ❌ = 157**.
**Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the
remaining gaps the rsync-parity wave left open (delete timing, wire counters and
output, codec breadth, general `-R`/`-d`, the filter grammar (the unsupported
@@ -81,8 +83,8 @@ output, codec breadth, general `-R`/`-d`, the filter grammar (the unsupported
rejected elsewhere — see the audit-cycle follow-up note above), receiver-side
name resolution, absolute basis dirs, and the remaining client quick wins) and
reclassified the inherently non-rsync rows as **divergent** (native daemon
config/auth, the non-interoperable batch container, `--fake-super`'s xattr
format, `-X`'s privileged namespaces, and the safe-subset device/privilege
config/auth, the non-interoperable batch container,
`-X`'s privileged namespaces, and the safe-subset device/privilege
flags). It moved `PROTOCOL_VERSION` three times (`2.23.0 → 2.24.0` delete
timing, `2.24.0 → 2.25.0` wire stats, `2.25.0 → 2.26.0` codecs). See the
**Parity Completion Wave (protocol 2.26.0)** section near the end for the full
@@ -121,10 +123,10 @@ Every one of those has an entry below with its remaining caveats.
|------|-------------------|-----------------|-------|
| `--stats` | Give transfer stats | ⚠️ Caveat | Prints transfer statistics. Protocol 2.25.0 populates the receiver-only counters the sender cannot observe (`Matched data`, `Number of deleted files`) from the receiver's `STATUS_STATS` report; the sender tracks the scanned file list per type so `Number of files` carries rsync's `(reg: X, dir: Y, link: Z, special: W)` breakdown (directories come from the scanner's captured directory list for `-a`/`-t`/`-p`, or from a lightweight traversed-directory counter on a plain `-r` run so the `dir:` category is present there too), `Number of regular files transferred` excludes symlinks/specials and up-to-date files, `Total file size` includes symlink target lengths, and `Total transferred file size` counts only transferred files. **Protocol 2.28.0 extends `STATUS_STATS`** with receiver-observed `literal_bytes` and the four `created_*` counters: `Number of created files` now carries rsync's `(reg/dir/link/special)` breakdown (the receiver reports which destination entries it newly created, including implicitly-created parent directories below the transfer root) and `Literal data` is exact for a delta transfer (the receiver counts the literal fragments it stored, not the whole source size) — all differential-tested in the sequential and `--threads` paths against rsync 3.4.1 for fresh-create, update and delta shapes. **Remaining divergences:** rsync's per-type breakdown on `Number of deleted files` is not reproduced; and `Total bytes sent`/`received` are FastSync wire bytes framed differently from rsync's, so they are not numerically comparable |
| `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units. **A lone `-h` with no transfer arguments prints help instead** (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer remains human-readable |
| `-i`, `--itemize-changes` | Per-file change summary | ✅ Parity | Prints rsync-style `>f+++++++++` lines to stdout only for files actually sent (also under `-j`/`--threads`); unchanged files print nothing, matching single-`-i` behavior |
| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision; **every** non-root directory is rendered as created (`cd+++++++++`) because the sender never probes a directory's destination state, so a pre-existing destination directory that rsync reports as unchanged (`.d..t......`) is still itemized as created — this is not limited to re-runs; an incremental re-run additionally itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`); and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
| `--progress` | Show progress | ⚠️ Caveat | Protocol 2.25.0 prints rsync-style per-file progress blocks (percent, transferred/total bytes, rate, elapsed, `(xfr#N, to-chk=M/T)`) fed by the receiver's `STATUS_STATS`, in both the sequential and `--threads` send paths. FastSync also prints rsync's leading `./` transfer-root line and, when progress is requested (`--progress`/`-P`/`--info=progress`) and not `--quiet`, runs a **paths-only metadata pre-scan** (no file reads, no hashing) that supplies rsync's file-list total `T` for the `to-chk` denominator and the directory names; `--delete-during`/`--delete-delay` reuse their existing keep-set pre-scan instead of walking twice, and non-progress runs are untouched. Per-directory name lines are emitted (trailing `/`), and symlink (` -> target`) and special entries are named too, so a **fresh multi-directory tree's name set and `to-chk` denominator match rsync 3.4.1** (differential test, sequential and `--threads`) and a **single-file transfer's name lines and deterministic frames remain byte-identical** to rsync. **Order parity (parity-2.29):** the sequential scanner now emits entries in rsync's sorted depth-first flist order (non-directories ascending, then directories ascending), so the interleaving and the `to-chk` numerator match rsync for the default single-threaded transfer (differential `test_parity_order.py`; `--threads` has no rsync analogue and stays unordered). **Remaining divergences:** the leading `./` root line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an ancestor directory line is emitted whenever a child transfers (rsync suppresses it when the directory itself is unchanged); on a re-run, entries without a quick-check (symlinks, empty directories) are still named where rsync stays silent; and the rate/ETA are wall-clock dependent |
| `-P` | Same as --partial --progress | ✅ Parity | Parses to `--partial` + `--progress`. The independent `--partial` retention semantics are rsync parity: an interrupted write retains the already-written temp at the destination (best-effort) so a later `--append`/`--append-verify` can resume. Progress presentation is owned by the `--progress` row; there is no separate `-P` divergence |
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible |
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally, every non-root directory renders as created because the sender does not probe directory destination state (so a pre-existing unchanged directory still shows `cd+++++++++`), and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
| `--log-file=FILE` | Log to file | ✅ Parity | `log_file` config field |
| `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` as the wire byte count) |
| `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Parity | Applies to displayed paths and protocol debug output |
@@ -148,7 +150,7 @@ Every one of those has an entry below with its remaining caveats.
| `--ignore-existing` | Skip updating existing files | ✅ Parity | `ignore_existing` config field (crosses the wire; receiver-side policy). Protocol 2.26.0 short-circuits in the per-file check **before any payload**: when the destination entry already exists, the receiver answers the skip during the incremental handshake instead of letting the sender stream data that would be discarded, so an existing 4 MiB destination costs only the config/check frames (verified with a counting proxy, matching rsync). The write-time paths (regular, delay-updates-staged, hardlink-sibling, special/device) still return `FILE_SAVE_SKIPPED` without overwriting, and `--backup` is disabled for skipped files. Like rsync, it does not apply to directories/symlinks. Combines with `-j`/`--threads` and `--delay-updates` |
| `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Parity | |
| `-x`, `--one-file-system` | Do not cross filesystem boundaries | ✅ Parity | Sender scanner captures the root device and does not descend into mount-point crossings (`st_dev` differs). **Protocol 2.23.0 matches rsync's entry emission:** the mount-point directory itself is emitted as a payload-less directory entry (so the destination gets an empty directory) while its contents are skipped; previously the crossing subdirectory was dropped entirely |
| `-F` | Add the default `.rsync-filter` rules | ✅ Parity | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (first match wins). **A single `-F` transfers the `.rsync-filter` files themselves, matching rsync; a repeated `-FF` additionally excludes them** (rsync 3.4.1's `-F`/`-FF` are exactly these two rules, with no `.cvsignore` branch). Unsupported/unparseable rules inside a per-directory file fail the scan with a clear error. **Residual (track 4a):** per-directory rules are still enforced receiver-side only through the sender-derived source-mirror protected prefixes; the base-rule receiver filter engine does not carry per-directory rules, so a destination-only entry matching ONLY a `.rsync-filter` rule is not yet shielded from `--delete` |
| `-F` | Add the default `.rsync-filter` rules | ⚠️ Caveat | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (first match wins). **A single `-F` transfers the `.rsync-filter` files themselves, matching rsync; a repeated `-FF` additionally excludes them** (rsync 3.4.1's `-F`/`-FF` are exactly these two rules, with no `.cvsignore` branch). Unsupported/unparseable rules inside a per-directory file fail the scan with a clear error. **Residual (track 4a):** per-directory rules are still enforced receiver-side only through the sender-derived source-mirror protected prefixes; the base-rule receiver filter engine does not carry per-directory rules, so a destination-only entry matching ONLY a `.rsync-filter` rule is not yet shielded from `--delete` |
## 4. Directory Options
@@ -159,7 +161,7 @@ Every one of those has an entry below with its remaining caveats.
| `--no-implied-dirs` | Don't send implied dirs with -R | ✅ Parity | With `-R`, rsync creates the ancestor directories implied by a listed path and, with `--no-implied-dirs`, omits their attributes from the transfer so they keep the destination's own state (or are created with default attributes when absent). Protocol 2.26.0 matches this: without `--files-from` the implied-dir walk applies per-attribute metadata only to explicitly transferred directories, and with `-R --files-from` a listed file whose parent is not itself listed is placed normally — the missing implied parent is created with default attributes (not the source's) and the file transfers with `rc 0`, exactly like rsync 3.4.1 (a differential test verifies the modes and mtimes with and without the flag). Works single-threaded and under `-j`/`--threads` |
| `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ✅ Parity | Protocol 2.26.0 implements rsync's one-level `-d` listing for `dir`, `dir/` and `.`: the source's immediate contents are transferred (files with content, directories as explicit entries), matching rsync's destination tree in a differential test. `--dirs --files-from` transfers exactly the listed items — a listed directory is created empty and a listed file with content — under the same `-R` layout rules. A plain recursive scan also recreates empty source directories now: the scanner emits a payload-less directory entry (with metadata) for every traversed directory that produced no transferred or descended child, unless `-m/--prune-empty-dirs` suppresses it or the run is `--files-from`/`--list-only` (a directory emptied by filtering is recreated too, matching rsync). Directory entries cross as `STATUS_MKDIR` and appear in the delete manifest, so `--delete` prunes correctly and an empty listed directory survives; an incoming directory replaces a destination regular file (rsync removes the non-directory and creates the directory), verified differentially. Directory times are applied at the end of the transfer; modes/ownership follow the per-attribute policy. Under `--delay-updates` directories are created immediately while only regular files are staged, exactly as rsync does |
| `--mkpath` | Create missing path components | ✅ Parity | Wire option (client → server). At connection start the server creates the client's destination root directory (and any missing leading components below its own authorized root) when `--mkpath` is set, failing the connection cleanly if it cannot. Without `--mkpath` a destination root that does not exist yet is rejected up front (rsync semantics), so the flag is the only way to transfer into a not-yet-created destination directory. Creation is confined by the same secure mkdir walk as file writes (`O_NOFOLLOW`, no `..`) |
| `--inc-recursive`, `--no-inc-recursive` | Incremental recursion mode | ❌ Divergent | rsync's man-page-only scanning-mode switch (and its short aliases). FastSync always performs a single full recursive scan, so both spellings are rejected as unknown options rather than accepted as a no-op; there is no incremental-recursion engine to toggle. A genuine implementation would be a scan-architecture change with no benefit for FastSync's push model |
| `--inc-recursive`, `--no-inc-recursive` | Incremental recursion mode | ✅ Parity | rsync's man-page-only scanning-mode switch. FastSync always performs a single full recursive scan, so both spellings are accepted as inert no-ops and the destination is identical whichever mode the caller requests — the same treatment as `-r`/`--recursive`, which is likewise a no-op. The switch is a scan-implementation detail with no observable effect on the final tree (rsync's own `--no-inc-recursive` selects a full scan, which is exactly FastSync's behavior) |
## 5. Transfer Modifications
@@ -182,7 +184,7 @@ Every one of those has an entry below with its remaining caveats.
| `--backup-dir=DIR` | Backup directory hierarchy | ✅ Parity | `backup_dir` config field |
| `--suffix=SUFFIX` | Backup suffix (default ~) | ✅ Parity | `suffix` config field |
| `--delay-updates` | Put updated files in place at end | ❌ Divergent | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication, and **`--force` is honored at publication** (protocol 2.23.0): a staged regular file or symlink may replace a destination directory that blocks it. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). **Reclassified Divergent (differential evidence):** the staging name is fixed and a delayed run wipes a pre-existing destination tree of that name at start even without `--delete`, whereas rsync uses its own internal temp name and leaves a genuine destination entry named `.fastsync-stage` untouched (`test_delay_updates_staging_name_collision_residual`); deletion also runs before publication while rsync's `--delay-updates` implies `--delete-after`. Works in single-threaded and `-j`/`--threads` modes |
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ❌ Divergent | `--temp-dir` with the rsync short `-T` (the timeout alias moved to long-only `--timeout`). A **relative** dir matches rsync exactly: it is resolved below the receive/destination root and must already exist (differentially verified: `rsync -a --temp-dir=scratch src/ dst/` and FastSync produce identical trees and an empty scratch dir). **Reclassified as a deliberate divergence because an absolute `--temp-dir` is rejected by the receiver** — it is resolved verbatim by rsync standalone (which will use `/tmp` or any other absolute directory, including one outside the destination), but FastSync's security-reviewed receiver confines the scratch dir to the authorized receive root and rejects any absolute path or one containing `..`. **Audit-cycle hardening:** the opened dir is additionally judged by the real path of its fd (`/proc/self/fd`), so a client-planted symlink under the receive root cannot redirect receiver scratch files outside the authorized root (an escaping target is refused with `EACCES`), while an in-root symlink to another filesystem — the `EXDEV` fallback case — still works. A differential test confirms rsync exits 0 using an absolute scratch dir while FastSync refuses before writing anything into it (the scratch dir stays empty). Its daemon mode also confines relative to the module, but standalone rsync's absolute-temp-dir behavior is not reproduced because it would let a client place receiver scratch files outside the sandbox. Temp copies use a unique name in the scratch dir and are atomically renamed into place; **on `EXDEV` (scratch dir and destination on different filesystems, reachable via a confined relative symlink) the receiver falls back to a non-atomic copy instead of aborting**, matching rsync. `--inplace` and `--partial-dir` writes bypass the scratch dir |
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ❌ Divergent | `--temp-dir` with the rsync short `-T` (the timeout alias moved to long-only `--timeout`). A **relative** dir matches rsync exactly: it is resolved below the receive/destination root and must already exist (differentially verified: `rsync -a --temp-dir=scratch src/ dst/` and FastSync produce identical trees and an empty scratch dir). An **absolute** dir is accepted when it canonicalizes (`realpath(3)`) inside the receive root, so an in-root absolute scratch path is usable and used (unit- and integration-tested for both the local batch apply and a live TCP transfer). **Remaining divergence:** an absolute `--temp-dir` that escapes the receive root is rejected, and so is a relative one containing `..` — rsync standalone resolves an absolute `--temp-dir` verbatim (it will use `/tmp` or any other directory, including one outside the destination), but FastSync's security-reviewed receiver confines the scratch dir to the authorized receive root and refuses an out-of-root path before writing anything. **Audit-cycle hardening:** the opened dir is additionally judged by the real path of its fd (`/proc/self/fd`), so a client-planted symlink under the receive root cannot redirect receiver scratch files outside the authorized root (an escaping target is refused with `EACCES`), while an in-root symlink to another filesystem — the `EXDEV` fallback case — still works. A differential test confirms rsync exits 0 using an out-of-root absolute scratch dir while FastSync refuses before writing anything into it (the scratch dir stays empty). Its daemon mode also confines relative to the module. Temp copies use a unique name in the scratch dir and are atomically renamed into place; **on `EXDEV` (scratch dir and destination on different filesystems, reachable via a confined relative symlink) the receiver falls back to a non-atomic copy instead of aborting**, matching rsync. `--inplace` and `--partial-dir` writes bypass the scratch dir |
| `--partial` | Keep partially transferred files | ✅ Parity | On a failed/interrupted write the already-written temp file is retained at the destination path (best-effort rename instead of unlink) so a later `--append`/`--append-verify` run can resume it. Retention never runs when no data was actually written or under `--ignore-existing`/`--existing` (the destination is not ours to overwrite), and it only ever renames the already-written temp. A failed rename falls back to the normal unlink |
| `--partial-dir=DIR` | Keep partial files in DIR | ✅ Parity | The working file is written under the confined partial directory (a relative dir below the receive root) and atomically renamed into place once complete, so an interrupted transfer leaves a resumable copy there and completed transfers do not linger under it. `--inplace` bypasses the partial dir (rsync parity), and combining `--inplace` with `--partial-dir` is now **rejected up front** with rsync's message (`--inplace cannot be used with --partial-dir`) instead of silently ignoring the partial dir. **Implies `--partial`** (audit-cycle fix, matching rsync 3.4.1, which sets `keep_partial` after option parsing): `--partial-dir=DIR` alone retains an interrupted transfer's partial, and the implication wins over an explicit `--no-partial` regardless of order |
@@ -191,7 +193,7 @@ Every one of those has an entry below with its remaining caveats.
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--delete` | Delete extraneous files from dest | ✅ Parity | `use_delete` config field. Deletion is always derived from the keep-set the sender actually transmitted (the per-directory `STATUS_DELETE_PLAN` set by default, or the whole-tree manifest for the late timings — never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. **Lockstep track 6 (protocol 2.28.0): plain `--delete` with no explicit timing flag now defaults to `--delete-during`**, exactly like rsync's `--del` (the client normalizes it to the existing `delete_during` wire bool; no new wire field). This frees destination space progressively during the transfer and avoids the whole-old+new-tree peak that could `ENOSPC` a tight destination. The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`. **Abort/ordering parity (parity-2.29):** the complete per-directory plan set is transmitted before the first data frame, so a mid-transfer abort has already applied every planned removal exactly like rsync's generator (which runs ahead of its throttled sender); `-d/--dirs` uses the same per-directory plans (the generator records only the directories whose direct children it enumerated, so an untraversed subdirectory's mirror is shielded); and the sorted depth-first traversal makes the removal order — and therefore the survivor set under a partial `--max-delete` — match rsync exactly (`test_delete_boundary_parity.py`, `test_parity_order.py`). By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent on the wire (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing |
| `--delete-before` | Delete before transfer | ⚠️ Caveat | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. **Phase-0 divergence (sharpened):** rsync builds the full file list first, so a source file created after that scan is NOT transferred and its destination extra is deleted; FastSync's single-threaded data pass re-scans the source, so the late file IS transferred (a safe superset), while FastSync `--threads` pipelines the scan and matches rsync |
| `--delete-before` | Delete before transfer | ✅ Parity | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. **Phase-0 divergence closed (no-wire):** both data passes now replay the exact file list the pre-scan built for the keep-set instead of re-reading the source — the single-threaded send loop and the `--threads` pipeline (whose scanner thread feeds the retained pre-scan chunks into the pipeline rather than re-scanning) — so a source file created after that scan is NOT transferred and its destination extra is deleted, exactly like rsync's single file list. The pre-scan captures the deferred directory times and the `--stats` directory count because no later scan runs (`test_delete_timing_parity.py::TestDeleteBeforeLateFileParity`, parametrized single-threaded vs `--threads=4`, differential vs rsync 3.4.1) |
| `--del`, `--delete-during` | Delete during transfer | ✅ Parity | Both spellings accepted; imply `--delete`, and since lockstep track 6 this is also the default timing of a plain `--delete`. **Protocol 2.24.0 implements per-directory delete plans:** as the sender reaches each source directory it streams a `STATUS_DELETE_PLAN` for that directory and the receiver removes that directory's extras (verified with a byte-slicing proxy). The one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) rides a dedicated config-only carrier frame with an `apply=false` flag, so it reaches the receiver even when the scope allows no directory plan at all (a `--files-from` list of bare files synchronizes no directory). **Abort/ordering parity (parity-2.29):** the complete plan set is transmitted before the first data frame, so on a mid-transfer abort every planned extra has already been removed exactly like rsync's generator (which runs ahead of its throttled sender); `-d/--dirs` no longer falls back to the end-of-transfer commit but records only the directories whose direct children it enumerated; and the sorted depth-first traversal makes the removal order — and the partial-`--max-delete` survivor set — identical to rsync (`test_delete_boundary_parity.py`, `test_parity_order.py`). `-R` plans are scoped to the transferred prefix subtree |
| `--delete-delay` | Find deletions during, delete after | ✅ Parity | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The **reported** deleted count advances only on an actual removal. **Fixed (no-wire):** the `--max-delete` budget is now charged on ACTUAL removals (an unlink/rmdir that succeeded), not at plan/snapshot time, and a queued directory is re-scanned at commit and removed recursively (content created after the plan included), matching rsync: a snapshotted entry that fails or is skipped consumes no budget, so a later extra rsync would delete is still deleted. The deferred snapshot list keeps an independent hard cap (`DELETE_PLAN_SERVER_LIMIT`) so it cannot grow without bound now that the budget is no longer charged while scanning. A `--max-delete=2` partial delete reports exactly 2 and exits 25 in both tools, and the refilled-directory differential (late content removed, directory removed, budget shared) now matches rsync 3.4.1 on both sides (`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`). Unit tests cover recursive removal, actual-removal charging, and the bounded deferred list. **Ordering parity (parity-2.29):** the sorted depth-first traversal plus the up-front plan set make the order in which extras are removed — and therefore the survivor set under a partial `--max-delete` — match rsync exactly (differential `test_parity_order.py::test_delete_delay_deletion_order_matches_rsync` and `::test_partial_max_delete_survivor_order_matches_rsync`) |
| `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. Selects the late whole-tree commit: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing. Since lockstep track 6 a plain `--delete` defaults to delete-during (rsync's `--del`); `--delete-after` — or the FastSync-only `--delete-commit` spelling, which selects the identical timing — is the explicit way to keep the old commit-style behavior |
@@ -337,10 +339,10 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
| `-E`, `--executability` | Preserve executability | ✅ Parity | Preserves executable permission bits (implies metadata preservation) |
| `--chmod=CHMOD` | Affect file permissions | ✅ Parity | Faithful port of rsync 3.4.1's `parse_chmod`/`tweak_mode`: numeric octal and symbolic `ugo`/`rwx` changes, `D`/`F` directory/file selectors, `X` (execute only on directories or already-executable files), `s`/`t` setuid/setgid/sticky, and append semantics — repeated clauses and repeated `--chmod` options accumulate in order (joined with commas). The changes are applied to the new mode **without sanitization** (matching rsync), except that setuid/setgid/sticky are masked when the connection forbids super-user activities (audit-cycle fix, see `-p`), and `--chmod` does **not** imply `-p` (rsync parity). Applied to files and directories on the receiver |
| `-A`, `--acls` | Preserve ACLs | ✅ Parity | Implemented on Linux via the POSIX-ACL xattr representation: the sender captures the `system.posix_acl_access` / `system.posix_acl_default` xattrs and the receiver re-applies them fd-relative. A differential test with `setfacl` confirms the complete access and default ACL sets (including `mask`) are identical to rsync's on a directory. libacl is not required; a `fsetxattr` an unprivileged receiver may not perform is logged and skipped, never fatal. Only the `system.posix_acl_*` namespaces plus `user.*` are ever applied; privileged namespaces are never applied. Implies metadata transmission |
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Divergent | Deliberately restricted to unprivileged `user.*` extended attributes plus the two POSIX ACL xattrs; `security.*` (SELinux, capabilities, ...) and `trusted.*` are **never** captured or applied — a client can never force a privileged attribute onto the destination, and the receiver independently re-validates every incoming name against the whitelist. This is a security-policy divergence from rsync, which can preserve the privileged namespaces with the needed privilege; implementing them would defeat FastSync's privilege-escalation guard. `user.*` capture/apply matches rsync in a differential test. Payloads are bounded on both ends. Incompatible with `-s` |
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Divergent | Deliberately restricted to unprivileged `user.*` extended attributes plus the two POSIX ACL xattrs; `security.*` (SELinux, capabilities, ...) and `trusted.*` are **never** captured or applied — a client can never force a privileged attribute onto the destination, and the receiver independently re-validates every incoming name against the whitelist. This is a security-policy divergence from rsync, which can preserve the privileged namespaces with the needed privilege; implementing them would defeat FastSync's privilege-escalation guard. `user.*` capture/apply matches rsync in a differential test. Payloads are bounded on both ends. Incompatible with `-s`. **Symlink xattrs are now carried (protocol 2.29.0):** a symlink entry appends the same bounded trailing xattr block to its `STATUS_SYMLINK` frame as every other entry kind, captured with `llistxattr`/`lgetxattr` so the link's OWN attributes are read and never the referent's, and re-applied no-follow with `lsetxattr` through the already-confined parent directory (`fsetxattr` cannot target a symlink: there is no `*at` xattr syscall and an `O_PATH` fd is rejected). On Linux the VFS refuses to associate xattrs with a symlink at all — every `lsetxattr` on a link fails with `EPERM` for `user.*`, `trusted.*` and `security.*`, even as root, verified in the CI container — so on FastSync's supported platforms the captured block is always empty and the apply is a no-op; the wire block is present for correctness and for a filesystem/platform that does support symlink xattrs. rsync 3.4.1's `--fake-super` is not a counterexample: it stores a symlink as a regular file whose `user.rsync.%stat` records the `S_IFLNK` mode bits, not an xattr on a real symlink. The row stays divergent only for the never-preserved privileged namespaces above |
| `-H`, `--hard-links` | Preserve hard links | ✅ Parity | Files on the source that share an inode (`st_dev`+`st_ino`, e.g. a `cp -al` tree) are re-created as hard links to one another on the destination, so duplicate links stay deduplicated and only the first member's data is sent (later members are transmitted as payload-less `STATUS_HARDLINK` frames). The receiver links each sibling to the first member's installed file with an atomic link + rename; on `link()` failure it falls back to a byte-identical local copy of the first member, never a partial/corrupt file. Requires the sequential scan for ordering (the first member is always emitted and installed before any sibling is linked). Works single-threaded and under `-j`/`--threads`, `--inplace`, `--delay-updates` (links staged and published by rename) and `--partial`. Crosses the wire (`preserve_hard_links` bool; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0**, peers must match). Incompatible with `-s` (chunk serialization) and `--append`/`--append-verify`, rejected up front with a distinct error. See the Phase-4 hard-links notes below |
| `-D` | Same as --devices --specials | ✅ Parity | Implies `--devices --specials`. `-D` was unassigned in FastSync (verified: no collision), so it is free to imply both device-node and special-file preservation. As of protocol 2.23.0 `--specials` genuinely covers **both FIFOs and unix sockets**, so `-D` covers the full rsync set. See the `--devices`/`--specials` rows and the Phase-4 devices notes below |
| `--devices` | Preserve device files | ❌ Divergent | Recreates char/block device nodes with `mknodat` (type + rdev strictly validated, confined fd-relative below the receive root), but only when the receiver has `CAP_MKNOD`: a non-root receiver logs a warning and skips the entry instead of erroring, so a transfer with devices never aborts. Deliberate privilege-model divergence from rsync, which errors when it cannot create the node. `--specials` (FIFOs and unix sockets) is unprivileged and remains parity |
| `--devices` | Preserve device files | ⚠️ Caveat | Recreates char/block device nodes with `mknodat` (type + rdev strictly validated, confined fd-relative below the receive root). A device whose `mknodat` fails with `EPERM`/`EACCES` (no `CAP_MKNOD`, or super-user activity forbidden) is a **per-entry failure**: FastSync logs `cannot create device ...` (rsync logs `mknod ... failed`), counts it, **continues with the remaining files**, and ends the run with a non-OK terminal status. rsync parity: rsync likewise continues and exits partial (23). Residuals: (1) FastSync's default AUTO still *attempts* the node on a non-root receiver and therefore reports the per-entry failure, whereas rsync without `--super` silently ignores `--devices` and skips the non-regular entry with exit 0 — use `--no-super` for rsync's silent-skip behavior; (2) FastSync's process exit code for a receiver-side per-entry failure is the general error code 1, not rsync's partial 23 (a client exit-code-mapping residual that applies to every receiver file error, not just this branch); (3) with `--remove-source-files`, the non-OK terminal status means successfully transferred sources are not removed on a partial run. `--specials` (FIFOs and unix sockets) keeps the unprivileged skip path and remains parity |
| `--specials` | Preserve special files | ✅ Parity | **FIFO and unix-socket recreation work** (protocol 2.23.0): FIFOs are recreated with `mkfifoat`, and sockets with `mknodat(..., S_IFSOCK)` — the latter is unprivileged on Linux because it materializes the socket *node*, not a live bound socket, so it is a real, assertable behavior under CI (it matches rsync, which also recreates a socket by `mknod`). Node creation is confined below the receive root (fd-relative parent; no `..`, no symlink follow) and type/rdev are validated strictly; a matching existing node is left in place and an unrelated entry is never replaced. Crosses the wire like `--devices` (the `STATUS_SPECIAL` frame). See the Phase-4 devices notes |
| `--copy-devices` | Copy device contents as file | ❌ Divergent | Copies a device/FIFO's reported `st_size` into an ordinary regular file and never reads an unbounded pseudo-device, so `--sendfile` cannot hang and the run always succeeds. Deliberate safe divergence from rsync's dd-like unbounded device read, which can block; the dangerous behavior will not be implemented |
| `--write-devices` | Write to devices as files | ❌ Divergent | Writes only into an existing char/block node under the confined receive root (`O_NOFOLLOW` + `O_NONBLOCK`); a missing, symlinked, FIFO-with-no-reader, non-device, or otherwise unusable destination is skipped with a warning rather than allowed or aborted. Deliberate confinement divergence from rsync's more permissive behavior |
@@ -349,10 +351,10 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
| `-O`, `--omit-dir-times` | Omit dirs from --times | ✅ Parity | Real modifier now that FastSync preserves directory times. With metadata on, the scanner captures every traversed source directory's mtime (and atime under `-U`) and the sender transmits them in trailing `STATUS_DIR_TIMES` frame(s) **after all file data and the optional delete manifest** (chunked at the receiver's `MAX_MANIFEST_ENTRIES` per-frame cap); a dir-time entry only RECORDS metadata and never creates the directory (an empty source directory is created by the separate `STATUS_MKDIR` entry the scanner now emits, and `-m/--prune-empty-dirs` suppresses that; the trailing dir-time simply re-applies the metadata). The receiver defers applying them until its delete / `--delay-updates` publication phases have committed, so writing or removing a child never clobbers a parent directory's mtime (rsync applies directory times at the end for exactly this reason). When `-O` is set (the boolean crosses the wire) the receiver does not apply any of them; without `-O` an `-a`/`--preserve` transfer now restores directory times (reversing the old "never preserves dir times" divergence). Wire change: the terminal `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
| `-J`, `--omit-link-times` | Omit symlinks from --times | ✅ Parity | Real modifier now that FastSync preserves symlink times. Symlink entries already carried their metadata on `STATUS_SYMLINK`; the receiver now applies it with **no-follow primitives only** (`utimensat(..., AT_SYMLINK_NOFOLLOW)`, plus best-effort `fchmodat(..., AT_SYMLINK_NOFOLLOW)` and policy-gated `fchownat(..., AT_SYMLINK_NOFOLLOW)`), so the link itself is stamped without ever dereferencing it, confined fd-relative below the authorized receive root. A symlink has no children, so the times are applied immediately at creation. When `-J` is set (the boolean crosses the wire) the receiver skips the timestamps (mode/ownership are unaffected); without `-J` an `-a`/`-l` transfer restores symlink mtimes. Wire change alongside `-O`: the shared `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
| `--super` | Receiver attempts super-user activities | ❌ Divergent | Safe-subset privilege model. `--super` permits the receiver to attempt already-confined super-user activities (ownership application, char/block device-node creation, `--write-devices`); `--no-super` forbids them even for root; `auto` keeps the historical best-effort attempt. **FastSync never elevates** — no `setuid`/`seteuid`/`setgid` — and `--super` never bypasses the confinement floor, so it diverges from rsync's real elevation. A server `--no-super` veto forces it off for every connection; a privileged standalone listener defaults off without `--allow-super`; daemon modules opt in with `client owner = yes` |
| `--fake-super` | Store/recover privileged attrs via xattrs | ❌ Divergent | Records the resolved `uid:gid:mode:mtime_sec:mtime_nsec` in a reserved `user.fastsync.stat` xattr and immediately replays mode/times fd-relative, but **never performs a real `chown`** (the owner is recorded for a later privileged restore). The on-disk key and format are FastSync-native, not rsync's `user.rsync.%stat%`, so recordings are not interoperable with rsync — the same class as the native auth and batch formats. Implies metadata transmission; incompatible with `-s` |
| `--fake-super` | Store/recover privileged attrs via xattrs | ⚠️ Caveat | Writes rsync 3.4.1's reserved `user.rsync.%stat` xattr with rsync's exact value grammar `<octal st_mode with S_IFMT> <rdev_major>,<rdev_minor> <uid>:<gid>` (e.g. `104711 0,0 1234:5678`), recording the RESOLVED owner (the `--chown`/`--usermap`/`--groupmap`/`--copy-as` mapping when active, else the source's own id) plus the full mode and rdev; it **never performs a real `chown`**. mtime is carried by the file's own timestamp, exactly as rsync does it (there is no mtime field). The receiver parses the same grammar and replays the permission bits fd-relative, stripping the recorded special bits on disk exactly like rsync's fake-super receiver. Regular files are interoperable with real rsync 3.4.1 in both directions (the differential test has rsync read a FastSync fake-super tree and re-emit the identical record). Char/block devices **are** faked: a device is written as a regular empty file and its `user.rsync.%stat` records the real `rdev` (e.g. `20644 1,3 0:0`), never `mknod`'d, on both privileged and unprivileged receivers, exactly as rsync does. The record parser range-checks every field (mode/rdev/uid/gid) and rejects malformed records cleanly. Residual: directories are not yet faked — no `%stat` record is written for a directory. Implies metadata transmission; incompatible with `-s` |
| `--open-noatime` | Avoid changing access time when opening files | ✅ Parity | Sender-side policy: the sender opens source files with `O_NOATIME` (Linux) when reading them for transfer, so the open/read does NOT bump the source's on-disk access time. Degrades safely when `O_NOATIME` is unavailable (not defined) or refused (`EPERM`, since it needs `CAP_FOWNER` or file ownership): the code falls back to a normal open, so the data always transfers — only the atime-bump is skipped. It does not itself capture/preserve atime; it only avoids modifying it. **Client-only, never crosses the wire.** Exposed as `file_open_for_read()` and applied to both the buffered data path and the sendfile path |
| `--numeric-ids` | Do not map uid/gid by name | ✅ Parity | **A mapping modifier only:** when ownership is being applied it uses the transmitted numeric uid/gid directly, skipping the name lookup. It does **not** request ownership application on its own — combine it with `-o`/`-g`, `-a`, or an explicit map (`--chown`/`--usermap`/`--groupmap`) — and it does not need any metadata flag merely to parse. Ownership is only applied when metadata (hence the source uid/gid) is actually transmitted (see the Phase-4 identity notes) |
| `--usermap=STRING` | Map usernames | ✅ Parity | Opt-in ownership application. Comma-separated `FROM:TO` rules evaluated in order, first match wins. `FROM` accepts a source-resolved user name, an `@N`/bare `N` numeric id, an inclusive `LOW-HIGH` id range, `*`, or an empty field (ids with no source name). `TO` accepts a receiver-resolved **name** (protocol 2.26.0 resolves it on the receiving side against the receiver's account database, matching rsync), an `@N`/bare `N` id, or `*` (the receiving process's euid). Rules travel as resolved numeric pairs plus an optional TO name; the receiver applies a matching rule, else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup, via fd-relative `fchown`. Malformed specs are clear errors. Implies metadata; only effective where the receiver can chown (otherwise a warning) |
| `--usermap=STRING` | Map usernames | ✅ Parity | Opt-in ownership application. Comma-separated `FROM:TO` rules evaluated in order, first match wins. `FROM` accepts a source-resolved user name, a name **glob** (`*`/`?`/`[...]`, expanded sender-side at CLI-parse time against the sender's passwd/group database and collapsed into numeric `LOW-HIGH` ranges, bounded by `MAX_IDENTITY_MAP`), an `@N`/bare `N` numeric id, an inclusive `LOW-HIGH` id range, `*`, or an empty field (ids with no source name). `TO` accepts a receiver-resolved **name** (protocol 2.26.0 resolves it on the receiving side against the receiver's account database, matching rsync), an `@N`/bare `N` id, or `*` (the receiving process's euid). Rules travel as resolved numeric pairs plus an optional TO name; the receiver applies a matching rule, else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup, via fd-relative `fchown`. Malformed specs are clear errors. Implies metadata; only effective where the receiver can chown (otherwise a warning) |
| `--groupmap=STRING` | Map group names | ✅ Parity | Same rules and receiver-side `TO`-name resolution as `--usermap`, applied to the group (gid) side |
| `--chown=USER:GROUP` | Map owner and group | ✅ Parity | Opt-in ownership override. Forms `USER:GROUP`, `USER`, `:GROUP`; `*` means the current user/group as appropriate; `@N`/bare `N` ids; a name may escape `:` as `\:`. A name that resolves on the sender is sent as an id; an unresolvable name is carried as a receiver-resolved `TO` name (protocol 2.26.0), matching rsync's receiver-side resolution. Equivalent to a trailing `*:*` usermap+groupmap rule (an explicit map match wins). Conflicts with `--usermap`/`--groupmap` on the same side are a clear configuration error. Implies metadata; a non-root receiver warns and continues (rsync parity) |
| `--copy-as=USER[:GROUP]` | Perform the copy as another user/group | ❌ Divergent | Close-refusal safe subset. FastSync never switches process credentials (its receiver is multithreaded, so a real `setuid`/`setgid` would be unsafe); instead the receiver forces the ownership of every entry it writes to the client-resolved ids through the confined fd-relative identity path. A privileged (root) receiver is required: an unprivileged receiver refuses the whole transfer at the config handshake, before any data, rather than produce wrong ownership. Deliberate divergence from rsync's real identity switching; a daemon refuses it unless the module sets `client owner = yes` |
@@ -406,7 +408,7 @@ match, exactly as prior phases did).
is refused) also re-applies the incoming (or, for `-H`, the first member's)
xattrs and the `--fake-super` stat, so attributes are preserved rather than
silently dropped when the link fails.
- **Reserved fake-super key is receiver-only:** the `user.fastsync.stat` key is
- **Reserved fake-super key is receiver-only:** the `user.rsync.%stat` key is
excluded from sender capture AND from receiver application, so it can only be
written by the receiver's own `--fake-super` handling. A source file that
already carries such a record is never forwarded on a plain `-X` run, so it
@@ -415,15 +417,19 @@ match, exactly as prior phases did).
Applying an ACL is owner-privileged: `fsetxattr` failure (e.g. non-root,
unsupported filesystem) is logged (collapsed to one line per file) and never
fatal.
- **`--fake-super`**: see the row above; the reserved key is `user.fastsync.stat`
with the documented `uid:gid:mode:mtime_sec:mtime_nsec` (mode octal) format.
**Replay exists**: after each stored record the receiver immediately re-applies
the recorded mode and times fd-relative (`fake_super_restore_fd`), but it
deliberately never performs a real `chown` — `--fake-super` only *records*
the resolved owner (the active `--chown`/`--usermap`/`--groupmap`/`--copy-as`
mapping when one is in effect, otherwise the source's own id) for a later
privileged restore. The recording format diverges from rsync's
`user.rsync.%stat%`; no cross-tool conversion is attempted.
- **`--fake-super`**: see the row above; the reserved key is rsync's own
`user.rsync.%stat` with rsync 3.4.1's exact `<octal st_mode> <rdev_major>,
<rdev_minor> <uid>:<gid>` value (mtime is not stored — the file's own
timestamp carries it, exactly as rsync does). **Replay exists**: after each
stored record the receiver immediately re-applies the recorded permission bits
fd-relative (`fake_super_restore_fd`, with the recorded special bits stripped
on disk exactly like rsync), but it deliberately never performs a real
`chown` — `--fake-super` only *records* the resolved owner (the active
`--chown`/`--usermap`/`--groupmap`/`--copy-as` mapping when one is in effect,
otherwise the source's own id) for a later privileged restore. Because the
key and grammar are rsync's, a regular-file fake-super tree is interoperable
with rsync 3.4.1 in both directions; directories and device nodes are not yet
faked.
- **Chunk serialization (`-s`) incompatibility:** the per-file xattr block rides
the streaming per-file frame, which `-s` replaces with a fixed buffer format,
so `-X` / `-A` combined with `-s` is rejected up front on both ends (mirroring
@@ -497,8 +503,10 @@ names, `--chown` names) are resolved to numbers at CLI parse time against the
**client (sender) machine's** account databases; this reproduces rsync's
semantics on a shared-account source/destination and is documented for a
genuinely different destination. The interesting named-value subset is
supported (`*` FROM wildcard, `*` TO = current user, `@N`/bare-`N` numerics); a
lone-`@` "use the FROM value unchanged" rsync form is not implemented. Also
supported (FROM name globs `*`/`?`/`[...]` expanded sender-side against the
passwd/group database and bounded by `MAX_IDENTITY_MAP`, `*` FROM wildcard,
`*` TO = current user, `@N`/bare-`N` numerics); a lone-`@` "use the FROM value
unchanged" rsync form is not implemented. Also
unlike rsync, plain `-M` never applies ownership and `--usermap`/`--groupmap`/
`--chown` each imply metadata preservation so the source uid/gid actually travel
(the flags only take effect where ownership is being preserved/applied).
@@ -550,18 +558,22 @@ marker + rdev so `--devices/--specials` also work under `-s`. `PROTOCOL_VERSION`
was bumped **2.12.0 → 2.13.0** (peers must match, exactly as prior phases did).
**Privilege gating (the crux):** making a device node requires `CAP_MKNOD` (root).
CI runs the integration suite as a NON-ROOT user (via setpriv), so `mknod` fails
with `EPERM`. The receiver treats this as a graceful, logged *skip of the entry*
returned as a success/skip outcome — the whole transfer NEVER aborts just because
the environment cannot create the node. `mkfifo` (FIFOs) is unprivileged, so
`--specials` FIFO creation is a real, assertable behavior under CI. **Sockets are
recreated too** (protocol 2.23.0) with `mknodat(..., S_IFSOCK)`: Linux allows an
unprivileged `mknod` of a socket node because no live bound socket is created,
so a source socket materializes as a socket-type filesystem entry exactly as
rsync does. The "device actually created" integration assertions are guarded to
run only as root. User-facing expectation: point `--devices` at devices and a
non-root receiver will faithfully skip them while transferring everything else;
`--specials` recreates FIFOs and socket nodes for any receiver.
When the receiver attempts a device `mknod` and the kernel refuses with
`EPERM`/`EACCES`, FastSync now reports a genuine transfer error (the receiver's
outcome aggregation fails the entry), matching rsync, which logs
`mknod ... failed` and exits partial (23) whenever it attempts the node (as root
or with `--super`); with `--no-super` the device entry is pre-skipped instead.
Only `mkfifo` (FIFOs) is unprivileged, so `--specials` FIFO creation is a real,
assertable behavior under CI. **Sockets are recreated too** (protocol 2.23.0)
with `mknodat(..., S_IFSOCK)`: Linux allows an unprivileged `mknod` of a socket
node because no live bound socket is created, so a source socket materializes as
a socket-type filesystem entry exactly as rsync does. The "device actually
created" integration assertions are guarded to run only as root; a root runner
additionally drops the receiver to an unprivileged user (setpriv) to assert the
`CAP_MKNOD` failure surfaces as a failed transfer rather than a silent skip.
User-facing expectation: point `--devices` at devices and a receiver without
`CAP_MKNOD` reports the failure, while `--specials` recreates FIFOs and socket
nodes for any receiver.
**Confinement & validation:** a special/device node is created with
`mknodat`/`mkfifoat` on the parent directory opened fd-relative below the receive
@@ -599,7 +611,7 @@ warning + skip, never a system-clobbering write or an abort.
| `-L`, `--copy-links` | Transform symlink to referent | ✅ Parity | Sender-side: every symlink is replaced by its referent's content. A referent that cannot be read, including a broken symlink, makes the run exit 23 (`RERR_PARTIAL`) like rsync while the rest of the tree still transfers, in both the sequential and `--threads` paths (differential test). The transferred tree matches rsync |
| `--copy-unsafe-links` | Transform unsafe symlinks | ✅ Parity | Sender-side: only symlinks whose target is unsafe (absolute or escaping via `..`, matching rsync's `unsafe_symlink()` semantics) are dereferenced into their referent; safe links stay symlinks. A broken unsafe referent makes the run exit 23 like rsync (differential test), while a safe broken symlink is not dereferenced and exits 0 |
| `--safe-links` | Ignore symlinks outside tree | ✅ Parity | Sender-side: a symlink whose target is unsafe is not transmitted at all (skipped), matching rsync's `--safe-links`. Because FastSync applies this while scanning the source, the receiver does not need to repeat it (`safe_links` config field) |
| `--munge-links` | Munge symlinks for safety | ✅ Parity | Sender rewrites each transmitted symlink target with rsync's `/rsyncd-munged/` prefix; the receiver strips the marker (only when the negotiated `munge_links` policy is on, so a source link that genuinely begins with the marker round-trips verbatim) and restores the exact real target. Unlike rsync, FastSync prefixes on the *sender* and un-munges on the receiver, but the wire result and the stored marker match rsync. See the Phase-4 symlink-trust notes |
| `--munge-links` | Munge symlinks for safety | ✅ Parity | The **receiver** munges: it prefixes each stored symlink target with rsync's `/rsyncd-munged/` marker (only when the negotiated `munge_links` policy is on, so a source link that genuinely begins with the marker round-trips verbatim). The **sender** un-munges a source target that already begins with the marker before transmitting, so a munged tree round-trips through the receiver's re-munging exactly like rsync. Matching rsync, the prefix is applied on the receiver and stripped on the sender; the wire result and the stored marker match rsync. See the Phase-4 symlink-trust notes |
| `-k`, `--copy-dirlinks` | Transform symlink to dir | ✅ Parity | A symlink whose referent is a directory is dereferenced and recursed as a real directory; a symlink to a regular file stays a symlink. Sender-side only. See the Phase-4 symlink-trust notes |
| `-K`, `--keep-dirlinks` | Treat symlinked dir as dir | ✅ Parity | On the receiver, an existing destination symlink-to-a-directory is used as that directory (followed) instead of being replaced; it is followed only when it resolves to a directory that stays beneath the receive root. See the Phase-4 symlink-trust notes |
@@ -649,14 +661,15 @@ was bumped **2.12.0 → 2.13.0** (peers must match, exactly as prior phases did)
divergence for `--delete` over an existing symlinked dir). Without `-K` the
destination symlink is not followed (the O_NOFOLLOW walk fails the write),
which is the safe default.
- **`--munge-links`** (sender rewrite; crosses the wire so the receiver
unmunges): every transmitted symlink target is prefixed with rsync's marker
`SYMLINK_MUNGE_PREFIX` = `/rsyncd-munged/`; the receiver strips the marker
(only when the negotiated `munge_links` policy is on — a plain `-l` run never
strips the prefix, so a source symlink that genuinely begins with
`/rsyncd-munged/` round-trips verbatim) and restores the exact real target.
This matches rsync's stored marker and its both-ends-negotiated model, with the
prefix applied on the sender rather than the receiver. The link *value* is
- **`--munge-links`** (receiver rewrite; crosses the wire so the receiver
munges): every stored symlink target is prefixed with rsync's marker
`SYMLINK_MUNGE_PREFIX` = `/rsyncd-munged/` by the **receiver**; the sender
un-munges a source target that already begins with the marker before
transmitting, so a munged tree round-trips verbatim. The marker is applied only
when the negotiated `munge_links` policy is on — a plain `-l` run never
prefixes, so a source symlink that genuinely begins with
`/rsyncd-munged/` round-trips verbatim. This matches rsync's stored marker and
its both-ends-negotiated model, with the prefix applied on the receiver. The link *value* is
otherwise stored verbatim; the *placement* path still goes through
`file_symlink_at_secure`'s confined fd walk (`has_path_traversal` on the
destination path, no symlink follow). When no symlink is being transmitted
@@ -725,8 +738,8 @@ targets verbatim, matching rsync.
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--daemon` | Run as rsync daemon | ❌ Divergent | Wave A: a real persistent listener. `fastsync-server --daemon --config FILE` (plus `--no-detach` to stay foreground; without it the listener detaches to the background after binding) reads a FastSync-native module config file and serves each connection confined to the requested module's `path` root (never a client-chosen root; every client-chosen-ownership/super-user request (`--numeric-ids`/`--chown`/`--usermap`/`--groupmap`/`--fake-super`/`--copy-as`/explicit `--super`) is refused unless the module opts in with `client owner = yes`, and the operator `--no-super` veto is honored). TCP/TLS via the existing `--tls` stack; plaintext still requires `--allow-unauthenticated` (same secure default as the standalone server). Client destinations use rsync's `host::module/path` form. Wire/protocol: the config frame gained a trailing daemon-module string and `PROTOCOL_VERSION` was bumped **2.14.0 → 2.15.0** (see the Daemon Mode notes below). Daemon mode is built in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding |
| `--config=FILE` | Alternate rsyncd.conf file | ❌ Divergent | Wave A: selects the daemon config file. Default when omitted (in `--daemon` mode): `~/.config/fastsync/fastsyncd.conf` if it exists, else `/etc/fastsyncd.conf`. The grammar is FastSync-native (documented in the Daemon Mode notes below) and strictly rejects unknown keys so a typo can never silently change what a module serves; requires `--daemon` |
| `--dparam=OVERRIDE` | Override global daemon config | ❌ Divergent | Wave A: overrides one global scalar from the command line (`--dparam port=8734` and `--dparam=KEY=VALUE` both work). Limited to the global keys the grammar defines (`port`, `motd file`, `address`, `max connections`, `max connections per host`, `auth failure delay`, `auth lockout threshold`, `auth lockout duration`, `hosts allow`, `hosts deny`); keys are case-insensitive and unknown keys/invalid values are rejected. Requires `--daemon` |
| `--config=FILE` | Alternate rsyncd.conf file | ❌ Divergent | Wave A: selects the daemon config file. Default when omitted (in `--daemon` mode): `~/.config/fastsync/fastsyncd.conf` if it exists, else `/etc/fastsyncd.conf`. The grammar is FastSync-native (documented in the Daemon Mode notes below) and still strictly rejects a genuinely unknown key so a typo can never silently change what a module serves; requires `--daemon`. **rsync 3.4.1 key subset accepted:** the common rsyncd.conf GLOBAL keys (`port`, `address`, `motd file`, `max connections`, `hosts allow`/`hosts deny`, plus the inert `pid file`, `log file`, `socket options`/`sockopts`, `listen backlog`, `syslog facility`, `syslog tag`, `log format`, `use chroot`, `uid`, `gid`, `timeout`, `max verbosity`/`min verbosity`, `lock file`, `transfer logging`, `strict modes`, `reverse lookup`/`forward lookup`, `ignore errors`, `ignore nonreadable`, `dont compress`) and MODULE keys (`path`, `read only`, `max connections`, `auth users`, `hosts allow`/`hosts deny`, plus the inert `comment`, `use chroot`, `uid`/`gid`/`daemon uid`/`daemon gid`, `exclude`, `include`, `exclude from`/`include from`, `filter`, `secrets file`, `auth digest`, `max verbosity`/`min verbosity`, `lock file`, `transfer logging`, `log file`/`log format`/`syslog facility`/`syslog tag`, `timeout`, `strict modes`, `numeric ids`, `fake super`, `munge symlinks`, `write only`, `list`, `dont compress`, `charset`, `refuse options`, `incoming chmod`/`outgoing chmod`, `open noatime`, `max size`/`min size`, `temp dir`, `pre-xfer exec`/`post-xfer exec`, `name converter`, `proxy protocol`/`proxy protocol hosts`, `reverse lookup`/`forward lookup`, `ignore errors`, `ignore nonreadable`) are recognized. Keys with a FastSync equivalent map onto it. Modules are **read-only by default**, exactly like rsync: `read only = no` (or `write only = yes`, which FastSync maps to writability because it is push-only) opts a module in; a global `read only` sets the default for later modules, and an explicit module value always wins. Keys with no FastSync equivalent load **inert** (no effect) rather than failing the whole config, and every inert key whose intent is access control (`secrets file`, `refuse options`, `exclude`/`include`/`filter`, `max size`/`min size`, `pre-xfer exec`/`post-xfer exec`, `incoming chmod`/`outgoing chmod`, `name converter`, `use chroot`, `uid`/`gid`, ...) emits a startup **WARN** naming the key (and module), so an operator cannot mistake an unenforced restriction for an enforced one. Residual: the native grammar still differs from rsync's (no `\` line continuation, `%VAR%` expansion, `[global]` re-entry, or inline `#` comments), and the inert keys are genuinely not enforced — in particular a daemon-side `exclude`/`filter` is NOT applied and `secrets file` is NOT read (use `path`, `--password-file`, and client-side filters instead) |
| `--dparam=OVERRIDE` | Override global daemon config | ❌ Divergent | Wave A: overrides one global scalar from the command line (`--dparam port=8734` and `--dparam=KEY=VALUE` both work). Reuses the exact same global-key dispatch as `--config`, so it accepts the native global keys (`port`, `motd file`, `address`, `read only`, `max connections`, `max connections per host`, `auth failure delay`, `auth lockout threshold`, `auth lockout duration`, `hosts allow`, `hosts deny`), the recognized inert rsync global keys, and rsync's compact spellings (`motdfile`, `pidfile`, `logfile`); keys are case-insensitive. `read only` sets the global default and re-applies it to every module that did not set its own value; the default is `yes` (rsync modules are read-only unless `read only = no` / `write only = yes`), so `--dparam read only=no` is required to make modules without their own value writable, and an inert security global key (`use chroot`, `uid`, `gid`, `strict modes`) emits the same startup **WARN** as `--config`. Genuinely unknown keys and invalid values are rejected. Requires `--daemon` |
| `--no-detach` | Don't detach from parent | ✅ Parity | Wave A: with `--daemon`, keeps the listener in the foreground (what integration tests use). Without it the daemonizes (fork/setsid, stdio redirected to /dev/null) after the listening socket is bound. Requires `--daemon` |
| `--password-file=FILE` | Read daemon password from file | ❌ Divergent | A7 daemon auth. Client: `--password-file` supplies `user:password` for a `host::module/path` destination (the username is taken from this file, so `user@host::module` stays rejected); the literal password is held client-side only for the SCRAM handshake and wiped at teardown. Server (`fastsync-server --daemon --password-file FILE`): the salted-PBKDF2 verifier store that modules with `auth users` are verified against. **Neither the password nor any replayable bearer value crosses the wire or is stored server-side** — the store holds a per-user salt plus derived keys, and the daemon proves the secret with a per-connection nonce challenge. The file must be private to its owner: both the client and server verify the exact inode they read (open-then-`fstat`, so the check cannot be raced) and refuse a `--password-file`/`--early-input` that is not owned by the current user or grants any group/other permission bit (mode 0600), mirroring the TLS private-key check. A process-substitution pipe (`--early-input <(vault ...)`) is still accepted when it satisfies those checks. **Hardening follow-up:** the file is opened with `O_NOFOLLOW`, so a symlinked credential path fails closed (`ELOOP`) instead of being followed before the owner/mode gate; literal fd-backed paths (`/dev/fd/<digits>`, `/proc/self/fd/<digits>`, which is what a bash process substitution passes) are exempt, so process substitution still works. A FIFO/process-substitution read now waits under a bounded ~3 s deadline for its writer, so a slow producer works while a connected-but-silent FIFO fails instead of hanging. See the Daemon Mode notes below for the file formats and the plaintext/TLS caveat |
| `--early-input=FILE` | Use FILE for daemon early exec | ❌ Divergent | Server-only (requires `--daemon`): a second credential-store file, same new-format grammar as `--password-file`, read before the listener accepts connections (a secrets-manager / process-substitution source). Its entries layer over `--password-file`: byte-identical verifiers dedupe, a conflicting verifier for the same user is a startup error. Opened with the same `O_NOFOLLOW` hardening as `--password-file` (a symlinked path fails closed with `ELOOP`; fd-backed `/dev/fd/N`/`/proc/self/fd/N` process-substitution paths are exempt) and a FIFO read is bound-waited (~3 s) so a slow producer works while a writer-less FIFO cannot hang. A daemon whose modules declare `auth users` must be given at least one of the two, or it refuses to start (fail closed) |
@@ -734,7 +747,7 @@ targets verbatim, matching rsync.
**Daemon Mode notes (Wave A protocol 2.15.0; A7 auth protocol 2.19.0; MOTD no bump):** FastSync daemon mode is supported in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding.
- **Config grammar** (`fastsyncd.conf`): line-based; an implicit global section first, then `[module]` sections. Keys are case-insensitive, values are trimmed and may be wrapped in one layer of double quotes (`path = "/srv/my dir"`). `#` and `;` at the start of a line (after leading whitespace) are full-line comments; inline comments and `\` continuations are not supported. Lines are bounded (4096 chars), and at most 256 `[module]` sections are accepted. Global keys: `port` (default 873), `motd file` (the daemon sends its bounded, escaped content to a client after the module gate/auth accepts, unless the client passes `--no-motd`), `address` (optional bind address), `max connections` (positive integer cap on concurrent connections, default 100; 0/negative/garbage is a parse error), `max connections per host` (concurrent-connection cap per source IP, default 0 = unlimited), `auth failure delay` (milliseconds to sleep after a failed authentication, default 500; 0 disables, capped at 5000), `auth lockout threshold` (failed authentications from one source before lockout, default 10; 0 disables), `auth lockout duration` (seconds a locked-out source is refused, default 300), `hosts allow` and `hosts deny` (comma- and/or whitespace-separated host access patterns — see the host access control note below). Module keys: `path` (required; the daemon-side authorized root for that module), `read only` (yes/no/true/false/1/0, default no), `client owner` (yes/no/true/false/1/0, default no; opts the module into client-chosen ownership — see below), `auth users` (comma list), `max connections` (optional per-module cap, 0 = unlimited; enforced across all connection children), `hosts allow`/`hosts deny` (per-module host access lists). **Unknown keys and malformed lines are parse-and-reject errors** (never silently ignored), so a typo cannot change what a module serves.
- **Config grammar** (`fastsyncd.conf`): line-based; an implicit global section first, then `[module]` sections. Keys are case-insensitive, values are trimmed and may be wrapped in one layer of double quotes (`path = "/srv/my dir"`). `#` and `;` at the start of a line (after leading whitespace) are full-line comments; inline comments and `\` continuations are not supported. Lines are bounded (4096 chars), and at most 256 `[module]` sections are accepted. Global keys: `port` (default 873), `motd file` (the daemon sends its bounded, escaped content to a client after the module gate/auth accepts, unless the client passes `--no-motd`), `address` (optional bind address), `max connections` (positive integer cap on concurrent connections, default 100; 0/negative/garbage is a parse error), `max connections per host` (concurrent-connection cap per source IP, default 0 = unlimited), `auth failure delay` (milliseconds to sleep after a failed authentication, default 500; 0 disables, capped at 5000), `auth lockout threshold` (failed authentications from one source before lockout, default 10; 0 disables), `auth lockout duration` (seconds a locked-out source is refused, default 300), `hosts allow` and `hosts deny` (comma- and/or whitespace-separated host access patterns — see the host access control note below). Module keys: `path` (required; the daemon-side authorized root for that module), `read only` (yes/no/true/false/1/0, default yes — rsync modules are read-only unless `read only = no`/`write only = yes`), `client owner` (yes/no/true/false/1/0, default no; opts the module into client-chosen ownership — see below), `auth users` (comma list), `max connections` (optional per-module cap, 0 = unlimited; enforced across all connection children), `hosts allow`/`hosts deny` (per-module host access lists). **Unknown keys and malformed lines are parse-and-reject errors** (never silently ignored), so a typo cannot change what a module serves. To reduce the rsync divergence, the parser additionally accepts the common rsync 3.4.1 GLOBAL and MODULE keys: the keys with a FastSync equivalent (`path`, `read only`, `max connections`, `auth users`, `hosts allow`/`hosts deny`, and the global `port`/`address`/`motd file`) map onto it, a global `read only` becomes the default for modules defined after it, and the keys with no FastSync equivalent (e.g. `pid file`, `log file`, `use chroot`, `uid`/`gid`, `comment`, `exclude`/`include`, `max verbosity`, `lock file`, `transfer logging`, `timeout`, `secrets file`) are recognized and loaded **inert** (accepted-but-ignored) instead of failing the whole file. `--dparam` reuses the same dispatch, so it also accepts the inert rsync global keys and the compact spellings `motdfile`/`pidfile`/`logfile`. A key outside both sets is still rejected. The inert keys are genuinely not enforced: a daemon-side `exclude`/`include`/`filter` is not applied and a `secrets file` is not read (use `--password-file`/`--early-input`), so an rsync config that relies on those must be edited rather than trusted.
- **Host access control (`hosts allow`/`hosts deny`):** both keys accept a comma- and/or whitespace-separated list of patterns and may appear globally and/or per module (multiple config-file lines append; a `--dparam` override replaces). Supported patterns are `*` (match all), an IPv4 or IPv6 literal (`10.0.0.1`, `2001:db8::1`), and an IPv4/IPv6 CIDR (`10.0.0.0/8`, `2001:db8::/32`). Hostname patterns are **not** supported: because the peer is always a numeric address and no reverse DNS is performed, a hostname/glob pattern would silently never match, so it is rejected at load time (fail-closed) instead of being accepted as a dead rule. An IPv4 peer on a dual-stack IPv6 listener is normalized from its `::ffff:a.b.c.d` form so IPv4 patterns match it. rsync-like semantics: a matching `hosts deny` rejects; if any `hosts allow` entries exist, a peer matching none of them is rejected; deny takes precedence over allow. The daemon enforces the global list first, then the selected module's list, **before authentication** in `server_module_gate`, with an audit log line naming the peer, the module and the outcome. The numeric peer address is obtained with `getpeername`+`inet_ntop` (`utils_fd_peer_ip`, handling both address families); when it cannot be obtained a module with any ACL fails closed (refused), while an ACL-free module continues and logs at debug. A malformed pattern (e.g. an out-of-range CIDR prefix) is a parse error at load time.
- **Connection caps, shared registry and auth lockout:** the global `max connections` key (default 100) is plumbed into the listener (`transport_tcp.c`), which rejects a connection once the accept-loop parent's active-child count reaches it; the IPv4/IPv6 peer is logged for every accepted connection. Because the listener forks one child per connection, the per-module `max connections` cap, the global `max connections per host` cap, and the auth-failure counter live in a fixed-size registry carved from an anonymous shared mapping (`daemon_limits.c`, `mmap(MAP_SHARED|MAP_ANONYMOUS)`) created by the parent before the accept loop, so every forked child shares the same counters (C11 atomics only — never a pthread lock, which can deadlock in a forked child). The parent reserves a registry slot per accepted connection and the child records the selected module and source IP once known; the parent's `SIGCHLD` handler reclaims the slot when the child dies (including `SIGKILL`) and re-derives the per-module and per-source occupancy counts from the surviving REGISTERED slots, so a child killed mid-registration cannot leak a count. The per-source table has a bounded lifetime: an entry with no live connection is reclaimed after its lockout expires or it has been idle (300 s); if the table is genuinely full the per-source cap/lockout fails open for new sources (per-module cap and ACLs still apply) with a rate-limited warning. The per-module cap (0 = unlimited) is enforced after the module lookup and before auth; per-source identity reuses the normalized numeric peer address (`utils_fd_peer_ip`, IPv4-mapped IPv6 collapsed to IPv4), and a trusted loopback peer (127.0.0.0/8 / `::1`, `utils_fd_peer_is_local`) is exempt from the per-source cap and the auth lockout because all local clients share one address (the per-module/global caps still apply). Clients behind a shared NAT/proxy address likewise share one per-source budget and lockout counter. A failed authentication increments the shared per-source failure count and, once `auth lockout threshold` (default 10; 0 disables) is reached, the source is refused for `auth lockout duration` seconds (default 300) before any challenge is sent, even when the next attempt is handled by a different forked child; a successful authentication clears the counter. On a failed authentication the per-connection child still sleeps the global `auth failure delay` (default 500 ms, 0 disables, capped at 5000) via `nanosleep`, rate-limiting online guessing without delaying a success. A missing registry (allocation failure) degrades to the global cap and host ACLs rather than refusing to start.
- **Module selection & confinement:** the client requests a module with an rsync-style `host::module[/path]` destination. The module name crosses the wire as a trailing string on the config frame (bumping `PROTOCOL_VERSION` 2.14.0 → 2.15.0; the bump is required because the config-frame layout changed and the strict same-version handshake is what prevents a peer from desynchronizing on the new trailing field). The daemon looks the module up in ITS OWN config and uses the module's `path` as the authorized root through the exact same `configure_authorization` confinement the standalone server applies to `--destination-root` (`file_open_secure_parent`, `has_path_traversal`, `path_is_within`); the client never supplies the root, every client-chosen-ownership/super-user request is refused unless the module declares `client owner = yes` (the daemon's per-module opt-in, see below), and the operator `--no-super` veto forces super-user activities off for every daemon connection. The client's `/path` part is relative inside the module and is rejected if absolute or if it contains `..`. Unknown modules are refused before any data moves (the run fails cleanly at the config handshake). An absolute destination and a module request against a non-daemon server are also refused.
@@ -791,7 +804,7 @@ modes or links.
| `--stop-after=MINS` | Stop after N minutes | ✅ Parity | Client-only sender stop deadline (Phase 6): computing `--stop-after=MINS` (a positive minute count; 0/negative/garbage rejected) and `--stop-at=TIME` (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`; a past time stops immediately). The transfer stops ELEGANTLY at the next chunk boundary: everything already fully sent is kept and applied, the run returns 0, and --delete (late/delete-after timing) does NOT wipe the destination — when the scan is cut short the partial keep-set manifest is suppressed with a warning (the delete walk is skipped rather than acting on an incomplete keep-set, so unscanned source mirrors survive). `--delete-before`/`--delete-during` still run their complete pre-scan (which ignores the deadline). Local client-only fields: never serialized into the wire config frame, so no PROTOCOL_VERSION bump. `--stop-after` uses CLOCK_MONOTONIC; `--stop-at` uses the wall clock. Works single-threaded and under `-j`/`--threads` (multithreaded). Divergence: rsync computes `--stop-after` from the run start; FastSync likewise. When both are given, the earlier of the two deadlines wins (checked per iteration). See the Phase-6 stop notes below |
| `--stop-at=TIME` | Stop at specified time | ✅ Parity | Deadline transfer stop (client-only, never serialized). Protocol 2.26.0 accepts rsync's full date/time grammar (`2030-12-31T23:59`, `2030/12/31T23:59`, `2030-12-31`, `12-31`, `14:00`, `:59`, `1`) in addition to FastSync's `HH:MM[:SS]` and `now+N[smhd]`; a past time stops immediately. Everything already transferred is kept and an early stop suppresses the late `--delete` keep-set so unscanned source mirrors survive. Works single-threaded and under `-j`/`--threads` |
| `--fsync` | Fsync every written file before publication | ✅ Parity | |
| `--protocol=NUM` | Force older protocol version | ❌ Divergent | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.28.0) with no downgrade/backward-compat code paths, so `--protocol=2.28.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.27.0`/`2.26.0`/`2.25.0`/`2.24.0`/`2.23.0`/`2.22.0`/`2.21.0`/`2.20.0`/`2.19.0`/`2.18.0`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
| `--protocol=NUM` | Force older protocol version | ❌ Divergent | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.29.0) with no downgrade/backward-compat code paths, so `--protocol=2.29.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.28.0`/`2.27.0`/`2.26.0`/`2.25.0`/`2.24.0`/`2.23.0`/`2.22.0`/`2.21.0`/`2.20.0`/`2.19.0`/`2.18.0`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
| `--iconv=CONVERT_SPEC` | Charset conversion | ✅ Parity | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, matching rsync's rule that the spec "stays the same whether you're pushing or pulling": on a PUSH the destination end's charset is the spec's REMOTE half, so the default receiver writes the wire bytes verbatim, and only a server started with its own `--iconv` (the daemon `charset` analog) declares a different destination charset and converts REMOTE→that LOCAL (rsync push parity, differential-tested with and without a server `--iconv`). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front; protocol 2.26.0 additionally accepts `--iconv=.` (the locale's default charset for both directions), `--iconv=-` and `--no-iconv` (disable conversion). Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below |
| `--checksum-seed=NUM` | Set checksum seed | ✅ Parity | Sets the seed for FastSync's whole-file xxHash digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). **As of protocol 2.23.0 a seed of `0` — the default when the flag is unset — is randomized per transfer and the chosen seed is sent to the receiver**, exactly like rsync, so two runs against different content do not share a predictable seed; an explicit non-zero seed is used verbatim, so an explicit seed deterministically reproduces every computed digest on BOTH endpoints (the seed crosses in the config frame). `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta` |
| `--secluded-args`, `-s` | Use protocol to send args | ❌ Divergent | Accepted for CLI compatibility (including the rsync short `-s`, Phase 7 Wave A) but a documented **no-op / divergence**. rsync's `-s` protects arguments from shell expansion by shipping them over the protocol; FastSync never passes remote arguments through a shell expansion boundary in the first place — its SSH transport builds the remote argv as **single-quote-escaped shell words** (`ssh_build_remote_command`), so the injection/leak that `-s` guards against does not exist and there is nothing to "seclude". Implementing a true arg-send protocol would mean replacing the argv-based SSH launch with an in-band argument channel, a large redesign of the transport that buys no security here. Chunk serialization remains the long-only `--chunk-serialization`. |
@@ -843,7 +856,7 @@ These features are moderate because they affect traversal, temporary files, mani
| `--relative`, `-R`; `--no-implied-dirs`; `--dirs`, `-d`; `--mkpath` | M | Extend path-list construction and destination directory creation while preserving traversal safety. |
| `--temp-dir`, `-T` | M | Separate temporary-file placement from FastSync's timeout alias and define collision, permissions, and cleanup rules. |
| `--delay-updates` | L | Stage all successful updates and publish them at completion, including crash and cancellation cleanup. |
| `--files-from=FILE`; `--from0`, `-0`; `--filter=RULE`, `-f`; `-F`; `--cvs-exclude`, `-C` | L | Build a complete filter/parser layer and integrate it with scanner pruning, manifests, and delete behavior. `-f` conflicts with FastSync sendfile mode. |
| `--files-from=FILE`; `--from0`, `-0`; `--filter=RULE`, `-f`; `-F`; `--cvs-exclude`, `-C` | L | Build a complete filter/parser layer and integrate it with scanner pruning, manifests, and delete behavior. (Done: `-f` is bound to `--filter`; the old sendfile conflict is gone, since sendfile is long-only `--sendfile`.) |
| `--list-only`; `--itemize-changes`, `-i`; `--out-format=FORMAT`; `--log-file-format=FMT` | M | Add a structured change-event model so output modes share one source of truth. |
### Phase 3: Deletion, Comparison, and Delta Compatibility
@@ -931,7 +944,7 @@ These are the last compatibility items and the closing phase toward rsync flag p
| `-T` / `--timeout` | `-T` = `--temp-dir` | → `--timeout` (long-only) |
| `-a` / `--archive` (= `-c -m -M`) | `-a` = `-rlptD` | → becomes **real rsync `-a`** after the renames |
**Wave B — Output & filesystem completion (✅ implemented).** `-S`/`--sparse` (`⚠️→✅`): real hole preservation — a sparse-aware writer (`write_all_sparse`) skips all-zero runs ≥ 4096 bytes with `lseek(SEEK_CUR)` and `ftruncate`s the final size, wired into both the atomic temp+rename store and `--inplace` receiver-side with **no wire change** (the full file image is already in memory; the ftruncate presize is kept). `-P` (`⚠️→✅`): interrupted-write retention — on a save failure after data reached the temp fd, `--partial` now renames the already-written temp to the destination path (best-effort; falls through to the normal unlink on failure, never retains when `--partial` is off) so a later `--append`/`--append-verify` run can resume. `--block-size=SIZE` (`⚠️→✅`): promoted after verification — `--block-size` is now an alias for `--delta-block`, both set `config->delta_block_size`, which the delta engine already honored end-to-end (`delta_signature_create_seeded` + `delta_apply`); out-of-range values keep the default. `--fake-super` (`⚠️→✅`): added `fake_super_restore_fd` to parse and re-apply the recorded `user.fastsync.stat` record fd-relative (mode/time only — protocol 2.23.0: **never a real chown**; the resolved owner is recorded for a later privileged restore); a save under `--fake-super` now re-applies the recorded attrs instead of only recording them, with the recording format unchanged. `--stderr=client` (`⚠️→❌ Divergent`): FastSync has no rsync client-message channel, and `client` is rejected at CLI parse — the rejection is the documented behavior (unit-tested). `-N`/`--crtimes` (`⚠️→❌ Divergent`): birth-times cannot be set by any portable fs call (`utimensat` sets only atime/mtime); capture/transmit stays, setting is impossible, the flag is accepted and safely inert. Review-hardening (post-eval): fake-super replay applies the mode through the shared `metadata_mode_for_policy` helper (protocol 2.23.0: exactly the source mode under `-p`, with no masking); `--sparse` takes precedence over `--preallocate` (posix_fallocate skipped so holes survive) — **reversed by the parity-completion wave: `--preallocate` now wins, matching rsync**; `--partial` retention is disabled for `--no_replace` (ignore/existing) and only marks a write-attempt after the actual write begins; `--block-size=SIZE`/`--delta-block=SIZE` inline forms are accepted.
**Wave B — Output & filesystem completion (✅ implemented).** `-S`/`--sparse` (`⚠️→✅`): real hole preservation — a sparse-aware writer (`write_all_sparse`) skips all-zero runs ≥ 4096 bytes with `lseek(SEEK_CUR)` and `ftruncate`s the final size, wired into both the atomic temp+rename store and `--inplace` receiver-side with **no wire change** (the full file image is already in memory; the ftruncate presize is kept). `-P` (`⚠️→✅`): interrupted-write retention — on a save failure after data reached the temp fd, `--partial` now renames the already-written temp to the destination path (best-effort; falls through to the normal unlink on failure, never retains when `--partial` is off) so a later `--append`/`--append-verify` run can resume. `--block-size=SIZE` (`⚠️→✅`): promoted after verification — `--block-size` is now an alias for `--delta-block`, both set `config->delta_block_size`, which the delta engine already honored end-to-end (`delta_signature_create_seeded` + `delta_apply`); out-of-range values keep the default. `--fake-super` (`⚠️→✅`): added `fake_super_restore_fd` to parse and re-apply the recorded `user.fastsync.stat` record fd-relative (mode/time only — protocol 2.23.0: **never a real chown**; the resolved owner is recorded for a later privileged restore); a save under `--fake-super` now re-applies the recorded attrs instead of only recording them. (The later fake-super xattr-interop pass replaced that native `user.fastsync.stat` format with rsync's `user.rsync.%stat` grammar — see the row and Phase-4 notes.) `--stderr=client` (`⚠️→❌ Divergent`): FastSync has no rsync client-message channel, and `client` is rejected at CLI parse — the rejection is the documented behavior (unit-tested). `-N`/`--crtimes` (`⚠️→❌ Divergent`): birth-times cannot be set by any portable fs call (`utimensat` sets only atime/mtime); capture/transmit stays, setting is impossible, the flag is accepted and safely inert. Review-hardening (post-eval): fake-super replay applies the mode through the shared `metadata_mode_for_policy` helper (protocol 2.23.0: exactly the source mode under `-p`, with no masking); `--sparse` takes precedence over `--preallocate` (posix_fallocate skipped so holes survive) — **reversed by the parity-completion wave: `--preallocate` now wins, matching rsync**; `--partial` retention is disabled for `--no_replace` (ignore/existing) and only marks a write-attempt after the actual write begins; `--block-size=SIZE`/`--delta-block=SIZE` inline forms are accepted.
**Wave C — Devices & special files (finalize statuses + tests) (✅ implemented).** The four special-file rows are finalized with coverage tests. `--devices`, `--copy-devices`, and `--write-devices` are **✅ Implemented**, each with a documented, safety-driven divergence: device-node creation is privilege-gated, so a receiver without `CAP_MKNOD` skips that entry with a warning (a per-entry skip, never a transfer failure); `--copy-devices` copies a device/FIFO's reported size into an ordinary regular file (a size-bounded safe divergence from rsync's unbounded dd-like read); `--write-devices` writes only into an existing char/block node under the confined receive root and skips every unusable target rather than clobbering or aborting. `--specials` reclassified from **⛔ Impossible/Divergence** to **✅ Parity** in protocol 2.23.0: **FIFO recreation works** (unprivileged `mkfifo`) **and unix sockets are recreated** with `mknod(S_IFSOCK)`, which Linux permits unprivileged (the flag previously assumed sockets were impossible — see the `--specials` row). Tests assert FIFO recreation, socket recreation, the regular-file result of `--copy-devices`, the skipped/missing and non-device `--write-devices` targets, and (root-gated) real device-node creation; a root runner additionally drops the receiver to an unprivileged user to assert the `CAP_MKNOD` skip is graceful. (The parity-completion wave later reclassified `--devices`, `--copy-devices`, and `--write-devices` as explicit **❌ Divergent** rows, because their safe subsets are deliberately not rsync's behavior; the implementation itself is unchanged.)
@@ -951,7 +964,7 @@ These are the last compatibility items and the closing phase toward rsync flag p
**Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity.
**Honest status after the parity 2.29 cycle (protocol 2.28.0, no wire change), updated by the parity cycle 2.29 pass and the audit-cycle follow-ups.** ✅ Parity 119 / ⚠️ Caveat 11 / ❌ Divergent 27 = 157 rows. The 2.29 cycle closed the scanner-order, delete-timing, relative-basis, and fuzzy-eligibility residuals (moving `-n`/`--delete`/`--del`/`--delete-delay` to ✅) and improved the `--info`/`--stats`/`--debug` partial rows; the remaining ⚠️ rows are `--info`, `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `--delete-before`, `--filter`, the three basis-dir options, and `-y/--fuzzy`. Earlier: **Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP
**Honest status after the parity 2.29 cycle (protocol 2.29.0 since the symlink-xattr wire wave, which adds no config-frame field and leaves this matrix unchanged), updated by the parity cycle 2.29 pass, the audit-cycle follow-ups, the triage cycle, and a later no-wire parity pass.** ✅ Parity 119 / ⚠️ Caveat 14 / ❌ Divergent 24 = 157 rows. The no-wire parity pass accepted `--inc-recursive`/`--no-inc-recursive` as inert no-ops (❌ → ✅, since FastSync's full scan is rsync's `--no-inc-recursive` and the destination is identical), narrowed the `--temp-dir` divergence by accepting an absolute path that canonicalizes inside the receive root (the row stays ❌ for out-of-root absolute paths), closed the `--delete-before` phase-0 divergence (⚠️ → ✅: both the single-threaded and the `--threads` data passes now replay the pre-scan file list, so a source file created after the scan is neither transferred nor kept, matching rsync), and moved `--fake-super` and `--devices` ❌ → ⚠️ (`--fake-super` now writes/reads rsync's exact `user.rsync.%stat` key and `<octal-mode> <rdev_major>,<rdev_minor> <uid>:<gid>` grammar, interoperating with real rsync 3.4.1 for regular files and faking char/block devices as regular files carrying the real rdev; `--devices` now logs a failed device `mknod` as a per-entry failure that continues the transfer instead of a silent non-root skip — see those rows for the remaining directory-faking and exit-code residuals). A review pass then hardened the fake-super stat parser (strict range-checked parsing), made rsync-style daemon modules read-only by default with a startup warning for accepted-but-unenforced access-control keys, and extended the `--delete-before` replay to the `--threads` path. The 2.29 cycle closed the scanner-order, delete-timing, relative-basis, and fuzzy-eligibility residuals (moving `-n`/`--delete`/`--del`/`--delete-delay` to ✅) and improved the `--info`/`--stats`/`--debug` partial rows; the triage cycle moved `-F` and `-i`/`--itemize-changes` ✅ → ⚠️ for their documented residuals. The remaining ⚠️ rows are `--info`, `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `-i`, `--filter`, `-F`, the three basis-dir options, `-y/--fuzzy`, `--fake-super`, and `--devices`. Earlier: **Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP
and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel /
receiver filter engine); the wire parity-track-4a pass later added that
receiver filter engine, flipping `--filter=RULE` back to ✅ (see above; the
@@ -990,13 +1003,17 @@ integration tests unless it is explicitly listed as a limitation.
### Checksums and compression
- **`--checksum-choice`/`--cc`** accepts `xxh64` (default), `xxhash`, `xxh3`,
`xxh128`, `md5`, and `auto`; `md4`, `sha1`, `none`, and the two-name
`transfer,pre-transfer` form are **rejected by name**.
- **`--checksum-choice`/`--cc`** accepts the full rsync 3.4.1 set: `xxh64`
(default), `xxhash`, `xxh3`, `xxh128`, `md5`, `md4`, `sha1`, `none`, the
two-name `transfer,pre-transfer` form, and `auto` (which honors
`RSYNC_CHECKSUM_LIST` before the compiled-in order). A genuinely unknown name
is still rejected by name, matching rsync.
- **`--checksum-seed=0` is randomized per transfer** (the chosen seed is sent to
the receiver), matching rsync; an explicit non-zero seed is used verbatim.
- **`--compress-choice`/`--zc`** accepts `zstd` (default), `none`, and `auto`;
rsync's `lz4`/`zlib`/`zlibx` are **rejected by name**.
- **`--compress-choice`/`--zc`** accepts the full rsync 3.4.1 set: `zstd`
(default), `lz4`, `zlib`, `zlibx`, `none`, and `auto` (which honors
`RSYNC_COMPRESS_LIST` before the compiled-in order). A genuinely unknown name
is still rejected by name, matching rsync.
- **`--skip-compress`** uses rsync 3.4.1's built-in default suffix list when no
list is supplied; an explicit list replaces it.
- **`--no-whole-file`** is accepted as the rsync spelling that clears
@@ -1012,8 +1029,15 @@ integration tests unless it is explicitly listed as a limitation.
### Filesystem and deletion semantics
- **`--temp-dir` is confined to the receive root on the receiver:** a relative
dir resolves below it; an absolute path or one containing `..` is rejected.
An `EXDEV` install falls back to a non-atomic copy instead of aborting.
dir resolves below it, and an absolute path is accepted only when
`realpath(3)` confirms it is inside the canonical receive root; an
out-of-root absolute path or one containing `..` is rejected.
An `EXDEV` install falls back to a non-atomic copy instead of aborting. (The
confined receiver path cannot be mount-tested in the CI container — no
`CAP_SYS_ADMIN` and unprivileged user namespaces are disabled — so the
cross-filesystem fallback is exercised end-to-end through the unconfined local
`--read-batch` apply against a `/dev/shm` scratch dir, in
`tests/integration/test_temp_dir_exdev.py`.)
- **Deletion scoping:** the manifest carries the synchronized directories, so
the extras walk only visits their subtrees; `--files-from` subsets no longer
delete untransmitted paths outside the listed directories.
@@ -1035,35 +1059,42 @@ integration tests unless it is explicitly listed as a limitation.
- **`--numeric-ids` is a mapping modifier only** — it changes *how* ids map, not
*whether* ownership is applied; combine it with `-o`/`-g`, `-a`, or an
explicit map.
- **`--usermap`/`--groupmap`** support names, `@N`/bare `N` ids, inclusive
`LOW-HIGH` ranges, `*`, empty-`FROM` (unnamed ids), and receiver-resolved `TO`
names.
- **`--usermap`/`--groupmap`** support names, FROM name **globs**
(`*`/`?`/`[...]`, expanded sender-side against the passwd/group database and
bounded by `MAX_IDENTITY_MAP`), `@N`/bare `N` ids, inclusive `LOW-HIGH` ranges,
`*`, empty-`FROM` (unnamed ids), and receiver-resolved `TO` names.
- **`--chown` conflicts with `--usermap`/`--groupmap` on the same side** and is a
clear configuration error (matching rsync) instead of an order-dependent
winner.
- **`--fake-super` never real-chowns.** It records the *resolved* owner (the
active mapping, else the source id) in `user.fastsync.stat` for a later
privileged restore and replays only mode/times. Directory ownership and
directory xattrs/ACLs are preserved alongside file entries.
active mapping, else the source id) in rsync's `user.rsync.%stat` for a later
privileged restore and replays only the permission bits (mtime travels through
the normal metadata path). Directory ownership and
directory xattrs/ACLs are preserved alongside file entries, though directories
themselves are not yet given a `%stat%` record.
- **`--chmod`** implements rsync's `D`/`F`/`X` selectors, `s`/`t`, append
semantics, does not imply `-p`, and applies its changes without sanitization.
### Symlinks and special files
- **`-l`/`--links` stores symlink targets verbatim** (absolute and `..`-bearing
targets included), matching rsync. `--safe-links`, `--copy-unsafe-links`, and
`--munge-links` (which now uses rsync's `/rsyncd-munged/` marker) match rsync
and are applied sender-side.
targets included), matching rsync. `--safe-links` and `--copy-unsafe-links`
match rsync and are applied sender-side; `--munge-links` (which now uses
rsync's `/rsyncd-munged/` marker) matches rsync too but is applied
**receiver-side** (the sender un-munges an already-marked source target).
- **`--specials` recreates unix sockets** with `mknodat(..., S_IFSOCK)`, so
`-D`/`--devices --specials` now covers the full rsync node set.
- **`--copy-devices`** is implemented (see its caveat below).
### Output
- **`-i`/`--out-format`** print rsync-style change lines; **`--list-only`**
- **`-i`/`--out-format`** print rsync-style change lines, including the
transfer-root `./` and per-directory `cd...`/`.d..t...` lines; **`--list-only`**
scans the source only and contacts no server; **`-h`** uses rsync's decimal
units; **`--progress`** is an aggregate line; **`--stats`** prints the counters
FastSync can observe locally (receiver-only counters are 0).
units; **`--progress`** prints rsync-style per-file progress blocks;
**`--stats`** prints the transfer-statistics block, whose receiver-only
counters (`Matched data`, `Number of deleted files`) are populated from the
receiver's `STATUS_STATS` report.
- **Server `--port`** is an alias of the `-p <port>` TCP listen port
(`--dparam port=` overrides the daemon config).
@@ -1076,8 +1107,9 @@ These remain after the wave; they are the reasons a row above is ⚠️.
link-following tool can follow a link outside the receive root. Use
`--safe-links` when the source is untrusted. `--trust-sender` does **not**
affect symlink targets.
- **`--temp-dir` absolute/foreign-filesystem paths are rejected by the
receiver** (rsync's daemon also confines; standalone rsync differs).
- **`--temp-dir` out-of-root absolute and foreign-filesystem paths are rejected
by the receiver** (an absolute path that canonicalizes inside the receive root
is accepted; rsync's daemon also confines; standalone rsync differs).
- **`--copy-devices` reads a bounded `st_size`** rather than rsync's unbounded
device read.
- **A broken symlink referent under `--copy-links`/`--copy-unsafe-links` exits 0**
@@ -1085,8 +1117,10 @@ These remain after the wave; they are the reasons a row above is ⚠️.
- **New directories without `-p` still use FastSync's `0755` creation default**
rather than `source & ~umask`; directory metadata is only applied when a
directory attribute is requested.
- **`--stats` receiver-only counters** (matched data, file-list bytes, deleted
count) are reported as 0; `--progress` is an aggregate line, not per-file.
- **`--stats` byte totals** (`Total bytes sent`/`received`) are FastSync wire
bytes framed differently from rsync's, so they are not numerically comparable;
the remaining `--stats`/`--progress` divergences are the ones named in their
rows (per-type deleted-file breakdown, root-line/ancestor suppression).
- **`--password-file`/`--early-input`/`--hash-credentials`/`--iterations` are
FastSync-native** (SCRAM/PBKDF2), not rsync semantics; the batch format is not
rsync-interoperable. Credential files are opened with `O_NOFOLLOW` (a symlinked
@@ -1214,8 +1248,9 @@ These remain after the wave; the individual rows carry the precise wording.
`--ignore-errors` exits 23 but its EACCES differential is not
exercised in CI.
- **`--delay-updates`** uses a fixed staging name with an advisory lock and
deletes before publication; **`--temp-dir`** rejects absolute/foreign paths
(deliberately confined, see the row); **`--remote-option`** is SSH-only.
deletes before publication; **`--temp-dir`** rejects out-of-root absolute and
foreign paths (in-root absolute paths are accepted; deliberately confined, see
the row); **`--remote-option`** is SSH-only.
**`--iconv`** now matches rsync's push direction (destination charset = the
spec's REMOTE half; a server `--iconv` overrides it).
- **Basis dirs** now use rsync's metadata quick-check by default (track 5a) and
@@ -1225,17 +1260,18 @@ These remain after the wave; the individual rows carry the precise wording.
name heuristic, but its candidate eligibility is bounded by the delta
engine (both files ≥ 16 KiB, size ratio ≤ 10×), a narrower window than
rsync's, so the selected basis — and the `--stats` bandwidth counters —
can differ while the tree stays byte-exact; and **`--bwlimit`** rejects rsync's
`0`/decimal/suffixed rates.
- **`--inc-recursive`/`--no-inc-recursive`** are not implemented (rejected).
can differ while the tree stays byte-exact.
- **`--inc-recursive`/`--no-inc-recursive`** are accepted as inert no-ops:
FastSync always performs a single full recursive scan (equivalent to
rsync's `--no-inc-recursive`), so the destination is identical either way.
### Intentional divergences (explicit ❌ rows)
Native daemon config/auth (`--daemon`, `--config`, `--dparam`,
`--password-file`, `--early-input`, `--hash-credentials`/`--iterations`), the
non-interoperable batch container (`--write-batch`/`--only-write-batch`/
`--read-batch`), `--fake-super`'s native xattr format, `-X`'s privileged
namespaces, `--devices`/`--copy-devices`/`--write-devices`'s safe subsets,
`--read-batch`), `-X`'s privileged
namespaces, `--copy-devices`/`--write-devices`'s safe subsets,
`--super`/`--copy-as`'s refusal to elevate or switch credentials, and the
`-s`/`--secluded-args`/`--protect-args`/`--old-args` accepted no-ops.
+12 -5
View File
@@ -141,6 +141,10 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
update = 'h';
else if (created)
update = (event->is_directory || event->is_symlink || event->is_special) ? 'c' : '>';
else if (event->is_directory)
/* rsync: an existing directory that only has attribute changes carries no
transfer, so the update column is `.` rather than `>`. */
update = '.';
else
update = '>';
code[0] = update;
@@ -168,12 +172,15 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
code[11] = '\0';
}
/* rsync %n: the transfer-relative name, with a trailing slash for directories. */
/* rsync %n: the transfer-relative name, with a trailing slash for directories.
* The transfer root is `.` (so `%n` renders `./`), matching rsync's root entry. */
static bool append_name(StrBuf* buf, const ChangeEvent* event) {
if (!strbuf_append(buf, event->name != NULL ? event->name : ""))
const char* name = event->name != NULL ? event->name : "";
if (event->is_directory && name[0] == '\0')
return strbuf_append(buf, "./");
if (!strbuf_append(buf, name))
return false;
if (event->is_directory && (event->name == NULL || event->name[0] == '\0' ||
event->name[strlen(event->name) - 1] != '/'))
if (event->is_directory && name[strlen(name) - 1] != '/')
return strbuf_append_char(buf, '/');
return true;
}
@@ -245,7 +252,7 @@ static char* change_render_name_uptodate(const ChangeEvent* event) {
* resolves to xxh128, so an explicit selection and the default both render the
* selected algorithm's digest. */
static ChecksumAlgo out_format_checksum_algo(const Config* config) {
return (ChecksumAlgo)config->checksum_transfer_algo;
return (ChecksumAlgo)config->cli.checksum_transfer_algo;
}
/* Render a digest as rsync's sum_as_hex: xxh128 prints the HIGH 64-bit half
+47 -31
View File
@@ -170,7 +170,7 @@ static int set_positive_int_option(int* dest, const char* value, const char* opt
* name is a hard error with rsync's exit code 4, never a silent no-op. */
static int set_compression_choice(Config* config, const char* value) {
if (!value) {
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
int algo;
@@ -178,7 +178,7 @@ static int set_compression_choice(Config* config, const char* value) {
algo = compression_choice_resolve();
if (algo < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
} else {
@@ -189,7 +189,7 @@ static int set_compression_choice(Config* config, const char* value) {
"--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, "
"lz4, zlib, zlibx, none or auto",
value);
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
const char* canonical = compression_algo_name((CompressionAlgo)algo);
@@ -226,7 +226,7 @@ static int resolve_checksum_name(const char* name, size_t len, int* out) {
* resolves to FastSync's negotiated default (xxh128). */
static int set_checksum_choice(Config* config, const char* value) {
if (!value) {
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
const char* comma = strchr(value, ',');
@@ -244,7 +244,7 @@ static int set_checksum_choice(Config* config, const char* value) {
"--checksum-choice '%s' is invalid; FastSync supports xxh64 (or xxhash), xxh128, "
"xxh3, md5, md4, sha1, none or auto, optionally as 'transfer,pre-transfer'",
value);
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
int negotiated = -1;
@@ -252,7 +252,7 @@ static int set_checksum_choice(Config* config, const char* value) {
negotiated = checksum_choice_resolve();
if (negotiated < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
}
@@ -264,8 +264,8 @@ static int set_checksum_choice(Config* config, const char* value) {
pre = negotiated;
config->checksum_algo = pre;
config->checksum_transfer_algo = transfer;
config->checksum_choice_set = true;
config->cli.checksum_transfer_algo = transfer;
config->cli.checksum_choice_set = true;
/* rsync: "none" for the transfer checksum forces --whole-file. */
if (transfer == (int)CHECKSUM_ALGO_NONE)
config->whole_file = true;
@@ -962,8 +962,18 @@ static const OptionEntry OPTION_TABLE[] = {
/* rsync -r/--recursive: FastSync is always recursive, so this is a
* faithful no-op (accepted silently, never consumes an argument). */
{"--recursive", "-r", OPT_NOOP, 0},
/* rsync's incremental-recursion scan-mode switch. FastSync always performs
* a single full recursive scan, so both spellings are accepted as no-ops:
* the destination is identical whichever mode the caller requests.
* --no-inc-recursive is handled before the generic --no-* negation branch
* (see cli_handle_pre_negation) but is registered here for discoverability. */
{"--inc-recursive", NULL, OPT_NOOP, 0},
{"--no-inc-recursive", NULL, OPT_NOOP, 0},
{"--update", "-u", OPT_FLAG, offsetof(Config, update)},
{"--old-args", NULL, OPT_FLAG, offsetof(Config, old_args)},
/* rsync's --old-args: accepted for CLI compatibility as a documented no-op
* (the remote server path is always safely quoted; see usage.c). It is
* recognized but stores no Config field. */
{"--old-args", NULL, OPT_NOOP, 0},
{"--rsh", "-e", OPT_STRING, offsetof(Config, rsh_command)},
{"--blocking-io", NULL, OPT_FLAG, offsetof(Config, blocking_io)},
{"--links", "-l", OPT_FLAG, offsetof(Config, follow_symlinks)},
@@ -1196,12 +1206,12 @@ static int apply_negation(Config* config, const char* arg) {
config->preserve_times = false;
config->preserve_owner = false;
config->preserve_group = false;
config->metadata_explicitly_disabled = true;
config->cli.metadata_explicitly_disabled = true;
/* --no-preserve is an explicit opt-out of the whole bundle: record it so
* the --incremental/--delta auto-preserve in cli_finalize_config does not
* silently re-enable perms/times. */
config->preserve_perms_explicit_off = true;
config->preserve_times_explicit_off = true;
config->cli.preserve_perms_explicit_off = true;
config->cli.preserve_times_explicit_off = true;
return 0;
}
*(bool*)((char*)config + entry->offset) = false;
@@ -1209,9 +1219,9 @@ static int apply_negation(Config* config, const char* arg) {
* auto-preserve the OTHER attribute without undoing this one. A later
* -p/-t sets the attribute directly; this flag only gates the implication. */
if (entry->offset == offsetof(Config, preserve_perms))
config->preserve_perms_explicit_off = true;
config->cli.preserve_perms_explicit_off = true;
else if (entry->offset == offsetof(Config, preserve_times))
config->preserve_times_explicit_off = true;
config->cli.preserve_times_explicit_off = true;
return 0;
}
@@ -1384,6 +1394,12 @@ static bool cli_handle_pre_negation(CliParseCtx* ctx) {
ctx->no_delta = true;
else if (strcmp(arg, "--no-incremental") == 0)
ctx->no_incremental = true;
/* Real rsync option names that merely start with "--no-" and are inert
* no-ops (e.g. --no-inc-recursive) are registered as OPT_NOOP entries;
* accept them before the generic negation table would reject the name. */
const OptionEntry* noop = find_table_option(arg);
if (noop && noop->kind == OPT_NOOP)
return true;
if (apply_negation(config, arg) != 0) {
ctx->exit_code = -1;
return true;
@@ -1440,7 +1456,7 @@ static bool cli_handle_range_time_options(CliParseCtx* ctx) {
ctx->exit_code = -1;
return true;
}
config->stop_at_set = true;
config->cli.stop_at_set = true;
return true;
}
if (strcmp(arg, "--stop-at") == 0) {
@@ -1455,7 +1471,7 @@ static bool cli_handle_range_time_options(CliParseCtx* ctx) {
ctx->exit_code = -1;
return true;
}
config->stop_at_set = true;
config->cli.stop_at_set = true;
return true;
}
const char* threads_prefix = "--compress-threads=";
@@ -1526,7 +1542,7 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
return true;
}
if (entry->offset == offsetof(Config, compression_level))
config->compression_level_set = true;
config->cli.compression_level_set = true;
if (entry->offset == offsetof(Config, chmod_spec)) {
mode_t ignored;
if (!chmod_apply(0, config->chmod_spec, &ignored)) {
@@ -1539,7 +1555,7 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
defaults to 127.0.0.1, so a value check cannot distinguish it). Used
by --dry-run to route an explicit remote target to the server. */
if (entry->offset == offsetof(Config, server_host))
config->server_host_set = true;
config->cli.server_host_set = true;
}
} else if (apply_table_option(config, entry, NULL) != 0) {
ctx->exit_code = -1;
@@ -1813,7 +1829,7 @@ static bool cli_handle_transfer_flags(CliParseCtx* ctx) {
return true;
}
config->compression_level = (int)level;
config->compression_level_set = true;
config->cli.compression_level_set = true;
log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level);
ctx->i++;
}
@@ -1877,7 +1893,7 @@ static int set_server_port_option(Config* config, const char* value, const char*
return -1;
}
config->server_port = port;
config->server_port_set = true;
config->cli.server_port_set = true;
return 0;
}
@@ -2648,7 +2664,7 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
int resolved = compression_choice_resolve();
if (resolved < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
config->compression_algo = resolved;
@@ -2659,7 +2675,7 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
* clamped to the codec's range, otherwise the codec's own default is used. */
if (config->use_compression) {
CompressionAlgo algo = (CompressionAlgo)config->compression_algo;
config->compression_level = config->compression_level_set
config->compression_level = config->cli.compression_level_set
? compression_clamp_level(algo, config->compression_level)
: compression_default_level(algo);
log_debug_message(LOG_DEBUG_UTIL, "Client compression: %s (level %d)",
@@ -2668,22 +2684,22 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
/* The negotiated checksum is always resolved (rsync negotiates one for the
* delta strong sum even without --checksum): RSYNC_CHECKSUM_LIST first, then
* the compiled-in order. An explicit --checksum-choice already set it. */
if (!config->checksum_choice_set) {
if (!config->cli.checksum_choice_set) {
int resolved = checksum_choice_resolve();
if (resolved < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
config->checksum_algo = resolved;
config->checksum_transfer_algo = resolved;
config->cli.checksum_transfer_algo = resolved;
}
/* rsync parity: "none" as the pre-transfer checksum cannot be combined with
* --checksum (exit 4). The check runs here because --checksum may appear on
* either side of --checksum-choice. */
if (config->checksum && config->checksum_algo == (int)CHECKSUM_ALGO_NONE) {
log_message(LOG_LEVEL_ERROR, "Invalid checksum-choice for --checksum: none");
config->cli_exit_code = 4;
config->cli.cli_exit_code = 4;
return -1;
}
@@ -2762,11 +2778,11 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
* explicitly negated them (--no-perms/--no-times/--no-preserve). This runs
* BEFORE the derived use_metadata bit so the transport frame is still sent
* for the incremental/delta handshake even when both attributes were negated
* via --no-preserve (metadata_explicitly_disabled handles that opt-out). */
if (preserve_implied && !config->metadata_explicitly_disabled) {
if (!config->preserve_perms_explicit_off)
* via --no-preserve (cli.metadata_explicitly_disabled handles that opt-out). */
if (preserve_implied && !config->cli.metadata_explicitly_disabled) {
if (!config->cli.preserve_perms_explicit_off)
config->preserve_perms = true;
if (!config->preserve_times_explicit_off)
if (!config->cli.preserve_times_explicit_off)
config->preserve_times = true;
}
@@ -3153,7 +3169,7 @@ int main(int argc, char* argv[]) {
int parse_ret = parse_args(config, argc, argv, positional_args, &positional_count);
if (parse_ret != 0) {
if (parse_ret < 0)
exit_code = config->cli_exit_code ? config->cli_exit_code : 1;
exit_code = config->cli.cli_exit_code ? config->cli.cli_exit_code : 1;
goto cleanup;
}
+1 -1
View File
@@ -31,7 +31,7 @@ bool dry_run_targets_server(const Config* config) {
return true;
if (config->module && config->module[0] != '\0')
return true;
if (config->server_host_set || config->server_port_set)
if (config->cli.server_host_set || config->cli.server_port_set)
return true;
if (config->use_tls)
return true;
+375 -26
View File
@@ -243,17 +243,31 @@ void transfer_stats_note_transferred(TransferStats* stats, const File* file) {
#define RSYNC_PROGRESS_IO_WINDOW (32ULL * 1024ULL)
/* Paths-only pre-count of the source file list, built once at transfer start
* when progress output is requested. rsync's `to-chk` denominator is the whole
* file list -- every regular file, directory, symlink and special plus the
* transfer root -- while the streaming scan never emits directories. A
* metadata-only walk (no file reads, no hashing) supplies that total and the
* directory names, so the opt-in pass leaves non-progress runs untouched. */
* when progress output or -i/--out-format needs it. rsync's `to-chk`
* denominator is the whole file list -- every regular file, directory, symlink
* and special plus the transfer root -- while the streaming scan only emits
* empty directories. A metadata-only walk (no file reads, no hashing) supplies
* that total and a metadata-bearing File for every directory, so --progress can
* name them and -i/--out-format can itemize them without a second full scan. */
/* One directory in the pre-count, keyed by its transfer-relative display name
* ("" is the transfer root). `file` is owned by ProgressPrecount.dir_files and
* carries the source metadata needed by -i/--out-format (%M/%B/%U/%G). */
typedef struct {
char* name; /* owned */
File* file;
} DirRef;
typedef struct {
unsigned long long total;
ArrayList* dir_paths; /* owned char* in transfer-relative display form */
ArrayList* dir_files; /* owned File* captured during the metadata walk */
ArrayList* dir_refs; /* owned DirRef*, sorted by name for prefix lookup */
} ProgressPrecount;
static bool g_progress_active;
/* True when -i/--out-format need the pre-counted directory entries fed into the
* change-event stream (independent of --progress). */
static bool g_change_dirs_active;
static unsigned long long g_progress_xferred;
static unsigned long long g_progress_index;
static unsigned long long g_progress_total;
@@ -270,14 +284,101 @@ bool progress_requested(const Config* config) {
(config->show_progress || (config->info_level & LOG_INFO_PROGRESS) != 0);
}
static void dir_ref_destroy(void* item) {
DirRef* ref = (DirRef*)item;
if (ref == NULL)
return;
free(ref->name);
free(ref);
}
/* Sort DirRef pointers by their transfer-relative name for binary search. */
static int dir_ref_compare(const void* left, const void* right) {
const DirRef* a = *(const DirRef* const*)left;
const DirRef* b = *(const DirRef* const*)right;
return strcmp(a->name, b->name);
}
/* Look up the pre-counted directory File for a transfer-relative name ("" is
* the transfer root). Returns NULL when no pre-count was built or the name is
* not a known directory. */
static File* progress_dir_lookup(const char* name) {
if (name == NULL || g_progress_precount.dir_refs == NULL)
return NULL;
ArrayList* refs = g_progress_precount.dir_refs;
size_t lo = 0;
size_t hi = (size_t)refs->size;
while (lo < hi) {
size_t mid = lo + (hi - lo) / 2;
DirRef* ref = (DirRef*)refs->items[mid];
int cmp = strcmp(ref->name, name);
if (cmp < 0)
lo = mid + 1;
else if (cmp > 0)
hi = mid;
else
return ref->file;
}
return NULL;
}
static void progress_precount_dispose(ProgressPrecount* p) {
if (p->dir_paths != NULL) {
array_list_delete(p->dir_paths);
p->dir_paths = NULL;
}
if (p->dir_files != NULL) {
array_list_delete(p->dir_files);
p->dir_files = NULL;
}
if (p->dir_refs != NULL) {
array_list_delete(p->dir_refs);
p->dir_refs = NULL;
}
p->total = 0;
}
/* Record the transfer root's pre-transfer state for -i/--out-format. The
* receive root always exists, so rsync never marks it `cd`; its only observable
* change is its timestamp, which FastSync cannot observe remotely. Force a time
* mismatch so the root renders rsync's `.d..t...... ./` rather than the `cd`
* a zeroed destination state would produce. */
static void progress_precount_mark_root(File* root) {
if (root == NULL)
return;
root->dest_state.known = true;
root->dest_state.existed = true;
root->dest_state.mode = root->metadata != NULL ? root->metadata->mode : 0;
root->dest_state.uid = root->metadata != NULL ? root->metadata->uid : 0;
root->dest_state.gid = root->metadata != NULL ? root->metadata->gid : 0;
root->dest_state.size = 0;
root->dest_state.mtime_sec = (root->metadata != NULL ? root->metadata->mtime_sec : 0) - 3600;
root->dest_state.mtime_nsec = root->metadata != NULL ? root->metadata->mtime_nsec : 0;
}
/* Append one DirRef (name -> file) to the pre-count, marking the transfer
* root's destination state. Returns false on allocation failure. */
static bool progress_precount_add_ref(ProgressPrecount* p, const Config* config, File* file) {
const char* rel = delete_display_path(config, file_wire_path(file));
char* name = rel != NULL ? str_dup(rel) : NULL;
if (name == NULL)
return false;
DirRef* ref = malloc(sizeof(*ref));
if (ref == NULL) {
free(name);
return false;
}
ref->name = name;
ref->file = file;
if (name[0] == '\0')
progress_precount_mark_root(file);
if (!array_list_add(p->dir_refs, ref)) {
dir_ref_destroy(ref);
return false;
}
return true;
}
void client_progress_cleanup(void) {
if (g_progress_dir_index_valid) {
path_index_free(&g_progress_dir_index);
@@ -293,6 +394,7 @@ void client_progress_cleanup(void) {
}
progress_precount_dispose(&g_progress_precount);
g_progress_active = false;
g_change_dirs_active = false;
g_progress_total = 0;
g_progress_index = 0;
g_progress_xferred = 0;
@@ -395,6 +497,11 @@ void print_delete_reports(const Config* config, const ArrayList* paths) {
fflush(stdout);
}
/* Emit every not-yet-seen ancestor directory of `rel`, outermost first, in the
* order rsync's depth-first flist walk visits them. With -i/--out-format each
* ancestor becomes a real change line (`cd+++++++++ sub/`, `.d..t...... ./`)
* rendered by the shared itemize code; otherwise it is the `--info=name` /
* --progress directory name line. */
static void client_progress_emit_ancestors(const Config* config, const char* rel) {
if (!g_progress_dir_index_valid || !g_progress_emitted_valid || g_progress_emitted_keys == NULL ||
rel == NULL)
@@ -413,9 +520,15 @@ static void client_progress_emit_ancestors(const Config* config, const char* rel
char* key = str_dup(prefix);
if (key != NULL && array_list_add(g_progress_emitted_keys, key)) {
str_hash_set_insert_ref(&g_progress_emitted, key);
char* escaped = output_escape(prefix, config->eight_bit_output);
printf("%s/\n", escaped ? escaped : prefix);
free(escaped);
if (g_change_dirs_active) {
const File* dir = progress_dir_lookup(prefix);
if (dir != NULL)
change_emit_dir_sent(config, dir);
} else {
char* escaped = output_escape(prefix, config->eight_bit_output);
printf("%s/\n", escaped ? escaped : prefix);
free(escaped);
}
g_progress_index++;
} else {
free(key);
@@ -425,6 +538,20 @@ static void client_progress_emit_ancestors(const Config* config, const char* rel
}
}
/* Feed a transferred entry's ancestor directories into the change-event stream
* before the entry's own line, so -i/--out-format and --progress report
* directories in rsync's depth-first order. Every directory is an ancestor of
* some emitted entry (a file, symlink, special, hard link or the empty-directory
* entry the scanner emits for a leaf), so this covers the whole tree. */
void client_change_emit_ancestors(const Config* config, const File* file) {
if (config == NULL || file == NULL)
return;
if (!g_progress_active && !g_change_dirs_active)
return;
const char* rel = delete_display_path(config, file_wire_path(file));
client_progress_emit_ancestors(config, rel);
}
/* rsync's --info=name/progress line for one entry: transfer-relative name (a
* trailing slash for directories) plus the ` -> target` symlink suffix. */
static char* progress_entry_line(const File* file, const char* rel) {
@@ -462,7 +589,7 @@ void client_progress_begin(const Config* config) {
g_progress_active = progress_requested(config);
g_progress_xferred = 0;
g_progress_index = 1; /* the transfer root is file-list entry #0 */
if (!g_progress_active) {
if (!g_progress_active && !g_change_dirs_active) {
/* `--info=flist` prints rsync's file-list header even without progress. */
if (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST)) {
printf("sending incremental file list\n");
@@ -470,11 +597,21 @@ void client_progress_begin(const Config* config) {
}
return;
}
printf("sending incremental file list\n");
/* rsync prints the transfer-root directory's name before the first file when
that directory is created; FastSync mirrors the source root below the
receive root and creates it on a fresh destination, so emit it here. */
printf("./\n");
/* -i/--out-format alone do not print the header, but --progress always does
and --info=flist does under any output mode (rsync prints it for
`-i --info=flist` and `--out-format=... --info=flist` too). */
if (g_progress_active || (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST)))
printf("sending incremental file list\n");
/* rsync prints the transfer-root directory before the first entry. Under
-i/--out-format it is the root change line (`.d..t...... ./`); otherwise it
is the plain --info=name / --progress name line. */
if (g_change_dirs_active) {
const File* root = progress_dir_lookup("");
if (root != NULL)
change_emit_dir_sent(config, root);
} else {
printf("./\n");
}
fflush(stdout);
}
@@ -487,7 +624,6 @@ void client_progress_file(const Config* config, const File* file) {
unsigned long long size = file->data->size;
if (!config->itemize_changes && config->out_format == NULL) {
const char* rel = delete_display_path(config, file_wire_path(file));
client_progress_emit_ancestors(config, rel);
char* escaped = output_escape(rel, config->eight_bit_output);
printf("%s\n", escaped ? escaped : (rel ? rel : ""));
free(escaped);
@@ -510,7 +646,6 @@ void client_progress_name(const Config* config, const File* file) {
return;
const char* rel = delete_display_path(config, file_wire_path(file));
if (!config->itemize_changes && config->out_format == NULL) {
client_progress_emit_ancestors(config, rel);
char* line = progress_entry_line(file, rel ? rel : "");
if (line != NULL) {
char* escaped = output_escape(line, config->eight_bit_output);
@@ -545,17 +680,152 @@ static bool progress_precount_add_dir(ProgressPrecount* p, const char* path) {
return false;
}
static int progress_path_compare(const void* left, const void* right) {
const char* const* a = (const char* const*)left;
const char* const* b = (const char* const*)right;
return strcmp(*a, *b);
}
/* Sort the collected directory paths and drop duplicates so a large
* --files-from list (many entries sharing an implied ancestor) cannot grow the
* list without bound. */
static void progress_precount_dedup_dirs(ArrayList* dir_paths) {
if (dir_paths == NULL || dir_paths->size < 2)
return;
qsort(dir_paths->items, (size_t)dir_paths->size, sizeof(char*), progress_path_compare);
int write = 0;
for (int read = 0; read < dir_paths->size; read++) {
char* current = (char*)dir_paths->items[read];
if (write > 0 && strcmp((char*)dir_paths->items[write - 1], current) == 0) {
free(current);
continue;
}
dir_paths->items[write++] = current;
}
dir_paths->size = write;
}
/* Create a metadata-bearing directory File for the transfer-relative directory
* `rel` ("" is the transfer root), stat'ing it below config->send_directory.
* The -d/--files-from dirs generator never traverses directories, so this
* synthesizes the metadata the recursive scanner captures through
* scanner_capture_dir_time, letting -i/--out-format render %M/%B/%U/%G and the
* transfer-root/ancestor lines identically on both paths. Returns NULL when
* the path cannot be stat'd as a directory or on allocation failure (the line
* is then simply omitted, exactly as before). */
static File* progress_precount_make_dir(const Config* config, const char* rel) {
if (config == NULL || config->send_directory == NULL)
return NULL;
char* fs_path = (rel == NULL || rel[0] == '\0') ? str_dup(config->send_directory)
: path_cat(config->send_directory, rel);
if (fs_path == NULL)
return NULL;
struct stat st;
if (stat(fs_path, &st) != 0 || !S_ISDIR(st.st_mode)) {
free(fs_path);
return NULL;
}
File* file = file_create(fs_path);
if (file == NULL) {
free(fs_path);
return NULL;
}
file->is_dir = true;
file->metadata =
file_metadata_create(fs_path, &st, config->preserve_atimes, config->preserve_crtimes);
file->send_path = str_dup(rel != NULL ? rel : "");
free(fs_path);
if (file->metadata == NULL || file->send_path == NULL) {
file_destroy(file);
return NULL;
}
return file;
}
/* The -d/--files-from dirs generator neither traverses nor records directories,
* so its metadata walk captures no Files. Synthesize the transfer root and
* every listed/implied directory from `entry_rels` so -i/--out-format emits the
* same root and ancestor lines the recursive scan does. `root_emitted` is true
* when the generator itself emits the root entry (bare `-d <dir>`), whose
* data-pass line must not be duplicated. Returns false only on allocation
* failure. */
static bool progress_precount_synthesize_dirs(const Config* config, ProgressPrecount* out,
const ArrayList* entry_rels, bool root_emitted) {
for (int i = 0; i < entry_rels->size; i++) {
const char* rel = (const char*)entry_rels->items[i];
if (rel == NULL)
continue;
size_t len = strlen(rel);
for (size_t j = 1; j < len; j++) {
if (rel[j] != '/')
continue;
/* --no-implied-dirs: rsync neither creates nor itemizes an implied parent,
so only explicitly listed directories get a line. */
if (config->no_implied_dirs)
break;
char* prefix = malloc(j + 1);
if (prefix == NULL)
return false;
memcpy(prefix, rel, j);
prefix[j] = '\0';
if (!progress_precount_add_dir(out, prefix)) {
free(prefix);
return false;
}
free(prefix);
}
}
progress_precount_dedup_dirs(out->dir_paths);
for (int i = 0; i < out->dir_paths->size; i++) {
const char* rel = (const char*)out->dir_paths->items[i];
File* dir = progress_precount_make_dir(config, rel);
if (dir == NULL)
continue;
if (!array_list_add(out->dir_files, dir)) {
file_destroy(dir);
return false;
}
}
/* Emit the transfer root only when the generator actually emitted an entry:
--prune-empty-dirs (or an empty --files-from list) transfers nothing, and
rsync prints no root line then either. */
if (!root_emitted && entry_rels->size > 0) {
File* root = progress_precount_make_dir(config, "");
if (root != NULL && !array_list_add(out->dir_files, root)) {
file_destroy(root);
return false;
}
}
return true;
}
/* Metadata-only walk collecting the full file-list total and every directory
* name. It uses its own scanner (fresh filter compilation and hard-link table)
* so the data pass's link-group state is never perturbed. */
static bool progress_precount_scan(const Config* config, ProgressPrecount* out) {
out->dir_paths = array_list_create(free);
if (out->dir_paths == NULL)
out->dir_files = array_list_create(file_destroy);
out->dir_refs = array_list_create(dir_ref_destroy);
if (out->dir_paths == NULL || out->dir_files == NULL || out->dir_refs == NULL) {
progress_precount_dispose(out);
return false;
}
out->total = 0;
/* The -d/--files-from dirs generator never calls scanner_capture_dir_time, so
the walk below captures no directory Files. Record every emitted entry's
transfer-relative name so the implied ancestors can be synthesized once the
walk is done. */
bool synthesize = g_change_dirs_active && config->dirs;
ArrayList* entry_rels = synthesize ? array_list_create(free) : NULL;
if (synthesize && entry_rels == NULL) {
progress_precount_dispose(out);
return false;
}
bool root_emitted = false;
PreparedScanner prepared;
memset(&prepared, 0, sizeof(prepared));
if (!prepare_scanner(config, 0, &prepared)) {
array_list_delete(entry_rels);
progress_precount_dispose(out);
return false;
}
@@ -568,12 +838,14 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
local.preserve_xattrs = false;
local.preserve_acls = false;
local.checksum = false;
local.capture_dir_times = false;
/* Capture one metadata-bearing File per traversed directory (including the
transfer root) so -i/--out-format can render %M/%B/%U/%G for directories. */
local.capture_dir_times = true;
local.excluded_paths = NULL;
local.size_skipped_paths = NULL;
local.synced_dirs = NULL;
local.plan_dirs = NULL;
local.dir_entries = NULL;
local.dir_entries = out->dir_files;
local.dir_entries_mutex = NULL;
local.hardlinks = NULL;
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
@@ -584,8 +856,23 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
out->total += (unsigned long long)chunk->element_count;
for (int i = 0; i < chunk->element_count && ok; i++) {
const File* f = chunk->items[i];
if (f != NULL && f->is_dir)
ok = progress_precount_add_dir(out, delete_display_path(config, file_wire_path(f)));
if (f == NULL)
continue;
const char* rel = delete_display_path(config, file_wire_path(f));
if (f->is_dir) {
if (rel != NULL && rel[0] == '\0')
root_emitted = true;
ok = progress_precount_add_dir(out, rel);
if (!ok)
break;
}
if (synthesize && rel != NULL) {
char* dup = str_dup(rel);
if (dup == NULL || !array_list_add(entry_rels, dup)) {
free(dup);
ok = false;
}
}
}
chunk_destroy(chunk);
}
@@ -595,9 +882,30 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
}
prepared_scanner_destroy(&prepared);
if (!ok) {
array_list_delete(entry_rels);
progress_precount_dispose(out);
return false;
}
if (synthesize) {
bool synth_ok = progress_precount_synthesize_dirs(config, out, entry_rels, root_emitted);
array_list_delete(entry_rels);
if (!synth_ok) {
progress_precount_dispose(out);
return false;
}
}
/* Build the name -> File lookup from the captured directory Files. */
for (int i = 0; i < out->dir_files->size; i++) {
File* f = (File*)out->dir_files->items[i];
if (f == NULL)
continue;
if (!progress_precount_add_ref(out, config, f)) {
progress_precount_dispose(out);
return false;
}
}
if (out->dir_refs->size > 1)
qsort(out->dir_refs->items, (size_t)out->dir_refs->size, sizeof(DirRef*), dir_ref_compare);
out->total += 1; /* the transfer root "." */
return true;
}
@@ -609,9 +917,26 @@ static bool progress_precount_from_plan_dirs(const Config* config, const ArrayLi
unsigned long long non_dir_count,
ProgressPrecount* out) {
out->dir_paths = array_list_create(free);
if (out->dir_paths == NULL)
out->dir_files = array_list_create(file_destroy);
out->dir_refs = array_list_create(dir_ref_destroy);
if (out->dir_paths == NULL || out->dir_files == NULL || out->dir_refs == NULL) {
progress_precount_dispose(out);
return false;
}
out->total = non_dir_count + 1;
/* The delete pre-scan's plan list omits the transfer root, so synthesize its
entry here; it is only used for the root change line. */
File* root = file_create("");
if (root == NULL || !array_list_add(out->dir_files, root)) {
file_destroy(root);
progress_precount_dispose(out);
return false;
}
root->is_dir = true;
if (!progress_precount_add_ref(out, config, root)) {
progress_precount_dispose(out);
return false;
}
for (int i = 0; i < plan_dirs->size; i++) {
const char* path = (const char*)plan_dirs->items[i];
const char* rel = config->send_directory != NULL
@@ -621,7 +946,25 @@ static bool progress_precount_from_plan_dirs(const Config* config, const ArrayLi
progress_precount_dispose(out);
return false;
}
File* dir = file_create("");
if (dir == NULL) {
progress_precount_dispose(out);
return false;
}
dir->is_dir = true;
dir->send_path = str_dup(rel != NULL ? rel : "");
if (dir->send_path == NULL || !array_list_add(out->dir_files, dir)) {
file_destroy(dir);
progress_precount_dispose(out);
return false;
}
if (!progress_precount_add_ref(out, config, dir)) {
progress_precount_dispose(out);
return false;
}
}
if (out->dir_refs->size > 1)
qsort(out->dir_refs->items, (size_t)out->dir_refs->size, sizeof(DirRef*), dir_ref_compare);
out->total += (unsigned long long)out->dir_paths->size;
return true;
}
@@ -633,11 +976,17 @@ void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
unsigned long long plan_non_dir_count) {
client_progress_cleanup();
g_progress_active = progress_requested(config);
if (!g_progress_active)
g_change_dirs_active = config->itemize_changes || config->out_format != NULL;
if (!g_progress_active && !g_change_dirs_active)
return;
bool ok = plan_dirs != NULL ? progress_precount_from_plan_dirs(
config, plan_dirs, plan_non_dir_count, &g_progress_precount)
: progress_precount_scan(config, &g_progress_precount);
/* -i/--out-format render directory metadata (%M/%B/%U/%G) that only the
metadata walk captures; the --delete-during/--delete-delay plan list has no
metadata, so prefer the walk whenever a change line is rendered. Pure
--progress keeps reusing the plan list and its cheaper path-only pass. */
bool ok = (plan_dirs != NULL && !g_change_dirs_active)
? progress_precount_from_plan_dirs(config, plan_dirs, plan_non_dir_count,
&g_progress_precount)
: progress_precount_scan(config, &g_progress_precount);
if (!ok) {
g_progress_total = 0;
return;
+25 -9
View File
@@ -381,21 +381,28 @@ bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* s
paths, loading and sending nothing. --delete-before/--delete-during need the
complete keep-set manifest before the first data byte, so it is built by a
dedicated pre-scan pass and transmitted early; the data pass then re-scans
with a fresh scanner. A source I/O error is fatal unless the options carry
--ignore-errors, in which case the scan continues past the unreadable
directory and *io_error_out reports it (the caller still performs the
deletion but reports the run as errored). */
with a fresh scanner. --delete-before additionally replays this very scan as
its data pass (rsync's single file list), so `chunks_out` (optional) retains
the scanned Chunk objects for the caller to send instead of destroying them;
the caller owns the list and must give it a chunk_destroy destructor. A
source I/O error is fatal unless the options carry --ignore-errors, in which
case the scan continues past the unreadable directory and *io_error_out
reports it (the caller still performs the deletion but reports the run as
errored). */
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
DeletePlanSender* plans, bool* io_error_out,
unsigned long long* non_dir_count_out) {
unsigned long long* non_dir_count_out, ArrayList* chunks_out,
bool emit_nonreg) {
if (io_error_out)
*io_error_out = false;
if (non_dir_count_out)
*non_dir_count_out = 0;
ScannerOptions local = *options;
/* The pre-scan is a paths-only pass with no client output; it must not emit
--info=nonreg lines (the data pass does that once). */
local.note_nonreg = false;
/* The pre-scan is normally a paths-only pass with no client output: it must
not emit --info=nonreg lines because the data pass re-scans and emits them
once. When the caller replays this scan as the data pass (--delete-before)
there is no later scan, so it opts in and the lines are emitted here. */
local.note_nonreg = emit_nonreg && options->note_nonreg;
DirectoryScanner* scanner = directory_scanner_create_with_options(config->send_directory, &local);
if (!scanner)
return false;
@@ -430,7 +437,16 @@ bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayL
break;
}
}
chunk_destroy(chunk);
if (chunks_out) {
/* Retain the chunk for the caller's data pass; ownership moves with it. */
if (!array_list_add(chunks_out, chunk)) {
ok = false;
chunk_destroy(chunk);
break;
}
} else {
chunk_destroy(chunk);
}
}
if (ok) {
/* Keep every traversed source directory, including empty ones, so a plan
+143 -30
View File
@@ -654,7 +654,11 @@ static bool send_symlink_entry(const Client* client, File* file, const Config* c
if (!send_status(fd, STATUS_SYMLINK) || !send_wire_str(fd, file_wire_path(file)) ||
!send_wire_str(fd, file->symlink_target))
return false;
return !config->use_metadata || metadata_send(fd, file->metadata);
if (config->use_metadata && !metadata_send(fd, file->metadata))
return false;
/* Symlink xattrs/ACLs (-X/-A) ride the same trailing block as regular files
and directories when the xattr transport was negotiated. */
return !config->use_xattrs || xattr_send(fd, file->xattrs);
}
// Send a single file directly via sendfile (non-incremental path).
@@ -838,6 +842,10 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (chunk->items[i] == NULL)
continue;
transfer_stats_note_entry(stats, chunk->items[i]);
/* The chunk-serialization path emits no --progress name lines, so only
feed -i/--out-format its ancestor directory lines here. */
if (config->itemize_changes || config->out_format != NULL)
client_change_emit_ancestors(config, chunk->items[i]);
if (chunk->items[i]->is_dir)
change_emit_dir_sent(config, chunk->items[i]);
else
@@ -859,6 +867,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
source to remove and no incremental check. */
if (!send_directory_entry(client, f, config))
return -1;
client_change_emit_ancestors(config, f);
change_emit_dir_sent(config, f);
client_progress_name(config, f);
continue;
@@ -873,6 +882,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
!send_int(client->file_descriptor, f->link_group) ||
!send_wire_str(client->file_descriptor, f->hardlink_target))
return -1;
client_change_emit_ancestors(config, f);
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
@@ -881,6 +891,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (f->is_symlink) {
if (!send_symlink_entry(client, f, config))
return -1;
client_change_emit_ancestors(config, f);
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
@@ -890,6 +901,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
if (f->is_special) {
if (!file_send_special(f, client->file_descriptor, config->use_metadata))
return -1;
client_change_emit_ancestors(config, f);
change_emit_file_sent(config, f);
client_progress_name(config, f);
continue;
@@ -913,6 +925,7 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
return -1;
}
transfer_stats_note_transferred(stats, f);
client_change_emit_ancestors(config, f);
change_emit_file_sent_bytes(config, f, protocol_bytes_written() - bytes_before,
protocol_bytes_read() - read_before);
client_progress_file(config, f);
@@ -1139,6 +1152,43 @@ send_fail:
static int scan_directory_multithreaded(void* pipeline_context) {
PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
protocol_session_bind(&context->allocation_session);
if (context->prescan_chunks != NULL) {
/* --delete-before replays the pre-scan that built the early keep-set as the
data pass (rsync builds one file list). Feed the retained chunks straight
into the pipeline instead of re-reading the source, so a file created
after the pre-scan is neither transferred nor kept. The chunk also
carries the directory times captured by that scan (there is no later
scan), so no scanner is created here. */
bool failed = false;
for (int i = 0; i < context->prescan_chunks->size; i++) {
Chunk* chunk = (Chunk*)context->prescan_chunks->items[i];
/* Move ownership out of the retained list so a cleanup here never
double-frees a chunk the queue now owns. */
context->prescan_chunks->items[i] = NULL;
if (chunk == NULL)
continue;
if (!queue_enqueue_multithreaded_cancel(
context->queue_scanner, chunk, &context->mutex_scanner,
&context->condition_not_empty_scanner, &context->condition_not_full_scanner,
&context->cancelled)) {
chunk_destroy(chunk);
failed = true;
break;
}
}
mtx_lock(&context->mutex_scanner);
context->scanner_done = true;
cnd_broadcast(&context->condition_not_empty_scanner);
cnd_broadcast(&context->condition_not_full_scanner);
mtx_unlock(&context->mutex_scanner);
if (failed) {
pipeline_cancel(context);
protocol_session_unbind();
return thrd_error;
}
protocol_session_unbind();
return thrd_success;
}
PreparedScanner prepared;
/* -j/--threads=N sizes the parallel scanner's worker pool; 0 (bare -j) lets
* the scanner apply its built-in default. */
@@ -1385,6 +1435,10 @@ typedef struct {
Client* client;
DirectoryScanner* scanner;
ArrayList* manifest;
/* --delete-before: the pre-scan that built the keep-set, retained as the data
pass's file list (owning Chunk*; consumed chunks are NULLed as they are
sent). NULL in every other mode, where the data pass scans normally. */
ArrayList* prescan_chunks;
DeletePlanSender* plan_sender;
ArrayList* remove_sources;
ArrayList* dir_entries;
@@ -1469,12 +1523,23 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
if (state->delete_early) {
/* Pass 1: collect the complete keep-set (paths only, no data loaded) and
transmit it now, before any file data. The receiver removes extras and
acks; the transfer aborts here if the deletion could not commit. */
acks; the transfer aborts here if the deletion could not commit. The
scanned chunks are retained so the data pass can replay this exact list
instead of re-reading the source (rsync builds one file list and never
transfers a file created after it). */
ArrayList* early_manifest = array_list_create(free);
if (!early_manifest)
ArrayList* prescan_chunks = array_list_create(chunk_destroy);
if (!early_manifest || !prescan_chunks) {
array_list_delete(early_manifest);
array_list_delete(prescan_chunks);
return false;
}
/* No later scan runs for --delete-before, so this pass must also capture the
deferred directory times and the --stats directory count. */
state->prepared.options.dir_entries = state->dir_entries;
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
bool prescan_ok = scan_paths_only(config, &state->prepared.options, early_manifest, NULL,
&state->had_scan_io, NULL);
&state->had_scan_io, NULL, prescan_chunks, true);
bool early_ok = false;
bool skip_delete = false;
if (prescan_ok) {
@@ -1505,8 +1570,13 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
state->prepared.options.excluded_paths = NULL;
state->prepared.options.size_skipped_paths = NULL;
state->prepared.options.synced_dirs = NULL;
if (!prescan_ok || (!early_ok && !skip_delete))
if (!prescan_ok || (!early_ok && !skip_delete)) {
array_list_delete(prescan_chunks);
return false;
}
/* Adopt the captured scan as the data pass's file list (including when an
I/O error suppressed only the deletion: the list is still complete). */
state->prescan_chunks = prescan_chunks;
} else if (state->delete_per_dir) {
/* --delete-during/--delete-delay: build one plan per source directory from a
path-only pre-scan and transmit the COMPLETE plan set now, before any data,
@@ -1518,8 +1588,9 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
if (!state->plan_sender || !state->plan_dirs)
return false;
state->prepared.options.plan_dirs = state->plan_dirs;
bool prescan_ok = scan_paths_only(config, &state->prepared.options, NULL, state->plan_sender,
&state->had_scan_io, &state->per_dir_non_dir_count);
bool prescan_ok =
scan_paths_only(config, &state->prepared.options, NULL, state->plan_sender,
&state->had_scan_io, &state->per_dir_non_dir_count, NULL, false);
bool plans_ok = false;
bool skip_delete = false;
if (prescan_ok) {
@@ -1567,7 +1638,10 @@ static bool send_files_prepare_delete(Config* config, SendFilesState* state) {
* runs the shared cleanup). */
static bool send_files_run(Config* config, SendFilesState* state) {
Client* client = state->client;
if (progress_requested(config))
/* --progress needs the file-list total; -i/--out-format needs the directory
entries. Either way one paths-only pre-count supplies both, and a
--delete-during/--delete-delay pre-scan is reused when present. */
if (progress_requested(config) || config->itemize_changes || config->out_format != NULL)
client_progress_prepare(config, state->plan_dirs, state->per_dir_non_dir_count);
/* Phase 6: compute the client-only stop deadline once at transfer start. The
early-delete pre-scan above deliberately ignores it so the keep-set (and
@@ -1578,26 +1652,44 @@ static bool send_files_run(Config* config, SendFilesState* state) {
now_mono.tv_nsec = 0;
}
state->stop = stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins,
config->stop_at_set, config->stop_at, now_mono);
config->cli.stop_at_set, config->stop_at, now_mono);
state->prepared.options.stop_condition = &state->stop;
/* The early-delete pre-scan above already ran; only the data pass should feed
the directory-time list (otherwise every directory would be captured
twice). */
state->prepared.options.dir_entries = state->dir_entries;
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
state->scanner =
directory_scanner_create_with_options(config->send_directory, &state->prepared.options);
if (!state->scanner)
return false;
/* --delete-before reuses the pre-scan that built the keep-set as the data
pass's file list, so a source file created after that scan is neither
transferred nor kept (rsync builds one file list). That pre-scan captured
the deferred directory times and the --stats directory count because no
later scan runs; every other mode opens a fresh data scanner here. */
if (state->prescan_chunks == NULL) {
state->prepared.options.dir_entries = state->dir_entries;
state->prepared.options.dir_count = config->stats ? &state->dir_count : NULL;
state->scanner =
directory_scanner_create_with_options(config->send_directory, &state->prepared.options);
if (!state->scanner)
return false;
}
Chunk* current_chunk;
int prescan_index = 0;
memset(&state->transfer_stats, 0, sizeof(state->transfer_stats));
state->start = time(NULL);
client_progress_begin(config);
/* True when the stop deadline cut the scan short so the keep-set manifest is
only a prefix of the source. */
bool send_failed = false;
while ((current_chunk = directory_scanner_next(state->scanner)) != NULL) {
while (true) {
if (state->prescan_chunks != NULL) {
if (prescan_index >= state->prescan_chunks->size)
break;
/* Move ownership out of the retained list so chunk_destroy below (and the
cleanup tail for an early exit) never double-frees it. */
current_chunk = (Chunk*)state->prescan_chunks->items[prescan_index];
state->prescan_chunks->items[prescan_index] = NULL;
prescan_index++;
} else {
current_chunk = directory_scanner_next(state->scanner);
if (current_chunk == NULL)
break;
}
/* Graceful abort (Ctrl-C/SIGTERM): notify the receiver and clean up. The
session is active (config_send already succeeded); a send failure here is
fine because the client is exiting anyway. */
@@ -1653,12 +1745,16 @@ static bool send_files_run(Config* config, SendFilesState* state) {
/* Completion tail: send the late delete manifest and captured directory times,
* finalize the receiver handshake, remove transferred sources and report stats.
* Returns the rsync-compatible exit code. */
static int send_files_finalize(Config* config, SendFilesState* state) {
static int send_files_finalize(const Config* config, SendFilesState* state) {
Client* client = state->client;
if (directory_scanner_failed(state->scanner))
return 1;
if (directory_scanner_had_io_error(state->scanner))
state->had_scan_io = true;
/* A --delete-before run replays the pre-scan and owns no data scanner; its
I/O-error verdict was already recorded by that pre-scan. */
if (state->scanner != NULL) {
if (directory_scanner_failed(state->scanner))
return 1;
if (directory_scanner_had_io_error(state->scanner))
state->had_scan_io = true;
}
/* An abort that arrived after the last chunk must still stop the completion
tail (manifest/finalize) rather than let it run to success. */
if (client_abort_pending()) {
@@ -1762,6 +1858,8 @@ static int send_files_finalize(Config* config, SendFilesState* state) {
static void send_files_cleanup(SendFilesState* state) {
if (state->manifest)
array_list_delete(state->manifest);
if (state->prescan_chunks)
array_list_delete(state->prescan_chunks);
if (state->plan_sender)
delete_plan_sender_destroy(state->plan_sender);
if (state->excluded)
@@ -1916,8 +2014,8 @@ int send_files_multithreaded(Config* config) {
now_mono.tv_nsec = 0;
}
context->stop_condition =
stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins, config->stop_at_set,
config->stop_at, now_mono);
stop_condition_make(config->stop_after_mins > 0, config->stop_after_mins,
config->cli.stop_at_set, config->stop_at, now_mono);
bool collect_excluded = config->use_delete && !config->delete_excluded;
unsigned long long pre_scan_non_dir = 0;
if (config->use_delete) {
@@ -1975,13 +2073,27 @@ int send_files_multithreaded(Config* config) {
if (prepared_ok)
prepared.options.plan_dirs = context->plan_dirs;
} else {
/* --delete-before: retain the pre-scan chunks as the pipeline's data
pass (rsync's single file list) so a source file created after the
scan is not transferred. No later scan runs, so this pass must also
capture the deferred directory times and the --stats directory
count. */
context->manifest = array_list_create(free);
prepared_ok = prepared_ok && context->manifest != NULL;
context->prescan_chunks = array_list_create(chunk_destroy);
prepared_ok = prepared_ok && context->manifest != NULL && context->prescan_chunks != NULL;
if (prepared_ok) {
prepared.options.dir_entries = context->dir_entries;
prepared.options.dir_entries_mutex = &context->dir_entries_mutex;
prepared.options.dir_count = config->stats ? &context->dir_count : NULL;
if (!append_implied_dir_times(config, context->dir_entries))
prepared_ok = false;
}
}
bool prebuilt =
prepared_ok &&
scan_paths_only(config, &prepared.options, context->manifest, context->delete_plans,
&context->scan_had_io_error, &pre_scan_non_dir);
&context->scan_had_io_error, &pre_scan_non_dir, context->prescan_chunks,
context->prescan_chunks != NULL);
prepared_scanner_destroy(&prepared);
if (per_dir && prebuilt) {
const char* walk_root = delete_plan_walk_root(config, context->synced_dirs);
@@ -2045,9 +2157,10 @@ int send_files_multithreaded(Config* config) {
return 1;
}
/* --progress/--info=progress: pre-count the file list for rsync's to-chk
denominator, reusing a --delete-during/--delete-delay pre-scan when one
denominator; -i/--out-format: pre-count the directory entries. One pass
supplies both, reusing a --delete-during/--delete-delay pre-scan when one
already ran. */
if (progress_requested(config))
if (progress_requested(config) || config->itemize_changes || config->out_format != NULL)
client_progress_prepare(config, context->plan_dirs, pre_scan_non_dir);
thrd_t scanner, loader, sender;
+5 -1
View File
@@ -44,7 +44,8 @@ const char* delete_plan_walk_root(const Config* config, const ArrayList* synced_
bool files_from_list_check(const Config* config, ArrayList* missing_dest, int* skipped_out);
bool scan_paths_only(const Config* config, const ScannerOptions* options, ArrayList* manifest,
DeletePlanSender* plans, bool* io_error_out,
unsigned long long* non_dir_count_out);
unsigned long long* non_dir_count_out, ArrayList* chunks_out,
bool emit_nonreg);
/* client_report.c */
void log_server_rejection(const char* context);
@@ -64,6 +65,9 @@ void client_progress_cleanup(void);
void client_progress_begin(const Config* config);
void client_progress_file(const Config* config, const File* file);
void client_progress_name(const Config* config, const File* file);
/* Emit a transferred entry's ancestor directories (as -i/--out-format change
* lines or --progress name lines) before the entry's own line. */
void client_change_emit_ancestors(const Config* config, const File* file);
void client_progress_uptodate(const Config* config, const File* file);
void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
unsigned long long plan_non_dir_count);
+6 -2
View File
@@ -282,11 +282,15 @@ bool entry_passes_selection(const FileListSet* file_list, const FilterRuleList*
}
/* Best-effort capture of the file's whitelisted xattrs (-X/-A). A failure to
* read xattrs is non-fatal: the file is transferred without them. */
* read xattrs is non-fatal: the file is transferred without them. A symlink
* entry reads the LINK's own xattrs (never the referent's) with the no-follow
* variant; on Linux the VFS refuses xattrs on symlinks, so that yields NULL. */
void scanner_capture_xattrs(const DirectoryScanner* scanner, File* file) {
if (!scanner || !file || !(scanner->options.preserve_xattrs || scanner->options.preserve_acls))
return;
file->xattrs = xattr_capture_path(file->path, scanner->options.preserve_acls);
file->xattrs = file->is_symlink
? xattr_capture_path_nofollow(file->path, scanner->options.preserve_acls)
: xattr_capture_path(file->path, scanner->options.preserve_acls);
}
/* Apply --hard-links (-H) detection to one regular File. On a sibling (a
+3 -1
View File
@@ -428,7 +428,9 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
}
if ((options->preserve_xattrs || options->preserve_acls) &&
!(file->link_group != 0 && !file->link_first))
file->xattrs = xattr_capture_path(file->path, options->preserve_acls);
file->xattrs = file->is_symlink
? xattr_capture_path_nofollow(file->path, options->preserve_acls)
: xattr_capture_path(file->path, options->preserve_acls);
if (!array_list_add(root_files, file)) {
free(rel);
file_destroy(file);
+8 -4
View File
@@ -26,6 +26,9 @@ void print_usage(void) {
printf(" owner, group, devices and specials; not\n");
printf(" compression/multithreading\n");
printf(" -r, --recursive Recurse into directories (FastSync is always recursive)\n");
printf(" --inc-recursive Accepted for rsync CLI compatibility; no effect (FastSync\n");
printf(" always performs a full scan, so the destination is identical)\n");
printf(" --no-inc-recursive Accepted for rsync CLI compatibility; no effect\n");
printf(" -n, --dry-run Show what would be transferred\n");
printf(" --remove-source-files Remove regular source files after successful transfer\n");
printf(" -p, --perms Preserve permission bits\n");
@@ -205,10 +208,11 @@ void print_usage(void) {
printf(" -A, --acls Preserve POSIX ACLs (the system.posix_acl_* xattrs;\n");
printf(" setting an ACL the receiver is not permitted to\n");
printf(" set is warned and skipped, never fatal)\n");
printf(" --fake-super Store the source uid/gid/mode/mtime in a reserved\n");
printf(" user.fastsync.stat xattr on each written file and\n");
printf(" re-apply it (fd-relative) on a privileged run; the\n");
printf(" recording format diverges from rsync's user.rsync.%%stat%%\n");
printf(" --fake-super Store the source mode/rdev/uid/gid in rsync's\n");
printf(" reserved user.rsync.%%stat xattr on each written\n");
printf(" file (interoperable with rsync); it never performs a\n");
printf(" real chown, so an unprivileged receiver records the\n");
printf(" privileged stat for a later restore\n");
printf(" --super Permit the receiver to attempt super-user activities\n");
printf(" (char/block device-node creation, --write-devices)\n");
printf(" within the confined receive root. Never elevates\n");
+336 -195
View File
@@ -53,7 +53,13 @@ bool receiver_send_final_success(int fd, const Config* config, const ReceiverOut
return send_status(fd, final_status);
size_t count = outcomes ? outcomes->count : 0;
for (size_t i = 0; i < count; i++) {
Status per_file = outcomes->entries[i] == FILE_SAVE_WRITTEN ? STATUS_NEXT : STATUS_OK;
Status per_file;
if (outcomes->entries[i] == FILE_SAVE_WRITTEN)
per_file = STATUS_NEXT;
else if (outcomes->entries[i] == FILE_SAVE_FAILED)
per_file = STATUS_ERROR;
else
per_file = STATUS_OK;
if (!send_status(fd, per_file))
return false;
}
@@ -303,6 +309,264 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
return receiver_process_pending(config, file_descriptor, sink, NULL, NULL);
}
/* Per-connection state threaded through the status handlers below. The parked
keep-set / per-directory session live here so one teardown helper can release
them on every exit path. */
typedef struct {
Config* config;
int fd;
const ReceiverSink* sink;
DeleteManifest** pending_manifest;
DeletePlanSession** pending_plans;
/* Parked keep-set for the late/commit timing. Every exit path frees it
exactly once; the only exception is the successful FINISHED handoff, which
transfers ownership to *pending_manifest (used by the -m receiver). */
DeleteManifest* deferred_manifest;
/* Per-directory delete session for --delete-during/--delete-delay. During the
loop it applies plans inline (during) or snapshots their extras (delay); on
a successful FINISHED it is either committed here or handed to
*pending_plans so the -m caller commits after its disk writer drained. */
DeletePlanSession* plan_session;
bool early_delete;
bool per_dir_delete;
bool delete_limit_noted;
} ReceiverPendingState;
/* Outcome of one frame handler. NEXT reads the following status frame; FAIL
tears the connection down without a peer STATUS_ERROR; ERROR tears it down
and (when the sink owns error reporting) emits STATUS_ERROR. */
typedef enum {
RECEIVER_STEP_NEXT,
RECEIVER_STEP_FAIL,
RECEIVER_STEP_ERROR,
} ReceiverStep;
static ReceiverStep receiver_handle_keepalive(ReceiverPendingState* state) {
if (!send_status(state->fd, STATUS_KEEPALIVE))
return RECEIVER_STEP_FAIL;
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_abort(ReceiverPendingState* state) {
(void)state;
log_message(LOG_LEVEL_INFO, "Received abort from client, cleaning up");
return RECEIVER_STEP_FAIL;
}
static ReceiverStep receiver_handle_check(ReceiverPendingState* state) {
bool skipped = false;
bool would_transfer = false;
File* file = receive_incremental_check_ex(state->fd, state->config, &skipped, &would_transfer);
if (state->config->dry_run) {
/* Server-contacting --dry-run: the reply has already been sent
(STATUS_OK = up to date, STATUS_DRY_RUN_TRANSFER = would transfer) and
nothing may be stored. Both flags false means a genuine protocol
error (STATUS_ERROR already sent or sent by receive_error below). */
if (!skipped && !would_transfer)
return RECEIVER_STEP_ERROR;
} else if (!skipped && (!file || !state->sink->store_file(file, state->sink->context))) {
return RECEIVER_STEP_ERROR;
}
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_chunk(ReceiverPendingState* state) {
Chunk* chunk = receive_chunk_data(state->fd, state->config);
if (!chunk || !receiver_process_chunk(chunk, state->sink))
return RECEIVER_STEP_ERROR;
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_check_batch(ReceiverPendingState* state) {
if (!receiver_process_batch(state->config, state->fd))
return RECEIVER_STEP_FAIL;
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_mkdir(ReceiverPendingState* state) {
File* dir = file_receive_directory(state->fd, state->config);
if (!dir || !state->sink->store_file(dir, state->sink->context))
return RECEIVER_STEP_ERROR;
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_dir_times(const ReceiverPendingState* state) {
if (!receiver_process_dir_times(state->fd, state->config, state->sink))
return RECEIVER_STEP_ERROR;
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_hardlink(ReceiverPendingState* state) {
File* file = file_receive_hardlink(state->fd);
if (!file || !state->sink->store_file(file, state->sink->context))
return RECEIVER_STEP_ERROR;
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_symlink(ReceiverPendingState* state) {
File* sym = file_receive_symlink(state->fd, state->config);
if (!sym || !state->sink->store_file(sym, state->sink->context))
return RECEIVER_STEP_ERROR;
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_special(ReceiverPendingState* state) {
File* file = file_receive_special(state->fd);
if (!file || !state->sink->store_file(file, state->sink->context))
return RECEIVER_STEP_ERROR;
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_manifest(ReceiverPendingState* state) {
Config* config = state->config;
int fd = state->fd;
const ReceiverSink* sink = state->sink;
DeleteManifest* manifest = receive_manifest_entries(fd);
if (!manifest)
return RECEIVER_STEP_FAIL; /* receive_manifest_entries already sent STATUS_ERROR */
if (config->dry_run) {
/* Server-contacting --dry-run mutates nothing, so a keep-set manifest
is consumed and discarded. The early-delete mode still needs its ACK
so a sender blocked on the delete handshake is not left hanging.
When would-delete reporting is armed, enumerate (read-only) the
destination extras so the terminal STATUS_STATS frame can list them. */
if (config->use_delete && sink->would_delete) {
size_t count = 0;
if (!manifest_would_delete_list(config, manifest, sink->would_delete, &count))
log_message(LOG_LEVEL_WARNING, "dry-run: could not enumerate would-delete paths");
}
delete_manifest_free(manifest);
if (state->early_delete && !send_status(fd, STATUS_OK))
return RECEIVER_STEP_FAIL;
return RECEIVER_STEP_NEXT;
}
if (state->early_delete) {
/* --delete-before: the whole-tree manifest is authoritative the moment
it arrives, before any file data. Delete now and acknowledge so the
sender only starts streaming once the deletion committed (or failed).
A later transfer failure does not restore these deletions. A
--max-delete-capped commit still succeeds and the transfer proceeds;
the terminal success frame reports the cap. */
size_t deleted = 0;
DeletePathObserver observer =
(config->report_deletes && sink->deleted_paths) ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion =
(config->use_delete || config->delete_missing_args)
? manifest_delete_all_observed(config, manifest, &deleted, observer,
(void*)sink->deleted_paths)
: DELETE_COMMIT_OK;
receiver_tally_deleted(sink, deleted);
delete_manifest_free(manifest);
if (deletion == DELETE_COMMIT_ERROR) {
send_status(fd, STATUS_ERROR);
return RECEIVER_STEP_FAIL;
}
if (deletion == DELETE_COMMIT_LIMIT_REACHED && sink->note_delete_limit)
sink->note_delete_limit(sink->context);
if (!send_status(fd, STATUS_OK))
return RECEIVER_STEP_FAIL;
} else if (config->use_delete || config->delete_missing_args) {
/* Plain --delete / --delete-after and the --delete-missing-args
exact-path deletions: hold the manifest and commit it only after
STATUS_FINISHED. The per-directory modes never send this frame. */
if (state->deferred_manifest) {
log_message(LOG_LEVEL_ERROR, "Received a second delete manifest");
delete_manifest_free(state->deferred_manifest);
state->deferred_manifest = NULL;
delete_manifest_free(manifest);
send_status(fd, STATUS_ERROR);
return RECEIVER_STEP_FAIL;
}
state->deferred_manifest = manifest;
} else {
delete_manifest_free(manifest);
}
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_delete_plan(ReceiverPendingState* state) {
Config* config = state->config;
int fd = state->fd;
const ReceiverSink* sink = state->sink;
if (!state->per_dir_delete) {
log_message(LOG_LEVEL_ERROR, "Received a per-directory delete plan without a per-dir "
"delete timing");
send_status(fd, STATUS_ERROR);
return RECEIVER_STEP_FAIL;
}
if (!state->plan_session) {
state->plan_session = delete_plan_session_create(config);
if (state->plan_session && config->report_deletes && sink->deleted_paths)
delete_plan_session_set_delete_observer(state->plan_session, receiver_record_deleted_path,
(void*)sink->deleted_paths);
}
if (!state->plan_session || delete_plan_session_receive(state->plan_session, config, fd) != 0)
return RECEIVER_STEP_FAIL;
if (delete_plan_session_limit_reached(state->plan_session) && !state->delete_limit_noted &&
sink->note_delete_limit) {
sink->note_delete_limit(sink->context);
state->delete_limit_noted = true;
}
return RECEIVER_STEP_NEXT;
}
static ReceiverStep receiver_handle_file(ReceiverPendingState* state) {
File* file = file_receive(state->config, state->fd);
if (!file) {
log_message(LOG_LEVEL_ERROR, "Failed to receive file");
return RECEIVER_STEP_ERROR;
}
if (!state->sink->store_file(file, state->sink->context))
return RECEIVER_STEP_ERROR;
return RECEIVER_STEP_NEXT;
}
/* One dispatch per admitted frame type; STATUS_NEXT (and any other
data-bearing status) falls through to the regular file receiver. */
static ReceiverStep receiver_dispatch_status(ReceiverPendingState* state, Status status) {
switch (status) {
case STATUS_KEEPALIVE:
return receiver_handle_keepalive(state);
case STATUS_ABORT:
return receiver_handle_abort(state);
case STATUS_CHECK:
return receiver_handle_check(state);
case STATUS_CHUNK:
return receiver_handle_chunk(state);
case STATUS_CHECK_BATCH:
return receiver_handle_check_batch(state);
case STATUS_MKDIR:
return receiver_handle_mkdir(state);
case STATUS_DIR_TIMES:
return receiver_handle_dir_times(state);
case STATUS_HARDLINK:
return receiver_handle_hardlink(state);
case STATUS_SYMLINK:
return receiver_handle_symlink(state);
case STATUS_SPECIAL:
return receiver_handle_special(state);
case STATUS_MANIFEST:
return receiver_handle_manifest(state);
case STATUS_DELETE_PLAN:
return receiver_handle_delete_plan(state);
default:
return receiver_handle_file(state);
}
}
/* Release the parked keep-set / per-directory session exactly once on every
failure exit. Never commit a deletion for a failed stream. */
static void receiver_drop_pending(ReceiverPendingState* state) {
if (state->deferred_manifest) {
delete_manifest_free(state->deferred_manifest);
state->deferred_manifest = NULL;
}
if (state->plan_session) {
delete_plan_session_destroy(state->plan_session);
state->plan_session = NULL;
}
}
/* Runs the whole receive loop. The delete manifest may legitimately arrive
either FIRST (--delete-before / --delete-during: the sender transmits the
validated keep-set before any file data) or LAST (--delete-after /
@@ -312,9 +576,9 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
deletion has committed (or failed); in the late modes the manifest is held
and the deletion is committed only after the terminal STATUS_FINISHED proves
the whole transfer succeeded. A plain --delete defaults to the per-directory
delete-during plan mode (no manifest at all). See
receiver_process_pending() for how the -m receiver defers that commit until
its disk writer has drained. */
delete-during plan mode (no manifest at all). See the per-frame handlers
above for how the -m receiver defers that commit until its disk writer has
drained. */
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
Status status;
@@ -329,166 +593,29 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
last_progress = session_start;
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
return -1;
bool early_delete = config_delete_timing_early(config);
bool per_dir_delete = config_delete_timing_per_dir(config);
/* Parked keep-set for the late/commit timing. Every exit path below frees it
exactly once; the only exception is the successful FINISHED handoff, which
transfers ownership to *pending_manifest (used by the -m receiver). */
DeleteManifest* deferred_manifest = NULL;
/* Per-directory delete session for --delete-during/--delete-delay. During the
loop it applies plans inline (during) or snapshots their extras (delay); on
a successful FINISHED it is either committed here or handed to
*pending_plans so the -m caller commits after its disk writer drained. */
DeletePlanSession* plan_session = NULL;
bool delete_limit_noted = false;
ReceiverPendingState state = {
.config = config,
.fd = file_descriptor,
.sink = sink,
.pending_manifest = pending_manifest,
.pending_plans = pending_plans,
.deferred_manifest = NULL,
.plan_session = NULL,
.early_delete = config_delete_timing_early(config),
.per_dir_delete = config_delete_timing_per_dir(config),
.delete_limit_noted = false,
};
bool notify_peer = false;
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK ||
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH ||
status == STATUS_MKDIR || status == STATUS_MANIFEST || status == STATUS_HARDLINK ||
status == STATUS_SYMLINK || status == STATUS_SPECIAL || status == STATUS_DIR_TIMES ||
status == STATUS_DELETE_PLAN) {
if (status == STATUS_KEEPALIVE) {
if (!send_status(file_descriptor, STATUS_KEEPALIVE))
goto fail;
goto next_status;
}
if (status == STATUS_ABORT) {
log_message(LOG_LEVEL_INFO, "Received abort from client, cleaning up");
ReceiverStep step = receiver_dispatch_status(&state, status);
if (step == RECEIVER_STEP_FAIL)
goto fail;
}
if (status == STATUS_CHECK) {
bool skipped = false;
bool would_transfer = false;
File* file = receive_incremental_check_ex(file_descriptor, config, &skipped, &would_transfer);
if (config->dry_run) {
/* Server-contacting --dry-run: the reply has already been sent
(STATUS_OK = up to date, STATUS_DRY_RUN_TRANSFER = would transfer) and
nothing may be stored. Both flags false means a genuine protocol
error (STATUS_ERROR already sent or sent by receive_error below). */
if (!skipped && !would_transfer)
goto receive_error;
} else if (!skipped && (!file || !sink->store_file(file, sink->context))) {
goto receive_error;
}
} else if (status == STATUS_CHUNK) {
Chunk* chunk = receive_chunk_data(file_descriptor, config);
if (!chunk || !receiver_process_chunk(chunk, sink))
goto receive_error;
} else if (status == STATUS_CHECK_BATCH) {
if (!receiver_process_batch(config, file_descriptor))
goto fail;
goto next_status;
} else if (status == STATUS_MKDIR) {
File* dir = file_receive_directory(file_descriptor, config);
if (!dir || !sink->store_file(dir, sink->context))
goto receive_error;
} else if (status == STATUS_DIR_TIMES) {
if (!receiver_process_dir_times(file_descriptor, config, sink))
goto receive_error;
} else if (status == STATUS_HARDLINK) {
File* file = file_receive_hardlink(file_descriptor);
if (!file || !sink->store_file(file, sink->context))
goto receive_error;
} else if (status == STATUS_SYMLINK) {
File* sym = file_receive_symlink(file_descriptor, config);
if (!sym || !sink->store_file(sym, sink->context))
goto receive_error;
} else if (status == STATUS_SPECIAL) {
File* file = file_receive_special(file_descriptor);
if (!file || !sink->store_file(file, sink->context))
goto receive_error;
} else if (status == STATUS_MANIFEST) {
DeleteManifest* manifest = receive_manifest_entries(file_descriptor);
if (!manifest)
goto fail; /* receive_manifest_entries already sent STATUS_ERROR */
if (config->dry_run) {
/* Server-contacting --dry-run mutates nothing, so a keep-set manifest
is consumed and discarded. The early-delete mode still needs its ACK
so a sender blocked on the delete handshake is not left hanging.
When would-delete reporting is armed, enumerate (read-only) the
destination extras so the terminal STATUS_STATS frame can list them. */
if (config->use_delete && sink->would_delete) {
size_t count = 0;
if (!manifest_would_delete_list(config, manifest, sink->would_delete, &count))
log_message(LOG_LEVEL_WARNING, "dry-run: could not enumerate would-delete paths");
}
delete_manifest_free(manifest);
if (early_delete && !send_status(file_descriptor, STATUS_OK))
goto fail;
goto next_status;
}
if (early_delete) {
/* --delete-before: the whole-tree manifest is authoritative the moment
it arrives, before any file data. Delete now and acknowledge so the
sender only starts streaming once the deletion committed (or failed).
A later transfer failure does not restore these deletions. A
--max-delete-capped commit still succeeds and the transfer proceeds;
the terminal success frame reports the cap. */
size_t deleted = 0;
DeletePathObserver observer =
(config->report_deletes && sink->deleted_paths) ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion =
(config->use_delete || config->delete_missing_args)
? manifest_delete_all_observed(config, manifest, &deleted, observer,
(void*)sink->deleted_paths)
: DELETE_COMMIT_OK;
receiver_tally_deleted(sink, deleted);
delete_manifest_free(manifest);
if (deletion == DELETE_COMMIT_ERROR) {
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
if (deletion == DELETE_COMMIT_LIMIT_REACHED && sink->note_delete_limit)
sink->note_delete_limit(sink->context);
if (!send_status(file_descriptor, STATUS_OK))
goto fail;
} else if (config->use_delete || config->delete_missing_args) {
/* Plain --delete / --delete-after and the --delete-missing-args
exact-path deletions: hold the manifest and commit it only after
STATUS_FINISHED. The per-directory modes never send this frame. */
if (deferred_manifest) {
log_message(LOG_LEVEL_ERROR, "Received a second delete manifest");
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
delete_manifest_free(manifest);
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
deferred_manifest = manifest;
} else {
delete_manifest_free(manifest);
}
goto next_status;
} else if (status == STATUS_DELETE_PLAN) {
if (!per_dir_delete) {
log_message(LOG_LEVEL_ERROR, "Received a per-directory delete plan without a per-dir "
"delete timing");
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
if (!plan_session) {
plan_session = delete_plan_session_create(config);
if (plan_session && config->report_deletes && sink->deleted_paths)
delete_plan_session_set_delete_observer(plan_session, receiver_record_deleted_path,
(void*)sink->deleted_paths);
}
if (!plan_session || delete_plan_session_receive(plan_session, config, file_descriptor) != 0)
goto fail;
if (delete_plan_session_limit_reached(plan_session) && !delete_limit_noted &&
sink->note_delete_limit) {
sink->note_delete_limit(sink->context);
delete_limit_noted = true;
}
goto next_status;
} else {
File* file = file_receive(config, file_descriptor);
if (!file) {
log_message(LOG_LEVEL_ERROR, "Failed to receive file");
goto receive_error;
}
if (!sink->store_file(file, sink->context))
goto receive_error;
}
next_status:
if (step == RECEIVER_STEP_ERROR)
goto receive_error;
if (!receive_status(file_descriptor, &status))
goto receive_error;
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
@@ -507,19 +634,19 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
disk writer may still be draining; the caller commits after the writer has
joined so no extra file is removed unless the transfer is known to have
succeeded. */
if (deferred_manifest) {
if (pending_manifest) {
*pending_manifest = deferred_manifest;
deferred_manifest = NULL;
if (state.deferred_manifest) {
if (state.pending_manifest) {
*state.pending_manifest = state.deferred_manifest;
state.deferred_manifest = NULL;
} else {
size_t deleted = 0;
DeletePathObserver observer =
(config->report_deletes && sink->deleted_paths) ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion = manifest_delete_all_observed(
config, deferred_manifest, &deleted, observer, (void*)sink->deleted_paths);
config, state.deferred_manifest, &deleted, observer, (void*)sink->deleted_paths);
receiver_tally_deleted(sink, deleted);
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
delete_manifest_free(state.deferred_manifest);
state.deferred_manifest = NULL;
if (deletion == DELETE_COMMIT_ERROR) {
send_status(file_descriptor, STATUS_ERROR);
goto fail;
@@ -533,28 +660,28 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
nothing yet and applies its decompressed snapshot here. The -m receiver
hands the session to its caller instead, which commits after the disk
writer drained. */
if (plan_session) {
if (state.plan_session) {
if (config->report_deletes && sink->deleted_paths)
delete_plan_session_set_delete_observer(plan_session, receiver_record_deleted_path,
delete_plan_session_set_delete_observer(state.plan_session, receiver_record_deleted_path,
(void*)sink->deleted_paths);
if (pending_plans) {
*pending_plans = plan_session;
plan_session = NULL;
if (state.pending_plans) {
*state.pending_plans = state.plan_session;
state.plan_session = NULL;
} else if (config->dry_run) {
/* Central dry-run no-op: never commit a deletion for a -n run. */
delete_plan_session_destroy(plan_session);
plan_session = NULL;
delete_plan_session_destroy(state.plan_session);
state.plan_session = NULL;
} else {
DeleteCommitResult deletion = delete_plan_session_commit(plan_session, config);
bool limit = delete_plan_session_limit_reached(plan_session);
receiver_tally_deleted(sink, delete_plan_session_deleted(plan_session));
delete_plan_session_destroy(plan_session);
plan_session = NULL;
DeleteCommitResult deletion = delete_plan_session_commit(state.plan_session, config);
bool limit = delete_plan_session_limit_reached(state.plan_session);
receiver_tally_deleted(sink, delete_plan_session_deleted(state.plan_session));
delete_plan_session_destroy(state.plan_session);
state.plan_session = NULL;
if (deletion == DELETE_COMMIT_ERROR) {
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
if (limit && !delete_limit_noted && sink->note_delete_limit)
if (limit && !state.delete_limit_noted && sink->note_delete_limit)
sink->note_delete_limit(sink->context);
}
}
@@ -568,26 +695,14 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
}
return 0;
receive_error:
notify_peer = true;
fail:
/* Failure exits that must not (or already did) report a STATUS_ERROR. The
parked keep-set/session is dropped: never commit a deletion for a failed
stream. */
if (deferred_manifest) {
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
}
if (plan_session)
delete_plan_session_destroy(plan_session);
return -1;
receive_error:
if (deferred_manifest) {
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
}
if (plan_session)
delete_plan_session_destroy(plan_session);
if (sink->send_error)
receiver_drop_pending(&state);
if (notify_peer && sink->send_error)
send_status(file_descriptor, STATUS_ERROR);
return -1;
}
@@ -610,6 +725,11 @@ typedef struct {
ArrayList* would_delete;
/* --info=del: actually-removed paths collected during the delete commit. */
ArrayList* deleted_paths;
/* Per-run count of entries that failed to materialize without aborting the
stream (currently ONLY a --devices mknod EPERM/EACCES). A nonzero count
makes the terminal frame carry a non-OK status so the client exits
non-zero, matching rsync's continue-and-exit-partial behavior. */
size_t failed_entries;
} ReceiverSaveContext;
static bool receiver_save_file(File* file, void* context_pointer) {
@@ -633,6 +753,11 @@ static bool receiver_save_file(File* file, void* context_pointer) {
count as matched data in the end-of-transfer report. */
if (result != FILE_SAVE_ERROR && file->matched_bytes > 0)
context->stats.matched_data += file->matched_bytes;
/* --devices parity: a device node the receiver could not mknod (EPERM/EACCES)
is counted per-run but does not abort the transfer. The terminal frame
turns a nonzero count into a non-OK status so the client exits non-zero. */
if (result == FILE_SAVE_FAILED)
context->failed_entries++;
/* Protocol 2.28.0: receiver-observed literal bytes and the created-entry
breakdown (regular/dir/link/special) for the `--stats` report. */
if (result == FILE_SAVE_WRITTEN)
@@ -666,9 +791,25 @@ static void receiver_note_delete_limit(void* context_pointer) {
context->delete_limit_reached = true;
}
/* Terminal status for a run. A capped --delete limit wins (rsync exit 25);
otherwise any per-entry failure (for example an unprivileged --devices
mknod) makes the terminal frame non-OK so the client exits non-zero. rsync
reports 23 here; mapping the client's exact exit code to 23 is a separate,
pre-existing concern. A clean run keeps STATUS_OK. */
static Status receiver_final_status(bool delete_limit_reached, size_t failed_entries) {
if (delete_limit_reached)
return STATUS_DELETE_LIMIT;
return failed_entries > 0 ? STATUS_ERROR : STATUS_OK;
}
static bool receiver_send_success_frame(int fd, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
Status final_status = context->delete_limit_reached ? STATUS_DELETE_LIMIT : STATUS_OK;
if (context->failed_entries > 0)
log_message(LOG_LEVEL_WARNING,
"%zu entr%s failed to materialize; continuing (partial transfer)",
context->failed_entries, context->failed_entries == 1 ? "y" : "ies");
Status final_status =
receiver_final_status(context->delete_limit_reached, context->failed_entries);
if (!receiver_send_stats_frame(fd, context->config, &context->stats, context->would_delete,
context->deleted_paths))
return false;
+8
View File
@@ -29,6 +29,7 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
context->deferred_manifest = NULL;
context->deferred_plans = NULL;
context->delete_limit_reached = false;
context->failed_entries = 0;
memset(&context->stats, 0, sizeof(context->stats));
context->would_delete = NULL;
context->deleted_paths = NULL;
@@ -264,6 +265,13 @@ int write_thread(void* pipeline_context) {
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
mtx_unlock(&context->mutex);
}
/* --devices parity: a device node that could not be mknod'ed is counted
per-run but does NOT abort the transfer. */
if (result == FILE_SAVE_FAILED) {
mtx_lock(&context->mutex);
context->failed_entries++;
mtx_unlock(&context->mutex);
}
if (result == FILE_SAVE_ERROR) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
+5
View File
@@ -64,6 +64,11 @@ typedef struct PipelineContextReceiver {
/* --info=del actually-removed path list, collected by the deferred delete
commit in server.c and reported in the STATUS_STATS frame. */
struct ArrayList* deleted_paths;
/* Per-run count of entries that failed to materialize without aborting the
stream (currently ONLY a --devices mknod EPERM/EACCES). write_thread
increments it under `mutex`; server.c turns a nonzero count into a non-OK
terminal status so the client exits non-zero. */
size_t failed_entries;
} PipelineContextReceiver;
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
+275 -186
View File
@@ -705,69 +705,85 @@ static const char* server_module_gate(const Config* config, void* context) {
return module_gate_install_root(config, module);
}
void handler(int file_descriptor) {
SSL* ssl = io_get_ssl();
/* Per-connection state threaded through the handler phase helpers below. The
* fields are a faithful split of the former handler() locals: the protocol
* session, the config-frame gate context, the accepted config, the optional
* multithreaded pipeline context and the teardown bookkeeping all live here so
* the single `done` epilogue in handler() can release them exactly as before. */
typedef struct ServerSession {
int fd;
SSL* ssl;
ProtocolSession session;
protocol_session_init(&session, file_descriptor, file_descriptor);
protocol_session_set_ssl(&session, ssl);
protocol_session_bind(&session);
ModuleGateContext gate_ctx;
gate_ctx.ssl = ssl;
gate_ctx.fd = file_descriptor;
gate_ctx.super_mode_override = -1;
gate_ctx.has_peer_ip = false;
gate_ctx.peer_ip[0] = '\0';
gate_ctx.is_local = false;
/* All teardown state starts empty so the single `done` epilogue is safe to
* reach from any error path (including before the config frame arrives). */
Config* config = NULL;
PipelineContextReceiver* context = NULL;
char* joined_destination = NULL;
bool charset_ready = false;
config = config_receive_with_validate(file_descriptor, server_module_gate, &gate_ctx);
if (config == NULL) {
Config* config;
PipelineContextReceiver* context;
char* joined_destination;
bool charset_ready;
} ServerSession;
/* Phase 1 -- config receipt + validation. Receives the client config frame
* through the module gate, applies the super-mode override the gate recorded
* exactly once, and installs the per-connection protocol/compression state.
* Returns false when the config frame was refused (the gate has already
* answered the client); the caller jumps to the shared `done` epilogue. */
static bool server_accept_config(ServerSession* state) {
state->config = config_receive_with_validate(state->fd, server_module_gate, &state->gate_ctx);
if (state->config == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to receive config");
goto done;
return false;
}
/* Apply the super-mode veto the gate decided on (operator --no-super, or a
* daemon module without the `client owner = yes` opt-in) exactly once, so
* every downstream gate (identity_apply_ownership via privilege_super_permitted,
* device-node creation) sees SUPER_MODE_OFF. The gate never mutated the
* received config. */
if (gate_ctx.super_mode_override != -1)
config->super_mode = (SuperMode)gate_ctx.super_mode_override;
if (state->gate_ctx.super_mode_override != -1)
state->config->super_mode = (SuperMode)state->gate_ctx.super_mode_override;
/* Install the codec this connection negotiated before the receiver/writer
* threads start (the server forks per connection, so the process-global
* codec is private to this session). */
compression_set_algo((CompressionAlgo)config->compression_algo);
compression_set_algo((CompressionAlgo)state->config->compression_algo);
/* If the client requested ownership but the effective super mode forbids it
* (operator --no-super, a privileged standalone receiver's secure default, or
* a daemon module without `client owner = yes`), say so ONCE per connection so
* a successful -a/-o/-g transfer is not mistaken for preserved ownership. */
if (config->super_mode == SUPER_MODE_OFF && identity_ownership_requested(config))
if (state->config->super_mode == SUPER_MODE_OFF && identity_ownership_requested(state->config))
log_message(LOG_LEVEL_WARNING,
"requested ownership will NOT be applied: super-user activities are disabled "
"for this connection (operator veto, or module without `client owner = yes`)");
protocol_set_8_bit_output(config->eight_bit_output);
protocol_set_8_bit_output(state->config->eight_bit_output);
/* Server-side per-message protocol deadline for every frame from here on.
* `timeout` is not serialized, so this is the server's own config (the server
* has no --timeout CLI and defaults it to 0). A client's --timeout tightens
* only that client's own protocol I/O; the server floors its own deadline at
* SERVER_IO_TIMEOUT_SEC so a silent peer can never hold a session slot
* forever (the socket layer gets the same floor at startup). */
protocol_session_set_io_timeout(&session, protocol_server_io_timeout_sec(config->timeout));
protocol_session_set_io_timeout(&state->session,
protocol_server_io_timeout_sec(state->config->timeout));
return true;
}
/* Phase 2 -- security gates. The ORDER here is load-bearing and must not be
* merged or reordered: transport/authentication (plaintext refusal, TLS
* client-CN verification), then daemon-root confinement (absolute-destination
* rejection, traversal + within-authorized-root), then delete/force
* authorization -- exactly the sequence the former handler() used. Returns
* false after logging the matching rejection; the caller jumps to the shared
* `done` epilogue. */
static bool server_apply_security_gates(ServerSession* state) {
Config* config = state->config;
const char* authorized_root = utils_get_authorized_root_path();
if (!authorized_root) {
log_message(LOG_LEVEL_ERROR, "No server-side destination root configured");
goto done;
return false;
}
if (!allow_unauthenticated && ssl == NULL) {
if (!allow_unauthenticated && state->ssl == NULL) {
log_message(LOG_LEVEL_ERROR, "Rejected unauthenticated plaintext connection");
goto done;
return false;
}
if (ssl && required_client_cn && !tls_client_identity_allowed(ssl)) {
if (state->ssl && required_client_cn && !tls_client_identity_allowed(state->ssl)) {
log_message(LOG_LEVEL_ERROR, "Rejected TLS client with unauthorized identity");
goto done;
return false;
}
/* Daemon mode: the module's root is the authorized root (installed by
server_module_gate), and the client's destination is a MODULE-RELATIVE
@@ -777,27 +793,27 @@ void handler(int file_descriptor) {
if (g_daemon_conf && config->receive_root_directory && config->receive_root_directory[0] == '/') {
log_message(LOG_LEVEL_ERROR, "Rejected absolute daemon destination (must be relative to the "
"selected module root)");
goto done;
return false;
}
char* destination = config->receive_root_directory;
if (destination && destination[0] != '/')
joined_destination = path_cat(authorized_root, destination);
if (joined_destination)
destination = joined_destination;
state->joined_destination = path_cat(authorized_root, destination);
if (state->joined_destination)
destination = state->joined_destination;
if (!destination || has_path_traversal(destination) ||
!path_is_within_root(authorized_root, destination)) {
log_message(LOG_LEVEL_ERROR, "Rejected destination outside authorized root");
free(joined_destination);
joined_destination = NULL;
goto done;
free(state->joined_destination);
state->joined_destination = NULL;
return false;
}
if (joined_destination) {
if (state->joined_destination) {
free(config->receive_root_directory);
config->receive_root_directory = joined_destination;
joined_destination = NULL;
config->receive_root_directory = state->joined_destination;
state->joined_destination = NULL;
}
if (!config->receive_root_directory) {
goto done;
return false;
}
config->use_delete = config->use_delete && allow_delete;
/* --force (receiver-side) is deletion authority too: it lets an incoming
@@ -807,6 +823,18 @@ void handler(int file_descriptor) {
* --delete-missing-args, so a client cannot use --force to bypass the delete
* policy. */
config->force_delete = config->force_delete && allow_delete;
return true;
}
/* Phase 3 -- session preparation. Installs the negotiated conversion, applies
* the remaining deletion policy, materializes the destination root (--mkpath),
* creates the --delay-updates staging tree, snapshots the identity policy, and
* publishes the --keep-dirlinks/--trust-sender globals and the daemon MOTD.
* All of it must happen before any receiver/writer thread is spawned. Returns
* false after logging the matching failure; the caller jumps to the shared
* `done` epilogue. */
static bool server_prepare_session(ServerSession* state) {
Config* config = state->config;
/* --iconv (protocol 2.16.0): install the receiver-side wire->local conversion
now that the client's full CONVERT_SPEC has been received and validated,
before any received file name is decoded. The server's own --iconv (if
@@ -817,14 +845,14 @@ void handler(int file_descriptor) {
if (!charset_wire_init_receiver(config->iconv_spec, server_iconv_spec)) {
log_message(LOG_LEVEL_ERROR,
"--iconv: unsupported charset conversion requested (LOCAL[,REMOTE])");
goto done;
return false;
}
charset_ready = true;
state->charset_ready = true;
}
/* --delete-missing-args deletes destination mirrors receiver-side, so it is
deletion and stays gated by the same --allow-delete server policy. When
the server policy is off the flag is inert (the missing entries are still
skipped via its implied --ignore-missing-args, but nothing is deleted). */
* deletion and stays gated by the same --allow-delete server policy. When
* the server policy is off the flag is inert (the missing entries are still
* skipped via its implied --ignore-missing-args, but nothing is deleted). */
config->delete_missing_args = config->delete_missing_args && allow_delete;
/* --mkpath: create the destination root (and its missing leading components)
* before anything else; without it the root must pre-exist. The precondition
@@ -839,7 +867,7 @@ void handler(int file_descriptor) {
log_message(LOG_LEVEL_ERROR, "destination root is not available: %s",
escaped_root ? escaped_root : "<allocation failed>");
free(escaped_root);
goto done;
return false;
}
/* A --delay-updates transfer stages under a private 0700 directory inside
the receive root. Create it up front (wiping leftovers of any previously
@@ -849,7 +877,7 @@ void handler(int file_descriptor) {
config->delay_context = delay_updates_context_create(config->receive_root_directory);
if (!config->delay_context || !delay_updates_prepare(config->delay_context)) {
log_message(LOG_LEVEL_ERROR, "Failed to initialize --delay-updates staging area");
goto done;
return false;
}
}
/* Preserve the negotiated identity policy for the fd-relative ownership
@@ -859,7 +887,7 @@ void handler(int file_descriptor) {
rather than silently applying the wrong ownership policy. */
if (!identity_set_active(config)) {
log_message(LOG_LEVEL_ERROR, "Failed to activate identity policy");
goto done;
return false;
}
/* Persist the negotiated --keep-dirlinks policy once, here at config-accept,
before any multithreaded receiver/writer threads are spawned, so the
@@ -887,140 +915,201 @@ void handler(int file_descriptor) {
Wave C note in config.h). */
if (g_daemon_conf) {
char* motd = motd_read_file(g_daemon_conf->global.motd_file);
if (!motd_send(file_descriptor, motd ? motd : "")) {
if (!motd_send(state->fd, motd ? motd : "")) {
free(motd);
log_message(LOG_LEVEL_ERROR, "Failed to send daemon MOTD");
goto done;
return false;
}
free(motd);
}
if (config->use_multithreading) {
Queue* q = queue_create(100, file_destroy);
if (q == NULL)
goto done;
context = pipeline_context_receiver_create(config, q, file_descriptor, ssl);
if (context == NULL) {
queue_destroy(q);
goto done;
}
protocol_session_set_max_alloc(&context->session, config->max_alloc);
protocol_session_set_io_timeout(&context->session,
protocol_server_io_timeout_sec(config->timeout));
atomic_store(&context->session.total_allocated_bytes,
atomic_load(&session.total_allocated_bytes));
pipeline_context_receiver_set_queue_byte_limit(context, RECEIVER_QUEUE_MAX_BYTES);
thrd_t receiver = {0};
thrd_t writer = {0};
bool receiver_created = thrd_create(&receiver, receive_thread, context) == thrd_success;
bool writer_created = false;
if (receiver_created)
writer_created = thrd_create(&writer, write_thread, context) == thrd_success;
if (!receiver_created || !writer_created) {
log_perror("Error creating Threads");
if (receiver_created) {
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
/* Unblock a worker parked in socket I/O without closing the fd (the
* child owns the single close). shutdown() only affects sockets; for
* the --stdio pipe the receiver's per-message poll timeout still
* bounds the join, so do nothing there rather than close a descriptor
* another thread may still be using. */
struct stat fd_stat;
if (fstat(file_descriptor, &fd_stat) == 0 && S_ISSOCK(fd_stat.st_mode))
shutdown(file_descriptor, SHUT_RDWR);
thrd_join(receiver, NULL);
}
if (writer_created)
thrd_join(writer, NULL);
goto done;
}
int receiver_result;
int writer_result;
thrd_join(receiver, &receiver_result);
thrd_join(writer, &writer_result);
bool transfer_ok = receiver_result == thrd_success && writer_result == thrd_success;
if (transfer_ok && !config->dry_run) {
/* Commit-style (late) deletion: receive_thread handed the keep-set
manifest here instead of deleting while write_thread might still be
draining, so by now every file is on disk and the whole transfer is
known to have succeeded. Remove the extras before publishing a
--delay-updates run; the walker skips the staging directory. A
server-contacting --dry-run deletes nothing (no manifest is sent). */
if (context->deferred_manifest) {
size_t deleted = 0;
DeletePathObserver observer = config->report_deletes ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion = manifest_delete_all_observed(
config, context->deferred_manifest, &deleted, observer, (void*)context->deleted_paths);
context->stats.deleted_files += deleted;
if (deletion == DELETE_COMMIT_ERROR) {
transfer_ok = false;
} else if (deletion == DELETE_COMMIT_LIMIT_REACHED) {
/* The transfer still succeeds; the terminal frame reports the capped
deletion so the sender exits 25 like rsync. */
context->delete_limit_reached = true;
}
delete_manifest_free(context->deferred_manifest);
context->deferred_manifest = NULL;
}
/* --delete-delay: receive_thread snapshotted each plan's extras as it
arrived; with the disk writer drained, commit the deferred removals.
--delete-during already applied its plans on the receive thread. */
if (context->deferred_plans) {
/* Defence in depth (the enclosing block already excludes dry-run): a
-n run never commits a deletion. */
if (config->report_deletes)
delete_plan_session_set_delete_observer(
context->deferred_plans, receiver_record_deleted_path, (void*)context->deleted_paths);
DeleteCommitResult deletion =
config->dry_run ? DELETE_COMMIT_OK
: delete_plan_session_commit(context->deferred_plans, config);
context->stats.deleted_files += delete_plan_session_deleted(context->deferred_plans);
if (deletion == DELETE_COMMIT_ERROR) {
transfer_ok = false;
} else if (deletion == DELETE_COMMIT_LIMIT_REACHED) {
context->delete_limit_reached = true;
}
delete_plan_session_destroy(context->deferred_plans);
context->deferred_plans = NULL;
}
}
if (transfer_ok && !config->dry_run) {
/* --delay-updates: receive_thread has finished the whole protocol stream
(including manifest/delete handling) and write_thread has drained its
queue, so every staged file is complete. Publish atomically before the
success/outcome frame so a --remove-source-files sender only learns of
files that were actually installed. */
if (config->delay_updates && config->delay_context &&
!delay_updates_publish(config->delay_context, config)) {
transfer_ok = false;
}
/* P7 Wave D: all writers have joined and the late deletion (and
--delay-updates publication) has committed above, so it is finally safe
to stamp directory times; a directory's mtime must not be clobbered by
its children or by an extra removal. */
if (transfer_ok)
dir_metadata_list_apply(&context->dir_times, config->receive_root_directory, config);
}
if (transfer_ok) {
Status final_status = context->delete_limit_reached ? STATUS_DELETE_LIMIT : STATUS_OK;
/* Emit the optional wire-stats record first (protocol 2.25.0), then the
success/outcome frame, exactly like the single-threaded receiver. */
if (!receiver_send_stats_frame(file_descriptor, config, &context->stats,
context->would_delete, context->deleted_paths) ||
!receiver_send_final_success(file_descriptor, config, &context->outcomes, final_status))
transfer_ok = false;
} else {
send_error_detail(file_descriptor, "transfer failed on receiver");
}
if (!transfer_ok)
log_message(LOG_LEVEL_ERROR, "Transfer failed");
} else {
if (receiver_receive_files(config, file_descriptor) != 0)
log_message(LOG_LEVEL_ERROR, "Transfer failed");
return true;
}
/* Phase 4a -- transfer via the multithreaded receiver. Spawns the receive/write
* thread pair, joins them, then commits the late deletion, --delay-updates
* publication and directory times before emitting the terminal stats/success
* frame. On any failure the helper just returns; the caller's `done` epilogue
* releases the pipeline context (which owns the config and queue) exactly as the
* former inline code did. */
static void server_run_mt_receiver(ServerSession* state) {
Config* config = state->config;
Queue* q = queue_create(100, file_destroy);
if (q == NULL)
return;
state->context = pipeline_context_receiver_create(config, q, state->fd, state->ssl);
if (state->context == NULL) {
queue_destroy(q);
return;
}
protocol_session_set_max_alloc(&state->context->session, config->max_alloc);
protocol_session_set_io_timeout(&state->context->session,
protocol_server_io_timeout_sec(config->timeout));
atomic_store(&state->context->session.total_allocated_bytes,
atomic_load(&state->session.total_allocated_bytes));
pipeline_context_receiver_set_queue_byte_limit(state->context, RECEIVER_QUEUE_MAX_BYTES);
thrd_t receiver = {0};
thrd_t writer = {0};
bool receiver_created = thrd_create(&receiver, receive_thread, state->context) == thrd_success;
bool writer_created = false;
if (receiver_created)
writer_created = thrd_create(&writer, write_thread, state->context) == thrd_success;
if (!receiver_created || !writer_created) {
log_perror("Error creating Threads");
if (receiver_created) {
mtx_lock(&state->context->mutex);
atomic_store(&state->context->cancelled, true);
cnd_broadcast(&state->context->condition_not_full);
cnd_broadcast(&state->context->condition_not_empty);
mtx_unlock(&state->context->mutex);
/* Unblock a worker parked in socket I/O without closing the fd (the
* child owns the single close). shutdown() only affects sockets; for
* the --stdio pipe the receiver's per-message poll timeout still
* bounds the join, so do nothing there rather than close a descriptor
* another thread may still be using. */
struct stat fd_stat;
if (fstat(state->fd, &fd_stat) == 0 && S_ISSOCK(fd_stat.st_mode))
shutdown(state->fd, SHUT_RDWR);
thrd_join(receiver, NULL);
}
if (writer_created)
thrd_join(writer, NULL);
return;
}
int receiver_result;
int writer_result;
thrd_join(receiver, &receiver_result);
thrd_join(writer, &writer_result);
bool transfer_ok = receiver_result == thrd_success && writer_result == thrd_success;
PipelineContextReceiver* context = state->context;
if (transfer_ok && !config->dry_run) {
/* Commit-style (late) deletion: receive_thread handed the keep-set
manifest here instead of deleting while write_thread might still be
draining, so by now every file is on disk and the whole transfer is
known to have succeeded. Remove the extras before publishing a
--delay-updates run; the walker skips the staging directory. A
server-contacting --dry-run deletes nothing (no manifest is sent). */
if (context->deferred_manifest) {
size_t deleted = 0;
DeletePathObserver observer = config->report_deletes ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion = manifest_delete_all_observed(
config, context->deferred_manifest, &deleted, observer, (void*)context->deleted_paths);
context->stats.deleted_files += deleted;
if (deletion == DELETE_COMMIT_ERROR) {
transfer_ok = false;
} else if (deletion == DELETE_COMMIT_LIMIT_REACHED) {
/* The transfer still succeeds; the terminal frame reports the capped
deletion so the sender exits 25 like rsync. */
context->delete_limit_reached = true;
}
delete_manifest_free(context->deferred_manifest);
context->deferred_manifest = NULL;
}
/* --delete-delay: receive_thread snapshotted each plan's extras as it
arrived; with the disk writer drained, commit the deferred removals.
--delete-during already applied its plans on the receive thread. */
if (context->deferred_plans) {
/* Defence in depth (the enclosing block already excludes dry-run): a
-n run never commits a deletion. */
if (config->report_deletes)
delete_plan_session_set_delete_observer(
context->deferred_plans, receiver_record_deleted_path, (void*)context->deleted_paths);
DeleteCommitResult deletion =
config->dry_run ? DELETE_COMMIT_OK
: delete_plan_session_commit(context->deferred_plans, config);
context->stats.deleted_files += delete_plan_session_deleted(context->deferred_plans);
if (deletion == DELETE_COMMIT_ERROR) {
transfer_ok = false;
} else if (deletion == DELETE_COMMIT_LIMIT_REACHED) {
context->delete_limit_reached = true;
}
delete_plan_session_destroy(context->deferred_plans);
context->deferred_plans = NULL;
}
}
if (transfer_ok && !config->dry_run) {
/* --delay-updates: receive_thread has finished the whole protocol stream
(including manifest/delete handling) and write_thread has drained its
queue, so every staged file is complete. Publish atomically before the
success/outcome frame so a --remove-source-files sender only learns of
files that were actually installed. */
if (config->delay_updates && config->delay_context &&
!delay_updates_publish(config->delay_context, config)) {
transfer_ok = false;
}
/* P7 Wave D: all writers have joined and the late deletion (and
--delay-updates publication) has committed above, so it is finally safe
to stamp directory times; a directory's mtime must not be clobbered by
its children or by an extra removal. */
if (transfer_ok)
dir_metadata_list_apply(&context->dir_times, config->receive_root_directory, config);
}
if (transfer_ok) {
if (context->failed_entries > 0)
log_message(LOG_LEVEL_WARNING,
"%zu entr%s failed to materialize; continuing (partial transfer)",
context->failed_entries, context->failed_entries == 1 ? "y" : "ies");
Status final_status = context->delete_limit_reached
? STATUS_DELETE_LIMIT
: (context->failed_entries > 0 ? STATUS_ERROR : STATUS_OK);
/* Emit the optional wire-stats record first (protocol 2.25.0), then the
success/outcome frame, exactly like the single-threaded receiver. */
if (!receiver_send_stats_frame(state->fd, config, &context->stats, context->would_delete,
context->deleted_paths) ||
!receiver_send_final_success(state->fd, config, &context->outcomes, final_status))
transfer_ok = false;
} else {
send_error_detail(state->fd, "transfer failed on receiver");
}
if (!transfer_ok)
log_message(LOG_LEVEL_ERROR, "Transfer failed");
}
/* Phase 4b -- transfer via the single-threaded receiver. Failure is logged
* exactly as before; the caller's `done` epilogue then releases the config. */
static void server_run_st_receiver(ServerSession* state) {
if (receiver_receive_files(state->config, state->fd) != 0)
log_message(LOG_LEVEL_ERROR, "Transfer failed");
}
/* Phase 4 dispatch -- choose the receiver implementation the config asks for.
* Both helpers own their success/failure logging; the caller falls through to
* the shared `done` epilogue either way. */
static void server_run_transfer(ServerSession* state) {
if (state->config->use_multithreading)
server_run_mt_receiver(state);
else
server_run_st_receiver(state);
}
void handler(int file_descriptor) {
/* Single per-connection state; every phase helper below advances it and
* returns false on a logged failure. All teardown state starts empty so the
* single `done` epilogue is safe to reach from any error path (including
* before the config frame arrives). */
ServerSession state;
state.fd = file_descriptor;
state.ssl = io_get_ssl();
protocol_session_init(&state.session, file_descriptor, file_descriptor);
protocol_session_set_ssl(&state.session, state.ssl);
protocol_session_bind(&state.session);
state.gate_ctx.ssl = state.ssl;
state.gate_ctx.fd = file_descriptor;
state.gate_ctx.super_mode_override = -1;
state.gate_ctx.has_peer_ip = false;
state.gate_ctx.peer_ip[0] = '\0';
state.gate_ctx.is_local = false;
state.config = NULL;
state.context = NULL;
state.joined_destination = NULL;
state.charset_ready = false;
if (!server_accept_config(&state))
goto done;
if (!server_apply_security_gates(&state))
goto done;
if (!server_prepare_session(&state))
goto done;
server_run_transfer(&state);
done:
/* Single cleanup epilogue: every error path jumps here, so the iconv
@@ -1029,22 +1118,22 @@ done:
* connection fd is deliberately NOT closed here -- the child functions own
* its single close (plain_child_fn / tls_child_fn), and the --stdio call
* site must leave stdin/stdout open. */
if (charset_ready)
if (state.charset_ready)
charset_wire_free();
/* The delay-updates staging tree is released by config_delete (which the
branch below always reaches), so it is cleaned exactly once. */
identity_clear_active();
protocol_session_unbind();
if (context != NULL) {
if (state.context != NULL) {
/* context owns both the config and the queue it was created with. */
pipeline_context_receiver_destroy(context);
context = NULL;
config = NULL;
pipeline_context_receiver_destroy(state.context);
state.context = NULL;
state.config = NULL;
} else {
config_delete(config);
config = NULL;
config_delete(state.config);
state.config = NULL;
}
free(joined_destination);
free(state.joined_destination);
}
#ifndef FASTSYNC_SERVER_AS_LIB
+12 -12
View File
@@ -20,9 +20,9 @@
static void config_set_defaults(Config* config) {
config->scanner_threads = 0;
config->metadata_explicitly_disabled = false;
config->preserve_perms_explicit_off = false;
config->preserve_times_explicit_off = false;
config->cli.preserve_perms_explicit_off = false;
config->cli.preserve_times_explicit_off = false;
config->cli.metadata_explicitly_disabled = false;
config->show_progress = false;
config->compression_threads = 0;
config->ssh_port = 22;
@@ -44,8 +44,8 @@ static void config_set_defaults(Config* config) {
config->tls_ca = NULL;
config->server_host = str_dup("127.0.0.1");
config->server_port = 8080;
config->server_port_set = false;
config->server_host_set = false;
config->cli.server_port_set = false;
config->cli.server_host_set = false;
/* rsync defaults: --timeout=0 (I/O timeouts disabled) and --contimeout=60.
* A value of 0 disables the client's own deadline on both the socket layer
* (tcp_set_timeouts) and the protocol layer
@@ -68,10 +68,10 @@ static void config_set_defaults(Config* config) {
config->human_readable = false;
config->ignore_errors = false;
config->ignore_missing_args = false;
config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
config->cli_exit_code = 0;
config->compression_level_set = false;
config->checksum_choice_set = false;
config->cli.checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
config->cli.cli_exit_code = 0;
config->cli.compression_level_set = false;
config->cli.checksum_choice_set = false;
config->filters = NULL;
config->files_from = NULL;
config->files_from_set = NULL;
@@ -85,7 +85,6 @@ static void config_set_defaults(Config* config) {
config->rsh_command = NULL;
config->blocking_io = false;
config->outbuf = OUTBUF_BLOCK;
config->old_args = false;
config->remote_options = NULL;
config->remote_option_count = 0;
config->address = NULL;
@@ -101,7 +100,7 @@ static void config_set_defaults(Config* config) {
config->trust_sender = false;
config->stop_after_mins = 0;
config->stop_at = 0;
config->stop_at_set = false;
config->cli.stop_at_set = false;
config->write_batch = NULL;
config->only_write_batch = NULL;
config->read_batch = NULL;
@@ -342,7 +341,8 @@ bool config_derived_use_metadata(const Config* config) {
config->chown_uid_set || config->chown_gid_set || config->usermap_count > 0 ||
config->groupmap_count > 0 || config->update)
return true;
return (config->use_incremental || config->use_delta) && !config->metadata_explicitly_disabled;
return (config->use_incremental || config->use_delta) &&
!config->cli.metadata_explicitly_disabled;
}
bool config_has_basis(const Config* config) {
+69 -46
View File
@@ -83,7 +83,7 @@ typedef struct {
typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF = 2 } SuperMode;
/* ===========================================================================
* Config wire-field table (single source of truth for protocol 2.28.0).
* Config wire-field table (single source of truth for protocol 2.29.0).
*
* Every field below crosses the wire. The table is the ONLY place a
* serialized field is named: config.h expands CONFIG_WIRE_FIELDS() to declare
@@ -342,22 +342,60 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
CONFIG_WIRE_CODEC_FIELDS(X) \
CONFIG_WIRE_PROTECT_FIELDS(X)
/* Client-only, CLI-parse bookkeeping (never serialized). These members exist
* only so the client command-line parser can record HOW an option was
* specified (explicitly set, explicitly negated, or a parser-requested exit
* code); no other module and no wire peer ever needs them. Grouping them in
* one nested member keeps the public Config free of client-CLI-only state. */
typedef struct {
/* Set when the user explicitly turned an attribute off with --no-perms /
* --no-times (long or short form). --incremental/--delta historically
* auto-enabled mode and mtime preservation; these flags let
* cli_finalize_config restore that behavior while still honoring the
* explicit per-attribute negation. A later -p/-t re-enables the attribute
* directly, so the flag only prevents the incremental/delta implication,
* never a POSITIVE request. */
bool preserve_perms_explicit_off;
bool preserve_times_explicit_off;
/* Set by --no-preserve, the explicit opt-out of the whole preservation
* bundle, so the --incremental/--delta auto-preserve implication stays off. */
bool metadata_explicitly_disabled;
/* True when --server-port/--port was explicitly given. --dry-run uses it to
* decide whether a real server handshake was requested, so a plain local
* destination (no explicit port) keeps the existing client-side dry-run
* behavior instead of dialing the default 127.0.0.1:8080. */
bool server_port_set;
/* True when --server-host was explicitly given, and distinct from the
* "127.0.0.1" default: --dry-run uses it to route an explicit remote target
* to the server so it reports receiver state exactly like a real run,
* instead of silently running the client-side manifest. */
bool server_host_set;
/* Codec-negotiation CLI state. The effective pre-transfer checksum is
* Config->checksum_algo (serialized); checksum_transfer_algo is the rsync
* "transfer" half of a two-name --checksum-choice form (validated and used
* only to mirror rsync's whole-file forcing, since FastSync's per-block
* strong hash is fixed). cli_exit_code carries a parser-requested process
* exit status (rsync uses 4 for an unsupported checksum/compress algorithm)
* so main() can mirror it. */
int checksum_transfer_algo;
int cli_exit_code;
/* "The user explicitly chose" bits. They let the per-codec default level /
* checksum list be applied only when the corresponding rsync option was
* omitted (an explicit --compress-level / --checksum-choice always wins). */
bool compression_level_set;
bool checksum_choice_set;
/* True when --stop-at was given. */
bool stop_at_set;
} ConfigCliParse;
typedef struct Config {
/* -j/--threads=N: number of parallel scanner worker threads for the -m
* pipeline. 0 (the default, also set by bare -j/--threads) means "use the
* scanner's built-in default" (4). CLIENT-ONLY: it is a local scheduling
* concern and is NEVER serialized into the wire config frame. */
int scanner_threads;
bool metadata_explicitly_disabled;
/* CLIENT-ONLY (never serialized; not in CONFIG_WIRE_FIELDS). Set when the
* user explicitly turned an attribute off with --no-perms / --no-times (long
* or short form). --incremental/--delta historically auto-enabled mode and
* mtime preservation; these flags let cli_finalize_config restore that
* behavior while still honoring the explicit per-attribute negation. A
* later -p/-t re-enables the attribute directly, so the flag only prevents
* the incremental/delta implication, never a POSITIVE request. */
bool preserve_perms_explicit_off;
bool preserve_times_explicit_off;
/* Client-only CLI-parse bookkeeping (never serialized). See ConfigCliParse. */
ConfigCliParse cli;
bool show_progress;
int compression_threads;
int ssh_port;
@@ -378,18 +416,6 @@ typedef struct Config {
bool use_tls;
char* server_host;
int server_port;
/* True when --server-port/--port was explicitly given. CLIENT-ONLY (never
* serialized): --dry-run uses it to decide whether a real server handshake
* was requested, so a plain local destination (no explicit port) keeps the
* existing client-side dry-run behavior instead of dialing the default
* 127.0.0.1:8080. */
bool server_port_set;
/* True when --server-host was explicitly given. CLIENT-ONLY (never
* serialized), and distinct from the "127.0.0.1" default: --dry-run uses it
* to route an explicit remote target to the server so it reports receiver
* state exactly like a real run, instead of silently running the client-side
* manifest. */
bool server_host_set;
char* tls_cert;
char* tls_key;
char* tls_ca;
@@ -435,22 +461,6 @@ typedef struct Config {
* enters the keep-set. Implied by --delete-missing-args. */
bool ignore_missing_args;
/* Codec-negotiation CLI state (all client-only, never serialized). The
* effective pre-transfer checksum is Config->checksum_algo (serialized);
* checksum_transfer_algo is the rsync "transfer" half of a two-name
* --checksum-choice form (validated and used only to mirror rsync's
* whole-file forcing, since FastSync's per-block strong hash is fixed).
* cli_exit_code carries a parser-requested process exit status (rsync uses 4
* for an unsupported checksum/compress algorithm) so main() can mirror it. */
int checksum_transfer_algo;
int cli_exit_code;
/* Client-only "the user explicitly chose" bits. They let the per-codec
* default level / checksum list be applied only when the corresponding
* rsync option was omitted (an explicit --compress-level / --checksum-choice
* always wins). Never serialized. */
bool compression_level_set;
bool checksum_choice_set;
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
// never serialized to the wire (the receiver must not learn them).
ArrayList* filters; /* --filter=RULE rule strings, in order */
@@ -486,7 +496,6 @@ typedef struct Config {
/* --outbuf mode (OutbufMode): stdout/stderr buffering. Client-only launch
* concern: NEVER crosses the wire. */
int outbuf;
bool old_args;
/* --remote-option=OPT (Phase 5, long form only): one or more extra command-line
* options to append to the REMOTE server invocation over SSH. CLIENT-ONLY:
* they are composed into the remote command line by ssh_build_remote_command()
@@ -556,7 +565,6 @@ typedef struct Config {
* process and are NEVER serialized into the config frame. */
int stop_after_mins; /* --stop-after=MINS minutes; 0 when unset */
time_t stop_at; /* --stop-at=... absolute wall-clock deadline */
bool stop_at_set; /* true when --stop-at was given */
/* Client-only residual-batch paths. A residual batch is a self-contained
* single-file record of the whole source tree (full file images using the
@@ -727,11 +735,13 @@ typedef struct Config {
* --copy-as) imply it. */
/* fake_super */
/* --fake-super: receiver-only. When set, each written file additionally gets
* a reserved user.fastsync.stat xattr recording the RESOLVED uid/gid (the
* rsync's reserved user.rsync.%stat xattr recording the RESOLVED uid/gid (the
* source's own when no ownership request is active, else the --chown/--usermap
* result) plus mode/mtime so a later privileged restore could re-apply them.
* It NEVER real-chowns: the point is to record the source ownership on an
* unprivileged receiver. Crosses the wire. */
* result) plus the full mode and rdev, in rsync 3.4.1's grammar, so the tree is
* interoperable and a later privileged restore could re-apply them. mtime is
* carried by the file's own timestamp, exactly as rsync does it. It NEVER
* real-chowns: the point is to record the source ownership on an unprivileged
* receiver. Crosses the wire. */
/* module */
/* Daemon module selection (Wave A, protocol 2.15.0). Client-composed from a
* host::module/path destination; NULL or "" means "no module" (the ordinary
@@ -1061,7 +1071,20 @@ typedef struct Config {
* filter rules so the receiver can protect DESTINATION-ONLY entries from
* --delete with `protect`/`risk` rules (rsync parity). The block appends after
* compression_algo; see CONFIG_WIRE_PROTECT_FIELDS. */
#define PROTOCOL_VERSION "2.28.0"
/* (10) Symlink xattrs/ACLs (protocol 2.29.0): the config-frame LAYOUT is
* unchanged (the derived use_xattrs bit already crosses the wire), but the
* STATUS_SYMLINK frame BODY grows a trailing bounded xattr block when -X/-A is
* negotiated -- exactly the block STATUS_MKDIR, STATUS_DIR_TIMES and regular
* files already carry. The sender captures the symlink's OWN xattrs with
* llistxattr/lgetxattr (so it can never attach the REFERENT's attributes to the
* link) and the receiver re-applies them to the link itself with lsetxattr on a
* confined /proc/self/fd/<parent>/<leaf> path (there is no *at xattr syscall and
* fsetxattr cannot target a symlink). A 2.28 peer that does not consume the new
* trailing block would desynchronize after every symlink, so the protocol
* version must bump; the strict same-version handshake (config_receive rejects a
* mismatched version before parsing anything else) keeps a 2.29 client and a
* 2.28 server from ever reaching that state. */
#define PROTOCOL_VERSION "2.29.0"
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
#define MAX_BASIS_DIRS 64
+216 -1
View File
@@ -1,5 +1,6 @@
#include "daemon_conf.h"
#include "credentials.h"
#include "log.h"
#include "utils.h"
#include <arpa/inet.h>
#include <ctype.h>
@@ -33,6 +34,158 @@ static bool key_equals(const char* key, const char* canonical) {
return strcasecmp(key, canonical) == 0;
}
/* True when `key` matches one of the NUL-terminated names in `list`. */
static bool key_in_list(const char* key, const char* const* list, size_t count) {
for (size_t i = 0; i < count; i++) {
if (strcasecmp(key, list[i]) == 0)
return true;
}
return false;
}
/* rsync 3.4.1 rsyncd.conf GLOBAL keys accepted in the pre-module section that
* have no FastSync equivalent. They are recognized and documented as inert:
* accepting a real rsync config must not fail on a logging/process key, but a
* silently-reinterpreted key is never invented. `pidfile`/`logfile` are the
* compact --dparam spellings rsync documents. The same list is used by the
* `--dparam` dispatch (apply_global_key), so there is a single impl. */
static const char* const kRsyncInertGlobalKeys[] = {
"pid file",
"pidfile",
"log file",
"logfile",
"socket options",
"sockopts",
"listen backlog",
"syslog facility",
"syslog tag",
"log format",
"use chroot",
"uid",
"gid",
"timeout",
"max verbosity",
"min verbosity",
"lock file",
"transfer logging",
"strict modes",
"reverse lookup",
"forward lookup",
"ignore errors",
"ignore nonreadable",
"dont compress",
};
/* rsync 3.4.1 rsyncd.conf MODULE keys accepted in a [module] section that have
* no FastSync equivalent (accepted-and-documented inert). Keys with a FastSync
* meaning (`path`, `read only`, `write only`, `auth users`, `max connections`,
* `hosts allow`/`hosts deny`, `client owner`) are handled by apply_module_key
* before this list is consulted. Security-relevant keys (`exclude`, `filter`,
* `secrets file`, `refuse options`, ...) are inert, so a daemon-side filter or
* rsync secrets file is NOT enforced: each is loudly warned about at load time
* (see kRsyncUnenforcedModuleSecurityKeys) and documented as a residual in
* RSYNC_COMPAT.md. */
static const char* const kRsyncInertModuleKeys[] = {
"comment",
"use chroot",
"daemon chroot",
"uid",
"gid",
"daemon uid",
"daemon gid",
"exclude",
"include",
"exclude from",
"include from",
"filter",
"max verbosity",
"min verbosity",
"lock file",
"transfer logging",
"log file",
"log format",
"syslog facility",
"syslog tag",
"timeout",
"secrets file",
"auth digest",
"strict modes",
"numeric ids",
"fake super",
"munge symlinks",
"list",
"dont compress",
"charset",
"refuse options",
"incoming chmod",
"outgoing chmod",
"open noatime",
"max size",
"min size",
"temp dir",
"pre-xfer exec",
"post-xfer exec",
"name converter",
"proxy protocol",
"proxy protocol hosts",
"reverse lookup",
"forward lookup",
"ignore errors",
"ignore nonreadable",
};
/* Subset of the inert rsync keys whose intent is access control (data
* visibility, credential source, transfer hooks, daemon privilege), plus the
* global keys that shape the daemon's privilege/identity. These load for
* rsync-config compatibility, but because FastSync ignores them an operator
* migrating a hardened rsyncd.conf must not believe the restriction applies.
* The loader emits one LOG_LEVEL_WARNING per occurrence naming the key (and the
* module, for a module key). `write only` is deliberately absent: it is mapped
* onto writability instead (FastSync is push-only, so a write-only module is
* simply writable). */
static const char* const kRsyncUnenforcedModuleSecurityKeys[] = {
"secrets file",
"auth digest",
"refuse options",
"exclude",
"include",
"exclude from",
"include from",
"filter",
"max size",
"min size",
"pre-xfer exec",
"post-xfer exec",
"incoming chmod",
"outgoing chmod",
"name converter",
"use chroot",
"daemon chroot",
"uid",
"gid",
"daemon uid",
"daemon gid",
"munge symlinks",
"fake super",
"strict modes",
"proxy protocol",
"proxy protocol hosts",
};
static const char* const kRsyncUnenforcedGlobalSecurityKeys[] = {
"use chroot",
"uid",
"gid",
"strict modes",
};
#define kRsyncInertGlobalCount (sizeof(kRsyncInertGlobalKeys) / sizeof(kRsyncInertGlobalKeys[0]))
#define kRsyncInertModuleCount (sizeof(kRsyncInertModuleKeys) / sizeof(kRsyncInertModuleKeys[0]))
#define kRsyncUnenforcedModuleSecurityCount \
(sizeof(kRsyncUnenforcedModuleSecurityKeys) / sizeof(kRsyncUnenforcedModuleSecurityKeys[0]))
#define kRsyncUnenforcedGlobalSecurityCount \
(sizeof(kRsyncUnenforcedGlobalSecurityKeys) / sizeof(kRsyncUnenforcedGlobalSecurityKeys[0]))
static bool parse_bool_value(const char* value, bool* out) {
if (strcasecmp(value, "yes") == 0 || strcasecmp(value, "true") == 0 || strcmp(value, "1") == 0) {
*out = true;
@@ -250,6 +403,10 @@ DaemonConf* daemon_conf_create(void) {
if (!conf)
return NULL;
conf->global.port = DAEMON_CONF_DEFAULT_PORT;
/* rsync modules are READ-ONLY unless `read only = no` (or `write only = yes`)
* is set, so FastSync must default the same way: a migrated rsyncd.conf that
* omits `read only` is served read-only, never writable. */
conf->global.read_only_default = true;
conf->global.max_connections = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS;
conf->global.auth_failure_delay_ms = DAEMON_CONF_DEFAULT_AUTH_FAILURE_DELAY_MS;
conf->global.max_connections_per_host = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS_PER_HOST;
@@ -324,7 +481,7 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
char* err, size_t err_size) {
if (key_equals(key, "port"))
return store_port(&conf->global.port, value, err, err_size);
if (key_equals(key, "motd file")) {
if (key_equals(key, "motd file") || key_equals(key, "motdfile")) {
if (!store_string(&conf->global.motd_file, value)) {
set_error(err, err_size, "out of memory parsing 'motd file'");
return false;
@@ -338,6 +495,25 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
}
return true;
}
/* rsync allows the `read only` module key in the global section as the
* default for modules defined after it. Map it to that default (a later
* --dparam re-applies it to modules that did not set their own value) so a
* global `read only = yes` cannot be silently dropped into a writable
* default. */
if (key_equals(key, "read only")) {
bool parsed;
if (!parse_bool_value(value, &parsed)) {
set_error(err, err_size, "global 'read only' must be yes/no (or true/false/1/0), got '%s'",
value);
return false;
}
conf->global.read_only_default = parsed;
for (int i = 0; i < conf->module_count; i++) {
if (!conf->modules[i].read_only_explicit)
conf->modules[i].read_only = parsed;
}
return true;
}
if (key_equals(key, "max connections"))
return store_max_connections(&conf->global.max_connections, value, NULL, err, err_size);
if (key_equals(key, "max connections per host"))
@@ -360,6 +536,15 @@ static bool apply_global_key(DaemonConf* conf, char* key, const char* value, boo
if (key_equals(key, "hosts deny"))
return store_host_list(&conf->global.hosts_deny, &conf->global.hosts_deny_count, value,
"hosts deny", NULL, replace_hosts, err, err_size);
/* A recognized rsync global key with no FastSync equivalent loads inert. */
if (key_in_list(key, kRsyncInertGlobalKeys, kRsyncInertGlobalCount)) {
if (key_in_list(key, kRsyncUnenforcedGlobalSecurityKeys, kRsyncUnenforcedGlobalSecurityCount))
log_message(LOG_LEVEL_WARNING,
"daemon config: global key '%s' is accepted for rsync compatibility but is NOT "
"enforced by FastSync; the restriction it expresses will not be applied",
key);
return true;
}
set_error(err, err_size, "unknown global key '%s'", key);
return false;
}
@@ -388,6 +573,26 @@ static bool apply_module_key(DaemonModule* module, char* key, char* value, char*
return false;
}
module->read_only = parsed;
module->read_only_explicit = true;
return true;
}
/* rsync's `write only = yes` makes the module client-writable. FastSync has
* no read/pull path, so mapping it to writability is the exact
* security-relevant effect; set `read_only_explicit` so a global default
* cannot override the module's explicit choice. `write only = no` is the
* rsync default and leaves the module's read-only state untouched. */
if (key_equals(key, "write only")) {
bool parsed;
if (!parse_bool_value(value, &parsed)) {
set_error(err, err_size,
"module '%s': 'write only' must be yes/no (or true/false/1/0), got '%s'",
module->name, value);
return false;
}
if (parsed) {
module->read_only = false;
module->read_only_explicit = true;
}
return true;
}
if (key_equals(key, "client owner")) {
@@ -457,6 +662,15 @@ static bool apply_module_key(DaemonModule* module, char* key, char* value, char*
if (key_equals(key, "hosts deny"))
return store_host_list(&module->hosts_deny, &module->hosts_deny_count, value, "hosts deny",
module->name, false, err, err_size);
/* A recognized rsync module key with no FastSync equivalent loads inert. */
if (key_in_list(key, kRsyncInertModuleKeys, kRsyncInertModuleCount)) {
if (key_in_list(key, kRsyncUnenforcedModuleSecurityKeys, kRsyncUnenforcedModuleSecurityCount))
log_message(LOG_LEVEL_WARNING,
"daemon config: module '%s' key '%s' is accepted for rsync compatibility but is "
"NOT enforced by FastSync; the restriction it expresses will not be applied",
module->name, key);
return true;
}
set_error(err, err_size, "unknown key '%s' in module '%s'", key, module->name);
return false;
}
@@ -507,6 +721,7 @@ static int open_module(DaemonConf* conf, int* current_module, const char* name,
}
conf->modules = grown;
memset(&conf->modules[conf->module_count], 0, sizeof(DaemonModule));
conf->modules[conf->module_count].read_only = conf->global.read_only_default;
conf->modules[conf->module_count].name = str_dup(name);
if (!conf->modules[conf->module_count].name) {
set_error(err, err_size, "out of memory adding module '%s'", name);
+39 -13
View File
@@ -17,7 +17,20 @@
* DAEMON_CONF_MAX_LINE all fail the whole load with a clear, line-numbered
* error instead of being silently ignored. This keeps a typo from silently
* changing what a module serves.
*/
*
* rsync compatibility: to reduce the divergence from rsync 3.4.1's rsyncd.conf
* grammar, the parser also ACCEPTS the common rsync GLOBAL and MODULE keys.
* Keys with a FastSync equivalent are mapped onto it (the native spellings are
* unchanged; `read only` defaults to yes like rsync, and `write only = yes`
* opts a module into writability). Keys with no FastSync equivalent are
* accepted and documented as inert (they load successfully but have no effect)
* rather than failing the whole config; the accepted inert set is listed in
* kRsyncInertGlobalKeys / kRsyncInertModuleKeys in daemon_conf.c and in
* RSYNC_COMPAT.md. Every inert key whose intent is access control is loudly
* warned about at load time (kRsyncUnenforced*SecurityKeys) so an operator
* migrating a hardened rsyncd.conf is never misled into believing the
* restriction is enforced. A key outside both the FastSync-native grammar and
* the recognized rsync subset is still rejected as unknown. */
/* A daemon module's configured root is used exactly like the standalone
* server's --destination-root: the daemon confines every connection that
@@ -42,15 +55,21 @@
* store refuses (fail closed) rather than falling open; see server.c. Auth is
* never bypassed by ignoring the list. */
typedef struct DaemonModule {
char* name; /* module name, as the client requests it */
char* path; /* module root (daemon-side authorized root) */
bool read_only; /* `read only = yes/no`; default no */
bool client_owner; /* `client owner = yes/no`; default no. Per-module opt-in
that lets this module's clients choose ownership
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/
--copy-as) and request explicit --super super-user
activities. Without it the daemon refuses all of them. */
char** auth_users; /* `auth users = a,b`; Wave B credential list */
char* name; /* module name, as the client requests it */
char* path; /* module root (daemon-side authorized root) */
bool read_only; /* `read only = yes/no`; defaults to the global `read only`
default (rsync allows it in the global section), which is
itself default YES (rsync modules are read-only unless
`read only = no` / `write only = yes` opts in) */
bool read_only_explicit; /* set when this module set its own `read only` or
`write only = yes`, so a later global default (from a
`--dparam read only=`) does not override it */
bool client_owner; /* `client owner = yes/no`; default no. Per-module opt-in
that lets this module's clients choose ownership
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/
--copy-as) and request explicit --super super-user
activities. Without it the daemon refuses all of them. */
char** auth_users; /* `auth users = a,b`; Wave B credential list */
int auth_user_count;
/* `max connections = N` (optional per-module cap). 0 means unlimited. The
* per-connection child records the selected module in the shared registry
@@ -69,6 +88,10 @@ typedef struct DaemonConfGlobals {
int port; /* `port`, default DAEMON_CONF_DEFAULT_PORT (873) */
char* motd_file; /* `motd file`, may be NULL */
char* address; /* `address` (optional bind address), may be NULL */
bool read_only_default; /* global `read only` default for modules defined
after it (rsync allows the module key in the
global section); default YES to match rsync's
read-only modules */
int max_connections; /* `max connections`, default
DAEMON_CONF_DEFAULT_MAX_CONNECTIONS (100) */
int auth_failure_delay_ms; /* `auth failure delay`, milliseconds; default
@@ -152,10 +175,13 @@ const DaemonModule* daemon_conf_find_module(const DaemonConf* conf, const char*
bool daemon_module_name_valid(const char* name);
/* Parse one --dparam=KEY=VALUE (or "--dparam KEY=VALUE") override string and
* apply it to the global keys only. Keys are case-insensitive and limited to
* the global keys defined by the grammar (port, motd file, address,
* apply it to the global keys only. Keys are case-insensitive and cover the
* global keys defined by the grammar (port, motd file, address, read only,
* max connections, max connections per host, auth failure delay,
* auth lockout threshold, auth lockout duration, hosts allow, hosts deny).
* auth lockout threshold, auth lockout duration, hosts allow, hosts deny) plus
* the recognized inert rsync global keys and the compact rsync spellings
* (`motdfile`, `pidfile`, `logfile`). Applying `read only` sets the global
* default and re-applies it to every module that did not set its own value.
* Returns 0 on success, -1 on error (err filled). */
int daemon_conf_apply_dparam(DaemonConf* conf, const char* assignment, char* err, size_t err_size);
+656
View File
@@ -0,0 +1,656 @@
#include "delete.h"
#include "delay_updates.h"
#include "filter.h"
#include "log.h"
#include "utils.h"
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
/* Build the keep-set index from the exact manifest entries only. A lookup of
`rel` succeeds iff `rel` is a kept entry, a kept directory, or an ancestor
directory of kept content (the old is_dir_in_manifest predicate); the sorted
view answers "is an ancestor of kept content" without materializing any
per-component prefix copy, so the index is O(manifest size) memory. */
static bool build_keep_index(const ArrayList* manifest, PathIndex* index) {
if (!manifest || manifest->size <= 0)
return path_index_build(index, NULL, 0);
return path_index_build(index, (const char* const*)manifest->items, (size_t)manifest->size);
}
static bool keep_is_dir(const PathIndex* index, const char* rel_path) {
return path_index_contains(index, rel_path) || path_index_has_descendant(index, rel_path);
}
static bool keep_is_file(const PathIndex* index, const char* rel_path) {
return path_index_contains(index, rel_path);
}
/* True when child_rel is, or lies below, a protected entry. A prefix "a"
therefore protects "a" and "a/b/c" but not "ab". Entries with top_level_only
set only protect DIRECT children of the receive root (at_root); nested
directories that share such a name stay ordinary destination content. */
bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSkipEntry* skips,
int skip_count) {
for (int i = 0; i < skip_count; i++) {
if (skips[i].top_level_only && !at_root)
continue;
size_t prefix_len = strlen(skips[i].prefix);
if (strncmp(child_rel, skips[i].prefix, prefix_len) == 0 &&
(child_rel[prefix_len] == '\0' || child_rel[prefix_len] == '/'))
return true;
}
return false;
}
/* Per-run deletion budget and tallies. `max_delete` is the cap on the number
of entries the walker may remove (SIZE_MAX = unlimited); once it is reached
the remaining extras are counted in `skipped` and left in place, matching
rsync's partial --max-delete behavior. */
typedef struct {
size_t max_delete;
size_t deleted;
size_t skipped;
bool limit_hit;
} DeleteBudget;
/* True when direct children of the directory named by `rel` may be removed.
With no synchronization info (dirs == NULL) the whole tree is deletable; when
a dirs index is supplied only its exact entries are (the receive root is the
"." sentinel). */
static bool is_synced_dir(const PathIndex* dirs, const char* rel) {
if (!dirs)
return true;
return path_index_contains(dirs, rel[0] == '\0' ? "." : rel);
}
/* Unsigned byte-wise string compare, matching rsync's u_strcmp (a signed
strcmp would order bytes >= 0x80 differently). */
static int delete_name_cmp(const char* a, const char* b) {
const unsigned char* pa = (const unsigned char*)a;
const unsigned char* pb = (const unsigned char*)b;
while (*pa != '\0' && *pa == *pb) {
pa++;
pb++;
}
return (int)*pa - (int)*pb;
}
bool delete_dir_entries_collect(int dirfd, DeleteDirEntry** out, size_t* count,
bool* operation_ok) {
*out = NULL;
*count = 0;
if (operation_ok)
*operation_ok = true;
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (scanfd < 0)
return false;
DIR* dir = fdopendir(scanfd);
if (!dir) {
close(scanfd);
return false;
}
DeleteDirEntry* entries = NULL;
size_t used = 0;
size_t capacity = 0;
bool ok = true;
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT && operation_ok)
*operation_ok = false;
continue;
}
if (used == capacity) {
size_t next = capacity == 0 ? 16 : capacity * 2;
DeleteDirEntry* grown = realloc(entries, next * sizeof(*grown));
if (!grown) {
ok = false;
break;
}
entries = grown;
capacity = next;
}
entries[used].name = str_dup(entry->d_name);
if (!entries[used].name) {
ok = false;
break;
}
entries[used].is_dir = S_ISDIR(st.st_mode);
used++;
}
closedir(dir);
if (!ok) {
delete_dir_entries_free(entries, used);
return false;
}
*out = entries;
*count = used;
return true;
}
void delete_dir_entries_free(DeleteDirEntry* entries, size_t count) {
if (!entries)
return;
for (size_t i = 0; i < count; i++)
free(entries[i].name);
free(entries);
}
/* rsync's extraneous-entry order: subdirectories before files, each group in
descending name order. */
int delete_dir_entry_cmp_desc(const void* a, const void* b) {
const DeleteDirEntry* ea = a;
const DeleteDirEntry* eb = b;
if (ea->is_dir != eb->is_dir)
return ea->is_dir ? -1 : 1;
return -delete_name_cmp(ea->name, eb->name);
}
/* rsync's kept-subdirectory order: plain ascending name. */
int delete_dir_entry_cmp_asc(const void* a, const void* b) {
const DeleteDirEntry* ea = a;
const DeleteDirEntry* eb = b;
return delete_name_cmp(ea->name, eb->name);
}
/* How the shared classification/descent walk disposes of an extra it has
identified. LIST records the destination-relative path without touching disk
(the -n/--dry-run would-delete enumeration); DELETE unlinks/rmdirs it, charges
the shared --max-delete budget and notifies the observer. Both modes classify
and traverse identically, so the dry-run enumeration and the real deletion
cannot drift. */
typedef enum { DELETE_WALK_MODE_DELETE, DELETE_WALK_MODE_LIST } DeleteWalkMode;
typedef struct {
DeleteWalkMode mode;
DeleteBudget* budget; /* DELETE mode */
ArrayList* out; /* LIST mode: receives strdup'd relative paths */
size_t* recorded; /* LIST mode */
DeletePathObserver observer; /* DELETE mode */
void* observer_context; /* DELETE mode */
} DeleteWalkState;
/* Remove the extras directly inside the directory open on `dirfd` (DELETE mode)
or record the paths that WOULD be removed (LIST mode), recursing into every
child directory so kept content below a synchronized prefix is reached.
`all_removed` reports whether every child entry was removed (so the caller may
rmdir this directory). A child directory is never removed when it is itself a
synchronized directory or holds kept content; with a dirs index supplied,
direct children of a non-synchronized directory are never extras at all (they
are left in place but still descended into). Symlinks are unlinked like any
other non-directory extra (never followed).
Entries are processed in rsync's order (extraneous subdirectories in
descending name order, then extraneous files, then kept subdirectories in
ascending order) rather than readdir() order, so `--max-delete` leaves the
same survivors and the `--info=del`/dry-run line order matches rsync. */
static bool delete_walk_fd(int dirfd, const char* rel_path, const PathIndex* keep,
const PathIndex* dirs, DeleteWalkState* state,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, bool parent_deletable,
bool* all_removed) {
DeleteDirEntry* entries = NULL;
size_t count = 0;
bool collect_ok = true;
if (!delete_dir_entries_collect(dirfd, &entries, &count, &collect_ok))
return false;
bool operation_ok = collect_ok;
bool local_survives = false;
bool* shielded = calloc(count ? count : 1, sizeof(bool));
bool* is_extra = calloc(count ? count : 1, sizeof(bool));
if (!shielded || !is_extra) {
free(shielded);
free(is_extra);
delete_dir_entries_free(entries, count);
return false;
}
/* A directory is deletable when it or ANY ancestor is synchronized; the
`parent_deletable` flag carries that down the recursion so dest-only
directories below a synchronized root are removed wholesale. */
bool deletable = parent_deletable || is_synced_dir(dirs, rel_path);
bool at_root = rel_path[0] == '\0';
/* Reproduce rsync's traversal order: extraneous subdirectories in descending
name order, then extraneous files in descending name order, and kept
subdirectories only afterwards (ascending). Sorting up front also fixes the
identity of the survivors under a partial --max-delete. */
if (count > 1)
qsort(entries, count, sizeof(*entries), delete_dir_entry_cmp_desc);
size_t dir_count = 0;
while (dir_count < count && entries[dir_count].is_dir)
dir_count++;
/* Classify every entry up front (the verdict does not depend on processing
order) so the ordered passes below can act on it. */
for (size_t i = 0; i < count; i++) {
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
/* A --delay-updates run keeps its staging directory as a direct child of
the receive root, and basis-dir snapshots live below it too. Their
contents are not manifest entries, so descending into them would delete
every staged / basis file as an "extra". Only the staging name (a
top-level-only prefix) and the basis prefixes are protected: a nested
destination directory that happens to be called .fastsync-stage is
ordinary content. */
if (path_under_skip_prefix(child_rel, at_root, skips, skip_count)) {
shielded[i] = true;
local_survives = true;
} else if (protect_rules &&
filter_rules_apply_side(protect_rules, child_rel, entries[i].name, entries[i].is_dir,
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
/* A first-match protect rule shields the extra; for a directory the whole
subtree is shielded (rsync prunes an excluded directory), so do not
descend. */
shielded[i] = true;
local_survives = true;
} else if (entries[i].is_dir) {
bool child_synced = dirs && path_index_contains(dirs, child_rel);
is_extra[i] = deletable && !child_synced && !keep_is_dir(keep, child_rel);
if (!is_extra[i])
local_survives = true;
} else {
is_extra[i] = deletable && !keep_is_file(keep, child_rel);
if (!is_extra[i])
local_survives = true;
}
free(child_rel);
}
/* Pass 1: extraneous subdirectories, descending. */
for (size_t i = 0; i < dir_count; i++) {
if (!is_extra[i])
continue;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
int childfd = openat(dirfd, entries[i].name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!delete_walk_fd(childfd, child_rel, keep, dirs, state, skips, skip_count, protect_rules,
deletable, &child_all_removed))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
if (child_all_removed && deletable) {
if (state->mode == DELETE_WALK_MODE_LIST) {
/* Record the directory with rsync's trailing slash. */
size_t len = strlen(child_rel);
char* copy = malloc(len + 2);
if (!copy) {
operation_ok = false;
} else {
memcpy(copy, child_rel, len);
copy[len] = '/';
copy[len + 1] = '\0';
if (!array_list_add(state->out, copy)) {
free(copy);
operation_ok = false;
} else {
(*state->recorded)++;
}
}
} else if (state->budget->deleted >= state->budget->max_delete) {
state->budget->limit_hit = true;
state->budget->skipped++;
local_survives = true;
} else if (unlinkat(dirfd, entries[i].name, AT_REMOVEDIR) != 0) {
/* ENOENT: already gone (fine). ENOTEMPTY/EEXIST: the directory still
holds entries the walker leaves in place (a protected excluded
prefix, a kept file the manifest protects, a symlink); rsync leaves
such a directory behind, so this is not an error. Only genuine I/O
failures abort the deletion. */
if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST)
operation_ok = false;
local_survives = true;
} else {
state->budget->deleted++;
/* rsync reports a removed directory with a trailing slash. */
if (state->observer) {
size_t len = strlen(child_rel);
char* with_slash = malloc(len + 2);
if (with_slash) {
memcpy(with_slash, child_rel, len);
with_slash[len] = '/';
with_slash[len + 1] = '\0';
state->observer(state->observer_context, with_slash);
free(with_slash);
} else {
state->observer(state->observer_context, child_rel);
}
}
}
} else {
local_survives = true;
}
free(child_rel);
}
/* Pass 2: extraneous files, descending. */
for (size_t i = dir_count; i < count; i++) {
if (!is_extra[i])
continue;
if (state->mode == DELETE_WALK_MODE_LIST) {
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
char* copy = str_dup(child_rel);
if (!copy || !array_list_add(state->out, copy)) {
free(copy);
operation_ok = false;
} else {
(*state->recorded)++;
}
free(child_rel);
} else if (state->budget->deleted >= state->budget->max_delete) {
state->budget->limit_hit = true;
state->budget->skipped++;
local_survives = true;
} else if (unlinkat(dirfd, entries[i].name, 0) != 0) {
if (errno != ENOENT)
operation_ok = false;
local_survives = true;
} else {
state->budget->deleted++;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (child_rel) {
if (state->observer)
state->observer(state->observer_context, child_rel);
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
free(escaped_path);
}
free(child_rel);
}
}
/* Pass 3: kept subdirectories, ascending (rsync descends into these only
after the parent's own extras have been handled). */
for (size_t i = dir_count; i-- > 0;) {
if (is_extra[i] || shielded[i])
continue;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
int childfd = openat(dirfd, entries[i].name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!delete_walk_fd(childfd, child_rel, keep, dirs, state, skips, skip_count, protect_rules,
deletable, &child_all_removed))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
/* A kept/synchronized directory is never removed. */
local_survives = true;
free(child_rel);
}
free(shielded);
free(is_extra);
delete_dir_entries_free(entries, count);
*all_removed = !local_survives;
return operation_ok;
}
/* Open the receive root following the same authorized-root confinement the
walker uses, or dest_root directly when no authorized root is installed. */
static int open_destination_root(const char* dest_root) {
int root_fd = utils_get_authorized_root_fd();
if (root_fd >= 0) {
if (utils_get_authorized_root_path())
return utils_open_authorized_destination(dest_root);
if (dest_root == NULL)
return dup(root_fd);
return -1;
}
return open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out) {
if (count_out)
*count_out = 0;
if (!manifest || !out)
return false;
PathIndex keep;
if (!build_keep_index(manifest, &keep))
return false;
PathIndex dirs;
bool have_dirs = synced_dirs != NULL;
if (have_dirs &&
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
path_index_free(&keep);
return false;
}
int rootfd = open_destination_root(dest_root);
if (rootfd < 0) {
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
return false;
}
bool all_removed = false;
size_t recorded = 0;
DeleteWalkState state = {.mode = DELETE_WALK_MODE_LIST,
.budget = NULL,
.out = out,
.recorded = &recorded,
.observer = NULL,
.observer_context = NULL};
bool ok = delete_walk_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &state, skips, skip_count,
protect_rules, false, &all_removed);
if (close(rootfd) != 0)
ok = false;
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
if (count_out)
*count_out = recorded;
return ok;
}
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer,
void* observer_context) {
if (deleted_out)
*deleted_out = 0;
if (skipped_out)
*skipped_out = 0;
if (!manifest)
return DELETE_WALK_ERROR;
/* Index the keep-set (and the synchronized-dir set, when supplied) once so
membership is answered in O(path length) instead of scanning every entry
for every destination entry. */
PathIndex keep;
if (!build_keep_index(manifest, &keep))
return DELETE_WALK_ERROR;
PathIndex dirs;
bool have_dirs = synced_dirs != NULL;
if (have_dirs &&
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
path_index_free(&keep);
return DELETE_WALK_ERROR;
}
int rootfd = open_destination_root(dest_root);
if (rootfd < 0) {
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
return DELETE_WALK_ERROR;
}
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
bool all_removed = false;
DeleteWalkState state = {.mode = DELETE_WALK_MODE_DELETE,
.budget = &budget,
.out = NULL,
.recorded = NULL,
.observer = observer,
.observer_context = observer_context};
bool ok = delete_walk_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &state, skips, skip_count,
protect_rules, false, &all_removed);
if (close(rootfd) != 0)
ok = false;
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
if (deleted_out)
*deleted_out = budget.deleted;
if (skipped_out)
*skipped_out = budget.skipped;
if (!ok)
return DELETE_WALK_ERROR;
return budget.limit_hit ? DELETE_WALK_LIMIT_REACHED : DELETE_WALK_OK;
}
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, size_t* deleted_out,
size_t* skipped_out) {
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
skip_count, protect_rules, deleted_out, skipped_out, NULL,
NULL);
}
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL, NULL) ==
DELETE_WALK_OK;
}
/* Build the delete-walk protection prefix for one basis directory. The walker
compares paths relative to the receive root, so a relative entry is already
in the right form; an absolute entry that lies below the root is converted to
its root-relative form, and one outside the root returns NULL (the walk
cannot reach it, and it is not protected data beneath the root). Exposed so
tests can exercise the root-of-"/" child mapping directly. */
char* delete_basis_relative(const Config* config, const char* path) {
if (!path)
return NULL;
if (path[0] != '/')
return str_dup(path);
const char* root = config->receive_root_directory;
if (!root || root[0] != '/')
return NULL;
size_t root_len = strlen(root);
while (root_len > 1 && root[root_len - 1] == '/')
root_len--;
if (strncmp(path, root, root_len) != 0)
return NULL;
if (root_len == 1) {
/* `root` is "/" (the only single-character absolute root): every absolute
path is below it, and the child relative form is everything after the
leading '/'. */
if (path[1] == '\0')
return NULL; /* identical to the root, not a child */
return str_dup(path + 1);
}
if (path[root_len] != '/')
return NULL; /* identical or a sibling sharing a name prefix */
return str_dup(path + root_len + 1);
}
bool delete_skips_build(const Config* config, const ArrayList* protected_paths,
const ArrayList* size_skipped, bool basis_root_relative,
DeleteSkipSet* out) {
if (!out)
return false;
out->entries = NULL;
out->owned_prefixes = NULL;
out->count = 0;
out->owned_count = 0;
if (!config)
return false;
int protected_count = protected_paths ? protected_paths->size : 0;
int size_skipped_count = size_skipped ? size_skipped->size : 0;
int count =
(config->delay_updates ? 1 : 0) + config->basis_count + protected_count + size_skipped_count;
if (count == 0)
return true;
out->entries = calloc((size_t)count, sizeof(DeleteSkipEntry));
if (!out->entries)
return false;
if (basis_root_relative && config->basis_count > 0) {
out->owned_prefixes = calloc((size_t)config->basis_count, sizeof(char*));
if (!out->owned_prefixes) {
free(out->entries);
out->entries = NULL;
return false;
}
out->owned_count = config->basis_count;
}
int idx = 0;
if (config->delay_updates) {
out->entries[idx].prefix = DELAY_UPDATES_STAGING_DIR;
out->entries[idx].top_level_only = true;
idx++;
}
for (int i = 0; i < config->basis_count; i++) {
const char* prefix = config->basis_dirs[i].path;
if (basis_root_relative) {
/* An absolute basis outside the receive root is unreachable by this walk,
so it contributes no protection prefix (and no slot). */
char* relative = delete_basis_relative(config, config->basis_dirs[i].path);
if (!relative)
continue;
out->owned_prefixes[i] = relative;
prefix = relative;
}
out->entries[idx].prefix = prefix;
out->entries[idx].top_level_only = false;
idx++;
}
for (int i = 0; i < protected_count; i++) {
out->entries[idx].prefix = (const char*)protected_paths->items[i];
out->entries[idx].top_level_only = false;
idx++;
}
for (int i = 0; i < size_skipped_count; i++) {
out->entries[idx].prefix = (const char*)size_skipped->items[i];
out->entries[idx].top_level_only = false;
idx++;
}
out->count = idx;
return true;
}
void delete_skips_free(DeleteSkipSet* set) {
if (!set)
return;
if (set->owned_prefixes) {
for (int i = 0; i < set->owned_count; i++)
free(set->owned_prefixes[i]);
}
free(set->owned_prefixes);
free(set->entries);
set->entries = NULL;
set->owned_prefixes = NULL;
set->count = 0;
set->owned_count = 0;
}
+151
View File
@@ -0,0 +1,151 @@
#ifndef DELETE_H
#define DELETE_H
#include "array_list.h"
#include "config.h"
#include <stdbool.h>
#include <stddef.h>
/* Delete engine.
*
* This module owns destination-relative delete traversal: the ordered directory
* walker that reproduces rsync's extraneous-entry order, the skip-prefix
* protection set shared by every delete pass, and the read-only enumeration
* that mirrors the walker for -n/--dry-run. The budgeted manifest commit
* (delete_commit.c) and the per-directory delete plans (delete_plan.c) are
* built on the primitives exported here. */
/* Result of a bounded extra-file deletion run. */
typedef enum {
/* Every extra entry was removed (or there were none). */
DELETE_WALK_OK = 0,
/* The numeric cap for this run was reached before every extra was removed.
The walker removed exactly the entries the cap allowed and skipped (without
removing) the rest, matching rsync's partial --max-delete behavior. */
DELETE_WALK_LIMIT_REACHED,
/* A traversal or unlink failure aborted the deletion (partial removal is
possible, mirroring the delete pass). */
DELETE_WALK_ERROR
} DeleteWalkResult;
/* One protected entry for the delete walker. When top_level_only is true the
prefix is skipped only as a DIRECT child of dest_root (the --delay-updates
staging directory, which must not hide genuine extras inside a nested
destination directory that happens to share the staging name); otherwise the
prefix is skipped at any depth (the --compare-dest/--copy-dest/--link-dest
basis trees, and the sender-side protected filter-excluded prefixes, which
are never destination content). */
typedef struct {
const char* prefix;
bool top_level_only;
} DeleteSkipEntry;
/* A built skip-prefix set. `entries`/`count` are what path_under_skip_prefix()
consumes. `owned_prefixes` holds any prefix strings the builder had to
allocate (root-relative basis-dir conversions); it is NULL when every prefix
is borrowed from the config or the caller's lists. Release with
delete_skips_free(). */
typedef struct {
DeleteSkipEntry* entries;
char** owned_prefixes;
int count;
int owned_count;
} DeleteSkipSet;
/* True when child_rel is, or lies below, one of the protected entries (a prefix
"a" protects "a" and "a/b/c" but not "ab"; top_level_only entries protect
only DIRECT children of the destination root, i.e. child_rel has no '/'). */
bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSkipEntry* skips,
int skip_count);
/* One destination-directory entry collected up front so the delete walkers can
reproduce rsync's traversal order instead of readdir() order. rsync processes
a directory's extraneous subdirectories first (descending name, depth-first),
then its extraneous files (descending name), and only afterwards descends into
its kept subdirectories (ascending name). */
typedef struct {
char* name;
bool is_dir;
} DeleteDirEntry;
/* Collect the entries of the directory open on `dirfd` (excluding "." and ".."),
stat'ing each with AT_SYMLINK_NOFOLLOW. On success *out is a malloc'd array of
*count entries whose names the caller frees with delete_dir_entries_free().
Returns false on an allocation/readdir failure; a vanished entry (ENOENT) is
skipped, any other stat failure is reported through *operation_ok while the
walk continues. */
bool delete_dir_entries_collect(int dirfd, DeleteDirEntry** out, size_t* count, bool* operation_ok);
void delete_dir_entries_free(DeleteDirEntry* entries, size_t count);
/* Sort comparators: `_desc` orders subdirectories before files and each group by
descending name (rsync's extraneous-entry order); `_asc` orders plain ascending
name (rsync's kept-subdirectory order). */
int delete_dir_entry_cmp_desc(const void* a, const void* b);
int delete_dir_entry_cmp_asc(const void* a, const void* b);
/* Remove files/dirs/symlinks under dest_root that are not listed in manifest
without ever descending into a protected prefix (see DeleteSkipEntry). When
`synced_dirs` is non-NULL, extras are only removed directly inside a directory
whose destination-relative path is an exact entry in that list (the receive
root is the "." sentinel); directories outside the synchronized set are still
descended into so kept content below a listed directory is preserved, but
nothing in them is removed. A NULL `synced_dirs` keeps the legacy behavior of
treating the whole destination tree as deletable. `max_delete` caps the
number of removed entries (SIZE_MAX = unlimited): the walker removes up to the
cap and returns DELETE_WALK_LIMIT_REACHED when more extras remained.
`deleted_out`/`skipped_out` optionally receive the number of entries removed
and the number skipped because of the cap. */
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, size_t* deleted_out,
size_t* skipped_out);
/* Optional per-deletion observer: called for each destination-relative path
actually removed (a file, symlink, or directory), in removal order, so the
receiver can stream rsync's `--info=del`/`--info=remove` lines. */
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
* observer; the observer is invoked only for entries truly removed. When
* `protect_rules` is non-NULL its receiver-side verdict is evaluated for every
* candidate extra: a first-match PROTECT leaves the entry (and, for a
* directory, its whole subtree) in place, while RISK/NONE fall through to the
* ordinary skip-prefix/keep-set logic. */
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer,
void* observer_context);
/* Read-only companion to delete_extras_limited: walk the destination exactly as
the delete pass would and APPEND (strdup'd) destination-relative paths that
WOULD be removed, without touching disk. Used for -n/--dry-run --delete
would-delete reporting. Returns true on a clean walk; the caller owns the
strings appended to `out` and receives their count in *count_out. */
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out);
bool delete_extras(const char* dest_root, const ArrayList* manifest);
/* Build the delete walk's skip-prefix set from the config's --delay-updates
staging directory, its --compare-dest/--copy-dest/--link-dest basis dirs, and
the caller-supplied protection lists, in that order. `protected_paths` and
`size_skipped` are borrowed (may be NULL); every entry in them is protected at
any depth. The staging directory is protected only as a DIRECT child of the
receive root. `basis_root_relative` selects how a basis path becomes a
prefix: true converts an absolute path under the receive root to its
root-relative form (the whole-tree commit walk; an unreachable path
contributes no slot), false keeps the configured path verbatim (the
per-directory plan walk). On success the caller releases `*out` with
delete_skips_free(); returns false on allocation failure. */
bool delete_skips_build(const Config* config, const ArrayList* protected_paths,
const ArrayList* size_skipped, bool basis_root_relative,
DeleteSkipSet* out);
void delete_skips_free(DeleteSkipSet* set);
/* Convert one basis-directory path to the receive-root-relative protection
prefix the delete walker uses (NULL when it lies outside the root). Exposed
for unit tests of the root-of-"/" and normalization edge cases. */
char* delete_basis_relative(const Config* config, const char* path);
#endif
+40 -187
View File
@@ -126,38 +126,6 @@ typedef struct {
bool limit_hit;
} DeleteBudgetState;
/* Build the delete-walk protection prefix for one basis directory. The walker
compares paths relative to the receive root, so a relative entry is already
in the right form; an absolute entry that lies below the root is converted to
its root-relative form, and one outside the root returns NULL (the walk
cannot reach it, and it is not protected data beneath the root). Exposed so
tests can exercise the root-of-"/" child mapping directly. */
char* file_receive_basis_delete_relative(const Config* config, const char* path) {
if (!path)
return NULL;
if (path[0] != '/')
return str_dup(path);
const char* root = config->receive_root_directory;
if (!root || root[0] != '/')
return NULL;
size_t root_len = strlen(root);
while (root_len > 1 && root[root_len - 1] == '/')
root_len--;
if (strncmp(path, root, root_len) != 0)
return NULL;
if (root_len == 1) {
/* `root` is "/" (the only single-character absolute root): every absolute
path is below it, and the child relative form is everything after the
leading '/'. */
if (path[1] == '\0')
return NULL; /* identical to the root, not a child */
return str_dup(path + 1);
}
if (path[root_len] != '/')
return NULL; /* identical or a sibling sharing a name prefix */
return str_dup(path + root_len + 1);
}
/* Remove every destination entry under the receive root that is not in the
keep-set, bounded by the shared budget (a smaller client --max-delete=NUM
replaces the server hard bound; rsync deletes up to the bound and skips the
@@ -168,61 +136,19 @@ char* file_receive_basis_delete_relative(const Config* config, const char* path)
alternate basis directories are never destination content and are skipped at
any depth. Returns true unless a traversal/unlink error aborted the walk;
the budget's limit_hit/skipped fields report a cap-stopped run. */
static bool delete_extras_budgeted_observed(const Config* config, DeleteManifest* manifest,
static bool delete_extras_budgeted_observed(const Config* config, const DeleteManifest* manifest,
DeleteBudgetState* budget, DeletePathObserver observer,
void* observer_context) {
if (!config || !manifest || !manifest->keeps)
return false;
fprintf(stderr, "Deleting files not in manifest...\n");
/* Protected entries:
- the --delay-updates staging name, protected only as a DIRECT child of the
receive root (a nested destination directory that happens to be named
.fastsync-stage is ordinary content);
- alternate basis directories (--compare-dest / --copy-dest / --link-dest)
at any depth: they are extra comparison snapshots the user pointed at,
not destination content, and deleting them would destroy the very files a
--link-dest run just linked into place;
- the sender-side protected prefixes (source paths excluded by filters and
paths pruned by --max-size/--min-size), at any depth, so their destination
mirror survives --delete unless --delete-excluded opts back into removing
the filter-excluded ones (size-pruned entries are always protected). */
int skip_count = (config->delay_updates ? 1 : 0) + config->basis_count +
(manifest->protected ? manifest->protected->size : 0);
DeleteSkipEntry* skips = NULL;
char** owned_prefixes = NULL;
int used = 0;
if (skip_count > 0) {
skips = calloc((size_t)skip_count, sizeof(DeleteSkipEntry));
owned_prefixes = calloc((size_t)config->basis_count, sizeof(char*));
if (!skips || (config->basis_count > 0 && !owned_prefixes)) {
free(skips);
free(owned_prefixes);
return false;
}
int idx = 0;
if (config->delay_updates) {
skips[idx].prefix = DELAY_UPDATES_STAGING_DIR;
skips[idx].top_level_only = true;
idx++;
}
for (int i = 0; i < config->basis_count; i++) {
/* An absolute basis outside the receive root is unreachable by this walk,
so it contributes no protection prefix (and no slot). */
char* prefix = file_receive_basis_delete_relative(config, config->basis_dirs[i].path);
if (!prefix)
continue;
owned_prefixes[i] = prefix;
skips[idx].prefix = prefix;
skips[idx].top_level_only = false;
idx++;
}
for (int i = 0; i < manifest->protected->size; i++) {
skips[idx].prefix = (const char*)manifest->protected->items[i];
skips[idx].top_level_only = false;
idx++;
}
used = idx;
}
/* Protected entries: the --delay-updates staging name (only as a DIRECT child
of the receive root), the alternate basis directories and the sender-side
protected prefixes (filter-excluded and size-pruned source mirrors), all at
any depth. See delete_skips_build(). */
DeleteSkipSet skips;
if (!delete_skips_build(config, manifest->protected, NULL, true, &skips))
return false;
/* Clamp rather than subtract: an accounting bug where deleted already exceeds
max_delete must never underflow into an effectively unlimited budget. */
size_t remaining;
@@ -235,14 +161,9 @@ static bool delete_extras_budgeted_observed(const Config* config, DeleteManifest
size_t deleted = 0;
size_t skipped = 0;
DeleteWalkResult result = delete_extras_limited_observed(
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips, used,
config->protect_rules, &deleted, &skipped, observer, observer_context);
if (owned_prefixes) {
for (int i = 0; i < config->basis_count; i++)
free(owned_prefixes[i]);
}
free(owned_prefixes);
free(skips);
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips.entries,
skips.count, config->protect_rules, &deleted, &skipped, observer, observer_context);
delete_skips_free(&skips);
budget->deleted += deleted;
budget->skipped += skipped;
if (result == DELETE_WALK_LIMIT_REACHED) {
@@ -256,7 +177,7 @@ static bool delete_extras_budgeted_observed(const Config* config, DeleteManifest
return true;
}
static bool delete_extras_budgeted(const Config* config, DeleteManifest* manifest,
static bool delete_extras_budgeted(const Config* config, const DeleteManifest* manifest,
DeleteBudgetState* budget) {
return delete_extras_budgeted_observed(config, manifest, budget, NULL, NULL);
}
@@ -294,7 +215,8 @@ static void prefixed_delete_observer(void* context, const char* rel) {
--max-delete budget: once it is exhausted the remaining requests are skipped
and counted. Returns false only on a genuine error (a confinement failure on
a validated path or an I/O error), which fails the run. */
static bool delete_missing_args_budgeted_observed(const Config* config, DeleteManifest* manifest,
static bool delete_missing_args_budgeted_observed(const Config* config,
const DeleteManifest* manifest,
DeleteBudgetState* budget,
DeletePathObserver observer,
void* observer_context) {
@@ -303,35 +225,12 @@ static bool delete_missing_args_budgeted_observed(const Config* config, DeleteMa
if (!manifest->missing || manifest->missing->size == 0)
return true;
fprintf(stderr, "Deleting destination mirrors of missing source arguments...\n");
int skip_count = (config->delay_updates ? 1 : 0) + config->basis_count;
DeleteSkipEntry* skips = NULL;
char** owned_prefixes = NULL;
int used = 0;
if (skip_count > 0) {
skips = calloc((size_t)skip_count, sizeof(DeleteSkipEntry));
owned_prefixes = calloc((size_t)config->basis_count, sizeof(char*));
if (!skips || (config->basis_count > 0 && !owned_prefixes)) {
free(skips);
free(owned_prefixes);
return false;
}
int idx = 0;
if (config->delay_updates) {
skips[idx].prefix = DELAY_UPDATES_STAGING_DIR;
skips[idx].top_level_only = true;
idx++;
}
for (int i = 0; i < config->basis_count; i++) {
char* prefix = file_receive_basis_delete_relative(config, config->basis_dirs[i].path);
if (!prefix)
continue;
owned_prefixes[i] = prefix;
skips[idx].prefix = prefix;
skips[idx].top_level_only = false;
idx++;
}
used = idx;
}
/* The staging directory and basis snapshots stay protected exactly as in the
extras walker (the missing-args path overrides the ordinary protected
prefixes, so those are not passed here). */
DeleteSkipSet skips;
if (!delete_skips_build(config, NULL, NULL, true, &skips))
return false;
bool ok = true;
for (int i = 0; i < manifest->missing->size; i++) {
const char* rel = (const char*)manifest->missing->items[i];
@@ -343,7 +242,7 @@ static bool delete_missing_args_budgeted_observed(const Config* config, DeleteMa
continue;
}
bool at_root = strchr(rel, '/') == NULL;
if (path_under_skip_prefix(rel, at_root, skips, used)) {
if (path_under_skip_prefix(rel, at_root, skips.entries, skips.count)) {
char* escaped = output_escape(rel, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING,
"missing-args path '%s' is protected (staging directory or basis snapshot); "
@@ -472,96 +371,49 @@ static bool delete_missing_args_budgeted_observed(const Config* config, DeleteMa
if (!ok)
break;
}
if (owned_prefixes) {
for (int i = 0; i < config->basis_count; i++)
free(owned_prefixes[i]);
}
free(owned_prefixes);
free(skips);
delete_skips_free(&skips);
return ok;
}
/* Public wrappers used outside the commit path (and by unit tests): no
--max-delete budget. */
bool manifest_would_delete_list(const Config* config, DeleteManifest* manifest, ArrayList* out,
size_t* count_out) {
bool manifest_would_delete_list(const Config* config, const DeleteManifest* manifest,
ArrayList* out, size_t* count_out) {
if (count_out)
*count_out = 0;
if (!config || !manifest || !manifest->keeps || !out)
return false;
int skip_count = (config->delay_updates ? 1 : 0) + config->basis_count +
(manifest->protected ? manifest->protected->size : 0);
DeleteSkipEntry* skips = NULL;
char** owned_prefixes = NULL;
int used = 0;
if (skip_count > 0) {
skips = calloc((size_t)skip_count, sizeof(DeleteSkipEntry));
owned_prefixes = calloc((size_t)config->basis_count, sizeof(char*));
if (!skips || (config->basis_count > 0 && !owned_prefixes)) {
free(skips);
free(owned_prefixes);
return false;
}
int idx = 0;
if (config->delay_updates) {
skips[idx].prefix = DELAY_UPDATES_STAGING_DIR;
skips[idx].top_level_only = true;
idx++;
}
for (int i = 0; i < config->basis_count; i++) {
/* Normalize exactly like the real commit path: a relative entry is
already root-relative, an absolute one inside the receive root is
converted, and one outside contributes no protection prefix. */
char* prefix = file_receive_basis_delete_relative(config, config->basis_dirs[i].path);
if (!prefix)
continue;
owned_prefixes[i] = prefix;
skips[idx].prefix = prefix;
skips[idx].top_level_only = false;
idx++;
}
for (int i = 0; i < manifest->protected->size; i++) {
skips[idx].prefix = (const char*)manifest->protected->items[i];
skips[idx].top_level_only = false;
idx++;
}
used = idx;
}
DeleteSkipSet skips;
if (!delete_skips_build(config, manifest->protected, NULL, true, &skips))
return false;
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
skips, used, config->protect_rules, out, count_out);
if (owned_prefixes) {
for (int i = 0; i < config->basis_count; i++)
free(owned_prefixes[i]);
}
free(owned_prefixes);
free(skips);
skips.entries, skips.count, config->protect_rules, out, count_out);
delete_skips_free(&skips);
return ok;
}
bool manifest_delete_extras(const Config* config, DeleteManifest* manifest) {
bool manifest_delete_extras(const Config* config, const DeleteManifest* manifest) {
DeleteBudgetState budget = {
.max_delete = SIZE_MAX, .deleted = 0, .skipped = 0, .limit_hit = false};
return delete_extras_budgeted(config, manifest, &budget);
}
bool manifest_delete_missing_args(const Config* config, DeleteManifest* manifest) {
bool manifest_delete_missing_args(const Config* config, const DeleteManifest* manifest) {
DeleteBudgetState budget = {
.max_delete = SIZE_MAX, .deleted = 0, .skipped = 0, .limit_hit = false};
return delete_missing_args_budgeted_observed(config, manifest, &budget, NULL, NULL);
}
bool manifest_delete_missing_args_limited(const Config* config, DeleteManifest* manifest,
bool manifest_delete_missing_args_limited(const Config* config, const DeleteManifest* manifest,
size_t max_delete, size_t* deleted, size_t* skipped,
bool* limit_hit) {
return manifest_delete_missing_args_limited_observed(config, manifest, max_delete, deleted,
skipped, limit_hit, NULL, NULL);
}
bool manifest_delete_missing_args_limited_observed(const Config* config, DeleteManifest* manifest,
size_t max_delete, size_t* deleted,
size_t* skipped, bool* limit_hit,
DeletePathObserver observer,
void* observer_context) {
bool manifest_delete_missing_args_limited_observed(
const Config* config, const DeleteManifest* manifest, size_t max_delete, size_t* deleted,
size_t* skipped, bool* limit_hit, DeletePathObserver observer, void* observer_context) {
DeleteBudgetState budget = {
.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
bool ok =
@@ -582,17 +434,18 @@ bool manifest_delete_missing_args_limited_observed(const Config* config, DeleteM
removal fail). The ordinary extras walk then runs when --delete is active.
Both draw from one --max-delete budget; the result reports a cap-stopped
(partial) commit distinctly so the client can exit 25 like rsync. */
DeleteCommitResult manifest_delete_all(const Config* config, DeleteManifest* manifest) {
DeleteCommitResult manifest_delete_all(const Config* config, const DeleteManifest* manifest) {
return manifest_delete_all_counted(config, manifest, NULL);
}
DeleteCommitResult manifest_delete_all_counted(const Config* config, DeleteManifest* manifest,
DeleteCommitResult manifest_delete_all_counted(const Config* config, const DeleteManifest* manifest,
size_t* deleted) {
return manifest_delete_all_observed(config, manifest, deleted, NULL, NULL);
}
DeleteCommitResult manifest_delete_all_observed(const Config* config, DeleteManifest* manifest,
size_t* deleted, DeletePathObserver observer,
DeleteCommitResult manifest_delete_all_observed(const Config* config,
const DeleteManifest* manifest, size_t* deleted,
DeletePathObserver observer,
void* observer_context) {
if (deleted)
*deleted = 0;
+14 -19
View File
@@ -3,7 +3,7 @@
#include "array_list.h"
#include "config.h"
#include "utils.h"
#include "delete.h"
#include <stdbool.h>
/* Delete-commit module: delete-manifest receive plus the budgeted extras and
@@ -44,7 +44,7 @@ DeleteManifest* receive_manifest_entries(int fd);
protected-prefix skips). `--max-delete` and `--force` are honored here. The
caller decides WHEN to run it based on the negotiated delete timing. Returns
false (and the transfer fails) when the deletion cannot be committed. */
bool manifest_delete_extras(const Config* config, DeleteManifest* manifest);
bool manifest_delete_extras(const Config* config, const DeleteManifest* manifest);
/* --delete-missing-args exact-path deletions: remove each destination mirror
in `manifest->missing` (never blocked by the protected prefixes, staging dir
and basis dirs excluded). A regular file/symlink is unlinked; an empty
@@ -53,22 +53,20 @@ bool manifest_delete_extras(const Config* config, DeleteManifest* manifest);
parity). A missing path is a no-op. Returns false only on a genuine
confinement or I/O error (the run then fails); tolerated per-path cases are
reported and skipped. */
bool manifest_delete_missing_args(const Config* config, DeleteManifest* manifest);
bool manifest_delete_missing_args(const Config* config, const DeleteManifest* manifest);
/* Budgeted form of manifest_delete_missing_args for the per-directory delete
session: each removed mirror draws from `max_delete` (SIZE_MAX = unlimited)
and the tallies are accumulated into `*deleted`/`*skipped`. `*limit_hit` is set
when the budget stopped the pass with entries left over. Returns false only
on a genuine deletion error. */
bool manifest_delete_missing_args_limited(const Config* config, DeleteManifest* manifest,
bool manifest_delete_missing_args_limited(const Config* config, const DeleteManifest* manifest,
size_t max_delete, size_t* deleted, size_t* skipped,
bool* limit_hit);
/* Observer-aware form of manifest_delete_missing_args_limited: `observer` (may
be NULL) is invoked for every destination-relative path truly removed. */
bool manifest_delete_missing_args_limited_observed(const Config* config, DeleteManifest* manifest,
size_t max_delete, size_t* deleted,
size_t* skipped, bool* limit_hit,
DeletePathObserver observer,
void* observer_context);
bool manifest_delete_missing_args_limited_observed(
const Config* config, const DeleteManifest* manifest, size_t max_delete, size_t* deleted,
size_t* skipped, bool* limit_hit, DeletePathObserver observer, void* observer_context);
/* Outcome of committing a delete manifest. LIMIT_REACHED reports rsync's
partial --max-delete result: the budget allowed some deletions and the rest
were skipped (the run still stores all file data but the client exits 25). */
@@ -84,15 +82,16 @@ typedef enum {
share one --max-delete budget. Returns DELETE_COMMIT_OK when nothing was to
do or everything committed, DELETE_COMMIT_LIMIT_REACHED when the budget
stopped part of the work, or DELETE_COMMIT_ERROR on a genuine failure. */
DeleteCommitResult manifest_delete_all(const Config* config, DeleteManifest* manifest);
DeleteCommitResult manifest_delete_all(const Config* config, const DeleteManifest* manifest);
/* Like manifest_delete_all, but reports how many destination entries the commit
removed (for the end-of-transfer wire stats). `deleted` may be NULL. */
DeleteCommitResult manifest_delete_all_counted(const Config* config, DeleteManifest* manifest,
DeleteCommitResult manifest_delete_all_counted(const Config* config, const DeleteManifest* manifest,
size_t* deleted);
/* Observer-aware form of manifest_delete_all_counted: `observer` (may be NULL)
is invoked for every destination-relative path truly removed. */
DeleteCommitResult manifest_delete_all_observed(const Config* config, DeleteManifest* manifest,
size_t* deleted, DeletePathObserver observer,
DeleteCommitResult manifest_delete_all_observed(const Config* config,
const DeleteManifest* manifest, size_t* deleted,
DeletePathObserver observer,
void* observer_context);
/* -n/--dry-run --delete would-delete reporting: walk the destination exactly as
@@ -100,11 +99,7 @@ DeleteCommitResult manifest_delete_all_observed(const Config* config, DeleteMani
WOULD be removed to `out`, without touching disk. Uses the same staging-dir,
basis-dir and protected-prefix skips as the real commit. Returns true on a
clean walk; `*count_out` receives the number of paths appended. */
bool manifest_would_delete_list(const Config* config, DeleteManifest* manifest, ArrayList* out,
size_t* count_out);
/* Convert one basis-directory path to the receive-root-relative protection
prefix the delete walker uses (NULL when it lies outside the root). Exposed
for unit tests of the root-of-"/" and normalization edge cases. */
char* file_receive_basis_delete_relative(const Config* config, const char* path);
bool manifest_would_delete_list(const Config* config, const DeleteManifest* manifest,
ArrayList* out, size_t* count_out);
#endif
+11 -38
View File
@@ -2,6 +2,7 @@
#include "charset.h"
#include "delay_updates.h"
#include "delete.h"
#include "file.h"
#include "log.h"
#include "utils.h"
@@ -601,9 +602,8 @@ static int open_plan_dir(const Config* config, const char* dir) {
return fd;
}
typedef struct PlanSkips {
DeleteSkipEntry* entries;
int count;
typedef struct {
DeleteSkipSet set;
/* Receiver-side delete-protection rules received on the config frame (NULL
when the sender sent none). Evaluated per extra so a protect/risk rule is
honored under --delete-during/--delete-delay exactly like the whole-tree
@@ -613,39 +613,12 @@ typedef struct PlanSkips {
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
PlanSkips* out) {
out->entries = NULL;
out->count = 0;
out->protect_rules = config->protect_rules;
int count = (config->delay_updates ? 1 : 0) + config->basis_count +
session->protected_prefixes->size + session->size_skipped->size;
if (count == 0)
return true;
out->entries = calloc((size_t)count, sizeof(DeleteSkipEntry));
if (!out->entries)
return false;
int idx = 0;
if (config->delay_updates) {
out->entries[idx].prefix = DELAY_UPDATES_STAGING_DIR;
out->entries[idx].top_level_only = true;
idx++;
}
for (int i = 0; i < config->basis_count; i++) {
out->entries[idx].prefix = config->basis_dirs[i].path;
out->entries[idx].top_level_only = false;
idx++;
}
for (int i = 0; i < session->protected_prefixes->size; i++) {
out->entries[idx].prefix = (const char*)session->protected_prefixes->items[i];
out->entries[idx].top_level_only = false;
idx++;
}
for (int i = 0; i < session->size_skipped->size; i++) {
out->entries[idx].prefix = (const char*)session->size_skipped->items[i];
out->entries[idx].top_level_only = false;
idx++;
}
out->count = idx;
return true;
/* The per-directory plan walk keeps each basis path verbatim (it does not
convert an absolute under-root path to its root-relative form, unlike the
whole-tree commit walk). */
return delete_skips_build(config, session->protected_prefixes, session->size_skipped, false,
&out->set);
}
static bool budget_available(const DeletePlanSession* session) {
@@ -796,7 +769,7 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
operation_ok = false;
continue;
}
if (path_under_skip_prefix(child_rel, at_root, skips->entries, skips->count)) {
if (path_under_skip_prefix(child_rel, at_root, skips->set.entries, skips->set.count)) {
shielded[i] = true;
local_survives = true;
free(child_rel);
@@ -883,7 +856,7 @@ static bool apply_plan_dir(DeletePlanSession* session, const Config* config, con
bool survives = false;
bool ok = process_children(dirfd, dir, dirs, files, strcmp(dir, ".") == 0, false, &skips, session,
&survives);
free(skips.entries);
delete_skips_free(&skips.set);
close(dirfd);
if (!ok)
log_message(LOG_LEVEL_ERROR, "deletion failed while removing extraneous files");
@@ -1024,7 +997,7 @@ static bool apply_deferred_path(DeletePlanSession* session, const Config* config
}
bool survives = false;
bool ok = process_children(dirfd, rel, NULL, NULL, false, true, &skips, session, &survives);
free(skips.entries);
delete_skips_free(&skips.set);
close(dirfd);
if (!ok) {
close(parent_fd);
+1
View File
@@ -3,6 +3,7 @@
#include "array_list.h"
#include "config.h"
#include "delete.h"
#include "file_receive.h"
#include "protocol.h"
#include "utils.h"
+31 -24
View File
@@ -1182,21 +1182,24 @@ int file_open_temp_dir(const char* dir_path) {
* destination file) and best-effort: a per-attribute or privilege failure is
* logged and skipped, never fatal. */
static void restore_extra_fd(int fd, const FileMetadata* metadata, const FileXattrList* xattrs,
bool fake_super, FileAttrPolicy policy) {
bool fake_super, FileAttrPolicy policy, uint32_t fake_super_rdev_major,
uint32_t fake_super_rdev_minor) {
xattr_apply_fd(fd, xattrs);
if (fake_super && metadata) {
/* Record the ownership that WOULD have been applied: when an explicit
ownership request (--chown/--usermap/--groupmap/--copy-as or -o/-g) is
active, the resolved mapping; otherwise the source's own id. The real
chown is suppressed (identity_apply_ownership early-returns under
--fake-super) so recording never defeats the flag. Mode/mtime are still
replayed (policy-gated) so unprivileged --fake-super keeps working. */
--fake-super) so recording never defeats the flag. The recorded stat is
rsync's format; the permission bits are replayed (policy-gated) so
unprivileged --fake-super keeps working while mtime comes from the
normal metadata path above. */
uint32_t store_uid;
uint32_t store_gid;
identity_resolve_storage_ids((int32_t)metadata->uid, (int32_t)metadata->gid, &store_uid,
&store_gid);
fake_super_store_fd(fd, store_uid, store_gid, (uint32_t)metadata->mode, metadata->mtime_sec,
metadata->mtime_nsec);
fake_super_store_fd(fd, store_uid, store_gid, (uint32_t)metadata->mode, fake_super_rdev_major,
fake_super_rdev_minor);
fake_super_restore_fd(fd, policy);
}
}
@@ -1206,7 +1209,8 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace, bool use_fsync,
const char* temp_dir, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, unsigned* dirs_created, const char* count_floor) {
bool keep_partial, unsigned* dirs_created, const char* count_floor,
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor) {
char* leaf = NULL;
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
if (dirfd < 0)
@@ -1313,7 +1317,8 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
}
}
if (ok)
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, fake_super_rdev_major,
fake_super_rdev_minor);
if (ok && use_fsync)
ok = fsync(fd) == 0;
}
@@ -1431,7 +1436,8 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
}
}
if (ok)
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, fake_super_rdev_major,
fake_super_rdev_minor);
if (ok && use_fsync)
ok = fsync(fd) == 0;
}
@@ -1499,7 +1505,8 @@ file_to_disk_secure_impl(const char* path, const void* data, unsigned long long
"non-atomic copy into the destination directory");
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, update, no_replace, use_fsync, NULL, xattrs, fake_super,
keep_partial, dirs_created, count_floor);
keep_partial, dirs_created, count_floor, fake_super_rdev_major,
fake_super_rdev_minor);
}
return ok;
}
@@ -1509,7 +1516,7 @@ bool file_to_disk_secure(const char* path, const void* data, unsigned long long
FileAttrPolicy policy, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, false, false, false, temp_dir, NULL, false, false, NULL,
NULL);
NULL, 0, 0);
}
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
@@ -1518,7 +1525,7 @@ bool file_to_disk_secure_update(const char* path, const void* data, unsigned lon
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, true, false, false, temp_dir, NULL, false, false, NULL,
NULL);
NULL, 0, 0);
}
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
@@ -1527,7 +1534,7 @@ bool file_to_disk_secure_with_fsync(const char* path, const void* data,
FileAttrPolicy policy, bool use_fsync, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, false, false, use_fsync, temp_dir, NULL, false, false,
NULL, NULL);
NULL, NULL, 0, 0);
}
bool file_to_disk_secure_no_replace(const char* path, const void* data,
@@ -1536,7 +1543,7 @@ bool file_to_disk_secure_no_replace(const char* path, const void* data,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
policy, false, true, false, temp_dir, NULL, false, false, NULL,
NULL);
NULL, 0, 0);
}
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
@@ -1551,19 +1558,19 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
bool fake_super, bool keep_partial, const char* temp_dir) {
return file_to_disk_secure_attrs_counted(path, data, data_size, inplace, sparse, preallocate,
metadata, policy, update, no_replace, use_fsync, xattrs,
fake_super, keep_partial, temp_dir, NULL, NULL);
fake_super, keep_partial, temp_dir, NULL, NULL, 0, 0);
}
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, const char* temp_dir,
unsigned* dirs_created, const char* count_floor) {
bool file_to_disk_secure_attrs_counted(
const char* path, const void* data, unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata, FileAttrPolicy policy, bool update,
bool no_replace, bool use_fsync, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, const char* temp_dir, unsigned* dirs_created, const char* count_floor,
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, update, no_replace, use_fsync, temp_dir, xattrs,
fake_super, keep_partial, dirs_created, count_floor);
fake_super, keep_partial, dirs_created, count_floor,
fake_super_rdev_major, fake_super_rdev_minor);
}
/* Atomic --link-dest install. The destination is replaced (via a temporary
@@ -1693,7 +1700,7 @@ static bool file_copy_basis_stream_impl(const char* path, const char* basis_path
wrote = false;
}
if (wrote)
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
restore_extra_fd(fd, metadata, xattrs, fake_super, policy, 0, 0);
if (wrote && use_fsync)
wrote = fsync(fd) == 0;
if (close(fd) != 0)
@@ -1842,7 +1849,7 @@ static bool file_to_disk_secure_link_impl(const char* path, const char* basis_pa
return true;
return file_to_disk_secure_attrs_counted(
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
xattrs, fake_super, false, temp_dir, dirs_created, count_floor);
xattrs, fake_super, false, temp_dir, dirs_created, count_floor, 0, 0);
}
if (scratch_dirfd >= 0)
+6 -7
View File
@@ -182,13 +182,12 @@ bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
* confined secure walk had to create that lie strictly below `count_floor` (a
* receive-root-relative prefix, or NULL for all). Used to reproduce rsync's
* `Number of created files` directory count on a fresh destination. */
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, const char* temp_dir,
unsigned* dirs_created, const char* count_floor);
bool file_to_disk_secure_attrs_counted(
const char* path, const void* data, unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata, FileAttrPolicy policy, bool update,
bool no_replace, bool use_fsync, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, const char* temp_dir, unsigned* dirs_created, const char* count_floor,
uint32_t fake_super_rdev_major, uint32_t fake_super_rdev_minor);
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
const void* data, unsigned long long data_size,
bool preallocate, const FileMetadata* metadata,
+8
View File
@@ -482,6 +482,14 @@ File* file_receive_symlink(int file_descriptor, const Config* config) {
return NULL;
}
}
/* Symlink xattrs/ACLs (-X/-A) arrive in the same trailing block as the other
entry kinds; the block is present iff use_xattrs (which itself implies
use_metadata, so the metadata frame above is always consumed first). */
if (config && !receive_file_xattrs(file, file_descriptor, config)) {
file_destroy(file);
free(target);
return NULL;
}
file->is_symlink = true;
file->symlink_target = target;
return file;
+203 -45
View File
@@ -2,6 +2,7 @@
#include <ctype.h>
#include <dirent.h>
#include <fcntl.h>
#include <limits.h>
#include <libgen.h>
#include <stdio.h>
#include <stdlib.h>
@@ -198,6 +199,56 @@ static FileSaveResult hardlink_sibling_absent_first(const char* destination_path
return FILE_SAVE_ERROR;
}
/* Resolve a user-supplied --temp-dir against the receive `root`.
*
* A relative, traversal-free name is joined below the root (the historical
* behavior). An absolute path is canonicalized with realpath(3) and accepted
* only when it lies inside the canonicalized receive root; this is the parity
* win over rejecting every absolute path, without weakening the confinement
* invariant: an absolute path that escapes the root (including one reached
* through a symlinked component) is still refused. A `..` component in a
* relative name is likewise refused. The root itself is treated as an
* absolute path free of `..`; its realpath() resolves any symlinks so the
* prefix comparison is against one canonical form.
*
* Logs a clear error on rejection (the scratch dir must stay confined) and
* returns a newly allocated scratch path, or NULL on rejection/allocation
* failure. */
static char* file_save_resolve_temp_dir(const char* root, const char* temp_dir) {
if (temp_dir[0] != '/') {
if (has_path_traversal(temp_dir)) {
log_message(
LOG_LEVEL_ERROR,
"receiver rejected --temp-dir '%s': a '..' component would escape the receive root",
temp_dir);
return NULL;
}
return path_cat(root, temp_dir);
}
char canonical_temp[PATH_MAX];
char canonical_root[PATH_MAX];
if (!realpath(temp_dir, canonical_temp)) {
log_message(LOG_LEVEL_ERROR,
"receiver rejected --temp-dir '%s': could not resolve the absolute path (%s)",
temp_dir, strerror(errno));
return NULL;
}
if (!realpath(root, canonical_root)) {
log_message(LOG_LEVEL_ERROR,
"receiver rejected --temp-dir '%s': could not resolve the receive root (%s)",
temp_dir, strerror(errno));
return NULL;
}
if (strcmp(canonical_root, "/") != 0 && !path_is_within_root(canonical_root, canonical_temp)) {
log_message(LOG_LEVEL_ERROR,
"receiver rejected --temp-dir '%s': an absolute temp dir must be inside the "
"receive root '%s'",
temp_dir, canonical_root);
return NULL;
}
return str_dup(canonical_temp);
}
/* Install a --hard-links/-H sibling: the destination entry is atomically
replaced (temp + rename) with a hard link to the group's first member. The
first member is guaranteed already installed at `hardlink_target` under the
@@ -303,17 +354,13 @@ static FileSaveResult file_save_hardlink_sibling(const char* root_directory, con
free(destination_path);
return absent_result;
}
/* Resolve a relative --temp-dir under the destination root, exactly as the
* primary save path does; an absolute or `..`-escaping value is rejected. */
/* Resolve the --temp-dir under the destination root, exactly as the primary
* save path does: a relative dir joins below the root, an absolute dir is
* accepted only when it canonicalizes inside the root, and any escaping value
* is rejected. */
char* resolved_temp = NULL;
if (cfg->temp_dir) {
if (cfg->temp_dir[0] == '/' || has_path_traversal(cfg->temp_dir)) {
free(content);
free(first_disk);
free(destination_path);
return FILE_SAVE_ERROR;
}
resolved_temp = path_cat(root_directory, cfg->temp_dir);
resolved_temp = file_save_resolve_temp_dir(root_directory, cfg->temp_dir);
if (!resolved_temp) {
free(content);
free(first_disk);
@@ -353,11 +400,14 @@ bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode) {
/* ---- Device/special node RECREATION (--devices/--specials), receiver side ----
*
* Privilege gating: making a real device node requires CAP_MKNOD (root); making
* a FIFO works unprivileged (mkfifo). When the receiver lacks the capability,
* mknodat() fails with EPERM and the entry is SKIPPED with a warning -- the
* whole transfer must NOT abort just because the environment cannot make the
* node. CI runs non-root, so device creation is expected to skip there and
* only a FIFO is honestly assertable unprivileged.
* a FIFO works unprivileged (mkfifo). A device node whose mknodat() fails with
* EPERM/EACCES is a PER-ENTRY failure (rsync parity: rsync reports the mknod
* failure, still transfers the rest, and exits partial, code 23), reported as
* FILE_SAVE_FAILED so the receiver counts it and continues. Only the
* unprivileged FIFO/socket (--specials) path keeps the best-effort skip,
* because those are normally creatable without privilege and a failure there is
* environmental. CI runs non-root, so device creation is expected to fail
* there; only a FIFO is honestly assertable unprivileged.
*
* Confinement: the parent directory is opened fd-relative below the receive
* root (file_open_secure_parent: O_NOFOLLOW, no "..", root-checked) and the
@@ -495,13 +545,32 @@ static FileSaveResult file_save_special_to_disk(const char* root_directory, cons
node_kind, escaped_path ? escaped_path : "<allocation failed>");
free(escaped_path);
} else if (errno == EPERM || errno == EACCES) {
/* Missing CAP_MKNOD / parent write permission: the environment cannot
create the node, so skip instead of failing the whole run. */
char* escaped_path = output_escape(file->path, log_get_8_bit_output());
const char* shown_path = escaped_path ? escaped_path : "<allocation failed>";
if (is_char || is_blk) {
/* rsync parity: a device node that cannot be created (no CAP_MKNOD, or
* super-user activities not permitted) is a per-entry failure. rsync
* logs `mknod ".../node" failed: ...`, still transfers the remaining
* files, and exits partial (23); FastSync logs it, counts it, and
* continues rather than aborting the stream. FIFO/socket creation
* (--specials) keeps the best-effort skip path below. */
log_message(LOG_LEVEL_ERROR,
"cannot create %s %s: %s\n"
" --devices node creation needs privilege (CAP_MKNOD)",
node_kind, shown_path, strerror(errno));
free(escaped_path);
close(parent_fd);
free(leaf);
free(destination);
return FILE_SAVE_FAILED;
}
/* Missing CAP_MKNOD / parent write permission for a FIFO/socket: the
environment cannot create the node, so skip instead of failing the
whole run. */
log_message(LOG_LEVEL_WARNING,
"skipping %s: cannot create %s node (%s)\n"
" --devices/--specials node creation needs privilege (CAP_MKNOD)",
escaped_path ? escaped_path : "<allocation failed>", node_kind, strerror(errno));
" --specials node creation needs privilege (CAP_MKNOD)",
shown_path, node_kind, strerror(errno));
free(escaped_path);
} else {
char* escaped_path = output_escape(file->path, log_get_8_bit_output());
@@ -714,27 +783,85 @@ static FileSaveResult file_save_directory_to_disk(const FileSavePlan* plan, bool
return FILE_SAVE_ERROR;
bool dir_existed = file_path_exists_secure(dir_path);
bool ok = file_ensure_directory_secure(dir_path);
/* One confined, no-follow descriptor drives ownership/mode/xattr/timestamp
application so none of them can follow a same-named symlink planted after
the mkdir. This mirrors the O_DIRECTORY|O_NOFOLLOW fd that
dir_metadata_list_apply() opens for the recursive path; the fd is reached
through the already-confined parent. */
char* leaf = NULL;
int parent_fd = -1;
int dir_fd = -1;
if (ok) {
parent_fd = file_open_secure_parent(dir_path, &leaf, false);
if (parent_fd >= 0)
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
/* P7 Wave E: apply the negotiated ownership to the directory ITSELF (not
just the files inside it). --copy-as and every explicit identity policy
own every entry, so a directory must not keep the receiver's owner while
its children get the policy owner. Applied no-follow on the confined
parent fd after the mkdir; identity_apply_ownership_link() is itself a
no-op unless an identity policy is active. */
parent fd; identity_apply_ownership_link() is itself a no-op unless an
identity policy is active. Ownership runs before the mode because a chown
clears setuid/setgid. A failed REQUIRED --copy-as ownership fails the
entry; every other policy stays best-effort. */
if (ok && file->metadata && identity_active_enabled()) {
char* leaf = NULL;
int parent_fd = file_open_secure_parent(dir_path, &leaf, false);
if (parent_fd >= 0) {
if (!identity_apply_ownership_link(parent_fd, leaf, (int32_t)file->metadata->uid,
(int32_t)file->metadata->gid))
ok = false;
close(parent_fd);
} else if (identity_copy_as_active()) {
/* The directory exists (ok) but its required --copy-as ownership could
not be applied because the confined parent could not be opened. */
ok = false;
}
free(leaf);
} else if (ok && identity_copy_as_active()) {
ok = false;
}
/* The final source MODE is deliberately NOT applied inline. A restrictive
source mode (for example 0555) would make the directory unwritable before
its children are created, so a non-root receiver fails each child with
EACCES. The receiver feeds every is_dir entry -- including this explicit
--dirs/STATUS_MKDIR one -- into the deferred DirTimeList, and
dir_metadata_list_apply() stamps the exact mode once the whole transfer has
finished, exactly as it does for the recursive path. Leaving the directory
at its creation mode keeps it writable for the children until then.
The xattrs below are still applied inline so a direct
file_save_to_disk_full() caller (which has no deferred pass) also gets
--dirs directory xattrs. Because the inline mode is absent, the inline
order here is ownership, then xattrs, then timestamps; the recursive path
(which DOES apply a mode) orders them times, mode, xattrs -- the difference
is intentional, and the deferred pass re-stamps mode and xattrs last.
Best-effort: a per-attribute failure is logged and skipped by
xattr_apply_fd(), never fatal. */
if (ok && plan->config && plan->config->use_xattrs && dir_fd >= 0 && file->xattrs)
xattr_apply_fd(dir_fd, file->xattrs);
/* Timestamps last so no later inline ownership/xattr change is mistaken for a
content update; the deferred pass re-stamps them after every child write.
-J/--omit-dir-times suppresses the directory mtime; --atimes/-U applies
only when the source atime is valid, exactly as the recursive path. */
if (ok && file->metadata && plan->config && plan->config->preserve_times &&
!plan->config->omit_dir_times) {
struct timespec times[2] = {
{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
{.tv_sec = file->metadata->mtime_sec, .tv_nsec = file->metadata->mtime_nsec}};
if (plan->config->preserve_atimes && file->metadata->atime_valid) {
times[0].tv_sec = file->metadata->atime_sec;
times[0].tv_nsec = file->metadata->atime_nsec;
}
if (parent_fd >= 0 && utimensat(parent_fd, leaf, times, AT_SYMLINK_NOFOLLOW) != 0) {
int saved_errno = errno;
char* escaped_path = output_escape(dir_path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Failed to set directory timestamps on %s: %s",
escaped_path ? escaped_path : "<allocation failed>", strerror(saved_errno));
free(escaped_path);
}
}
if (dir_fd >= 0)
close(dir_fd);
if (parent_fd >= 0)
close(parent_fd);
free(leaf);
free(dir_path);
if (ok && created && !dir_existed)
*created = true;
@@ -796,6 +923,21 @@ static FileSaveResult file_save_symlink_to_disk(const FileSavePlan* plan, bool*
ok = file_restore_symlink_metadata(link_path, file->metadata, link_policy,
config->omit_link_times);
}
/* -X/-A: apply the symlink's OWN xattrs with a no-follow primitive. The
confined parent directory is the anchor and the final component is applied
with lsetxattr, so the referent is never touched. Best-effort: on Linux
the VFS refuses xattrs on symlinks, so this is normally a no-op. Hoist the
empty-list check so the common Linux case (NULL/empty xattrs) does not pay
an open/close of the parent per symlink. */
if (ok && config && config->use_xattrs && file->xattrs && file->xattrs->count > 0) {
char* leaf = NULL;
int parent_fd = file_open_secure_parent(link_path, &leaf, false);
if (parent_fd >= 0) {
xattr_apply_path_nofollow(parent_fd, leaf, file->xattrs, config->preserve_acls);
close(parent_fd);
}
free(leaf);
}
if (ok && created && !link_existed)
*created = true;
free(link_path);
@@ -810,6 +952,14 @@ static bool file_save_try_special_dispatch(const FileSavePlan* plan, bool* creat
/* Device/special node (--devices/--specials): recreate the node instead of
writing content (privilege-gated, confined, rdev-validated). */
if (file->is_special) {
/* Under --fake-super rsync never mknod()s a device: it writes a regular
empty file and records the real rdev in user.rsync.%stat. Fall through to
the ordinary writer so the device round-trips (its S_IFMT mode bits and
rdev are parked in the record). Without --fake-super the node is
recreated (or, when privilege is refused, handled per-entry). */
mode_t special_mode = file->metadata ? file->metadata->mode : 0;
if (config && config->fake_super && (S_ISCHR(special_mode) || S_ISBLK(special_mode)))
return false;
*out = file_save_special_to_disk(plan->root_directory, file, config, created);
return true;
}
@@ -838,17 +988,17 @@ static bool file_save_try_special_dispatch(const FileSavePlan* plan, bool* creat
and disk paths. Returns false on an invalid/escaping option or an
allocation failure (the caller routes to the cleanup epilogue). */
static bool file_save_resolve_paths(FileSavePlan* plan) {
/* These options arrive from the client. --backup-dir, --partial-dir and
--temp-dir are names below the server root, never independent filesystem
roots: an absolute or `..`-escaping value is rejected outright (rsync's
daemon confines temp-dir to the module the same way). A relative temp dir
is resolved under the receive root below; if that resolution still lands on
a different filesystem than the destination the install falls back to a
non-atomic copy (see file_to_disk_secure_impl), never an abort. */
/* These options arrive from the client. --backup-dir and --partial-dir are
names below the server root, never independent filesystem roots: an
absolute or `..`-escaping value is rejected outright. --temp-dir is
resolved by file_save_resolve_temp_dir below: a relative name joins below
the root, an absolute name is accepted only when it canonicalizes inside
the root, and any escaping value is rejected. If the resolved scratch dir
still lands on a different filesystem than the destination the install
falls back to a non-atomic copy (see file_to_disk_secure_impl), never an
abort. */
if ((plan->backup_dir && (plan->backup_dir[0] == '/' || has_path_traversal(plan->backup_dir))) ||
(plan->partial_dir &&
(plan->partial_dir[0] == '/' || has_path_traversal(plan->partial_dir))) ||
(plan->temp_dir && (plan->temp_dir[0] == '/' || has_path_traversal(plan->temp_dir))))
(plan->partial_dir && (plan->partial_dir[0] == '/' || has_path_traversal(plan->partial_dir))))
return false;
if (plan->backup_dir &&
!(plan->confined_backup = path_cat(plan->root_directory, plan->backup_dir)))
@@ -856,6 +1006,9 @@ static bool file_save_resolve_paths(FileSavePlan* plan) {
if (plan->partial_dir &&
!(plan->confined_partial = path_cat(plan->root_directory, plan->partial_dir)))
return false;
if (plan->temp_dir &&
!(plan->confined_temp = file_save_resolve_temp_dir(plan->root_directory, plan->temp_dir)))
return false;
const char* actual_root = plan->use_partial_root ? plan->confined_partial : plan->root_directory;
plan->destination_path = path_cat(plan->root_directory, plan->file->path);
@@ -981,7 +1134,7 @@ static bool file_save_install_data(FileSavePlan* plan, const FileMetadata* metad
config && config->preallocate, metadata, plan->policy, config && config->update,
config && config->ignore_existing, config && config->use_fsync, file->xattrs,
config ? config->fake_super : false, config ? config->partial : false, plan->confined_temp,
created_dirs, count_floor);
created_dirs, count_floor, (uint32_t)file->rdev_major, (uint32_t)file->rdev_minor);
}
free(count_floor);
return ok;
@@ -1088,17 +1241,22 @@ FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File*
/* A configured --temp-dir sends the temporary working copy to a scratch
directory; the engine then atomically renames the completed file into the
final destination directory. A relative temp dir is resolved under the
receive root and must already exist (an absolute or `..`-escaping value was
rejected above); the engine falls back to a non-atomic copy on EXDEV. The
partial-dir flow already keeps its working copy in a separate directory and
--inplace writes directly, so neither diverts through the scratch dir
(matching rsync, where --inplace/--partial-dir supersede --temp-dir). */
bool use_temp_dir = plan.temp_dir != NULL && !plan.inplace && !plan.use_partial_root;
final destination directory. The scratch path was confined to the receive
root (and canonicalized) in file_save_resolve_paths and must already exist;
the engine falls back to a non-atomic copy on EXDEV. The partial-dir flow
already keeps its working copy in a separate directory and --inplace writes
directly, so neither diverts through the scratch dir (matching rsync, where
--inplace/--partial-dir supersede --temp-dir). */
/* --inplace and --partial-dir supersede --temp-dir in rsync, so the scratch
dir is not used on those paths. The value was still validated/confined by
file_save_resolve_paths; drop the resolved path so it is never handed to the
install engine. */
if (plan.confined_temp && (plan.inplace || plan.use_partial_root)) {
free(plan.confined_temp);
plan.confined_temp = NULL;
}
bool use_temp_dir = plan.confined_temp != NULL;
if (use_temp_dir) {
plan.confined_temp = path_cat(root_directory, plan.temp_dir);
if (!plan.confined_temp)
goto out;
/* A user-supplied trailing slash would leave the scratch path ending in
"/", which has no final component to create/open. Normalize it away. */
size_t temp_len = strlen(plan.confined_temp);
+10 -2
View File
@@ -12,8 +12,16 @@
/* Outcome of a single file_save_to_disk operation. The receiver needs to
distinguish "written" from "skipped" so --remove-source-files can be told
which sources were actually stored. */
typedef enum { FILE_SAVE_ERROR = 0, FILE_SAVE_WRITTEN = 1, FILE_SAVE_SKIPPED = 2 } FileSaveResult;
which sources were actually stored. FILE_SAVE_FAILED is a per-entry failure
(for example a device node that mknodat() refused with EPERM/EACCES): it is
logged and counted by the receiver but does NOT abort the transfer, matching
rsync's continue-and-exit-partial behavior. */
typedef enum {
FILE_SAVE_ERROR = 0,
FILE_SAVE_WRITTEN = 1,
FILE_SAVE_SKIPPED = 2,
FILE_SAVE_FAILED = 3
} FileSaveResult;
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
+7 -3
View File
@@ -124,8 +124,12 @@ bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_meta
}
/* sendfile cannot encrypt TLS records. Keep the framing identical but
route encrypted transfers through the deadline-aware IO layer. */
if (io_get_ssl() != NULL) {
route encrypted transfers through the deadline-aware IO layer. Resolve
the transport from the bound session, not the thread-local io_ssl: a
worker thread running a TLS transfer has its SSL only on the session it
bound, so io_get_ssl() would be NULL there and the raw sendfile() path
would be taken on an encrypted socket. */
if (protocol_current_ssl() != NULL) {
unsigned char buffer[64 * 1024];
unsigned long long remaining = file_size;
bool ok = true;
@@ -185,7 +189,7 @@ bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_meta
return false;
}
protocol_note_bytes_written((unsigned long long)sent);
protocol_throttle_bytes((size_t)sent);
protocol_throttle_bytes(file_descriptor, (size_t)sent);
}
close(fd);
+163 -11
View File
@@ -3,6 +3,7 @@
#include "utils.h"
#include <errno.h>
#include <fcntl.h>
#include <fnmatch.h>
#include <grp.h>
#include <limits.h>
#include <pwd.h>
@@ -12,6 +13,8 @@
#include <sys/stat.h>
#include <unistd.h>
static bool identity_id_fits_int32(unsigned long id);
/* The active identity snapshot lives in a per-process global. The TCP server
* forks one child process per connection, so a connection never shares this
* with another; within a connection the multithreaded receiver reads it without
@@ -419,17 +422,10 @@ static int identity_parse_from(const char* token, bool is_group, int32_t* out_fr
/* Not a numeric LOW-HIGH range: fall through and treat as a name (a
* hyphenated account name like "wayne-smith" must still resolve). */
}
/* A sender-side name. A wildcard other than the bare '*' is matched by rsync
* against the sender's names; because FastSync transmits numeric ids only, the
* receiver cannot evaluate it, so reject rather than silently mis-match. */
if (identity_token_has_glob(token)) {
log_message(LOG_LEVEL_ERROR,
"%smap FROM '%s': name wildcards other than '*' are not supported "
"(FastSync transmits numeric ids, so sender names are unavailable on the "
"receiver)",
is_group ? "--group" : "--user", token);
return -1;
}
/* A sender-side name. A FROM name wildcard other than the bare '*' is handled
* by identity_expand_from_glob() in the caller (it expands against the
* sender's account database at CLI-parse time), so this function only sees the
* bare '*' or a literal name here. */
int32_t id;
if (identity_resolve_token(token, is_group, &id) != 0)
return -1;
@@ -486,6 +482,138 @@ static int identity_append_rule(IdentityMap** map, int* count, const IdentityMap
return 0;
}
/* True when `lo` and `hi` are adjacent ids (no overflow at INT32_MAX). */
static bool identity_ids_adjacent(int32_t lo, int32_t hi) {
return lo < INT32_MAX && hi == lo + 1;
}
static int identity_id_cmp(const void* a, const void* b) {
int32_t x = *(const int32_t*)a;
int32_t y = *(const int32_t*)b;
return (x > y) - (x < y);
}
static bool identity_ids_push(int32_t** ids, size_t* count, size_t* cap, int32_t id) {
if (*count == *cap) {
size_t grown_cap = *cap ? *cap * 2 : 16;
int32_t* grown = realloc(*ids, grown_cap * sizeof(int32_t));
if (!grown)
return false;
*ids = grown;
*cap = grown_cap;
}
(*ids)[(*count)++] = id;
return true;
}
/* Expand a FROM name wildcard (rsync's match against sender-side account names)
* into one rule per contiguous run of matching numeric ids, all sharing the same
* TO side. FastSync transmits numeric ids only, so the wildcard must be
* resolved here -- at CLI-parse time -- against the SENDER's passwd/group
* database; the receiver has no sender names to match. Contiguous matched ids
* are collapsed into a single LOW-HIGH range (a range of adjacent ids contains
* exactly the ids it spans, so this is semantically exact). Returns 0 on
* success, -1 on an allocation failure, a wildcard that matches no sender
* account, or an expansion that would push the map past MAX_IDENTITY_MAP. */
static int identity_expand_from_glob(Config* config, const char* glob, bool is_group,
const IdentityMap* to_rule) {
const char* optname = is_group ? "--groupmap" : "--usermap";
size_t cap = 0;
size_t n = 0;
int32_t* ids = NULL;
bool alloc_failed = false;
if (is_group) {
setgrent();
struct group* gr;
while ((gr = getgrent()) != NULL) {
if (fnmatch(glob, gr->gr_name, 0) != 0)
continue;
if (!identity_id_fits_int32((unsigned long)gr->gr_gid))
continue;
if (!identity_ids_push(&ids, &n, &cap, (int32_t)gr->gr_gid)) {
alloc_failed = true;
break;
}
}
endgrent();
} else {
setpwent();
struct passwd* pw;
while ((pw = getpwent()) != NULL) {
if (fnmatch(glob, pw->pw_name, 0) != 0)
continue;
if (!identity_id_fits_int32((unsigned long)pw->pw_uid))
continue;
if (!identity_ids_push(&ids, &n, &cap, (int32_t)pw->pw_uid)) {
alloc_failed = true;
break;
}
}
endpwent();
}
if (alloc_failed) {
free(ids);
log_message(LOG_LEVEL_ERROR, "%s: memory allocation failed expanding FROM '%s'", optname, glob);
return -1;
}
if (n == 0) {
free(ids);
log_message(LOG_LEVEL_ERROR, "%s FROM '%s': no source account name matches the wildcard",
optname, glob);
return -1;
}
qsort(ids, n, sizeof(int32_t), identity_id_cmp);
size_t unique = 0;
for (size_t i = 0; i < n; i++) {
if (unique == 0 || ids[unique - 1] != ids[i])
ids[unique++] = ids[i];
}
n = unique;
int runs = 0;
for (size_t i = 0; i < n; i++) {
if (i == 0 || !identity_ids_adjacent(ids[i - 1], ids[i]))
runs++;
}
IdentityMap** map = is_group ? &config->groupmap : &config->usermap;
int* count = is_group ? &config->groupmap_count : &config->usermap_count;
if (*count > MAX_IDENTITY_MAP - runs) {
log_message(LOG_LEVEL_ERROR,
"%s FROM '%s': the name wildcard expands to %d rule(s), which would exceed "
"the maximum of %d map rules",
optname, glob, runs, MAX_IDENTITY_MAP);
free(ids);
return -1;
}
for (size_t i = 0; i < n;) {
size_t j = i;
while (j + 1 < n && identity_ids_adjacent(ids[j], ids[j + 1]))
j++;
IdentityMap rule;
rule.from = ids[i];
rule.from_hi = ids[j];
rule.to = to_rule->to;
rule.to_name = to_rule->to_name ? str_dup(to_rule->to_name) : NULL;
if (to_rule->to_name && !rule.to_name) {
free(ids);
return -1;
}
if (identity_append_rule(map, count, &rule) != 0) {
free(rule.to_name);
free(ids);
return -1;
}
i = j + 1;
}
free(ids);
return 0;
}
int identity_parse_map(Config* config, const char* value, bool is_group) {
if (!config || !value || *value == '\0') {
log_message(LOG_LEVEL_ERROR, "%smap requires a value", is_group ? "--group" : "--user");
@@ -509,6 +637,30 @@ int identity_parse_map(Config* config, const char* value, bool is_group) {
char* to_token = colon + 1;
IdentityMap parsed;
memset(&parsed, 0, sizeof(parsed));
/* A FROM name wildcard (anything with a glob metacharacter other than the
* bare '*') is expanded against the sender's account database here, while
* the sender's passwd/group DB is still available; the resulting numeric
* rules travel on the wire like an explicit list. The TO side is parsed
* first so every expanded rule shares it. */
if (strcmp(from_token, "*") != 0 && identity_token_has_glob(from_token)) {
if (identity_parse_to(to_token, is_group, &parsed.to, &parsed.to_name) != 0) {
log_message(LOG_LEVEL_ERROR, "%s could not parse TO '%s' in '%s'", optname, to_token,
value);
free(list);
return -1;
}
if (identity_expand_from_glob(config, from_token, is_group, &parsed) != 0) {
free(parsed.to_name);
free(list);
return -1;
}
/* Every rule emitted by the expansion took its own str_dup of the name,
* so the parse-time copy is unreachable on success: release it here (the
* failure path above already does). `parsed.to_name` is NULL for a
* numeric TO. */
free(parsed.to_name);
continue;
}
if (identity_parse_from(from_token, is_group, &parsed.from, &parsed.from_hi) != 0) {
log_message(LOG_LEVEL_ERROR,
"%s could not resolve FROM '%s' in '%s' (a name must exist on the "
+8 -2
View File
@@ -25,8 +25,14 @@
/* Parse one --usermap= / --groupmap= value (comma-separated FROM:TO rules,
* first match wins) into config->usermap / config->groupmap. is_group selects
* the group tables and name databases. Returns 0 on success, -1 on a
* malformed spec or an unresolvable name (never a silent no-op). */
* the group tables and name databases. A FROM name wildcard (containing `*`,
* `?` or `[...]`, but not the bare `*`) is expanded against the SENDER's
* account database at parse time into one or more numeric id/range rules
* (contiguous ids collapse to a range) sharing the same TO, because only
* numeric ids cross the wire; the expansion is capped at MAX_IDENTITY_MAP and a
* wildcard matching no account is an error. Returns 0 on success, -1 on a
* malformed spec, an unresolvable name, an unmatched wildcard, or a map that
* would exceed MAX_IDENTITY_MAP (never a silent no-op). */
int identity_parse_map(Config* config, const char* value, bool is_group);
/* Parse --chown=USER:GROUP. Supports USER:GROUP, USER (owner only), :GROUP
+3
View File
@@ -37,6 +37,7 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
context->scan_had_io_error = false;
context->remove_source_files = NULL;
context->early_delete = false;
context->prescan_chunks = NULL;
context->delete_plans = NULL;
context->delete_suppressed = false;
context->scan_stopped_early = false;
@@ -194,6 +195,8 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
if (context->manifest) {
array_list_delete(context->manifest);
}
if (context->prescan_chunks)
array_list_delete(context->prescan_chunks);
if (context->delete_plans)
delete_plan_sender_destroy(context->delete_plans);
if (context->excluded_paths)
+7
View File
@@ -78,6 +78,13 @@ typedef struct {
path-only pre-scan on the calling thread and the pipeline scanner must not
append to it. Set once before the worker threads start. */
bool early_delete;
/* --delete-before: the path-only pre-scan that built the early keep-set,
retained as the pipeline's file list (owning Chunk*; consumed and NULLed by
the scanner thread) so the data pass replays rsync's single file list
instead of re-reading the source. NULL in every other mode, where the
scanner thread scans normally. Set once before the worker threads start
and freed with the context. */
ArrayList* prescan_chunks;
/* Non-NULL for --delete-during/--delete-delay: the per-directory plan set
prebuilt by the path-only pre-scan on the calling thread. The sender
thread transmits the root plan before any data and the remaining plans
+180 -82
View File
@@ -38,6 +38,129 @@ static atomic_ullong io_bytes_read = 0;
static unsigned long long global_bwlimit(void);
/* ------------------------------------------------------------------------- *
* Transport vtable implementations.
*
* Each op performs exactly one transfer attempt. WANT_READ/WANT_WRITE and an
* EINTR-interrupted syscall are reported as PROTOCOL_IO_RETRY (with
* *wait_events set to the poll event the caller must wait on); a clean peer
* close is PROTOCOL_IO_CLOSED and anything else is PROTOCOL_IO_ERROR. This
* keeps every WANT_READ/WANT_WRITE and EINTR retry exactly where it was before
* the vtable was introduced, just moved behind the function pointer.
* ------------------------------------------------------------------------- */
static ssize_t plain_io_send(ProtocolSession* session, const void* data, size_t size,
short* wait_events) {
ssize_t written = write(session->write_fd, data, size);
if (written < 0) {
if (errno == EINTR)
return PROTOCOL_IO_RETRY;
return PROTOCOL_IO_ERROR;
}
if (written == 0)
return PROTOCOL_IO_ERROR;
*wait_events = POLLOUT;
return written;
}
static ssize_t plain_io_recv(ProtocolSession* session, void* data, size_t size,
short* wait_events) {
ssize_t received = read(session->read_fd, data, size);
if (received < 0) {
if (errno == EINTR)
return PROTOCOL_IO_RETRY;
return PROTOCOL_IO_ERROR;
}
if (received == 0)
return PROTOCOL_IO_CLOSED;
*wait_events = POLLIN;
return received;
}
static bool plain_io_has_pending(const ProtocolSession* session) {
(void)session;
return false;
}
static ssize_t tls_io_send(ProtocolSession* session, const void* data, size_t size,
short* wait_events) {
/* SSL_write takes an int length; clamp a >INT_MAX request into chunks so the
* size_t downcast can never truncate into a negative/partial write. */
size_t chunk = size > (size_t)INT_MAX ? (size_t)INT_MAX : size;
ssize_t written = SSL_write(session->ssl, data, (int)chunk);
if (written <= 0) {
int ssl_err = SSL_get_error(session->ssl, (int)written);
if (ssl_err == SSL_ERROR_WANT_WRITE) {
*wait_events = POLLOUT;
return PROTOCOL_IO_RETRY;
}
if (ssl_err == SSL_ERROR_WANT_READ) {
*wait_events = POLLIN;
return PROTOCOL_IO_RETRY;
}
/* A signal (e.g. Ctrl-C) interrupts the blocking TLS write: retry so the
* send loop can observe the abort flag at the next checkpoint. Only an
* actual negative return is an interrupted syscall; a 0-byte SSL_write is
* not a valid EINTR retry. */
if (written < 0 && ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
return PROTOCOL_IO_RETRY;
return PROTOCOL_IO_ERROR;
}
*wait_events = POLLOUT;
return written;
}
static ssize_t tls_io_recv(ProtocolSession* session, void* data, size_t size, short* wait_events) {
/* SSL_read takes an int length; clamp a >INT_MAX request into chunks
* (mirrors the send path) so the size_t downcast can never truncate into a
* negative/partial read. */
size_t chunk = size > (size_t)INT_MAX ? (size_t)INT_MAX : size;
ssize_t received = SSL_read(session->ssl, data, (int)chunk);
if (received <= 0) {
int ssl_err = SSL_get_error(session->ssl, (int)received);
if (ssl_err == SSL_ERROR_WANT_WRITE) {
*wait_events = POLLOUT;
return PROTOCOL_IO_RETRY;
}
if (ssl_err == SSL_ERROR_WANT_READ) {
*wait_events = POLLIN;
return PROTOCOL_IO_RETRY;
}
/* A signal interrupts the blocking TLS read: retry (mirrors the send path)
* so the loop reaches its next abort/deadline checkpoint. Only an actual
* negative return is an interrupted syscall: a 0-byte SSL_read is an
* unexpected EOF (the peer closed without close_notify), which OpenSSL also
* reports as SSL_ERROR_SYSCALL with errno possibly still EINTR from an
* earlier interrupted poll/read. Retrying that would busy-spin the
* status-read loop until its deadline, so classify it as closed instead. */
if (received < 0 && ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
return PROTOCOL_IO_RETRY;
/* A zero-length SSL_read is the peer's clean close_notify (or EOF without
* one); report it distinctly so the caller can log it as a close. */
if (received == 0)
return PROTOCOL_IO_CLOSED;
return PROTOCOL_IO_ERROR;
}
*wait_events = POLLIN;
return received;
}
static bool tls_io_has_pending(const ProtocolSession* session) {
return session->ssl != NULL && SSL_pending(session->ssl) > 0;
}
static const ProtocolIoOps plain_io_ops = {
.send = plain_io_send,
.recv = plain_io_recv,
.has_pending = plain_io_has_pending,
};
static const ProtocolIoOps tls_io_ops = {
.send = tls_io_send,
.recv = tls_io_recv,
.has_pending = tls_io_has_pending,
};
static bool protocol_reserve_memory(ProtocolSession* session, size_t charge) {
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
while (true) {
@@ -79,6 +202,7 @@ void io_set_fds(int read_fd, int write_fd) {
legacy_io_session.read_fd = read_fd;
legacy_io_session.write_fd = write_fd;
legacy_io_session.ssl = NULL;
legacy_io_session.ops = &plain_io_ops;
legacy_io_session.eight_bit_output = false;
atomic_store(&legacy_io_session.total_allocated_bytes, 0);
legacy_io_session.max_alloc = DEFAULT_MAX_ALLOC;
@@ -91,6 +215,7 @@ void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd)
memset(session, 0, sizeof(*session));
session->read_fd = read_fd;
session->write_fd = write_fd;
session->ops = &plain_io_ops;
session->max_alloc = DEFAULT_MAX_ALLOC;
session->io_timeout_sec = RECEIVE_TIMEOUT_SEC;
atomic_init(&session->total_allocated_bytes, 0);
@@ -158,8 +283,12 @@ void protocol_session_unbind(void) {
}
void protocol_session_set_ssl(ProtocolSession* session, SSL* ssl) {
if (session)
session->ssl = ssl;
if (!session)
return;
session->ssl = ssl;
/* Select the transport dispatch once, here, instead of branching on the SSL
* pointer inside every I/O loop. */
session->ops = ssl ? &tls_io_ops : &plain_io_ops;
}
static void bw_mutex_init(void) {
@@ -270,6 +399,20 @@ SSL* io_get_ssl(void) {
return io_ssl;
}
SSL* protocol_current_ssl(void) {
/* The bound session is the authoritative transport for a worker thread: it
* was explicitly handed to protocol_session_bind() and carries its own SSL,
* whereas io_ssl is thread-local and NULL in a thread that never performed
* the handshake. Only a session whose selected dispatch is TLS may supply
* the SSL: a bound plaintext session has ssl == NULL and must not shadow a
* live thread-local io_ssl, or file_send.c would take the raw sendfile(2)
* path on a socket this thread is driving with TLS. With no TLS session
* bound (plaintext session, or the fd-shim path), fall back to io_ssl. */
if (bound_session && bound_session->ops == &tls_io_ops && bound_session->ssl)
return bound_session->ssl;
return io_ssl;
}
unsigned long long protocol_bytes_written(void) {
return atomic_load(&io_bytes_written);
}
@@ -298,15 +441,19 @@ static ProtocolSession* legacy_session(int read_fd, int write_fd) {
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
}
legacy_io_session.ssl = io_ssl;
legacy_io_session.ops = io_ssl ? &tls_io_ops : &plain_io_ops;
return &legacy_io_session;
}
/* Pace an out-of-band write that bypassed protocol_send_n_data (the plaintext
* sendfile fast path). The bound/legacy session is resolved exactly as
* send_n_data resolves it, so the same token-bucket state is throttled and the
* TLS and plaintext transports share identical --bwlimit semantics. */
void protocol_throttle_bytes(size_t bytes) {
bw_throttle_session(legacy_session(-1, -1), bytes);
* sendfile fast path). The bound/legacy session is resolved exactly as the
* preceding send_n_data(fd, ...) resolved it, so the same token-bucket state is
* throttled and the TLS and plaintext transports share identical --bwlimit
* semantics. Passing the wire fd (rather than -1) is essential: the sendfile
* send left legacy_io_session.write_fd bound to it, so resolving with -1 would
* mismatch, re-initialize the session and hand out a second first-call burst. */
void protocol_throttle_bytes(int file_descriptor, size_t bytes) {
bw_throttle_session(legacy_session(-1, file_descriptor), bytes);
}
bool send_n_data(int file_descriptor, const void* data, size_t data_size) {
@@ -332,8 +479,9 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
if (!data && data_size != 0)
return false;
log_debug_message(LOG_DEBUG_IO, " Sending n Data: %zu", data_size);
if (!session)
if (!session || !session->ops)
return false;
const ProtocolIoOps* ops = session->ops;
/* A non-positive session timeout disables the deadline entirely (rsync's
* --timeout=0 default); poll then blocks until the socket becomes writable. */
int timeout_sec = session->io_timeout_sec > 0 ? session->io_timeout_sec : 0;
@@ -359,36 +507,16 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
continue;
if (pfd.revents & (POLLERR | POLLNVAL))
return false;
ssize_t bytes_send;
if (session->ssl) {
/* SSL_write takes an int length; clamp a >INT_MAX request into chunks so
* the size_t downcast can never truncate into a negative/partial write. */
size_t ssl_chunk = chunk > (size_t)INT_MAX ? (size_t)INT_MAX : chunk;
bytes_send = SSL_write(session->ssl, (const char*)data + total_bytes_send, (int)ssl_chunk);
} else {
bytes_send = write(fd, (const char*)data + total_bytes_send, chunk);
}
ssize_t bytes_send =
ops->send(session, (const char*)data + total_bytes_send, chunk, &wait_events);
if (bytes_send == PROTOCOL_IO_RETRY)
continue;
if (bytes_send <= 0) {
if (session->ssl) {
int ssl_err = SSL_get_error(session->ssl, (int)bytes_send);
if (ssl_err == SSL_ERROR_WANT_WRITE || ssl_err == SSL_ERROR_WANT_READ) {
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
continue;
}
/* A signal (e.g. Ctrl-C) interrupts the blocking TLS write: retry so
the send loop can observe the abort flag at the next checkpoint. */
if (ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
continue;
} else if (errno == EINTR) {
continue;
}
log_message(LOG_LEVEL_ERROR, "Could not send data");
return false;
}
bw_throttle_session(session, (size_t)bytes_send);
total_bytes_send += bytes_send;
if (session->ssl)
wait_events = POLLOUT;
}
log_debug_message(LOG_DEBUG_IO, " Send n Data: %zd", total_bytes_send);
atomic_fetch_add(&io_bytes_written, (unsigned long long)total_bytes_send);
@@ -421,14 +549,15 @@ bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_s
static bool protocol_receive_n_data_until(ProtocolSession* session, void* data, size_t data_size,
const struct timespec* deadline) {
log_debug_message(LOG_DEBUG_IO, " Receiving n Data: %zu", data_size);
if (!session)
if (!session || !session->ops)
return false;
const ProtocolIoOps* ops = session->ops;
int fd = session->read_fd;
size_t total_bytes_received = 0;
short wait_events = POLLIN;
while (total_bytes_received < data_size) {
if (!session->ssl || SSL_pending(session->ssl) == 0) {
if (!ops->has_pending(session)) {
struct pollfd pfd = {.fd = fd, .events = wait_events};
/* A NULL deadline means "wait indefinitely" (timeout disabled). */
int poll_result = poll(&pfd, 1, deadline ? deadline_remaining_ms(deadline) : -1);
@@ -446,43 +575,19 @@ static bool protocol_receive_n_data_until(ProtocolSession* session, void* data,
return false;
}
ssize_t bytes_received;
if (session->ssl) {
/* SSL_read takes an int length; clamp a >INT_MAX request into chunks
* (mirrors the send path) so the size_t downcast can never truncate into
* a negative/partial read. */
size_t ssl_chunk = data_size - total_bytes_received > (size_t)INT_MAX
? (size_t)INT_MAX
: data_size - total_bytes_received;
bytes_received = SSL_read(session->ssl, (char*)data + total_bytes_received, (int)ssl_chunk);
} else {
bytes_received =
read(fd, (char*)data + total_bytes_received, data_size - total_bytes_received);
ssize_t bytes_received = ops->recv(session, (char*)data + total_bytes_received,
data_size - total_bytes_received, &wait_events);
if (bytes_received == PROTOCOL_IO_RETRY)
continue;
if (bytes_received == PROTOCOL_IO_CLOSED) {
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
return false;
}
if (bytes_received <= 0) {
if (session->ssl) {
int ssl_err = SSL_get_error(session->ssl, (int)bytes_received);
if (ssl_err == SSL_ERROR_WANT_WRITE || ssl_err == SSL_ERROR_WANT_READ) {
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
continue;
}
/* A signal interrupts the blocking TLS read: retry (mirrors the send
path and protocol_read_status_until) so the loop reaches its next
abort/deadline checkpoint instead of failing spuriously. */
if (ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
continue;
} else if (errno == EINTR) {
continue;
}
if (bytes_received == 0)
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
else
log_message(LOG_LEVEL_ERROR, "Could not receive bytes");
log_message(LOG_LEVEL_ERROR, "Could not receive bytes");
return false;
}
total_bytes_received += (size_t)bytes_received;
if (session->ssl)
wait_events = POLLIN;
}
log_debug_message(LOG_DEBUG_IO, " Received n Data: %zu", total_bytes_received);
atomic_fetch_add(&io_bytes_read, (unsigned long long)total_bytes_received);
@@ -842,11 +947,14 @@ bool protocol_receive_status_timed(ProtocolSession* session, Status* status, int
* reply across a frame boundary. Returns false on timeout/EOF/error. */
static bool protocol_read_status_until(ProtocolSession* session, Status* status,
const struct timespec* deadline) {
if (!session || !session->ops)
return false;
const ProtocolIoOps* ops = session->ops;
Status received = STATUS_ERROR;
size_t got = 0;
short wait_events = POLLIN;
while (got < sizeof(Status)) {
if (!session->ssl || SSL_pending(session->ssl) == 0) {
if (!ops->has_pending(session)) {
int remaining_ms = deadline ? deadline_remaining_ms(deadline) : -1;
if (remaining_ms == 0) {
log_message(LOG_LEVEL_ERROR, "Receive timeout while reading status");
@@ -866,21 +974,11 @@ static bool protocol_read_status_until(ProtocolSession* session, Status* status,
if (pfd.revents & (POLLERR | POLLNVAL))
return false;
}
ssize_t bytes_received;
if (session->ssl)
bytes_received = SSL_read(session->ssl, (char*)&received + got, sizeof(Status) - got);
else
bytes_received = read(session->read_fd, (char*)&received + got, sizeof(Status) - got);
ssize_t bytes_received =
ops->recv(session, (char*)&received + got, sizeof(Status) - got, &wait_events);
if (bytes_received == PROTOCOL_IO_RETRY)
continue;
if (bytes_received <= 0) {
if (session->ssl) {
int ssl_err = SSL_get_error(session->ssl, (int)bytes_received);
if (ssl_err == SSL_ERROR_WANT_READ || ssl_err == SSL_ERROR_WANT_WRITE) {
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
continue;
}
}
if (bytes_received < 0 && errno == EINTR)
continue;
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving status");
return false;
}
@@ -909,7 +1007,7 @@ bool protocol_receive_status_keepalive(ProtocolSession* session, Status* status,
while (true) {
if (abort_check && abort_check())
return false;
if (!session->ssl || SSL_pending(session->ssl) == 0) {
if (!session->ops || !session->ops->has_pending(session)) {
int remaining_ms = deadline_remaining_ms(&deadline);
if (remaining_ms <= 0) {
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", timeout_sec);
+53 -7
View File
@@ -50,16 +50,48 @@
typedef struct ssl_st SSL;
typedef struct ProtocolSession ProtocolSession;
/*
* Transport vtable: the per-session set of I/O primitives the three protocol
* loops (send, receive, status-read) dispatch through. The ops are selected
* once, when the session is initialized or its SSL is installed, so the loops
* never branch on the transport at runtime. A plaintext session uses the
* read()/write() ops; a TLS session uses the SSL_read()/SSL_write() ops.
*
* `send`/`recv` attempt exactly one transfer and return:
* > 0 bytes transferred,
* PROTOCOL_IO_RETRY no progress; poll on *wait_events and retry,
* PROTOCOL_IO_CLOSED peer closed the stream,
* PROTOCOL_IO_ERROR fatal transport error.
* `has_pending` reports bytes already buffered by the transport (a TLS record
* residue); the receive loops skip the poll() gate when it is true.
*/
typedef struct ProtocolIoOps {
ssize_t (*send)(ProtocolSession* session, const void* data, size_t size, short* wait_events);
ssize_t (*recv)(ProtocolSession* session, void* data, size_t size, short* wait_events);
bool (*has_pending)(const ProtocolSession* session);
} ProtocolIoOps;
/* Negative sentinels returned by ProtocolIoOps.send/recv (see above). */
enum {
PROTOCOL_IO_RETRY = -1,
PROTOCOL_IO_CLOSED = -2,
PROTOCOL_IO_ERROR = -3,
};
/*
* Explicit owner of protocol I/O. A session does not own the descriptors or
* SSL object; it only describes the transport used by a transfer. This makes
* it safe to pass the transport to a worker without relying on inherited
* thread-local state.
*/
typedef struct ProtocolSession {
struct ProtocolSession {
int read_fd;
int write_fd;
SSL* ssl;
/* Transport dispatch selected by protocol_session_init()/set_ssl(). */
const ProtocolIoOps* ops;
unsigned long long bwlimit;
long long bw_tokens;
long long bw_last_refill_sec;
@@ -75,7 +107,7 @@ typedef struct ProtocolSession {
* SO_RCVTIMEO/SO_SNDTIMEO. The server does not propagate a client 0 here: it
* installs protocol_server_io_timeout_sec() so its sessions keep a floor. */
int io_timeout_sec;
} ProtocolSession;
};
typedef int Status;
enum NET_STATUS {
@@ -216,6 +248,17 @@ void io_set_bwlimit(unsigned long long bytes_per_sec);
unsigned long long io_get_bwlimit(void);
void io_set_ssl(SSL* ssl);
SSL* io_get_ssl(void);
/* SSL object of the transport in effect on this thread: the currently bound
* session's SSL when a TLS session is bound, otherwise the legacy thread-local
* io_ssl. NULL for a plaintext transport. Unlike io_get_ssl(), this resolves
* worker threads that bound a TLS session via protocol_session_set_ssl()/
* protocol_session_bind() but never called io_set_ssl() themselves (C11
* _Thread_local state is not inherited by a new thread). A bound session only
* wins when its selected dispatch is TLS; a bound plaintext session (ssl ==
* NULL) falls back to io_ssl so it can never mask a live encrypted transport.
* Callers that must choose a TLS-only code path (e.g. file_send.c's sendfile
* fallback) must use this instead of io_get_ssl(). */
SSL* protocol_current_ssl(void);
/* Process-wide wire byte counters. protocol_send_n_data/protocol_receive_n_data
* update them; the zero-copy sendfile path reports through
@@ -225,11 +268,14 @@ unsigned long long protocol_bytes_written(void);
unsigned long long protocol_bytes_read(void);
void protocol_note_bytes_written(unsigned long long bytes);
/* Apply --bwlimit pacing to bytes written outside protocol_send_n_data (the
* plaintext zero-copy sendfile fast path). Resolves the bound/legacy session
* exactly as send_n_data does and runs the same token-bucket throttle, so the
* sendfile transport is paced identically to the buffered/TLS paths. A no-op
* when the effective session has no bandwidth limit. */
void protocol_throttle_bytes(size_t bytes);
* plaintext zero-copy sendfile fast path). `file_descriptor` is the wire fd
* the bytes were written to, so the legacy session is resolved exactly as the
* preceding send_n_data call resolved it (the bound TLS session still wins when
* set); resolving with the same fd avoids re-initializing the legacy session
* and granting a second first-call burst. Runs the same token-bucket throttle,
* so the sendfile transport is paced identically to the buffered/TLS paths. A
* no-op when the effective session has no bandwidth limit. */
void protocol_throttle_bytes(int file_descriptor, size_t bytes);
void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd);
/* Transitional bridge for helpers whose signatures still carry only an fd. */
-635
View File
@@ -625,641 +625,6 @@ bool format_human_bytes(unsigned long long bytes, char* buffer, size_t buffer_si
return written >= 0 && (size_t)written < buffer_size;
}
/* Build the keep-set index from the exact manifest entries only. A lookup of
`rel` succeeds iff `rel` is a kept entry, a kept directory, or an ancestor
directory of kept content (the old is_dir_in_manifest predicate); the sorted
view answers "is an ancestor of kept content" without materializing any
per-component prefix copy, so the index is O(manifest size) memory. */
static bool build_keep_index(const ArrayList* manifest, PathIndex* index) {
if (!manifest || manifest->size <= 0)
return path_index_build(index, NULL, 0);
return path_index_build(index, (const char* const*)manifest->items, (size_t)manifest->size);
}
static bool keep_is_dir(const PathIndex* index, const char* rel_path) {
return path_index_contains(index, rel_path) || path_index_has_descendant(index, rel_path);
}
static bool keep_is_file(const PathIndex* index, const char* rel_path) {
return path_index_contains(index, rel_path);
}
/* True when child_rel is, or lies below, a protected entry. A prefix "a"
therefore protects "a" and "a/b/c" but not "ab". Entries with top_level_only
set only protect DIRECT children of the receive root (at_root); nested
directories that share such a name stay ordinary destination content. */
bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSkipEntry* skips,
int skip_count) {
for (int i = 0; i < skip_count; i++) {
if (skips[i].top_level_only && !at_root)
continue;
size_t prefix_len = strlen(skips[i].prefix);
if (strncmp(child_rel, skips[i].prefix, prefix_len) == 0 &&
(child_rel[prefix_len] == '\0' || child_rel[prefix_len] == '/'))
return true;
}
return false;
}
/* Per-run deletion budget and tallies. `max_delete` is the cap on the number
of entries the walker may remove (SIZE_MAX = unlimited); once it is reached
the remaining extras are counted in `skipped` and left in place, matching
rsync's partial --max-delete behavior. */
typedef struct {
size_t max_delete;
size_t deleted;
size_t skipped;
bool limit_hit;
} DeleteBudget;
/* True when direct children of the directory named by `rel` may be removed.
With no synchronization info (dirs == NULL) the whole tree is deletable; when
a dirs index is supplied only its exact entries are (the receive root is the
"." sentinel). */
static bool is_synced_dir(const PathIndex* dirs, const char* rel) {
if (!dirs)
return true;
return path_index_contains(dirs, rel[0] == '\0' ? "." : rel);
}
/* Unsigned byte-wise string compare, matching rsync's u_strcmp (a signed
strcmp would order bytes >= 0x80 differently). */
static int delete_name_cmp(const char* a, const char* b) {
const unsigned char* pa = (const unsigned char*)a;
const unsigned char* pb = (const unsigned char*)b;
while (*pa != '\0' && *pa == *pb) {
pa++;
pb++;
}
return (int)*pa - (int)*pb;
}
bool delete_dir_entries_collect(int dirfd, DeleteDirEntry** out, size_t* count,
bool* operation_ok) {
*out = NULL;
*count = 0;
if (operation_ok)
*operation_ok = true;
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (scanfd < 0)
return false;
DIR* dir = fdopendir(scanfd);
if (!dir) {
close(scanfd);
return false;
}
DeleteDirEntry* entries = NULL;
size_t used = 0;
size_t capacity = 0;
bool ok = true;
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT && operation_ok)
*operation_ok = false;
continue;
}
if (used == capacity) {
size_t next = capacity == 0 ? 16 : capacity * 2;
DeleteDirEntry* grown = realloc(entries, next * sizeof(*grown));
if (!grown) {
ok = false;
break;
}
entries = grown;
capacity = next;
}
entries[used].name = str_dup(entry->d_name);
if (!entries[used].name) {
ok = false;
break;
}
entries[used].is_dir = S_ISDIR(st.st_mode);
used++;
}
closedir(dir);
if (!ok) {
delete_dir_entries_free(entries, used);
return false;
}
*out = entries;
*count = used;
return true;
}
void delete_dir_entries_free(DeleteDirEntry* entries, size_t count) {
if (!entries)
return;
for (size_t i = 0; i < count; i++)
free(entries[i].name);
free(entries);
}
/* rsync's extraneous-entry order: subdirectories before files, each group in
descending name order. */
int delete_dir_entry_cmp_desc(const void* a, const void* b) {
const DeleteDirEntry* ea = a;
const DeleteDirEntry* eb = b;
if (ea->is_dir != eb->is_dir)
return ea->is_dir ? -1 : 1;
return -delete_name_cmp(ea->name, eb->name);
}
/* rsync's kept-subdirectory order: plain ascending name. */
int delete_dir_entry_cmp_asc(const void* a, const void* b) {
const DeleteDirEntry* ea = a;
const DeleteDirEntry* eb = b;
return delete_name_cmp(ea->name, eb->name);
}
/* Remove the extras directly inside the directory open on `dirfd`, recursing
into every child directory so kept content below a synchronized prefix is
reached. `all_removed` reports whether every child entry was removed (so the
caller may rmdir this directory). A child directory is never removed when it
is itself a synchronized directory or holds kept content; with a dirs index
supplied, direct children of a non-synchronized directory are never extras at
all (they are left in place but still descended into). Symlinks are unlinked
like any other non-directory extra (never followed).
Entries are processed in rsync's order (extraneous subdirectories in
descending name order, then extraneous files, then kept subdirectories in
ascending order) rather than readdir() order, so `--max-delete` leaves the
same survivors and the `--info=del`/dry-run line order matches rsync. */
static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
const PathIndex* dirs, DeleteBudget* budget,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, bool parent_deletable,
bool* all_removed, DeletePathObserver observer,
void* observer_context) {
DeleteDirEntry* entries = NULL;
size_t count = 0;
bool collect_ok = true;
if (!delete_dir_entries_collect(dirfd, &entries, &count, &collect_ok))
return false;
bool operation_ok = collect_ok;
bool local_survives = false;
bool* shielded = calloc(count ? count : 1, sizeof(bool));
bool* is_extra = calloc(count ? count : 1, sizeof(bool));
if (!shielded || !is_extra) {
free(shielded);
free(is_extra);
delete_dir_entries_free(entries, count);
return false;
}
/* A directory is deletable when it or ANY ancestor is synchronized; the
`parent_deletable` flag carries that down the recursion so dest-only
directories below a synchronized root are removed wholesale. */
bool deletable = parent_deletable || is_synced_dir(dirs, rel_path);
bool at_root = rel_path[0] == '\0';
/* Reproduce rsync's traversal order: extraneous subdirectories in descending
name order, then extraneous files in descending name order, and kept
subdirectories only afterwards (ascending). Sorting up front also fixes the
identity of the survivors under a partial --max-delete. */
if (count > 1)
qsort(entries, count, sizeof(*entries), delete_dir_entry_cmp_desc);
size_t dir_count = 0;
while (dir_count < count && entries[dir_count].is_dir)
dir_count++;
/* Classify every entry up front (the verdict does not depend on processing
order) so the ordered passes below can act on it. */
for (size_t i = 0; i < count; i++) {
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
/* A --delay-updates run keeps its staging directory as a direct child of
the receive root, and basis-dir snapshots live below it too. Their
contents are not manifest entries, so descending into them would delete
every staged / basis file as an "extra". Only the staging name (a
top-level-only prefix) and the basis prefixes are protected: a nested
destination directory that happens to be called .fastsync-stage is
ordinary content. */
if (path_under_skip_prefix(child_rel, at_root, skips, skip_count)) {
shielded[i] = true;
local_survives = true;
} else if (protect_rules &&
filter_rules_apply_side(protect_rules, child_rel, entries[i].name, entries[i].is_dir,
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
/* A first-match protect rule shields the extra; for a directory the whole
subtree is shielded (rsync prunes an excluded directory), so do not
descend. */
shielded[i] = true;
local_survives = true;
} else if (entries[i].is_dir) {
bool child_synced = dirs && path_index_contains(dirs, child_rel);
is_extra[i] = deletable && !child_synced && !keep_is_dir(keep, child_rel);
if (!is_extra[i])
local_survives = true;
} else {
is_extra[i] = deletable && !keep_is_file(keep, child_rel);
if (!is_extra[i])
local_survives = true;
}
free(child_rel);
}
/* Pass 1: extraneous subdirectories, descending. */
for (size_t i = 0; i < dir_count; i++) {
if (!is_extra[i])
continue;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
int childfd = openat(dirfd, entries[i].name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count,
protect_rules, deletable, &child_all_removed, observer,
observer_context))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
if (child_all_removed && deletable) {
if (budget->deleted >= budget->max_delete) {
budget->limit_hit = true;
budget->skipped++;
local_survives = true;
} else if (unlinkat(dirfd, entries[i].name, AT_REMOVEDIR) != 0) {
/* ENOENT: already gone (fine). ENOTEMPTY/EEXIST: the directory still
holds entries the walker leaves in place (a protected excluded
prefix, a kept file the manifest protects, a symlink); rsync leaves
such a directory behind, so this is not an error. Only genuine I/O
failures abort the deletion. */
if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST)
operation_ok = false;
local_survives = true;
} else {
budget->deleted++;
/* rsync reports a removed directory with a trailing slash. */
if (observer) {
size_t len = strlen(child_rel);
char* with_slash = malloc(len + 2);
if (with_slash) {
memcpy(with_slash, child_rel, len);
with_slash[len] = '/';
with_slash[len + 1] = '\0';
observer(observer_context, with_slash);
free(with_slash);
} else {
observer(observer_context, child_rel);
}
}
}
} else {
local_survives = true;
}
free(child_rel);
}
/* Pass 2: extraneous files, descending. */
for (size_t i = dir_count; i < count; i++) {
if (!is_extra[i])
continue;
if (budget->deleted >= budget->max_delete) {
budget->limit_hit = true;
budget->skipped++;
local_survives = true;
} else if (unlinkat(dirfd, entries[i].name, 0) != 0) {
if (errno != ENOENT)
operation_ok = false;
local_survives = true;
} else {
budget->deleted++;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (child_rel) {
if (observer)
observer(observer_context, child_rel);
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
free(escaped_path);
}
free(child_rel);
}
}
/* Pass 3: kept subdirectories, ascending (rsync descends into these only
after the parent's own extras have been handled). */
for (size_t i = dir_count; i-- > 0;) {
if (is_extra[i] || shielded[i])
continue;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
int childfd = openat(dirfd, entries[i].name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count,
protect_rules, deletable, &child_all_removed, observer,
observer_context))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
/* A kept/synchronized directory is never removed. */
local_survives = true;
free(child_rel);
}
free(shielded);
free(is_extra);
delete_dir_entries_free(entries, count);
*all_removed = !local_survives;
return operation_ok;
}
/* Read-only mirror of delete_extras_fd: records the paths that WOULD be removed
without unlinking anything. A child directory is reported after its own
reportable children (depth-first), matching the delete pass's ordering. */
static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
const PathIndex* dirs, ArrayList* out, size_t* recorded,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, bool parent_deletable,
bool* all_removed) {
DeleteDirEntry* entries = NULL;
size_t count = 0;
bool collect_ok = true;
if (!delete_dir_entries_collect(dirfd, &entries, &count, &collect_ok))
return false;
bool operation_ok = collect_ok;
bool local_survives = false;
bool* shielded = calloc(count ? count : 1, sizeof(bool));
bool* is_extra = calloc(count ? count : 1, sizeof(bool));
if (!shielded || !is_extra) {
free(shielded);
free(is_extra);
delete_dir_entries_free(entries, count);
return false;
}
bool deletable = parent_deletable || is_synced_dir(dirs, rel_path);
bool at_root = rel_path[0] == '\0';
/* Mirror the delete walk's rsync order (extraneous subdirectories descending,
then extraneous files descending, then kept subdirectories ascending). */
if (count > 1)
qsort(entries, count, sizeof(*entries), delete_dir_entry_cmp_desc);
size_t dir_count = 0;
while (dir_count < count && entries[dir_count].is_dir)
dir_count++;
for (size_t i = 0; i < count; i++) {
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
if (path_under_skip_prefix(child_rel, at_root, skips, skip_count)) {
shielded[i] = true;
local_survives = true;
} else if (protect_rules &&
filter_rules_apply_side(protect_rules, child_rel, entries[i].name, entries[i].is_dir,
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
/* Mirror the delete walk: a protected entry is never reported as a
would-delete and a protected directory's subtree is not enumerated. */
shielded[i] = true;
local_survives = true;
} else if (entries[i].is_dir) {
bool child_synced = dirs && path_index_contains(dirs, child_rel);
is_extra[i] = deletable && !child_synced && !keep_is_dir(keep, child_rel);
if (!is_extra[i])
local_survives = true;
} else {
is_extra[i] = deletable && !keep_is_file(keep, child_rel);
if (!is_extra[i])
local_survives = true;
}
free(child_rel);
}
/* Pass 1: extraneous subdirectories, descending (recorded after contents). */
for (size_t i = 0; i < dir_count; i++) {
if (!is_extra[i])
continue;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
int childfd = openat(dirfd, entries[i].name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
protect_rules, deletable, &child_all_removed))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
if (child_all_removed && deletable) {
size_t len = strlen(child_rel);
char* copy = malloc(len + 2);
if (!copy) {
operation_ok = false;
} else {
memcpy(copy, child_rel, len);
copy[len] = '/';
copy[len + 1] = '\0';
if (!array_list_add(out, copy)) {
free(copy);
operation_ok = false;
} else {
(*recorded)++;
}
}
} else {
local_survives = true;
}
free(child_rel);
}
/* Pass 2: extraneous files, descending. */
for (size_t i = dir_count; i < count; i++) {
if (!is_extra[i])
continue;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
char* copy = str_dup(child_rel);
if (!copy || !array_list_add(out, copy)) {
free(copy);
operation_ok = false;
} else {
(*recorded)++;
}
free(child_rel);
}
/* Pass 3: kept subdirectories, ascending. */
for (size_t i = dir_count; i-- > 0;) {
if (is_extra[i] || shielded[i])
continue;
char* child_rel = path_cat((char*)rel_path, entries[i].name);
if (!child_rel) {
operation_ok = false;
continue;
}
int childfd = openat(dirfd, entries[i].name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
protect_rules, deletable, &child_all_removed))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
local_survives = true;
free(child_rel);
}
free(shielded);
free(is_extra);
delete_dir_entries_free(entries, count);
*all_removed = !local_survives;
return operation_ok;
}
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out) {
if (count_out)
*count_out = 0;
if (!manifest || !out)
return false;
PathIndex keep;
if (!build_keep_index(manifest, &keep))
return false;
PathIndex dirs;
bool have_dirs = synced_dirs != NULL;
if (have_dirs &&
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
path_index_free(&keep);
return false;
}
int rootfd;
int root_fd = utils_get_authorized_root_fd();
if (root_fd >= 0) {
if (utils_get_authorized_root_path())
rootfd = utils_open_authorized_destination(dest_root);
else if (dest_root == NULL)
rootfd = dup(root_fd);
else
rootfd = -1;
} else {
rootfd = open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
if (rootfd < 0) {
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
return false;
}
bool all_removed = false;
size_t recorded = 0;
bool ok = list_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, out, &recorded, skips,
skip_count, protect_rules, false, &all_removed);
if (close(rootfd) != 0)
ok = false;
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
if (count_out)
*count_out = recorded;
return ok;
}
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer,
void* observer_context) {
if (deleted_out)
*deleted_out = 0;
if (skipped_out)
*skipped_out = 0;
if (!manifest)
return DELETE_WALK_ERROR;
/* Index the keep-set (and the synchronized-dir set, when supplied) once so
membership is answered in O(path length) instead of scanning every entry
for every destination entry. */
PathIndex keep;
if (!build_keep_index(manifest, &keep))
return DELETE_WALK_ERROR;
PathIndex dirs;
bool have_dirs = synced_dirs != NULL;
if (have_dirs &&
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
path_index_free(&keep);
return DELETE_WALK_ERROR;
}
int rootfd;
int root_fd = utils_get_authorized_root_fd();
if (root_fd >= 0) {
if (utils_get_authorized_root_path())
rootfd = utils_open_authorized_destination(dest_root);
else if (dest_root == NULL)
rootfd = dup(root_fd);
else
rootfd = -1;
} else {
rootfd = open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
if (rootfd < 0) {
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
return DELETE_WALK_ERROR;
}
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
bool all_removed = false;
bool ok =
delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips, skip_count,
protect_rules, false, &all_removed, observer, observer_context);
if (close(rootfd) != 0)
ok = false;
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
if (deleted_out)
*deleted_out = budget.deleted;
if (skipped_out)
*skipped_out = budget.skipped;
if (!ok)
return DELETE_WALK_ERROR;
return budget.limit_hit ? DELETE_WALK_LIMIT_REACHED : DELETE_WALK_OK;
}
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, size_t* deleted_out,
size_t* skipped_out) {
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
skip_count, protect_rules, deleted_out, skipped_out, NULL,
NULL);
}
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL, NULL) ==
DELETE_WALK_OK;
}
bool has_path_traversal(const char* path) {
if (!path)
return true;
-94
View File
@@ -106,101 +106,7 @@ int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* sp
ssize_t utils_getdelim_bounded(FILE* stream, char** line, size_t* cap, int delim, size_t max_len);
char* path_cat(const char* path1, const char* path2);
bool glob_match(const char* pattern, const char* str);
/* Result of a bounded extra-file deletion run. */
typedef enum {
/* Every extra entry was removed (or there were none). */
DELETE_WALK_OK = 0,
/* The numeric cap for this run was reached before every extra was removed.
The walker removed exactly the entries the cap allowed and skipped (without
removing) the rest, matching rsync's partial --max-delete behavior. */
DELETE_WALK_LIMIT_REACHED,
/* A traversal or unlink failure aborted the deletion (partial removal is
possible, mirroring the delete pass). */
DELETE_WALK_ERROR
} DeleteWalkResult;
/* One protected entry for the delete walker. When top_level_only is true the
prefix is skipped only as a DIRECT child of dest_root (the --delay-updates
staging directory, which must not hide genuine extras inside a nested
destination directory that happens to share the staging name); otherwise the
prefix is skipped at any depth (the --compare-dest/--copy-dest/--link-dest
basis trees, and the sender-side protected filter-excluded prefixes, which
are never destination content). */
typedef struct {
const char* prefix;
bool top_level_only;
} DeleteSkipEntry;
/* True when child_rel is, or lies below, one of the protected entries (a prefix
"a" protects "a" and "a/b/c" but not "ab"; top_level_only entries protect
only DIRECT children of the destination root, i.e. child_rel has no '/'). */
bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSkipEntry* skips,
int skip_count);
/* One destination-directory entry collected up front so the delete walkers can
reproduce rsync's traversal order instead of readdir() order. rsync processes
a directory's extraneous subdirectories first (descending name, depth-first),
then its extraneous files (descending name), and only afterwards descends into
its kept subdirectories (ascending name). */
typedef struct {
char* name;
bool is_dir;
} DeleteDirEntry;
/* Collect the entries of the directory open on `dirfd` (excluding "." and ".."),
stat'ing each with AT_SYMLINK_NOFOLLOW. On success *out is a malloc'd array of
*count entries whose names the caller frees with delete_dir_entries_free().
Returns false on an allocation/readdir failure; a vanished entry (ENOENT) is
skipped, any other stat failure is reported through *operation_ok while the
walk continues. */
bool delete_dir_entries_collect(int dirfd, DeleteDirEntry** out, size_t* count, bool* operation_ok);
void delete_dir_entries_free(DeleteDirEntry* entries, size_t count);
/* Sort comparators: `_desc` orders subdirectories before files and each group by
descending name (rsync's extraneous-entry order); `_asc` orders plain ascending
name (rsync's kept-subdirectory order). */
int delete_dir_entry_cmp_desc(const void* a, const void* b);
int delete_dir_entry_cmp_asc(const void* a, const void* b);
/* Remove files/dirs/symlinks under dest_root that are not listed in manifest
without ever descending into a protected prefix (see DeleteSkipEntry). When
`synced_dirs` is non-NULL, extras are only removed directly inside a directory
whose destination-relative path is an exact entry in that list (the receive
root is the "." sentinel); directories outside the synchronized set are still
descended into so kept content below a listed directory is preserved, but
nothing in them is removed. A NULL `synced_dirs` keeps the legacy behavior of
treating the whole destination tree as deletable. `max_delete` caps the
number of removed entries (SIZE_MAX = unlimited): the walker removes up to the
cap and returns DELETE_WALK_LIMIT_REACHED when more extras remained.
`deleted_out`/`skipped_out` optionally receive the number of entries removed
and the number skipped because of the cap. */
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, size_t* deleted_out,
size_t* skipped_out);
/* Optional per-deletion observer: called for each destination-relative path
actually removed (a file, symlink, or directory), in removal order, so the
receiver can stream rsync's `--info=del`/`--info=remove` lines. */
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
* observer; the observer is invoked only for entries truly removed. When
* `protect_rules` is non-NULL its receiver-side verdict is evaluated for every
* candidate extra: a first-match PROTECT leaves the entry (and, for a
* directory, its whole subtree) in place, while RISK/NONE fall through to the
* ordinary skip-prefix/keep-set logic. */
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer,
void* observer_context);
/* Read-only companion to delete_extras_limited: walk the destination exactly as
the delete pass would and APPEND (strdup'd) destination-relative paths that
WOULD be removed, without touching disk. Used for -n/--dry-run --delete
would-delete reporting. Returns true on a clean walk; the caller owns the
strings appended to `out` and receives their count in *count_out. */
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out);
bool delete_extras(const char* dest_root, const ArrayList* manifest);
/* Open the existing destination directory at `dest_root`, confined to the
authorized root with an O_NOFOLLOW component walk (the same confinement the
deletion walker uses for its root). Returns a new fd the caller owns, or -1
+159 -38
View File
@@ -7,6 +7,7 @@
#include "utils.h"
#include "file_types.h"
#include <errno.h>
#include <limits.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
@@ -133,16 +134,23 @@ static bool xattr_name_is_posix_acl(const char* name) {
/* ---- SENDER: capture ---- */
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
/* The two syscall families differ only in whether the FINAL component is
* followed (`listxattr`/`getxattr` follow; `llistxattr`/`lgetxattr` do not), so
* one common implementation backs both public entry points. */
typedef ssize_t (*XattrListFn)(const char* path, char* list, size_t size);
typedef ssize_t (*XattrGetFn)(const char* path, const char* name, void* value, size_t size);
static FileXattrList* xattr_capture_common(const char* path, bool preserve_acls,
XattrListFn list_fn, XattrGetFn get_fn) {
if (!path)
return NULL;
ssize_t list_size = listxattr(path, NULL, 0);
ssize_t list_size = list_fn(path, NULL, 0);
if (list_size <= 0)
return NULL; /* no xattrs, ENOTSUP, or error: nothing appliable */
char* names = malloc((size_t)list_size);
if (!names)
return NULL;
ssize_t got = listxattr(path, names, (size_t)list_size);
ssize_t got = list_fn(path, names, (size_t)list_size);
if (got < 0) {
free(names);
return NULL;
@@ -165,7 +173,7 @@ FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
negotiated. Without it a plain -X capture never carries an ACL. */
if (!xattr_name_appliable(name, preserve_acls))
continue;
ssize_t value_size = getxattr(path, name, NULL, 0);
ssize_t value_size = get_fn(path, name, NULL, 0);
if (value_size < 0)
continue;
if (value_size > XATTR_VALUE_MAX)
@@ -178,7 +186,7 @@ FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
free(names);
return NULL;
}
ssize_t read_len = getxattr(path, name, buffer, (size_t)value_size);
ssize_t read_len = get_fn(path, name, buffer, (size_t)value_size);
if (read_len < 0 || read_len != value_size) {
free(buffer);
continue;
@@ -200,6 +208,14 @@ FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
return list;
}
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls) {
return xattr_capture_common(path, preserve_acls, listxattr, getxattr);
}
FileXattrList* xattr_capture_path_nofollow(const char* path, bool preserve_acls) {
return xattr_capture_common(path, preserve_acls, llistxattr, lgetxattr);
}
/* ---- WIRE ---- */
bool xattr_send(int fd, const FileXattrList* list) {
@@ -363,16 +379,74 @@ bool xattr_apply_fd(int fd, const FileXattrList* list) {
return true;
}
/* ---- --fake-super: park ownership/mode/mtime in a reserved xattr ---- */
/* Symlink counterpart of xattr_apply_fd(): target the link ITSELF, never its
* referent. fsetxattr cannot be used (no *at xattr syscall exists, and the
* kernel rejects xattr syscalls on an O_PATH descriptor), so the already-open,
* confinement-checked parent directory is addressed through /proc/self/fd and
* the final component is applied with lsetxattr, which does not follow it.
*
* The list is trusted to come from xattr_receive() (already whitelisted), but
* every name is re-validated here so this path-based primitive is confined on
* its own -- this is the only apply primitive that addresses a path, and the
* header promises a whitelisted apply. The apply is best-effort: if /proc is
* not mounted (the anchor cannot be formed) or the kernel refuses the set, the
* failure is skipped and never fails the transfer. See xattr.h for the bounded
* residual TOCTOU between link creation and lsetxattr. */
bool xattr_apply_path_nofollow(int parent_fd, const char* leaf, const FileXattrList* list,
bool preserve_acls) {
if (parent_fd < 0 || !leaf || leaf[0] == '\0' || strchr(leaf, '/') != NULL || !list)
return false;
if (list->count == 0)
return true;
char prefix[64];
int prefix_len = snprintf(prefix, sizeof(prefix), "/proc/self/fd/%d/", parent_fd);
if (prefix_len < 0 || (size_t)prefix_len >= sizeof(prefix))
return false;
size_t leaf_len = strlen(leaf);
char* path = malloc((size_t)prefix_len + leaf_len + 1);
if (!path)
return false;
memcpy(path, prefix, (size_t)prefix_len);
memcpy(path + prefix_len, leaf, leaf_len + 1);
bool warned = false;
int first_errno = 0;
for (int i = 0; i < list->count; i++) {
const FileXattr* xa = &list->items[i];
/* Defense in depth: re-validate against the receiver's full whitelist, so a
hand-crafted list can never apply a privileged namespace or the reserved
--fake-super key through this path-based primitive. */
if (!xattr_name_appliable(xa->name, preserve_acls))
continue;
if (lsetxattr(path, xa->name, xa->value, xa->value_len, 0) != 0) {
if (!warned) {
warned = true;
first_errno = errno;
}
}
}
if (warned)
log_message(LOG_LEVEL_WARNING,
"could not set one or more xattrs on the destination symlink: %s",
strerror(first_errno));
free(path);
return true;
}
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, int64_t mtime_sec,
int64_t mtime_nsec) {
/* ---- --fake-super: park ownership/mode/rdev in a reserved xattr ---- */
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, uint32_t rdev_major,
uint32_t rdev_minor) {
if (fd < 0)
return;
char record[128];
int len =
snprintf(record, sizeof(record), "%lu:%lu:%03o:%lld:%ld", (unsigned long)uid,
(unsigned long)gid, (unsigned)mode & 0777U, (long long)mtime_sec, (long)mtime_nsec);
/* rsync 3.4.1's exact grammar: "<octal full st_mode> <rdev_major>,<rdev_minor>
* <uid>:<gid>". The octal mode carries the S_IFMT bits (e.g. 0104711 for a
* setuid regular file, 020644 for a char device); the rdev pair is 0,0 for a
* non-device. No mtime field: rsync leaves the file's own timestamp in
* charge of mtime. This value is what rsync reads back to restore a
* fake-super tree, so the field order and separators must not change. */
char record[96];
int len = snprintf(record, sizeof(record), "%o %u,%u %u:%u", (unsigned)mode, (unsigned)rdev_major,
(unsigned)rdev_minor, (unsigned)uid, (unsigned)gid);
if (len <= 0 || (size_t)len >= sizeof(record))
return;
if (fsetxattr(fd, FAKESUPER_XATTR, record, (size_t)len, 0) != 0) {
@@ -381,11 +455,64 @@ void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, int6
}
}
/* --fake-super replay: read the freshly-stored record and re-apply mode/mtime
* fd-relative. The recorded uid/gid are retained for a later privileged
* restore but are NEVER chowned here: --fake-super only RECORDS ownership, it
* must not real-chown the recorded (resolved) owner. Mode/mtime still apply so
* unprivileged --fake-super keeps working. */
/* Parse rsync's `user.rsync.%stat` grammar strictly:
* "<octal st_mode> <rdev_major>,<rdev_minor> <uid>:<gid>"
* Every field is parsed with strtoul() so an out-of-range value is a clean
* rejection rather than the undefined behavior sscanf("%u") exhibited, each
* field is range-checked against the same bounds the wire validator uses, and
* the whole record must be consumed (only trailing whitespace is tolerated) so
* trailing garbage is refused. Returns false on any malformed input. */
static bool fake_super_parse_stat(const char* record, unsigned* mode_out, unsigned* rdev_major_out,
unsigned* rdev_minor_out, unsigned* uid_out, unsigned* gid_out) {
if (!record)
return false;
char* end = NULL;
const char* p = record;
errno = 0;
unsigned long mode = strtoul(p, &end, 8);
if (errno != 0 || end == p || mode > (unsigned long)UINT_MAX || *end != ' ')
return false;
p = end + 1;
errno = 0;
unsigned long rdev_major = strtoul(p, &end, 10);
if (errno != 0 || end == p || rdev_major > 0xffffUL || *end != ',')
return false;
p = end + 1;
errno = 0;
unsigned long rdev_minor = strtoul(p, &end, 10);
if (errno != 0 || end == p || rdev_minor > 0x00ffffffUL || *end != ' ')
return false;
p = end + 1;
errno = 0;
unsigned long uid = strtoul(p, &end, 10);
if (errno != 0 || end == p || uid > (unsigned long)UINT_MAX || *end != ':')
return false;
p = end + 1;
errno = 0;
unsigned long gid = strtoul(p, &end, 10);
if (errno != 0 || end == p || gid > (unsigned long)UINT_MAX)
return false;
p = end;
while (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r')
p++;
if (*p != '\0')
return false;
*mode_out = (unsigned)mode;
*rdev_major_out = (unsigned)rdev_major;
*rdev_minor_out = (unsigned)rdev_minor;
*uid_out = (unsigned)uid;
*gid_out = (unsigned)gid;
return true;
}
/* --fake-super replay: read the freshly-stored record and re-apply its
* permission bits fd-relative. The recorded uid/gid are retained for a later
* privileged restore but are NEVER chowned here: --fake-super only RECORDS
* ownership, it must not real-chown the recorded (resolved) owner. The
* recorded rdev is likewise parsed for grammar compatibility but is not acted
* on (device recreation is a separate, privilege-gated path). mtime is not in
* the record: the normal metadata path applies it (policy.times), exactly as
* rsync relies on the file's own timestamp. */
bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
if (fd < 0)
return false;
@@ -394,24 +521,25 @@ bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
if (len < 0)
return false; /* absent or filesystem without xattrs: silent no-op */
record[len] = '\0';
unsigned long ul_uid, ul_gid, ul_mode;
long long mtime_sec;
long mtime_nsec;
if (sscanf(record, "%lu:%lu:%lo:%lld:%ld", &ul_uid, &ul_gid, &ul_mode, &mtime_sec, &mtime_nsec) !=
5)
unsigned ul_mode, rdev_major, rdev_minor, ul_uid, ul_gid;
if (!fake_super_parse_stat(record, &ul_mode, &rdev_major, &rdev_minor, &ul_uid, &ul_gid))
return false; /* malformed record: skip, never fatal */
/* --fake-super NEVER performs a real chown: that would defeat the whole
point of the flag (record privileged ownership on an unprivileged receiver
for a later privileged restore). The uid/gid parsed above are retained in
the record for that later restore, but no ownership change happens here. */
/* --fake-super NEVER performs a real chown: that would defeat the whole point
of the flag (record privileged ownership on an unprivileged receiver for a
later privileged restore). The uid/gid parsed above are retained in the
record for that later restore, but no ownership change happens here. The
rdev is retained for the same reason. */
(void)rdev_major;
(void)rdev_minor;
(void)ul_uid;
(void)ul_gid;
/* Mode is applied only when the per-attribute policy asks for it, through the
SAME shared helper the normal metadata path uses (metadata_mode_for_policy):
under --perms the recorded source mode is copied exactly, including
group/other write and setuid/setgid/sticky bits (rsync parity), and the -E
rule derives exec bits from the destination's read bits exactly like
SAME shared helper the normal metadata path uses (metadata_mode_for_policy).
The recorded special bits are stripped first: rsync's fake-super receiver
stores the full mode in the xattr but never installs setuid/setgid/sticky on
the real file, so only the 0777 permission bits may be replayed. The -E
rule then derives exec bits from the destination's read bits exactly like
file_restore_metadata_fd. */
if (policy.perms || policy.executability) {
struct stat cur;
@@ -419,19 +547,12 @@ bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
if (fstat(fd, &cur) != 0) {
log_message(LOG_LEVEL_WARNING, "--fake-super: could not read destination mode: %s",
strerror(errno));
} else if (metadata_mode_for_policy((mode_t)ul_mode, cur.st_mode, policy, &want)) {
} else if (metadata_mode_for_policy((mode_t)(ul_mode & 0777U), cur.st_mode, policy, &want)) {
if (fchmod(fd, want) != 0)
log_message(LOG_LEVEL_WARNING,
"--fake-super: could not restore mode on destination file: %s",
strerror(errno));
}
}
if (policy.times) {
struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
{.tv_sec = (time_t)mtime_sec, .tv_nsec = mtime_nsec}};
if (futimens(fd, times) != 0)
log_message(LOG_LEVEL_WARNING,
"--fake-super: could not restore mtime on destination file: %s", strerror(errno));
}
return true;
}
+73 -23
View File
@@ -26,16 +26,22 @@
* and total bytes) on BOTH ends to prevent OOM/memory abuse; an oversized
* or malformed frame is a clean protocol rejection, never an allocation
* blowup.
* * Application is confined to the exact destination file descriptor
* (fsetxattr on the just-written fd), never a caller-controlled path.
* * Application is confined to the exact destination entry: fsetxattr on the
* just-written fd for regular files/directories, and for a symlink an
* lsetxattr on "/proc/self/fd/<parent_fd>/<leaf>" reached through the
* already-opened, confinement-checked parent directory -- never a
* caller-controlled path, and never following the link.
*/
/* Reserved key used by --fake-super to park the source's privileged ownership
* / mode / mtime on the destination file as an unprivileged user.* xattr, so a
* later privileged restore could re-apply them. Exact documented format:
* uid:gid:mode:mtime_sec:mtime_nsec (decimal, decimal, octal, dec, dec)
* e.g. "1000:1000:644:1765238400:0". */
#define FAKESUPER_XATTR "user.fastsync.stat"
* / mode / rdev on the destination file as an unprivileged user.* xattr, so the
* tree is interoperable with rsync 3.4.1 and a later privileged restore can
* re-apply them. This is rsync's own key and value grammar exactly:
* <octal st_mode with S_IFMT> <rdev_major>,<rdev_minor> <uid>:<gid>
* e.g. "104711 0,0 1234:5678" for a setuid regular file owned by 1234:5678,
* or "20644 1,3 111:222" for a char device. mtime is deliberately NOT part of
* the record: exactly like rsync, the file's own timestamp carries it. */
#define FAKESUPER_XATTR "user.rsync.%stat"
/* --- bounds --- */
#define XATTR_NAME_MAX 255 /* xattr names are limited to 255 bytes */
@@ -76,6 +82,17 @@ bool xattr_name_appliable(const char* name, bool preserve_acls);
* distinct from NULL. */
FileXattrList* xattr_capture_path(const char* path, bool preserve_acls);
/* Sender: like xattr_capture_path() but reads the xattrs of `path` ITSELF,
* never following a final symlink (llistxattr/lgetxattr). A symlink entry must
* use this so the scanner never captures the REFERENT's attributes onto the
* link (the path-following variant would). On Linux the VFS refuses to
* associate xattrs with symlinks at all, so this normally returns NULL; it is
* still correct and portable for a filesystem/platform that supports them.
* The same whitelist/bounds as xattr_capture_path() apply. Returns NULL when
* the link has no appliable xattrs (or the filesystem does not support them);
* an empty-but-valid list is never returned distinct from NULL. */
FileXattrList* xattr_capture_path_nofollow(const char* path, bool preserve_acls);
/* Wire: bounded serialization. xattr_send returns false on write failure; an
* empty/NULL list transmits a zero-count block. xattr_receive returns NULL and
* sets *ok = 0 on any malformed / oversized / non-whitelisted entry. When
@@ -91,24 +108,57 @@ FileXattrList* xattr_receive(int fd, int* ok, bool preserve_acls);
* true when apply was attempted (allowing callers to treat it as best-effort). */
bool xattr_apply_fd(int fd, const FileXattrList* list);
/* --fake-super: write the source uid/gid/mode/mtime record into the reserved
* FAKESUPER_XATTR on `fd`. Best-effort (logged, never fatal). Only meaningful
* when metadata was transmitted so the values exist. */
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, int64_t mtime_sec,
int64_t mtime_nsec);
/* Receiver: apply every entry to the symlink named by (parent_fd, leaf) WITHOUT
* following it, via lsetxattr() on the confined path
* "/proc/self/fd/<parent_fd>/<leaf>". Every incoming name is independently
* re-validated against xattr_name_appliable() with `preserve_acls`, exactly like
* xattr_apply_fd(): a non-whitelisted namespace (including the reserved
* --fake-super key) is skipped, so this primitive stays confined even if handed
* a hand-crafted list. A symlink cannot be targeted by the fd-relative
* fsetxattr() path: there is no *at() xattr syscall and the kernel rejects
* xattr syscalls on an O_PATH descriptor, so the already-opened,
* confinement-checked parent directory is the anchor and only the final
* component is the (no-follow) link. `leaf` must be a single path component.
*
* Portability: the "/proc/self/fd/<parent_fd>" anchor requires a mounted /proc.
* Where /proc is unavailable (or the fd cannot be addressed that way) the
* lsetxattr simply fails and is skipped -- the apply is best-effort exactly like
* xattr_apply_fd(), so no error is propagated and the transfer continues. A
* per-attribute failure (on Linux every set on a symlink fails with EPERM) is
* logged once and skipped, never fatal. Returns false only for an invalid
* anchor/list; true when an apply was attempted.
*
* Residual TOCTOU: `leaf` is a caller-supplied name resolved by path in the
* parent, so a local writer could replace the just-created symlink between its
* creation and lsetxattr(). This is bounded: it requires write access to the
* confinement-checked destination directory (already trusted), can only install
* a whitelisted user namespace or POSIX-ACL name, and never follows the link (a
* replacement symlink is still applied to as the final, no-follow component). */
bool xattr_apply_path_nofollow(int parent_fd, const char* leaf, const FileXattrList* list,
bool preserve_acls);
/* --fake-super: write the source uid/gid/mode/rdev record into the reserved
* FAKESUPER_XATTR on `fd`, using rsync 3.4.1's exact grammar (see the key
* comment above). `mode` is the full st_mode including its S_IFMT bits.
* Best-effort (logged, never fatal). Only meaningful when metadata was
* transmitted so the values exist. */
void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, uint32_t rdev_major,
uint32_t rdev_minor);
/* --fake-super replay: parse the FAKESUPER_XATTR record previously written on
* `fd` by fake_super_store_fd and re-apply mode/mtime fd-relative. The
* recorded uid/gid are deliberately NOT chowned for real: --fake-super only
* RECORDS ownership (the caller stores the resolved mapping via
* identity_resolve_storage_ids), it never performs a real chown. Best-effort:
* absence of the xattr or a malformed record is a silent no-op that never fails
* the transfer. The MODE leg is applied only when policy.perms||policy.
* executability and the MTIME leg only when policy.times, so the fake-super
* replay cannot bypass the per-attribute split; the mode follows the normal
* metadata path exactly (under --perms the source mode is copied verbatim,
* special and group/other write bits included).
* Returns true when the xattr was present and parsed. */
* `fd` by fake_super_store_fd and re-apply the recorded permission bits
* fd-relative. The recorded uid/gid are deliberately NOT chowned for real:
* --fake-super only RECORDS ownership (the caller stores the resolved mapping
* via identity_resolve_storage_ids), it never performs a real chown. The
* recorded rdev is retained for a later privileged restore but is not acted on
* here. Best-effort: absence of the xattr or a malformed record is a silent
* no-op that never fails the transfer. The MODE leg is applied only when
* policy.perms||policy.executability, and the recorded special bits
* (setuid/setgid/sticky) are NOT applied to the real file -- exactly like
* rsync's fake-super receiver, which stores the full mode in the xattr but
* strips the special bits on disk. mtime is not part of the record; the normal
* metadata path carries it (policy.times) exactly as rsync sets the file's own
* timestamp. Returns true when the xattr was present and parsed. */
bool fake_super_restore_fd(int fd, FileAttrPolicy policy);
#endif
+76 -10
View File
@@ -141,6 +141,7 @@ class DaemonManager:
def __init__(self):
self._proc = None
self._port = None
self.log_path = None
def start(self, config_path, port_override=None, extra_args=None, log_path=None):
self.stop()
@@ -154,7 +155,11 @@ class DaemonManager:
if extra_args:
cmd += extra_args
if log_path is None:
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
# A unique log per manager: several managers run in one xdist
# worker, and a shared log lets one daemon's truncate/write offset
# corrupt the other's appended lines (a flaky log assertion).
log_path = os.path.join(TEST_DATA_DIR, f"fastsyncd_{id(self):x}.log")
self.log_path = log_path
log = open(log_path, "w")
self._proc = subprocess.Popen(
cmd, stdout=log, stderr=log, stdin=subprocess.DEVNULL, start_new_session=True)
@@ -224,6 +229,7 @@ def daemon_env():
"\n"
"[files]\n"
"path = %s\n"
"read only = no\n"
"\n"
"[readonly]\n"
"path = %s\n"
@@ -231,18 +237,22 @@ def daemon_env():
"\n"
"[locked]\n"
"path = %s\n"
"read only = no\n"
"auth users = alice\n"
"\n"
"[team]\n"
"path = %s\n"
"read only = no\n"
"auth users = alice,bob\n"
"\n"
"[owner]\n"
"path = %s\n"
"read only = no\n"
"client owner = yes\n"
"\n"
"[denied]\n"
"path = %s\n"
"read only = no\n"
"hosts deny = 127.0.0.1\n"
% (config_port, FILES_MODULE, READONLY_MODULE, AUTH_MODULE, TEAM_MODULE, OWNER_MODULE,
DENIED_MODULE))
@@ -261,7 +271,8 @@ def daemon_env():
global DETACH_PORT
DETACH_PORT = _find_free_port()
with open(DETACH_CONF, "w") as f:
f.write("port = %d\n\n[detach]\npath = %s\n" % (DETACH_PORT, DETACH_MODULE))
f.write("port = %d\n\n[detach]\npath = %s\nread only = no\n"
% (DETACH_PORT, DETACH_MODULE))
yield
_kill_by_cmdline_marker(DETACH_CONF)
@@ -348,7 +359,8 @@ class TestDaemonModuleSelection:
the fix regresses."""
port = _find_free_port()
with open(UMASK_CONF, "w") as f:
f.write("port = %d\n\n[files]\npath = %s\n" % (port, FILES_MODULE))
f.write("port = %d\n\n[files]\npath = %s\nread only = no\n"
% (port, FILES_MODULE))
sub = os.path.join(FILES_MODULE, "umask_check")
shutil.rmtree(sub, ignore_errors=True)
os.makedirs(sub, exist_ok=True)
@@ -375,6 +387,58 @@ class TestDaemonModuleSelection:
proc.kill()
class TestRsyncConfigCompat:
"""A real rsyncd.conf can be pointed at FastSync: the common rsync GLOBAL
and MODULE keys are accepted, the ones with a FastSync equivalent (port,
path, read only, max connections) take effect, and the inert ones (pid
file, log file, comment, use chroot, uid, gid, exclude, timeout, ...) are
documented no-ops. --dparam accepts the same expanded key set."""
@pytest.mark.ci
def test_rsync_style_config_round_trip(self):
module = os.path.join(MODULE_ROOT, "rsync_style")
shutil.rmtree(module, ignore_errors=True)
os.makedirs(module, exist_ok=True)
port = _find_free_port()
conf = os.path.join(TEST_DATA_DIR, "fastsyncd_rsync_style.conf")
with open(conf, "w") as f:
f.write(
"# an rsync 3.4.1-style rsyncd.conf\n"
"pid file = /tmp/fastsyncd_rsync_style.pid\n"
"log file = /tmp/fastsyncd_rsync_style.log\n"
"socket options = TCP_NODELAY\n"
"use chroot = no\n"
"uid = nobody\n"
"gid = nogroup\n"
"timeout = 600\n"
"max verbosity = 2\n"
"transfer logging = yes\n"
"port = %d\n"
"\n"
"[rsync_style]\n"
"path = %s\n"
"comment = rsync-style module\n"
"use chroot = no\n"
"exclude = *.tmp\n"
"read only = no\n"
"max connections = 4\n"
% (port, module))
d = DaemonManager()
# --dparam borrows rsync's compact spelling; `pidfile` is inert but must
# not be rejected, proving dparam reuses the expanded global key set.
d.start(conf, extra_args=["--dparam", "pidfile=/tmp/rsync_style.pid"],
log_path=os.path.join(TEST_DATA_DIR, "fastsyncd_rsync_style.log"))
try:
result = _push("127.0.0.1::rsync_style", d.port)
assert result.returncode == 0, result.stderr or result.stdout
received = get_dest_received_dir(module, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"missing: {missing[:5]}"
assert not mismatches, f"mismatch: {mismatches[:5]}"
finally:
d.stop()
class TestDaemonRejection:
def _tree_files(self):
"""Snapshot every file path (module-relative) currently under the module
@@ -477,7 +541,7 @@ class TestDaemonRejection:
before any data lands. `accept` lists the log phrases that count as the
refusal (a non-root daemon refuses --copy-as earlier, at the privilege
check, so the caller accepts that phrase too)."""
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
log_path = daemon.log_path
before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
before_files = self._tree_files()
result, _ = run_client(SOURCE_DIR, f"127.0.0.1::{module}", port=daemon.port, flags=flags)
@@ -514,14 +578,13 @@ class TestDaemonRejection:
the refusal into a silent accept."""
port = _find_free_port()
d = DaemonManager()
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
try:
d.start(CONF_FILE, port_override=port,
extra_args=["--password-file", CRED_FILE, "--no-super"])
result, _ = run_client(SOURCE_DIR, "127.0.0.1::files", port=d.port,
flags=["--super", "--preserve"])
assert result.returncode != 0, "the --no-super daemon must refuse --super"
with open(log_path, "rb") as f:
with open(d.log_path, "rb") as f:
tail = f.read().decode("utf-8", "replace")
assert "client-chosen ownership" in tail, (
f"daemon did not log the --super refusal: {tail[-400:]!r}"
@@ -1010,7 +1073,7 @@ class TestDaemonAuthentication:
def test_auth_log_does_not_leak_password(self, daemon):
"""The daemon log must never contain the password or the store verifier."""
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
log_path = daemon.log_path
before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
_push_with_creds("127.0.0.1::locked", daemon.port, "alice", WRONG_PASS)
_push_with_creds("127.0.0.1::locked", daemon.port, "alice", ALICE_PASS)
@@ -1037,7 +1100,7 @@ class TestDaemonAuthentication:
_push_with_creds("127.0.0.1::locked", port, "alice", ALICE_PASS)
_push_with_creds("127.0.0.1::locked", port, "alice", WRONG_PASS)
time.sleep(0.3)
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
log_path = d.log_path
with open(log_path, "rb") as f:
log = f.read().decode("utf-8", "replace")
finally:
@@ -1072,7 +1135,8 @@ class TestDaemonMotd:
motd_line = "motd file = %s\n" % motd_path if motd_path else ""
os.makedirs(self.MOTD_MODULE, exist_ok=True)
with open(self.MOTD_CONF, "w") as f:
f.write("port = %d\n%s\n[files]\npath = %s\n" % (port, motd_line, self.MOTD_MODULE))
f.write("port = %d\n%s\n[files]\npath = %s\nread only = no\n"
% (port, motd_line, self.MOTD_MODULE))
d = DaemonManager()
d.start(self.MOTD_CONF, port_override=port)
return d, port
@@ -1256,12 +1320,12 @@ class TestDaemonTLSAuth:
_write_client_password_file(client_creds, "alice", ALICE_PASS)
d = DaemonManager()
port = _find_free_port()
log_path = os.path.join(TEST_DATA_DIR, "fastsyncd.log")
try:
d.start(CONF_FILE, port_override=port, extra_args=[
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
"--ca", certs["ca"], "--client-cn", "fastsync-client",
"--password-file", CRED_FILE])
log_path = d.log_path
before_files = _tree_file_count(AUTH_MODULE)
log_before = os.path.getsize(log_path) if os.path.exists(log_path) else 0
tls_flags = ["--tls",
@@ -1309,6 +1373,7 @@ class TestDaemonConnectionLimits:
"\n"
"[locked]\n"
"path = %s\n"
"read only = no\n"
"auth users = alice\n"
% (port, AUTH_MODULE))
d = DaemonManager()
@@ -1346,6 +1411,7 @@ class TestDaemonConnectionLimits:
"\n"
"[files]\n"
"path = %s\n"
"read only = no\n"
"max connections = 2\n"
% (port, FILES_MODULE))
d = DaemonManager()
+94 -3
View File
@@ -111,13 +111,20 @@ class _SlicingProxy:
"""
def __init__(self, target_port, forward_limit=None, hook=None, hook_after=0,
throttle=0.0, wait_for_reply=False):
throttle=0.0, wait_for_reply=False, hook_after_config_ack=False):
self.target = ("127.0.0.1", target_port)
self.forward_limit = forward_limit
self.hook = hook
self.hook_after = hook_after
self.throttle = throttle
self.wait_for_reply = wait_for_reply
# When set, the hook fires on the FIRST client->server bytes that follow
# the config-frame ack, BEFORE they are forwarded. For --delete-before
# those bytes are the keep-set manifest, so this runs the hook after the
# client's source pre-scan but before the receiver's delete ack releases
# the client into its data pass -- a deterministic late-file window.
self.hook_after_config_ack = hook_after_config_ack
self.config_acked = False
self.server_replied = threading.Event()
self.hook_called = threading.Event()
self.listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
@@ -165,6 +172,13 @@ class _SlicingProxy:
socks = []
break
data = data[:room]
if (self.hook_after_config_ack and self.config_acked and self.hook is not None
and not self.hook_called.is_set()):
# The first client bytes after the config ack are the
# pre-scan keep-set manifest: run the injection before
# forwarding so it is causally after the source scan.
self.hook()
self.hook_called.set()
backend.sendall(data)
forwarded += len(data)
self._maybe_hook(forwarded)
@@ -177,6 +191,7 @@ class _SlicingProxy:
client.sendall(data)
# Any server reply proves the receiver consumed the
# frames that precede it, so the hook barrier is met.
self.config_acked = True
self.server_replied.set()
self._maybe_hook(forwarded)
except OSError:
@@ -198,8 +213,11 @@ class _SlicingProxy:
def _maybe_hook(self, forwarded):
"""Fire the one-shot hook once its barrier is satisfied: enough client
bytes have been forwarded and, when ``wait_for_reply`` is set, the
server has sent a reply proving it processed the preceding frames."""
if self.hook is None or self.hook_called.is_set():
server has sent a reply proving it processed the preceding frames.
``hook_after_config_ack`` uses its own barrier (see ``_serve``), so the
byte/reply heuristic is bypassed entirely."""
if self.hook is None or self.hook_called.is_set() or self.hook_after_config_ack:
return
if forwarded < self.hook_after:
return
@@ -537,3 +555,76 @@ class TestDeleteDelayMaxDeleteRefilledDir:
assert os.path.isdir(later_dir), "later extra was not skipped by the budget"
# The one actual removal is reported.
assert _deleted_count(result.stdout) == 1, result.stdout
class TestDeleteBeforeLateFileParity:
"""rsync builds its file list once, so a source file created after that scan
is NOT transferred and its destination extra is deleted. FastSync used to
re-scan the source in its data pass (single-threaded) or pipeline a fresh
re-scan against the pre-scan keep-set (``--threads``) and would transfer the
late file (a safe superset); both paths now replay the pre-scan file list
instead, matching rsync.
The late file is injected through the config-ack barrier: the first client
bytes after the config ack are the pre-scan keep-set manifest, so the hook
runs causally after the source scan and before the receiver's delete ack
releases the client into its data pass.
For ``--threads`` the pipeline scanner runs concurrently with the sender, so
the injection must land while that re-scan is still in flight to be observed
by it. The source is therefore a tree of ``_N_DIRS`` directories: the
injection writes the late file into EVERY directory, so it is enough that
any one directory is still unscanned when the hook fires. The tree is sized
so the hook (a localhost round trip) lands long before a full scan finishes;
a re-scanning pipeline then transfers the late files for the directories it
has not yet reached, which the tree comparison catches.
"""
_N_DIRS = 2000
@requires_rsync
@pytest.mark.parametrize("mt", [False, True])
def test_late_source_file_not_transferred_and_extra_deleted(self, mt):
tag = f"dblate_mt{int(mt)}"
source = os.path.join(TEST_DATA_DIR, f"{tag}_src")
dest = os.path.join(TEST_DATA_DIR, f"{tag}_dst")
rsync_dst = os.path.join(TEST_DATA_DIR, f"{tag}_rsync_dst")
clean_dir(source)
clean_dir(dest)
clean_dir(rsync_dst)
for i in range(self._N_DIRS):
_write(os.path.join(source, f"dir{i:05d}", "keep.txt"), b"kept payload\n")
# Both destinations carry the would-be late file as an extra.
for root in (dest, rsync_dst):
received = get_dest_received_dir(root, source)
for i in range(self._N_DIRS):
_write(os.path.join(received, f"dir{i:05d}", "late.txt"), b"stale extra\n")
# rsync reference: the same source with no late file; the extras are
# removed and nothing is transferred for the (never-scanned) late paths.
rsync_result = _rsync(["-a", "--delete-before", source + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_tree = _tree(rsync_dst)
assert "dir00000/late.txt" not in rsync_tree
received = get_dest_received_dir(dest, source)
def hook():
# Runs after the pre-scan and before the receiver's delete ack.
for i in range(self._N_DIRS):
_write(os.path.join(source, f"dir{i:05d}", "late.txt"),
b"created after the scan\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
proxy = _SlicingProxy(server.port, hook=hook, hook_after_config_ack=True)
flags = ["--delete-before"] + (["--threads=4"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=proxy.port)
proxy.finish()
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert proxy.hook_called.is_set(), "late-file hook never fired"
assert _tree(received) == rsync_tree, (
f"late source {'multithreaded' if mt else 'single-threaded'} data pass re-scanned: "
f"{sum(1 for p in _tree(received) if p.endswith('late.txt'))} late files were "
"transferred"
)
+1 -1
View File
@@ -36,7 +36,7 @@ from common import ( # noqa: E402
verify_transfer,
)
PROTOCOL_VERSION = b"2.28.0"
PROTOCOL_VERSION = b"2.29.0"
STATUS_MANIFEST = 5
STATUS_OK = 0
+187 -27
View File
@@ -183,10 +183,11 @@ class TestDeviceSpecial:
)
@pytest.mark.setpriv
def test_devices_nonroot_receiver_skips_safely(self):
"""A receiver without CAP_MKNOD must skip a device entry with a warning
and never abort. A root runner drops the receiver (server) to nobody
via setpriv; on a non-root runner (or without setpriv) the test skips."""
def test_devices_nonroot_receiver_errors_like_rsync(self):
"""A receiver without CAP_MKNOD must report the failed device mknod as a
transfer error (rsync parity, partial failure) instead of silently
succeeding. A root runner drops the receiver (server) to nobody via
setpriv; on a non-root runner (or without setpriv) the test skips."""
if os.geteuid() != 0 or shutil.which("setpriv") is None:
pytest.skip("requires root + setpriv to run the receiver unprivileged")
self._setup()
@@ -202,16 +203,16 @@ class TestDeviceSpecial:
flags=["--devices"], port=port)
finally:
out, err = _stop_captured_server(server)
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:300]}"
received = get_dest_received_dir(DEVICE_DEST, DEVICE_SOURCE)
with open(os.path.join(received, "plain.txt")) as f:
assert f.read() == "regular content\n"
assert not os.path.lexists(os.path.join(received, "chardev")), (
"a receiver without CAP_MKNOD must skip the device node, not create it"
assert result.returncode != 0, (
f"a failed device mknod must be a transfer error like rsync (got exit 0): "
f"{(out + err)[:300]}"
)
assert ("cannot create device node" in (out + err)
or "device-node creation is not permitted" in (out + err)), (
f"receiver did not log the documented device skip: out={out!r} err={err!r}"
received = get_dest_received_dir(DEVICE_DEST, DEVICE_SOURCE)
assert not os.path.lexists(os.path.join(received, "chardev")), (
"a receiver without CAP_MKNOD must not create the device node"
)
assert "cannot create device" in (out + err), (
f"receiver did not log the device creation error: out={out!r} err={err!r}"
)
@pytest.mark.skipif(os.geteuid() != 0, reason="requires root to create device nodes")
@@ -2786,7 +2787,12 @@ class TestItemizeChanges:
flags=["--preserve", "-i", "--incremental"],
port=shared_server.port)
assert result.returncode == 0, f"incremental itemize failed: {result.stderr[:200]}"
itemized = [line for line in result.stdout.splitlines() if line and line[0] in ">.<c"]
# -i also emits the transfer-root and directory lines; only FILE entries
# matter here, so drop any line whose name has a trailing '/'.
itemized = [
line for line in result.stdout.splitlines()
if line and line[0] in ">.<c" and not line.rsplit(" ", 1)[-1].endswith("/")
]
assert itemized == [], f"unchanged files were itemized: {itemized[:5]}"
def test_multithreaded_emits_same_itemize_lines(self, shared_server):
@@ -6594,7 +6600,7 @@ class TestExtendedAttributes:
os.getxattr(os.path.join(received, "data.txt"), "user.foo")
def test_reserved_fake_super_key_not_forwarded(self, shared_server):
"""A source file that already carries the reserved user.fastsync.stat
"""A source file that already carries the reserved user.rsync.%stat
record must NOT have it planted on the receiver during a plain -X run
(it is receiver-only, so it cannot be spoofed for a later privileged
restore)."""
@@ -6604,7 +6610,7 @@ class TestExtendedAttributes:
fh.write(b"reserved\n")
if not _xattr_supported(f):
pytest.skip("filesystem does not support user xattrs")
os.setxattr(f, "user.fastsync.stat", b"0:0:644:0:0")
os.setxattr(f, "user.rsync.%stat", b"100644 0,0 0:0")
# A normal user.* attr still travels alongside.
os.setxattr(f, "user.keep", b"yes")
@@ -6614,7 +6620,7 @@ class TestExtendedAttributes:
received = get_dest_received_dir(dest, source)
assert os.getxattr(os.path.join(received, "data.txt"), "user.keep") == b"yes"
with pytest.raises(OSError):
os.getxattr(os.path.join(received, "data.txt"), "user.fastsync.stat")
os.getxattr(os.path.join(received, "data.txt"), "user.rsync.%stat")
@pytest.mark.ci
def test_xattrs_multithreaded(self, shared_server):
@@ -6631,6 +6637,68 @@ class TestExtendedAttributes:
received = get_dest_received_dir(dest, source)
assert os.getxattr(os.path.join(received, "data.txt"), "user.k") == b"v"
@pytest.mark.ci
def test_symlink_own_xattrs_never_referent(self, shared_server):
"""Protocol 2.29.0: a symlink's STATUS_SYMLINK frame carries a trailing
xattr block captured with llistxattr/lgetxattr (no follow) and applied
with lsetxattr on the link itself. Linux's VFS refuses to associate
xattrs with a symlink at all, so the portable guarantee asserted here is
the no-follow one: a referent that carries user.* must NOT have those
attributes appear on the destination symlink entry (the old
path-following capture would have copied the referent's attrs onto the
link). On a platform/filesystem that does support symlink xattrs the
full round-trip of the link's own attribute is asserted too."""
source, dest = self._source_and_dest("symlink_xattr")
target = os.path.join(source, "target.txt")
with open(target, "wb") as fh:
fh.write(b"referent payload\n")
if not _xattr_supported(target):
pytest.skip("filesystem does not support user xattrs")
os.setxattr(target, "user.referent-only", b"referent-value")
link = os.path.join(source, "link")
os.symlink("target.txt", link)
link_xattr_supported = False
try:
os.setxattr(link, "user.link-own", b"link-value", follow_symlinks=False)
link_xattr_supported = os.getxattr(
link, "user.link-own", follow_symlinks=False
) == b"link-value"
except (OSError, AttributeError, NotImplementedError):
link_xattr_supported = False
result, _ = run_client(source, dest, flags=["-aX"], port=shared_server.port)
assert result.returncode == 0, \
f"-aX symlink sync failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
dst_link = os.path.join(received, "link")
assert os.path.islink(dst_link), "destination link entry is not a symlink"
assert os.readlink(dst_link) == "target.txt"
# The no-follow guarantee. Checking only the link's own xattr list is
# vacuous on Linux (lsetxattr on a symlink always fails EPERM), so also
# prove the apply never followed the link: the destination REFERENT must
# keep its own user.* value untouched.
dst_target = os.path.join(received, "target.txt")
assert os.getxattr(dst_target, "user.referent-only") == b"referent-value", (
"the destination symlink apply followed the link and rewrote the "
"referent's xattr"
)
link_names = os.listxattr(dst_link, follow_symlinks=False)
assert "user.referent-only" not in link_names, (
"the destination symlink captured its REFERENT's xattr "
"(path-following capture bug)"
)
if sys.platform.startswith("linux"):
assert link_names == [], (
"Linux associates no xattrs with a symlink; the link entry must "
"carry none"
)
if link_xattr_supported:
assert os.getxattr(
dst_link, "user.link-own", follow_symlinks=False
) == b"link-value", "the symlink's own xattr did not round-trip"
@pytest.mark.ci
def test_acls_via_posix_acl_xattr(self, shared_server):
source, dest = self._source_and_dest("acl")
@@ -6703,10 +6771,17 @@ class TestExtendedAttributes:
assert result.returncode == 0, \
f"--fake-super sync failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
record = os.getxattr(os.path.join(received, "data.txt"), "user.fastsync.stat").decode()
fields = record.split(":")
assert len(fields) == 5
assert fields[0] == str(uid), f"reserved uid field {fields[0]} != source uid {uid}"
record = os.getxattr(os.path.join(received, "data.txt"), "user.rsync.%stat").decode()
# rsync 3.4.1 grammar: "<octal st_mode> <rdev_major>,<rdev_minor> <uid>:<gid>".
fields = record.split()
assert len(fields) == 3, f"unexpected rsync fake-super record {record!r}"
mode_field, rdev_field, owner_field = fields
assert rdev_field == "0,0", f"regular file rdev must be 0,0, got {rdev_field!r}"
assert int(mode_field, 8) & 0o170000 == stat.S_IFREG, (
f"recorded mode {mode_field!r} must carry S_IFREG"
)
assert owner_field.split(":")[0] == str(uid), \
f"recorded uid {owner_field!r} != source uid {uid}"
@pytest.mark.ci
def test_fake_super_records_resolved_chown_without_real_chown(self, shared_server):
@@ -6725,12 +6800,71 @@ class TestExtendedAttributes:
assert result.returncode == 0, \
f"--fake-super --chown sync failed: {(result.stderr or result.stdout)[:300]}"
dst = os.path.join(get_dest_received_dir(dest, source), "data.txt")
record = os.getxattr(dst, "user.fastsync.stat").decode().split(":")
assert record[0] == "33333", f"recorded owner {record[0]} != resolved 33333"
assert record[1] == "44444", f"recorded group {record[1]} != resolved 44444"
record = os.getxattr(dst, "user.rsync.%stat").decode().split()
owner = record[2].split(":")
assert owner == ["33333", "44444"], (
f"recorded owner {record[2]!r} != resolved 33333:44444"
)
st = os.stat(dst)
assert st.st_uid != 33333, "--fake-super must not real-chown the recorded owner"
@pytest.mark.ci
def test_fake_super_rsync_interop(self, shared_server):
"""A fake-super tree written by FastSync is readable by rsync 3.4.1:
rsync reads the `user.rsync.%stat` record (mode/rdev/uid:gid) and, when
it re-emits a fake-super tree, reproduces the same record. This pins
the on-disk key and value grammar against the real tool."""
rsync = shutil.which("rsync")
if rsync is None:
pytest.skip("rsync not installed")
source, dest = self._source_and_dest("fakesuper_interop")
f = os.path.join(source, "data.txt")
with open(f, "wb") as fh:
fh.write(b"interop\n")
if not _xattr_supported(f):
pytest.skip("filesystem does not support user xattrs")
# rsync's fake-super receiver only writes a %stat% record when it has
# something to fake; a root-owned file with a matching root stat is
# a no-op. When privileged, record a non-root owner so the round-trip
# actually exercises the parser (non-root CI already has a non-zero uid).
if os.geteuid() == 0:
try:
os.chown(f, 12345, 12346)
except OSError:
pass
# A setuid bit exercises the full st_mode encoding; set it AFTER any
# chown (chown clears setuid/setgid), and note that neither tool installs
# it on the real destination file.
os.chmod(f, 0o4711)
result, _ = run_client(source, dest, flags=["--fake-super"],
port=shared_server.port)
assert result.returncode == 0, \
f"--fake-super sync failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
rec = os.getxattr(os.path.join(received, "data.txt"), "user.rsync.%stat").decode()
rec_fields = rec.split()
assert len(rec_fields) == 3 and rec_fields[1] == "0,0", (
f"FastSync did not write rsync's stat grammar: {rec!r}"
)
assert int(rec_fields[0], 8) & 0o7777 == 0o4711, (
f"FastSync did not record the source mode in rsync's grammar: {rec!r}"
)
out = os.path.join(TEST_DATA_DIR, "fakesuper_interop_rsync")
clean_dir(out)
rs = subprocess.run([rsync, "-aX", "--fake-super",
received + "/", out + "/"],
capture_output=True, text=True, timeout=120)
assert rs.returncode == 0, (
f"rsync could not read FastSync's fake-super tree: {rs.stderr[:300]}"
)
out_rec = os.getxattr(os.path.join(out, "data.txt"), "user.rsync.%stat").decode()
assert out_rec == rec, (
"rsync re-emitted a different fake-super record; FastSync's grammar "
f"is not interoperable: ours={rec!r} rsync={out_rec!r}"
)
@pytest.mark.ci
def test_directory_xattrs_preserved(self, shared_server):
"""#286.3: -aX must preserve user.* xattrs on DIRECTORIES, not just files."""
@@ -6750,6 +6884,31 @@ class TestExtendedAttributes:
assert os.getxattr(received, "user.rootdir") == b"r"
assert os.getxattr(os.path.join(received, "sub"), "user.subdir") == b"s"
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_dirs_directory_xattr_applied(self, shared_server, mt):
"""#286.3: -d/-X must apply a transferred directory's user.* xattr at the
destination through the --dirs STATUS_MKDIR path (both the
single-threaded and -m/--threads receiver paths)."""
source, dest = self._source_and_dest("dirsxattr")
sub = os.path.join(source, "sub")
os.makedirs(sub)
if not _xattr_supported(sub):
pytest.skip("filesystem does not support user xattrs")
os.setxattr(sub, "user.dirsdir", b"dirs-value")
lst = os.path.join(TEST_DATA_DIR, "dirs_xattr_list.txt")
with open(lst, "wb") as fh:
fh.write(b"sub\n")
flags = ["--files-from", lst, "--dirs", "-R", "-X"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, \
f"--dirs -X sync failed: {(result.stderr or result.stdout)[:300]}"
received = os.path.join(dest, "sub")
assert os.path.isdir(received), "--dirs directory entry was not created"
assert os.getxattr(received, "user.dirsdir") == b"dirs-value", \
"the --dirs directory's user.* xattr was not applied at the destination"
@pytest.mark.ci
def test_directory_default_acl_preserved(self, shared_server):
"""#286.3: -aA must preserve a directory's default POSIX ACL (the
@@ -7225,9 +7384,10 @@ class TestCopyAs:
)
received = get_dest_received_dir(dest, source)
dst = os.path.join(received, "mixed.txt")
record = os.getxattr(dst, "user.fastsync.stat").decode().split(":")
assert (record[0], record[1]) == ("65534", "65534"), (
f"fake-super must record the resolved copy-as ownership: {record[:2]}"
record = os.getxattr(dst, "user.rsync.%stat").decode().split()
owner = record[2].split(":")
assert owner == ["65534", "65534"], (
f"fake-super must record the resolved copy-as ownership: {owner}"
)
st = os.lstat(dst)
assert (st.st_uid, st.st_gid) != (12345, 12346), (
+173 -5
View File
@@ -26,6 +26,17 @@ def _rsync(args):
)
def _file_entry_line(text):
"""The file entry line for a single-file transfer.
-i/--out-format emit the transfer-root (and directory) lines too, so the
file entry is not necessarily the first line; for the one-file corpora used
by the wire-counter tests it is the last non-empty line.
"""
lines = [line for line in text.splitlines() if line.strip()]
return lines[-1] if lines else ""
def _make_selection_tree(root):
clean_dir(root)
os.makedirs(os.path.join(root, "sub"))
@@ -184,6 +195,120 @@ class TestItemizeParity:
)
assert fast_lines == rsync_lines, f"rsync={rsync_lines} fastsync={fast_lines}"
@requires_rsync
@pytest.mark.ci
def test_itemize_directory_lines_match_rsync(self, shared_server):
"""#292: -i/--out-format emit rsync's directory lines (including the
transfer root) in rsync's depth-first order."""
source = os.path.join(TEST_DATA_DIR, "out_itemdir_src")
dest = os.path.join(TEST_DATA_DIR, "out_itemdir_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_itemdir_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub", "deep"))
os.makedirs(os.path.join(source, "emptydir"))
with open(os.path.join(source, "a.txt"), "wb") as fh:
fh.write(b"hello\n")
with open(os.path.join(source, "sub", "b.txt"), "wb") as fh:
fh.write(b"world\n")
with open(os.path.join(source, "sub", "deep", "d.txt"), "wb") as fh:
fh.write(b"deep\n")
clean_dir(dest)
clean_dir(rdst)
def dir_lines(text):
# Any line whose name ends with '/' is a directory entry.
return sorted(
line for line in text.splitlines()
if line.rsplit(" ", 1)[-1].endswith("/")
)
for fmt in (None, "%i %n%L"):
rsync_flags = ["-a", "-i"] if fmt is None else ["-a", "--out-format=" + fmt]
fast_flags = rsync_flags
clean_dir(rdst)
clean_dir(dest)
rsync_result = _rsync(rsync_flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=fast_flags,
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
expected = [l for l in dir_lines(rsync_result.stdout)
if not l.rsplit(" ", 1)[-1] == "./"]
fast = dir_lines(result.stdout)
assert [l for l in fast if not l.rsplit(" ", 1)[-1] == "./"] == expected, (
f"fmt={fmt} rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
assert ".d..t...... ./" in fast, f"missing root line: {result.stdout!r}"
@requires_rsync
@pytest.mark.ci
def test_itemize_info_flist_header_matches_rsync(self, shared_server):
"""`-i --info=flist` prints rsync's file-list header: the -i change
lines alone do not enable the flist category, but an explicit --info=flist
must not be suppressed when itemizing."""
source = os.path.join(TEST_DATA_DIR, "out_itemfl_src")
dest = os.path.join(TEST_DATA_DIR, "out_itemfl_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_itemfl_rdst")
_make_output_tree(source)
clean_dir(dest)
clean_dir(rdst)
flags = ["-a", "-i", "--info=flist"]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert "sending incremental file list" in rsync_result.stdout
assert "sending incremental file list" in result.stdout, result.stdout
# -i alone (no explicit --info=flist) must stay silent like rsync.
clean_dir(dest)
clean_dir(rdst)
rsync_plain = _rsync(["-a", "-i", source + "/", rdst + "/"])
plain, _ = run_client(source, dest, flags=["-a", "-i"],
port=shared_server.port)
assert "sending incremental file list" not in rsync_plain.stdout
assert "sending incremental file list" not in plain.stdout, plain.stdout
@requires_rsync
@pytest.mark.ci
def test_itemize_files_from_dirs_root_and_ancestors(self, shared_server):
"""The -d/--files-from dirs generator emits the transfer-root line and
rsync's implied ancestor directory lines. The generator traverses no
directories, so those must be synthesized from the listed entries."""
source = os.path.join(TEST_DATA_DIR, "out_itemff_src")
dest = os.path.join(TEST_DATA_DIR, "out_itemff_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_itemff_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub", "deep"))
with open(os.path.join(source, "sub", "deep", "d.txt"), "wb") as fh:
fh.write(b"deep\n")
clean_dir(dest)
clean_dir(rdst)
listing = os.path.join(TEST_DATA_DIR, "out_itemff.list")
with open(listing, "w") as fh:
fh.write("sub/deep/d.txt\n")
flags = ["-d", "-i", "--files-from=" + listing]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
def dir_lines(text):
return sorted(line for line in text.splitlines()
if line.rsplit(" ", 1)[-1].endswith("/"))
# rsync emits the implied parents (sub/, sub/deep/) but never the root
# here; FastSync emits the same set plus its unconditional root line.
expected = [line for line in dir_lines(rsync_result.stdout)
if not line.rsplit(" ", 1)[-1] == "./"]
fast = dir_lines(result.stdout)
assert [line for line in fast if not line.rsplit(" ", 1)[-1] == "./"] == expected, (
f"rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
assert "cd+++++++++ sub/" in fast, result.stdout
assert "cd+++++++++ sub/deep/" in fast, result.stdout
assert any(line.rsplit(" ", 1)[-1] == "./" for line in fast), result.stdout
@requires_rsync
@pytest.mark.ci
def test_itemize_modified_file_matches_rsync(self, shared_server):
@@ -262,6 +387,47 @@ class TestOutFormatParity:
if line:
assert pattern.match(line), f"bad %M format: {line!r}"
@requires_rsync
@pytest.mark.ci
def test_out_format_directory_metadata_with_delete_during(self):
"""--delete-during/--delete-delay reuse the per-directory plan pre-scan,
whose list carries no metadata. Directory %M/%B/%U/%G must still come
from the source, exactly as the plain recursive scan renders them."""
source = os.path.join(TEST_DATA_DIR, "out_fmtmeta_src")
dest = os.path.join(TEST_DATA_DIR, "out_fmtmeta_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_fmtmeta_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub", "deep"))
with open(os.path.join(source, "a.txt"), "wb") as fh:
fh.write(b"hello\n")
with open(os.path.join(source, "sub", "b.txt"), "wb") as fh:
fh.write(b"world\n")
clean_dir(dest)
clean_dir(rdst)
def dir_lines(text):
# Directory names are the last whitespace-separated token.
return sorted(line for line in text.splitlines()
if line.rsplit(" ", 1)[-1].endswith("/")
and line.rsplit(" ", 1)[-1] != "./")
for timing in ("--delete-during", "--delete-delay"):
for fmt in ("%M %n", "%B %n", "%U %G %n"):
clean_dir(dest)
clean_dir(rdst)
flags = ["-a", "--out-format=" + fmt, timing]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, result.stderr[:300]
assert dir_lines(result.stdout) == dir_lines(rsync_result.stdout), (
f"{timing} {fmt}: rsync={rsync_result.stdout!r} "
f"fastsync={result.stdout!r}"
)
assert "1970/" not in result.stdout, result.stdout
class TestListOnlyParity:
@requires_rsync
@@ -387,8 +553,8 @@ class TestWireStatsParity:
result, _ = run_client(source, dest, flags=["-a", "--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
rb, rl = (int(x) for x in rsync_result.stdout.split()[:2])
fb, fl = (int(x) for x in result.stdout.split()[:2])
rb, rl = (int(x) for x in _file_entry_line(rsync_result.stdout).split()[:2])
fb, fl = (int(x) for x in _file_entry_line(result.stdout).split()[:2])
assert rl == fl == 5000, (rsync_result.stdout, result.stdout)
assert rb > rl, f"rsync %b must include framing: {rsync_result.stdout!r}"
assert fb > fl, f"fastsync %b must include framing: {result.stdout!r}"
@@ -421,7 +587,9 @@ class TestWireStatsParity:
assert file_lines(result.stdout) == file_lines(rsync_result.stdout), (
f"rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
assert result.stdout.split()[0] == rsync_result.stdout.split()[0] == "16", (
fs_c = _file_entry_line(result.stdout).split()[0]
rs_c = _file_entry_line(rsync_result.stdout).split()[0]
assert fs_c == rs_c == "16", (
f"%c must be rsync's 16-byte sum header: {result.stdout!r}"
)
@@ -450,8 +618,8 @@ class TestWireStatsParity:
"--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
rs_c = int(rsync_result.stdout.split()[0])
fs_c = int(result.stdout.split()[0])
rs_c = int(_file_entry_line(rsync_result.stdout).split()[0])
fs_c = int(_file_entry_line(result.stdout).split()[0])
# No basis exists, so rsync still reports only its sum header.
assert rs_c == 16, rsync_result.stdout
# FastSync reports its own handshake bytes and is not aligned.
+3 -3
View File
@@ -133,14 +133,14 @@ def _seed_protocol_source(source):
class TestProtocol:
@pytest.mark.ci
def test_protocol_current_version_accepted(self, shared_server):
"""--protocol=2.28.0 (the current PROTOCOL_VERSION) is accepted and the
"""--protocol=2.29.0 (the current PROTOCOL_VERSION) is accepted and the
transfer completes normally."""
source = os.path.join(TEST_DATA_DIR, "proto_ok_src")
dest = os.path.join(TEST_DATA_DIR, "proto_ok_dst")
shutil.rmtree(dest, ignore_errors=True)
os.makedirs(dest)
_seed_protocol_source(source)
result, _ = run_client(source, dest, flags=["--protocol=2.28.0"],
result, _ = run_client(source, dest, flags=["--protocol=2.29.0"],
port=shared_server.port)
assert result.returncode == 0, \
f"--protocol current run failed: {(result.stderr or result.stdout)[:400]}"
@@ -157,7 +157,7 @@ class TestProtocol:
shutil.rmtree(dest, ignore_errors=True)
os.makedirs(dest)
_seed_protocol_source(source)
for bad in ("2.27.0", "2.26.0", "2.25.0", "2.24.0", "2.23.0", "2.22.0", "2.21.0", "2.20.0",
for bad in ("2.28.0", "2.27.0", "2.26.0", "2.25.0", "2.24.0", "2.23.0", "2.22.0", "2.21.0", "2.20.0",
"2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0", "216", "31"):
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
port=shared_server.port)
+4 -4
View File
@@ -353,10 +353,10 @@ class TestOwnershipRoot:
st = os.stat(dst)
assert st.st_uid != 12345, \
f"--fake-super -o must NOT real-chown the source owner, got uid={st.st_uid}"
record = os.getxattr(dst, "user.fastsync.stat").decode()
fields = record.split(":")
assert fields[0] == "12345", \
f"--fake-super must record the resolved owner, got {fields[0]}"
record = os.getxattr(dst, "user.rsync.%stat").decode()
owner = record.split()[2].split(":")
assert owner[0] == "12345", \
f"--fake-super must record the resolved owner, got {owner[0]}"
def test_o_applies_directory_owner(self, shared_server):
"""#286.2: -o must apply the source owner to DIRECTORIES too (the
+138
View File
@@ -0,0 +1,138 @@
"""Parity coverage for an absolute ``--temp-dir`` that lies inside the receive root.
FastSync confines the ``--temp-dir`` scratch directory to the receive root. It
previously rejected *every* absolute path; it now canonicalizes an absolute path
with ``realpath(3)`` and accepts it when it resolves inside the canonical receive
root (the destination is identical, so this is a pure parity win), while an
absolute path that escapes the root stays rejected with a clear error.
Two paths exercise the same receiver-side resolution:
* the local ``--read-batch`` apply (no network; the batch destination is the
receive root), and
* a real TCP transfer against the shared server (the destination root is the
client-supplied absolute path).
The out-of-root case asserts the run fails without writing a single scratch
file, so the confinement invariant is preserved.
"""
import os
import subprocess
import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import CLIENT_CMD, TEST_DATA_DIR, clean_dir, get_dest_received_dir, run_client
FILES = {
"top.txt": b"top level\n",
"sub/nested.txt": b"nested file\n" * 16,
}
def _run(args):
return subprocess.run(CLIENT_CMD + args, capture_output=True, text=True, timeout=180)
def _seed_source(root):
clean_dir(root)
for rel, data in FILES.items():
full = os.path.join(root, rel)
os.makedirs(os.path.dirname(full), exist_ok=True)
with open(full, "wb") as fh:
fh.write(data)
def _make_batch(tmp, source):
batch = os.path.join(tmp, "tree.batch")
result = _run(["--only-write-batch", batch, source])
assert result.returncode == 0, (result.stdout, result.stderr)
return batch
def _read(path):
with open(path, "rb") as fh:
return fh.read()
@pytest.mark.ci
def test_read_batch_absolute_temp_dir_inside_root_accepted(tmp_path):
source = os.path.join(tmp_path, "src")
dest = os.path.join(tmp_path, "dst")
_seed_source(source)
clean_dir(dest)
scratch = os.path.join(dest, "scratch")
os.makedirs(scratch)
# Stamp the scratch dir with an old mtime so the test can prove the receiver
# really created (and then removed) its temp file there: the directory mtime
# changes when an entry is created/removed, so a silently-ignored --temp-dir
# would leave the stamp untouched. An empty scratch dir alone does not
# distinguish "used and cleaned up" from "never used".
stale_mtime = 946684800 # 2000-01-01
os.utime(scratch, (stale_mtime, stale_mtime))
batch = _make_batch(str(tmp_path), source)
result = _run(["--read-batch", batch, dest, "--temp-dir", scratch])
assert result.returncode == 0, (result.stdout, result.stderr)
received = get_dest_received_dir(dest, source)
for rel, data in FILES.items():
assert _read(os.path.join(received, rel)) == data, f"content mismatch for {rel}"
assert os.listdir(scratch) == [], "scratch dir was not left clean"
assert os.stat(scratch).st_mtime != stale_mtime, (
"--temp-dir scratch dir was never written to (temp file not created there)"
)
@pytest.mark.ci
def test_read_batch_absolute_temp_dir_outside_root_rejected(tmp_path):
source = os.path.join(tmp_path, "src")
dest = os.path.join(tmp_path, "dst")
_seed_source(source)
clean_dir(dest)
outside = os.path.join(tmp_path, "outside")
os.makedirs(outside)
batch = _make_batch(str(tmp_path), source)
result = _run(["--read-batch", batch, dest, "--temp-dir", outside])
assert result.returncode != 0, "an absolute temp dir outside the receive root must be rejected"
assert os.listdir(outside) == [], "receiver wrote into an unconfined temp dir"
assert "temp-dir" in (result.stdout + result.stderr), (result.stdout, result.stderr)
def test_tcp_absolute_temp_dir_inside_root_accepted(shared_server):
source = os.path.join(TEST_DATA_DIR, "tempdir_abs_in_src")
dest = os.path.join(TEST_DATA_DIR, "tempdir_abs_in_dst")
_seed_source(source)
clean_dir(dest)
scratch = os.path.join(dest, "scratch")
os.makedirs(scratch)
# See test_read_batch_absolute_temp_dir_inside_root_accepted: the stale
# mtime makes actual scratch usage observable (the temp file creation and
# removal bump the directory mtime).
stale_mtime = 946684800 # 2000-01-01
os.utime(scratch, (stale_mtime, stale_mtime))
result, _ = run_client(source, dest, flags=["--temp-dir", scratch], port=shared_server.port)
assert result.returncode == 0, (result.stdout, result.stderr)[:300]
received = get_dest_received_dir(dest, source)
for rel, data in FILES.items():
assert _read(os.path.join(received, rel)) == data, f"content mismatch for {rel}"
assert os.listdir(scratch) == [], "scratch dir was not left clean"
assert os.stat(scratch).st_mtime != stale_mtime, (
"--temp-dir scratch dir was never written to (temp file not created there)"
)
def test_tcp_absolute_temp_dir_outside_root_rejected(shared_server):
source = os.path.join(TEST_DATA_DIR, "tempdir_abs_out_src")
dest = os.path.join(TEST_DATA_DIR, "tempdir_abs_out_dst")
_seed_source(source)
clean_dir(dest)
outside = os.path.join(TEST_DATA_DIR, "tempdir_abs_out_scratch")
clean_dir(outside)
result, _ = run_client(source, dest, flags=["--temp-dir", outside], port=shared_server.port)
assert result.returncode != 0, "an absolute temp dir outside the receive root must be rejected"
assert os.listdir(outside) == [], "receiver wrote into an unconfined temp dir"
+80
View File
@@ -0,0 +1,80 @@
"""End-to-end coverage for the `--temp-dir` EXDEV (cross-filesystem) fallback.
`file_to_disk_secure_impl` installs a completed temp file with `renameat(2)`;
when the scratch dir lives on a different filesystem the rename fails with
`EXDEV` and the engine retries with no scratch dir, writing the file directly in
the destination directory (a non-atomic copy), matching rsync.
The daemon receiver confines `--temp-dir` to the authorized receive root, so a
genuine cross-fs scratch there would require an in-root mount point. Bind/tmpfs
mounting is not permitted in the CI container (no `CAP_SYS_ADMIN`, and
unprivileged user namespaces are disabled), so this test reaches the exact same
code path through the local `--read-batch` apply instead: it has no
authorized-root confinement, so a relative `--temp-dir` that is a symlink to a
tmpfs (`/dev/shm`) is accepted and the final install then crosses filesystems.
"""
import os
import shutil
import subprocess
import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import CLIENT_CMD, get_dest_received_dir
TMPFS = "/dev/shm"
def _run(args):
return subprocess.run(CLIENT_CMD + args, capture_output=True, text=True, timeout=180)
def _read(path):
with open(path, "rb") as fh:
return fh.read()
def test_read_batch_temp_dir_cross_filesystem_fallback(tmp_path):
if not os.path.isdir(TMPFS):
pytest.skip("no /dev/shm tmpfs available to force a cross-filesystem install")
source = tmp_path / "src"
dest = tmp_path / "dst"
source.mkdir()
dest.mkdir()
files = {
"payload.bin": bytes(range(256)) * 64,
"sub/nested.txt": b"nested exdev fallback\n" * 8,
}
for rel, data in files.items():
full = source / rel
full.parent.mkdir(parents=True, exist_ok=True)
full.write_bytes(data)
batch = tmp_path / "tree.batch"
r = _run(["--only-write-batch", str(batch), str(source)])
assert r.returncode == 0, (r.stdout, r.stderr)
# A cross-filesystem scratch dir, reached through a relative --temp-dir
# symlink (the local batch apply performs no authorized-root confinement).
scratch = os.path.join(TMPFS, "fastsync_exdev_%d" % os.getpid())
shutil.rmtree(scratch, ignore_errors=True)
os.makedirs(scratch)
os.symlink(scratch, dest / "scratch")
try:
assert os.stat(scratch).st_dev != os.stat(dest).st_dev, (
"scratch and destination share a filesystem; EXDEV cannot be exercised"
)
r = _run(["--read-batch", str(batch), str(dest), "--temp-dir=scratch"])
assert r.returncode == 0, (r.stdout, r.stderr)
# The engine must report the non-atomic cross-fs fallback rather than
# silently claiming an atomic install.
assert "different filesystem" in (r.stdout + r.stderr), (r.stdout, r.stderr)
# The tree is still byte-exact and the scratch dir is left clean.
received = get_dest_received_dir(str(dest), str(source))
for rel, data in files.items():
assert _read(os.path.join(received, rel)) == data, f"content mismatch for {rel}"
assert os.listdir(scratch) == [], "cross-fs temp file was not cleaned up"
finally:
shutil.rmtree(scratch, ignore_errors=True)
+10 -3
View File
@@ -146,9 +146,16 @@ class TestTLSBasic:
assert not missing, f"Missing files: {missing}"
assert not mismatches, f"Mismatched files: {mismatches}"
@pytest.mark.xfail(reason="TLS multithreading has architectural limitations with per-thread SSL context")
@pytest.mark.ci
def test_tls_with_multithreading(self, certs):
"""TLS + multithreading."""
"""TLS + multithreading + --sendfile.
Exercises the TLS/sendfile interaction end to end: with --sendfile the
sender must route the file body through the buffered TLS path rather
than raw sendfile(2) on the encrypted socket. The focused decision
guard lives in tests/test_protocol.c
(test_tls_sendfile_decision_uses_buffered_path).
"""
clean_dir(DEST_DIR)
with ServerManager() as server:
server.start(extra_args=[
@@ -157,7 +164,7 @@ class TestTLSBasic:
])
result, dur = run_client(
SOURCE_DIR, DEST_DIR,
flags=["--threads", "--tls",
flags=["--threads", "--sendfile", "--tls",
"--cert", certs["client_cert"], "--key", certs["client_key"],
"--ca", certs["ca"]],
port=server.port,
+1 -1
View File
@@ -195,7 +195,7 @@ static void test_format_C_padding_uses_transfer_algo() {
{CHECKSUM_ALGO_NONE, 2},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
config->checksum_transfer_algo = cases[i].algo;
config->cli.checksum_transfer_algo = cases[i].algo;
char expected[64];
size_t n = 0;
expected[n++] = '[';
+278 -28
View File
@@ -10,6 +10,8 @@
#include "protocol.h"
#include "test_utils.h"
#include "utils.h"
#include <fnmatch.h>
#include <grp.h>
#include <pwd.h>
#include <stddef.h>
#include <stdint.h>
@@ -350,7 +352,7 @@ static void test_parse_args_protocol_accept_current() {
Config* cfg = valid_client_config();
EXPECT_NOT_NULL(cfg);
char* argv_equals[] = {"fastsync", "--source-dir", "/src",
"--dest-dir", "/dst", "--protocol=2.28.0"};
"--dest-dir", "/dst", "--protocol=2.29.0"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, argv_equals, positional_args, &positional_count), 0);
@@ -360,7 +362,7 @@ static void test_parse_args_protocol_accept_current() {
cfg = valid_client_config();
EXPECT_NOT_NULL(cfg);
char* argv_space[] = {"fastsync", "--source-dir", "/src", "--dest-dir",
"/dst", "--protocol", "2.28.0"};
"/dst", "--protocol", "2.29.0"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 7, argv_space, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->version, PROTOCOL_VERSION);
@@ -373,7 +375,7 @@ static void test_parse_args_protocol_rejects_other_versions() {
static const char* const bad_versions[] = {"2.17", "2.16", "2.15.0", "2.16.0", "2.17.0",
"2.18.0", "2.19.0", "2.20.0", "2.21.0", "2.22.0",
"2.23.0", "2.24.0", "2.25.0", "2.26.0", "2.27.0",
"216", "31", "abc", ""};
"2.28.0", "216", "31", "abc", ""};
for (size_t i = 0; i < sizeof(bad_versions) / sizeof(bad_versions[0]); i++) {
Config* cfg = valid_client_config();
EXPECT_NOT_NULL(cfg);
@@ -730,7 +732,7 @@ static void test_parse_args_port_alias() {
EXPECT_EQ_INT(cfg->server_port, 9000);
/* The default port is 8080; the explicit bit is what lets --dry-run tell an
explicit remote target from the default and route to the server. */
EXPECT_TRUE(cfg->server_port_set);
EXPECT_TRUE(cfg->cli.server_port_set);
config_delete(cfg);
cfg = config_create();
@@ -738,7 +740,7 @@ static void test_parse_args_port_alias() {
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_inline, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->server_port, 9001);
EXPECT_TRUE(cfg->server_port_set);
EXPECT_TRUE(cfg->cli.server_port_set);
config_delete(cfg);
cfg = config_create();
@@ -746,7 +748,7 @@ static void test_parse_args_port_alias() {
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_long, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->server_port, 9002);
EXPECT_TRUE(cfg->server_port_set);
EXPECT_TRUE(cfg->cli.server_port_set);
config_delete(cfg);
}
@@ -757,18 +759,18 @@ static void test_parse_args_server_host_sets_routing_bit() {
Config* cfg = config_create();
int positional_args[2];
int positional_count = 0;
EXPECT_FALSE(cfg->server_host_set);
EXPECT_FALSE(cfg->cli.server_host_set);
char* argv_space[] = {"fastsync", "--server-host", "example.test", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, argv_space, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->server_host, "example.test");
EXPECT_TRUE(cfg->server_host_set);
EXPECT_TRUE(cfg->cli.server_host_set);
config_delete(cfg);
cfg = config_create();
char* argv_inline[] = {"fastsync", "--server-host=example.test", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_inline, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->server_host_set);
EXPECT_TRUE(cfg->cli.server_host_set);
config_delete(cfg);
}
@@ -1726,7 +1728,7 @@ static void test_parse_args_no_preserve_blocks_implicit_metadata() {
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_FALSE(cfg->use_metadata);
EXPECT_TRUE(cfg->metadata_explicitly_disabled);
EXPECT_TRUE(cfg->cli.metadata_explicitly_disabled);
config_delete(cfg);
}
}
@@ -1777,7 +1779,7 @@ static void test_parse_args_checksum_choice_rejects_unsupported() {
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
EXPECT_EQ_INT(cfg->cli.cli_exit_code, 4);
config_delete(cfg);
}
}
@@ -1817,7 +1819,7 @@ static void test_parse_args_checksum_choice_new_algos() {
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv4, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, single[i]);
EXPECT_EQ_INT(cfg->checksum_transfer_algo, single[i]);
EXPECT_EQ_INT(cfg->cli.checksum_transfer_algo, single[i]);
config_delete(cfg);
}
@@ -1827,7 +1829,7 @@ static void test_parse_args_checksum_choice_new_algos() {
char* argv5[] = {"fastsync", "--cc=sha1,md4", "/checksum/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv5, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_transfer_algo, (int)CHECKSUM_ALGO_SHA1);
EXPECT_EQ_INT(cfg->cli.checksum_transfer_algo, (int)CHECKSUM_ALGO_SHA1);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD4);
config_delete(cfg);
@@ -1849,14 +1851,14 @@ static void test_parse_args_checksum_none_with_checksum_rejected() {
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
EXPECT_EQ_INT(cfg->cli.cli_exit_code, 4);
config_delete(cfg);
cfg = config_create();
char* argv2[] = {"fastsync", "--checksum", "--cc=md5,none", "/checksum/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv2, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
EXPECT_EQ_INT(cfg->cli.cli_exit_code, 4);
config_delete(cfg);
/* "none" as the TRANSFER checksum with a real pre-transfer checksum is
@@ -1958,7 +1960,7 @@ static void test_parse_args_compress_choice_parity() {
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
EXPECT_EQ_INT(cfg->cli.cli_exit_code, 4);
config_delete(cfg);
}
}
@@ -2078,6 +2080,9 @@ static void test_parse_args_temp_dir() {
config_delete(cfg);
}
/* --old-args is accepted for rsync CLI compatibility as a documented no-op (the
* remote server path is always safely quoted); it stores no Config field, so
* parsing it must simply succeed and leave the positional arguments intact. */
static void test_parse_args_old_args() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--old-args", "/src", "/dst"};
@@ -2085,7 +2090,7 @@ static void test_parse_args_old_args() {
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->old_args);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
@@ -2719,7 +2724,7 @@ static void test_parse_args_compression_env_list() {
cfg = config_create();
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
EXPECT_EQ_INT(cfg->cli.cli_exit_code, 4);
config_delete(cfg);
unsetenv("RSYNC_COMPRESS_LIST");
}
@@ -2734,7 +2739,7 @@ static void test_parse_args_checksum_env_list() {
setenv("RSYNC_CHECKSUM_LIST", "md5", 1);
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD5);
EXPECT_EQ_INT(cfg->checksum_transfer_algo, (int)CHECKSUM_ALGO_MD5);
EXPECT_EQ_INT(cfg->cli.checksum_transfer_algo, (int)CHECKSUM_ALGO_MD5);
config_delete(cfg);
/* An explicit --cc wins. */
@@ -2750,7 +2755,7 @@ static void test_parse_args_checksum_env_list() {
cfg = config_create();
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
EXPECT_EQ_INT(cfg->cli.cli_exit_code, 4);
config_delete(cfg);
unsetenv("RSYNC_CHECKSUM_LIST");
}
@@ -3534,6 +3539,243 @@ static void test_parse_args_usermap_rsync_forms() {
config_delete(cfg);
}
/* Independent oracle for the FROM name-glob tests: enumerate the sender's
* account database and fill `ids` with the DISTINCT ids whose name matches
* `glob`, sorted ascending. Returns the count, or -1 if the matching set
* exceeds `max` distinct ids -- production has no such bound on the number of
* candidates it scans, so a truncated set would under-count runs and flake on
* hosts with very large account databases. Callers must skip (not fail) on
* -1. */
static int cli_collect_glob_ids(const char* glob, bool is_group, int32_t* ids, int max) {
int n = 0;
bool overflow = false;
if (is_group) {
setgrent();
struct group* gr;
while ((gr = getgrent()) != NULL) {
if (fnmatch(glob, gr->gr_name, 0) != 0)
continue;
if ((unsigned long)gr->gr_gid > (unsigned long)INT32_MAX)
continue;
int32_t id = (int32_t)gr->gr_gid;
bool dup = false;
for (int i = 0; i < n; i++)
if (ids[i] == id)
dup = true;
if (dup)
continue;
if (n >= max) {
overflow = true;
break;
}
ids[n++] = id;
}
endgrent();
} else {
setpwent();
struct passwd* pw;
while ((pw = getpwent()) != NULL) {
if (fnmatch(glob, pw->pw_name, 0) != 0)
continue;
if ((unsigned long)pw->pw_uid > (unsigned long)INT32_MAX)
continue;
int32_t id = (int32_t)pw->pw_uid;
bool dup = false;
for (int i = 0; i < n; i++)
if (ids[i] == id)
dup = true;
if (dup)
continue;
if (n >= max) {
overflow = true;
break;
}
ids[n++] = id;
}
endpwent();
}
for (int i = 1; i < n; i++) {
int32_t key = ids[i];
int j = i - 1;
while (j >= 0 && ids[j] > key) {
ids[j + 1] = ids[j];
j--;
}
ids[j + 1] = key;
}
return overflow ? -1 : n;
}
static int cli_count_runs(const int32_t* ids, int n) {
int runs = 0;
for (int i = 0; i < n; i++) {
if (i == 0 || ids[i - 1] == INT32_MAX || ids[i] != ids[i - 1] + 1)
runs++;
}
return runs;
}
/* #294: a FROM name wildcard must expand, at CLI-parse time, against the
* sender's account database into numeric id/range rules. Prefer a prefix that
* matches >=2 DISTINCT NON-contiguous ids (exercising multi-rule expansion); if
* no such prefix exists on this host, fall back to one whose ids are contiguous
* (exercising range collapse). The expected rules are derived independently by
* enumerating the same database. */
static void test_parse_args_identity_map_from_name_glob(bool is_group) {
int32_t ids[512];
int chosen_n = 0;
int chosen_runs = 0;
char chosen_c = 0;
for (char c = 'a'; c <= 'z'; c++) {
const char glob[3] = {c, '*', '\0'};
int n = cli_collect_glob_ids(glob, is_group, ids, (int)(sizeof(ids) / sizeof(ids[0])));
if (n < 2)
continue; /* no matches, or the set overflowed the oracle's buffer */
int runs = cli_count_runs(ids, n);
if (runs > MAX_IDENTITY_MAP)
continue; /* production would reject this expansion; try another prefix */
if (runs >= 2 || chosen_c == 0) {
chosen_c = c;
chosen_n = n;
chosen_runs = runs;
}
if (runs >= 2)
break;
}
if (chosen_c == 0)
return; /* no multi-match prefix on this host (skipped, not failed) */
const char glob[3] = {chosen_c, '*', '\0'};
chosen_n = cli_collect_glob_ids(glob, is_group, ids, (int)(sizeof(ids) / sizeof(ids[0])));
if (chosen_n < 2)
return; /* account DB changed under us: skip, don't flake */
chosen_runs = cli_count_runs(ids, chosen_n);
EXPECT_TRUE(chosen_n >= 2);
char map_value[16];
snprintf(map_value, sizeof(map_value), "%s:@0", glob);
Config* cfg = config_create();
char* argv[] = {"fastsync", is_group ? "--groupmap" : "--usermap", map_value, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
int got = is_group ? cfg->groupmap_count : cfg->usermap_count;
EXPECT_EQ_INT(got, chosen_runs);
const IdentityMap* map = is_group ? cfg->groupmap : cfg->usermap;
/* Every matched id is covered by some expanded rule. */
for (int i = 0; i < chosen_n; i++) {
bool covered = false;
for (int r = 0; r < got; r++)
if (ids[i] >= map[r].from && ids[i] <= map[r].from_hi)
covered = true;
EXPECT_TRUE(covered);
}
/* Every id inside every expanded range is one the glob actually matched, so
* the range collapse cannot over-match a name that does not fit the glob. */
for (int r = 0; r < got; r++) {
EXPECT_EQ_INT(map[r].to, 0);
for (int32_t v = map[r].from; v <= map[r].from_hi; v++) {
bool expected = false;
for (int i = 0; i < chosen_n; i++)
if (ids[i] == v)
expected = true;
EXPECT_TRUE(expected);
if (v == INT32_MAX)
break;
}
}
config_delete(cfg);
}
static void test_parse_args_usermap_from_name_glob() {
test_parse_args_identity_map_from_name_glob(false);
}
static void test_parse_args_groupmap_from_name_glob() {
test_parse_args_identity_map_from_name_glob(true);
}
/* Regression for a leak in the FROM name-glob success path: the TO side is
* parsed into `parsed.to_name` before the glob is expanded, and every emitted
* rule takes its own str_dup of that name -- so the parse-time copy must be
* released before the branch continues. A numeric TO has to_name == NULL and
* cannot expose the leak, hence this uses a NAME TO. The name is resolved on
* the receiver (not here), so any well-formed non-glob name works. Run this
* under ASan/valgrind to catch the leak. */
static void test_parse_args_identity_map_from_name_glob_name_to(bool is_group) {
int32_t ids[512];
char chosen_c = 0;
for (char c = 'a'; c <= 'z'; c++) {
const char glob[3] = {c, '*', '\0'};
int n = cli_collect_glob_ids(glob, is_group, ids, (int)(sizeof(ids) / sizeof(ids[0])));
if (n < 2)
continue;
if (cli_count_runs(ids, n) > MAX_IDENTITY_MAP)
continue;
chosen_c = c;
break;
}
if (chosen_c == 0)
return; /* no multi-match prefix on this host (skipped, not failed) */
const char glob[3] = {chosen_c, '*', '\0'};
char map_value[32];
snprintf(map_value, sizeof(map_value), "%s:nobody", glob);
Config* cfg = config_create();
char* argv[] = {"fastsync", is_group ? "--groupmap" : "--usermap", map_value, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
int got = is_group ? cfg->groupmap_count : cfg->usermap_count;
EXPECT_TRUE(got >= 1);
const IdentityMap* map = is_group ? cfg->groupmap : cfg->usermap;
for (int r = 0; r < got; r++) {
EXPECT_EQ_INT(map[r].to, 0);
EXPECT_NOT_NULL(map[r].to_name);
if (map[r].to_name)
EXPECT_EQ_STR(map[r].to_name, "nobody");
}
config_delete(cfg);
}
static void test_parse_args_usermap_from_name_glob_name_to() {
test_parse_args_identity_map_from_name_glob_name_to(false);
}
static void test_parse_args_groupmap_from_name_glob_name_to() {
test_parse_args_identity_map_from_name_glob_name_to(true);
}
/* #294: an expansion that would push the map past MAX_IDENTITY_MAP must fail
* with a clear error rather than silently truncating. Prefill the map to the
* cap and then add a wildcard guaranteed to match at least the current user. */
static void test_parse_args_identity_map_from_name_glob_over_cap() {
const struct passwd* self = getpwuid(geteuid());
if (!self || self->pw_name[0] == '\0')
return;
char glob[8];
snprintf(glob, sizeof(glob), "%c*", self->pw_name[0]);
size_t need = (size_t)MAX_IDENTITY_MAP * 6 + strlen(glob) + 4 + 1;
char* value = malloc(need);
if (!value)
return;
size_t off = 0;
for (int i = 0; i < MAX_IDENTITY_MAP; i++)
off += (size_t)snprintf(value + off, need - off, "@0:@0,");
snprintf(value + off, need - off, "%s:@0", glob);
Config* cfg = config_create();
char* argv[] = {"fastsync", "--usermap", value, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
config_delete(cfg);
free(value);
}
/* #294: rsync refuses to mix --chown with --usermap/--groupmap on the same
* side (either order). --chown=USER conflicts with a prior --usermap;
* --chown=:GROUP conflicts with a prior --groupmap; the opposite side is fine. */
@@ -3713,7 +3955,8 @@ static void test_parse_args_rejects_malformed_identity() {
{"--usermap", "definitely_not_a_real_user_zzz:@1"},
{"--usermap", "0-"},
{"--usermap", "5-2:@1"},
{"--usermap", "roo*:@1"},
{"--usermap", "zzz_definitely_no_such_user_glob_zzz*:@1"},
{"--groupmap", "zzz_definitely_no_such_group_glob_zzz*:@1"},
{"--groupmap", "@1"},
{"--groupmap", "no_such_group_qqq:x"},
{"--chown", "a:b:c"},
@@ -4385,7 +4628,7 @@ static void test_parse_args_preserve_long_form() {
}
/* --no-perms/--no-times/--no-owner/--no-group (long and short) clear only
* their own attribute bit; they never set metadata_explicitly_disabled. */
* their own attribute bit; they never set cli.metadata_explicitly_disabled. */
static void test_parse_args_preserve_negations() {
struct {
const char* arg;
@@ -4413,7 +4656,7 @@ static void test_parse_args_preserve_negations() {
bool expected = all[j] != cases[i].offset;
EXPECT_TRUE(*(bool*)((char*)cfg + all[j]) == expected);
}
EXPECT_FALSE(cfg->metadata_explicitly_disabled);
EXPECT_FALSE(cfg->cli.metadata_explicitly_disabled);
/* -a's devices/specials keep the metadata frame on. */
EXPECT_TRUE(cfg->use_metadata);
config_delete(cfg);
@@ -4456,7 +4699,7 @@ static void test_parse_args_no_preserve_disables_bundle() {
EXPECT_FALSE(cfg->preserve_times);
EXPECT_FALSE(cfg->preserve_owner);
EXPECT_FALSE(cfg->preserve_group);
EXPECT_TRUE(cfg->metadata_explicitly_disabled);
EXPECT_TRUE(cfg->cli.metadata_explicitly_disabled);
EXPECT_TRUE(cfg->use_incremental);
EXPECT_FALSE(cfg->use_metadata);
config_delete(cfg);
@@ -4509,7 +4752,7 @@ static void test_parse_args_incremental_implies_preserve() {
EXPECT_EQ_INT(parse_args(cfg, 5, argv4, positional_args, &positional_count), 0);
EXPECT_FALSE(cfg->preserve_perms);
EXPECT_FALSE(cfg->preserve_times);
EXPECT_TRUE(cfg->metadata_explicitly_disabled);
EXPECT_TRUE(cfg->cli.metadata_explicitly_disabled);
EXPECT_FALSE(cfg->use_metadata);
config_delete(cfg);
}
@@ -4797,10 +5040,12 @@ static void test_parse_args_include_exclude_order() {
config_delete(cfg3);
}
/* OPT_NOOP compatibility flags (-s/--secluded-args, -r/--recursive) must never
* swallow the next argv: `fastsync -s SRC DST` keeps both positionals. */
/* OPT_NOOP compatibility flags (-s/--secluded-args, -r/--recursive, and the
* --inc-recursive/--no-inc-recursive scan-mode pair) must never swallow the
* next argv: `fastsync -s SRC DST` keeps both positionals. */
static void test_parse_args_noop_does_not_consume_argv() {
static const char* const noops[] = {"-s", "--secluded-args", "-r", "--recursive"};
static const char* const noops[] = {"-s", "--secluded-args", "-r",
"--recursive", "--inc-recursive", "--no-inc-recursive"};
for (size_t i = 0; i < sizeof(noops) / sizeof(noops[0]); i++) {
Config* cfg = config_create();
int positional_args[2];
@@ -4879,6 +5124,11 @@ void test_client_cli() {
test_parse_args_groupmap();
test_parse_args_usermap_name_resolution();
test_parse_args_usermap_rsync_forms();
test_parse_args_usermap_from_name_glob();
test_parse_args_groupmap_from_name_glob();
test_parse_args_usermap_from_name_glob_name_to();
test_parse_args_groupmap_from_name_glob_name_to();
test_parse_args_identity_map_from_name_glob_over_cap();
test_parse_args_identity_map_chown_conflict();
test_parse_args_chown();
test_parse_args_copy_as();
+12 -9
View File
@@ -2533,15 +2533,15 @@ static void test_config_derived_use_metadata() {
/* Incremental/delta imply metadata unless --no-preserve disabled it. */
c->use_incremental = true;
EXPECT_TRUE(config_derived_use_metadata(c));
c->metadata_explicitly_disabled = true;
c->cli.metadata_explicitly_disabled = true;
EXPECT_FALSE(config_derived_use_metadata(c));
c->metadata_explicitly_disabled = false;
c->cli.metadata_explicitly_disabled = false;
c->use_incremental = false;
c->use_delta = true;
EXPECT_TRUE(config_derived_use_metadata(c));
c->metadata_explicitly_disabled = true;
c->cli.metadata_explicitly_disabled = true;
EXPECT_FALSE(config_derived_use_metadata(c));
c->metadata_explicitly_disabled = false;
c->cli.metadata_explicitly_disabled = false;
c->use_delta = false;
/* Flags that must NOT imply metadata on their own. */
@@ -2924,7 +2924,7 @@ static void golden_config_populate(Config* c) {
array_list_add(c->filters, str_dup("- /sub/dir/"));
}
/* The pinned golden frame (protocol 2.28.0). The values below are the only
/* The pinned golden frame (protocol 2.29.0). The values below are the only
* thing that ties the generated table to the historical wire format; update
* them ONLY with a PROTOCOL_VERSION bump and a documented reason. The 2.24.0
* delete-plan wave changed only the version string; 2.25.0 appended the
@@ -2933,10 +2933,13 @@ static void golden_config_populate(Config* c) {
* appended the receiver-side delete-protection rule block (the STATUS_STATS
* body also grew, but that is not part of this frame). Track 5a appends the
* FastSync-only verify_basis bool to the basis block WITHOUT a version bump
* (project decision), so the frame grew by one int to 886 bytes. The
* byte-exact values are recomputed for the merged layout. */
* (project decision), so the frame grew by one int to 886 bytes. The 2.29.0
* symlink-xattr wave changes only the version string: the config-frame layout
* is unchanged (use_xattrs already crosses the wire); the STATUS_SYMLINK frame
* body grows instead. The byte-exact values are recomputed for the merged
* layout. */
#define GOLDEN_WIRE_LEN 886
#define GOLDEN_WIRE_HASH 5809509022716816757ULL
#define GOLDEN_WIRE_HASH 17827864270611927842ULL
static unsigned long long fnv1a_64(const unsigned char* buf, size_t len) {
unsigned long long h = 1469598103934665603ULL;
@@ -3018,7 +3021,7 @@ static unsigned long long capture_wire_hash(const Config* cfg, size_t* out_len)
return h;
}
/* Byte-for-byte wire compatibility guard (protocol 2.28.0). The expected hash
/* Byte-for-byte wire compatibility guard (protocol 2.29.0). The expected hash
* pins the pre-X-macro byte stream; the refactor MUST NOT change it. */
static void test_config_wire_golden() {
if (is_running_under_valgrind())
+353
View File
@@ -1,6 +1,7 @@
#include "test_daemon_conf.h"
#include "credentials.h"
#include "daemon_conf.h"
#include "log.h"
#include "test_utils.h"
#include <stdio.h>
#include <stdlib.h>
@@ -31,6 +32,9 @@ static void test_daemon_conf_create_defaults() {
EXPECT_EQ_INT(conf->global.port, DAEMON_CONF_DEFAULT_PORT);
EXPECT_NULL(conf->global.motd_file);
EXPECT_NULL(conf->global.address);
/* rsync modules are read-only unless they opt in, so the default must be
* true. */
EXPECT_TRUE(conf->global.read_only_default);
EXPECT_EQ_INT(conf->global.max_connections, DAEMON_CONF_DEFAULT_MAX_CONNECTIONS);
EXPECT_EQ_INT(conf->global.auth_failure_delay_ms, DAEMON_CONF_DEFAULT_AUTH_FAILURE_DELAY_MS);
EXPECT_EQ_INT(conf->global.max_connections_per_host,
@@ -626,6 +630,348 @@ static void test_daemon_conf_module_count_capped() {
EXPECT_TRUE(strstr(err, "too many modules") != NULL);
}
/* rsync rsyncd.conf compatibility: the common GLOBAL keys FastSync does not
* implement (pid file, log file, use chroot, uid/gid, timeout, ...) are
* accepted as documented inert keys, while the keys with a FastSync equivalent
* keep working and the compact rsync --dparam spellings (`pidfile`, `logfile`,
* `motdfile`) are recognized. A global `read only` is rsync's module default
* and must not be silently dropped. */
static void test_daemon_conf_rsync_global_keys() {
char* path;
char err[256];
EXPECT_EQ_INT(write_conf("pid file = /run/fastsyncd.pid\n"
"log file = /var/log/fastsyncd.log\n"
"socket options = TCP_NODELAY\n"
"listen backlog = 10\n"
"syslog facility = daemon\n"
"syslog tag = fastsyncd\n"
"use chroot = no\n"
"uid = nobody\n"
"gid = nogroup\n"
"timeout = 600\n"
"max verbosity = 3\n"
"lock file = /var/run/fastsyncd.lock\n"
"transfer logging = yes\n"
"strict modes = yes\n"
"reverse lookup = no\n"
"dont compress = *.gz\n"
"read only = yes\n"
"port = 8734\n"
"address = 127.0.0.1\n"
"pidfile = /run/other.pid\n"
"logfile = /var/log/other.log\n"
"motdfile = /etc/fastsync/motd.alt\n"
"\n"
"[pub]\n"
"path = /srv/pub\n"
"\n"
"[explicit]\n"
"path = /srv/explicit\n"
"read only = no\n",
&path),
0);
DaemonConf* conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(conf);
/* Mapped globals took effect; the compact aliases too. */
EXPECT_EQ_INT(conf->global.port, 8734);
EXPECT_EQ_STR(conf->global.address, "127.0.0.1");
EXPECT_EQ_STR(conf->global.motd_file, "/etc/fastsync/motd.alt");
/* The global `read only = yes` is the default for modules defined after it. */
EXPECT_TRUE(conf->global.read_only_default);
EXPECT_EQ_INT(conf->module_count, 2);
EXPECT_TRUE(conf->modules[0].read_only);
/* An explicit per-module value wins over the global default. */
EXPECT_FALSE(conf->modules[1].read_only);
daemon_conf_free(conf);
}
/* rsync module keys with no FastSync equivalent load inert; the keys with a
* FastSync meaning still map onto their native fields. */
static void test_daemon_conf_rsync_module_keys() {
char* path;
char err[256];
EXPECT_EQ_INT(write_conf("[data]\n"
"path = /srv/data\n"
"comment = Public data\n"
"use chroot = yes\n"
"uid = nobody\n"
"gid = nogroup\n"
"exclude = *.tmp\n"
"include = keep.tmp\n"
"exclude from = /etc/rsync.exclude\n"
"max verbosity = 2\n"
"lock file = /var/run/rsyncd.lock\n"
"transfer logging = yes\n"
"timeout = 300\n"
"secrets file = /etc/rsyncd.secrets\n"
"auth digest = sha256\n"
"numeric ids = yes\n"
"write only = no\n"
"list = yes\n"
"dont compress = *.gz\n"
"refuse options = delete\n"
"read only = yes\n"
"max connections = 5\n"
"hosts allow = 10.0.0.0/8\n"
"auth users = alice\n",
&path),
0);
DaemonConf* conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(conf);
EXPECT_EQ_STR(conf->modules[0].path, "/srv/data");
EXPECT_TRUE(conf->modules[0].read_only);
EXPECT_EQ_INT(conf->modules[0].max_connections, 5);
EXPECT_EQ_INT(conf->modules[0].hosts_allow_count, 1);
EXPECT_EQ_STR(conf->modules[0].hosts_allow[0], "10.0.0.0/8");
EXPECT_EQ_INT(conf->modules[0].auth_user_count, 1);
EXPECT_EQ_STR(conf->modules[0].auth_users[0], "alice");
daemon_conf_free(conf);
}
/* A genuinely unknown key is still rejected in both contexts, so accepting the
* rsync subset did not turn typos into silent no-ops. */
static void test_daemon_conf_rsync_unknown_keys_rejected() {
char* path;
char err[256];
EXPECT_EQ_INT(write_conf("bogus rsync key = 1\n", &path), 0);
const DaemonConf* conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NULL(conf);
EXPECT_TRUE(strstr(err, "unknown global key") != NULL);
EXPECT_EQ_INT(write_conf("[m]\npath = /x\nnot a real key = 1\n", &path), 0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NULL(conf);
EXPECT_TRUE(strstr(err, "unknown key 'not a real key'") != NULL);
}
/* A recognized rsync key with an invalid value is still a clear parse error. */
static void test_daemon_conf_rsync_read_only_invalid() {
char* path;
char err[256];
EXPECT_EQ_INT(write_conf("read only = maybe\n[m]\npath = /x\n", &path), 0);
const DaemonConf* conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NULL(conf);
EXPECT_TRUE(strstr(err, "read only") != NULL);
EXPECT_EQ_INT(write_conf("[m]\npath = /x\nread only = maybe\n", &path), 0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NULL(conf);
EXPECT_TRUE(strstr(err, "read only") != NULL);
}
/* --dparam reuses the same global dispatch: it accepts the compact rsync
* spellings and the inert rsync global keys, and `read only` sets the default
* for modules that did not set their own value. */
static void test_daemon_conf_dparam_rsync_keys() {
DaemonConf* conf = daemon_conf_create();
EXPECT_NOT_NULL(conf);
char err[256];
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "pidfile=/run/x.pid", err, sizeof(err)), 0);
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "pid file=/run/y.pid", err, sizeof(err)), 0);
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "logfile=/tmp/x.log", err, sizeof(err)), 0);
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "log file=/tmp/y.log", err, sizeof(err)), 0);
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "motdfile=/tmp/alt.motd", err, sizeof(err)), 0);
EXPECT_EQ_STR(conf->global.motd_file, "/tmp/alt.motd");
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "timeout=600", err, sizeof(err)), 0);
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "use chroot=no", err, sizeof(err)), 0);
/* A module already parsed without an explicit `read only` takes the
* --dparam default; an explicit module value is preserved. */
{
char* path;
EXPECT_EQ_INT(write_conf("[plain]\npath = /p\n[explicit]\npath = /e\nread only = no\n", &path),
0);
DaemonConf* loaded = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(loaded);
EXPECT_EQ_INT(daemon_conf_apply_dparam(loaded, "read only=yes", err, sizeof(err)), 0);
EXPECT_TRUE(loaded->global.read_only_default);
EXPECT_TRUE(loaded->modules[0].read_only);
EXPECT_FALSE(loaded->modules[1].read_only);
daemon_conf_free(loaded);
}
/* Invalid values and genuinely unknown keys are still rejected. */
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "read only=maybe", err, sizeof(err)), -1);
EXPECT_EQ_INT(daemon_conf_apply_dparam(conf, "definitely not rsync=1", err, sizeof(err)), -1);
EXPECT_TRUE(strstr(err, "unknown global key") != NULL);
daemon_conf_free(conf);
}
/* rsync modules default to READ-ONLY; `read only = no` / `write only = yes`
* opt a module into writability, and an explicit module value wins over a
* global default. */
static void test_daemon_conf_read_only_default_and_opt_in() {
char* path;
char err[256];
DaemonConf* conf;
/* A module that never mentions read only/write only is READ-ONLY, matching
* rsync (a migrated rsyncd.conf must not be served writable). */
EXPECT_EQ_INT(write_conf("[m]\npath = /x\n", &path), 0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(conf);
EXPECT_TRUE(conf->modules[0].read_only);
EXPECT_FALSE(conf->modules[0].read_only_explicit);
daemon_conf_free(conf);
/* `read only = no` opts in to writable. */
EXPECT_EQ_INT(write_conf("[m]\npath = /x\nread only = no\n", &path), 0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(conf);
EXPECT_FALSE(conf->modules[0].read_only);
EXPECT_TRUE(conf->modules[0].read_only_explicit);
daemon_conf_free(conf);
/* `write only = yes` opts in to writable (FastSync is push-only). */
EXPECT_EQ_INT(write_conf("[m]\npath = /x\nwrite only = yes\n", &path), 0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(conf);
EXPECT_FALSE(conf->modules[0].read_only);
EXPECT_TRUE(conf->modules[0].read_only_explicit);
daemon_conf_free(conf);
/* `write only = no` is rsync's default and does not undo read-only. */
EXPECT_EQ_INT(write_conf("[m]\npath = /x\nwrite only = no\n", &path), 0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(conf);
EXPECT_TRUE(conf->modules[0].read_only);
EXPECT_FALSE(conf->modules[0].read_only_explicit);
daemon_conf_free(conf);
/* A global `read only = no` is the default for later modules; an explicit
* module `read only`/`write only = yes` still wins. */
EXPECT_EQ_INT(write_conf("read only = no\n[a]\npath = /a\n"
"[b]\npath = /b\nread only = yes\n"
"[c]\npath = /c\nwrite only = yes\n",
&path),
0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(conf);
EXPECT_FALSE(conf->global.read_only_default);
EXPECT_FALSE(conf->modules[0].read_only);
EXPECT_TRUE(conf->modules[1].read_only);
EXPECT_FALSE(conf->modules[2].read_only);
daemon_conf_free(conf);
/* A global `read only = yes` keeps modules without an explicit value
* read-only. */
EXPECT_EQ_INT(write_conf("read only = yes\n[a]\npath = /a\n"
"[b]\npath = /b\nread only = no\n"
"[c]\npath = /c\nwrite only = yes\n",
&path),
0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NOT_NULL(conf);
EXPECT_TRUE(conf->global.read_only_default);
EXPECT_TRUE(conf->modules[0].read_only);
EXPECT_FALSE(conf->modules[1].read_only);
EXPECT_FALSE(conf->modules[2].read_only);
daemon_conf_free(conf);
/* An invalid `write only` value is a clear parse error. */
EXPECT_EQ_INT(write_conf("[m]\npath = /x\nwrite only = maybe\n", &path), 0);
conf = daemon_conf_load(path, err, sizeof(err));
free(path);
EXPECT_NULL(conf);
EXPECT_TRUE(strstr(err, "write only") != NULL);
}
/* Security-relevant rsync keys are accepted for migration but have no FastSync
* effect, so loading must warn loudly (naming the key and module) rather than
* letting an operator believe the restriction is enforced. */
static void test_daemon_conf_unenforced_security_keys_warned() {
char* path;
char err[256];
EXPECT_EQ_INT(write_conf("use chroot = yes\n"
"uid = nobody\n"
"[m]\n"
"path = /x\n"
"read only = no\n"
"secrets file = /etc/rsyncd.secrets\n"
"refuse options = delete\n"
"exclude = *.tmp\n"
"max size = 1M\n"
"pre-xfer exec = /bin/true\n"
"incoming chmod = F644\n"
"name converter = sh\n",
&path),
0);
set_log_level(LOG_LEVEL_WARNING);
FILE* log_capture = tmpfile();
EXPECT_NOT_NULL(log_capture);
log_set_file(log_capture);
DaemonConf* conf = daemon_conf_load(path, err, sizeof(err));
fflush(log_capture);
rewind(log_capture);
log_set_file(NULL);
free(path);
/* Inert security keys must never fail the load. */
EXPECT_NOT_NULL(conf);
EXPECT_FALSE(conf->modules[0].read_only);
bool saw_secrets = false;
bool saw_refuse = false;
bool saw_filter = false;
bool saw_size = false;
bool saw_hook = false;
bool saw_chmod = false;
bool saw_converter = false;
bool saw_global_chroot = false;
bool saw_global_uid = false;
char line[512];
while (fgets(line, sizeof(line), log_capture) != NULL) {
if (strstr(line, "NOT enforced") == NULL)
continue;
if (strstr(line, "module 'm'") && strstr(line, "secrets file"))
saw_secrets = true;
if (strstr(line, "module 'm'") && strstr(line, "refuse options"))
saw_refuse = true;
if (strstr(line, "module 'm'") && strstr(line, "exclude"))
saw_filter = true;
if (strstr(line, "module 'm'") && strstr(line, "max size"))
saw_size = true;
if (strstr(line, "module 'm'") && strstr(line, "pre-xfer exec"))
saw_hook = true;
if (strstr(line, "module 'm'") && strstr(line, "incoming chmod"))
saw_chmod = true;
if (strstr(line, "module 'm'") && strstr(line, "name converter"))
saw_converter = true;
if (strstr(line, "global key 'use chroot'"))
saw_global_chroot = true;
if (strstr(line, "global key 'uid'"))
saw_global_uid = true;
}
fclose(log_capture);
EXPECT_TRUE(saw_secrets);
EXPECT_TRUE(saw_refuse);
EXPECT_TRUE(saw_filter);
EXPECT_TRUE(saw_size);
EXPECT_TRUE(saw_hook);
EXPECT_TRUE(saw_chmod);
EXPECT_TRUE(saw_converter);
EXPECT_TRUE(saw_global_chroot);
EXPECT_TRUE(saw_global_uid);
daemon_conf_free(conf);
}
void test_daemon_conf() {
test_daemon_conf_create_defaults();
test_daemon_conf_full_parse();
@@ -645,4 +991,11 @@ void test_daemon_conf() {
test_daemon_conf_module_count_capped();
test_daemon_hosts_allowed();
test_daemon_module_name_valid();
test_daemon_conf_rsync_global_keys();
test_daemon_conf_rsync_module_keys();
test_daemon_conf_rsync_unknown_keys_rejected();
test_daemon_conf_rsync_read_only_invalid();
test_daemon_conf_dparam_rsync_keys();
test_daemon_conf_read_only_default_and_opt_in();
test_daemon_conf_unenforced_security_keys_warned();
}
+131 -6
View File
@@ -9,7 +9,9 @@
#include "charset.h"
#include "utils.h"
#include "protocol.h"
#include "xattr.h"
#include "test_utils.h"
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <stdlib.h>
@@ -17,7 +19,11 @@
#include <sys/stat.h>
#include <sys/sysmacros.h>
#include <sys/wait.h>
#include <sys/xattr.h>
#include <time.h>
#ifdef __linux__
#include <sys/prctl.h>
#endif
#include <unistd.h>
static void test_file_create() {
@@ -339,12 +345,17 @@ static void test_file_save_to_disk_temp_dir_confined() {
const char* root = "test_temp_confine_tmp";
const char* dest_file = "test_temp_confine_tmp/file.txt";
char outside[PATH_MAX];
char inside_abs[PATH_MAX];
snprintf(outside, sizeof(outside), "/tmp/fastsync_temp_outside_%d", (int)getpid());
unlink(dest_file);
rmdir("test_temp_confine_tmp/scratch");
rmdir("test_temp_confine_tmp/abs_scratch");
rmdir(root);
mkdir(root, 0755);
mkdir("test_temp_confine_tmp/scratch", 0755);
mkdir("test_temp_confine_tmp/abs_scratch", 0755);
if (!realpath("test_temp_confine_tmp/abs_scratch", inside_abs))
EXPECT_FAIL("realpath(abs_scratch) failed; inside_abs would be uninitialized");
mkdir(outside, 0755);
File* f = file_create("file.txt");
@@ -364,6 +375,13 @@ static void test_file_save_to_disk_temp_dir_confined() {
config->temp_dir = str_dup("../escape");
EXPECT_EQ_INT(file_save_to_disk_full(root, f, config), FILE_SAVE_ERROR);
EXPECT_EQ_INT(access(dest_file, F_OK), -1);
/* An absolute temp dir that canonicalizes INSIDE the receive root is
accepted and used (the parity win); destination is still written. */
free(config->temp_dir);
config->temp_dir = str_dup(inside_abs);
EXPECT_EQ_INT(file_save_to_disk_full(root, f, config), FILE_SAVE_WRITTEN);
EXPECT_EQ_INT(access(dest_file, F_OK), 0);
unlink(dest_file);
free(config->temp_dir);
config->temp_dir = str_dup("scratch");
EXPECT_EQ_INT(file_save_to_disk_full(root, f, config), FILE_SAVE_WRITTEN);
@@ -373,6 +391,7 @@ static void test_file_save_to_disk_temp_dir_confined() {
config_delete(config);
unlink(dest_file);
rmdir("test_temp_confine_tmp/scratch");
rmdir("test_temp_confine_tmp/abs_scratch");
rmdir(root);
rmdir(outside);
}
@@ -1315,6 +1334,110 @@ static void test_special_socket_recreated() {
rmdir(root);
}
/* --fake-super device round-trip (rsync parity): a char/block device must be
* materialized as a REGULAR empty file whose user.rsync.%stat records the real
* rdev -- never as an mknod'ed node -- even on a privileged receiver. This is
* the non-privileged unit counterpart to the setpriv integration test (which
* the PR gate excludes). */
static void test_fake_super_device_writes_regular_file_with_rdev() {
const char* root = "test_fake_super_dev_tmp";
const char* node = "test_fake_super_dev_tmp/cdev";
unlink(node);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->fake_super = true;
cfg->preserve_devices = true;
cfg->preserve_perms = true;
cfg->use_metadata = true;
cfg->use_xattrs = true;
FileMetadata meta;
memset(&meta, 0, sizeof(meta));
meta.mode = S_IFCHR | 0644;
File* f = file_create("cdev");
EXPECT_NOT_NULL(f);
f->is_special = true;
f->rdev_major = 1;
f->rdev_minor = 3;
f->metadata = &meta;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
struct stat st;
EXPECT_EQ_INT(lstat(node, &st), 0);
EXPECT_TRUE(S_ISREG(st.st_mode)); /* never a real device node */
EXPECT_EQ_INT((int)st.st_size, 0);
char value[128] = {0};
ssize_t got = getxattr(node, FAKESUPER_XATTR, value, sizeof(value) - 1);
EXPECT_TRUE(got > 0);
EXPECT_EQ_STR(value, "20644 1,3 0:0"); /* the REAL rdev, not 0,0 */
f->metadata = NULL;
file_destroy(f);
config_delete(cfg);
unlink(node);
rmdir(root);
}
/* A char/block device that mknodat() refuses (EPERM/EACCES on an unprivileged
* receiver) must be a PER-ENTRY failure -- FILE_SAVE_FAILED, which the receiver
* counts and continues past -- never the fatal FILE_SAVE_ERROR that aborts the
* stream. The unit suite normally runs as root, so drop the effective uid to
* make the kernel refusal deterministic. */
static void test_device_mknod_failure_is_per_entry() {
const char* root = "test_device_eperm_tmp";
const char* node = "test_device_eperm_tmp/cdev";
unlink(node);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0777), 0);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->preserve_devices = true;
cfg->use_metadata = true;
FileMetadata meta;
memset(&meta, 0, sizeof(meta));
meta.mode = S_IFCHR | 0644;
File* f = file_create("cdev");
EXPECT_NOT_NULL(f);
f->is_special = true;
f->rdev_major = 1;
f->rdev_minor = 3;
f->metadata = &meta;
uid_t saved = geteuid();
bool dropped = false;
if (saved == 0 && seteuid(65534) == 0)
dropped = true;
FileSaveResult result = file_save_to_disk_full(root, f, cfg);
if (dropped) {
EXPECT_EQ_INT(seteuid(saved), 0);
#ifdef __linux__
/* A setuid transition clears the process dumpable flag, which makes
* LeakSanitizer's ptrace-based thread suspension fail at exit. Restore it
* so the ASan/UBSan CI jobs can still run the leak check. */
(void)prctl(PR_SET_DUMPABLE, 1, 0, 0, 0);
#endif
}
EXPECT_EQ_INT(result, FILE_SAVE_FAILED);
/* Nothing was created: no device node and no regular-file fallback. */
struct stat st;
EXPECT_EQ_INT(lstat(node, &st), -1);
f->metadata = NULL;
file_destroy(f);
config_delete(cfg);
rmdir(root);
}
static void test_inplace_overwrite_truncates_shorter_payload() {
const char* root = "test_inplace_trunc_tmp";
const char* path = "test_inplace_trunc_tmp/big.txt";
@@ -2255,26 +2378,26 @@ static void test_basis_delete_relative_root_slash() {
EXPECT_NOT_NULL(cfg);
cfg->receive_root_directory = str_dup("/");
char* rel = file_receive_basis_delete_relative(cfg, "/a");
char* rel = delete_basis_relative(cfg, "/a");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "a");
free(rel);
rel = file_receive_basis_delete_relative(cfg, "/a/b");
rel = delete_basis_relative(cfg, "/a/b");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "a/b");
free(rel);
/* The root itself is not a child. */
EXPECT_NULL(file_receive_basis_delete_relative(cfg, "/"));
EXPECT_NULL(delete_basis_relative(cfg, "/"));
/* A relative entry is already root-relative. */
rel = file_receive_basis_delete_relative(cfg, "x/y");
rel = delete_basis_relative(cfg, "x/y");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "x/y");
free(rel);
/* An absolute path outside a non-"/" root is unreachable. */
free(cfg->receive_root_directory);
cfg->receive_root_directory = str_dup("/root");
EXPECT_NULL(file_receive_basis_delete_relative(cfg, "/other/a"));
rel = file_receive_basis_delete_relative(cfg, "/root/a");
EXPECT_NULL(delete_basis_relative(cfg, "/other/a"));
rel = delete_basis_relative(cfg, "/root/a");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "a");
free(rel);
@@ -2396,6 +2519,8 @@ void test_file() {
test_new_file_mode_honors_source_and_umask();
test_special_fifo_mode_honors_source_and_umask();
test_special_socket_recreated();
test_fake_super_device_writes_regular_file_with_rdev();
test_device_mknod_failure_is_per_entry();
test_inplace_overwrite_truncates_shorter_payload();
test_inplace_refuses_fifo_destination();
test_inplace_refuses_device_destination();
+502 -2
View File
@@ -1,7 +1,17 @@
#include "protocol.h"
#include "file.h"
#include "test_utils.h"
#include "utils.h"
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <openssl/evp.h>
#include <openssl/ssl.h>
#include <openssl/x509.h>
#include <poll.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <time.h>
#include <unistd.h>
#include <threads.h>
@@ -715,7 +725,7 @@ static void test_protocol_throttle_bytes_paces() {
struct timespec start;
clock_gettime(CLOCK_MONOTONIC, &start);
protocol_throttle_bytes(150000);
protocol_throttle_bytes(-1, 150000);
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
long long elapsed_ms =
@@ -736,7 +746,7 @@ static void test_protocol_throttle_bytes_unlimited() {
struct timespec start;
clock_gettime(CLOCK_MONOTONIC, &start);
protocol_throttle_bytes(100000000ULL);
protocol_throttle_bytes(-1, 100000000ULL);
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
long long elapsed_ms =
@@ -746,6 +756,490 @@ static void test_protocol_throttle_bytes_unlimited() {
protocol_session_unbind();
}
/* Regression for the plaintext sendfile path: it calls protocol_throttle_bytes()
* immediately after send_n_data(), which already bound legacy_io_session.write_fd
* to the wire fd. Resolving the throttle session with (read=-1, write=-1)
* mismatched that fd and re-initialized the legacy session, granting a *second*
* first-call burst and discarding the accumulated debt. This drives the same
* sequence and asserts the debt from send_n_data carries into the throttle. */
static void test_protocol_throttle_bytes_legacy_same_session() {
const size_t payload = 150000; /* 1.5x the 100 KB burst at --bwlimit=1 MB/s */
unsigned char* buffer = malloc(payload);
EXPECT_TRUE(buffer != NULL);
memset(buffer, 0, payload);
io_set_fds(-1, -1);
io_set_bwlimit(1000000ULL);
int fd = open("/dev/null", O_WRONLY);
EXPECT_TRUE(fd >= 0);
struct timespec start;
clock_gettime(CLOCK_MONOTONIC, &start);
/* send_n_data() consumes the whole 100 KB burst and sleeps ~50 ms. */
EXPECT_TRUE(send_n_data(fd, buffer, payload));
/* The throttle must share that session, so the 150 KB is all debt and sleeps
~150 ms (total ~200 ms). A re-initialized session would hand out a fresh
100 KB burst and sleep only ~50 ms (total ~100 ms). */
protocol_throttle_bytes(fd, payload);
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
long long elapsed_ms =
(now.tv_sec - start.tv_sec) * 1000LL + (now.tv_nsec - start.tv_nsec) / 1000000LL;
EXPECT_TRUE(elapsed_ms >= 150);
close(fd);
free(buffer);
io_set_bwlimit(0);
io_set_fds(-1, -1);
}
/* ------------------------------------------------------------------------- *
* Transport-vtable dispatch tests.
* ------------------------------------------------------------------------- */
static int dispatch_send_calls;
static int dispatch_recv_calls;
static ssize_t counting_send(ProtocolSession* session, const void* data, size_t size,
short* wait_events) {
dispatch_send_calls++;
ssize_t written = write(session->write_fd, data, size);
if (written < 0)
return errno == EINTR ? PROTOCOL_IO_RETRY : PROTOCOL_IO_ERROR;
if (written == 0)
return PROTOCOL_IO_ERROR;
*wait_events = POLLOUT;
return written;
}
static ssize_t counting_recv(ProtocolSession* session, void* data, size_t size,
short* wait_events) {
dispatch_recv_calls++;
ssize_t received = read(session->read_fd, data, size);
if (received < 0)
return errno == EINTR ? PROTOCOL_IO_RETRY : PROTOCOL_IO_ERROR;
if (received == 0)
return PROTOCOL_IO_CLOSED;
*wait_events = POLLIN;
return received;
}
static bool counting_has_pending(const ProtocolSession* session) {
(void)session;
return false;
}
static const ProtocolIoOps counting_ops = {
.send = counting_send,
.recv = counting_recv,
.has_pending = counting_has_pending,
};
/* A plain-TCP socketpair session must route every byte through the ops table:
* installing a counting ops wrapper proves the send/receive loops dispatch via
* session->ops instead of branching on session->ssl. */
static void test_protocol_dispatch_via_ops() {
int sv[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, sv), 0);
ProtocolSession sender;
ProtocolSession receiver;
protocol_session_init(&sender, sv[0], sv[0]);
protocol_session_set_bwlimit(&sender, 0);
protocol_session_init(&receiver, sv[1], sv[1]);
protocol_session_set_bwlimit(&receiver, 0);
EXPECT_NOT_NULL(sender.ops);
EXPECT_NOT_NULL(receiver.ops);
dispatch_send_calls = 0;
dispatch_recv_calls = 0;
sender.ops = &counting_ops;
receiver.ops = &counting_ops;
const char payload[] = "dispatch-through-vtable";
EXPECT_TRUE(protocol_send_n_data(&sender, payload, sizeof(payload)));
char received[sizeof(payload)] = {0};
EXPECT_TRUE(protocol_receive_n_data(&receiver, received, sizeof(received)));
EXPECT_EQ_INT(memcmp(payload, received, sizeof(payload)), 0);
EXPECT_TRUE(dispatch_send_calls > 0);
EXPECT_TRUE(dispatch_recv_calls > 0);
close(sv[0]);
close(sv[1]);
}
/* Retry-contract tests: an op that reports PROTOCOL_IO_RETRY once (and hands the
* loop a switched wait event) must be retried rather than treated as a fatal
* error or a close. The send/receive loops had no unit coverage for this path
* even though every TLS WANT_READ/WANT_WRITE and EINTR retry relies on it. */
static int retry_send_calls;
static short retry_send_last_wait;
static int retry_recv_calls;
static short retry_recv_last_wait;
static ssize_t retry_once_send(ProtocolSession* session, const void* data, size_t size,
short* wait_events) {
retry_send_calls++;
if (retry_send_calls == 1) {
/* Simulate a WANT_READ-style retry: switch the poll event and make no
* progress. The send loop must consume this and retry. */
*wait_events = POLLIN;
return PROTOCOL_IO_RETRY;
}
ssize_t written = write(session->write_fd, data, size);
if (written < 0)
return PROTOCOL_IO_ERROR;
if (written == 0)
return PROTOCOL_IO_ERROR;
*wait_events = POLLOUT;
retry_send_last_wait = *wait_events;
return written;
}
static ssize_t retry_once_recv(ProtocolSession* session, void* data, size_t size,
short* wait_events) {
retry_recv_calls++;
if (retry_recv_calls == 1) {
*wait_events = POLLOUT;
return PROTOCOL_IO_RETRY;
}
ssize_t received = read(session->read_fd, data, size);
if (received < 0)
return PROTOCOL_IO_ERROR;
if (received == 0)
return PROTOCOL_IO_CLOSED;
*wait_events = POLLIN;
retry_recv_last_wait = *wait_events;
return received;
}
static const ProtocolIoOps retry_send_ops = {
.send = retry_once_send,
.recv = counting_recv,
.has_pending = counting_has_pending,
};
static const ProtocolIoOps retry_recv_ops = {
.send = counting_send,
.recv = retry_once_recv,
.has_pending = counting_has_pending,
};
static void test_protocol_io_retry_contract() {
const char payload[] = "retry-contract";
/* The send loop: the first attempt reports RETRY and switches the poll event
* to POLLIN. A pre-seeded readable byte on the *opposite* end of the
* socketpair keeps that poll immediately satisfiable, so the retry is the
* only thing under test. */
int send_sv[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, send_sv), 0);
char seed = 'x';
EXPECT_EQ_INT(write(send_sv[1], &seed, 1), 1);
ProtocolSession sender;
protocol_session_init(&sender, send_sv[0], send_sv[0]);
protocol_session_set_bwlimit(&sender, 0);
sender.ops = &retry_send_ops;
retry_send_calls = 0;
retry_send_last_wait = 0;
EXPECT_TRUE(protocol_send_n_data(&sender, payload, sizeof(payload)));
EXPECT_EQ_INT(retry_send_calls, 2);
EXPECT_EQ_INT(retry_send_last_wait, POLLOUT);
close(send_sv[0]);
close(send_sv[1]);
/* The receive loop: the first attempt reports RETRY and switches the poll
* event to POLLOUT, which a socketpair read fd satisfies immediately. */
int recv_sv[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, recv_sv), 0);
EXPECT_EQ_INT((int)write(recv_sv[0], payload, sizeof(payload)), (int)sizeof(payload));
ProtocolSession receiver;
protocol_session_init(&receiver, recv_sv[1], recv_sv[1]);
protocol_session_set_bwlimit(&receiver, 0);
receiver.ops = &retry_recv_ops;
retry_recv_calls = 0;
retry_recv_last_wait = 0;
char received[sizeof(payload)] = {0};
EXPECT_TRUE(protocol_receive_n_data(&receiver, received, sizeof(received)));
EXPECT_EQ_INT(memcmp(payload, received, sizeof(payload)), 0);
EXPECT_EQ_INT(retry_recv_calls, 2);
EXPECT_EQ_INT(retry_recv_last_wait, POLLIN);
close(recv_sv[0]);
close(recv_sv[1]);
}
typedef struct {
ProtocolSession* session;
SSL* expected_ssl;
SSL* resolved_ssl;
SSL* thread_local_ssl;
} SslResolverWorkerArg;
static int ssl_resolver_worker(void* arg) {
SslResolverWorkerArg* worker = arg;
protocol_session_bind(worker->session);
worker->resolved_ssl = protocol_current_ssl();
worker->thread_local_ssl = io_get_ssl();
protocol_session_unbind();
return thrd_success;
}
/* The worker-thread bug fix: a thread that bound a TLS session but never ran
* the handshake has io_ssl == NULL, yet protocol_current_ssl() must return the
* session's SSL so callers pick the TLS path. */
static void test_protocol_current_ssl_prefers_bound_session() {
SSL_CTX* ctx = SSL_CTX_new(TLS_method());
EXPECT_NOT_NULL(ctx);
SSL* ssl = SSL_new(ctx);
EXPECT_NOT_NULL(ssl);
int sv[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, sv), 0);
ProtocolSession session;
protocol_session_init(&session, sv[0], sv[0]);
const ProtocolIoOps* plain_ops = session.ops;
protocol_session_set_ssl(&session, ssl);
/* set_ssl must select a distinct (TLS) dispatch table; protocol_current_ssl
* only returns a bound session's SSL for TLS ops, so arg.resolved_ssl == ssl
* below also proves the bound session's ops are the TLS ops. */
EXPECT_NOT_NULL(plain_ops);
EXPECT_TRUE(session.ops != plain_ops);
EXPECT_TRUE(session.ssl == ssl);
/* Clear the calling thread's legacy SSL: only the bound session carries it. */
io_set_fds(-1, -1);
SslResolverWorkerArg arg = {
.session = &session, .expected_ssl = ssl, .resolved_ssl = NULL, .thread_local_ssl = ssl};
thrd_t worker;
EXPECT_EQ_INT(thrd_create(&worker, ssl_resolver_worker, &arg), thrd_success);
EXPECT_EQ_INT(thrd_join(worker, NULL), thrd_success);
EXPECT_TRUE(arg.resolved_ssl == arg.expected_ssl);
EXPECT_TRUE(arg.resolved_ssl == ssl);
EXPECT_NULL(arg.thread_local_ssl);
close(sv[0]);
close(sv[1]);
SSL_free(ssl);
SSL_CTX_free(ctx);
}
/* A bound plaintext session must NOT mask a live thread-local TLS transport:
* protocol_current_ssl() only trusts a bound session whose dispatch is TLS, so
* it falls back to io_ssl here. This is the safe direction for the sendfile
* decision -- returning NULL would let file_send.c take raw sendfile(2) on a
* socket this thread is encrypting. */
static void test_protocol_current_ssl_plaintext_bound_falls_back() {
SSL_CTX* ctx = SSL_CTX_new(TLS_method());
EXPECT_NOT_NULL(ctx);
SSL* ssl = SSL_new(ctx);
EXPECT_NOT_NULL(ssl);
int sv[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, sv), 0);
/* Live thread-local TLS, then a bound plaintext session: the plaintext
* session's NULL ssl must not shadow the encrypted transport. */
io_set_ssl(ssl);
ProtocolSession plain;
protocol_session_init(&plain, sv[0], sv[0]);
protocol_session_bind(&plain);
EXPECT_TRUE(protocol_current_ssl() == ssl);
protocol_session_unbind();
/* A bound TLS session still wins over a different thread-local TLS object. */
SSL* other = SSL_new(ctx);
EXPECT_NOT_NULL(other);
io_set_ssl(other);
ProtocolSession tls;
protocol_session_init(&tls, sv[0], sv[0]);
protocol_session_set_ssl(&tls, ssl);
protocol_session_bind(&tls);
EXPECT_TRUE(protocol_current_ssl() == ssl);
EXPECT_TRUE(protocol_current_ssl() != other);
protocol_session_unbind();
io_set_fds(-1, -1);
close(sv[0]);
close(sv[1]);
SSL_free(other);
SSL_free(ssl);
SSL_CTX_free(ctx);
}
/* ------------------------------------------------------------------------- *
* Genuine TLS + sendfile regression test.
*
* file_send_sendfile_with_skip() must route a TLS transfer through the
* buffered SSL path, resolved from the bound session, even in a worker thread
* whose thread-local io_ssl was never installed. This drives a real TLS
* handshake between two in-memory endpoints and calls the production
* file_send entry from a worker that bound a TLS session only: if the sendfile
* decision regresses to io_get_ssl() it sees NULL, takes raw sendfile(2), and
* copies the file's plaintext into the encrypted stream, so the peer's final
* SSL_read here fails. A tautology-free end-to-end decision guard.
* ------------------------------------------------------------------------- */
static void test_set_fd_nonblocking(int fd) {
int flags = fcntl(fd, F_GETFL, 0);
if (flags != -1)
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}
static SSL_CTX* test_tls_context_with_self_signed_cert(void) {
EVP_PKEY* key = EVP_PKEY_new();
EVP_PKEY_CTX* key_ctx = EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, NULL);
if (!key || !key_ctx) {
EVP_PKEY_free(key);
EVP_PKEY_CTX_free(key_ctx);
return NULL;
}
bool key_ok = EVP_PKEY_keygen_init(key_ctx) == 1 &&
EVP_PKEY_CTX_set_rsa_keygen_bits(key_ctx, 2048) == 1 &&
EVP_PKEY_keygen(key_ctx, &key) == 1;
EVP_PKEY_CTX_free(key_ctx);
X509* cert = key_ok ? X509_new() : NULL;
bool cert_ok = cert != NULL && X509_set_version(cert, 2) == 1 &&
ASN1_INTEGER_set(X509_get_serialNumber(cert), 1) == 1 &&
X509_gmtime_adj(X509_getm_notBefore(cert), 0) != NULL &&
X509_gmtime_adj(X509_getm_notAfter(cert), 3600) != NULL &&
X509_set_pubkey(cert, key) == 1;
if (cert_ok) {
X509_NAME* name = X509_get_subject_name(cert);
cert_ok = X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC, (unsigned char*)"localhost", -1,
-1, 0) == 1 &&
X509_set_issuer_name(cert, name) == 1 && X509_sign(cert, key, EVP_sha256()) > 0;
}
SSL_CTX* ctx = cert_ok ? SSL_CTX_new(TLS_method()) : NULL;
bool installed = ctx != NULL && SSL_CTX_use_certificate(ctx, cert) == 1 &&
SSL_CTX_use_PrivateKey(ctx, key) == 1;
if (ctx && !installed) {
SSL_CTX_free(ctx);
ctx = NULL;
}
if (ctx)
SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL);
X509_free(cert);
EVP_PKEY_free(key);
return ctx;
}
static bool test_tls_pump_handshake(SSL* ssl, int* done) {
int result = SSL_do_handshake(ssl);
if (result == 1) {
*done = 1;
return true;
}
int err = SSL_get_error(ssl, result);
return err == SSL_ERROR_WANT_READ || err == SSL_ERROR_WANT_WRITE;
}
static bool test_tls_read_exact(SSL* ssl, void* out, size_t size) {
char* bytes = out;
size_t got = 0;
while (got < size) {
int result = SSL_read(ssl, bytes + got, (int)(size - got));
if (result > 0) {
got += (size_t)result;
continue;
}
int err = SSL_get_error(ssl, result);
if (err != SSL_ERROR_WANT_READ && err != SSL_ERROR_WANT_WRITE)
return false;
struct pollfd pfd = {.fd = SSL_get_fd(ssl),
.events = err == SSL_ERROR_WANT_READ ? POLLIN : POLLOUT};
if (poll(&pfd, 1, 5000) <= 0)
return false;
}
return true;
}
typedef struct {
ProtocolSession* session;
File* file;
int fd;
bool ok;
} TlsSendfileWorkerArg;
static int tls_sendfile_worker(void* arg) {
TlsSendfileWorkerArg* worker = arg;
/* Deliberately never call io_set_ssl(): the bound session is the only
* transport this thread has, exactly like a worker in the -m pipeline. */
protocol_session_bind(worker->session);
worker->ok =
file_send_sendfile_with_skip(worker->file, worker->fd, false, 0, false, NULL, 0, 0, false);
protocol_session_unbind();
return thrd_success;
}
static void test_tls_sendfile_decision_uses_buffered_path() {
const char content[] = "tls-sendfile-regression-payload";
const char* path = "test_tls_sendfile_regression.bin";
EXPECT_TRUE(file_write_to_disk(path, content, sizeof(content), false, false));
SSL_CTX* ctx = test_tls_context_with_self_signed_cert();
EXPECT_NOT_NULL(ctx);
SSL* server_ssl = SSL_new(ctx);
SSL* client_ssl = SSL_new(ctx);
EXPECT_NOT_NULL(server_ssl);
EXPECT_NOT_NULL(client_ssl);
int sv[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, sv), 0);
test_set_fd_nonblocking(sv[0]);
test_set_fd_nonblocking(sv[1]);
EXPECT_EQ_INT(SSL_set_fd(server_ssl, sv[0]), 1);
EXPECT_EQ_INT(SSL_set_fd(client_ssl, sv[1]), 1);
SSL_set_accept_state(server_ssl);
SSL_set_connect_state(client_ssl);
int server_done = 0;
int client_done = 0;
for (int i = 0; i < 1000 && !(server_done && client_done); i++) {
bool server_ok = server_done || test_tls_pump_handshake(server_ssl, &server_done);
bool client_ok = client_done || test_tls_pump_handshake(client_ssl, &client_done);
if (!server_ok || !client_ok)
break;
}
EXPECT_TRUE(server_done && client_done);
File* file = file_create(path);
EXPECT_NOT_NULL(file);
file->data->size = sizeof(content);
ProtocolSession session;
protocol_session_init(&session, sv[0], sv[0]);
protocol_session_set_bwlimit(&session, 0);
protocol_session_set_ssl(&session, server_ssl);
TlsSendfileWorkerArg arg = {.session = &session, .file = file, .fd = sv[0], .ok = false};
thrd_t worker;
EXPECT_EQ_INT(thrd_create(&worker, tls_sendfile_worker, &arg), thrd_success);
EXPECT_EQ_INT(thrd_join(worker, NULL), thrd_success);
EXPECT_TRUE(arg.ok);
/* The peer must be able to decrypt the whole framing: size header and the
* file body, both produced through the TLS transport. */
unsigned long long wire_size = 0;
EXPECT_TRUE(test_tls_read_exact(client_ssl, &wire_size, sizeof(wire_size)));
EXPECT_EQ_INT((int)wire_size, (int)sizeof(content));
char received[sizeof(content)] = {0};
EXPECT_TRUE(test_tls_read_exact(client_ssl, received, sizeof(received)));
EXPECT_EQ_INT(memcmp(received, content, sizeof(content)), 0);
file_destroy(file);
close(sv[0]);
close(sv[1]);
SSL_free(server_ssl);
SSL_free(client_ssl);
SSL_CTX_free(ctx);
unlink(path);
}
void test_protocol() {
test_send_receive_n_data();
test_send_receive_n_data_zero();
@@ -778,4 +1272,10 @@ void test_protocol() {
test_data_create_starts_uncharged_and_unowned();
test_protocol_throttle_bytes_paces();
test_protocol_throttle_bytes_unlimited();
test_protocol_throttle_bytes_legacy_same_session();
test_protocol_dispatch_via_ops();
test_protocol_io_retry_contract();
test_protocol_current_ssl_prefers_bound_session();
test_protocol_current_ssl_plaintext_bound_falls_back();
test_tls_sendfile_decision_uses_buffered_path();
}
+1
View File
@@ -1,4 +1,5 @@
#include "test_shared_utils.h"
#include "delete.h"
#include "utils.h"
#include "protocol.h"
#include "test_utils.h"
+228 -1
View File
@@ -3,11 +3,13 @@
#include "test_utils.h"
#include "transport_tcp.h"
#include <arpa/inet.h>
#include <dirent.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <unistd.h>
/* -4/-6 map to a getaddrinfo ai_family hint: -4 -> AF_INET, -6 -> AF_INET6,
* and neither -> AF_UNSPEC. Both flags together are rejected earlier (in
@@ -246,6 +248,227 @@ static void test_tcp_nodelay_default_and_override() {
server_delete(&s);
}
/* Count the process's open descriptors via /proc/self/fd. The opendir
* descriptor is itself counted and closed before returning, so repeated calls
* are consistent and a before/after delta reflects only the code under test. */
static int count_open_fds(void) {
DIR* dir = opendir("/proc/self/fd");
if (!dir)
return -1;
int count = 0;
const struct dirent* ent;
while ((ent = readdir(dir)) != NULL) {
if (strcmp(ent->d_name, ".") == 0 || strcmp(ent->d_name, "..") == 0)
continue;
count++;
}
closedir(dir);
return count;
}
/* True when two sockaddrs name the same endpoint (family, address, and port).
* Comparing only the IP would let a connection to a different port on the same
* host pass, so the port is part of the identity. */
static bool sockaddr_same_endpoint(const struct sockaddr_storage* a,
const struct sockaddr_storage* b) {
if (a->ss_family != b->ss_family)
return false;
if (a->ss_family == AF_INET) {
const struct sockaddr_in* ia = (const struct sockaddr_in*)a;
const struct sockaddr_in* ib = (const struct sockaddr_in*)b;
return ia->sin_port == ib->sin_port && ia->sin_addr.s_addr == ib->sin_addr.s_addr;
}
if (a->ss_family == AF_INET6) {
const struct sockaddr_in6* ia = (const struct sockaddr_in6*)a;
const struct sockaddr_in6* ib = (const struct sockaddr_in6*)b;
return ia->sin6_port == ib->sin6_port &&
memcmp(&ia->sin6_addr, &ib->sin6_addr, sizeof(ia->sin6_addr)) == 0;
}
return false;
}
/* Bind + listen on the SECOND address getaddrinfo returns for "localhost", so
* the first candidate is connection-refused and the shared connect loop must
* fall back to a later one. On success the actual bound endpoint is written to
* out_bound/out_bound_len (the caller asserts the winning connect landed on it).
* Returns the listener fd and its port, or -1 when this host does not resolve
* localhost to at least two addresses (the test then skips rather than claiming
* coverage it does not have). */
static int bind_second_localhost_address(int* out_port, struct sockaddr_storage* out_bound,
socklen_t* out_bound_len) {
struct addrinfo hints;
memset(&hints, 0, sizeof(hints));
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
struct addrinfo* res = NULL;
if (getaddrinfo("localhost", "0", &hints, &res) != 0 || !res)
return -1;
const struct addrinfo* chosen = res->ai_next;
if (!chosen) {
freeaddrinfo(res);
return -1;
}
int fd = socket(chosen->ai_family, chosen->ai_socktype, chosen->ai_protocol);
if (fd < 0) {
freeaddrinfo(res);
return -1;
}
int opt = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
if (bind(fd, chosen->ai_addr, chosen->ai_addrlen) != 0 || listen(fd, 1) != 0) {
close(fd);
freeaddrinfo(res);
return -1;
}
struct sockaddr_storage bound;
socklen_t bound_len = sizeof(bound);
if (getsockname(fd, (struct sockaddr*)&bound, &bound_len) != 0) {
close(fd);
freeaddrinfo(res);
return -1;
}
if (out_bound)
*out_bound = bound;
if (out_bound_len)
*out_bound_len = bound_len;
if (bound.ss_family == AF_INET6)
*out_port = ntohs(((struct sockaddr_in6*)&bound)->sin6_port);
else
*out_port = ntohs(((struct sockaddr_in*)&bound)->sin_port);
freeaddrinfo(res);
return fd;
}
/* #219 AC3: when the first getaddrinfo candidate is refused, the connect loop
* must fall back to the next address and end with exactly ONE open descriptor
* (proving the failed attempt's fd was closed before the retry). */
static void test_tcp_connect_falls_back_to_next_address() {
int port = 0;
struct sockaddr_storage bound;
int listener = bind_second_localhost_address(&port, &bound, NULL);
if (listener < 0)
return; /* localhost is single-address on this host: cannot exercise fallback */
/* The /proc/self/fd delta is unreliable under valgrind (its own lazy fd
* activity perturbs the baseline), so only the functional assertions run
* there; the fd-count checks are skipped. */
bool check_fds = !is_running_under_valgrind();
int before = check_fds ? count_open_fds() : -1;
Client* c = client_create();
EXPECT_NOT_NULL(c);
EXPECT_TRUE(client_connect(c, "localhost", port));
EXPECT_TRUE(c->file_descriptor >= 0);
/* The winning candidate must be the endpoint we bound (the second
* getaddrinfo entry). Without this, a re-resolution that dropped the second
* address would make the test pass without ever exercising fallback. */
EXPECT_TRUE(sockaddr_same_endpoint(&c->address, &bound));
if (check_fds && before >= 0)
EXPECT_EQ_INT(count_open_fds(), before + 1);
client_disconnect(c);
if (check_fds && before >= 0)
EXPECT_EQ_INT(count_open_fds(), before);
client_delete(c);
close(listener);
}
/* #219 AC3: a connect that fails on every candidate leaves at most one
* descriptor (the last failed attempt) and none after client_disconnect. */
static void test_tcp_connect_failed_attempts_do_not_leak_fds() {
if (is_running_under_valgrind())
return; /* /proc/self/fd delta is perturbed by valgrind's own lazy fds */
/* Keep an ephemeral loopback port bound (but NOT listening) for the whole
* assertion: the port stays occupied by our own socket, so the kernel
* deterministically refuses a connect() to it. This closes the bind/close/
* connect TOCTOU window in which a parallel test could claim the port. */
int probe = socket(AF_INET, SOCK_STREAM, 0);
EXPECT_TRUE(probe >= 0);
struct sockaddr_in addr;
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
addr.sin_port = 0;
EXPECT_EQ_INT(bind(probe, (struct sockaddr*)&addr, sizeof(addr)), 0);
socklen_t addr_len = sizeof(addr);
EXPECT_EQ_INT(getsockname(probe, (struct sockaddr*)&addr, &addr_len), 0);
int port = ntohs(addr.sin_port);
int before = count_open_fds();
Client* c = client_create();
EXPECT_NOT_NULL(c);
/* The literal loopback address has a single getaddrinfo candidate -- the one
* our bound socket owns -- so the connect is deterministically refused. */
EXPECT_FALSE(client_connect(c, "127.0.0.1", port));
if (before >= 0)
EXPECT_TRUE(count_open_fds() <= before + 1);
client_disconnect(c);
if (before >= 0)
EXPECT_EQ_INT(count_open_fds(), before);
client_delete(c);
close(probe);
}
/* #219 AC3: the shared tcp_connect_socket_ex() (used by both the plain and TLS
* entry points) must install the --contimeout as SO_RCVTIMEO/SO_SNDTIMEO before
* connecting. Calling it directly lets us observe the pre-connect state (the
* plain wrapper later overrides the receive timeout with the IO --timeout). */
static void test_tcp_connect_socket_ex_applies_contimeout() {
Server* s = server_create(0);
EXPECT_NOT_NULL(s);
EXPECT_EQ_INT(listen(s->file_descriptor, 1), 0);
struct sockaddr_in bound;
socklen_t bound_len = sizeof(bound);
EXPECT_EQ_INT(getsockname(s->file_descriptor, (struct sockaddr*)&bound, &bound_len), 0);
int port = ntohs(bound.sin_port);
tcp_set_timeouts(30, 7);
Client* c = client_create();
EXPECT_NOT_NULL(c);
TcpConnectOptions opts;
memset(&opts, 0, sizeof(opts));
EXPECT_TRUE(tcp_connect_socket_ex(c, "127.0.0.1", port, &opts));
struct timeval tv;
socklen_t tv_len = sizeof(tv);
EXPECT_EQ_INT(getsockopt(c->file_descriptor, SOL_SOCKET, SO_RCVTIMEO, &tv, &tv_len), 0);
EXPECT_EQ_INT((int)tv.tv_sec, 7);
tv_len = sizeof(tv);
EXPECT_EQ_INT(getsockopt(c->file_descriptor, SOL_SOCKET, SO_SNDTIMEO, &tv, &tv_len), 0);
EXPECT_EQ_INT((int)tv.tv_sec, 7);
client_disconnect(c);
client_delete(c);
tcp_set_timeouts(30, 10);
server_delete(&s);
}
/* The plain wrapper applies the post-connect IO --timeout, which supersedes the
* contimeout installed during connect. */
static void test_tcp_connect_post_timeout_applied() {
Server* s = server_create(0);
EXPECT_NOT_NULL(s);
EXPECT_EQ_INT(listen(s->file_descriptor, 1), 0);
struct sockaddr_in bound;
socklen_t bound_len = sizeof(bound);
EXPECT_EQ_INT(getsockname(s->file_descriptor, (struct sockaddr*)&bound, &bound_len), 0);
int port = ntohs(bound.sin_port);
tcp_set_timeouts(5, 7);
Client* c = client_create();
EXPECT_NOT_NULL(c);
EXPECT_TRUE(client_connect(c, "127.0.0.1", port));
struct timeval tv;
socklen_t tv_len = sizeof(tv);
EXPECT_EQ_INT(getsockopt(c->file_descriptor, SOL_SOCKET, SO_RCVTIMEO, &tv, &tv_len), 0);
EXPECT_EQ_INT((int)tv.tv_sec, 5);
tv_len = sizeof(tv);
EXPECT_EQ_INT(getsockopt(c->file_descriptor, SOL_SOCKET, SO_SNDTIMEO, &tv, &tv_len), 0);
EXPECT_EQ_INT((int)tv.tv_sec, 5);
client_disconnect(c);
client_delete(c);
tcp_set_timeouts(30, 10);
server_delete(&s);
}
void test_transport_tcp() {
test_server_create_ephemeral();
test_server_delete_null();
@@ -263,4 +486,8 @@ void test_transport_tcp() {
test_server_create_bind_address();
test_server_create_bind_ipv6();
test_tcp_nodelay_default_and_override();
test_tcp_connect_falls_back_to_next_address();
test_tcp_connect_failed_attempts_do_not_leak_fds();
test_tcp_connect_socket_ex_applies_contimeout();
test_tcp_connect_post_timeout_applied();
}
+51
View File
@@ -3,8 +3,11 @@
#include "test_utils.h"
#include "transport_tcp.h"
#include "transport_tls.h"
#include <dirent.h>
#include <netinet/in.h>
#include <openssl/ssl.h>
#include <string.h>
#include <sys/socket.h>
#include <unistd.h>
static void test_tls_global_init() {
@@ -71,9 +74,57 @@ static void test_server_create_tls_empty_certs() {
EXPECT_NULL(s);
}
/* Count the process's open descriptors via /proc/self/fd (see the TCP tests). */
static int tls_count_open_fds(void) {
DIR* dir = opendir("/proc/self/fd");
if (!dir)
return -1;
int count = 0;
const struct dirent* ent;
while ((ent = readdir(dir)) != NULL) {
if (strcmp(ent->d_name, ".") == 0 || strcmp(ent->d_name, "..") == 0)
continue;
count++;
}
closedir(dir);
return count;
}
/* #219 AC3: client_connect_tls_ex() reuses the shared tcp_connect_socket_ex()
* for the TCP connect, and a later TLS-setup failure must release that
* descriptor. Passing no CA path makes create_ssl_ctx() fail deterministically
* AFTER a successful TCP connect, so the cleanup path is exercised without a
* TLS handshake or a certificate. (The multi-address fallback itself is covered
* by the shared tcp_connect_socket_ex() tests in test_transport_tcp.c, which the
* TLS entry point calls.) */
static void test_client_connect_tls_releases_fd_on_setup_failure() {
Server* s = server_create(0);
EXPECT_NOT_NULL(s);
EXPECT_EQ_INT(listen(s->file_descriptor, 1), 0);
struct sockaddr_in bound;
socklen_t bound_len = sizeof(bound);
EXPECT_EQ_INT(getsockname(s->file_descriptor, (struct sockaddr*)&bound, &bound_len), 0);
int port = ntohs(bound.sin_port);
/* The /proc/self/fd delta is unreliable under valgrind (its own lazy fd
* activity perturbs the baseline); keep the functional assertions and skip
* only the count checks there. */
bool check_fds = !is_running_under_valgrind();
int before = check_fds ? tls_count_open_fds() : -1;
Client* c = client_create();
EXPECT_NOT_NULL(c);
EXPECT_FALSE(client_connect_tls(c, "127.0.0.1", port, NULL, NULL, NULL));
EXPECT_TRUE(c->file_descriptor == -1);
if (check_fds && before >= 0)
EXPECT_EQ_INT(tls_count_open_fds(), before);
client_delete(c);
server_delete(&s);
}
void test_transport_tls() {
test_tls_global_init();
test_server_create_tls_without_certs();
test_client_connect_tls_fail();
test_client_connect_tls_releases_fd_on_setup_failure();
test_server_create_tls_empty_certs();
}
+367 -9
View File
@@ -1,15 +1,21 @@
#include "test_xattr.h"
#include "xattr.h"
#include "charset.h"
#include "config.h"
#include "file.h"
#include "file_receive.h"
#include "file_save.h"
#include "identity.h"
#include "metadata.h"
#include "protocol.h"
#include "scanner_internal.h"
#include "test_utils.h"
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <sys/xattr.h>
@@ -155,8 +161,8 @@ static void test_xattr_capture_and_appliable() {
EXPECT_TRUE(xattr_name_appliable("user.foo", false));
EXPECT_TRUE(xattr_name_appliable("user.foo", true));
/* The reserved fake-super key is receiver-only and never forwarded/applied. */
EXPECT_FALSE(xattr_name_appliable("user.fastsync.stat", false));
EXPECT_FALSE(xattr_name_appliable("user.fastsync.stat", true));
EXPECT_FALSE(xattr_name_appliable("user.rsync.%stat", false));
EXPECT_FALSE(xattr_name_appliable("user.rsync.%stat", true));
/* B4: the ACL names require --acls; -X alone must not authorize them. */
EXPECT_FALSE(xattr_name_appliable("system.posix_acl_access", false));
EXPECT_FALSE(xattr_name_appliable("system.posix_acl_default", false));
@@ -350,9 +356,10 @@ static void test_xattr_capture_filters_acls() {
}
/* --fake-super replay: fake_super_store_fd records the source stat into the
* reserved xattr, and fake_super_restore_fd re-applies mode/mtime (and owner,
* when the process may) fd-relative. Restore must also be a safe no-op with no
* xattr present. Guarded on filesystem xattr support. */
* reserved xattr, and fake_super_restore_fd re-applies the permission bits
* fd-relative (mtime travels through the normal metadata path; the owner is
* never chowned). Restore must also be a safe no-op with no xattr present.
* Guarded on filesystem xattr support. */
static void test_fake_super_restore() {
const char* path = "test_fake_super_restore.txt";
unlink(path);
@@ -372,7 +379,7 @@ static void test_fake_super_restore() {
FileAttrPolicy policy = {true, true, false, false, true};
EXPECT_FALSE(fake_super_restore_fd(fd, policy));
fake_super_store_fd(fd, 1001, 1002, 0751, 1700000000, 123456789);
fake_super_store_fd(fd, 1001, 1002, S_IFREG | 0751, 0, 0);
EXPECT_TRUE(fake_super_restore_fd(fd, policy));
struct stat st;
EXPECT_EQ_INT(fstat(fd, &st), 0);
@@ -380,7 +387,7 @@ static void test_fake_super_restore() {
/* Strict rsync parity: -p restores the recorded mode exactly, including
group/other write (a recorded 0666 restores as 0666). */
fake_super_store_fd(fd, 1001, 1002, 0666, 1700000000, 0);
fake_super_store_fd(fd, 1001, 1002, S_IFREG | 0666, 0, 0);
EXPECT_TRUE(fake_super_restore_fd(fd, policy));
EXPECT_EQ_INT(fstat(fd, &st), 0);
EXPECT_EQ_INT((int)(st.st_mode & 0777), 0666);
@@ -400,6 +407,76 @@ static void test_fake_super_restore() {
unlink(path);
}
/* The stored record is rsync 3.4.1's exact grammar
* "<octal st_mode with S_IFMT> <rdev_major>,<rdev_minor> <uid>:<gid>"
* so a fake-super tree is readable by rsync. Also pins two rsync parity
* rules: the special bits are stored in the record but NOT applied to the real
* file, and a device record's rdev round-trips through the parser. Guarded on
* filesystem xattr support. */
static void test_fake_super_rsync_format() {
const char* path = "test_fake_super_format.txt";
unlink(path);
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
if (fd < 0)
return;
bool has_xattr = setxattr(path, "user.fastsync.xprobe", "p", 1, 0) == 0;
if (has_xattr)
removexattr(path, "user.fastsync.xprobe");
if (!has_xattr) {
close(fd);
unlink(path);
return; /* skip silently when the filesystem has no xattr support */
}
/* A setuid regular file: the full st_mode (with S_IFMT + special bits) is
recorded, rdev is 0,0, and the owner is uid:gid. */
fake_super_store_fd(fd, 1234, 5678, S_IFREG | 04711, 0, 0);
char value[128];
ssize_t got = fgetxattr(fd, FAKESUPER_XATTR, value, sizeof(value));
EXPECT_EQ_INT((int)got, 20);
EXPECT_TRUE(got == 20 && memcmp(value, "104711 0,0 1234:5678", 20) == 0);
/* The special bits in the record are NOT installed on the real file. */
FileAttrPolicy policy = {true, true, false, false, true};
EXPECT_TRUE(fake_super_restore_fd(fd, policy));
struct stat st;
EXPECT_EQ_INT(fstat(fd, &st), 0);
EXPECT_EQ_INT((int)(st.st_mode & 07777), 0711);
EXPECT_EQ_INT((int)(st.st_mode & (S_ISUID | S_ISGID | S_ISVTX)), 0);
/* A device record (char 1,3, uid 111, gid 222) parses without error and
still never real-chowns or installs the device's mode bits verbatim. */
EXPECT_EQ_INT((int)fsetxattr(fd, FAKESUPER_XATTR, "20644 1,3 111:222", 17, 0), 0);
struct stat before;
fstat(fd, &before);
EXPECT_TRUE(fake_super_restore_fd(fd, policy));
fstat(fd, &st);
EXPECT_EQ_INT((int)(st.st_mode & 0777), 0644);
EXPECT_EQ_INT((int)st.st_uid, (int)before.st_uid);
EXPECT_EQ_INT((int)st.st_gid, (int)before.st_gid);
/* Hardened parser: an out-of-range field (previously UB via sscanf("%u")),
a missing field, or trailing garbage is rejected cleanly instead of being
silently accepted. */
const char* malformed[] = {
"20644 65536,3 111:222", /* major > 0xffff */
"20644 1,16777216 111:222", /* minor > 0xffffff */
"20644 1,3 111:222 trailing", /* trailing garbage */
"20644 1,3 111", /* missing gid */
"20644 1,3 4294967296:222", /* uid > UINT_MAX */
"20644 1,3 111:4294967296", /* gid > UINT_MAX */
"99999999999999999999 1,3 0:0", /* mode overflow */
"", /* empty record */
};
for (size_t i = 0; i < sizeof(malformed) / sizeof(malformed[0]); i++) {
EXPECT_EQ_INT((int)fsetxattr(fd, FAKESUPER_XATTR, malformed[i], strlen(malformed[i]), 0), 0);
EXPECT_FALSE(fake_super_restore_fd(fd, policy));
}
close(fd);
unlink(path);
}
/* --fake-super must NEVER perform a real chown: fake_super_restore_fd applies
* only mode/mtime and leaves the entry's uid/gid exactly as they were, even
* when an explicit ownership policy is active and super_mode permits it. This
@@ -420,7 +497,7 @@ static void test_fake_super_no_real_chown() {
}
struct stat before;
EXPECT_EQ_INT(fstat(fd, &before), 0);
fake_super_store_fd(fd, 12345, 12346, 0755, 1700000000, 0);
fake_super_store_fd(fd, 12345, 12346, S_IFREG | 0755, 0, 0);
Config* c = config_create();
FileAttrPolicy policy = {true, true, false, false, true};
@@ -540,8 +617,287 @@ static void test_xattr_list_clone() {
xattr_list_free(clone);
}
/* #286.3: an explicit directory entry (--dirs, STATUS_MKDIR) that carries a
* captured user.* xattr must have it applied fd-relative by the directory
* install path itself -- not only by the receiver's deferred DirTimeList, which
* a direct file_save_to_disk_full() caller does not use. */
static void test_file_save_directory_applies_xattrs() {
const char* root = "test_save_dir_xattr_tmp";
const char* leaf = "subdir";
const char* path = "test_save_dir_xattr_tmp/subdir";
rmdir(path);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
/* The working directory may be a filesystem without user xattrs (e.g. some
tmpfs mounts): skip cleanly rather than fail the suite. */
if (setxattr(root, "user.fastsync-dirprobe", "p", 1, 0) != 0) {
rmdir(root);
return;
}
removexattr(root, "user.fastsync-dirprobe");
File* dir = file_create(leaf);
EXPECT_NOT_NULL(dir);
dir->is_dir = true;
FileXattrList* xattrs = xattr_list_new();
EXPECT_NOT_NULL(xattrs);
EXPECT_TRUE(xattr_list_append(xattrs, "user.dirxattr", "dirvalue", 8));
dir->xattrs = xattrs;
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->use_metadata = true;
config->use_xattrs = true;
config->preserve_xattrs = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, dir, config), FILE_SAVE_WRITTEN);
EXPECT_EQ_INT(access(path, F_OK), 0);
char value[32];
ssize_t got = getxattr(path, "user.dirxattr", value, sizeof(value));
EXPECT_EQ_INT((int)got, 8);
EXPECT_TRUE(got == 8 && memcmp(value, "dirvalue", 8) == 0);
file_destroy(dir);
config_delete(config);
removexattr(path, "user.dirxattr");
rmdir(path);
rmdir(root);
}
/* Symlink xattrs (protocol 2.29.0): the no-follow capture must read the LINK's
* OWN attributes and never the REFERENT's. Linux's VFS refuses to associate
* xattrs with a symlink at all, so the nofollow capture returns NULL while the
* path-following capture sees the referent's attribute -- which is exactly the
* bug the no-follow variant exists to prevent (a symlink entry must not carry
* its target's attributes). Guarded on filesystem xattr support. */
static void test_xattr_capture_symlink_nofollow() {
const char* target = "test_symlink_xattr_capture_target";
const char* link = "test_symlink_xattr_capture_link";
unlink(link);
unlink(target);
int fd = open(target, O_WRONLY | O_CREAT | O_TRUNC, 0600);
if (fd < 0)
return;
bool has_xattr = setxattr(target, "user.symref", "referent", 8, 0) == 0;
close(fd);
if (!has_xattr) {
unlink(target);
return; /* filesystem without xattr support */
}
if (symlink(target, link) != 0) {
unlink(target);
return;
}
/* The no-follow capture must never pick up the referent's attributes. */
FileXattrList* nofollow = xattr_capture_path_nofollow(link, false);
EXPECT_NULL(nofollow);
/* The path-following capture does, proving the referent really carries one
and that the no-follow variant differs. */
FileXattrList* follow = xattr_capture_path(link, false);
bool saw = false;
for (int i = 0; follow && i < follow->count; i++) {
if (strcmp(follow->items[i].name, "user.symref") == 0)
saw = true;
}
EXPECT_TRUE(saw);
xattr_list_free(follow);
xattr_list_free(nofollow);
unlink(link);
unlink(target);
}
/* Symlink xattrs (protocol 2.29.0): the no-follow apply must target the LINK,
* never its referent. On Linux the LSETXATTR is refused (the VFS does not
* allow symlink xattrs), but the critical guarantee is observable: the
* referent's attributes are UNCHANGED. A regression from lsetxattr to the
* path-following setxattr would rewrite the referent here and fail this test. */
static void test_xattr_apply_path_nofollow_does_not_follow() {
const char* root = "test_symlink_xattr_apply_tmp";
const char* target = "test_symlink_xattr_apply_tmp/target";
const char* link = "test_symlink_xattr_apply_tmp/link";
unlink(link);
unlink(target);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
int tfd = open(target, O_WRONLY | O_CREAT | O_TRUNC, 0600);
if (tfd < 0) {
rmdir(root);
return;
}
bool has_xattr = setxattr(target, "user.orig", "orig", 4, 0) == 0;
close(tfd);
if (!has_xattr) {
unlink(target);
rmdir(root);
return; /* filesystem without xattr support */
}
EXPECT_EQ_INT(symlink("target", link), 0);
FileXattrList* list = xattr_list_new();
EXPECT_NOT_NULL(list);
EXPECT_TRUE(xattr_list_append(list, "user.orig", "hacked", 6));
EXPECT_TRUE(xattr_list_append(list, "user.added", "x", 1));
/* Invalid anchors are refused before any syscall (no fd/leaf/list). */
EXPECT_FALSE(xattr_apply_path_nofollow(-1, "link", list, false));
EXPECT_FALSE(xattr_apply_path_nofollow(0, "", list, false));
EXPECT_FALSE(xattr_apply_path_nofollow(0, "a/b", list, false));
EXPECT_FALSE(xattr_apply_path_nofollow(0, "link", NULL, false));
/* The confined parent directory is the anchor; the final component is the
link. Best-effort: returns true even when the kernel refuses. */
int dir_fd = open(root, O_RDONLY | O_DIRECTORY);
EXPECT_TRUE(dir_fd >= 0);
EXPECT_TRUE(xattr_apply_path_nofollow(dir_fd, "link", list, false));
close(dir_fd);
/* The referent must be untouched: a following apply would have set user.orig
to "hacked" and created user.added on the target. */
char buf[16];
ssize_t got = getxattr(target, "user.orig", buf, sizeof(buf));
EXPECT_EQ_INT(4, (int)got);
if (got == 4)
EXPECT_TRUE(memcmp(buf, "orig", 4) == 0);
EXPECT_TRUE(getxattr(target, "user.added", buf, sizeof(buf)) < 0);
/* If the platform DOES support symlink xattrs, they must have landed on the
link itself; on Linux the VFS refuses them, so the link stays empty. */
if (llistxattr(link, NULL, 0) > 0) {
ssize_t n = lgetxattr(link, "user.added", buf, sizeof(buf));
EXPECT_EQ_INT(1, (int)n);
if (n == 1)
EXPECT_TRUE(buf[0] == 'x');
}
xattr_list_free(list);
unlink(link);
unlink(target);
rmdir(root);
}
/* Protocol 2.29.0 scanner wiring: scanner_capture_xattrs() must choose the
* NO-FOLLOW capture for a symlink entry, so the link's FileXattrList never
* carries the REFERENT's user.* attributes. xattr_capture_path_nofollow() is
* already covered directly above; this exercises the scanner CALL SITE, which is
* what makes the no-follow variant actually reach symlink entries. If the
* scanner regressed to the path-following capture, file->xattrs would contain
* user.symref and this test fails. Guarded on filesystem xattr support. */
static void test_scanner_symlink_capture_is_nofollow() {
const char* target = "test_scanner_symlink_xattr_target";
const char* link = "test_scanner_symlink_xattr_link";
unlink(link);
unlink(target);
int fd = open(target, O_WRONLY | O_CREAT | O_TRUNC, 0600);
if (fd < 0)
return;
bool has_xattr = setxattr(target, "user.symref", "referent", 8, 0) == 0;
close(fd);
if (!has_xattr) {
unlink(target);
return; /* filesystem without xattr support */
}
if (symlink(target, link) != 0) {
unlink(target);
return;
}
DirectoryScanner scanner;
memset(&scanner, 0, sizeof(scanner));
scanner.options.preserve_xattrs = true;
File* file = file_create(link);
EXPECT_NOT_NULL(file);
file->is_symlink = true;
scanner_capture_xattrs(&scanner, file);
/* The referent's attribute must not appear on the symlink's captured list. */
bool leaked = false;
for (int i = 0; file->xattrs && i < file->xattrs->count; i++) {
if (strcmp(file->xattrs->items[i].name, "user.symref") == 0)
leaked = true;
}
EXPECT_FALSE(leaked);
/* On Linux the VFS associates no xattrs with a symlink, so the capture is
NULL (never an empty-but-valid list). */
EXPECT_NULL(file->xattrs);
file_destroy(file);
unlink(link);
unlink(target);
}
/* Protocol 2.29.0: a STATUS_SYMLINK frame followed by an -X/-A xattr block is
* decoded by file_receive_symlink() with the block attached to the File. This
* is the wire round-trip for the new trailing symlink xattr block. */
static void run_recv_symlink_with_xattrs(int fd) {
/* Stack-allocated so the forked child leaks nothing at _exit() (a
config_create() in the parent would be inherited and never freed here). */
Config config;
memset(&config, 0, sizeof(config));
config.use_metadata = true;
config.use_xattrs = true;
config.preserve_xattrs = true;
File* file = file_receive_symlink(fd, &config);
if (!file)
_exit(1);
bool ok = file->is_symlink && file->symlink_target != NULL &&
strcmp(file->symlink_target, "target") == 0;
ok = ok && file->xattrs != NULL && file->xattrs->count == 1 &&
strcmp(file->xattrs->items[0].name, "user.sym") == 0 &&
file->xattrs->items[0].value_len == 3 && memcmp(file->xattrs->items[0].value, "sym", 3) == 0;
file_destroy(file);
_exit(ok ? 0 : 1);
}
static void test_symlink_frame_carries_xattrs() {
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
io_set_bwlimit(0);
run_recv_symlink_with_xattrs(p[0]);
}
close(p[0]);
io_set_fds(p[1], p[1]);
io_set_bwlimit(0);
FileMetadata m;
memset(&m, 0, sizeof(m));
m.mode = S_IFLNK | 0777;
m.uid = (uint32_t)geteuid();
m.gid = (uint32_t)getegid();
m.mtime_sec = 1700000000;
FileXattrList* list = xattr_list_new();
EXPECT_NOT_NULL(list);
EXPECT_TRUE(xattr_list_append(list, "user.sym", "sym", 3));
/* Exactly the sender's order: path, target, metadata, xattr block. */
bool wrote = send_wire_str(p[1], "link") && send_wire_str(p[1], "target") &&
metadata_send(p[1], &m) && xattr_send(p[1], list);
xattr_list_free(list);
close(p[1]);
int status = 0;
waitpid(pid, &status, 0);
EXPECT_TRUE(wrote);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
void test_xattr() {
test_xattr_list_clone();
test_xattr_capture_symlink_nofollow();
test_scanner_symlink_capture_is_nofollow();
test_xattr_apply_path_nofollow_does_not_follow();
test_symlink_frame_carries_xattrs();
test_xattr_wire_roundtrip();
test_xattr_reject_privileged_namespace();
test_xattr_reject_oversized_value();
@@ -551,6 +907,8 @@ void test_xattr() {
test_xattr_receive_drops_acl_without_preserve_acls();
test_link_copy_fallback_preserves_xattrs();
test_fake_super_restore();
test_fake_super_rsync_format();
test_fake_super_no_real_chown();
test_fake_super_storage_resolution();
}
test_file_save_directory_applies_xattrs();
}