Release v2.26.0 #284

Merged
TapTap merged 210 commits from dev into main 2026-09-18 19:05:52 +02:00
5 changed files with 258 additions and 18 deletions
Showing only changes of commit 6269ae54e5 - Show all commits
+2 -2
View File
@@ -679,7 +679,7 @@ now transmits targets (the prior behavior was broken/partial); its status moved
| `--stop-after=MINS` | Stop after N minutes | ✅ Implemented | Client-only sender stop deadline (Phase 6): computing `--stop-after=MINS` (a positive minute count; 0/negative/garbage rejected) and `--stop-at=TIME` (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`; a past time stops immediately). The transfer stops ELEGANTLY at the next chunk boundary: everything already fully sent is kept and applied, the run returns 0, and --delete (late/delete-after timing) does NOT wipe the destination — when the scan is cut short the partial keep-set manifest is suppressed with a warning (the delete walk is skipped rather than acting on an incomplete keep-set, so unscanned source mirrors survive). `--delete-before`/`--delete-during` still run their complete pre-scan (which ignores the deadline). Local client-only fields: never serialized into the wire config frame, so no PROTOCOL_VERSION bump. `--stop-after` uses CLOCK_MONOTONIC; `--stop-at` uses the wall clock. Works single-threaded and under `-j`/`--threads` (multithreaded). Divergence: rsync computes `--stop-after` from the run start; FastSync likewise. When both are given, the earlier of the two deadlines wins (checked per iteration). See the Phase-6 stop notes below | | `--stop-after=MINS` | Stop after N minutes | ✅ Implemented | Client-only sender stop deadline (Phase 6): computing `--stop-after=MINS` (a positive minute count; 0/negative/garbage rejected) and `--stop-at=TIME` (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`; a past time stops immediately). The transfer stops ELEGANTLY at the next chunk boundary: everything already fully sent is kept and applied, the run returns 0, and --delete (late/delete-after timing) does NOT wipe the destination — when the scan is cut short the partial keep-set manifest is suppressed with a warning (the delete walk is skipped rather than acting on an incomplete keep-set, so unscanned source mirrors survive). `--delete-before`/`--delete-during` still run their complete pre-scan (which ignores the deadline). Local client-only fields: never serialized into the wire config frame, so no PROTOCOL_VERSION bump. `--stop-after` uses CLOCK_MONOTONIC; `--stop-at` uses the wall clock. Works single-threaded and under `-j`/`--threads` (multithreaded). Divergence: rsync computes `--stop-after` from the run start; FastSync likewise. When both are given, the earlier of the two deadlines wins (checked per iteration). See the Phase-6 stop notes below |
| `--stop-at=TIME` | Stop at specified time | ✅ Implemented | Same feature as `--stop-after` (deadline transfer stop), absolute wall-clock form (`HH:MM[:SS]` or `now+N[smhd]`). See the row above and the Phase-6 stop notes | | `--stop-at=TIME` | Stop at specified time | ✅ Implemented | Same feature as `--stop-after` (deadline transfer stop), absolute wall-clock form (`HH:MM[:SS]` or `now+N[smhd]`). See the row above and the Phase-6 stop notes |
| `--fsync` | Fsync every written file before publication | ✅ Implemented | | | `--fsync` | Fsync every written file before publication | ✅ Implemented | |
| `--protocol=NUM` | Force older protocol version | ✅ Implemented | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.20.0) with no downgrade/backward-compat code paths, so `--protocol=2.20.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.19.0`/`2.18.0`/`2.18`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below | | `--protocol=NUM` | Force older protocol version | ✅ Implemented | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.21.0) with no downgrade/backward-compat code paths, so `--protocol=2.21.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.20.0`/`2.19.0`/`2.18.0`/`2.18`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
| `--iconv=CONVERT_SPEC` | Charset conversion | ✅ Implemented | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, and the receiver converts each wire filename REMOTE→LOCAL before creating/writing. The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front. Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below | | `--iconv=CONVERT_SPEC` | Charset conversion | ✅ Implemented | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, and the receiver converts each wire filename REMOTE→LOCAL before creating/writing. The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front. Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below |
| `--checksum-seed=NUM` | Set checksum seed | ✅ Implemented | Sets the seed for FastSync's whole-file xxHash64 digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). An explicit seed deterministically changes every computed digest on BOTH endpoints (sender and receiver share the seed via the config frame, protocol 2.10.0), so identical runs with the same seed skip the same files and a changed seed changes the digests — the explicit-seed path that makes xxHash comparisons deterministic. `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta`. Divergence from rsync: the default is seed 0, and FastSync never randomizes the seed (rsync uses a random per-transfer seed when `--checksum-seed` is unset); FastSync's unset default therefore reproduces its historical byte-for-byte behavior | | `--checksum-seed=NUM` | Set checksum seed | ✅ Implemented | Sets the seed for FastSync's whole-file xxHash64 digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). An explicit seed deterministically changes every computed digest on BOTH endpoints (sender and receiver share the seed via the config frame, protocol 2.10.0), so identical runs with the same seed skip the same files and a changed seed changes the digests — the explicit-seed path that makes xxHash comparisons deterministic. `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta`. Divergence from rsync: the default is seed 0, and FastSync never randomizes the seed (rsync uses a random per-transfer seed when `--checksum-seed` is unset); FastSync's unset default therefore reproduces its historical byte-for-byte behavior |
| `--secluded-args`, `-s` | Use protocol to send args | ⛔ Impossible/Divergence | Accepted for CLI compatibility (including the rsync short `-s`, Phase 7 Wave A) but a documented **no-op / divergence**. rsync's `-s` protects arguments from shell expansion by shipping them over the protocol; FastSync never passes remote arguments through a shell expansion boundary in the first place — its SSH transport builds the remote argv as **single-quote-escaped shell words** (`ssh_build_remote_command`), so the injection/leak that `-s` guards against does not exist and there is nothing to "seclude". Implementing a true arg-send protocol would mean replacing the argv-based SSH launch with an in-band argument channel, a large redesign of the transport that buys no security here. Chunk serialization remains the long-only `--chunk-serialization`. | | `--secluded-args`, `-s` | Use protocol to send args | ⛔ Impossible/Divergence | Accepted for CLI compatibility (including the rsync short `-s`, Phase 7 Wave A) but a documented **no-op / divergence**. rsync's `-s` protects arguments from shell expansion by shipping them over the protocol; FastSync never passes remote arguments through a shell expansion boundary in the first place — its SSH transport builds the remote argv as **single-quote-escaped shell words** (`ssh_build_remote_command`), so the injection/leak that `-s` guards against does not exist and there is nothing to "seclude". Implementing a true arg-send protocol would mean replacing the argv-based SSH launch with an in-band argument channel, a large redesign of the transport that buys no security here. Chunk serialization remains the long-only `--chunk-serialization`. |
@@ -795,7 +795,7 @@ These are the hardest compatibility items because they require durable formats o
**Phase 6, Wave B (iconv) shipping note (PROTOCOL 2.15.0 → 2.16.0):** `--iconv=LOCAL[,REMOTE]` converts file NAMES at the wire boundary (never content). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field (empty→NULL canonicalized), so both ends share the same wire charset interpretation; this required the PROTOCOL bump because the frame is a strict ordered sequence and a peer that does not parse the new trailing field would desynchronize. Each end derives LOCAL (its own charset) and REMOTE (the wire charset): the sender opens LOCAL→REMOTE and converts every transmitted filename; the receiver opens REMOTE→LOCAL and converts every received filename before creating/writing. Conversion is applied at every wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest keep/protected/missing entries, the incremental-check path, and the embedded `-s`/chunk-blob path). A name it cannot convert (EILSEQ/EINVAL) is failed cleanly with a logged `--iconv: cannot convert file name ...` and is never written truncated/mangled. Validation probes both directions up front (both the sender local→remote and the receiver remote→local, and, for a server/daemon with its own `--iconv`, the client-REMOTE→server-LOCAL pair) so an unusable spec is rejected before the connection rather than mid-transfer, and NUL-emitting target charsets (utf-16/utf-32/ucs-2) are refused because filenames cannot contain NUL. Divergence documented upstream: the receiver does NOT half-swap; the wire charset always comes from the sender's REMOTE half, so a server whose local charset differs from the client's LOCAL must declare it with its own `--iconv`. Conversion is process-global and runs on a single thread per process (sender thread / receiver-loop thread), initialized before worker threads start and freed after they join. **Phase 6, Wave B (iconv) shipping note (PROTOCOL 2.15.0 → 2.16.0):** `--iconv=LOCAL[,REMOTE]` converts file NAMES at the wire boundary (never content). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field (empty→NULL canonicalized), so both ends share the same wire charset interpretation; this required the PROTOCOL bump because the frame is a strict ordered sequence and a peer that does not parse the new trailing field would desynchronize. Each end derives LOCAL (its own charset) and REMOTE (the wire charset): the sender opens LOCAL→REMOTE and converts every transmitted filename; the receiver opens REMOTE→LOCAL and converts every received filename before creating/writing. Conversion is applied at every wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest keep/protected/missing entries, the incremental-check path, and the embedded `-s`/chunk-blob path). A name it cannot convert (EILSEQ/EINVAL) is failed cleanly with a logged `--iconv: cannot convert file name ...` and is never written truncated/mangled. Validation probes both directions up front (both the sender local→remote and the receiver remote→local, and, for a server/daemon with its own `--iconv`, the client-REMOTE→server-LOCAL pair) so an unusable spec is rejected before the connection rather than mid-transfer, and NUL-emitting target charsets (utf-16/utf-32/ucs-2) are refused because filenames cannot contain NUL. Divergence documented upstream: the receiver does NOT half-swap; the wire charset always comes from the sender's REMOTE half, so a server whose local charset differs from the client's LOCAL must declare it with its own `--iconv`. Conversion is process-global and runs on a single thread per process (sender thread / receiver-loop thread), initialized before worker threads start and freed after they join.
**Phase 6, Wave C (protocol-version) shipping note (no PROTOCOL_VERSION change):** `--protocol=NUM` lets the client force the wire protocol version for a transfer. FastSync's protocol is a single lockstep format: the config frame is a strict ordered sequence and the server requires the client's version string to equal `PROTOCOL_VERSION` exactly (`config_receive_with_validate`, src/shared/config.c) — there are no older-format code paths and no downgrade/negotiation machinery, so a lower/higher/virtual version can never be spoken. The honest contract is therefore: `--protocol=2.20.0` (the current `PROTOCOL_VERSION`, as of the packed-metadata wave) is accepted and stored into the client's `version` claim (which `config_send` already transmits), and every other value — `2.19.0`, `2.18.0`, `2.18`, `2.17.0`, `2.16.0`, `2.15.0`, `3.0.0`, rsync-integer spellings like `216`/`31`, garbage, empty — is rejected up front in `validate_config()` before any connection, with a clear error that FastSync supports only its current wire protocol and cannot speak an older or virtual one. Implementation is client-only: a server-side `--protocol` is intentionally not added because the server has no negotiation (it only enforces exact match), and it could only ever be the current version. This preserves (and slightly tightens) existing validation: the client now also refuses to launch with a version it cannot actually speak, rather than only the server rejecting it later. A genuine downgrade would require a per-version compatibility layer for every frame/feature added since (append 2.10, preallocate 2.11, hardlinks 2.12, devices/specials/symlink-trust/xattr 2.13, remote-option 2.14, daemon module/auth 2.15, iconv 2.16, dir/symlink times 2.17, privilege flags --super/--copy-as 2.18, SCRAM daemon auth 2.19, packed metadata 2.20) and is intentionally out of scope — documented divergences from rsync's integer-negotiated downgrade remain. **Phase 6, Wave C (protocol-version) shipping note (no PROTOCOL_VERSION change):** `--protocol=NUM` lets the client force the wire protocol version for a transfer. FastSync's protocol is a single lockstep format: the config frame is a strict ordered sequence and the server requires the client's version string to equal `PROTOCOL_VERSION` exactly (`config_receive_with_validate`, src/shared/config.c) — there are no older-format code paths and no downgrade/negotiation machinery, so a lower/higher/virtual version can never be spoken. The honest contract is therefore: `--protocol=2.21.0` (the current `PROTOCOL_VERSION`, as of the server-contacting dry-run wave) is accepted and stored into the client's `version` claim (which `config_send` already transmits), and every other value — `2.20.0`, `2.19.0`, `2.18.0`, `2.18`, `2.17.0`, `2.16.0`, `2.15.0`, `3.0.0`, rsync-integer spellings like `216`/`31`, garbage, empty — is rejected up front in `validate_config()` before any connection, with a clear error that FastSync supports only its current wire protocol and cannot speak an older or virtual one. Implementation is client-only: a server-side `--protocol` is intentionally not added because the server has no negotiation (it only enforces exact match), and it could only ever be the current version. This preserves (and slightly tightens) existing validation: the client now also refuses to launch with a version it cannot actually speak, rather than only the server rejecting it later. A genuine downgrade would require a per-version compatibility layer for every frame/feature added since (append 2.10, preallocate 2.11, hardlinks 2.12, devices/specials/symlink-trust/xattr 2.13, remote-option 2.14, daemon module/auth 2.15, iconv 2.16, dir/symlink times 2.17, privilege flags --super/--copy-as 2.18, SCRAM daemon auth 2.19, packed metadata 2.20) and is intentionally out of scope — documented divergences from rsync's integer-negotiated downgrade remain.
**Phase-1/2 selection-and-update status correction (docs):** `-I/--ignore-times`, `--size-only`, `-@/--modify-window`, `--existing`, `--ignore-existing`, `-u/--update`, `-W/--whole-file`, and `--compress-threads` were previously listed as not-implemented in this document but are in fact fully implemented and tested on `dev`. This pass corrects the matrix to match the code. The realistic model of these is that FastSync is a *sender-driven* whole-tree copy, so the size+mtime quick-check and all three receiver-policy skips (`--existing`, `--ignore-existing`, `-u`) are evaluated against the **destination** on the receiver side, and their booleans cross the wire in the config frame. `-I`/`--size-only`/`--modify-window` modify the `--incremental` per-file `STATUS_CHECK` handshake's match predicate (`-I` disables the mtime leg and forces transfer; `--size-only` drops only the mtime leg; `--modify-window` adds tolerance to `metadata_mtime_matches`); they require `--incremental` (or a basis dir) to have a handshake to affect, mirroring how they only matter where a quick-check exists in rsync. `--existing`/`--ignore-existing`/`-u` are receiver write-time policies (skipping the write / newer-destination guard) applied across the regular-file, `--delay-updates`-staged, hardlink-sibling, and special/device paths; `-u` implies `-M` metadata and uses a second-then-nanosecond strict `>` newer check; both correctly influence `--remove-source-files` (a skipped source is not removed). `-W/--whole-file` disables block-level delta (opt-in via `--delta`), folded into the wire `use_delta` so no protocol bump was needed, and makes `--fuzzy` inert; `--append`/`--append-verify` are rejected with `-W`. `--compress-threads=NUM` (1..64, client-only, never crosses the wire) sizes the zstd compression worker pool. No code was changed by this correction; the implementation had landed in earlier merge waves (feat/ignore-times, feat/ignore-existing via the newer `file_to_disk_secure_no_replace`/`linkat EEXIST` path, feat/size-only, feat/modify-window, feat/whole-file, feat/update, compression-threads). **Phase-1/2 selection-and-update status correction (docs):** `-I/--ignore-times`, `--size-only`, `-@/--modify-window`, `--existing`, `--ignore-existing`, `-u/--update`, `-W/--whole-file`, and `--compress-threads` were previously listed as not-implemented in this document but are in fact fully implemented and tested on `dev`. This pass corrects the matrix to match the code. The realistic model of these is that FastSync is a *sender-driven* whole-tree copy, so the size+mtime quick-check and all three receiver-policy skips (`--existing`, `--ignore-existing`, `-u`) are evaluated against the **destination** on the receiver side, and their booleans cross the wire in the config frame. `-I`/`--size-only`/`--modify-window` modify the `--incremental` per-file `STATUS_CHECK` handshake's match predicate (`-I` disables the mtime leg and forces transfer; `--size-only` drops only the mtime leg; `--modify-window` adds tolerance to `metadata_mtime_matches`); they require `--incremental` (or a basis dir) to have a handshake to affect, mirroring how they only matter where a quick-check exists in rsync. `--existing`/`--ignore-existing`/`-u` are receiver write-time policies (skipping the write / newer-destination guard) applied across the regular-file, `--delay-updates`-staged, hardlink-sibling, and special/device paths; `-u` implies `-M` metadata and uses a second-then-nanosecond strict `>` newer check; both correctly influence `--remove-source-files` (a skipped source is not removed). `-W/--whole-file` disables block-level delta (opt-in via `--delta`), folded into the wire `use_delta` so no protocol bump was needed, and makes `--fuzzy` inert; `--append`/`--append-verify` are rejected with `-W`. `--compress-threads=NUM` (1..64, client-only, never crosses the wire) sizes the zstd compression worker pool. No code was changed by this correction; the implementation had landed in earlier merge waves (feat/ignore-times, feat/ignore-existing via the newer `file_to_disk_secure_no_replace`/`linkat EEXIST` path, feat/size-only, feat/modify-window, feat/whole-file, feat/update, compression-threads).
+30
View File
@@ -43,6 +43,9 @@ from common import (
_find_free_port, _find_free_port,
_wait_for_port, _wait_for_port,
) )
# The dry-run no-mutation contract is asserted with the same structural snapshot
# (mode/inode/mtime/xattr/content) the feature suite uses.
from test_features import _snapshot_tree
SOURCE_DIR = os.path.join(TEST_DATA_DIR, "daemon_source") SOURCE_DIR = os.path.join(TEST_DATA_DIR, "daemon_source")
MODULE_ROOT = os.path.join(TEST_DATA_DIR, "daemon_modules") MODULE_ROOT = os.path.join(TEST_DATA_DIR, "daemon_modules")
@@ -355,6 +358,33 @@ class TestDaemonRejection:
assert result.returncode != 0 assert result.returncode != 0
assert self._tree_files() == before, "read-only rejection wrote under the module root" assert self._tree_files() == before, "read-only rejection wrote under the module root"
@pytest.mark.ci
def test_read_only_module_allows_dry_run(self, daemon):
"""A server-contacting --dry-run IS a read-only wire operation, so a
`read only = yes` module is the safest dry-run target and must accept it
while writing nothing."""
result, _ = run_client(SOURCE_DIR, "127.0.0.1::readonly", flags=["--dry-run"],
port=daemon.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert "Dry run:" in result.stdout, result.stdout[:200]
assert _tree_file_count(READONLY_MODULE) == 0, "read-only dry-run wrote a file"
@pytest.mark.ci
def test_module_dry_run_mutates_nothing(self, daemon):
"""A daemon-module dry-run reports would-transfer entries but leaves the
module tree structurally identical (mode/inode/mtime/xattr/content)."""
result = _push("127.0.0.1::files", daemon.port)
assert result.returncode == 0, result.stderr or result.stdout
before = _snapshot_tree(FILES_MODULE)
# --ignore-times forces every regular file to be reported as
# would-transfer, so the dry-run exercises the receiver decision rather
# than an all-skip shortcut -- while still mutating nothing.
result, _ = run_client(SOURCE_DIR, "127.0.0.1::files",
flags=["--dry-run", "--ignore-times"], port=daemon.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert "Dry run:" in result.stdout, result.stdout[:200]
assert _snapshot_tree(FILES_MODULE) == before, "daemon dry-run mutated the module root"
def test_unknown_module_rejected(self, daemon): def test_unknown_module_rejected(self, daemon):
result = _push("127.0.0.1::no-such-module", daemon.port) result = _push("127.0.0.1::no-such-module", daemon.port)
assert result.returncode != 0 assert result.returncode != 0
+194 -12
View File
@@ -370,26 +370,55 @@ class TestDryRun:
assert not mismatches, f"Mismatch: {mismatches}" assert not mismatches, f"Mismatch: {mismatches}"
def _snapshot_tree(root): def _snapshot_xattrs(path):
"""Return {relpath: (size, mtime_ns, content_bytes)} for a directory tree. """Return a stable, comparable tuple of (name, value) xattr pairs.
Used to prove a dry-run left the destination byte-for-byte and Returns None when the platform/filesystem does not expose xattrs so both
timestamp-for-timestamp unchanged. Returns an empty dict for a missing snapshots agree on "unavailable" instead of one being treated as changed."""
root so "nothing was created" is also observable.""" try:
names = os.listxattr(path, follow_symlinks=False)
except (AttributeError, OSError):
return None
if not names:
return ()
pairs = []
for name in sorted(names):
try:
value = os.getxattr(path, name, follow_symlinks=False)
except OSError:
value = None
pairs.append((name, value))
return tuple(pairs)
def _snapshot_tree(root):
"""Return a structural snapshot of a directory tree.
Every entry (including directories) is recorded as
(inode, mtime_ns, mode, xattrs, kind-specific payload) so a dry-run that
touched a mode, inode, mtime, xattr, or content is observable. Regular
files carry their size+bytes, symlinks their target, and special entries
(FIFO/socket/device) their size only -- opening a special file could block.
Returns an empty dict for a missing root so "nothing was created" is also
observable."""
snapshot = {} snapshot = {}
if not os.path.exists(root): if not os.path.exists(root):
return snapshot return snapshot
for dirpath, _dirnames, filenames in os.walk(root): for dirpath, dirnames, filenames in os.walk(root):
for name in filenames: for name in list(dirnames) + filenames:
path = os.path.join(dirpath, name) path = os.path.join(dirpath, name)
rel = os.path.relpath(path, root) rel = os.path.relpath(path, root)
st = os.lstat(path) st = os.lstat(path)
entry = [st.st_ino, st.st_mtime_ns, stat.S_IMODE(st.st_mode), _snapshot_xattrs(path)]
if stat.S_ISLNK(st.st_mode): if stat.S_ISLNK(st.st_mode):
snapshot[rel] = ("symlink", os.readlink(path), st.st_mtime_ns) entry.append(("symlink", os.readlink(path)))
continue elif stat.S_ISREG(st.st_mode):
with open(path, "rb") as fh: with open(path, "rb") as fh:
data = fh.read() data = fh.read()
snapshot[rel] = (st.st_size, st.st_mtime_ns, data) entry += [st.st_size, data]
else:
entry.append(st.st_size)
snapshot[rel] = tuple(entry)
return snapshot return snapshot
@@ -461,6 +490,10 @@ class TestRemoteDryRun:
@pytest.mark.ci @pytest.mark.ci
def test_remote_dry_run_mkpath_does_not_create_root(self, shared_server): def test_remote_dry_run_mkpath_does_not_create_root(self, shared_server):
"""A wire dry_run cannot make --mkpath create anything, and it cannot
relax the precondition either: a nonexistent root is rejected (a real
run without the created root is impossible in dry-run) while nothing is
created."""
source = os.path.join(TEST_DATA_DIR, "remote_dry_mk_src") source = os.path.join(TEST_DATA_DIR, "remote_dry_mk_src")
dest = os.path.join(TEST_DATA_DIR, "remote_dry_mk_dst") dest = os.path.join(TEST_DATA_DIR, "remote_dry_mk_dst")
self._seed(source) self._seed(source)
@@ -469,8 +502,7 @@ class TestRemoteDryRun:
result, _ = run_client(source, dest, flags=["--dry-run", "--mkpath"], result, _ = run_client(source, dest, flags=["--dry-run", "--mkpath"],
port=shared_server.port) port=shared_server.port)
assert result.returncode == 0, f"exit {result.returncode}: {result.stderr[:300]}" assert result.returncode != 0, "dry-run --mkpath accepted a nonexistent receive root"
assert "changed.txt" in result.stdout
assert not os.path.exists(dest), "dry-run --mkpath created the destination root" assert not os.path.exists(dest), "dry-run --mkpath created the destination root"
@pytest.mark.ci @pytest.mark.ci
@@ -534,6 +566,156 @@ class TestRemoteDryRun:
with open(os.path.join(received, "changed.txt"), "rb") as f: with open(os.path.join(received, "changed.txt"), "rb") as f:
assert f.read() == b"updated payload for the real transfer\n" assert f.read() == b"updated payload for the real transfer\n"
@pytest.mark.ci
def test_remote_dry_run_delay_updates_mutates_nothing(self, shared_server):
"""--delay-updates stages under the receive root; a dry-run must neither
create that staging tree nor publish anything (mode/inode/mtime intact)."""
source = os.path.join(TEST_DATA_DIR, "remote_dry_delay_src")
dest = os.path.join(TEST_DATA_DIR, "remote_dry_delay_dst")
self._seed(source)
clean_dir(dest)
result, _ = run_client(source, dest, flags=["--delay-updates"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:200]
with open(os.path.join(source, "changed.txt"), "wb") as f:
f.write(b"changed for delay-updates dry-run\n")
before = _snapshot_tree(dest)
result, _ = run_client(source, dest, flags=["--dry-run", "--delay-updates"],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert "changed.txt" in result.stdout, result.stdout
assert _snapshot_tree(dest) == before, "delay-updates dry-run mutated the destination"
@pytest.mark.ci
def test_remote_dry_run_backup_mutates_nothing(self, shared_server):
"""--backup would rename the old file aside; a dry-run must not."""
source = os.path.join(TEST_DATA_DIR, "remote_dry_backup_src")
dest = os.path.join(TEST_DATA_DIR, "remote_dry_backup_dst")
self._seed(source)
clean_dir(dest)
result, _ = run_client(source, dest, port=shared_server.port)
assert result.returncode == 0, result.stderr[:200]
with open(os.path.join(source, "changed.txt"), "wb") as f:
f.write(b"changed for backup dry-run\n")
before = _snapshot_tree(dest)
result, _ = run_client(source, dest, flags=["--dry-run", "--backup"],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert "changed.txt" in result.stdout, result.stdout
assert _snapshot_tree(dest) == before, "--backup dry-run mutated the destination"
@pytest.mark.ci
def test_remote_dry_run_symlink_mutates_nothing(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remote_dry_symlink_src")
dest = os.path.join(TEST_DATA_DIR, "remote_dry_symlink_dst")
self._seed(source)
os.symlink("changed.txt", os.path.join(source, "link"))
clean_dir(dest)
result, _ = run_client(source, dest, flags=["-a"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:200]
received = get_dest_received_dir(dest, source)
assert os.path.islink(os.path.join(received, "link"))
# Re-point the source link so the entry is genuinely stale, then prove a
# dry-run leaves the destination link target, inode, and mtime untouched.
os.unlink(os.path.join(source, "link"))
os.symlink("keep.txt", os.path.join(source, "link"))
before = _snapshot_tree(dest)
result, _ = run_client(source, dest, flags=["-a", "--dry-run"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert _snapshot_tree(dest) == before, "symlink dry-run mutated the destination"
assert os.readlink(os.path.join(received, "link")) == "changed.txt"
@pytest.mark.ci
def test_remote_dry_run_hardlink_mutates_nothing(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remote_dry_hardlink_src")
dest = os.path.join(TEST_DATA_DIR, "remote_dry_hardlink_dst")
clean_dir(source)
clean_dir(dest)
with open(os.path.join(source, "h1.txt"), "wb") as f:
f.write(b"hardlinked payload\n")
os.link(os.path.join(source, "h1.txt"), os.path.join(source, "h2.txt"))
result, _ = run_client(source, dest, flags=["-H"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:200]
received = get_dest_received_dir(dest, source)
assert os.stat(os.path.join(received, "h1.txt")).st_ino == \
os.stat(os.path.join(received, "h2.txt")).st_ino
# Change the shared inode; both names are now stale in the destination.
with open(os.path.join(source, "h1.txt"), "wb") as f:
f.write(b"changed hardlinked payload\n")
before = _snapshot_tree(dest)
result, _ = run_client(source, dest, flags=["-H", "--dry-run"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert _snapshot_tree(dest) == before, "hardlink dry-run mutated the destination"
@pytest.mark.ci
def test_remote_dry_run_fifo_special_mutates_nothing(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remote_dry_fifo_src")
dest = os.path.join(TEST_DATA_DIR, "remote_dry_fifo_dst")
clean_dir(source)
clean_dir(dest)
with open(os.path.join(source, "plain.txt"), "wb") as f:
f.write(b"plain\n")
os.mkfifo(os.path.join(source, "existing.fifo"))
result, _ = run_client(source, dest, flags=["--specials"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:200]
received = get_dest_received_dir(dest, source)
assert stat.S_ISFIFO(os.lstat(os.path.join(received, "existing.fifo")).st_mode)
os.mkfifo(os.path.join(source, "new.fifo"))
before = _snapshot_tree(dest)
result, _ = run_client(source, dest, flags=["--specials", "--dry-run"],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert not os.path.exists(os.path.join(received, "new.fifo")), \
"dry-run created a FIFO on the receiver"
assert _snapshot_tree(dest) == before, "special-node dry-run mutated the destination"
@pytest.mark.ci
def test_read_batch_with_dry_run_is_refused(self, shared_server):
"""A dry-run of a local batch apply is meaningless (and must not become a
mutation escape hatch): the CLI rejects the combination up front."""
source = os.path.join(TEST_DATA_DIR, "remote_dry_batch_src")
dest = os.path.join(TEST_DATA_DIR, "remote_dry_batch_dst")
self._seed(source)
clean_dir(dest)
result, _ = run_client(source, dest, flags=["--read-batch=/nonexistent.batch", "--dry-run"],
port=shared_server.port)
assert result.returncode != 0, "read-batch + dry-run was accepted"
combined = (result.stderr or "") + (result.stdout or "")
assert "cannot be combined" in combined or "--dry-run" in combined, combined[:300]
@pytest.mark.ci
def test_remote_dry_run_bad_root_fails_like_real_run(self, shared_server):
"""A wire dry_run must not relax the destination-root precondition: a
missing or non-directory root that fails a real run fails a dry-run too,
and the dry-run must not create/replace anything."""
source = os.path.join(TEST_DATA_DIR, "remote_dry_badroot_src")
self._seed(source)
missing = os.path.join(TEST_DATA_DIR, "remote_dry_badroot_missing")
shutil.rmtree(missing, ignore_errors=True)
real, _ = run_client(source, missing, port=shared_server.port)
assert real.returncode != 0, "real run accepted a missing receive root"
assert not os.path.exists(missing), "real run created the missing root"
dry, _ = run_client(source, missing, flags=["--dry-run"], port=shared_server.port)
assert dry.returncode != 0, "dry-run accepted a missing receive root a real run rejects"
assert not os.path.exists(missing), "dry-run created the missing receive root"
fileroot = os.path.join(TEST_DATA_DIR, "remote_dry_badroot_file")
shutil.rmtree(fileroot, ignore_errors=True)
with open(fileroot, "wb") as f:
f.write(b"i am a regular file, not a directory\n")
real, _ = run_client(source, fileroot, port=shared_server.port)
assert real.returncode != 0, "real run accepted a regular-file receive root"
dry, _ = run_client(source, fileroot, flags=["--dry-run"], port=shared_server.port)
assert dry.returncode != 0, "dry-run accepted a regular-file receive root a real run rejects"
with open(fileroot, "rb") as f:
assert f.read() == b"i am a regular file, not a directory\n", \
"dry-run clobbered a regular-file receive root"
class TestRemoveSourceFiles: class TestRemoveSourceFiles:
def test_removes_only_transferred_regular_files(self, shared_server): def test_removes_only_transferred_regular_files(self, shared_server):
+28
View File
@@ -590,6 +590,9 @@ static void test_parse_args_port_alias() {
int positional_count = 0; int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv_space, positional_args, &positional_count), 0); EXPECT_EQ_INT(parse_args(cfg, 5, argv_space, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->server_port, 9000); EXPECT_EQ_INT(cfg->server_port, 9000);
/* The default port is 8080; the explicit bit is what lets --dry-run tell an
explicit remote target from the default and route to the server. */
EXPECT_TRUE(cfg->server_port_set);
config_delete(cfg); config_delete(cfg);
cfg = config_create(); cfg = config_create();
@@ -597,6 +600,7 @@ static void test_parse_args_port_alias() {
positional_count = 0; positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_inline, positional_args, &positional_count), 0); EXPECT_EQ_INT(parse_args(cfg, 4, argv_inline, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->server_port, 9001); EXPECT_EQ_INT(cfg->server_port, 9001);
EXPECT_TRUE(cfg->server_port_set);
config_delete(cfg); config_delete(cfg);
cfg = config_create(); cfg = config_create();
@@ -604,6 +608,29 @@ static void test_parse_args_port_alias() {
positional_count = 0; positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_long, positional_args, &positional_count), 0); EXPECT_EQ_INT(parse_args(cfg, 4, argv_long, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->server_port, 9002); EXPECT_EQ_INT(cfg->server_port, 9002);
EXPECT_TRUE(cfg->server_port_set);
config_delete(cfg);
}
/* An explicit --server-host must set its own routing bit (the field itself
* defaults to 127.0.0.1, so a value check cannot distinguish an explicit host
* from the default); --dry-run uses it to route to the server. */
static void test_parse_args_server_host_sets_routing_bit() {
Config* cfg = config_create();
int positional_args[2];
int positional_count = 0;
EXPECT_FALSE(cfg->server_host_set);
char* argv_space[] = {"fastsync", "--server-host", "example.test", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, argv_space, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->server_host, "example.test");
EXPECT_TRUE(cfg->server_host_set);
config_delete(cfg);
cfg = config_create();
char* argv_inline[] = {"fastsync", "--server-host=example.test", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_inline, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->server_host_set);
config_delete(cfg); config_delete(cfg);
} }
@@ -3301,6 +3328,7 @@ void test_client_cli() {
test_parse_args_non_numeric_port(); test_parse_args_non_numeric_port();
test_parse_args_invalid_server_port(); test_parse_args_invalid_server_port();
test_parse_args_port_alias(); test_parse_args_port_alias();
test_parse_args_server_host_sets_routing_bit();
test_parse_args_threads(); test_parse_args_threads();
test_client_abort_flag(); test_client_abort_flag();
test_parse_args_invalid_compression_level(); test_parse_args_invalid_compression_level();
+2 -2
View File
@@ -2399,7 +2399,7 @@ static void golden_config_populate(Config* c) {
c->modify_window = 3; c->modify_window = 3;
c->compress_choice = str_dup("zstd"); c->compress_choice = str_dup("zstd");
/* "u=rwx,go=rx" is the same 11 bytes as the original "u=rwX,go=rX" (so the /* "u=rwx,go=rx" is the same 11 bytes as the original "u=rwX,go=rX" (so the
* frame stays 633 bytes) but X is not in FastSync's chmod grammar, and the * frame stays 637 bytes) but X is not in FastSync's chmod grammar, and the
* receive-side golden validates the frame. */ * receive-side golden validates the frame. */
c->chmod_spec = str_dup("u=rwx,go=rx"); c->chmod_spec = str_dup("u=rwx,go=rx");
c->skip_compress_set = true; c->skip_compress_set = true;
@@ -2531,7 +2531,7 @@ static unsigned long long capture_wire_hash(const Config* cfg, size_t* out_len)
return h; return h;
} }
/* Byte-for-byte wire compatibility guard (protocol 2.20.0). The expected hash /* Byte-for-byte wire compatibility guard (protocol 2.21.0). The expected hash
* pins the pre-X-macro byte stream; the refactor MUST NOT change it. */ * pins the pre-X-macro byte stream; the refactor MUST NOT change it. */
static void test_config_wire_golden() { static void test_config_wire_golden() {
if (is_running_under_valgrind()) if (is_running_under_valgrind())