Merge branch 'fix/parity-fs' into feat/parity-fixes

# Conflicts:
#	RSYNC_COMPAT.md
This commit is contained in:
2026-09-18 21:19:31 +02:00
22 changed files with 1684 additions and 175 deletions
+77
View File
@@ -0,0 +1,77 @@
# Integration tests
The integration suite drives the built `build/server` and `build/client`
against local corpora. Unit tests live in `tests/` (the custom C framework);
the Python suite here covers the full transfer pipeline, transports, features,
and rsync parity.
## Running
```bash
# Full suite (excludes privilege-dependent tests on CI runners)
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"
# Fast PR subset only
python3 -m pytest tests/integration/ -n 4 --dist=load -m ci
```
The tests expect `build/server` and `build/client` (configure/build with CMake
first); `common.py` derives `BUILD_DIR` from the repository root.
## Differential rsync-parity gate
`test_differential_parity.py` runs the **same** transfer with real
`rsync 3.4.1` and with FastSync over separate destinations, then compares:
- the destination trees — relative paths, file content hashes, symlink
targets, modes (where the case is about perms), and hard-link grouping;
- the normalized stdout for output-oriented flags (`-i`,
`--out-format=...`, `--stats`), after stripping volatile fields
(timings, rates, wire byte counts) and directory-only itemize lines that
FastSync's recursive scanner documents as absent.
FastSync mirrors the absolute source path under its receive root (see
`get_dest_received_dir`); the harness normalizes that layout (and the
`-R`/`--files-from` layouts) before comparing.
```bash
# Fast subset that guards the ✅ surface on pull requests
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci
# Full set (all ✅ cases plus the documented ⚠️/❌ residuals)
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity
```
The suite skips cleanly when `rsync` is not installed.
## Allowlist (`parity_caveats.py`)
`parity_caveats.py` is the single data-driven allowlist of known differences.
Each entry maps a case id to the aspects that may differ (`tree`, `stdout`,
`extra`, `rc`) and cites the governing row in `RSYNC_COMPAT.md`:
```python
CAVEATS = {
"min_size": {
"tree": "recursive transfer does not create a source directory that "
"becomes empty after --min-size filtering. ref: RSYNC_COMPAT.md "
"`-d/--dirs` row and completion-wave residual.",
},
}
```
A differential mismatch in an aspect that is **not** listed fails the gate with
a readable tree/stdout diff.
If a case is allowlisted but now matches rsync, the gate emits a loud warning
naming the stale entry — that is the parity burn-down signal. Run with
`FASTSYNC_PARITY_STRICT=1` to make stale entries fail instead (the full CI
parity job sets this). To add a residual:
1. Reproduce it with `-m parity` and read the failure's tree/stdout diff.
2. Confirm it is a documented `⚠️`/`❌` residual (or get the `✅` row
reclassified) and cite the row.
3. Add the case id and aspect(s) to `CAVEATS`, keeping the reason concise.
Do not allowlist an undocumented divergence from a `✅` row — fix it or get the
row reclassified first.
+46
View File
@@ -0,0 +1,46 @@
"""Data-driven allowlist for the differential rsync-parity gate.
Every entry maps a case id (see ``test_differential_parity.py``) to the aspects
that are *known* to differ from ``rsync 3.4.1`` and the documented reason. A
differential mismatch in an aspect that is **not** listed here fails the gate.
Aspect keys
-----------
``tree`` destination tree differs (paths, file hashes, symlink targets,
modes, hardlink grouping)
``stdout`` normalized output for ``-i`` / ``--stats`` / ``--out-format``
``extra`` a case-specific assertion differs (basis/inode checks, ...)
``rc`` exit status differs
Burn-down
---------
If a case is listed here but now matches rsync, the gate emits a loud
``pytest`` warning naming the stale entry: delete the entry (and, when the
underlying row in ``RSYNC_COMPAT.md`` is now parity, update that row). Set
``FASTSYNC_PARITY_STRICT=1`` to turn stale entries into failures in CI.
Keep the values concise but cite the governing row so the entry can be
re-triaged when the row moves.
"""
# case id -> {aspect: "reason (ref: RSYNC_COMPAT.md ...)"}
CAVEATS = {
# --max-delete stops the extras walk part-way and exits 25 in both
# implementations; which of the remaining extras survives depends on
# deletion order, which neither tool specifies. The exit code and the
# number of survivors match (asserted implicitly by the harness's rc
# comparison and the one-for-one diff below).
"max_delete": {
"tree": "which destination extras survive a partial --max-delete abort "
"is deletion-order dependent and unspecified; rc=25 and the "
"number of survivors match rsync. ref: RSYNC_COMPAT.md "
"`--max-delete=NUM` row.",
},
}
# Accepted aspect names (guards against typos in this file).
ASPECTS = ("tree", "stdout", "extra", "rc")
def caveat_for(case_id: str) -> dict:
return CAVEATS.get(case_id, {})
+502
View File
@@ -0,0 +1,502 @@
"""Differential rsync-parity harness.
Runs the SAME transfer with real ``rsync`` and with FastSync over separate
destinations and compares the resulting trees and (optionally) normalized
stdout. ``test_differential_parity.py`` drives this module with a table of
cases; ``parity_caveats.py`` is the data-driven allowlist of documented
residuals.
Design notes
------------
FastSync mirrors the *absolute* source path below its receive root, while
rsync copies the source contents directly into the destination. ``Case.layout``
tells the harness which pair of directory roots to compare:
* ``MIRROR`` -- rsync ``DEST/`` vs FastSync ``DEST/<abs-src>/`` (the common
case; matches ``common.get_dest_received_dir``).
* ``MIRROR_ABS`` -- ``rsync -R`` without a cut lays the full absolute path
under the destination, so rsync ``DEST/<abs-src>/`` is compared against the
same FastSync mirror path.
* ``RELATIVE`` -- ``rsync -R --files-from`` lays bare relative paths under the
destination and FastSync does the same, so both destination roots compare
directly.
Only ``tests/integration/common.py`` is used to reach the build products and the
server manager; the harness never duplicates that plumbing.
"""
import difflib
import hashlib
import os
import re
import shutil
import subprocess
import sys
from dataclasses import dataclass
from typing import Callable, Dict, List, Optional, Tuple
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402 (path bootstrap above)
TEST_DATA_DIR,
clean_dir,
get_dest_received_dir,
run_client,
)
RSYNC = shutil.which("rsync")
# Comparison layouts (see module docstring).
MIRROR = "mirror"
MIRROR_ABS = "mirror_abs"
RELATIVE = "relative"
# stdout comparators.
STDOUT_NONE = None
STDOUT_ITEMIZE = "itemize"
STDOUT_OUTFMT = "outfmt"
STDOUT_STATS = "stats"
# rsync --stats lines that are protocol-independent and must match exactly.
# Deliberately excluded: the per-type "Number of files"/"Number of created
# files" breakdown and Total bytes sent/received (documented residual, see the
# `--stats` row in RSYNC_COMPAT.md).
STATS_KEYS = (
"Number of deleted files",
"Number of regular files transferred",
"Total file size",
"Total transferred file size",
"Literal data",
"Matched data",
"File list size",
)
_ITEMIZE_RE = re.compile(r"^(<|>|c|h|\.|\*)[fdLDS][.+\-][.+\-][.+\-][.+\-]")
@dataclass
class Case:
"""One differential scenario: a corpus, a flag set, and how to compare."""
id: str
corpus: str
flags: List[str]
fastsync_flags: Optional[List[str]] = None
layout: str = MIRROR
server_args: Tuple[str, ...] = ("--allow-super",)
seed: Optional[Callable] = None
stdout: Optional[str] = STDOUT_NONE
compare_modes: bool = False
compare_hardlinks: bool = False
ignore_paths: Tuple[str, ...] = ()
extra_check: Optional[Callable] = None
files_from: Optional[Tuple[str, ...]] = None
# rsync receives ``src + "/"``; FastSync mirrors the path it is given, so a
# trailing-slash-sensitive case must hand FastSync the same form.
fs_src_suffix: str = ""
ci: bool = False
ref: str = ""
def fs_flags(self) -> List[str]:
return list(self.flags if self.fastsync_flags is None else self.fastsync_flags)
# ---------------------------------------------------------------------------
# Corpora
# ---------------------------------------------------------------------------
# Deterministic mtimes so quick-check decisions are reproducible.
_SRC_MTIME = 1_600_000_000
def _write(path: str, data: bytes, mode: Optional[int] = None) -> None:
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(data)
os.utime(path, (_SRC_MTIME, _SRC_MTIME))
if mode is not None:
os.chmod(path, mode)
def _set_mode(path: str, mode: int) -> None:
os.chmod(path, mode)
def corpus_basic(root: str) -> None:
"""Regular files + nested dirs (dirs are implied by their files)."""
clean_dir(root)
_write(os.path.join(root, "a.txt"), b"hello world\n")
_write(os.path.join(root, "sub", "b.bin"),
bytes((i * 7) & 0xFF for i in range(5000)))
_write(os.path.join(root, "sub", "deep", "c.txt"), "w\u00f6rld\n".encode())
def corpus_unicode(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "uni \u00f1\u6587.txt"), b"unicode\n")
_write(os.path.join(root, "sub", "sp ace \u00e9.dat"), b"spaced\n")
_set_mode(os.path.join(root, "sub"), 0o750)
def corpus_links(root: str) -> None:
corpus_basic(root)
os.symlink("a.txt", os.path.join(root, "rel_link"))
os.symlink("/etc/hostname", os.path.join(root, "abs_link"))
os.symlink("nowhere/target", os.path.join(root, "broken_link"))
def corpus_hardlinks(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "h1.txt"), b"hardlinked payload\n")
os.link(os.path.join(root, "h1.txt"), os.path.join(root, "h2.txt"))
_write(os.path.join(root, "other.txt"), b"other\n")
def corpus_sparse(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "small.txt"), b"small\n")
sparse = os.path.join(root, "sparse.bin")
with open(sparse, "wb") as fh:
fh.seek(1024 * 1024 - 1)
fh.write(b"\0")
os.utime(sparse, (_SRC_MTIME, _SRC_MTIME))
def corpus_filters(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "keep.txt"), b"keep\n")
_write(os.path.join(root, "drop.log"), b"log\n")
_write(os.path.join(root, "sub", "keep2.txt"), b"keep2\n")
_write(os.path.join(root, "sub", "drop2.log"), b"log2\n")
_write(os.path.join(root, "sub", "data.bin"), b"bin\n")
def corpus_empty_dir(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "keep.txt"), b"keep\n")
os.makedirs(os.path.join(root, "emptydir"), exist_ok=True)
os.utime(os.path.join(root, "emptydir"), (_SRC_MTIME, _SRC_MTIME))
_write(os.path.join(root, "nonempty", "f.txt"), b"f\n")
def corpus_relative(root: str) -> None:
"""Tree for the -R/--files-from cases."""
clean_dir(root)
_write(os.path.join(root, "a.txt"), b"a\n")
_write(os.path.join(root, "b.txt"), b"b\n")
_write(os.path.join(root, "sub", "x.txt"), b"x\n")
_write(os.path.join(root, "sub", "y.txt"), b"y\n")
os.makedirs(os.path.join(root, "dir1"), exist_ok=True)
os.utime(os.path.join(root, "dir1"), (_SRC_MTIME, _SRC_MTIME))
_write(os.path.join(root, "dir1", "keep.txt"), b"keep\n")
def corpus_iconv(root: str) -> None:
"""Latin-1 (ISO-8859-1) encoded filenames, matching the --iconv direction."""
clean_dir(root)
for rel, data in ((b"caf\xe9.txt", b"caf\xe9\n"),
(os.path.join(b"sub", b"\xfcber.txt"), b"\xfcber\n")):
full = os.path.join(os.fsencode(root), rel)
os.makedirs(os.path.dirname(full), exist_ok=True)
with open(full, "wb") as fh:
fh.write(data)
os.utime(full, (_SRC_MTIME, _SRC_MTIME))
CORPORA: Dict[str, Callable[[str], None]] = {
"basic": corpus_basic,
"unicode": corpus_unicode,
"links": corpus_links,
"hardlinks": corpus_hardlinks,
"sparse": corpus_sparse,
"filters": corpus_filters,
"empty_dir": corpus_empty_dir,
"relative": corpus_relative,
"iconv": corpus_iconv,
}
# ---------------------------------------------------------------------------
# Tree snapshotting / comparison
# ---------------------------------------------------------------------------
def snapshot(root: str, compare_modes: bool = False) -> Dict[str, tuple]:
"""Map relative path -> descriptor for every entry below ``root``.
Files hash their contents with SHA-256 (structural comparison, so differing
quick-check metadata cannot mask a payload difference). Symlinks record
their target. Empty directories are included (as ``("dir", ...)``) so the
recursive-empty-directory residual is observable.
"""
out: Dict[str, tuple] = {}
if not os.path.isdir(root):
return out
def describe(path: str) -> Optional[tuple]:
st = os.lstat(path)
if os.path.islink(path):
return ("link", os.readlink(path))
if os.path.isdir(path):
mode = oct(st.st_mode & 0o7777) if compare_modes else None
return ("dir", mode)
h = hashlib.sha256()
with open(path, "rb") as fh:
for chunk in iter(lambda: fh.read(65536), b""):
h.update(chunk)
mode = oct(st.st_mode & 0o7777) if compare_modes else None
return ("file", h.hexdigest()[:16], mode)
# The comparison root itself is not part of the tree diff: a no-transfer
# result legitimately leaves FastSync's mirror directory absent while rsync
# leaves an existing (empty) destination root.
for dirpath, dirnames, filenames in os.walk(root, followlinks=False):
dirnames.sort()
for name in sorted(dirnames):
p = os.path.join(dirpath, name)
rel = os.path.relpath(p, root)
if os.path.islink(p):
out[rel] = ("link", os.readlink(p))
dirnames.remove(name)
else:
out[rel] = describe(p)
for name in sorted(filenames):
p = os.path.join(dirpath, name)
out[os.path.relpath(p, root)] = describe(p)
return out
def _hardlink_groups(root: str) -> Dict[str, str]:
"""Assign a stable group letter to each inode shared by >1 regular file."""
inodes: Dict[tuple, List[str]] = {}
for dirpath, _dirs, filenames in os.walk(root, followlinks=False):
for name in filenames:
p = os.path.join(dirpath, name)
if os.path.islink(p):
continue
st = os.lstat(p)
if st.st_nlink > 1:
inodes.setdefault((st.st_dev, st.st_ino), []).append(
os.path.relpath(p, root))
groups: Dict[str, str] = {}
for i, (_key, members) in enumerate(sorted(inodes.items())):
for rel in members:
groups[rel] = chr(ord("A") + i)
return groups
def _drop_ignored(tree: Dict[str, tuple], ignore_paths) -> Dict[str, tuple]:
if not ignore_paths:
return tree
out = {}
for rel, desc in tree.items():
if any(rel == ig or rel.startswith(ig.rstrip("/") + "/") for ig in ignore_paths):
continue
out[rel] = desc
return out
def tree_diff(rsync_root: str, fs_root: str, case: Case) -> List[str]:
"""Return a list of human-readable differences (empty when identical)."""
rtree = _drop_ignored(snapshot(rsync_root, case.compare_modes), case.ignore_paths)
ftree = _drop_ignored(snapshot(fs_root, case.compare_modes), case.ignore_paths)
if case.compare_hardlinks:
rgroups = _hardlink_groups(rsync_root)
fgroups = _hardlink_groups(fs_root)
else:
rgroups = fgroups = {}
diffs: List[str] = []
for rel in sorted(set(rtree) | set(ftree)):
r = rtree.get(rel)
f = ftree.get(rel)
if r == f:
continue
if r is None:
diffs.append(f"+ fastsync-only: {rel!r} {f}")
elif f is None:
diffs.append(f"- rsync-only: {rel!r} {r}")
else:
diffs.append(f"~ differs: {rel!r} rsync={r} fastsync={f}")
if case.compare_hardlinks:
for rel in sorted(set(rgroups) | set(fgroups)):
if rgroups.get(rel) != fgroups.get(rel):
diffs.append(
f"~ hardlink group: {rel!r} rsync={rgroups.get(rel)} "
f"fastsync={fgroups.get(rel)}")
return diffs
# ---------------------------------------------------------------------------
# stdout normalization
# ---------------------------------------------------------------------------
def _parse_bytes(text: str) -> str:
m = re.match(r"([\d,]+)", text.strip())
return m.group(1).replace(",", "") if m else text.strip()
def normalize_stdout(text: str, mode: Optional[str]) -> object:
if mode == STDOUT_ITEMIZE:
lines = []
for line in (text or "").splitlines():
line = line.rstrip()
if not line:
continue
if line.startswith("*deleting"):
lines.append(line)
continue
if not _ITEMIZE_RE.match(line):
continue
# Directories are not transfer entries in FastSync's recursive
# scanner, so rsync's `cd+++++++++ name/` lines have no counterpart
# (documented recursive-empty-dir residual). Compare file/link
# itemization only.
if line.rsplit(" ", 1)[-1].endswith("/"):
continue
lines.append(line)
return sorted(lines)
if mode == STDOUT_OUTFMT:
lines = []
for line in (text or "").splitlines():
line = line.rstrip()
if not line:
continue
# Directory entries are emitted by rsync but not by FastSync's
# recursive scanner (documented residual). Tokens are either
# `%n %l` (path first) or `%i %n` (path last); drop a line when
# either end-token is a directory path.
first = line.split(" ", 1)[0]
last = line.rsplit(" ", 1)[-1]
if first.endswith("/") or last.endswith("/"):
continue
lines.append(line)
return sorted(lines)
if mode == STDOUT_STATS:
found = {}
for line in (text or "").splitlines():
for key in STATS_KEYS:
if line.startswith(key + ":"):
found[key] = _parse_bytes(line.split(":", 1)[1])
return found
# raw
return sorted(l.rstrip() for l in (text or "").splitlines() if l.strip())
def stdout_diff(rsync_out: str, fs_out: str, mode: Optional[str]) -> List[str]:
r = normalize_stdout(rsync_out, mode)
f = normalize_stdout(fs_out, mode)
if r == f:
return []
if mode == STDOUT_STATS:
return [f"stats rsync={r}", f"stats fastsync={f}"]
return list(difflib.unified_diff(
[str(x) for x in r], [str(x) for x in f],
fromfile="rsync", tofile="fastsync", lineterm=""))
# ---------------------------------------------------------------------------
# Running one case
# ---------------------------------------------------------------------------
def run_rsync(src: str, rdst: str, flags: List[str]) -> subprocess.CompletedProcess:
args = [RSYNC] + list(flags) + [src + "/", rdst + "/"]
return subprocess.run(
args, capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"), timeout=180)
def run_fastsync(src: str, fdst: str, flags: List[str], port: int):
return run_client(src, fdst, flags=list(flags), port=port)
def run_differential( # noqa: PLR0913 (explicit scenario parameters)
src: str,
rdst: str,
fdst: str,
rs_flags: List[str],
fs_flags: List[str],
server,
layout: str = MIRROR,
seed: Optional[Callable] = None,
stdout: Optional[str] = STDOUT_NONE,
compare_modes: bool = False,
compare_hardlinks: bool = False,
ignore_paths: Tuple[str, ...] = (),
extra_check: Optional[Callable] = None,
files_from: Optional[Tuple[str, ...]] = None,
fs_src_suffix: str = "",
) -> Dict[str, object]:
"""Run one rsync/FastSync pair and return the diff aspects.
Returned dict keys: ``rsync_rc``, ``fastsync_rc``, ``rsync_stderr``,
``fastsync_stderr``, ``tree``, ``stdout``, ``extra``.
"""
clean_dir(rdst)
clean_dir(fdst)
abs_src = os.path.abspath(src)
rel = abs_src.lstrip(os.sep)
if layout == RELATIVE:
rroot, froot = rdst, fdst
elif layout == MIRROR_ABS:
rroot, froot = os.path.join(rdst, rel), get_dest_received_dir(fdst, src)
else:
rroot, froot = rdst, get_dest_received_dir(fdst, src)
if seed:
seed(src, rroot, froot)
rs_flags = list(rs_flags)
fs_flags = list(fs_flags)
if files_from is not None:
list_path = os.path.join(TEST_DATA_DIR, "parity_" +
os.path.basename(src) + ".list")
write_list(list_path, files_from)
rs_flags.append(f"--files-from={list_path}")
fs_flags.append(f"--files-from={list_path}")
rs = run_rsync(src, rdst, rs_flags)
fs_result, _ = run_fastsync(src + fs_src_suffix, fdst, fs_flags, server.port)
class _View:
"""Adapter so tree_diff/extra_check keep the Case-shaped interface."""
def __init__(self) -> None:
self.compare_modes = compare_modes
self.compare_hardlinks = compare_hardlinks
self.ignore_paths = ignore_paths
result = {
"rsync_rc": rs.returncode,
"fastsync_rc": fs_result.returncode,
"rsync_stderr": rs.stderr,
"fastsync_stderr": fs_result.stderr or fs_result.stdout,
"tree": tree_diff(rroot, froot, _View()),
"stdout": [],
"extra": [],
}
if stdout is not None:
result["stdout"] = stdout_diff(rs.stdout, fs_result.stdout, stdout)
if extra_check:
result["extra"] = list(extra_check(src, rroot, froot, rs, fs_result) or [])
return result
def execute_case(case: Case, server) -> Dict[str, object]:
"""Run a table-driven case and return the diff aspects."""
tag = case.id
src = os.path.join(TEST_DATA_DIR, f"parity_{tag}_src")
rdst = os.path.join(TEST_DATA_DIR, f"parity_{tag}_rdst")
fdst = os.path.join(TEST_DATA_DIR, f"parity_{tag}_fdst")
CORPORA[case.corpus](src)
return run_differential(
src, rdst, fdst,
case.flags, case.fs_flags(), server,
layout=case.layout, seed=case.seed, stdout=case.stdout,
compare_modes=case.compare_modes, compare_hardlinks=case.compare_hardlinks,
ignore_paths=case.ignore_paths, extra_check=case.extra_check,
files_from=case.files_from, fs_src_suffix=case.fs_src_suffix,
)
def write_list(path: str, entries) -> str:
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8") as fh:
for e in entries:
fh.write(e + "\n")
return path
@@ -0,0 +1,520 @@
"""Differential rsync-parity gate.
Runs real ``rsync 3.4.1`` and FastSync over the same corpora and flags, then
compares the destination trees and the normalized output of the
output-oriented flags. This is the executable counterpart of
``RSYNC_COMPAT.md``: the fast subset (``-m parity_ci``) guards the ✅ surface on
every pull request, and the full set (``-m parity``) burns the documented
⚠️/❌ residuals down.
Known, documented differences live in ``parity_caveats.py``; anything else
fails with a readable tree/stdout diff. A stale allowlist entry is reported
loudly (and fails when ``FASTSYNC_PARITY_STRICT=1``).
Run locally::
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity
"""
import os
import shutil
import sys
import warnings
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402
ServerManager,
TEST_DATA_DIR,
clean_dir,
)
from parity_caveats import ASPECTS, caveat_for # noqa: E402
import parity_harness as H # noqa: E402
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
# `--allow-super` matches the rest of the integration suite; `--allow-delete`
# is needed only by the delete cases.
SUPER = ("--allow-super",)
DELETE = ("--allow-super", "--allow-delete")
_OLD_MTIME = 1_500_000_000
parity = pytest.mark.parity
parity_ci = pytest.mark.parity_ci
@pytest.fixture(scope="session")
def parity_server_factory():
"""Lazily start one server per distinct extra-argument set, per xdist worker."""
servers = {}
def get(extra):
key = tuple(extra)
if key not in servers:
s = ServerManager()
s.start(extra_args=list(extra))
servers[key] = s
return servers[key]
yield get
for s in servers.values():
s.stop()
def _pin(path, mtime):
os.utime(path, (mtime, mtime))
def _mk(path, data, mtime=None):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(data)
if mtime is not None:
_pin(path, mtime)
# --- destination seeds ------------------------------------------------------
def seed_extras(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "extra.txt"), b"extra\n")
_mk(os.path.join(root, "extradir", "z.txt"), b"z\n")
def seed_update(_src, rroot, froot):
for root in (rroot, froot):
p = os.path.join(root, "a.txt")
_mk(p, b"destination is newer and longer\n", 2_000_000_000)
def seed_ignore_existing(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "a.txt"), b"destination-kept\n", _OLD_MTIME)
def seed_append(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "a.txt"), b"hello ", _OLD_MTIME)
def seed_backup(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "a.txt"), b"OLD-CONTENT\n", _OLD_MTIME)
def seed_size_only(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "a.txt"), b"XXXXXXXXXXX\n", _OLD_MTIME)
def seed_delete_excluded(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "drop.log"), b"stale log\n", _OLD_MTIME)
_mk(os.path.join(root, "extra.txt"), b"extra\n", _OLD_MTIME)
_mk(os.path.join(root, "keep.txt"), b"keep\n", _OLD_MTIME)
def seed_max_delete(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
_mk(os.path.join(root, "extra2.txt"), b"e2\n", _OLD_MTIME)
def max_delete_count_check(_src, rroot, froot, _rs, _fs):
"""The exact survivor set is order-dependent; the count must still match."""
r = H.snapshot(rroot)
f = H.snapshot(froot)
if len(r) != len(f):
return [f"survivor count differs: rsync={len(r)} fastsync={len(f)}"]
return []
# --- case table -------------------------------------------------------------
_CASES = [
# --- core archive / recursion -----------------------------------------
H.Case("archive", "basic", ["-a"], ci=True, ref="-a/--archive"),
H.Case("recursive", "basic", ["-r"], ci=True, ref="-r/--recursive"),
H.Case("unicode_names", "unicode", ["-a"], ci=True, ref="-a unicode names"),
H.Case("links_archive", "links", ["-a"], ci=True, ref="-l/--links"),
H.Case("copy_links", "links", ["-aL"], ref="-L/--copy-links"),
H.Case("hardlinks", "hardlinks", ["-a", "-H"], compare_hardlinks=True,
ci=True, ref="-H/--hard-links"),
H.Case("hardlinks_without_H", "hardlinks", ["-a"], compare_hardlinks=True,
ref="hardlinks without -H"),
H.Case("sparse", "sparse", ["-a", "-S"], ref="-S/--sparse"),
# --- compression / checksums ------------------------------------------
H.Case("compress_zstd", "basic", ["-a", "-z"], ci=True, ref="-z/--compress"),
H.Case("checksum", "basic", ["-a", "-c"], ref="-c/--checksum"),
H.Case("checksum_choice_xxh64", "basic",
["-a", "-c", "--checksum-choice=xxh64"], ref="--checksum-choice"),
# --- selection --------------------------------------------------------
H.Case("exclude", "filters", ["-a", "--exclude=*.log"], ci=True,
ref="--exclude"),
H.Case("include_exclude", "filters",
["-a", "--include=*.txt", "--exclude=*"], ci=True,
ref="--include/--exclude ordering"),
H.Case("filter_rules", "filters",
["-a", "-f", "- *.log", "-f", "+ *.txt", "-f", "- *"],
ref="--filter/-f grammar"),
H.Case("max_size", "basic", ["-a", "--max-size=1000"], ref="--max-size"),
H.Case("min_size", "basic", ["-a", "--min-size=1000"], ref="--min-size"),
# --- output-oriented --------------------------------------------------
H.Case("stats", "basic", ["-a", "--stats"], stdout=H.STDOUT_STATS,
ci=True, ref="--stats"),
H.Case("itemize", "links", ["-a", "-i"], stdout=H.STDOUT_ITEMIZE,
ci=True, ref="-i/--itemize-changes"),
H.Case("out_format_n_l", "basic", ["-a", "--out-format=%n %l"],
stdout=H.STDOUT_OUTFMT, ref="--out-format %n %l"),
H.Case("out_format_i_n", "basic", ["-a", "--out-format=%i %n"],
stdout=H.STDOUT_OUTFMT, ref="--out-format %i %n"),
# --- transfer modifications -------------------------------------------
H.Case("update", "basic", ["-a", "--update"], seed=seed_update,
ref="-u/--update"),
H.Case("ignore_existing", "basic", ["-a", "--ignore-existing"],
seed=seed_ignore_existing, ci=True, ref="--ignore-existing"),
H.Case("size_only", "basic",
["-a", "--size-only"], fastsync_flags=["-a", "--incremental", "--size-only"],
seed=seed_size_only, ref="--size-only"),
H.Case("append", "basic", ["-a", "--append"], seed=seed_append,
ref="--append"),
H.Case("append_verify", "basic", ["-a", "--append-verify"], seed=seed_append,
ref="--append-verify"),
H.Case("backup", "basic", ["-a", "--backup"], seed=seed_backup,
ref="--backup"),
H.Case("chmod", "basic", ["-a", "--chmod=Fu+rwx"], compare_modes=True,
ci=True, ref="--chmod"),
# --- deletion ---------------------------------------------------------
H.Case("delete", "basic", ["-a", "--delete"], seed=seed_extras,
server_args=DELETE, ci=True, ref="--delete"),
H.Case("delete_before", "basic", ["-a", "--delete-before"], seed=seed_extras,
server_args=DELETE, ref="--delete-before"),
H.Case("delete_during", "basic", ["-a", "--delete-during"], seed=seed_extras,
server_args=DELETE, ref="--delete-during"),
H.Case("delete_delay", "basic", ["-a", "--delete-delay"], seed=seed_extras,
server_args=DELETE, ref="--delete-delay"),
H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras,
server_args=DELETE, ref="--delete-after"),
H.Case("delete_excluded", "filters",
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
H.Case("max_delete", "basic", ["-a", "--delete", "--max-delete=1"],
seed=seed_max_delete, server_args=DELETE,
extra_check=max_delete_count_check, ref="--max-delete"),
# --- relative / dirs --------------------------------------------------
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
compare_modes=True, ref="-R/--relative"),
H.Case("relative_no_implied_dirs", "basic",
["-a", "-R", "--no-implied-dirs"], layout=H.MIRROR_ABS,
compare_modes=True, ref="--no-implied-dirs"),
H.Case("files_from", "relative", ["--dirs", "-R"],
files_from=("dir1", "sub/x.txt"), layout=H.RELATIVE, ci=True,
ref="-d/--dirs + --files-from"),
H.Case("dirs_plain", "basic", ["-d"], fs_src_suffix="/",
ref="-d/--dirs (plain)"),
H.Case("empty_dirs_recursive", "empty_dir", ["-a"],
ref="recursive empty-directory residual"),
H.Case("empty_dirs_files_from", "empty_dir", ["--dirs", "-R"],
files_from=("emptydir",), layout=H.RELATIVE, ci=True,
ref="-d/--dirs explicit empty directory"),
# --- codecs -----------------------------------------------------------
H.Case("iconv_identity", "basic", ["-a", "--iconv=UTF-8,UTF-8"],
ref="--iconv identity"),
H.Case("iconv_convert", "iconv",
["-a", "--iconv=ISO-8859-1,UTF-8"],
server_args=("--allow-super", "--iconv=UTF-8"),
ref="--iconv conversion (receiver declares its own charset)"),
# rsync's spec is LOCAL,REMOTE and the destination end's charset is REMOTE
# on a push, so a default server writes the wire (UTF-8) names verbatim.
H.Case("iconv_default_server", "iconv",
["-a", "--iconv=ISO-8859-1,UTF-8"],
ref="--iconv push direction (default receiver charset = REMOTE)"),
# --- partial ----------------------------------------------------------
H.Case("partial_complete", "basic", ["-a", "--partial"], ref="--partial"),
]
# Cases that must always be tolerated (documented ⚠️/❌ residuals) get an
# allowlist entry; the table below stays the exact ✅ surface.
ALL_CASES = _CASES
def _params():
out = []
for case in ALL_CASES:
marks = [parity]
if case.ci:
marks.append(parity_ci)
out.append(pytest.param(case, id=case.id, marks=marks))
return out
def _aspects_to_check(result):
return {
"tree": result["tree"],
"stdout": result["stdout"],
"extra": result["extra"],
}
def _assert_no_unexpected(case_id, mismatches, caveat, ref=""):
unexpected = {a: v for a, v in mismatches.items() if v and a not in caveat}
if unexpected:
lines = [f"differential parity mismatch for case {case_id!r}:"]
lines.append(f" ref: {ref or 'see RSYNC_COMPAT.md'}")
for aspect, detail in unexpected.items():
lines.append(f" --- {aspect} ---")
lines.extend(" " + str(d) for d in detail)
lines.append("If this is a documented residual, add it to "
"tests/integration/parity_caveats.py with a RSYNC_COMPAT.md "
"reference. Do not allowlist an undocumented divergence.")
pytest.fail("\n".join(lines))
stale = [a for a in ASPECTS
if a in caveat and a != "rc" and not mismatches.get(a)]
if stale:
msg = (f"stale parity allowlist entry for case {case_id!r}, aspect(s) "
f"{stale}: FastSync now matches rsync. Remove it from "
f"parity_caveats.py (and update RSYNC_COMPAT.md if the row moved).")
if os.environ.get("FASTSYNC_PARITY_STRICT") == "1":
pytest.fail(msg)
warnings.warn(msg, stacklevel=2)
@requires_rsync
@pytest.mark.parametrize("case", _params())
def test_differential_case(case, parity_server_factory):
server = parity_server_factory(case.server_args)
result = H.execute_case(case, server)
caveat = caveat_for(case.id)
mismatches = _aspects_to_check(result)
if result["rsync_rc"] != result["fastsync_rc"]:
mismatches["rc"] = [
f"rsync rc={result['rsync_rc']} fastsync rc={result['fastsync_rc']} "
f"(rsync stderr: {result['rsync_stderr'][:200]!r}, "
f"fastsync stderr: {result['fastsync_stderr'][:200]!r})"]
_assert_no_unexpected(case.id, mismatches, caveat, ref=case.ref)
# ---------------------------------------------------------------------------
# Multi-run and setup-heavy scenarios (kept as explicit tests)
# ---------------------------------------------------------------------------
def _result_aspects(result):
return _aspects_to_check(result)
_STANDALONE_REFS = {
"incremental_modified": "-i/--itemize-changes + incremental second run",
"compare_dest": "--compare-dest",
"link_dest": "--link-dest",
"added_and_deleted": "--delete across two runs",
"added_and_deleted_seed": "--delete across two runs",
"one_file_system": "-x/--one-file-system",
}
def _run_and_check(case_id, result, ref=""):
mismatches = _result_aspects(result)
if result["rsync_rc"] != result["fastsync_rc"]:
mismatches["rc"] = [
f"rsync rc={result['rsync_rc']} fastsync rc={result['fastsync_rc']} "
f"(rsync stderr: {result['rsync_stderr'][:200]!r}, "
f"fastsync stderr: {result['fastsync_stderr'][:200]!r})"]
_assert_no_unexpected(case_id, mismatches, caveat_for(case_id),
ref=ref or _STANDALONE_REFS.get(case_id, ""))
@requires_rsync
@parity
def test_incremental_modified_file(parity_server_factory):
"""A second run sends only the modified file; destinations stay identical."""
case_id = "incremental_modified"
src = os.path.join(TEST_DATA_DIR, "parity_inc_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_inc_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_inc_fdst")
H.CORPORA["basic"](src)
server = parity_server_factory(SUPER)
# Seed both destinations with the initial content.
H.run_differential(src, rdst, fdst, ["-a"], ["-a"], server,
extra_check=lambda *a: [])
with open(os.path.join(src, "a.txt"), "wb") as fh:
fh.write(b"hello world, now modified and longer\n")
_pin(os.path.join(src, "a.txt"), 1_650_000_000)
result = H.run_differential(
src, rdst, fdst, ["-a", "-i"], ["-a", "-i", "--incremental"], server,
stdout=H.STDOUT_ITEMIZE)
_run_and_check(case_id, result)
def _seed_basis(rel_entries):
def seed(src, rroot, froot):
for root in (rroot, froot):
os.makedirs(root, exist_ok=True)
for rel, data in rel_entries.items():
_mk(os.path.join(root, rel), data)
return seed
@requires_rsync
@parity
def test_compare_dest_skips_basis(parity_server_factory):
"""--compare-dest: a file present in the basis is not copied."""
case_id = "compare_dest"
src = os.path.join(TEST_DATA_DIR, "parity_cmpd_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"basis-content\n")
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
# rsync resolves --compare-dest relative to the destination dir; FastSync
# resolves it under the receive root and appends the mirrored source path.
def seed(_src, rroot, froot):
_mk(os.path.join(rroot, "basis", "f.txt"), b"basis-content\n")
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"basis-content\n")
def extra(_src, rroot, froot, _rs, _fs):
out = []
for label, root in (("rsync", rroot), ("fastsync", froot)):
if os.path.exists(os.path.join(root, "f.txt")):
out.append(f"{label} copied a file that is present in the "
f"compare basis")
return out
result = H.run_differential(
src, rdst, fdst,
["-a", "--compare-dest=basis"],
["-a", f"--compare-dest={os.path.join(fdst, 'basis')}", "--incremental"],
server, seed=seed, ignore_paths=("basis",), extra_check=extra)
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_link_dest_hardlinks_basis(parity_server_factory):
"""--link-dest: an unchanged file is hard-linked to the basis, not copied."""
case_id = "link_dest"
src = os.path.join(TEST_DATA_DIR, "parity_linkd_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_linkd_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_linkd_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n")
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n")
def extra(_src, rroot, froot, _rs, _fs):
r_basis = os.stat(os.path.join(rroot, "basis", "f.txt")).st_ino
f_basis = os.stat(os.path.join(fdst, "basis", rel, "f.txt")).st_ino
out = []
for label, root, basis in (("rsync", rroot, r_basis),
("fastsync", froot, f_basis)):
target = os.path.join(root, "f.txt")
if not os.path.exists(target):
out.append(f"{label}: f.txt missing")
elif os.stat(target).st_ino != basis:
out.append(f"{label}: f.txt is not hard-linked to the basis")
return out
result = H.run_differential(
src, rdst, fdst,
["-a", "--link-dest=basis"],
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental"],
server, seed=seed, ignore_paths=("basis",), extra_check=extra)
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_added_and_deleted_between_runs(parity_server_factory):
"""A source deletion and addition sync correctly under --delete."""
case_id = "added_and_deleted"
src = os.path.join(TEST_DATA_DIR, "parity_addel_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_addel_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_addel_fdst")
server = parity_server_factory(DELETE)
H.CORPORA["basic"](src)
seed = seed_extras
result = H.run_differential(
src, rdst, fdst, ["-a", "--delete"], ["-a", "--delete"], server,
seed=seed)
_run_and_check(case_id + "_seed", result)
os.remove(os.path.join(src, "a.txt"))
_mk(os.path.join(src, "added.txt"), b"added between runs\n")
result = H.run_differential(
src, rdst, fdst, ["-a", "--delete", "-i"],
["-a", "--delete", "-i", "--incremental"], server,
stdout=H.STDOUT_ITEMIZE)
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_one_file_system(parity_server_factory):
"""-x emits the mount-point directory but not its contents."""
case_id = "one_file_system"
local = os.stat(".")
shm = "/dev/shm"
if not os.path.isdir(shm):
pytest.skip("/dev/shm not available")
if os.stat(shm).st_dev == local.st_dev:
pytest.skip("no cross-device filesystem available")
src = os.path.join(TEST_DATA_DIR, "parity_ofs_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_ofs_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_ofs_fdst")
clean_dir(src)
_mk(os.path.join(src, "keep.txt"), b"keep\n")
probe = os.path.join(shm, f"fastsync_ofs_{os.getpid()}")
shutil.rmtree(probe, ignore_errors=True)
os.makedirs(probe)
_mk(os.path.join(probe, "inside.txt"), b"cross\n")
try:
os.symlink(probe, os.path.join(src, "nested_link"))
server = parity_server_factory(SUPER)
result = H.run_differential(
src, rdst, fdst,
["-a", "--copy-links", "-x"],
["-a", "--copy-links", "-x"], server)
_run_and_check(case_id, result)
finally:
shutil.rmtree(probe, ignore_errors=True)
@requires_rsync
@parity
def test_parity_caveats_reference_known_cases():
"""Every allowlist entry must name a real case id and aspect."""
from parity_caveats import CAVEATS
known = {c.id for c in ALL_CASES} | {
"incremental_modified", "compare_dest", "link_dest",
"added_and_deleted", "added_and_deleted_seed", "one_file_system",
}
problems = []
for case_id, entry in CAVEATS.items():
if case_id not in known:
problems.append(f"unknown case id in parity_caveats.py: {case_id!r}")
for aspect in entry:
if aspect not in ASPECTS:
problems.append(
f"{case_id!r}: unknown aspect {aspect!r} (expected {ASPECTS})")
assert not problems, "\n".join(problems)
+167 -26
View File
@@ -583,6 +583,55 @@ class TestRemoteDryRun:
assert os.path.exists(extra), f"{flags} deleted an extra in dry-run"
assert _snapshot_tree(received) == before, f"{flags} mutated the destination"
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
def test_dry_run_delete_lines_over_report_residual(self):
"""Documented residual (RSYNC_COMPAT.md `-n/--dry-run` row): FastSync's
dry-run would-delete report includes the file that is merely being
updated (derived from the receiver's STATUS_STATS extras) and, unlike
rsync, also reports an excluded-but-protected extra. rsync `-n -i
--delete` lists only genuine extras. Pins the residual that keeps the
row Divergent."""
source = os.path.join(TEST_DATA_DIR, "dryrep_src")
rdst = os.path.join(TEST_DATA_DIR, "dryrep_rdst")
fdst = os.path.join(TEST_DATA_DIR, "dryrep_fdst")
clean_dir(source)
clean_dir(rdst)
clean_dir(fdst)
with open(os.path.join(source, "a.txt"), "wb") as fh:
fh.write(b"new content\n")
os.utime(os.path.join(source, "a.txt"), (1_700_000_000, 1_700_000_000))
for root in (rdst, fdst):
with open(os.path.join(root, "a.txt"), "wb") as fh:
fh.write(b"old\n")
for name, data in (("extra.log", b"log\n"), ("extra.txt", b"extra\n")):
with open(os.path.join(root, name), "wb") as fh:
fh.write(data)
for p in (os.path.join(root, "a.txt"), os.path.join(root, "extra.log"),
os.path.join(root, "extra.txt")):
os.utime(p, (1_500_000_000, 1_500_000_000))
r = subprocess.run(["rsync", "-an", "-i", "--delete", "--exclude=*.log",
source + "/", rdst + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r.returncode == 0, r.stderr
rsync_del = {l.split(None, 1)[1] for l in r.stdout.splitlines()
if l.startswith("*deleting")}
assert rsync_del == {"extra.txt"}, f"unexpected rsync deleting set: {rsync_del}"
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, fdst,
flags=["-a", "-n", "-i", "--delete", "--exclude=*.log"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fs_del = {l.split(None, 1)[1] for l in (result.stdout or "").splitlines()
if l.startswith("*deleting")}
# Documented over-report: the transferred/updated file and the excluded
# extra appear in FastSync's would-delete set.
assert "a.txt" in fs_del, "residual changed: FastSync no longer over-reports the update"
assert "extra.log" in fs_del, "residual changed: FastSync no longer reports excluded extra"
@pytest.mark.ci
def test_remote_dry_run_quiet_is_silent(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remote_dry_quiet_src")
@@ -2376,6 +2425,61 @@ class TestTempDir:
assert not mismatches, f"Mismatch: {mismatches}"
self._assert_clean_scratch(os.path.join(dest, "scratch"))
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
def test_relative_temp_dir_matches_rsync_absolute_rejected(self):
"""Differential: a relative --temp-dir is resolved under the destination
by both (rsync 3.4.1 and FastSync), producing identical trees. An
absolute --temp-dir is used verbatim by rsync standalone, but the
receiver deliberately confines it to the receive root (security
invariant), so FastSync rejects it without writing outside the root.
"""
source = self._make_source("tempdir_diff_src")
rdst = os.path.join(TEST_DATA_DIR, "tempdir_diff_rdst")
fdst = os.path.join(TEST_DATA_DIR, "tempdir_diff_fdst")
clean_dir(rdst)
clean_dir(fdst)
os.makedirs(os.path.join(rdst, "scratch"), exist_ok=True)
os.makedirs(os.path.join(fdst, "scratch"), exist_ok=True)
r = subprocess.run(["rsync", "-a", "--temp-dir=scratch", source + "/", rdst + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r.returncode == 0, r.stderr
with ServerManager() as server:
server.start()
result, _ = run_client(source, fdst, flags=["--temp-dir=scratch"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
# rsync lays the source contents directly in rdst; FastSync mirrors the
# absolute source path below fdst. Compare the mirrored content trees
# (the scratch dir lives at each destination root).
rtree = sorted(os.path.relpath(os.path.join(dp, n), rdst)
for dp, dn, fn in os.walk(rdst)
for n in dn + fn if os.path.join(dp, n) != os.path.join(rdst, "scratch"))
mirror = get_dest_received_dir(fdst, source)
ftree = sorted(os.path.relpath(os.path.join(dp, n), mirror)
for dp, dn, fn in os.walk(mirror) for n in dn + fn)
assert rtree == ftree, f"relative temp-dir tree mismatch: {rtree} != {ftree}"
assert _walk_tmp_files(os.path.join(rdst, "scratch")) == []
assert _walk_tmp_files(os.path.join(fdst, "scratch")) == []
# Absolute temp dir: rsync accepts it; FastSync rejects it safely.
abs_scratch = os.path.join(TEST_DATA_DIR, "tempdir_diff_abs")
clean_dir(abs_scratch)
rdst2 = os.path.join(TEST_DATA_DIR, "tempdir_diff_rdst2")
clean_dir(rdst2)
r2 = subprocess.run(["rsync", "-a", "--temp-dir=" + abs_scratch, source + "/", rdst2 + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r2.returncode == 0, r2.stderr
fdst2 = os.path.join(TEST_DATA_DIR, "tempdir_diff_fdst2")
clean_dir(fdst2)
with ServerManager() as server:
server.start()
result2, _ = run_client(source, fdst2, flags=["--temp-dir", abs_scratch],
port=server.port)
assert result2.returncode != 0, "an absolute --temp-dir must be rejected (confined)"
assert os.listdir(abs_scratch) == [], "receiver wrote into an unconfined temp dir"
def test_default_behavior_has_no_scratch_dir(self, shared_server):
source = self._make_source("tempdir_default_src")
dest = os.path.join(TEST_DATA_DIR, "tempdir_default_dst")
@@ -2837,6 +2941,41 @@ class TestDelayUpdates:
assert not os.path.isdir(os.path.join(delay_dest, self.STAGING)), \
"staging directory left behind after a successful delayed transfer"
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
def test_delay_updates_staging_name_collision_residual(self):
"""Documented residual (RSYNC_COMPAT.md `--delay-updates` row): FastSync
uses a fixed `.fastsync-stage` staging name and wipes a pre-existing tree
of that name at the start of a delayed run (crash-leftover cleanup),
even without `--delete`; rsync leaves a genuine destination entry of that
name untouched. Pins the divergence that keeps the row Divergent."""
source = self._make_source("delay_collide_src")
rdst = os.path.join(TEST_DATA_DIR, "delay_collide_rdst")
fdst = os.path.join(TEST_DATA_DIR, "delay_collide_fdst")
clean_dir(rdst)
clean_dir(fdst)
for root in (rdst, fdst):
with open(os.path.join(root, "top.txt"), "wb") as fh:
fh.write(b"old\n")
stage = os.path.join(root, self.STAGING)
os.makedirs(stage, exist_ok=True)
with open(os.path.join(stage, "keepme.txt"), "wb") as fh:
fh.write(b"genuine user data\n")
r = subprocess.run(["rsync", "-a", "--delay-updates", source + "/", rdst + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r.returncode == 0, r.stderr
assert os.path.exists(os.path.join(rdst, self.STAGING, "keepme.txt")), \
"rsync removed an unrelated destination entry named like the staging dir"
with ServerManager() as server:
server.start()
result, _ = run_client(source, fdst, flags=["--delay-updates"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert not os.path.exists(os.path.join(fdst, self.STAGING)), \
"FastSync did not wipe the reserved staging name (residual changed)"
@pytest.mark.parametrize("mt", [False, True])
def test_delay_updates_incremental_rerun_no_leftovers(self, shared_server, mt):
source = self._make_source("delay_rerun_src")
@@ -3523,23 +3662,25 @@ class TestMissingArgs:
class TestNoImpliedDirs:
"""--no-implied-dirs (only meaningful with -R + --files-from) refuses to
place a listed file whose parent directory is not itself listed."""
"""--no-implied-dirs (meaningful with -R) omits the source metadata of a
listed path's implied parent directories but still creates those parents
with default attributes, matching rsync 3.4.1."""
def _make(self):
return _make_relative_source("noimplied_src")
@pytest.mark.parametrize("mt", [False, True])
def test_implied_dir_only_fails_entry(self, shared_server, mt):
def test_implied_dir_created_with_default_attrs(self, shared_server, mt):
source = self._make()
dest = os.path.join(TEST_DATA_DIR, "noimplied_dst")
clean_dir(dest)
lst = _write_rel_list(b"a/b.txt\n") # "a" itself is not listed
flags = ["--files-from", lst, "-R", "--no-implied-dirs"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode != 0, "implied parent directory was not rejected"
assert "--no-implied-dirs" in (result.stderr or result.stdout)
assert not os.path.exists(os.path.join(dest, "a", "b.txt"))
assert result.returncode == 0, \
f"implied parent directory was not created: {result.stderr[:200]}"
assert os.path.isdir(os.path.join(dest, "a")), "implied parent 'a' was not created"
assert _read_file(os.path.join(dest, "a", "b.txt")) == b"nested\n"
@pytest.mark.parametrize("mt", [False, True])
def test_listed_dir_allows_file(self, shared_server, mt):
@@ -3666,9 +3807,10 @@ class TestDirs:
files.extend(os.path.relpath(os.path.join(root, n), mirror) for n in names)
assert files == [], f"--dirs descended into contents: {files}"
def test_dirs_listed_dir_colliding_with_file_fails(self, shared_server):
"""A listed directory that already exists as a regular file at the
destination fails the transfer cleanly instead of clobbering the file."""
def test_dirs_listed_dir_replaces_blocking_file(self, shared_server):
"""rsync parity: a listed directory replaces a regular file already at
its destination path (rsync removes the non-directory and creates the
directory)."""
source = self._make()
dest = os.path.join(TEST_DATA_DIR, "dirs_coll_dst")
clean_dir(dest)
@@ -3678,8 +3820,10 @@ class TestDirs:
lst = _write_rel_list(b"dir1\n")
result, _ = run_client(source, dest, flags=["--files-from", lst, "--dirs", "-R"],
port=shared_server.port)
assert result.returncode != 0, "dir entry over an existing file did not fail"
assert os.path.isfile(blocker), "blocking regular file was clobbered"
assert result.returncode == 0, \
f"dir entry over an existing file failed: {(result.stderr or result.stdout)[:300]}"
assert os.path.isdir(blocker) and not os.path.islink(blocker), \
"blocking regular file was not replaced by the incoming directory"
class TestMkpath:
@@ -6696,11 +6840,10 @@ class TestDirectoryAndSymlinkTimes:
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_preserve_does_not_create_empty_source_dir(self, shared_server, mt):
"""P7 Wave D #1: a captured-but-EMPTY source directory is never created
at the destination. The scanner records its time (it is transmitted via
STATUS_DIR_TIMES), but the receiver treats that entry as record-only, so
`-a` keeps the documented "empty dirs are never transferred" behavior."""
def test_preserve_creates_empty_source_dir(self, shared_server, mt):
"""rsync parity: a recursive `-a` transfer recreates an empty source
directory at the destination (the scanner emits it as an explicit
directory entry)."""
source = os.path.join(TEST_DATA_DIR, f"empty_dir_{'m' if mt else 's'}_src")
dest = os.path.join(TEST_DATA_DIR, f"empty_dir_{'m' if mt else 's'}_dst")
clean_dir(source)
@@ -6711,8 +6854,8 @@ class TestDirectoryAndSymlinkTimes:
flags = ["-a"] + (["--threads"] if mt else [])
received = self._run(source, dest, flags, shared_server)
assert os.path.isfile(os.path.join(received, "keep.txt")), "regular file missing"
assert not os.path.lexists(os.path.join(received, "empty_sub")), \
f"-a created an empty source directory at {received}/empty_sub"
assert os.path.isdir(os.path.join(received, "empty_sub")), \
f"-a did not recreate the empty source directory at {received}/empty_sub"
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
@@ -6735,10 +6878,10 @@ class TestDirectoryAndSymlinkTimes:
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_collision_at_dir_time_path_does_not_abort(self, shared_server, mt):
"""P7 Wave D #1: a pre-existing regular file at a source-empty-dir's
mirror path must not abort the transfer (the old mkdir failed and failed
the run) and must not be clobbered."""
def test_collision_at_empty_dir_path_replaces_blocker(self, shared_server, mt):
"""rsync parity: a pre-existing regular file at a source empty-dir's
mirror path is replaced by the incoming directory (rsync removes the
non-directory and creates the directory); the run succeeds."""
source = os.path.join(TEST_DATA_DIR, f"dirtime_collide_{'m' if mt else 's'}_src")
dest = os.path.join(TEST_DATA_DIR, f"dirtime_collide_{'m' if mt else 's'}_dst")
clean_dir(source)
@@ -6757,10 +6900,8 @@ class TestDirectoryAndSymlinkTimes:
assert result.returncode == 0, \
f"-a aborted on a pre-existing file at an empty-dir path: " \
f"{(result.stderr or result.stdout)[:400]}"
assert os.path.isfile(blocker) and not os.path.islink(blocker), \
"the pre-existing blocker was replaced by a directory"
with open(blocker, "rb") as fh:
assert fh.read() == b"pre-existing blocker\n", "the blocker file was clobbered"
assert os.path.isdir(blocker) and not os.path.islink(blocker), \
"the pre-existing blocker was not replaced by the incoming directory"
assert os.path.isfile(os.path.join(received, "keep.txt")), "regular file missing"
+34 -23
View File
@@ -1,10 +1,12 @@
"""--iconv=CONVERT_SPEC file-NAME charset conversion integration tests.
The client converts every source file name from LOCAL to REMOTE before it goes
on the wire, and the receiver converts it back from REMOTE to LOCAL, so a
source tree using one charset can be written into a destination tree using
another (rsync compatibility; content bytes are never touched).
rsync's spec is ``--iconv=LOCAL,REMOTE`` (the order is the same push or pull).
The sender converts each source name from LOCAL to REMOTE for the wire, and on
a PUSH the receiver's charset is the spec's REMOTE half, so it writes the wire
bytes verbatim (only a server with its own ``--iconv`` declares a different
destination charset and re-converts). Content bytes are never touched.
"""
import codecs
import os
import shutil
@@ -16,6 +18,11 @@ LATIN1_NAME = b"caf\xe9.txt"
UTF8_NAME = "caf\u00e9.txt".encode("utf-8")
def _to_utf8(name_bytes):
"""The UTF-8 encoding of a name that is stored as ISO-8859-1 bytes."""
return codecs.encode(codecs.decode(name_bytes, "iso-8859-1"), "utf-8")
def _make(tag):
source = os.path.join(TEST_DATA_DIR, f"iconv_{tag}_src")
dest = os.path.join(TEST_DATA_DIR, f"iconv_{tag}_dst")
@@ -41,10 +48,10 @@ def _dest_file(source, dest, name):
@pytest.mark.ci
def test_iconv_latin1_roundtrip(shared_server):
"""A source file whose name is ISO-8859-1 bytes is transferred with
--iconv=iso-8859-1,utf-8 and lands on the destination with the ORIGINAL
latin1 name (the wire carried it as UTF-8)."""
def test_iconv_latin1_to_utf8_dest(shared_server):
"""rsync push parity: --iconv=iso-8859-1,utf-8 converts a latin1 source name
to the spec's REMOTE (UTF-8) on the wire and the default receiver writes it
verbatim, so the destination name is UTF-8 (not the source's latin1)."""
source, dest = _make("latin1")
_place_bytes(source, LATIN1_NAME)
@@ -53,8 +60,10 @@ def test_iconv_latin1_roundtrip(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
dst = _dest_file(source, dest, LATIN1_NAME)
assert os.path.exists(dst), f"dest latin1-named file not found under {dest}"
dst = _dest_file(source, dest, UTF8_NAME)
assert os.path.exists(dst), f"dest UTF-8-named file not found under {dest}"
assert not os.path.exists(_dest_file(source, dest, LATIN1_NAME)), \
"destination kept the latin1 name instead of the wire (UTF-8) charset"
@pytest.mark.ci
@@ -153,7 +162,7 @@ def test_iconv_expanding_name_growth(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert os.path.exists(_dest_file(source, dest, name_bytes))
assert os.path.exists(_dest_file(source, dest, _to_utf8(name_bytes)))
def test_iconv_symlink_path_and_target(shared_server):
@@ -170,11 +179,13 @@ def test_iconv_symlink_path_and_target(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
dst_target = _dest_file(source, dest, target)
dst_link = _dest_file(source, dest, b"link\xe9")
assert os.path.exists(dst_target), "dest latin1 target file missing"
assert os.path.islink(dst_link), "dest latin1 symlink missing"
assert os.readlink(dst_link) == target, "symlink target not preserved/decoded"
utf8_target = _to_utf8(target)
utf8_link = _to_utf8(b"link\xe9")
dst_target = _dest_file(source, dest, utf8_target)
dst_link = _dest_file(source, dest, utf8_link)
assert os.path.exists(dst_target), "dest UTF-8 target file missing"
assert os.path.islink(dst_link), "dest UTF-8 symlink missing"
assert os.readlink(dst_link) == utf8_target, "symlink target not wire-converted"
with open(dst_link, "rb") as fh:
assert fh.read() == b"t\n"
@@ -198,8 +209,8 @@ def test_iconv_hardlink_path_and_target(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
dst_a = _dest_file(source, dest, a)
dst_b = _dest_file(source, dest, b)
dst_a = _dest_file(source, dest, _to_utf8(a))
dst_b = _dest_file(source, dest, _to_utf8(b))
assert os.path.exists(dst_a) and os.path.exists(dst_b)
assert os.stat(dst_a).st_ino == os.stat(dst_b).st_ino, \
"hard-link relationship not preserved across the transfer"
@@ -221,16 +232,16 @@ def test_iconv_delete_manifest_consistent(shared_server):
flags = ["--iconv=iso-8859-1,utf-8"]
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert os.path.exists(_dest_file(source, dest, keep))
assert os.path.exists(_dest_file(source, dest, gone))
assert os.path.exists(_dest_file(source, dest, _to_utf8(keep)))
assert os.path.exists(_dest_file(source, dest, _to_utf8(gone)))
os.remove(os.path.join(os.fsencode(source), gone))
result, _ = run_client(
source, dest, flags=flags + ["--delete"], port=server.port
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert os.path.exists(_dest_file(source, dest, keep)), "kept file deleted"
assert not os.path.exists(_dest_file(source, dest, gone)), \
assert os.path.exists(_dest_file(source, dest, _to_utf8(keep))), "kept file deleted"
assert not os.path.exists(_dest_file(source, dest, _to_utf8(gone))), \
"missing file was not deleted"
@@ -247,4 +258,4 @@ def test_iconv_chunk_serialization_blob(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert os.path.exists(_dest_file(source, dest, name))
assert os.path.exists(_dest_file(source, dest, _to_utf8(name)))
@@ -769,6 +769,47 @@ class TestVerifyAndFlip:
assert os.stat(dest_file).st_ino == os.stat(basis_file).st_ino, \
"--link-dest must hard-link to the basis file"
@requires_rsync
def test_basis_dir_size_only_content_residual(self, shared_server):
"""Documented residual (RSYNC_COMPAT.md basis-dir rows): FastSync
xxHash-verifies a basis hit, while rsync's `--size-only` quick check
trusts the size alone. With a same-size, different-content basis,
rsync links/copies the wrong basis content while FastSync transfers the
source. This test pins both observed behaviors (FastSync is stricter,
so the rows are reclassified Divergent)."""
source = self._src("basissz")
rdest = self._dst("basissz_r")
fdest = self._dst("basissz_f")
with open(os.path.join(source, "f.txt"), "wb") as fh:
fh.write(b"AAAA\n")
OLD = 1_400_000_000
# rsync basis at the transfer-relative path (relative to the dest dir).
os.makedirs(os.path.join(rdest, "basis"), exist_ok=True)
with open(os.path.join(rdest, "basis", "f.txt"), "wb") as fh:
fh.write(b"BBBB\n")
os.utime(os.path.join(rdest, "basis", "f.txt"), (OLD, OLD))
rs = _rsync(["-a", "--size-only", "--link-dest=basis", source + "/", rdest + "/"])
assert rs.returncode == 0, rs.stderr
with open(os.path.join(rdest, "f.txt"), "rb") as fh:
assert fh.read() == b"BBBB\n", "rsync --size-only did not trust the basis size"
# FastSync basis is relative to the receive root; the file mirrors the
# source path.
rel = os.path.abspath(source).lstrip(os.sep)
basis = os.path.join(fdest, "basis", rel)
os.makedirs(basis, exist_ok=True)
with open(os.path.join(basis, "f.txt"), "wb") as fh:
fh.write(b"BBBB\n")
os.utime(os.path.join(basis, "f.txt"), (OLD, OLD))
received = get_dest_received_dir(fdest, source)
result, _ = run_client(source, fdest,
flags=["-a", "--size-only", "--link-dest=basis", "--incremental"],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
with open(os.path.join(received, "f.txt"), "rb") as fh:
assert fh.read() == b"AAAA\n", \
"FastSync must verify the basis content and transfer the source"
class TestIgnoreExistingShortCircuit:
"""#9: --ignore-existing is decided by the receiver during the per-file
@@ -111,6 +111,45 @@ class TestRelativeGeneral:
assert int(rs.st_mtime) == int(fs.st_mtime), \
f"mtime mismatch for {rel} with {extra}"
@requires_rsync
@pytest.mark.ci
def test_no_implied_dirs_files_from_matches_rsync(self, shared_server):
"""-R --no-implied-dirs --files-from: a listed file whose parent is not
itself listed still transfers; the implied parent is created with
default attributes (rsync 3.4.1 parity)."""
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_nidff_src"))
# Make the implied parent unmistakably non-default on the source so a
# wrongly-applied attribute would be observable.
os.chmod(os.path.join(source, "foo"), 0o700)
os.chmod(os.path.join(source, "foo", "bar"), 0o711)
os.utime(os.path.join(source, "foo"), (978307200, 978307200))
os.utime(os.path.join(source, "foo", "bar"), (978307200, 978307200))
lst = os.path.join(TEST_DATA_DIR, "sel_nidff_list")
with open(lst, "w") as fh:
fh.write("foo/bar/baz/f.txt\n")
dest = os.path.join(TEST_DATA_DIR, "sel_nidff_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_nidff_rdst")
clean_dir(dest)
clean_dir(rdst)
r = _rsync(["-rlpt", "-R", "--no-implied-dirs", "--files-from=" + lst,
source + "/", rdst + "/"])
assert r.returncode == 0, r.stderr
result, _ = run_client(source, dest, flags=[
"-rlpt", "-R", "--no-implied-dirs", "--files-from", lst],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert _tree(rdst) == _tree(dest), "implied-parent layout mismatch"
# The implied parents exist on both sides and carry the run-time default
# attributes, not the source's (non-default) ones.
for rel in ("foo", "foo/bar", "foo/bar/baz"):
rs = os.stat(os.path.join(rdst, rel))
fs = os.stat(os.path.join(dest, rel))
assert (rs.st_mode & 0o7777) == (fs.st_mode & 0o7777), \
f"mode mismatch for implied {rel}"
# The listed file is transferred with its content.
with open(os.path.join(dest, "foo", "bar", "baz", "f.txt"), "rb") as fh:
assert fh.read() == b"deep\n"
class TestDirsOneLevel:
"""#13: -d with a trailing slash (or '.') lists the source's immediate
+17 -3
View File
@@ -148,10 +148,23 @@ static void test_iconv_wire_sender_converts_local_to_remote() {
charset_wire_free();
}
static void test_iconv_wire_receiver_converts_remote_to_local() {
/* rsync push parity: with no server --iconv the destination charset is the
* client spec's REMOTE half, so the receiver writes the wire bytes verbatim. */
static void test_iconv_wire_receiver_default_writes_remote() {
EXPECT_TRUE(charset_wire_init_receiver("utf-8,iso-8859-1", NULL));
char* local = charset_wire_apply("caf\xe9");
EXPECT_NOT_NULL(local);
EXPECT_EQ_INT(strcmp(local, "caf\xe9"), 0);
free(local);
charset_wire_free();
}
/* A server that declares its own --iconv LOCAL converts wire(REMOTE) into that
* declared charset (the daemon "charset" analog). */
static void test_iconv_wire_receiver_server_local_override() {
EXPECT_TRUE(charset_wire_init_receiver("utf-8,iso-8859-1", "utf-8"));
char* local = charset_wire_apply("caf\xe9");
EXPECT_NOT_NULL(local);
EXPECT_EQ_INT(strcmp(local, "caf\xc3\xa9"), 0);
free(local);
charset_wire_free();
@@ -179,7 +192,7 @@ static void test_iconv_wire_str_roundtrip() {
close(p[1]);
io_set_fds(p[0], p[0]);
charset_wire_free();
charset_wire_init_receiver("utf-8,iso-8859-1", NULL);
charset_wire_init_receiver("utf-8,iso-8859-1", "utf-8");
char* got = receive_wire_str(p[0]);
bool ok = got != NULL && strcmp(got, "caf\xc3\xa9") == 0;
free(got);
@@ -211,7 +224,8 @@ void test_iconv() {
test_iconv_exact_fill_no_overflow();
test_iconv_growth_expanding_name();
test_iconv_wire_sender_converts_local_to_remote();
test_iconv_wire_receiver_converts_remote_to_local();
test_iconv_wire_receiver_default_writes_remote();
test_iconv_wire_receiver_server_local_override();
test_iconv_wire_disabled_passthrough();
// This subtest forks to exercise the wire string handshake; the instrumented
// parent is too slow under valgrind for the child's blocking reads.