Release v2.28.0 #304

Merged
TapTap merged 39 commits from dev into main 2026-09-20 01:17:26 +02:00
19 changed files with 546 additions and 62 deletions
Showing only changes of commit eb7e3fd2e0 - Show all commits
+18 -2
View File
@@ -8,8 +8,8 @@
- **Release PR #284 (`dev` -> `main`)** open, CI green (run 553).
`main` is protected: it needs review/approval to merge.
https://gitea.tap-tap.win/TapTap/FastSync/pulls/284
- **`PROTOCOL_VERSION` = `"2.26.0"`** (`src/shared/config.h`); CMake
`project(FastFileTransfer VERSION 2.26.0)`.
- **`PROTOCOL_VERSION` = `"2.27.0"`** (`src/shared/config.h`); CMake
`project(FastFileTransfer VERSION 2.27.0)`.
- Working tree clean; no wave worktrees remain.
## What landed this session
@@ -41,6 +41,22 @@
6. **Rsync-parity wave (protocol 2.23.0)** on `feat/rsync-parity`: rsync short options/clustering/attached values (`-r`/`-b`/`-L`/`-B`, `-av`, `-aAX`, `-B1000`, `-essh`, `-MOPT`), `-c` checksum quick-check, `--checksum-choice`/`--compress-choice` validation and seed randomization, rsync timeout/max-alloc defaults, temp-dir confinement + `EXDEV` fallback, ownership/mapping parity (numeric-ids modifier, map ranges/`*`/empty-FROM, `--chown`+map conflicts, fake-super resolved-owner record), verbatim symlink storage with rsync `--safe-links`/`--munge-links`, socket recreation under `--specials`, `--chmod` 3.4.1 semantics, and delete scoping + `--max-delete` partial/exit-25. Wire: appended delete-manifest synchronized-directory section and `STATUS_DELETE_LIMIT`.
7. **Parity-completion wave (protocol 2.24.0 → 2.26.0)** on `feat/parity-completion`: per-directory delete plans (`STATUS_DELETE_PLAN`) for `--delete-during`/`--delete-delay`; receiver `STATUS_STATS` counters feeding `--stats`/`--progress` and `--out-format %b/%c/%C`, plus `-n --delete` lines; `lz4`/`zlib`/`zlibx` compression and `md4`/`sha1`/`none` checksums with `auto` negotiation (default `xxh128`/`zstd`); general `-R`/`--no-implied-dirs`/`-d`; the full filter grammar (`merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` + modifiers) and corrected `-F`/`-FF`; receiver-side `--chown`/map TO-name resolution; absolute basis dirs + `--link-dest` relink; receiver-side `--ignore-existing` short-circuit; `--preallocate` over `--sparse` via `fallocate(2)`; `--iconv=.`/`-`/`--no-iconv`; lone `-h` help; aliases `--ignore-non-existing`/`--protect-args`/`--msgs2stderr`; and the full `--info`/`--debug` vocabulary. `RSYNC_COMPAT.md` reclassifies the matrix to 106 ✅ / 27 ⚠️ / 23 ❌; the later rsync-parity-stats pass (`fix/parity-stats`) moves it to 107 ✅ / 25 ⚠️ / 24 ❌ (see item 8).
8. **rsync-parity-stats pass** on `fix/parity-stats` (no wire change, `PROTOCOL_VERSION` stays `2.26.0`): `--delete-delay` now counts/budgets only entries actually removed (a refilled deferred directory that survives `ENOTEMPTY` is not counted; `--max-delete` partial-delete count matches rsync); `--stats` gained the `(reg/dir/link/special)` `Number of files` breakdown and now counts only regular files actually stored for `Number of regular files transferred`/transferred size/literal data (up-to-date re-runs report 0); `Total file size` includes symlink target lengths; `--progress` prints the leading `./` root line and counts it in `to-chk` so a single-file transfer matches rsync; and `%C` uses the selected transfer checksum with `checksum_digest_file` supporting md4/sha1/none, byte-identical to rsync for every algorithm. `--out-format` reclassified ❌ (`%b`/delta-`%c` are protocol-specific). Differential + regression tests added; full suite + ASan + clang-format + cppcheck clean.
9. **Option-parity wave (protocol 2.26.0 → 2.27.0, on `fix/parity-options`):**
`--bwlimit` now ports rsync 3.4.1's units/quantization and paces like its
leaky bucket; `--ignore-errors` reproduces rsync's default (an I/O error
skips deletion unless the flag is set; the readable tree still transfers and
the run exits 23) across every delete timing; the `--info` categories with a
FastSync event (`name`/`flist`/`del`/`remove`/`nonreg`/`progress`) emit
rsync's line format, with real-run `deleting`/`*deleting` lines carried over
the new trailing config bool `report_deletes` (golden wire updated by
`tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
over daemon/TCP (no argv channel in FastSync's binary config handshake;
rsync-daemon differential pins the rsync behavior) and receiver-side
`protect`/`risk` re-derivation for destination-only entries (would need a
receiver filter engine; differential pins the divergence). Matrix now
**109 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
`--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
## Next steps
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
+1 -1
View File
@@ -789,7 +789,7 @@ before the module list, before authentication, and the connecting peer address
## Protocol and Security
FastSync protocol version `2.26.0` is shared by the client and server. The
FastSync protocol version `2.27.0` is shared by the client and server. The
current protocol is sender-driven and includes configuration negotiation,
including the maximum allocation limit, incremental checks, checksums,
manifests, keep-alives, abort handling, per-file remove-source results, and
+23 -9
View File
@@ -6,9 +6,9 @@ This document maps rsync's full feature set to FastSync's current implementation
| Status | Count | Description |
|--------|-------|-------------|
| ✅ Parity | 107 | Reproduces rsync's semantics for this option's scope |
| ⚠️ Caveat | 25 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
| ❌ Divergent | 24 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
| ✅ Parity | 109 | Reproduces rsync's semantics for this option's scope |
| ⚠️ Caveat | 21 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
| ❌ Divergent | 26 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
| **Total** | **156** | One row per rsync option/feature group; a row may name several spellings |
This matrix reports honest rsync parity, not "implemented" as a synonym for
@@ -21,6 +21,20 @@ the batch container, `--fake-super`'s xattr format, `--copy-as` credential
switching), or impossible (`-N`/`--crtimes`). The counts are derived from the
rows below; update them together with the table.
**Option wave (protocol 2.26.0 → 2.27.0).** A differential pass against rsync
3.4.1 over the remaining option caveats. `--bwlimit` now parses rsync's units
exactly and paces like rsync's leaky bucket; `--ignore-errors` reproduces
rsync's default (an I/O error skips deletion unless the flag is set, while the
readable tree still transfers and the run exits 23); the `--info` categories
that map to a FastSync event (`name`, `flist`, `del`, `remove`, `nonreg`,
`progress`) now emit rsync's line format, including real-run `deleting PATH` /
`*deleting` lines carried over a new `report_deletes` wire bool; and two
genuinely non-interoperable residuals are reclassified divergent (`-M` over a
daemon/TCP connection, which FastSync's binary config handshake has no argv
channel for, and receiver-side `protect`/`risk` re-derivation for
destination-only entries, which would need a receiver filter engine). That moves
the matrix to **109 ✅ / 21 ⚠️ / 26 ❌ = 156**.
**Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the
remaining gaps the rsync-parity wave left open (delete timing, wire counters and
output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side
@@ -51,7 +65,7 @@ Every one of those has an entry below with its remaining caveats.
| `-q`, `--quiet` | Suppress non-error messages | ✅ Parity | Suppresses client output while preserving errors |
| `--help` | Show help | ✅ Parity | Prints usage and exits. A lone `-h` with no other transfer arguments also prints help (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer keeps its rsync meaning of `--human-readable` (see that row) |
| `-V`, `--version` | Print version | ✅ Parity | |
| `--info=FLAGS` | Fine-grained info verbosity | ⚠️ Caveat | Protocol 2.26.0 accepts rsync 3.4.1's full `--info` vocabulary — `backup`, `copy`, `del`, `flist`, `misc`, `mount`, `name`, `nonreg`, `progress`, `remove`, `skip`, `stats`, `symsafe`, `all`, `none` — with optional level suffixes (`--info=stats2`), so a valid rsync invocation is never rejected up front. The categories that map to a FastSync channel emit (`copy`, `name`, `misc`, `skip`, `stats`); the remaining rsync categories are accepted silently, with no output. `none` suppresses info output, explicit flags override `--verbose`, and a genuinely unknown name is still rejected by name (matching rsync). **Caveat:** many accepted rsync categories produce no output (e.g. `del`, `flist`, `remove`, `progress`, `symsafe`, `mount`, `nonreg`, `backup`), so e.g. `--info=progress` is accepted for CLI compatibility only; `name` maps to the `copy` channel rather than rsync's per-file name output |
| `--info=FLAGS` | Fine-grained info verbosity | ⚠️ Caveat | Accepts rsync 3.4.1's full `--info` vocabulary — `backup`, `copy`, `del`, `flist`, `misc`, `mount`, `name`, `nonreg`, `progress`, `remove`, `skip`, `stats`, `symsafe`, `all`, `none` — with optional level suffixes (`--info=stats2`), so a valid rsync invocation is never rejected up front. Protocol 2.27.0 wires the categories that map to a real FastSync event, matching rsync's line format: `name` prints the updated entry names (with the ` -> target` link suffix), `flist` prints `sending incremental file list`, `del` prints `deleting PATH` (or `*deleting PATH` under `-i`/`--out-format`) for both dry-run would-delete and real deletions (real runs carry the removed paths over the new `report_deletes` wire bool), `remove` prints `sender removed PATH`, `nonreg` prints `skipping non-regular file "NAME"`, `progress` drives the per-file progress output, and `copy`/`misc`/`skip`/`stats` keep their existing channels. `none` suppresses info output, explicit flags override `--verbose`, and a genuinely unknown name is still rejected by name (matching rsync). **Caveat:** the categories with no client-observable event stay accepted-but-silent — `symsafe`, `mount`, and `backup` (the backup happens on the receiver, which FastSync's protocol does not echo back); `name` level 2 (`is uptodate` lines) and the leading `./` root name line are not emitted; and `skip` maps to FastSync's sender-side skip logging rather than rsync's receiver-side "not creating new file" lines |
| `--debug=FLAGS` | Fine-grained debug verbosity | ⚠️ Caveat | Protocol 2.26.0 accepts rsync 3.4.1's full `--debug` vocabulary with optional level suffixes. FastSync emits for its own channels (`io`, `proto`, `pack`, `util`, plus the aliases `hl`/`owner`); the rsync-only categories (`acl`, `filter`, `send`, ...) are accepted silently. `--debug=help` lists the flags; a genuinely unknown name is rejected by name. **Caveat:** most accepted rsync categories produce no output (e.g. `acl`, `filter`, `send`, `flist`, `del`, `deltasum`, `hash`, `recv`, `time`), so they are accepted for CLI compatibility only |
| `--stderr=MODE` | Change stderr output mode | ❌ Divergent | `errors` (default) and `all` are supported; `client` is rejected with a clear error (`--stderr=client is not supported`) because FastSync has no rsync client-message channel — the rejection itself is the documented behavior (Phase 7 Wave B decision). The modes that exist work; the missing rsync channel cannot be emulated without a wire change |
| `--msgs2stderr`, `--no-msgs2stderr` | Deprecated `--stderr` aliases | ⚠️ Caveat | `--msgs2stderr` maps to `--stderr=all` (supported, matching rsync). `--no-msgs2stderr` is rsync's spelling of `--stderr=client`, which FastSync has no client-message channel for, so it maps to the errors-only default instead of reproducing rsync's client mode. See `--stderr=MODE` |
@@ -81,7 +95,7 @@ Every one of those has an entry below with its remaining caveats.
|------|-------------------|-----------------|-------|
| `--exclude-from=FILE` | Read exclude patterns from file | ✅ Parity | Reads patterns from file |
| `--include-from=FILE` | Read include patterns from file | ✅ Parity | Reads patterns from file |
| `--filter=RULE` | Add file-filtering rule | ⚠️ Caveat | The short `-f` **is** bound to `--filter` (the old FastSync sendfile conflict is gone; sendfile is long-only `--sendfile`), and `-f RULE`, `-f=RULE`, `--filter=RULE` and the two-argument form all parse. Protocol 2.26.0 implements rsync's filter grammar: `+`/`-`, `include`/`exclude`, a leading `/` anchor (to the transfer root or a `.rsync-filter` file's directory), a trailing `/` dir-only rule, and the `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R` and `clear`/`!` words, including the `:`/`.` modifiers. First match wins; the filter layer is independent of `--exclude`/`--include`. **Remaining divergence:** the receiver-mirror protection a `protect`/`risk` rule produces is derived from the sender's source traversal, so a rule that would match only a destination-only entry is not re-derived on the receiver; destination-only deletion protection continues to come from the ordinary sender-derived protected-prefix mechanism |
| `--filter=RULE` | Add file-filtering rule | ❌ Divergent | The short `-f` **is** bound to `--filter` (the old FastSync sendfile conflict is gone; sendfile is long-only `--sendfile`), and `-f RULE`, `-f=RULE`, `--filter=RULE` and the two-argument form all parse. Protocol 2.26.0 implements rsync's filter grammar: `+`/`-`, `include`/`exclude`, a leading `/` anchor (to the transfer root or a `.rsync-filter` file's directory), a trailing `/` dir-only rule, and the `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R` and `clear`/`!` words, including the `:`/`.` modifiers. First match wins; the filter layer is independent of `--exclude`/`--include`. **Reclassified because the receiver-side `protect`/`risk` semantics cannot be reproduced:** rsync maintains an independent filter engine on the receiver and re-applies every rule to the destination during deletion, so a `P *.log` rule protects a destination-only `extra.log` (differential test vs rsync 3.4.1 pins this). FastSync is sender-derived: its delete protection is the set of source paths the scan actually pruned, so a rule that matches only a destination-only entry is never re-derived and the extra is deleted. Closing this would require shipping the whole (including per-directory merge) filter grammar to, and re-implementing rsync's dual-sided engine on, the receiver — a protocol/architecture change out of proportion to the residual |
| `--files-from=FILE` | Read source file list from file | ✅ Parity | Entries are paths relative to the source root (leading `./` stripped, `..`/absolute rejected at parse time, blank lines ignored; NUL-delimited with `-0`). A listed regular file is transferred; a listed directory transfers its whole subtree (FastSync recursion is always on). Non-listed paths are pruned by the scanner; the delete manifest is scoped to the listed directory subtrees. A listed entry that does not exist is a hard error unless `--ignore-missing-args`/`--delete-missing-args` is given. **An empty list is a zero-transfer success (exit 0), matching rsync 3.4.1** — the earlier claim that rsync reports "no source files specified" was wrong. Scalability note: `file_list_affects` is O(list size) per scanned entry, so a very large list against a huge tree is quadratic (the documented bound) |
| `-0`, `--from0` | Delimit *-from files with NULs | ✅ Parity | `--files-from` entries become NUL-delimited; the flag may appear before or after `--files-from` on the command line. NUL mode preserves entry bytes exactly (trailing CR/LF are part of the name; only newline mode trims them) |
| `--max-size=SIZE` | Skip files larger than SIZE | ✅ Parity | `max_size` in scanner |
@@ -142,7 +156,7 @@ Every one of those has an entry below with its remaining caveats.
| `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. The delete-after timing is also what plain `--delete` does: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing |
| `--delete-excluded` | Also delete excluded files | ⚠️ Caveat | `delete_excluded` config field. Under `--delete` FastSync protects (rsync's default) the destination mirror of paths the sender's source scan pruned by the user-selection rules — the `--filter`/`-F`/`-C` layer and the legacy `--exclude`/`--include` layer. The sender transmits those concrete pruned paths as **protected prefixes** in the delete-manifest frame (see the Phase-3 notes below); the walker never descends into or removes them. `--delete-excluded` opts back in: the sender sends an empty protected list, so the excluded destination mirrors become ordinary extras and are removed. **`--max-size`/`--min-size` pruned mirrors are a separate, always-on protection** (protocol 2.23.0, rsync parity): size-pruned source mirrors survive `--delete` even with `--delete-excluded`. Divergences (documented): protection is derived only from what the source scan actually pruned — a stray destination-only file that happens to match an exclude rule is not protected (FastSync never re-applies rules to the destination, keeping deletion sender-derived) |
| `--max-delete=NUM` | Max files to delete | ✅ Parity | `max_delete` config field (default -1 = no client limit; 0 = delete nothing). **Protocol 2.23.0 matches rsync's partial semantics:** the receiver deletes up to NUM entries (regular files, symlinks and empty directories; each directory removal counts as one) and then **stops deleting, skips the rest, and reports the run as partial**. The client prints a "deletions stopped due to `--max-delete` limit" message and exits **25** (rsync's `RERR_PARTIAL`), not a hard failure — the transfer itself succeeded. NUM only applies together with `--delete` (it is inert otherwise, matching rsync). A client NUM below the server hard bound `MAX_SERVER_DELETE_COUNT` (100000) replaces it; a NUM above it never raises that cap. Deleting an entire destination with no limit is still bounded by the server's 100000-entry ceiling. `--delete-missing-args` exact-path deletions and the ordinary extras walk draw from the same budget, matching rsync |
| `--ignore-errors` | Delete even with I/O errors | ⚠️ Caveat | Sender-side, client-only config field. rsync suppresses `--delete` when the transfer had I/O errors; FastSync's equivalent is a source-scan I/O error (an unreadable directory, e.g. EACCES). By default the scan aborts the run so no deletion happens. With `--ignore-errors` the scan continues past the unreadable directory, the readable tree is transferred, deletion still runs (the unreadable directory's mirror is treated as an extra), and the run exits 23 (`RERR_PARTIAL`), matching rsync. **Remaining divergence / uncertainty:** the EACCES differential is not exercised in CI because the runner is root (mode 000 is still readable), so this rests on source inspection plus the setpriv integration test |
| `--ignore-errors` | Delete even with I/O errors | ✅ Parity | Sender-side, client-only config field. Matches rsync's semantics exactly: an unreadable source subdirectory is always skipped so the readable tree transfers (the transfer root itself stays fatal), and the run reports rsync's partial-transfer exit **23**. Deletion policy follows rsync: by default an I/O error suppresses deletion (`IO error encountered -- skipping file deletion`), while `--ignore-errors` lets the deletion commit. The decision applies to every timing (`--delete`, `--delete-before`, `--delete-during`, `--delete-delay`, `--delete-after`) in both the sequential and `--threads` send paths. Differential-tested against rsync 3.4.1 with both tools run as an unprivileged user (mode-000 source directory); the reference build's root-only gate still excludes the EACCES differential, but the setpriv differential test exercises it. The piece that stays FastSync-specific is documented under the recursive-empty-directory residual: FastSync never emits an unreadable (or empty) directory entry, so that mirror is an extra that a run with `--ignore-errors` removes, where rsync emits the directory and keeps its mirror |
| `--force` | Force deletion of non-empty dirs | ✅ Parity | `force_delete` receiver config field (crosses the wire). rsync's `--force` lets an incoming non-directory replace a destination directory; FastSync implements exactly that: when a regular file (or symlink) is written to a path that is currently a (possibly non-empty) destination directory, `--force` removes that directory tree first — confined to the receive root and symlink-safe (O_NOFOLLOW fd walk, symlinks removed by name, never followed) — so the install can place the file. **Protocol 2.23.0 honors `--force` on the `--delay-updates` publication path too**, not only the immediate-install path. Without `--force` such a write fails and the run aborts. Gated by the server `--allow-delete` policy (a client cannot use `--force` to remove a destination tree on a server that forbids deletion) |
| `-m`, `--prune-empty-dirs` | Prune empty dir chains | ✅ Parity | `-m`/`--prune-empty-dirs` (Phase 7 Wave A freed the rsync short `-m`; FastSync multithreading is now `-j`/`--threads`). FastSync's recursive transfer records directory times but never CREATES an empty directory (a `STATUS_DIR_TIMES` entry is record-only, and `--dirs` empty entries are pruned by this flag), so empty directories are inherently never transferred (which is rsync's `-m` behavior) and truly-empty destination directory chains are removed by `--delete` regardless of this flag. The flag's additional real effect is on the `--dirs` explicit directory-entry generator: a plain `-d <empty-dir>` run omits the empty source directory's entry, so nothing is created at the destination (no `STATUS_MKDIR`, no `-i`/`--out-format` change line, and an existing empty mirror becomes an extra that `--delete` prunes). Explicitly `--files-from`-listed directories always pass through (documented `--files-from` behavior). A directory that still holds an excluded-but-protected file survives, matching the `--delete-excluded` default |
@@ -659,8 +673,8 @@ targets verbatim, matching rsync.
| `--address=ADDRESS` | Bind address for outgoing socket | ✅ Parity | Binds the outgoing client socket to a local source address before `connect()` (resolved with the same `-4`/`-6` family hints as the destination). Local socket concern: never crosses the wire |
| `-4`, `--ipv4` | Prefer IPv4 | ✅ Parity | Forces `AF_INET` in the `getaddrinfo` hints for client destination/source resolution and the server bind (see the Phase 5, Wave B note). Mutually exclusive with `-6` |
| `-6`, `--ipv6` | Prefer IPv6 | ✅ Parity | Forces `AF_INET6` in the `getaddrinfo` hints for client destination/source resolution and the server bind. Mutually exclusive with `-4` |
| `--remote-option=OPT`, `-M` | Send an option only to the remote side | ⚠️ Caveat | Each value is appended to the remote server invocation over SSH as an individually single-quote-escaped shell word in `ssh_build_remote_command()`. Values are validated (non-empty, no control characters) and shell metacharacters cannot break out of the quoting (`;`, `&`, `\|`, <code>`</code>, `$`, `(`, `)`, quotes are neutralized), so a value cannot inject an arbitrary remote command and a subsequent `--` on the client line cannot be turned into one. The short `-M` form (`-M OPT`, `-M=OPT`, and rsync-style attached `-MOPT`) is available, matching rsync; metadata mode moved to long-only `--preserve`. **Divergence:** `-M` is only meaningful for the SSH transport (`user@host:path`); a daemon (`host::module/path`) or local TCP destination **rejects** it (there is no remote command line to append to), whereas rsync applies it to its own remote process on every transport. The options never cross the binary config frame |
| `--bwlimit=RATE` | Limit I/O bandwidth | ⚠️ Caveat | Token-bucket throttling of the transfer I/O (Kibibytes/second). **Divergence:** FastSync accepts only a positive integer; rsync additionally accepts `0` (no limit) and decimal/suffixed rates (`1.5`, `1.5m`, `100K`), so those rsync spellings are rejected. The limit is a local I/O concern and is not negotiated on the wire |
| `--remote-option=OPT`, `-M` | Send an option only to the remote side | ❌ Divergent | Each value is appended to the remote server invocation over SSH as an individually single-quote-escaped shell word in `ssh_build_remote_command()`. Values are validated (non-empty, no control characters) and shell metacharacters cannot break out of the quoting (`;`, `&`, `\|`, <code>`</code>, `$`, `(`, `)`, quotes are neutralized), so a value cannot inject an arbitrary remote command and a subsequent `--` on the client line cannot be turned into one. The short `-M` form (`-M OPT`, `-M=OPT`, and rsync-style attached `-MOPT`) is available, matching rsync; metadata mode moved to long-only `--preserve`. **Reclassified because the daemon/TCP case cannot be reproduced:** `-M` is only meaningful for the SSH transport (`user@host:path`); a daemon (`host::module/path`) or local TCP destination **rejects** it, whereas rsync forwards it to its own remote process on every transport. A differential test starts a real rsync daemon and shows `-M--totally-bogus` reaching the remote parser (`unknown option`) while a valid `-M--safe-links` is accepted. FastSync's daemon handshake is a fixed binary config frame with no per-connection argv channel; adding one would let a client set arbitrary server-side options (the same class of divergence as the native daemon config/auth), so the safe subset stays SSH-only |
| `--bwlimit=RATE` | Limit I/O bandwidth | ✅ Parity | A faithful port of rsync 3.4.1's `parse_size_arg(bwlimit_arg, 'K', "bwlimit", 512, -1, True)`: a bare value is KiB/s, `K`/`M`/`G`/`T`/`P` are binary suffixes, `KB`/`MB` are decimal, `KiB`/`MiB` are binary, decimals are accepted and quantized to whole KiB exactly like rsync's `(size + 512) / 1024`, `0` (or an empty value) means "no limit", and any other value below the 512-byte floor is rejected. The token bucket's burst capacity is ~100 ms of bandwidth, matching the point at which rsync's leaky bucket starts sleeping, so a throttled transfer paces like rsync (4 MiB at `--bwlimit=1024`/`2048` matches rsync within ~4%). Differential-tested: the accept/reject matrix and the wall-clock rate both match rsync 3.4.1. The limit is a local I/O concern and is not negotiated on the wire |
## 14. Daemon Mode
@@ -733,7 +747,7 @@ modes or links.
| `--stop-after=MINS` | Stop after N minutes | ✅ Parity | Client-only sender stop deadline (Phase 6): computing `--stop-after=MINS` (a positive minute count; 0/negative/garbage rejected) and `--stop-at=TIME` (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`; a past time stops immediately). The transfer stops ELEGANTLY at the next chunk boundary: everything already fully sent is kept and applied, the run returns 0, and --delete (late/delete-after timing) does NOT wipe the destination — when the scan is cut short the partial keep-set manifest is suppressed with a warning (the delete walk is skipped rather than acting on an incomplete keep-set, so unscanned source mirrors survive). `--delete-before`/`--delete-during` still run their complete pre-scan (which ignores the deadline). Local client-only fields: never serialized into the wire config frame, so no PROTOCOL_VERSION bump. `--stop-after` uses CLOCK_MONOTONIC; `--stop-at` uses the wall clock. Works single-threaded and under `-j`/`--threads` (multithreaded). Divergence: rsync computes `--stop-after` from the run start; FastSync likewise. When both are given, the earlier of the two deadlines wins (checked per iteration). See the Phase-6 stop notes below |
| `--stop-at=TIME` | Stop at specified time | ✅ Parity | Deadline transfer stop (client-only, never serialized). Protocol 2.26.0 accepts rsync's full date/time grammar (`2030-12-31T23:59`, `2030/12/31T23:59`, `2030-12-31`, `12-31`, `14:00`, `:59`, `1`) in addition to FastSync's `HH:MM[:SS]` and `now+N[smhd]`; a past time stops immediately. Everything already transferred is kept and an early stop suppresses the late `--delete` keep-set so unscanned source mirrors survive. Works single-threaded and under `-j`/`--threads` |
| `--fsync` | Fsync every written file before publication | ✅ Parity | |
| `--protocol=NUM` | Force older protocol version | ❌ Divergent | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.26.0) with no downgrade/backward-compat code paths, so `--protocol=2.26.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.25.0`/`2.24.0`/`2.23.0`/`2.22.0`/`2.21.0`/`2.20.0`/`2.19.0`/`2.18.0`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
| `--protocol=NUM` | Force older protocol version | ❌ Divergent | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.27.0) with no downgrade/backward-compat code paths, so `--protocol=2.27.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.26.0`/`2.25.0`/`2.24.0`/`2.23.0`/`2.22.0`/`2.21.0`/`2.20.0`/`2.19.0`/`2.18.0`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
| `--iconv=CONVERT_SPEC` | Charset conversion | ⚠️ Caveat | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, and the receiver converts each wire filename REMOTE→LOCAL before creating/writing. The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front; protocol 2.26.0 additionally accepts `--iconv=.` (the locale's default charset for both directions), `--iconv=-` and `--no-iconv` (disable conversion). Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below |
| `--checksum-seed=NUM` | Set checksum seed | ✅ Parity | Sets the seed for FastSync's whole-file xxHash digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). **As of protocol 2.23.0 a seed of `0` — the default when the flag is unset — is randomized per transfer and the chosen seed is sent to the receiver**, exactly like rsync, so two runs against different content do not share a predictable seed; an explicit non-zero seed is used verbatim, so an explicit seed deterministically reproduces every computed digest on BOTH endpoints (the seed crosses in the config frame). `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta` |
| `--secluded-args`, `-s` | Use protocol to send args | ❌ Divergent | Accepted for CLI compatibility (including the rsync short `-s`, Phase 7 Wave A) but a documented **no-op / divergence**. rsync's `-s` protects arguments from shell expansion by shipping them over the protocol; FastSync never passes remote arguments through a shell expansion boundary in the first place — its SSH transport builds the remote argv as **single-quote-escaped shell words** (`ssh_build_remote_command`), so the injection/leak that `-s` guards against does not exist and there is nothing to "seclude". Implementing a true arg-send protocol would mean replacing the argv-based SSH launch with an in-band argument channel, a large redesign of the transport that buys no security here. Chunk serialization remains the long-only `--chunk-serialization`. |
+1 -2
View File
@@ -70,8 +70,7 @@ static bool strbuf_append(StrBuf* buf, const char* text) {
}
bool change_list_enabled(const Config* config) {
return config != NULL &&
(config->itemize_changes || config->out_format != NULL ||
return config != NULL && (config->itemize_changes || config->out_format != NULL ||
(config->log_file != NULL && config->log_file_format != NULL) ||
(config->info_level & LOG_INFO_NAME) != 0);
}
+8 -8
View File
@@ -501,7 +501,9 @@ static bool is_accepted_debug_category(const char* name) {
static bool is_accepted_info_category(const char* name) {
static const char* const categories[] = {
"mount", "syms", "symsafe",
"mount",
"syms",
"symsafe",
};
for (size_t i = 0; i < sizeof(categories) / sizeof(categories[0]); i++) {
if (strcmp(name, categories[i]) == 0)
@@ -1846,9 +1848,9 @@ static int parse_bwlimit_value(const char* value, unsigned long long* bytes_per_
long long mult;
while (*arg >= '0' && *arg <= '9')
arg++;
if (*arg != '\0' &&
(*arg == '.' || *arg == localeconv()->decimal_point[0]))
for (arg++; *arg >= '0' && *arg <= '9'; arg++) {}
if (*arg != '\0' && (*arg == '.' || *arg == localeconv()->decimal_point[0]))
for (arg++; *arg >= '0' && *arg <= '9'; arg++) {
}
char suffix = *arg && *arg != '+' && *arg != '-' ? *arg++ : 'K';
switch (suffix) {
@@ -1932,8 +1934,7 @@ static int set_bwlimit_option(const char* value) {
if (parse_bwlimit_value(value, &bytes_per_sec) != 0)
return -1;
io_set_bwlimit(bytes_per_sec);
log_info_message(LOG_INFO_MISC, "Set bandwidth limit to %llu KB/s",
bytes_per_sec / 1024);
log_info_message(LOG_INFO_MISC, "Set bandwidth limit to %llu KB/s", bytes_per_sec / 1024);
return 0;
}
@@ -2651,8 +2652,7 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
/* --info=del on a real --delete run asks the receiver to report the paths it
actually removed; the report rides the STATUS_STATS path list, so the wire
stats frame must be negotiated too. */
config->report_deletes =
config->use_delete && !config->dry_run &&
config->report_deletes = config->use_delete && !config->dry_run &&
((config->info_level & LOG_INFO_DEL) != 0 || config->itemize_changes ||
config->out_format != NULL);
config->report_stats = config->stats || config->show_progress ||
+4 -3
View File
@@ -347,8 +347,8 @@ static void print_delete_reports(const Config* config, const ArrayList* paths) {
}
static void client_progress_begin(const Config* config) {
g_progress_active = (config->show_progress || info_flag_enabled(config, LOG_INFO_PROGRESS)) &&
!config->quiet;
g_progress_active =
(config->show_progress || info_flag_enabled(config, LOG_INFO_PROGRESS)) && !config->quiet;
g_progress_xferred = 0;
g_progress_seen = 0;
if (!g_progress_active) {
@@ -3307,7 +3307,8 @@ int send_files(Config* config) {
/* rsync default: a scan I/O error suppresses deletion unless
--ignore-errors, even in the late (commit) modes. Drop the keep-set so
the receiver removes nothing; the readable tree still transferred. */
bool late_delete = (manifest || config->delete_missing_args) && !delete_early && !delete_per_dir;
bool late_delete =
(manifest || config->delete_missing_args) && !delete_early && !delete_per_dir;
if (late_delete && !ignore_errors_allows_delete(config, had_scan_io)) {
log_message(LOG_LEVEL_WARNING, "IO error encountered -- skipping file deletion");
if (manifest) {
+2 -4
View File
@@ -417,8 +417,7 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
--max-delete-capped commit still succeeds and the transfer proceeds;
the terminal success frame reports the cap. */
size_t deleted = 0;
DeletePathObserver observer =
sink->deleted_paths ? receiver_record_deleted_path : NULL;
DeletePathObserver observer = sink->deleted_paths ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion =
(config->use_delete || config->delete_missing_args)
? manifest_delete_all_observed(config, manifest, &deleted, observer,
@@ -506,8 +505,7 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
deferred_manifest = NULL;
} else {
size_t deleted = 0;
DeletePathObserver observer =
sink->deleted_paths ? receiver_record_deleted_path : NULL;
DeletePathObserver observer = sink->deleted_paths ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion = manifest_delete_all_observed(
config, deferred_manifest, &deleted, observer, (void*)sink->deleted_paths);
receiver_tally_deleted(sink, deleted);
+5 -8
View File
@@ -956,11 +956,9 @@ void handler(int file_descriptor) {
server-contacting --dry-run deletes nothing (no manifest is sent). */
if (context->deferred_manifest) {
size_t deleted = 0;
DeletePathObserver observer =
config->report_deletes ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion =
manifest_delete_all_observed(config, context->deferred_manifest, &deleted, observer,
(void*)context->deleted_paths);
DeletePathObserver observer = config->report_deletes ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion = manifest_delete_all_observed(
config, context->deferred_manifest, &deleted, observer, (void*)context->deleted_paths);
context->stats.deleted_files += deleted;
if (deletion == DELETE_COMMIT_ERROR) {
transfer_ok = false;
@@ -979,9 +977,8 @@ void handler(int file_descriptor) {
/* Defence in depth (the enclosing block already excludes dry-run): a
-n run never commits a deletion. */
if (config->report_deletes)
delete_plan_session_set_delete_observer(context->deferred_plans,
receiver_record_deleted_path,
(void*)context->deleted_paths);
delete_plan_session_set_delete_observer(
context->deferred_plans, receiver_record_deleted_path, (void*)context->deleted_paths);
DeleteCommitResult deletion =
config->dry_run ? DELETE_COMMIT_OK
: delete_plan_session_commit(context->deferred_plans, config);
+2 -2
View File
@@ -268,8 +268,8 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
* real (non-dry-run) deletion. It is set only when --info=del is requested with
* --delete; the transfer decision itself is unchanged. */
#define CONFIG_WIRE_OUTPUT_FIELDS(X) \
X(report_dest_info, bool, false, BOOL) X(report_stats, bool, false, BOOL) \
X(report_deletes, bool, false, BOOL)
X(report_dest_info, bool, false, BOOL) \
X(report_stats, bool, false, BOOL) X(report_deletes, bool, false, BOOL)
/* Codec-negotiation wave (protocol 2.26.0). compression_algo is the concrete
* codec the client selected for this transfer (a CompressionAlgo id) and is the
+4 -4
View File
@@ -791,8 +791,8 @@ static bool apply_missing(DeletePlanSession* session, const Config* config) {
size_t deleted = 0;
size_t skipped = 0;
bool limit = false;
bool ok = manifest_delete_missing_args_limited_observed(
config, &manifest, remaining, &deleted, &skipped, &limit, session->observer,
bool ok = manifest_delete_missing_args_limited_observed(config, &manifest, remaining, &deleted,
&skipped, &limit, session->observer,
session->observer_context);
session->deleted += deleted;
session->planned += deleted;
@@ -893,8 +893,8 @@ static bool apply_deferred_path(DeletePlanSession* session, const Config* config
return ok;
}
void delete_plan_session_set_delete_observer(DeletePlanSession* session, DeletePathObserver observer,
void* context) {
void delete_plan_session_set_delete_observer(DeletePlanSession* session,
DeletePathObserver observer, void* context) {
if (!session)
return;
session->observer = observer;
+2 -2
View File
@@ -84,7 +84,7 @@ size_t delete_plan_session_deleted(const DeletePlanSession* session);
truly removes (including the deferred --delete-delay commit), so the receiver
can report rsync's `deleting PATH` lines through the terminal STATUS_STATS
record. Pass NULL/0 to clear. */
void delete_plan_session_set_delete_observer(DeletePlanSession* session, DeletePathObserver observer,
void* context);
void delete_plan_session_set_delete_observer(DeletePlanSession* session,
DeletePathObserver observer, void* context);
#endif
+5 -6
View File
@@ -3261,10 +3261,9 @@ static bool delete_extras_budgeted_observed(const Config* config, DeleteManifest
remaining = budget->max_delete - budget->deleted;
size_t deleted = 0;
size_t skipped = 0;
DeleteWalkResult result =
delete_extras_limited_observed(config->receive_root_directory, manifest->keeps, manifest->dirs,
remaining, skips, used, &deleted, &skipped, observer,
observer_context);
DeleteWalkResult result = delete_extras_limited_observed(
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips, used,
&deleted, &skipped, observer, observer_context);
if (owned_prefixes) {
for (int i = 0; i < config->basis_count; i++)
free(owned_prefixes[i]);
@@ -3592,8 +3591,8 @@ bool manifest_delete_missing_args_limited_observed(const Config* config, DeleteM
void* observer_context) {
DeleteBudgetState budget = {
.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
bool ok = delete_missing_args_budgeted_observed(config, manifest, &budget, observer,
observer_context);
bool ok =
delete_missing_args_budgeted_observed(config, manifest, &budget, observer, observer_context);
if (deleted)
*deleted = budget.deleted;
if (skipped)
+2 -1
View File
@@ -132,7 +132,8 @@ bool manifest_delete_missing_args_limited(const Config* config, DeleteManifest*
bool manifest_delete_missing_args_limited_observed(const Config* config, DeleteManifest* manifest,
size_t max_delete, size_t* deleted,
size_t* skipped, bool* limit_hit,
DeletePathObserver observer, void* observer_context);
DeletePathObserver observer,
void* observer_context);
/* Outcome of committing a delete manifest. LIMIT_REACHED reports rsync's
partial --max-delete result: the budget allowed some deletions and the rest
were skipped (the run still stores all file data but the client exits 25). */
+2 -1
View File
@@ -876,7 +876,8 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer, void* observer_context) {
DeletePathObserver observer,
void* observer_context) {
if (deleted_out)
*deleted_out = 0;
if (skipped_out)
+2 -1
View File
@@ -145,7 +145,8 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer, void* observer_context);
DeletePathObserver observer,
void* observer_context);
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
+455
View File
@@ -0,0 +1,455 @@
"""Differential parity tests for the option wave (bwlimit, --info=*, -M,
--ignore-errors, --filter protect).
Every differential here runs the SAME scenario with real ``rsync 3.4.1`` and
with fastsync and compares the observable result, so the modules are skipped
when rsync is unavailable. The privilege-dependent --ignore-errors differential
drops the client to an unprivileged uid so a mode-000 source directory is
genuinely unreadable; it is marked ``setpriv`` (run as root locally, excluded
from the root PR gate exactly like the other privilege tests).
"""
import os
import shutil
import subprocess
import sys
import time
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402
CLIENT_CMD,
TEST_DATA_DIR,
ServerManager,
clean_dir,
get_dest_received_dir,
run_client,
)
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
def _rsync(args, timeout=120, as_nobody=False):
env = dict(os.environ, LC_ALL="C")
cmd = [RSYNC] + args
if as_nobody:
cmd = ["setpriv", "--reuid=65534", "--regid=65534", "--clear-groups"] + cmd
return subprocess.run(cmd, capture_output=True, text=True, env=env, timeout=timeout)
def _write(path, content):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(content)
class TestBwlimitParity:
"""--bwlimit must accept rsync 3.4.1's spellings and pace like it."""
ACCEPTED = ["100", "0", "1.5", "100K", "100KB", "100KiB", "1M", "1MB", "1m", "1G", "512"]
REJECTED = ["-1", "abc", "1x", "1 000"]
@requires_rsync
@pytest.mark.ci
def test_parse_acceptance_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "bwp_src")
clean_dir(source)
_write(os.path.join(source, "f.txt"), b"payload\n")
for value in self.ACCEPTED + self.REJECTED:
rdst = os.path.join(TEST_DATA_DIR, "bwp_rdst")
clean_dir(rdst)
rsync_result = _rsync(["-a", "--bwlimit=" + value, source + "/", rdst + "/"])
dest = os.path.join(TEST_DATA_DIR, "bwp_dst")
clean_dir(dest)
result, _ = run_client(source, dest, flags=["-a", "--bwlimit=" + value],
port=shared_server.port)
assert (result.returncode == 0) == (rsync_result.returncode == 0), (
f"--bwlimit={value}: fastsync rc={result.returncode} "
f"({(result.stderr or result.stdout)[:120]!r}) "
f"rsync rc={rsync_result.returncode} ({rsync_result.stderr[:120]!r})"
)
@requires_rsync
@pytest.mark.ci
def test_throttle_rate_matches_rsync(self, shared_server):
"""A 4 MiB transfer at --bwlimit=2048 (2 MiB/s) must take about the same
wall-clock time for both tools (~2 s with rsync's leaky bucket)."""
source = os.path.join(TEST_DATA_DIR, "bwt_src")
clean_dir(source)
_write(os.path.join(source, "big.bin"), os.urandom(4 * 1024 * 1024))
dest = os.path.join(TEST_DATA_DIR, "bwt_dst")
rdst = os.path.join(TEST_DATA_DIR, "bwt_rdst")
clean_dir(rdst)
start = time.monotonic()
rsync_result = _rsync(["-a", "--bwlimit=2048", source + "/", rdst + "/"])
rsync_secs = time.monotonic() - start
assert rsync_result.returncode == 0, rsync_result.stderr
clean_dir(dest)
result, fast_secs = run_client(source, dest, flags=["-a", "--bwlimit=2048"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
assert fast_secs > 1.0, f"fastsync throttled too little: {fast_secs:.2f}s"
# Both rendezvous near 2 s; allow a generous band for CI scheduling.
assert abs(fast_secs - rsync_secs) < 1.0, (
f"fastsync {fast_secs:.2f}s vs rsync {rsync_secs:.2f}s"
)
def _output_tree(root):
clean_dir(root)
os.makedirs(os.path.join(root, "sub"))
_write(os.path.join(root, "a.txt"), b"top\n")
_write(os.path.join(root, "sub", "b.txt"), b"nested\n")
os.symlink("a.txt", os.path.join(root, "link"))
class TestInfoParity:
"""The --info categories that map to a FastSync event must print rsync's
line format."""
@requires_rsync
@pytest.mark.ci
def test_info_flist_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_fl_src")
dest = os.path.join(TEST_DATA_DIR, "inf_fl_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_fl_rdst")
_output_tree(source)
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-a", "--info=flist", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--info=flist"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
assert "sending incremental file list" in result.stdout
assert "sending incremental file list" in rsync_result.stdout
@requires_rsync
@pytest.mark.ci
def test_info_name_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_nm_src")
dest = os.path.join(TEST_DATA_DIR, "inf_nm_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_nm_rdst")
_output_tree(source)
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-a", "--info=name", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--info=name"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
def entries(text):
# Compare the transferred entries only: rsync also prints the
# transfer-root `./` and every directory (FastSync records dirs),
# which are a separate documented divergence.
out = []
for line in text.splitlines():
if not line or line.startswith("sending ") or line.startswith("created "):
continue
if line == "./" or line.endswith("/"):
continue
out.append(line)
return sorted(out)
assert entries(result.stdout) == entries(rsync_result.stdout), (
f"rsync={entries(rsync_result.stdout)} fastsync={entries(result.stdout)}"
)
@requires_rsync
@pytest.mark.ci
def test_info_nonreg_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_nr_src")
dest = os.path.join(TEST_DATA_DIR, "inf_nr_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_nr_rdst")
clean_dir(source)
os.mkfifo(os.path.join(source, "fifo"))
_write(os.path.join(source, "a.txt"), b"a\n")
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-rlt", "--info=nonreg", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-rlt", "--info=nonreg"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("skipping non-regular"))
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("skipping non-regular"))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
assert fast_lines, "no non-regular skip line emitted"
@requires_rsync
@pytest.mark.ci
def test_info_del_real_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_dl_src")
dest = os.path.join(TEST_DATA_DIR, "inf_dl_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_dl_rdst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
clean_dir(rdst)
_write(os.path.join(rdst, "extra.txt"), b"x\n")
_write(os.path.join(rdst, "extra2.txt"), b"y\n")
rsync_result = _rsync(["-a", "--delete", "--info=del", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("deleting "))
clean_dir(dest)
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.txt"), b"x\n")
_write(os.path.join(received, "extra2.txt"), b"y\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=["-a", "--delete", "--info=del"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("deleting "))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
assert fast_lines, "no deletion lines emitted"
@requires_rsync
@pytest.mark.ci
def test_info_del_itemize_real_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_di_src")
dest = os.path.join(TEST_DATA_DIR, "inf_di_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_di_rdst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
clean_dir(rdst)
_write(os.path.join(rdst, "extra.txt"), b"x\n")
rsync_result = _rsync(["-a", "-i", "--delete", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("*deleting"))
clean_dir(dest)
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.txt"), b"x\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=["-a", "-i", "--delete"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("*deleting"))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
@requires_rsync
@pytest.mark.ci
def test_info_del_dry_run_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_dd_src")
dest = os.path.join(TEST_DATA_DIR, "inf_dd_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_dd_rdst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
clean_dir(rdst)
_write(os.path.join(rdst, "extra.txt"), b"x\n")
rsync_result = _rsync(["-a", "-n", "--delete", "--info=del", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("deleting "))
clean_dir(dest)
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.txt"), b"x\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=["-a", "-n", "--delete", "--info=del"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("deleting "))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
@requires_rsync
@pytest.mark.ci
def test_info_remove_matches_rsync(self, shared_server):
tag = "inf_rm"
rsync_src = os.path.join(TEST_DATA_DIR, f"{tag}_rsrc")
rsync_dst = os.path.join(TEST_DATA_DIR, f"{tag}_rdst")
fast_src = os.path.join(TEST_DATA_DIR, f"{tag}_fsrc")
fast_dst = os.path.join(TEST_DATA_DIR, f"{tag}_fdst")
for root in (rsync_src, rsync_dst, fast_src, fast_dst):
clean_dir(root)
_write(os.path.join(rsync_src, "a.txt"), b"a\n")
_write(os.path.join(rsync_src, "sub", "b.txt"), b"b\n")
_write(os.path.join(fast_src, "a.txt"), b"a\n")
_write(os.path.join(fast_src, "sub", "b.txt"), b"b\n")
rsync_result = _rsync(["-a", "--remove-source-files", "--info=remove",
rsync_src + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("sender removed "))
result, _ = run_client(fast_src, fast_dst,
flags=["-a", "--remove-source-files", "--info=remove"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("sender removed "))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
assert fast_lines, "no source-removal lines emitted"
class TestIgnoreErrorsParity:
"""--ignore-errors: a source I/O error skips deletion by default; the flag
lets deletion proceed. Both exit 23. Run the client as an unprivileged user
so the mode-000 directory is genuinely unreadable."""
@pytest.mark.setpriv
def test_delete_after_io_error_matches_rsync(self):
if os.geteuid() != 0 or shutil.which("setpriv") is None:
pytest.skip("requires root + setpriv to drop privileges for the client")
tag = f"ie_{os.getpid()}"
source = os.path.join(TEST_DATA_DIR, f"{tag}_src")
rsync_dst = os.path.join(TEST_DATA_DIR, f"{tag}_rdst")
dest = os.path.join(TEST_DATA_DIR, f"{tag}_dst")
clean_dir(source)
clean_dir(rsync_dst)
clean_dir(dest)
_write(os.path.join(source, "top.txt"), b"top\n")
_write(os.path.join(source, "locked", "blocked.txt"), b"blocked\n")
os.chmod(os.path.join(source, "locked"), 0)
os.chmod(TEST_DATA_DIR, 0o777)
os.chmod(source, 0o755)
os.chmod(rsync_dst, 0o777)
os.chmod(dest, 0o777)
try:
for ignore in (False, True):
flags = ["-a", "--delete-after"] + (["--ignore-errors"] if ignore else [])
# rsync side
_write(os.path.join(rsync_dst, "extra.txt"), b"x\n")
os.chmod(os.path.join(rsync_dst, "extra.txt"), 0o666)
rres = _rsync(flags + [source + "/", rsync_dst + "/"], as_nobody=True)
rsync_extra = os.path.exists(os.path.join(rsync_dst, "extra.txt"))
# fastsync side
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.txt"), b"x\n")
os.chmod(os.path.join(received, "extra.txt"), 0o666)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
fflags = (["--delete", "--ignore-errors"] if ignore else ["--delete"])
cmd = CLIENT_CMD + ["--source-dir", source, "--dest-dir", dest,
"--save-to-disk", "--server-port", str(server.port)] + fflags
fres = subprocess.run(
["setpriv", "--reuid=65534", "--regid=65534", "--clear-groups"] + cmd,
text=True, capture_output=True)
fast_extra = os.path.exists(os.path.join(received, "extra.txt"))
assert rres.returncode == 23, (ignore, rres.returncode, rres.stderr[:200])
assert fres.returncode == 23, (ignore, fres.returncode, fres.stderr[:200])
assert rsync_extra == fast_extra, (
f"ignore_errors={ignore}: rsync extra={rsync_extra} fastsync extra={fast_extra}"
)
assert fast_extra is (not ignore), (ignore, fast_extra)
finally:
os.chmod(os.path.join(source, "locked"), 0o755)
class TestRemoteOptionDaemon:
"""rsync forwards -M/--remote-option to its remote process over a daemon
connection; FastSync's daemon has no per-connection argv channel and rejects
it. This pins the documented divergence with evidence."""
@requires_rsync
def test_rsync_forwards_M_over_daemon_and_fastsync_rejects(self, tmp_path):
import socket
with socket.socket() as probe:
probe.bind(("127.0.0.1", 0))
port = probe.getsockname()[1]
module_root = tmp_path / "mod"
module_root.mkdir()
os.chmod(module_root, 0o777)
source = tmp_path / "src"
source.mkdir()
(source / "a.txt").write_bytes(b"hello\n")
conf = tmp_path / "rsyncd.conf"
conf.write_text(
f"port = {port}\nuse chroot = no\n[m]\npath = {module_root}\nread only = no\n"
)
daemon = subprocess.Popen(
[RSYNC, "--daemon", "--no-detach", "--port", str(port), "--config", str(conf)],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
deadline = time.monotonic() + 5
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.3):
break
except OSError:
time.sleep(0.05)
else:
pytest.skip("rsync daemon did not start")
# A well-formed -M option is forwarded and accepted by the daemon...
ok = _rsync(["-a", "-M--safe-links", source.as_posix() + "/",
f"rsync://127.0.0.1:{port}/m/"])
# ...and a bogus one is rejected ON THE REMOTE with "unknown option",
# which proves the option reached the daemon's parser.
bogus = _rsync(["-a", "-M--totally-bogus", source.as_posix() + "/",
f"rsync://127.0.0.1:{port}/m/"])
assert bogus.returncode != 0
assert "unknown option" in (bogus.stderr + bogus.stdout), bogus.stderr
del ok
finally:
daemon.terminate()
try:
daemon.wait(timeout=5)
except subprocess.TimeoutExpired:
daemon.kill()
# FastSync rejects -M for a non-SSH transport up front.
dest = os.path.join(TEST_DATA_DIR, "ro_dst")
clean_dir(dest)
result, _ = run_client(source.as_posix(), dest, flags=["-a", "-M--safe-links"])
assert result.returncode != 0
assert "remote-option" in (result.stderr + result.stdout)
class TestFilterProtectDivergence:
"""Documented residual: a protect rule that matches only a destination-only
entry is not re-derived on the receiver (FastSync derives delete protection
from the source scan), so rsync protects the extra but FastSync removes it."""
@requires_rsync
@pytest.mark.ci
def test_protect_dest_only_divergence(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "fpd_src")
dest = os.path.join(TEST_DATA_DIR, "fpd_dst")
rdst = os.path.join(TEST_DATA_DIR, "fpd_rdst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
clean_dir(rdst)
_write(os.path.join(rdst, "extra.log"), b"extra\n")
_write(os.path.join(rdst, "other.txt"), b"other\n")
rsync_result = _rsync(["-a", "--delete", "--filter=P *.log", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
assert os.path.exists(os.path.join(rdst, "extra.log")), "rsync did not protect extra.log"
clean_dir(dest)
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.log"), b"extra\n")
_write(os.path.join(received, "other.txt"), b"other\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest,
flags=["-a", "--delete", "--filter=P *.log"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
# Pin the known divergence: FastSync deletes the destination-only file.
assert not os.path.exists(os.path.join(received, "extra.log")), (
"FastSync now protects destination-only P matches; the --filter row may be "
"upgradable to full parity"
)
assert not os.path.exists(os.path.join(received, "other.txt"))
+2 -2
View File
@@ -1361,8 +1361,8 @@ static void test_parse_args_rsync_flag_vocabulary_accepted() {
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_BACKUP | LOG_INFO_DEL | LOG_INFO_FLIST |
LOG_INFO_NONREG | LOG_INFO_PROGRESS | LOG_INFO_REMOVE);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_BACKUP | LOG_INFO_DEL | LOG_INFO_FLIST | LOG_INFO_NONREG |
LOG_INFO_PROGRESS | LOG_INFO_REMOVE);
EXPECT_EQ_INT(cfg->debug_level, 0);
config_delete(cfg);
}
+2
View File
@@ -244,6 +244,7 @@ static void test_write_thread_done() {
* and queue_destroy which would double-free since we created them
* in this test. Let me just free the context directly. */
array_list_delete(ctx->would_delete);
array_list_delete(ctx->deleted_paths);
mtx_destroy(&ctx->mutex);
cnd_destroy(&ctx->condition_not_full);
cnd_destroy(&ctx->condition_not_empty);
@@ -329,6 +330,7 @@ static void test_receiver_enqueue_byte_budget() {
/* Tear down: the second file is still queued and is freed by queue_destroy. */
array_list_delete(ctx->would_delete);
array_list_delete(ctx->deleted_paths);
mtx_destroy(&ctx->mutex);
cnd_destroy(&ctx->condition_not_full);
cnd_destroy(&ctx->condition_not_empty);