Add test_out_format_c_delta_mode_divergence documenting that FastSync's
delta %c (its own handshake bytes) cannot match rsync's (16-byte sum header
plus per-block checksums); only the whole-file case is aligned. The test
pins both values so a future parity improvement is noticed.
- Output parity: add `File list size` to the strict --stats differential and
document the row-#3 residual with test_stats_file_count_breakdown_residual
(rsync's `Number of files`/`Number of created files` type breakdown is not
reproducible from what the sender knows: no directory accounting and no
per-entry destination-created state). The row stays a caveat.
- Add --threads variants for the --stats and --progress/-P differentials.
The `-n --delete --threads` variant is a documented xfail: the threaded
dry-run path does not consume the receiver's STATUS_STATS delete list yet.
- Replace the 0.2s sleep flake in the delete-timing proxy with a socket
barrier: the hook now fires only after the server sends a reply (proving it
processed the preceding per-directory delete plan), using --incremental +
--ignore-times to guarantee a mid-transfer handshake reply.
Blockers addressed together (shared scanner/delete-plan plumbing):
* #10: an empty in-scope source directory produced no plan keep entry, so the
receiver deleted the destination directory itself. The scanner now records
every traversed directory into a delete-plan sink, the plan sender keeps them,
and any directory whose plan the data stream never triggered is emitted after
the data so its extras are still removed. Differential tests cover
--delete-during and --delete-delay.
* #8: an invalid per-directory filter file was silently ignored when an earlier
merge file in the same directory existed; key the failure off the error text
(both sequential and parallel scanners) and fail the scan.
* #9: -R + --files-from receiver-protect rules recorded the source-relative
path; record the bare relative wire path in both scanners so the protected
destination mirror survives --delete.
* #5: the STATUS_STATS would-delete parser now validates each retained path and
enforces the shared MAX_MANIFEST_BYTES budget, and the --out-format dry-run
delete line is escaped like the itemize line.
* #11: drop the unused DELETE_PLAN_MAX_NAMES macro, log the delete-limit
warning once per session, roll back dir-merge names from a per-directory file
that fails to parse, and guard every filter error snprintf against err==NULL.
#10 leaves the empty directory itself kept and its extras removed, matching
rsync's final state on both per-directory timings.
rsync's %c counts the block-checksum bytes received: even a whole-file
transfer with no basis receives rsync's 16-byte sum header (append and
inplace included), while a dry run receives nothing. FastSync's
whole-file path has no equivalent header, so report 16 for parity when
delta is inactive, keep 0 for dry runs, and keep the real received bytes
when delta is active (FastSync's signature framing differs, so delta %c
stays divergent). Add a strict %c/%l/%n differential against rsync and
turn the %b check into a real rsync differential (semantics: both count
wire bytes and exceed %l; the exact values are protocol-specific).
Accept the full rsync 3.4.1 --info (backup, del, flist, mount, nonreg,
progress, remove, symsafe) and --debug (acl, backup, bind, chdir, connect,
cmd, del, deltasum, dup, exit, filter, flist, fuzzy, genr, hash, hlink,
iconv, nstr, own, recv, send, time) vocabularies, plus the historical
syms/hl/owner aliases, with level suffixes. Categories FastSync already
emits (copy/name/misc/skip/stats and io/proto/pack/util) still set their
log flags; the rest are accepted but silent. Unknown names remain
rejected by name, matching rsync. Update the CLI unit tests and the
rsync-parity integration tests (previously they required del/filter to be
rejected).
- Assert FastSync's sparse/preallocate block accounting matches rsync 3.4.1
for a hole file (preallocate wins over sparse, exactly like rsync).
- Assert --ignore-existing is answered by the receiver before the sender
transmits the payload (CountingProxy wire volume near-zero).
- CountingProxy.run gains a bounded join_timeout so tests that only need the
client->server count do not wait for the server's idle socket.
- Fix the stale protocol 2.24.0 comment in test_config.c (golden is 2.26.0).
The single-threaded and -m receivers never populated ReceiverStats.matched_data
or .deleted_files, so --stats always printed 0 for both even when rsync
reported nonzero. Track the bytes reconstructed from the basis file while
applying a delta, and tally the delete-commit counts (manifest and
per-directory sessions) into the receiver stats. The -m pipeline now carries
its own stats/would-delete fields and emits the STATUS_STATS frame before the
terminal success, so --threads finally reports the counters and renders
-n --delete lines.
Also normalize the -n --delete would-delete enumeration's absolute basis
prefixes exactly like the real commit path (fixing an over-report) and fix the
basis_delete_relative off-by-one when the receive root is '/'. Unit tests
cover the root mapping and the basis protection; integration tests cover
matched/deleted stats for both receivers and the --threads dry-run delete
lines.
delete_plan_session_commit() lacked the central no-mutation guard that
manifest_delete_all() has, so a server-contacting -n run (or a hostile plan
frame) could still remove --delete-missing-args mirrors on the per-directory
timing path. Return DELETE_COMMIT_OK immediately when the session is a
dry-run, and gate the receiver/server commit call sites too. Add a unit test
that streams a plan naming an existing destination file and asserts it
survives.
The -R prefix marker installed in synced_dirs was discarded when finalizing
the per-directory delete sender (--delete-during/--delete-delay), so the
up-front root plan was the receive root '.', whose keep list only held the
first prefix component. The receiver then deleted destination content
outside the transferred prefix (e.g. unrelated/keep.txt), a data-loss bug;
rsync keeps it.
Confine the walk to the -R prefix: send that prefix's plan as the root plan,
only transmit plans at or below it, and never emit the receive root plan for
a scoped run. Add a differential test covering both --delete-during and
--delete-delay.
Add tests/integration/test_codecs.py (accept/reject matrix and byte
differential against rsync 3.4.1 for every algorithm), extend the
checksum/compression unit tests with MD4/SHA1/none vectors and per-codec
round-trips, pin the new config golden (2.26.0), and update the version
strings and codec acceptance expectations.
Accept the full rsync 3.4.1 --compress-choice set (zstd/lz4/zlib/zlibx/
none/auto) and the two-name --checksum-choice TRANSFER,PRE-TRANSFER form,
including rsync's 'none' rules (rejected with --checksum at exit 4, and
forcing --whole-file as the transfer half) and unknown names at exit 4.
The checksum default becomes the auto-negotiated xxh128.
Negotiation is deterministic and symmetric: both peers run the same
preference resolver (rsync's --version order). The resolved
compression_algo crosses the wire as a new trailing config-frame int so
the receiver validates and installs the exact codec; an unsupported
choice is refused before STATUS_OK like rsync's failed negotiation.
Bump PROTOCOL_VERSION to 2.26.0.
Add real implementations for the rsync 3.4.1 checksum and compression
breadth: a self-contained MD4 (RFC 1320), OpenSSL-backed SHA1, a
no-digest mode, and LZ4/zlib codecs alongside zstd. Compressed buffers
are now self-describing (a leading codec id), so every existing
decompression call site keeps working through a process-global codec
selection. zlibx shares the zlib codec because FastSync compresses only
delta/token bytes (never matched file data), matching the 'x' intent.
- Only honor the --delete-missing-args exact paths when the server's
--allow-delete policy left delete_missing_args set.
- -d/--dirs does not recurse, so a per-directory plan would carry no child
information and could delete the contents of an untraversed directory; fall
back to the whole-tree end-of-transfer commit for that mode.
A -R source prune (--exclude/--max-size) must record the destination wire
path below the reconstructed prefix so --delete protects it; the parallel
root scan and the sequential skip path used the source path instead.
A general -R transfer places its files below the reconstructed prefix, so
marking the whole receive root as the delete scope deleted unrelated
sibling directories (data loss; rsync keeps them). Use the prefix itself
as the root marker when it is non-empty, in both the single-threaded and
multithreaded pipelines.
rsync prints the --stats block (with the (DRY RUN) suffix) for -n; route
the dry-run path through report_transfer_stats using the wire counters and
the STATUS_STATS receiver report.
The receiver emits the wire-stats frame before the per-file acks and the
terminal status; the client must consume it in that order or a combined
--stats --remove-source-files run desynchronizes.
Replace the aggregate stderr progress with rsync 3.4.1's per-file progress
block (name, 32 KiB first frame, final frame with (xfr#N, to-chk=X/Y)).
Add differential tests against real rsync for --out-format %C/%b, the
--progress frames, selected --stats lines and -n --delete lines.
Add the STATUS_STATS end-of-transfer receiver report (matched/deleted
counters plus a would-delete path list) behind the report_stats wire
bool, and a read-only delete_extras_list walker. --stats now renders
true wire byte totals and the receiver-reported deleted count; a
server-contacting -n --delete prints transfer-relative '*deleting' lines
matching rsync's itemize layout.
- Compare --delete-during/--delete-delay final state against rsync 3.4.1.
- Force a mid-transfer failure through a byte-slicing proxy: --delete-during has
removed the processed directory's extra, --delete-delay has not.
- Create a destination entry while the transfer is in flight: it survives
--delete-delay's snapshot but is removed by --delete-after's fresh end scan.
- Cover the --delete-delay type-conflict case now matching rsync.
Stream one delete plan per source directory from sender to receiver instead of
a single whole-tree keep-set manifest:
- --delete-during applies each directory's extras as its plan arrives, before
that directory's data (rsync's generator-order deletion).
- --delete-delay snapshots each directory's extras while the plan arrives and
commits the removals only after a fully-successful transfer, so files created
after the scan survive (matching rsync's delete-delay, not delete-after).
- Type conflicts (a destination file blocking a source directory, or vice
versa) are cleared immediately in both modes, so the nested write succeeds.
The plan carries the destination-relative directory, its kept child directory
names and its kept child file names; the first frame also carries the global
protected prefixes, size-skipped prefixes and --delete-missing-args paths.
--delete-before keeps the existing whole-tree early manifest; plain --delete and
--delete-after keep the end-of-transfer manifest commit.
Preserves the existing safety surface: protected/size-skipped prefixes and the
--delay-updates/basis skips are honored at any depth, deletion is scoped to the
synchronized directories (--files-from), MAX_SERVER_DELETE_COUNT and
--max-delete (partial + exit 25) are shared across plans, symlinks are never
followed, and paths are confined to the receive root.
A trailing slash (or trailing '/.', or a bare '.') now lists the source's
immediate contents -- files transferred, subdirectories created empty --
without recursing, while a bare directory still sends only its own entry.
The -R prefix applies to the generated entries and to the root entry.
Reconstruct the destination-relative prefix from the source spec outside
--files-from: cut at rsync's first '/./' (or a leading './'), normalize
later '.' components and trailing slashes. Apply it as each File's
send_path in the sequential and parallel scanners (root and worker paths,
files, one-file-system mount entries and directory-time capture).
Transmit the metadata of implied parent directories (prefix components
above the source root), suppressed by --no-implied-dirs, so parent attrs
match rsync in both the single-threaded and -m pipelines.
Bump PROTOCOL_VERSION to 2.25.0 and append a report_stats bool to the
config frame, add a STATUS_STATS status, and add process-wide wire byte
counters (protocol_bytes_written/read) for the client.
Render the rsync 3.4.1 --out-format %b (wire bytes sent) and %c (wire
bytes read back) tokens from per-file counter deltas, and %C (whole-file
xxh128 checksum, seed 0) via a new streaming checksum_digest_file().
TapTap
opencode