- --delete-delay: count/track only entries actually removed; a directory
refilled before commit (ENOTEMPTY) no longer inflates Number of deleted
files or the --max-delete budget (unit + integration + rsync differential).
- --stats: per-type Number of files breakdown; only stored regular files
count as transferred; transferred/literal byte totals and Total file size
(symlink target lengths) now match rsync for whole-file transfers.
- --progress: print the leading ./ root line and include the root entry in
the to-chk denominator (single-file output byte-identical to rsync).
- --out-format %C: use the selected transfer checksum and render every
algorithm exactly like rsync; checksum_digest_file gains md4/sha1/none.
Reclassify --out-format to Divergent (protocol-specific %b/delta-%c).
- Docs: RSYNC_COMPAT tally 107/25/24, HANDOFF update. No wire change.
Add tests/integration/test_differential_parity.py plus a shared
parity_harness.py and a data-driven parity_caveats.py allowlist. The gate
runs real rsync 3.4.1 and FastSync over the same corpora, compares the
destination trees (paths, hashes, symlink targets, modes, hard-link
grouping) and the normalized -i/--stats/--out-format output, and fails on
any difference not listed in the allowlist. Stale allowlist entries warn
(or fail under FASTSYNC_PARITY_STRICT=1) so the residual list shrinks.
Register parity/parity_ci markers and wire a fast PR job (parity_ci) plus a
push-only full job (parity, strict) into .gitea/workflows/ci.yaml.
Document the gate and the allowlist workflow in tests/integration/README.md.
The post-merge valgrind job on dev hangs. Root cause: the CI valgrind step
exports FASTSYNC_UNDER_VALGRIND=1, but nothing read it, and the
/proc/self/maps "vgpreload" probe is unreliable on valgrind 3.22 (the guest's
maps no longer list the tool's own libraries). So the fork-based unit tests
ran under valgrind anyway; tests that call io_set_fds() left the thread-local
read/write descriptors pointing at a closed test pipe, and a later
send_n_data()/receive_n_data() call was silently redirected to those stale fds
(legacy_session() prefers the globals, which the stdin/stdout SSH server
requires). Later tests only worked by fd-reuse luck; under valgrind the fd
numbers no longer coincide, so the read blocked forever on an empty pipe.
- test_utils.h: honor FASTSYNC_UNDER_VALGRIND (already set by ci.yaml) and keep
the maps scan as a best-effort fallback. Reset io_set_fds(-1, -1) at the
start of every RUN_TEST so one suite cannot leak descriptor redirection into
the next.
- test_iconv.c: skip the forking wire-string roundtrip under valgrind like the
other fork-based tests.
- config.c: initialize per_dir_filter_count in config_set_defaults. The field
was never initialized, so -F/-FF counting read uninitialized heap (valgrind:
conditional jump on uninitialised value at client_cli.c:1464) and could count
from garbage.
Verified with the CI-equivalent command (FASTSYNC_UNDER_VALGRIND=1 valgrind
--leak-check=full --show-leak-kinds=definite --error-exitcode=1): completes
with 0 errors (previously hung >80 min). Unit 43/43; full integration 729
passed; cppcheck and clang-format clean.
Address findings from the four-agent review of PR #298:
- file_create: zero the new File.matched_bytes. It was uninitialized
malloc memory, so the receiver could sum a garbage value into
STATUS_STATS "Matched data" (nondeterministic --stats divergence and
an uninitialized-heap disclosure on the wire).
- send_append: load the source into memory before hashing/copying the
prefix and tail. Files >64 MiB without compression (and --sendfile
runs) are streamed without loading, so --append/--append-verify
dereferenced a NULL data->data and crashed.
- filter_file_append: clamp the rollback to the surviving rule count.
A "clear" rule in a merge file frees every rule including the
caller's; the old rollback rewound count to rules_before and
resurrected freed pointers for a double free / UAF. Also roll back
when set_rule_owner fails instead of leaving owner-less rules.
- filter_rule_parse: reject the xattr-name filter modifier (x), which
was parsed and silently reinterpreted as a filename rule (affecting
what --delete protects). The p modifier stays accepted (existing
grammar test).
- Remove two dead functions: compression_default_algo and
change_render_itemize_code.
- tests: free ctx->would_delete in the two test_multiprocessing manual
teardowns (ASan leak, 1648 bytes/run).
- docs: correct the RSYNC_COMPAT/HANDOFF tally (156 rows: 106/27/23),
downgrade --info/--debug to caveat with their silent categories, add
%C-vs-xxh64 and --delete-delay count caveats, refresh stale xattr
mode comments, and document -p special-bit (setuid/setgid/sticky)
parity plus its mitigations.
Implicit parent directories were created with a hardcoded 0755, diverging from rsync's 0777 & ~umask whenever the process umask is not 022 (the CI runner uses 0). Matches rsync under any umask; verified with umask 0.
Reclassify the RSYNC_COMPAT matrix after the parity-completion wave (protocol
2.23.0 -> 2.26.0): 9 already-parity rows to parity, 17 inherently non-rsync
rows to divergent, and the genuine fixes to parity, recounting to
109 parity / 25 caveat / 23 divergent of 157 rows. Add rows for --bwlimit,
--partial, --partial-dir, --no-whole-file, --inc-recursive/--no-inc-recursive,
--protect-args and --msgs2stderr; fix the documented -f/filter, -F/.rsync-filter,
empty --files-from, and --preallocate/--sparse precedence bugs; add the Parity
Completion Wave section.
Refresh README/HANDOFF/release skill/cmake-expert to protocol 2.26.0 and the
zlib/lz4 + md4/sha1/none codecs, add the CHANGELOG 2.26.0 entry, and correct
the stale --max-delete --help wording.
Adding every traversed directory to the per-directory plan keep set must not
make an I/O-errored partial scan look non-empty. Count only transmitted file
entries for delete_plan_sender_empty(), so a scan that hit an unreadable
directory and found no files still refuses to delete.
Add test_out_format_c_delta_mode_divergence documenting that FastSync's
delta %c (its own handshake bytes) cannot match rsync's (16-byte sum header
plus per-block checksums); only the whole-file case is aligned. The test
pins both values so a future parity improvement is noticed.
- Output parity: add `File list size` to the strict --stats differential and
document the row-#3 residual with test_stats_file_count_breakdown_residual
(rsync's `Number of files`/`Number of created files` type breakdown is not
reproducible from what the sender knows: no directory accounting and no
per-entry destination-created state). The row stays a caveat.
- Add --threads variants for the --stats and --progress/-P differentials.
The `-n --delete --threads` variant is a documented xfail: the threaded
dry-run path does not consume the receiver's STATUS_STATS delete list yet.
- Replace the 0.2s sleep flake in the delete-timing proxy with a socket
barrier: the hook now fires only after the server sends a reply (proving it
processed the preceding per-directory delete plan), using --incremental +
--ignore-times to guarantee a mid-transfer handshake reply.
Blockers addressed together (shared scanner/delete-plan plumbing):
* #10: an empty in-scope source directory produced no plan keep entry, so the
receiver deleted the destination directory itself. The scanner now records
every traversed directory into a delete-plan sink, the plan sender keeps them,
and any directory whose plan the data stream never triggered is emitted after
the data so its extras are still removed. Differential tests cover
--delete-during and --delete-delay.
* #8: an invalid per-directory filter file was silently ignored when an earlier
merge file in the same directory existed; key the failure off the error text
(both sequential and parallel scanners) and fail the scan.
* #9: -R + --files-from receiver-protect rules recorded the source-relative
path; record the bare relative wire path in both scanners so the protected
destination mirror survives --delete.
* #5: the STATUS_STATS would-delete parser now validates each retained path and
enforces the shared MAX_MANIFEST_BYTES budget, and the --out-format dry-run
delete line is escaped like the itemize line.
* #11: drop the unused DELETE_PLAN_MAX_NAMES macro, log the delete-limit
warning once per session, roll back dir-merge names from a per-directory file
that fails to parse, and guard every filter error snprintf against err==NULL.
#10 leaves the empty directory itself kept and its extras removed, matching
rsync's final state on both per-directory timings.
rsync's %c counts the block-checksum bytes received: even a whole-file
transfer with no basis receives rsync's 16-byte sum header (append and
inplace included), while a dry run receives nothing. FastSync's
whole-file path has no equivalent header, so report 16 for parity when
delta is inactive, keep 0 for dry runs, and keep the real received bytes
when delta is active (FastSync's signature framing differs, so delta %c
stays divergent). Add a strict %c/%l/%n differential against rsync and
turn the %b check into a real rsync differential (semantics: both count
wire bytes and exceed %l; the exact values are protocol-specific).
Accept the full rsync 3.4.1 --info (backup, del, flist, mount, nonreg,
progress, remove, symsafe) and --debug (acl, backup, bind, chdir, connect,
cmd, del, deltasum, dup, exit, filter, flist, fuzzy, genr, hash, hlink,
iconv, nstr, own, recv, send, time) vocabularies, plus the historical
syms/hl/owner aliases, with level suffixes. Categories FastSync already
emits (copy/name/misc/skip/stats and io/proto/pack/util) still set their
log flags; the rest are accepted but silent. Unknown names remain
rejected by name, matching rsync. Update the CLI unit tests and the
rsync-parity integration tests (previously they required del/filter to be
rejected).
- Assert FastSync's sparse/preallocate block accounting matches rsync 3.4.1
for a hole file (preallocate wins over sparse, exactly like rsync).
- Assert --ignore-existing is answered by the receiver before the sender
transmits the payload (CountingProxy wire volume near-zero).
- CountingProxy.run gains a bounded join_timeout so tests that only need the
client->server count do not wait for the server's idle socket.
- Fix the stale protocol 2.24.0 comment in test_config.c (golden is 2.26.0).
The single-threaded and -m receivers never populated ReceiverStats.matched_data
or .deleted_files, so --stats always printed 0 for both even when rsync
reported nonzero. Track the bytes reconstructed from the basis file while
applying a delta, and tally the delete-commit counts (manifest and
per-directory sessions) into the receiver stats. The -m pipeline now carries
its own stats/would-delete fields and emits the STATUS_STATS frame before the
terminal success, so --threads finally reports the counters and renders
-n --delete lines.
Also normalize the -n --delete would-delete enumeration's absolute basis
prefixes exactly like the real commit path (fixing an over-report) and fix the
basis_delete_relative off-by-one when the receive root is '/'. Unit tests
cover the root mapping and the basis protection; integration tests cover
matched/deleted stats for both receivers and the --threads dry-run delete
lines.
delete_plan_session_commit() lacked the central no-mutation guard that
manifest_delete_all() has, so a server-contacting -n run (or a hostile plan
frame) could still remove --delete-missing-args mirrors on the per-directory
timing path. Return DELETE_COMMIT_OK immediately when the session is a
dry-run, and gate the receiver/server commit call sites too. Add a unit test
that streams a plan naming an existing destination file and asserts it
survives.
The -R prefix marker installed in synced_dirs was discarded when finalizing
the per-directory delete sender (--delete-during/--delete-delay), so the
up-front root plan was the receive root '.', whose keep list only held the
first prefix component. The receiver then deleted destination content
outside the transferred prefix (e.g. unrelated/keep.txt), a data-loss bug;
rsync keeps it.
Confine the walk to the -R prefix: send that prefix's plan as the root plan,
only transmit plans at or below it, and never emit the receive root plan for
a scoped run. Add a differential test covering both --delete-during and
--delete-delay.
Add tests/integration/test_codecs.py (accept/reject matrix and byte
differential against rsync 3.4.1 for every algorithm), extend the
checksum/compression unit tests with MD4/SHA1/none vectors and per-codec
round-trips, pin the new config golden (2.26.0), and update the version
strings and codec acceptance expectations.
Accept the full rsync 3.4.1 --compress-choice set (zstd/lz4/zlib/zlibx/
none/auto) and the two-name --checksum-choice TRANSFER,PRE-TRANSFER form,
including rsync's 'none' rules (rejected with --checksum at exit 4, and
forcing --whole-file as the transfer half) and unknown names at exit 4.
The checksum default becomes the auto-negotiated xxh128.
Negotiation is deterministic and symmetric: both peers run the same
preference resolver (rsync's --version order). The resolved
compression_algo crosses the wire as a new trailing config-frame int so
the receiver validates and installs the exact codec; an unsupported
choice is refused before STATUS_OK like rsync's failed negotiation.
Bump PROTOCOL_VERSION to 2.26.0.
Add real implementations for the rsync 3.4.1 checksum and compression
breadth: a self-contained MD4 (RFC 1320), OpenSSL-backed SHA1, a
no-digest mode, and LZ4/zlib codecs alongside zstd. Compressed buffers
are now self-describing (a leading codec id), so every existing
decompression call site keeps working through a process-global codec
selection. zlibx shares the zlib codec because FastSync compresses only
delta/token bytes (never matched file data), matching the 'x' intent.
- Only honor the --delete-missing-args exact paths when the server's
--allow-delete policy left delete_missing_args set.
- -d/--dirs does not recurse, so a per-directory plan would carry no child
information and could delete the contents of an untraversed directory; fall
back to the whole-tree end-of-transfer commit for that mode.
A -R source prune (--exclude/--max-size) must record the destination wire
path below the reconstructed prefix so --delete protects it; the parallel
root scan and the sequential skip path used the source path instead.
A general -R transfer places its files below the reconstructed prefix, so
marking the whole receive root as the delete scope deleted unrelated
sibling directories (data loss; rsync keeps them). Use the prefix itself
as the root marker when it is non-empty, in both the single-threaded and
multithreaded pipelines.
rsync prints the --stats block (with the (DRY RUN) suffix) for -n; route
the dry-run path through report_transfer_stats using the wire counters and
the STATUS_STATS receiver report.
The receiver emits the wire-stats frame before the per-file acks and the
terminal status; the client must consume it in that order or a combined
--stats --remove-source-files run desynchronizes.
Replace the aggregate stderr progress with rsync 3.4.1's per-file progress
block (name, 32 KiB first frame, final frame with (xfr#N, to-chk=X/Y)).
Add differential tests against real rsync for --out-format %C/%b, the
--progress frames, selected --stats lines and -n --delete lines.
Add the STATUS_STATS end-of-transfer receiver report (matched/deleted
counters plus a would-delete path list) behind the report_stats wire
bool, and a read-only delete_extras_list walker. --stats now renders
true wire byte totals and the receiver-reported deleted count; a
server-contacting -n --delete prints transfer-relative '*deleting' lines
matching rsync's itemize layout.
- Compare --delete-during/--delete-delay final state against rsync 3.4.1.
- Force a mid-transfer failure through a byte-slicing proxy: --delete-during has
removed the processed directory's extra, --delete-delay has not.
- Create a destination entry while the transfer is in flight: it survives
--delete-delay's snapshot but is removed by --delete-after's fresh end scan.
- Cover the --delete-delay type-conflict case now matching rsync.
TapTap
opencode