Where the fuzzy decision lives. Entirely receiver-side, inside receive_incremental_check (src/shared/file_receive.c). When a file must be
transferred and the destination's own content at the exact path cannot serve
as a delta basis (file absent, or outside the delta engine's size bounds), the
receiver searches the target's destination directory for a similar existing
regular file, computes the block signature over THAT file's bytes, and sends
the normal STATUS_DELTA_SIGNATURE. The sender never learns the basis was a
different file and needs no new logic — only the config frame grew a fuzzy
boolean, so PROTOCOL_VERSION was bumped 2.8.0 → 2.9.0 (peers must match).
Similarity heuristic (deterministic, simpler than rsync's; documented in
code + RSYNC_COMPAT.md). Candidates = sibling entries of the target opened O_NOFOLLOW/AT_SYMLINK_NOFOLLOW under the confined root (no symlinks
followed, nothing outside the destination root read/hashed); dotfiles, dirs,
the target's own name and stage/temp names excluded; size gate = delta_should_attempt (both ≥ 16 KiB, ≤ --delta-max, ratio ≤ 10×) rather
than rsync's ~1.5× size window; name gate = Levenshtein distance ≤ half the
longer basename; closest match wins (size tie-break then lexical); directory
scan capped at 4096 entries.
Byte-exactness argument. Reconstruction never depends on which bytes the
basis holds: block matches require the sender's Adler-32 + xxHash32 checksums
to agree (delta_compute/delta_apply validate every reference against the basis
size), and a basis that shares nothing makes the sender reply STATUS_NEXT
(whole-file transfer). A fuzzy basis can waste bandwidth, never corrupt.
When fuzzy applies / divergences. Only for files the receiver would
otherwise send whole; the destination's own file stays the preferred delta
basis when it exists and fits the delta size bounds. No similar candidate →
normal whole-file transfer (tested). FastSync delta is off by default (unlike
rsync), so --fuzzy implies --incremental + --delta; -W/--no-delta
leave it inert. --no-fuzzy negates via the generic negation machinery.
Integration (new TestFuzzy + CountingProxy byte counter): rename case
transfers a 2 MiB file in ~2% of its bytes with byte-exact output under --fuzzy, while no-fuzzy / no-candidate / dissimilar-sibling / --whole-file
runs send the whole file; -y alias; single-thread vs -m parity;
dest-holds-unsuitable-file falls back to the sibling; --delay-updates and --remove-source-files semantics preserved.
feat: --fuzzy/-y/--no-fuzzy similar-file delta basis
## Design summary
**Where the fuzzy decision lives.** Entirely receiver-side, inside
`receive_incremental_check` (src/shared/file_receive.c). When a file must be
transferred and the destination's own content at the exact path cannot serve
as a delta basis (file absent, or outside the delta engine's size bounds), the
receiver searches the target's destination directory for a similar existing
regular file, computes the block signature over THAT file's bytes, and sends
the normal `STATUS_DELTA_SIGNATURE`. The sender never learns the basis was a
different file and needs no new logic — only the config frame grew a `fuzzy`
boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match).
**Similarity heuristic (deterministic, simpler than rsync's; documented in
code + RSYNC_COMPAT.md).** Candidates = sibling entries of the target opened
`O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (no symlinks
followed, nothing outside the destination root read/hashed); dotfiles, dirs,
the target's own name and stage/temp names excluded; size gate =
`delta_should_attempt` (both ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×) rather
than rsync's ~1.5× size window; name gate = Levenshtein distance ≤ half the
longer basename; closest match wins (size tie-break then lexical); directory
scan capped at 4096 entries.
**Byte-exactness argument.** Reconstruction never depends on which bytes the
basis holds: block matches require the sender's Adler-32 + xxHash32 checksums
to agree (delta_compute/delta_apply validate every reference against the basis
size), and a basis that shares nothing makes the sender reply `STATUS_NEXT`
(whole-file transfer). A fuzzy basis can waste bandwidth, never corrupt.
**When fuzzy applies / divergences.** Only for files the receiver would
otherwise send whole; the destination's own file stays the preferred delta
basis when it exists and fits the delta size bounds. No similar candidate →
normal whole-file transfer (tested). FastSync delta is off by default (unlike
rsync), so `--fuzzy` implies `--incremental` + `--delta`; `-W`/`--no-delta`
leave it inert. `--no-fuzzy` negates via the generic negation machinery.
## Test coverage
- CLI unit: `--fuzzy`, `-y`, `--no-fuzzy` (order-independent), `-W`/`--no-delta`
interaction, `-s`/`-f` rejection, delta/incremental/metadata implication.
- Config wire round-trip of the new `fuzzy` field.
- Integration (new `TestFuzzy` + `CountingProxy` byte counter): rename case
transfers a 2 MiB file in ~2% of its bytes with byte-exact output under
`--fuzzy`, while no-fuzzy / no-candidate / dissimilar-sibling / `--whole-file`
runs send the whole file; `-y` alias; single-thread vs `-m` parity;
dest-holds-unsuitable-file falls back to the sibling; `--delay-updates` and
`--remove-source-files` semantics preserved.
-y/--fuzzy is receiver-side similar-file basis selection, so the receiver
must learn the flag: add a bool to Config, serialize it as a trailing field
on the config frame, and validate the received value. Because the config
frame layout changed, PROTOCOL_VERSION moves to 2.9.0 (peers must match).
Register --fuzzy (alias -y) as a boolean option and fuzzy as negatable so
--no-fuzzy works through the generic negation machinery. FastSync's delta
machinery is off by default (unlike rsync, where --fuzzy implies nothing
because delta is the default), so --fuzzy implies --incremental and --delta
unless --whole-file or an explicit --no-delta switched delta off (leaving
fuzzy inert, matching rsync where -W makes fuzzy irrelevant). Add help text.
When a file must be transferred and the destination holds no usable content
at the exact path (file absent, or outside the delta engine's size bounds),
the receiver now searches the target's own destination directory for an
existing regular file with a similar basename and uses it as the delta basis
via the existing receiver-driven STATUS_DELTA_SIGNATURE handshake. The
sender never learns the basis was another file, so no wire change beyond the
new config flag was required.
Heuristic (deterministic, simpler than rsync's, documented): candidates are
sibling entries confined below the root and opened O_NOFOLLOW (symlinks are
never followed, nothing outside the destination root is read); dotfiles,
directories, the target's own name and stage/temp names are excluded; size
gate is delta_should_attempt; name gate is a Levenshtein distance <= half
the longer basename; the closest candidate (size tie-break, then lexical) is
loaded; the scan is capped at 4096 entries.
Byte-exactness is independent of the basis: block matches are verified by
Adler-32 + xxHash32, delta_apply validates every reference, and a basis that
shares nothing makes the sender reply with a whole-file transfer. When no
candidate qualifies the normal whole-file transfer runs unchanged.
Unit: parse -y/--fuzzy and --no-fuzzy negation (order-independent), -W and
--no-delta leave fuzzy inert, --fuzzy implies --incremental/--delta, and
validate_config rejects fuzzy with -s (chunk serialization) and -f
(sendfile). Config: fuzzy survives the socketpair wire round-trip.
Integration (TestFuzzy, byte counts via a new CountingProxy helper): the
rename case transfers a 2 MiB file in a few percent of its size with byte-
exact output under --fuzzy, while the no-fuzzy, no-candidate, dissimilar-
sibling and --whole-file runs send the whole file; -y alias; -m parity;
dest-holds-unsuitable-file falls back to the sibling; --delay-updates and
--remove-source-files keep their semantics on fuzzy transfers.
Document the receiver-side decision location, the exact deterministic
similarity heuristic, the byte-exactness argument, when fuzzy applies (and
when it deliberately does not), the 2.8.0 -> 2.9.0 protocol bump, and the
divergences from rsync. Update the basis-dir rows' protocol references to
the now-current 2.9.0.
Review follow-ups on the --fuzzy candidate scan:
- Allocate the two DP rows once per directory scan instead of once per
candidate (4096-entry directories no longer do thousands of malloc pairs).
- Pre-prune before the DP with two cheap lower bounds on the edit distance:
the name length gap and the count of characters of one basename absent from
the other; a candidate whose gate (distance*2 <= longer) already fails on
the max of those bounds is skipped without running the DP.
- Trim the common prefix and non-overlapping common suffix before the DP so
it only runs over the differing middles.
- Note that the 4096 readdir cap bounds iterations, not per-entry DP cost,
and that the seen set is filesystem-order dependent (winner stays
deterministic via the total comparator).
- Open the chosen candidate with O_NONBLOCK so a name raced to a FIFO cannot
block the receive thread forever in open(2); the existing fstat S_ISREG gate
still rejects non-regular files. (The pre-existing basis_open_regular has
the same latent FIFO pattern and is intentionally left unchanged.)
- Free old_data in receive_delta_file's defensive NULL guard.
The --fuzzy implication previously forced use_incremental back on even when
the user passed --no-incremental, while --no-delta and -W were honored.
Track a --no-incremental latch (like the no_delta latch): with it set, do not
force the handshake on, and because delta needs the handshake, also suppress
the delta implication so no invalid '--delta requires --incremental' config
results. A --fuzzy --no-incremental run is therefore a plain default-mode
transfer (fuzzy inert), consistent with -W/--no-delta. Documented in the
usage text (this deliberately differs from the basis-dir options, which still
force incremental unconditionally).
Review follow-ups on the --fuzzy row: note that the receiver's block
signature is derived from a sibling file it may not otherwise send, exposing
destination sibling files to the sender at block granularity (the same
known-plaintext information class as the ordinary delta path); and document
that --fuzzy honors an explicit --no-incremental (unlike the basis-dir
options, which force it), with the delta implication suppressed accordingly.
Unit (CLI): --fuzzy --no-incremental (either order) stays a valid plain-mode
config -- no forced handshake, no delta implication; --fuzzy --no-delta stays
covered.
Integration (TestFuzzy):
- worthless basis: a sibling that passes the name+size gates but shares no
blocks makes the sender reply STATUS_NEXT; the whole file is consumed inside
the delta handshake with byte-exact output (no protocol desync).
- non-displacement: an existing exact-path destination file INSIDE the delta
size bounds (same size, different content, older mtime) is used as the delta
basis instead of a byte-identical similar sibling (whole-file wire cost).
- asymmetric bases: basis larger than source (prefix reuse) and basis smaller
than source (appended tail as literals) both reconstruct byte-exactly with a
small delta.
- --no-fuzzy end-to-end equals the no-flag whole-file behavior.
CountingProxy closes its listener socket (fd hygiene).
The suffix-trim loop using computed end offsets tripped cppcheck's
knownConditionTrueFalse value-range analysis (it unsoundly concluded the trims
always consume the whole middle). Rewrite it with explicit moving end indices
and add an inline suppression with a rationale for the residual false
positive; cppcheck --error-exitcode=1 is clean again. The trimming logic is
unchanged and was verified against a full DP reference over 200k random name
pairs.
Merged into dev via local merge (2FA blocks server-side merge). delete-policy ef13a70 + fuzzy ebab335. Independent c-review: delete-policy REQUEST CHANGES (2 blockers fixed: -m use-after-free, unreadable-root wipe under --ignore-errors) ; fuzzy APPROVE WITH NITS (perf/confinement/CLI findings fixed). dev CI run #487: all jobs success. Closing without server merge.
Merged into dev via local merge (2FA blocks server-side merge). delete-policy ef13a70 + fuzzy ebab335. Independent c-review: delete-policy REQUEST CHANGES (2 blockers fixed: -m use-after-free, unreadable-root wipe under --ignore-errors) ; fuzzy APPROVE WITH NITS (perf/confinement/CLI findings fixed). dev CI run #487: all jobs success. Closing without server merge.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
feat: --fuzzy/-y/--no-fuzzy similar-file delta basis
Design summary
Where the fuzzy decision lives. Entirely receiver-side, inside
receive_incremental_check(src/shared/file_receive.c). When a file must betransferred and the destination's own content at the exact path cannot serve
as a delta basis (file absent, or outside the delta engine's size bounds), the
receiver searches the target's destination directory for a similar existing
regular file, computes the block signature over THAT file's bytes, and sends
the normal
STATUS_DELTA_SIGNATURE. The sender never learns the basis was adifferent file and needs no new logic — only the config frame grew a
fuzzyboolean, so
PROTOCOL_VERSIONwas bumped 2.8.0 → 2.9.0 (peers must match).Similarity heuristic (deterministic, simpler than rsync's; documented in
code + RSYNC_COMPAT.md). Candidates = sibling entries of the target opened
O_NOFOLLOW/AT_SYMLINK_NOFOLLOWunder the confined root (no symlinksfollowed, nothing outside the destination root read/hashed); dotfiles, dirs,
the target's own name and stage/temp names excluded; size gate =
delta_should_attempt(both ≥ 16 KiB, ≤--delta-max, ratio ≤ 10×) ratherthan rsync's ~1.5× size window; name gate = Levenshtein distance ≤ half the
longer basename; closest match wins (size tie-break then lexical); directory
scan capped at 4096 entries.
Byte-exactness argument. Reconstruction never depends on which bytes the
basis holds: block matches require the sender's Adler-32 + xxHash32 checksums
to agree (delta_compute/delta_apply validate every reference against the basis
size), and a basis that shares nothing makes the sender reply
STATUS_NEXT(whole-file transfer). A fuzzy basis can waste bandwidth, never corrupt.
When fuzzy applies / divergences. Only for files the receiver would
otherwise send whole; the destination's own file stays the preferred delta
basis when it exists and fits the delta size bounds. No similar candidate →
normal whole-file transfer (tested). FastSync delta is off by default (unlike
rsync), so
--fuzzyimplies--incremental+--delta;-W/--no-deltaleave it inert.
--no-fuzzynegates via the generic negation machinery.Test coverage
--fuzzy,-y,--no-fuzzy(order-independent),-W/--no-deltainteraction,
-s/-frejection, delta/incremental/metadata implication.fuzzyfield.TestFuzzy+CountingProxybyte counter): rename casetransfers a 2 MiB file in ~2% of its bytes with byte-exact output under
--fuzzy, while no-fuzzy / no-candidate / dissimilar-sibling /--whole-fileruns send the whole file;
-yalias; single-thread vs-mparity;dest-holds-unsuitable-file falls back to the sibling;
--delay-updatesand--remove-source-filessemantics preserved.Merged into dev via local merge (2FA blocks server-side merge). delete-policy
ef13a70+ fuzzyebab335. Independent c-review: delete-policy REQUEST CHANGES (2 blockers fixed: -m use-after-free, unreadable-root wipe under --ignore-errors) ; fuzzy APPROVE WITH NITS (perf/confinement/CLI findings fixed). dev CI run #487: all jobs success. Closing without server merge.Pull request closed