Implements the Phase-3 "alternate basis directories" row: each flag is parsed,
repeatable, serialized to the receiver, and honored in the receiver's per-file
"unchanged" decision. Protocol version bumped 2.7.0 -> 2.8.0 (config wire gains
the basis-dir list).
Design (receiver-side match flow)
FastSync is sender-streams-data; only the receiver can see the basis dirs and the
current destination, so the basis decision lands in receive_incremental_check
(the --incremental STATUS_CHECK handshake). Supplying a basis dir therefore
implies --incremental on the client.
Per incoming file the sender already sends path/size/mtime; when basis dirs are
configured it also sends its xxHash64 (same frame slot as --checksum, gated
by config on both sides). After the normal destination quick-check fails, the
receiver probes each basis dir in command-line order. An exact match requires
equal size, equal mtime (unless --size-only; --ignore-times disables basis
matching like rsync), and equal content xxHash64 — so a hard link/copy is only
ever made from byte-identical content. On a match the receiver replies STATUS_OK
so the sender omits the data; on no match the existing delta / full-data path runs
byte-for-byte unchanged.
--compare-dest: skip data only when the destination does not already hold
the file (sparse destination, rsync parity). Never copies. A destination that
holds a different version falls back to a normal transfer (documented
divergence: rsync deletes the stale entry; FastSync keeps the mirror complete).
--copy-dest: materialize a real local copy of the basis file through the
normal atomic store engine (so --existing/--ignore-existing/--update/ --backup/--delay-updates/--partial-dir all still apply unchanged).
--link-dest: install an atomic hard link (temp linkat + rename), with a
byte-identical local-copy fallback for EXDEV/EPERM/unsupported filesystems —
never a corrupt or partial file. --delay-updates stages the link and publishes
by rename so the final entry stays a real hard link. Successful links keep the
basis inode's own attributes (metadata is never written through the shared
inode, so the basis file is never mutated).
Basis dirs are resolved relative to the destination root and confined below it
(absolute / .. / . values are rejected up front), reusing the --backup-dir
confinement model and the secure O_NOFOLLOW path helpers. They are excluded from --delete (a delete run must never remove the snapshot a link-dest run links
from).
Divergences from rsync (documented in RSYNC_COMPAT.md)
compare-dest never deletes a stale differing destination entry (rsync does).
Attribute-only differences are not re-applied on a basis hit (data is skipped,
so the sender never sends metadata).
Already-up-to-date destination files are not re-linked to a basis file.
--remove-source-files sources satisfied from a basis dir are retained.
Basis matching verifies content with xxHash64 (stricter than rsync's quick check).
Basis dirs require --incremental (implied) and are incompatible with -s.
Integration: compare-dest sparse skip + transfer of unsupported files,
copy-dest real-copy inode, link-dest hard-link inode/nlink, content-mismatch
fallback for all three, missing-basis no-op, -m multithreaded link-dest, --delay-updates link-dest (staging cleaned, real link), and --delete
leaving the basis dir untouched.
Clean strict-warning build; unit + full integration suites pass; clang-format +
cppcheck clean; ASan unit + basis integration pass.
## Alternate basis directories (`--compare-dest` / `--copy-dest` / `--link-dest`)
Implements the Phase-3 "alternate basis directories" row: each flag is parsed,
repeatable, serialized to the receiver, and honored in the receiver's per-file
"unchanged" decision. Protocol version bumped 2.7.0 -> 2.8.0 (config wire gains
the basis-dir list).
### Design (receiver-side match flow)
FastSync is sender-streams-data; only the receiver can see the basis dirs and the
current destination, so the basis decision lands in `receive_incremental_check`
(the `--incremental` STATUS_CHECK handshake). Supplying a basis dir therefore
implies `--incremental` on the client.
Per incoming file the sender already sends path/size/mtime; when basis dirs are
configured it **also sends its xxHash64** (same frame slot as `--checksum`, gated
by config on both sides). After the normal destination quick-check fails, the
receiver probes each basis dir in command-line order. An *exact match* requires
equal size, equal mtime (unless `--size-only`; `--ignore-times` disables basis
matching like rsync), **and** equal content xxHash64 — so a hard link/copy is only
ever made from byte-identical content. On a match the receiver replies `STATUS_OK`
so the sender omits the data; on no match the existing delta / full-data path runs
byte-for-byte unchanged.
- `--compare-dest`: skip data only when the destination does **not** already hold
the file (sparse destination, rsync parity). Never copies. A destination that
holds a *different* version falls back to a normal transfer (documented
divergence: rsync deletes the stale entry; FastSync keeps the mirror complete).
- `--copy-dest`: materialize a real **local copy** of the basis file through the
normal atomic store engine (so `--existing`/`--ignore-existing`/`--update`/
`--backup`/`--delay-updates`/`--partial-dir` all still apply unchanged).
- `--link-dest`: install an atomic **hard link** (temp `linkat` + rename), with a
byte-identical local-copy fallback for EXDEV/EPERM/unsupported filesystems —
never a corrupt or partial file. `--delay-updates` stages the link and publishes
by rename so the final entry stays a real hard link. Successful links keep the
basis inode's own attributes (metadata is never written through the shared
inode, so the basis file is never mutated).
Basis dirs are resolved relative to the destination root and confined below it
(absolute / `..` / `.` values are rejected up front), reusing the `--backup-dir`
confinement model and the secure O_NOFOLLOW path helpers. They are excluded from
`--delete` (a delete run must never remove the snapshot a link-dest run links
from).
### Divergences from rsync (documented in RSYNC_COMPAT.md)
- compare-dest never deletes a stale differing destination entry (rsync does).
- Attribute-only differences are not re-applied on a basis hit (data is skipped,
so the sender never sends metadata).
- Already-up-to-date destination files are not re-linked to a basis file.
- `--remove-source-files` sources satisfied from a basis dir are retained.
- Basis matching verifies content with xxHash64 (stricter than rsync's quick check).
- Basis dirs require `--incremental` (implied) and are incompatible with `-s`.
### Test coverage
- Unit: CLI parsing (both forms, repetition order, invalid paths, implied
incremental, `-s` rejection) and config wire round-trip (incl. receiver-side
rejection of escaping paths).
- Integration: compare-dest sparse skip + transfer of unsupported files,
copy-dest real-copy inode, link-dest hard-link inode/nlink, content-mismatch
fallback for all three, missing-basis no-op, `-m` multithreaded link-dest,
`--delay-updates` link-dest (staging cleaned, real link), and `--delete`
leaving the basis dir untouched.
- Clean strict-warning build; unit + full integration suites pass; clang-format +
cppcheck clean; ASan unit + basis integration pass.
Replace the vestigial single compare_dest/copy_dest/link_dest Config fields with
an ordered BasisDest list (type + path per entry) that is serialized to the
receiver and interpreted relative to the destination root. Paths must be
relative with no '.'/'..' components (confined like --backup-dir); trailing
slashes are normalized. Wire layout changes, so PROTOCOL_VERSION -> 2.8.0.
CLI: each flag is parsed in both --flag=DIR and --flag DIR forms, is
repeatable, and keeps command-line order as basis priority. Supplying any
basis dir implies --incremental (and therefore metadata) on the sender because
the unchanged decision is receiver-side; combining basis dirs with -s chunk
serialization is rejected in validate_config. Usage text updated.
The receiver's per-file incremental check now consults the ordered basis-dir
list whenever the destination is not already up to date. An exact basis match
requires equal size, equal mtime (unless --size-only; --ignore-times disables
basis matching like rsync), and an equal content xxHash64 -- the sender sends
its xxHash for every file whenever basis dirs are configured (not only under
--checksum), so a hard link or local copy is only ever made from byte-identical
content.
On a match:
- compare-dest: reply STATUS_OK and skip data only when the destination does
not already hold the file (sparse, rsync parity). A destination that holds
a DIFFERENT version falls back to a normal transfer instead of rsync's
delete, keeping the mirror complete.
- copy-dest: reply STATUS_OK and hand a synthetic File (bytes read from the
basis file, basis metadata) to the normal store sink, so the file is
installed as a real local copy through the existing atomic temp+rename
engine and honors --existing/--ignore-existing/--update/--backup/
--delay-updates/--partial-dir unchanged.
- link-dest: same, but File.basis_link records the basis path and the store
engine calls the new file_to_disk_secure_link(): an atomic temp hard link +
rename. Cross-filesystem/refused links fall back to a byte-identical local
copy (never a corrupt or partial file); the copy fallback applies metadata,
while a successful link keeps the basis inode's own attributes so the basis
file is never mutated.
Basis-materialized files carry File.skip so they are not acknowledged to a
--remove-source-files sender (the sender already saw STATUS_OK and keeps the
source). The no-match path is byte-for-byte identical to the existing delta /
full-data transfer.
Generalize the delete walker's protected-root-child skip into a prefix list.
The receiver now passes both the --delay-updates staging directory and every
basis-dir path, so a --delete run can never treat a basis snapshot (which a
--link-dest run just linked from) as destination content to remove.
- parse_args accepts each flag in both forms, keeps repetition order/types,
implies --incremental + metadata, and rejects absolute/escaping/degenerate
paths; validate_config rejects basis dirs combined with -s.
- config wire round-trips a mixed basis list and rejects escaping/absolute
paths on the receiver side.
- compare-dest skips an exact basis match (leaving a sparse destination) and
still transfers files the basis cannot satisfy; a content mismatch forces a
normal transfer.
- copy-dest materializes the unchanged file as a real local copy (distinct
inode) and transfers content mismatches.
- link-dest hard-links (asserted same inode/nlink to the DIR file) and falls
back to a normal transfer on content mismatch.
- a missing basis dir is a clean full-transfer no-op for all three flags.
- link-dest works through -m multithreading and --delay-updates (staged and
published as a real link, staging cleaned up).
- --delete removes genuine extras while leaving the basis dir untouched.
Document receiver-side basis semantics, relative-to-destination-root
confinement, content-verified matching, hard-link vs copy vs compare-only
behavior, cross-filesystem copy fallback, repetition/priority, --delete
exclusion, the implied --incremental and -s incompatibility, and each exact
divergence from rsync (no dest deletion on compare-dest stale entries, no
attribute re-application on basis hits, no re-linking of already-up-to-date
dest files, sources matched from basis dirs kept under --remove-source-files).
config_basis_path_valid and config_basis_append now share one normalizer
(basis_path_normalize): interior empty components (a//b) collapse, '.'
components and trailing slashes are dropped, and the stored form is exactly
the canonical relative path used by validation, the delete-walker prefix match
and the receiver's basis lookup. Degenerate inputs (empty, absolute, '..',
'.' that normalizes to nothing) stay rejected.
- copy-dest basis hits leaked the heap-allocated basis path: BASIS_DEST_COPY
did not transfer it (only LINK does) and returned before the basis cleanup.
basis_match_free is now called on every materialization return path (success
and send-failure) after content/link ownership is transferred.
- the --delete walker regression: the delay-updates staging name must be
protected only as a DIRECT child of the receive root, while basis dirs may
be skipped at any depth. delete_extras_limited now takes DeleteSkipEntry
entries carrying a top_level_only flag instead of a flat prefix list, so a
nested destination directory named .fastsync-stage is ordinary content again
(its extras are deleted) and a basis tree is still never removed.
Basis dirs imply the per-file incremental check, which (like every whole-file
payload path in FastSync) is bounded by MAX_RECEIVE_WHOLE_FILE_SIZE. A source
tree with a larger file used to abort the whole run mid-stream on the receiver
with no client-side diagnostic. With basis dirs configured the client now
preflights the scan (respecting filters/size rules) and fails up front with a
clear error naming the offending file before connecting, matching the
documented 256 MiB whole-file limit instead of aborting silently.
- unit: config basis-path normalization (trailing slash, a//b, ./x/./y collapse;
degenerate inputs rejected).
- integration: same-size/same-mtime/different-content fixtures prove the xxHash
gate -- the basis changed.txt now has the SAME byte size as the source so the
size short-circuit can no longer mask the hash comparison, plus a dedicated
parametrized same-size mismatch test asserting link/copy never use a
content-mismatched basis and compare-dest transfers.
- basis-dir priority is first-match-wins: two link-dest dirs (inode of the
first), and compare-dest before link-dest stays sparse while the reverse
order hard-links.
- --size-only links a same-content basis file with a different mtime;
--ignore-times never links even an exact match.
- --delay-updates + --delete removes extras inside a nested .fastsync-stage
dir (regression guard) while keeping the real staging dir and basis tree.
- a basis run containing a file above the whole-file limit fails up front with
a clear error and transfers nothing.
- --link-dest destination entries share the basis inode: a later --inplace run
against such a path mutates the basis snapshot through the shared inode
(recommend --copy-dest when the destination must stay independently writable).
- basis-hit files take mode/uid/gid and mtime from the basis file, not the
sender's metadata (with --size-only the mtime can differ from the source).
- --remove-source-files sources satisfied by a basis dir are retained.
- basis runs refuse files above the 256 MiB whole-file limit up front (FastSync
caps every whole-file payload path at 256 MiB; rsync supports arbitrary sizes);
the config frame always carries a basis-count field (protocol 2.8.0).
The seed run did not preserve timestamps, so the destination copy's mtime was
the write time; the incremental --remove-source-files rerun only skipped the
file when both writes happened to land in the same whole second, making the
test flaky (observed intermittently in local full-suite runs and on CI). Seed
with -M so the destination stores the source's exact mtime.
Merged into dev via local merge (2FA blocks server-side merge). Commits: delete-timing 265b1e7 (rsync delete timing, protocol 2.8.0) + basis-dest 01a5a93. Independent c-review APPROVE WITH NITS / REQUEST CHANGES; all findings fixed and re-verified. dev CI run #479: all jobs success (lint, build-and-test, sanitizers address+undefined, fuzz-build, coverage, valgrind). Closing without server merge.
Merged into dev via local merge (2FA blocks server-side merge). Commits: delete-timing 265b1e7 (rsync delete timing, protocol 2.8.0) + basis-dest 01a5a93. Independent c-review APPROVE WITH NITS / REQUEST CHANGES; all findings fixed and re-verified. dev CI run #479: all jobs success (lint, build-and-test, sanitizers address+undefined, fuzz-build, coverage, valgrind). Closing without server merge.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Alternate basis directories (
--compare-dest/--copy-dest/--link-dest)Implements the Phase-3 "alternate basis directories" row: each flag is parsed,
repeatable, serialized to the receiver, and honored in the receiver's per-file
"unchanged" decision. Protocol version bumped 2.7.0 -> 2.8.0 (config wire gains
the basis-dir list).
Design (receiver-side match flow)
FastSync is sender-streams-data; only the receiver can see the basis dirs and the
current destination, so the basis decision lands in
receive_incremental_check(the
--incrementalSTATUS_CHECK handshake). Supplying a basis dir thereforeimplies
--incrementalon the client.Per incoming file the sender already sends path/size/mtime; when basis dirs are
configured it also sends its xxHash64 (same frame slot as
--checksum, gatedby config on both sides). After the normal destination quick-check fails, the
receiver probes each basis dir in command-line order. An exact match requires
equal size, equal mtime (unless
--size-only;--ignore-timesdisables basismatching like rsync), and equal content xxHash64 — so a hard link/copy is only
ever made from byte-identical content. On a match the receiver replies
STATUS_OKso the sender omits the data; on no match the existing delta / full-data path runs
byte-for-byte unchanged.
--compare-dest: skip data only when the destination does not already holdthe file (sparse destination, rsync parity). Never copies. A destination that
holds a different version falls back to a normal transfer (documented
divergence: rsync deletes the stale entry; FastSync keeps the mirror complete).
--copy-dest: materialize a real local copy of the basis file through thenormal atomic store engine (so
--existing/--ignore-existing/--update/--backup/--delay-updates/--partial-dirall still apply unchanged).--link-dest: install an atomic hard link (templinkat+ rename), with abyte-identical local-copy fallback for EXDEV/EPERM/unsupported filesystems —
never a corrupt or partial file.
--delay-updatesstages the link and publishesby rename so the final entry stays a real hard link. Successful links keep the
basis inode's own attributes (metadata is never written through the shared
inode, so the basis file is never mutated).
Basis dirs are resolved relative to the destination root and confined below it
(absolute /
../.values are rejected up front), reusing the--backup-dirconfinement model and the secure O_NOFOLLOW path helpers. They are excluded from
--delete(a delete run must never remove the snapshot a link-dest run linksfrom).
Divergences from rsync (documented in RSYNC_COMPAT.md)
so the sender never sends metadata).
--remove-source-filessources satisfied from a basis dir are retained.--incremental(implied) and are incompatible with-s.Test coverage
incremental,
-srejection) and config wire round-trip (incl. receiver-siderejection of escaping paths).
copy-dest real-copy inode, link-dest hard-link inode/nlink, content-mismatch
fallback for all three, missing-basis no-op,
-mmultithreaded link-dest,--delay-updateslink-dest (staging cleaned, real link), and--deleteleaving the basis dir untouched.
cppcheck clean; ASan unit + basis integration pass.
The receiver's per-file incremental check now consults the ordered basis-dir list whenever the destination is not already up to date. An exact basis match requires equal size, equal mtime (unless --size-only; --ignore-times disables basis matching like rsync), and an equal content xxHash64 -- the sender sends its xxHash for every file whenever basis dirs are configured (not only under --checksum), so a hard link or local copy is only ever made from byte-identical content. On a match: - compare-dest: reply STATUS_OK and skip data only when the destination does not already hold the file (sparse, rsync parity). A destination that holds a DIFFERENT version falls back to a normal transfer instead of rsync's delete, keeping the mirror complete. - copy-dest: reply STATUS_OK and hand a synthetic File (bytes read from the basis file, basis metadata) to the normal store sink, so the file is installed as a real local copy through the existing atomic temp+rename engine and honors --existing/--ignore-existing/--update/--backup/ --delay-updates/--partial-dir unchanged. - link-dest: same, but File.basis_link records the basis path and the store engine calls the new file_to_disk_secure_link(): an atomic temp hard link + rename. Cross-filesystem/refused links fall back to a byte-identical local copy (never a corrupt or partial file); the copy fallback applies metadata, while a successful link keeps the basis inode's own attributes so the basis file is never mutated. Basis-materialized files carry File.skip so they are not acknowledged to a --remove-source-files sender (the sender already saw STATUS_OK and keeps the source). The no-match path is byte-for-byte identical to the existing delta / full-data transfer.Merged into dev via local merge (2FA blocks server-side merge). Commits: delete-timing
265b1e7(rsync delete timing, protocol 2.8.0) + basis-dest01a5a93. Independent c-review APPROVE WITH NITS / REQUEST CHANGES; all findings fixed and re-verified. dev CI run #479: all jobs success (lint, build-and-test, sanitizers address+undefined, fuzz-build, coverage, valgrind). Closing without server merge.Pull request closed