Source files sharing (st_dev,st_ino) are recreated as hard links on the
destination; only the first member's data crosses the wire (siblings ride a
payload-less STATUS_HARDLINK frame). Ordering requires the single-FIFO-writer
receiver + forced sequential scan (documented). link()-failure falls back to a
byte-identical local copy. Rejects -s/--append. PROTOCOL_VERSION 2.11.0->2.12.0.
Review fixes: delete the dead HardLinkRegistry (ordering holds by FIFO writer),
and --existing no longer aborts when the first member is absent but the sibling
exists (leaves the sibling in place).
Receiver allocates the destination file's full size before streaming data
(posix_fallocate preferred, ftruncate fallback on EOPNOTSUPP/ENOSYS) so an
out-of-space transfer fails fast instead of partway. Additive config bool
crossing the wire; PROTOCOL_VERSION 2.10.0 -> 2.11.0. Threaded through all
store paths (atomic, inplace, partial/delay-updates staging, link-dest copy
fallback). Review hardening: explicit lseek(0) before the data write so
correctness does not depend on posix_fallocate leaving the fd offset unchanged.
Receiver-side ownership application, opt-in and privilege-gated:
- OFF for every existing transfer (plain -M/--preserve still never applies
ownership); only triggers on an explicit identity flag + receiver permission.
- EPERM/EACCES warn-and-continue (never aborts); other fchown errors escalate.
- fd-relative fchown after the file is written (symlink-safe, confined).
- New src/shared/identity.{c,h}; config fields numeric_ids / chown uid/gid /
usermap + groupmap id-pair tables cross the wire; PROTOCOL_VERSION 2.10.0
-> 2.11.0. CLI in client_cli.c; per-connection snapshot in server.c.
- Review fixes: EPERM/EACCES-only warn-and-continue, prominent root-receiver
notice, identity_clear_active on early server error paths, --numeric-ids
kept inert standalone (removed from activation trigger set).
Adds real algorithm selection (xxh64 default, plus md5 via OpenSSL EVP) and a
64-bit seed for the per-file whole-file digest used by the --incremental/
--checksum handshake and basis-dir content verification. The seed also feeds
the delta path's per-block xxHash32 strong checksum (low 32 bits) so an
explicit seed deterministically changes those digests too. Sender and receiver
hash identically: the algorithm id and seed cross the config wire frame and the
STATUS_CHECK handshake now carries a length-prefixed, bounded digest instead of
a fixed 64-bit value. Unsupported algorithm names are rejected at parse time
(never a silent no-op). PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0; defaults
(xxh64, seed 0) preserve prior byte-for-byte behavior.
Implement rsync's append modes: when an existing destination file is SHORTER
than the source, the receiver negotiates a resume offset and only the tail is
transferred; the full file (retained prefix + tail) is rebuilt and installed
through the normal atomic store path, so the result is byte-identical to the
source whenever the prefix matches.
- --append: sends the tail without content-verifying the retained prefix
(rsync parity; the documented prefix-trust risk).
- --append-verify: verifies the retained prefix against the source's prefix
xxHash64 before appending and, on a mismatch, falls back to a clean full
transfer (never a corrupt prefix+tail blend).
New wire frames STATUS_APPEND / STATUS_APPEND_SIG / STATUS_APPEND_OK /
STATUS_APPEND_DATA; PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0 (peers must match).
Both flags imply --incremental and are incompatible with -s (chunk
serialization) and --whole-file (rejected up front). Respects --inplace,
--partial/--partial-dir and --delay-updates via the shared store engine.
New flags parse onto the config; --delete-missing-args implies
--ignore-missing-args (order-independent) and does NOT imply --delete (rsync:
independent of other delete processing). The files-from preflight now classifies
listed-but-missing entries instead of hard-failing: under the flags each is
skipped (logged + counted, never silent) and the run succeeds for the rest,
including the all-missing case; an empty list stays a hard error. Under
--delete-missing-args the missing entries' destination mirrors (bare relative
path with -R, full source mirror otherwise) ride the manifest's third section in
both the single-threaded and -m senders; --dirs listed-but-missing entries are
skipped in the scanner.
The --fuzzy implication previously forced use_incremental back on even when
the user passed --no-incremental, while --no-delta and -W were honored.
Track a --no-incremental latch (like the no_delta latch): with it set, do not
force the handshake on, and because delta needs the handshake, also suppress
the delta implication so no invalid '--delta requires --incremental' config
results. A --fuzzy --no-incremental run is therefore a plain default-mode
transfer (fuzzy inert), consistent with -W/--no-delta. Documented in the
usage text (this deliberately differs from the basis-dir options, which still
force incremental unconditionally).
Adds ignore_errors (client-only) and force_delete (wire) booleans, makes
max_delete default -1 (no client limit), and parses --delete-excluded,
--max-delete=NUM, --ignore-errors, --force, --prune-empty-dirs. Bumps
PROTOCOL_VERSION 2.8.0 -> 2.9.0 for the new on-the-wire force_delete field.
Register --fuzzy (alias -y) as a boolean option and fuzzy as negatable so
--no-fuzzy works through the generic negation machinery. FastSync's delta
machinery is off by default (unlike rsync, where --fuzzy implies nothing
because delta is the default), so --fuzzy implies --incremental and --delta
unless --whole-file or an explicit --no-delta switched delta off (leaving
fuzzy inert, matching rsync where -W makes fuzzy irrelevant). Add help text.
Usage help now gives --delete-during a complete description with --del on its
own line, and notes that timing flags imply --delete while timing+--no-delete
is rejected regardless of argument order. RSYNC_COMPAT.md documents: the
receiver's MAX_MANIFEST_ENTRIES/MAX_MANIFEST_BYTES caps now abort an early-mode
run before any data (previously only the deletion step failed), the extended
early-delete ACK deadline, and the order-independent flag-conflict policy.
Deletion timing is now real and selected by the four rsync flags plus the
plain --delete default. Wire protocol bumps to 2.8.0: two new config
booleans (delete_during, delete_delay) are serialized and validated, joining
the existing delete_before/delete_after.
- Early modes (--delete-before, --delete-during/--del): the sender pre-scans
the whole tree (paths only), transmits the keep-set manifest BEFORE any
file data, and the receiver removes extras and acks STATUS_OK; the sender
only streams data after the deletion committed. Deletion is thus performed
even if a later transfer phase fails (rsync delete-before/during are
destructive by definition). FastSync streams in a single scan so it cannot
interleave per-directory like rsync delete-during; --delete-during selects
the same engine mode as --delete-before (documented divergence).
- Late/commit modes (plain --delete, --delete-after, --delete-delay): the
manifest closes the data stream and deletion is committed only after
STATUS_FINISHED proves the whole transfer succeeded, preserving FastSync's
commit-style safety. --delete-delay converges with --delete-after because
FastSync never snapshots the destination during data flow (documented).
- The STATUS_MANIFEST frame is now self-delimiting and position-independent.
Single-threaded receivers delete before the success frame; the -m receiver
hands the keep-set to server.c, which commits the deletion only after the
disk writer thread has drained (fixes a delete-vs-in-flight-temp race).
- Every timing flag implies --delete; at most one timing flag is allowed.
- Each timing flag implies --delete, matching rsync; conflicts are rejected.
Replace the vestigial single compare_dest/copy_dest/link_dest Config fields with
an ordered BasisDest list (type + path per entry) that is serialized to the
receiver and interpreted relative to the destination root. Paths must be
relative with no '.'/'..' components (confined like --backup-dir); trailing
slashes are normalized. Wire layout changes, so PROTOCOL_VERSION -> 2.8.0.
CLI: each flag is parsed in both --flag=DIR and --flag DIR forms, is
repeatable, and keeps command-line order as basis priority. Supplying any
basis dir implies --incremental (and therefore metadata) on the sender because
the unchanged decision is receiver-side; combining basis dirs with -s chunk
serialization is rejected in validate_config. Usage text updated.
RSYNC_COMPAT Phase-2 row M: path-list construction and destination
directory creation while preserving traversal safety.
- -R/--relative with --files-from: transmit each listed entry under its
bare relative destination path (no source-root mirror). Files keep an
absolute local read path plus a separate wire/dest path (File.send_path);
manifest, incremental quick-check and change output follow the wire path,
so --delete and --remove-source-files stay consistent. -R without
--files-from is unchanged (full mirror).
- --no-implied-dirs: client-only, only meaningful with -R + --files-from.
A listed file whose parent dir is not itself (or via an ancestor)
explicitly listed cannot be placed; the run fails up front with a clear
error. No effect otherwise.
- --dirs/-d + --old-dirs/--old-d aliases: -d <dir> transmits the source
root as an explicit empty directory entry (STATUS_MKDIR frame); with
--files-from listed dirs are created empty and listed files transferred,
never descending. Works single-threaded, -m (sequential scanner in the
-m scan thread) and chunk-serialization (per-file type marker).
Directory entries appear in the delete manifest.
- --mkpath: new wire bool; server creates the destination root (and missing
leading components under its authorized root) at connection start. A
missing destination root is now rejected by default.
- Protocol bumped to 2.7.0 (mkpath wire field + STATUS_MKDIR + chunk type
marker). All receive paths funnel through file_save_to_disk_full which
creates directories via the secure confined mkdir engine; dir entries are
excluded from --remove-source-files outcome acknowledgements on both ends.
- Unit coverage: CLI parse (relative/dirs aliases/mkpath/no-implied-dirs),
config round-trip (relative + mkpath), scanner --dirs non-recursion and
-R send_path (sequential + parallel), receiver dir-entry save.
- Integration coverage: TestRelativeFilesFrom, TestNoImpliedDirs, TestDirs,
TestMkpath (single and -m).
- RSYNC_COMPAT: 4 rows move to Implemented (Summary 67/3/5/1/71 = 147).
Implement the RSYNC_COMPAT Phase-2 filter/parser feature group:
- --files-from=FILE (repeatable) plus -0/--from0 NUL delimiters: parse the
source file list relative to the source root into a shared read-only
allow-set; the scanner transfers listed files and the whole subtree of
listed directories and prunes everything else in single- and
multithreaded mode. Absolute/'..' entries and missing files are hard
CLI errors.
- --filter=RULE: rsync-style +/- rules (anchored '/', dir-only trailing '/',
word include/exclude forms) evaluated first-match-wins with a default of
include, as an independent layer from legacy --exclude/--include.
Unsupported directives (merge/hide/... ) are rejected explicitly. -f stays
sendfile.
- -C/--cvs-exclude: well-known rsync CVS default exclude set.
- -F: per-directory .rsync-filter files read during traversal and applied to
the owning directory's subtree (single + parallel), never transferred.
- Delete manifest still derives from what was actually sent.
Client-only config fields; no wire/protocol change. Adds unit coverage
(CLI parse, allow-set and filter scanning single+parallel) and integration
tests (TestFilesFrom, TestFilters). RSYNC_COMPAT matrix rows updated:
5 rows move to Implemented (Summary 62/3/5/1/76 = 147).
Stage every successfully written file under a private 0700 .fastsync-stage
directory inside the receive root and atomically publish all staged files
only after the whole protocol stream (manifest/delete handling included)
has completed, immediately before the success/outcome frame. On any
abort/error before publication nothing is installed and staging is removed;
a publish failure aborts the transfer with best-effort cleanup of the
remainder (already-published files are not rolled back). Crash leftovers
are wiped when the next delayed transfer starts.
Wire: new delay_updates config flag (selection-options block), protocol
version bumped to 2.6.0, client/server validation rejects --inplace.
CLI/usage/validation updated. Works in single-threaded and -m modes
(exactly one write_thread stages files; the staged-file registry is
mutex-protected; publication runs once after both threads join).
--existing/--ignore-existing/--update decide against the final destination
at stage time; --backup is deferred to publication. remove_source_files
outcomes are only sent after publication so skipped/unpublished sources are
never deleted. Default (no flag) behavior is unchanged.
Tests: config wire round-trip, CLI parse, --inplace rejection, new
test_delay_updates unit suite (27 suites total), and integration
TestDelayUpdates covering single/-m parity, incremental reruns, remove
source files, receiver-skip ordering, and a deterministic publish-failure
abort path.
Receiver writes temps into a confined scratch dir and atomically renames
into place; EXDEV aborts; inplace/partial bypass; thread-safe temp names.
Uses existing wire field; no protocol bump. Reviewed (c-review APPROVE
WITH NITS, all fixed); PR #262.
Implement rsync-style itemized output backed by one shared change-event
engine (src/client/change_list.c):
- -i/--itemize-changes prints ">f+++++++++ <path>" for files actually sent
(single-threaded and -m); unchanged files print nothing.
- --list-only prints an ls-style listing of files that would be transferred
without contacting the server or writing anything.
- --out-format=FORMAT prints a printf-style template per changed file
(tokens %%f %%n %%l %%b %%M %%%%; unknown escapes preserved).
- --log-file-format=FMT logs each transferred file when --log-file is set.
Events are emitted from the per-file sender path shared by both transfer
modes, so the single sender thread is the only reporter (no races).
Capture the transfer root's device (st_dev) at scanner creation and skip
descending into any subdirectory on a different device (a mount point).
Implemented sender/client-side only: sequential BFS and parallel (-m) root
scan apply the same scanner_same_filesystem decision; no wire/protocol change
and default behavior is unchanged. Unit tests cover the pure decision, same
device scanning in both modes, and CLI parsing; integration tests prove -x
leaves a single-filesystem tree byte-identical and, when root can mount a
tmpfs, skips a genuine cross-device subtree.
TapTap