Capture+transmit source atime (pre-read stat; O_NOATIME sender guard) and birth
time (statx STATX_BTIME); receiver restores atime with mtime (crtime not settable
portably -> transmitted, explicitly not applied). --open-noatime is client-only.
-O/-J documented as accepted no-ops (FastSync never preserves dir/symlink times).
Wire: metadata frame gains atime/crtime val+sec+nsec; PROTOCOL_VERSION
2.11.0->2.12.0. Review fixes: gate atime capture to Linux (no epoch clobber on
non-Linux), close fd on fdopen failure, honest -O/-J status (Compat no-op).
Source files sharing (st_dev,st_ino) are recreated as hard links on the
destination; only the first member's data crosses the wire (siblings ride a
payload-less STATUS_HARDLINK frame). Ordering requires the single-FIFO-writer
receiver + forced sequential scan (documented). link()-failure falls back to a
byte-identical local copy. Rejects -s/--append. PROTOCOL_VERSION 2.11.0->2.12.0.
Review fixes: delete the dead HardLinkRegistry (ordering holds by FIFO writer),
and --existing no longer aborts when the first member is absent but the sibling
exists (leaves the sibling in place).
Receiver allocates the destination file's full size before streaming data
(posix_fallocate preferred, ftruncate fallback on EOPNOTSUPP/ENOSYS) so an
out-of-space transfer fails fast instead of partway. Additive config bool
crossing the wire; PROTOCOL_VERSION 2.10.0 -> 2.11.0. Threaded through all
store paths (atomic, inplace, partial/delay-updates staging, link-dest copy
fallback). Review hardening: explicit lseek(0) before the data write so
correctness does not depend on posix_fallocate leaving the fd offset unchanged.
Receiver-side ownership application, opt-in and privilege-gated:
- OFF for every existing transfer (plain -M/--preserve still never applies
ownership); only triggers on an explicit identity flag + receiver permission.
- EPERM/EACCES warn-and-continue (never aborts); other fchown errors escalate.
- fd-relative fchown after the file is written (symlink-safe, confined).
- New src/shared/identity.{c,h}; config fields numeric_ids / chown uid/gid /
usermap + groupmap id-pair tables cross the wire; PROTOCOL_VERSION 2.10.0
-> 2.11.0. CLI in client_cli.c; per-connection snapshot in server.c.
- Review fixes: EPERM/EACCES-only warn-and-continue, prominent root-receiver
notice, identity_clear_active on early server error paths, --numeric-ids
kept inert standalone (removed from activation trigger set).
Adds real algorithm selection (xxh64 default, plus md5 via OpenSSL EVP) and a
64-bit seed for the per-file whole-file digest used by the --incremental/
--checksum handshake and basis-dir content verification. The seed also feeds
the delta path's per-block xxHash32 strong checksum (low 32 bits) so an
explicit seed deterministically changes those digests too. Sender and receiver
hash identically: the algorithm id and seed cross the config wire frame and the
STATUS_CHECK handshake now carries a length-prefixed, bounded digest instead of
a fixed 64-bit value. Unsupported algorithm names are rejected at parse time
(never a silent no-op). PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0; defaults
(xxh64, seed 0) preserve prior byte-for-byte behavior.
Implement rsync's append modes: when an existing destination file is SHORTER
than the source, the receiver negotiates a resume offset and only the tail is
transferred; the full file (retained prefix + tail) is rebuilt and installed
through the normal atomic store path, so the result is byte-identical to the
source whenever the prefix matches.
- --append: sends the tail without content-verifying the retained prefix
(rsync parity; the documented prefix-trust risk).
- --append-verify: verifies the retained prefix against the source's prefix
xxHash64 before appending and, on a mismatch, falls back to a clean full
transfer (never a corrupt prefix+tail blend).
New wire frames STATUS_APPEND / STATUS_APPEND_SIG / STATUS_APPEND_OK /
STATUS_APPEND_DATA; PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0 (peers must match).
Both flags imply --incremental and are incompatible with -s (chunk
serialization) and --whole-file (rejected up front). Respects --inplace,
--partial/--partial-dir and --delay-updates via the shared store engine.
New flags parse onto the config; --delete-missing-args implies
--ignore-missing-args (order-independent) and does NOT imply --delete (rsync:
independent of other delete processing). The files-from preflight now classifies
listed-but-missing entries instead of hard-failing: under the flags each is
skipped (logged + counted, never silent) and the run succeeds for the rest,
including the all-missing case; an empty list stays a hard error. Under
--delete-missing-args the missing entries' destination mirrors (bare relative
path with -R, full source mirror otherwise) ride the manifest's third section in
both the single-threaded and -m senders; --dirs listed-but-missing entries are
skipped in the scanner.
The --fuzzy implication previously forced use_incremental back on even when
the user passed --no-incremental, while --no-delta and -W were honored.
Track a --no-incremental latch (like the no_delta latch): with it set, do not
force the handshake on, and because delta needs the handshake, also suppress
the delta implication so no invalid '--delta requires --incremental' config
results. A --fuzzy --no-incremental run is therefore a plain default-mode
transfer (fuzzy inert), consistent with -W/--no-delta. Documented in the
usage text (this deliberately differs from the basis-dir options, which still
force incremental unconditionally).
Adds ignore_errors (client-only) and force_delete (wire) booleans, makes
max_delete default -1 (no client limit), and parses --delete-excluded,
--max-delete=NUM, --ignore-errors, --force, --prune-empty-dirs. Bumps
PROTOCOL_VERSION 2.8.0 -> 2.9.0 for the new on-the-wire force_delete field.
Register --fuzzy (alias -y) as a boolean option and fuzzy as negatable so
--no-fuzzy works through the generic negation machinery. FastSync's delta
machinery is off by default (unlike rsync, where --fuzzy implies nothing
because delta is the default), so --fuzzy implies --incremental and --delta
unless --whole-file or an explicit --no-delta switched delta off (leaving
fuzzy inert, matching rsync where -W makes fuzzy irrelevant). Add help text.
Deletion timing is now real and selected by the four rsync flags plus the
plain --delete default. Wire protocol bumps to 2.8.0: two new config
booleans (delete_during, delete_delay) are serialized and validated, joining
the existing delete_before/delete_after.
- Early modes (--delete-before, --delete-during/--del): the sender pre-scans
the whole tree (paths only), transmits the keep-set manifest BEFORE any
file data, and the receiver removes extras and acks STATUS_OK; the sender
only streams data after the deletion committed. Deletion is thus performed
even if a later transfer phase fails (rsync delete-before/during are
destructive by definition). FastSync streams in a single scan so it cannot
interleave per-directory like rsync delete-during; --delete-during selects
the same engine mode as --delete-before (documented divergence).
- Late/commit modes (plain --delete, --delete-after, --delete-delay): the
manifest closes the data stream and deletion is committed only after
STATUS_FINISHED proves the whole transfer succeeded, preserving FastSync's
commit-style safety. --delete-delay converges with --delete-after because
FastSync never snapshots the destination during data flow (documented).
- The STATUS_MANIFEST frame is now self-delimiting and position-independent.
Single-threaded receivers delete before the success frame; the -m receiver
hands the keep-set to server.c, which commits the deletion only after the
disk writer thread has drained (fixes a delete-vs-in-flight-temp race).
- Every timing flag implies --delete; at most one timing flag is allowed.
- Each timing flag implies --delete, matching rsync; conflicts are rejected.
Replace the vestigial single compare_dest/copy_dest/link_dest Config fields with
an ordered BasisDest list (type + path per entry) that is serialized to the
receiver and interpreted relative to the destination root. Paths must be
relative with no '.'/'..' components (confined like --backup-dir); trailing
slashes are normalized. Wire layout changes, so PROTOCOL_VERSION -> 2.8.0.
CLI: each flag is parsed in both --flag=DIR and --flag DIR forms, is
repeatable, and keeps command-line order as basis priority. Supplying any
basis dir implies --incremental (and therefore metadata) on the sender because
the unchanged decision is receiver-side; combining basis dirs with -s chunk
serialization is rejected in validate_config. Usage text updated.
B1: destination-root existence/creation mishandled two legitimate forms.
file_directory_exists_secure/file_ensure_directory_secure now normalize a
trailing-slash destination (so the last component is never an empty leaf)
and treat a destination equal to the already-open authorized root as present
(no spurious <root>/<basename> nested dir with --mkpath). Confinement and
O_NOFOLLOW probing are unchanged. Regression integration tests: existing
dest with trailing slash works with and without --mkpath; dest == authorized
root works and creates no stray nested dir.
W1: chunk serialize/deserialize round-trip unit test for a directory entry
mixed with a regular file, with and without metadata; --dirs coverage under
-s and -s -m.
W2: --list-only and --dry-run now print file_wire_path() so all outputs show
the -R transformed relative name, matching -i/--out-format.
W3: RSYNC_COMPAT -d/--dirs note: dirs are created immediately under
--delay-updates (only regular files are staged).
W4: --dirs generator flushes chunks by element count too, so a long
--files-from list of empty directories cannot exceed the per-chunk file cap.
N1: STATUS_MKDIR added to status_to_string.
N2: removed dead TestMkpath._transfer.
N3: removed redundant --no-implied-dirs OPTION_TABLE row.
N4: integration tests: --dirs --delete keeps the just-created empty dir (and
deletes extras); a listed dir colliding with a regular file at the dest fails
cleanly.
RSYNC_COMPAT Phase-2 row M: path-list construction and destination
directory creation while preserving traversal safety.
- -R/--relative with --files-from: transmit each listed entry under its
bare relative destination path (no source-root mirror). Files keep an
absolute local read path plus a separate wire/dest path (File.send_path);
manifest, incremental quick-check and change output follow the wire path,
so --delete and --remove-source-files stay consistent. -R without
--files-from is unchanged (full mirror).
- --no-implied-dirs: client-only, only meaningful with -R + --files-from.
A listed file whose parent dir is not itself (or via an ancestor)
explicitly listed cannot be placed; the run fails up front with a clear
error. No effect otherwise.
- --dirs/-d + --old-dirs/--old-d aliases: -d <dir> transmits the source
root as an explicit empty directory entry (STATUS_MKDIR frame); with
--files-from listed dirs are created empty and listed files transferred,
never descending. Works single-threaded, -m (sequential scanner in the
-m scan thread) and chunk-serialization (per-file type marker).
Directory entries appear in the delete manifest.
- --mkpath: new wire bool; server creates the destination root (and missing
leading components under its authorized root) at connection start. A
missing destination root is now rejected by default.
- Protocol bumped to 2.7.0 (mkpath wire field + STATUS_MKDIR + chunk type
marker). All receive paths funnel through file_save_to_disk_full which
creates directories via the secure confined mkdir engine; dir entries are
excluded from --remove-source-files outcome acknowledgements on both ends.
- Unit coverage: CLI parse (relative/dirs aliases/mkpath/no-implied-dirs),
config round-trip (relative + mkpath), scanner --dirs non-recursion and
-R send_path (sequential + parallel), receiver dir-entry save.
- Integration coverage: TestRelativeFilesFrom, TestNoImpliedDirs, TestDirs,
TestMkpath (single and -m).
- RSYNC_COMPAT: 4 rows move to Implemented (Summary 67/3/5/1/71 = 147).
Address an independent c-review of the files-from/filter feature:
- .rsync-filter precedence now matches rsync: evaluate the innermost
(current) directory's rules first, then ancestors, then the command-line
base (--filter/-C), so a deeper file's '+' can re-include what a shallower
'-' excluded (regression tests in both scan modes; single-thread and -m).
- --files-from: a listed entry missing on disk and an empty list are now hard
errors surfaced pre-transfer in send_files, send_files_multithreaded,
dry-run and --list-only; '.' (whole tree) and empty listed dirs stay valid.
- scanner_path_relative now handles a transfer root of / (previously the
scanner aborted on children of /).
- Reject unsupported rsync filter syntax explicitly (no silent no-ops):
+/- modifiers other than '/' (! C s r p x) and rules beginning with ':'/'.'
/'!' (merge/dir-merge/list-clear shorthands). Docs updated.
- -0/--from0 NUL mode preserves entry bytes (no CR/LF trimming); only newline
mode trims. Absolute-entry error message no longer includes the newline.
- --no-from0/--no-cvs-exclude registered as negatable booleans.
- RSYNC_COMPAT rows updated for the precedence, rejection list, NUL-mode
detail and the documented O(entries x files) scalability bound of the
allow-set (Summary unchanged: 62/3/5/1/76 = 147).
Implement the RSYNC_COMPAT Phase-2 filter/parser feature group:
- --files-from=FILE (repeatable) plus -0/--from0 NUL delimiters: parse the
source file list relative to the source root into a shared read-only
allow-set; the scanner transfers listed files and the whole subtree of
listed directories and prunes everything else in single- and
multithreaded mode. Absolute/'..' entries and missing files are hard
CLI errors.
- --filter=RULE: rsync-style +/- rules (anchored '/', dir-only trailing '/',
word include/exclude forms) evaluated first-match-wins with a default of
include, as an independent layer from legacy --exclude/--include.
Unsupported directives (merge/hide/... ) are rejected explicitly. -f stays
sendfile.
- -C/--cvs-exclude: well-known rsync CVS default exclude set.
- -F: per-directory .rsync-filter files read during traversal and applied to
the owning directory's subtree (single + parallel), never transferred.
- Delete manifest still derives from what was actually sent.
Client-only config fields; no wire/protocol change. Adds unit coverage
(CLI parse, allow-set and filter scanning single+parallel) and integration
tests (TestFilesFrom, TestFilters). RSYNC_COMPAT matrix rows updated:
5 rows move to Implemented (Summary 62/3/5/1/76 = 147).
Stage every successfully written file under a private 0700 .fastsync-stage
directory inside the receive root and atomically publish all staged files
only after the whole protocol stream (manifest/delete handling included)
has completed, immediately before the success/outcome frame. On any
abort/error before publication nothing is installed and staging is removed;
a publish failure aborts the transfer with best-effort cleanup of the
remainder (already-published files are not rolled back). Crash leftovers
are wiped when the next delayed transfer starts.
Wire: new delay_updates config flag (selection-options block), protocol
version bumped to 2.6.0, client/server validation rejects --inplace.
CLI/usage/validation updated. Works in single-threaded and -m modes
(exactly one write_thread stages files; the staged-file registry is
mutex-protected; publication runs once after both threads join).
--existing/--ignore-existing/--update decide against the final destination
at stage time; --backup is deferred to publication. remove_source_files
outcomes are only sent after publication so skipped/unpublished sources are
never deleted. Default (no flag) behavior is unchanged.
Tests: config wire round-trip, CLI parse, --inplace rejection, new
test_delay_updates unit suite (27 suites total), and integration
TestDelayUpdates covering single/-m parity, incremental reruns, remove
source files, receiver-skip ordering, and a deterministic publish-failure
abort path.
Receiver writes temps into a confined scratch dir and atomically renames
into place; EXDEV aborts; inplace/partial bypass; thread-safe temp names.
Uses existing wire field; no protocol bump. Reviewed (c-review APPROVE
WITH NITS, all fixed); PR #262.
Implement rsync-style itemized output backed by one shared change-event
engine (src/client/change_list.c):
- -i/--itemize-changes prints ">f+++++++++ <path>" for files actually sent
(single-threaded and -m); unchanged files print nothing.
- --list-only prints an ls-style listing of files that would be transferred
without contacting the server or writing anything.
- --out-format=FORMAT prints a printf-style template per changed file
(tokens %%f %%n %%l %%b %%M %%%%; unknown escapes preserved).
- --log-file-format=FMT logs each transferred file when --log-file is set.
Events are emitted from the per-file sender path shared by both transfer
modes, so the single sender thread is the only reporter (no races).
Capture the transfer root's device (st_dev) at scanner creation and skip
descending into any subdirectory on a different device (a mount point).
Implemented sender/client-side only: sequential BFS and parallel (-m) root
scan apply the same scanner_same_filesystem decision; no wire/protocol change
and default behavior is unchanged. Unit tests cover the pure decision, same
device scanning in both modes, and CLI parsing; integration tests prove -x
leaves a single-filesystem tree byte-identical and, when root can mount a
tmpfs, skips a genuine cross-device subtree.
- Correct misleading doc comments on set_string_option /
set_positive_int_option / set_nonneg_int_option (they return 0/-1,
not true/false).
- find_table_option_with_equals() now matches every OPTION_TABLE value
option (OPT_STRING/OPT_POS_INT/OPT_NONNEG_INT/OPT_ULL), so forms such
as --max-size=2G, --min-size=1K, --suffix=.bak, --timeout=30,
--max-depth=5, --backup-dir=X parse instead of dying as 'Unknown option'.
--max-size/--min-size now accept rsync-style binary suffixes (0 remains a
valid 'no limit' byte count). Existing special handling for
--compress-choice, --compress-level, --modify-window=, --chmod=,
--skip-compress=, --compress-threads= is preserved.
- Options that require a separate value (-p, --exclude, --include,
--delta-block, --delta-max, --server-port, --bwlimit, --chunk-size,
--log-file, --exclude-from, --include-from, -T, --skip-compress,
--compress-threads) now emit an explicit 'missing argument' diagnostic
instead of falling through to the generic 'Unknown option' branch when
given as the final argv entry.
- Add unit tests covering the = forms (--max-size=2G, --min-size=1K,
--suffix=.bak, --timeout=30, --max-depth=5, --backup-dir=X) and a clean
'missing argument' (not 'Unknown option') diagnostic for trailing
--exclude/--server-port/--skip-compress/-T.
#251 --remove-source-files deletes sources that were skipped receiver-side
(--existing/--ignore-existing/--update). The receiver now reports a
per-file outcome for every processed data file when the sender requests
removal; the client only unlinks sources the receiver actually wrote.
add remove_source_files to the wire config and bump the protocol to 2.5.0.
#252 --backup/--suffix/--backup-dir broken by NULL-vs-empty wire loss. Receivers
canonicalize the empty wire string back to NULL for backup_dir, temp_dir,
partial_dir and suffix, and --suffix is received unconditionally.
#253 --partial --partial-dir never installed completed files. file_save_to_disk
now renames a fully written partial-dir file into the real destination.
#255 STATUS_CHECK read the entire old file before the size/mtime quick check.
Old contents are only read when a checksum compare or delta needs them.
#256 receive_delta_file failure paths did not set *failed, so the caller sent
STATUS_NEXT and waited for a body that never came. Every NULL return now
marks the transfer failed.
#257 files >64 MiB could not transfer. Whole-file receive caps raised to the
256 MiB connection/allocation ceiling (chunk caps stay 64 MiB) and the
client ignores SIGPIPE so a server-side close surfaces as a clean error.
Unit tests added: config NULL-vs-empty round trip, incremental quick-check
skip/NEXT paths, delta oversize failure, partial-dir install, save-result
skip reporting.
TapTap