Wave D of Phase 7. Make -O/--omit-dir-times and -J/--omit-link-times real by
preserving directory and symlink times, and mark --secluded-args as an explicit
Impossible/Divergence no-op.
Wire: PROTOCOL_VERSION 2.16.0 -> 2.17.0. Adds a terminal STATUS_DIR_TIMES frame
(int count + (wire path, metadata) pairs) sent after all file data and the
optional delete manifest. STATUS_MKDIR also carries metadata for --dirs entries.
Config-frame layout is unchanged.
Sender: the recursive scanner captures every traversed source directory (both
DirectoryScanner and the parallel scanner root + workers, appends mutex-guarded)
into a shared list; the single-threaded and -m paths transmit it last.
Receiver: a DirTimeList accumulates received directory metadata and applies it
with fd-relative no-follow utimensat only at the very end -- after all children,
after the commit-style --delete, and after --delay-updates publication -- in the
single-threaded success frame and in server.c after the -m threads join. -O skips
the application. Symlink metadata is applied at link creation with
utimensat/fchownat/fchmodat AT_SYMLINK_NOFOLLOW; -J suppresses only link times.
identity_apply_ownership_link shares the identity resolver with the fd path.
Docs: -O/-J rows -> Implemented; --secluded-args -> Impossible/Divergence;
--protocol accepted/rejected values and Phase-6/7 notes updated.
Tests: unit (scanner dir capture, DirTimeList apply, symlink metadata, protocol
version values) and integration (dir mtime round-trip + -O, symlink mtime
round-trip + -J, independent suppression), parameterized over single/multithread.
- test_credentials.c: NUL-terminate the overlong-line stack buffer before
make_tmp_file's strlen() (was a stack-buffer-overflow READ under ASan);
still exercises the overlong-rejection path.
- Add redacted protocol string variants (protocol_send_str_redacted /
receive + fd send_str_redacted/receive_str_redacted) and use them for the
daemon auth username/digest so --verbose / LOG_DEBUG_ALL never logs a
replayable credential while other protocol strings keep their debug trace.
- credentials_verify/gate: replace byte-wise-short-circuiting strcmp with a
fixed-length constant-time username compare (closes user-enumeration oracle);
update doc comment to match.
- read_secret_file: preserve password exact bytes (only strip trailing CR/LF)
and burn the stack line buffer; document the whitespace behavior.
- test_server_cli.c: note the parser zero-inits opts on failure.
- Add debug-level daemon test asserting the digest never appears under --verbose.
PROTOCOL_VERSION stays 2.15.0.
Source files sharing (st_dev,st_ino) are recreated as hard links on the
destination; only the first member's data crosses the wire (siblings ride a
payload-less STATUS_HARDLINK frame). Ordering requires the single-FIFO-writer
receiver + forced sequential scan (documented). link()-failure falls back to a
byte-identical local copy. Rejects -s/--append. PROTOCOL_VERSION 2.11.0->2.12.0.
Review fixes: delete the dead HardLinkRegistry (ordering holds by FIFO writer),
and --existing no longer aborts when the first member is absent but the sibling
exists (leaves the sibling in place).
Implement rsync's append modes: when an existing destination file is SHORTER
than the source, the receiver negotiates a resume offset and only the tail is
transferred; the full file (retained prefix + tail) is rebuilt and installed
through the normal atomic store path, so the result is byte-identical to the
source whenever the prefix matches.
- --append: sends the tail without content-verifying the retained prefix
(rsync parity; the documented prefix-trust risk).
- --append-verify: verifies the retained prefix against the source's prefix
xxHash64 before appending and, on a mismatch, falls back to a clean full
transfer (never a corrupt prefix+tail blend).
New wire frames STATUS_APPEND / STATUS_APPEND_SIG / STATUS_APPEND_OK /
STATUS_APPEND_DATA; PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0 (peers must match).
Both flags imply --incremental and are incompatible with -s (chunk
serialization) and --whole-file (rejected up front). Respects --inplace,
--partial/--partial-dir and --delay-updates via the shared store engine.
The receiver performs the whole bounded deletion walk (up to
MAX_SERVER_DELETE_COUNT unlinks) before answering the delete-before/during
manifest, so its STATUS_OK reply can take far longer than the default 60 s
per-message receive window. Waiting with the default would make the sender
abort AFTER the deletion had already committed on the receiver. Add a timed
receive variant (receive_status_timed / protocol_receive_n_data_timed) and use
it for the early-manifest ACK with a 1 h explicit deadline; connection errors
and EOF still abort immediately.
B1: destination-root existence/creation mishandled two legitimate forms.
file_directory_exists_secure/file_ensure_directory_secure now normalize a
trailing-slash destination (so the last component is never an empty leaf)
and treat a destination equal to the already-open authorized root as present
(no spurious <root>/<basename> nested dir with --mkpath). Confinement and
O_NOFOLLOW probing are unchanged. Regression integration tests: existing
dest with trailing slash works with and without --mkpath; dest == authorized
root works and creates no stray nested dir.
W1: chunk serialize/deserialize round-trip unit test for a directory entry
mixed with a regular file, with and without metadata; --dirs coverage under
-s and -s -m.
W2: --list-only and --dry-run now print file_wire_path() so all outputs show
the -R transformed relative name, matching -i/--out-format.
W3: RSYNC_COMPAT -d/--dirs note: dirs are created immediately under
--delay-updates (only regular files are staged).
W4: --dirs generator flushes chunks by element count too, so a long
--files-from list of empty directories cannot exceed the per-chunk file cap.
N1: STATUS_MKDIR added to status_to_string.
N2: removed dead TestMkpath._transfer.
N3: removed redundant --no-implied-dirs OPTION_TABLE row.
N4: integration tests: --dirs --delete keeps the just-created empty dir (and
deletes extras); a listed dir colliding with a regular file at the dest fails
cleanly.
The per-connection memory budget (MAX_CONNECTION_MEMORY, 256 MiB) only
charged wire buffers via receive_data_limited. Decompression buffers and
per-file chunk copies were not accounted for, and the multithreaded
receiver could enqueue up to 100 files (each up to 64 MiB uncompressed)
ahead of a slow disk writer, retaining ~6.4 GiB per connection. A client
sending highly compressible chunks with little bandwidth could OOM the
host while the reserve never tripped.
Bound the receive pipeline by aggregate payload bytes instead of item
count alone:
- Export MAX_CONNECTION_MEMORY from protocol.h.
- PipelineContextReceiver tracks queued_bytes (payload bytes received but
not yet released by the disk writer, i.e. queued or in the writer's
hand) under the existing mutex.
- receiver enqueue now blocks while the queue is full by count OR when
adding the file would push queued_bytes over the configured byte limit,
applying backpressure to the sender instead of failing the transfer.
- The disk writer releases the byte budget after each file is freed and
signals the not-full condition.
- The server sets the byte ceiling to
MAX_CONNECTION_MEMORY - 2*MAX_CHUNK_SIZE so that the queued payloads
plus the transient wire/decompression buffers of the one in-flight
chunk stay within the per-connection budget.
The single-threaded receive path is already bounded: it writes files to
disk before reading the next chunk, so its transient is at most one
chunk's wire + decompressed + copied payload (~3 * MAX_CHUNK_SIZE, below
the budget). Wire buffers remain charged exactly once by
receive_data_limited; this change does not double charge them.
Adds a deterministic unit test in test_multiprocessing.c proving that an
enqueue which would exceed the byte budget blocks until the writer
releases bytes.
The Phase 1 merge conflict resolutions introduced formatting that failed
the CI lint job (clang-format 18.1.3). Reformatted with the exact CI
version; no functional changes.
- Fix memory leak in delta_deserialize() on BLOCK_MATCH error path
- Fix memory leak on malloc failure for literal data
- Add port range validation (1-65535) for -p/--port and --server-port
- Restore -V/--version flag with usage text
- Use MAX_DATA_PAYLOAD_SIZE consistently (remove local MAX_DATA_SIZE)
- Fix integer truncation in config_send/config_receive for delta_block_size
- Add log messages for malloc failures
- Extract config_set_defaults() helper to eliminate duplication
- Add 10 new CLI tests for argument parsing
- Restore PROTOCOL_VERSION to a forward-compatible 2.1.0 and document wire format
- Add NULL guard to config_delete
- Add pipeline cancellation flag and cancellation-aware queue helper
- Join running threads before destroying pipeline contexts on creation failure
- Fix NULL dereference and memory leaks in manifest/chunk handling
- Fix TLS/TCP socket fd leak on connect error paths
- Add compression-level range validation (1-22)
- Close previous log file before opening a new one
- Use getline for unbounded pattern-file lines
- Fix thread-unsafe localtime() and add log level bounds check
- Fix file_load_data to clean up data on read size mismatch
- Add hard ceiling to decompression buffer growth
- Fix mkdir_r bounds check and restore glob comments
- Add send_str NULL guard and mutex-protect bandwidth limiter
- Update AGENTS.md for per-thread io_ssl contract
- send_n_data: void* data → const void* data
- send_str: char* data → const char* data
- send_data: Data* data → const Data* data
- Remove duplicate typedef struct ssl_st SSL in protocol.h
- Add NULL check after malloc in directory_scanner_create
- Reformat all C/H files to match .clang-format (LLVM style)
- Fix 26 cppcheck const-correctness warnings (constParameterPointer,
constVariablePointer, constVariable)
- Update function declarations in headers to match const parameters
Add Tier 1 review automation to catch issues before human review:
- Add cppcheck and clang-format to CI lint job
- Add sanitizer matrix (ASan + TSan) CI job
- Enable -Wextra -Wpedantic -Werror in CI build
- Add SANITIZER and STRICT_WARNINGS CMake options
- Add .clang-format for consistent code style
- Update Dockerfile with cppcheck and clang-format
- Fix sign-compare and unused-parameter warnings for -Werror
TapTap