diff --git a/RSYNC_COMPAT.md b/RSYNC_COMPAT.md index ed3f622..5a6b5b8 100644 --- a/RSYNC_COMPAT.md +++ b/RSYNC_COMPAT.md @@ -123,10 +123,10 @@ Every one of those has an entry below with its remaining caveats. |------|-------------------|-----------------|-------| | `--stats` | Give transfer stats | ⚠️ Caveat | Prints transfer statistics. Protocol 2.25.0 populates the receiver-only counters the sender cannot observe (`Matched data`, `Number of deleted files`) from the receiver's `STATUS_STATS` report; the sender tracks the scanned file list per type so `Number of files` carries rsync's `(reg: X, dir: Y, link: Z, special: W)` breakdown (directories come from the scanner's captured directory list for `-a`/`-t`/`-p`, or from a lightweight traversed-directory counter on a plain `-r` run so the `dir:` category is present there too), `Number of regular files transferred` excludes symlinks/specials and up-to-date files, `Total file size` includes symlink target lengths, and `Total transferred file size` counts only transferred files. **Protocol 2.28.0 extends `STATUS_STATS`** with receiver-observed `literal_bytes` and the four `created_*` counters: `Number of created files` now carries rsync's `(reg/dir/link/special)` breakdown (the receiver reports which destination entries it newly created, including implicitly-created parent directories below the transfer root) and `Literal data` is exact for a delta transfer (the receiver counts the literal fragments it stored, not the whole source size) — all differential-tested in the sequential and `--threads` paths against rsync 3.4.1 for fresh-create, update and delta shapes. **Remaining divergences:** rsync's per-type breakdown on `Number of deleted files` is not reproduced; and `Total bytes sent`/`received` are FastSync wire bytes framed differently from rsync's, so they are not numerically comparable | | `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units. **A lone `-h` with no transfer arguments prints help instead** (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer remains human-readable | -| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an incremental re-run itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`) | +| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision; **every** non-root directory is rendered as created (`cd+++++++++`) because the sender never probes a directory's destination state, so a pre-existing destination directory that rsync reports as unchanged (`.d..t......`) is still itemized as created — this is not limited to re-runs; an incremental re-run additionally itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`); and directory attribute columns (`%M`/`%U`/`%G`) come from the source | | `--progress` | Show progress | ⚠️ Caveat | Protocol 2.25.0 prints rsync-style per-file progress blocks (percent, transferred/total bytes, rate, elapsed, `(xfr#N, to-chk=M/T)`) fed by the receiver's `STATUS_STATS`, in both the sequential and `--threads` send paths. FastSync also prints rsync's leading `./` transfer-root line and, when progress is requested (`--progress`/`-P`/`--info=progress`) and not `--quiet`, runs a **paths-only metadata pre-scan** (no file reads, no hashing) that supplies rsync's file-list total `T` for the `to-chk` denominator and the directory names; `--delete-during`/`--delete-delay` reuse their existing keep-set pre-scan instead of walking twice, and non-progress runs are untouched. Per-directory name lines are emitted (trailing `/`), and symlink (` -> target`) and special entries are named too, so a **fresh multi-directory tree's name set and `to-chk` denominator match rsync 3.4.1** (differential test, sequential and `--threads`) and a **single-file transfer's name lines and deterministic frames remain byte-identical** to rsync. **Order parity (parity-2.29):** the sequential scanner now emits entries in rsync's sorted depth-first flist order (non-directories ascending, then directories ascending), so the interleaving and the `to-chk` numerator match rsync for the default single-threaded transfer (differential `test_parity_order.py`; `--threads` has no rsync analogue and stays unordered). **Remaining divergences:** the leading `./` root line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an ancestor directory line is emitted whenever a child transfers (rsync suppresses it when the directory itself is unchanged); on a re-run, entries without a quick-check (symlinks, empty directories) are still named where rsync stays silent; and the rate/ETA are wall-clock dependent | | `-P` | Same as --partial --progress | ✅ Parity | Parses to `--partial` + `--progress`. The independent `--partial` retention semantics are rsync parity: an interrupted write retains the already-written temp at the destination (best-effort) so a later `--append`/`--append-verify` can resume. Progress presentation is owned by the `--progress` row; there is no separate `-P` divergence | -| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally and an incremental re-run may itemize directories/symlinks without a quick-check | +| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally, every non-root directory renders as created because the sender does not probe directory destination state (so a pre-existing unchanged directory still shows `cd+++++++++`), and directory attribute columns (`%M`/`%U`/`%G`) come from the source | | `--log-file=FILE` | Log to file | ✅ Parity | `log_file` config field | | `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` as the wire byte count) | | `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Parity | Applies to displayed paths and protocol debug output | diff --git a/src/client/client_report.c b/src/client/client_report.c index cb787c4..ff45260 100644 --- a/src/client/client_report.c +++ b/src/client/client_report.c @@ -597,7 +597,10 @@ void client_progress_begin(const Config* config) { } return; } - if (g_progress_active) + /* -i/--out-format alone do not print the header, but --progress always does + and --info=flist does under any output mode (rsync prints it for + `-i --info=flist` and `--out-format=... --info=flist` too). */ + if (g_progress_active || (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST))) printf("sending incremental file list\n"); /* rsync prints the transfer-root directory before the first entry. Under -i/--out-format it is the root change line (`.d..t...... ./`); otherwise it @@ -677,6 +680,125 @@ static bool progress_precount_add_dir(ProgressPrecount* p, const char* path) { return false; } +static int progress_path_compare(const void* left, const void* right) { + const char* const* a = (const char* const*)left; + const char* const* b = (const char* const*)right; + return strcmp(*a, *b); +} + +/* Sort the collected directory paths and drop duplicates so a large + * --files-from list (many entries sharing an implied ancestor) cannot grow the + * list without bound. */ +static void progress_precount_dedup_dirs(ArrayList* dir_paths) { + if (dir_paths == NULL || dir_paths->size < 2) + return; + qsort(dir_paths->items, (size_t)dir_paths->size, sizeof(char*), progress_path_compare); + int write = 0; + for (int read = 0; read < dir_paths->size; read++) { + char* current = (char*)dir_paths->items[read]; + if (write > 0 && strcmp((char*)dir_paths->items[write - 1], current) == 0) { + free(current); + continue; + } + dir_paths->items[write++] = current; + } + dir_paths->size = write; +} + +/* Create a metadata-bearing directory File for the transfer-relative directory + * `rel` ("" is the transfer root), stat'ing it below config->send_directory. + * The -d/--files-from dirs generator never traverses directories, so this + * synthesizes the metadata the recursive scanner captures through + * scanner_capture_dir_time, letting -i/--out-format render %M/%B/%U/%G and the + * transfer-root/ancestor lines identically on both paths. Returns NULL when + * the path cannot be stat'd as a directory or on allocation failure (the line + * is then simply omitted, exactly as before). */ +static File* progress_precount_make_dir(const Config* config, const char* rel) { + if (config == NULL || config->send_directory == NULL) + return NULL; + char* fs_path = (rel == NULL || rel[0] == '\0') ? str_dup(config->send_directory) + : path_cat(config->send_directory, rel); + if (fs_path == NULL) + return NULL; + struct stat st; + if (stat(fs_path, &st) != 0 || !S_ISDIR(st.st_mode)) { + free(fs_path); + return NULL; + } + File* file = file_create(fs_path); + if (file == NULL) { + free(fs_path); + return NULL; + } + file->is_dir = true; + file->metadata = + file_metadata_create(fs_path, &st, config->preserve_atimes, config->preserve_crtimes); + file->send_path = str_dup(rel != NULL ? rel : ""); + free(fs_path); + if (file->metadata == NULL || file->send_path == NULL) { + file_destroy(file); + return NULL; + } + return file; +} + +/* The -d/--files-from dirs generator neither traverses nor records directories, + * so its metadata walk captures no Files. Synthesize the transfer root and + * every listed/implied directory from `entry_rels` so -i/--out-format emits the + * same root and ancestor lines the recursive scan does. `root_emitted` is true + * when the generator itself emits the root entry (bare `-d