diff --git a/RSYNC_COMPAT.md b/RSYNC_COMPAT.md index ed3f622..5a6b5b8 100644 --- a/RSYNC_COMPAT.md +++ b/RSYNC_COMPAT.md @@ -123,10 +123,10 @@ Every one of those has an entry below with its remaining caveats. |------|-------------------|-----------------|-------| | `--stats` | Give transfer stats | ⚠️ Caveat | Prints transfer statistics. Protocol 2.25.0 populates the receiver-only counters the sender cannot observe (`Matched data`, `Number of deleted files`) from the receiver's `STATUS_STATS` report; the sender tracks the scanned file list per type so `Number of files` carries rsync's `(reg: X, dir: Y, link: Z, special: W)` breakdown (directories come from the scanner's captured directory list for `-a`/`-t`/`-p`, or from a lightweight traversed-directory counter on a plain `-r` run so the `dir:` category is present there too), `Number of regular files transferred` excludes symlinks/specials and up-to-date files, `Total file size` includes symlink target lengths, and `Total transferred file size` counts only transferred files. **Protocol 2.28.0 extends `STATUS_STATS`** with receiver-observed `literal_bytes` and the four `created_*` counters: `Number of created files` now carries rsync's `(reg/dir/link/special)` breakdown (the receiver reports which destination entries it newly created, including implicitly-created parent directories below the transfer root) and `Literal data` is exact for a delta transfer (the receiver counts the literal fragments it stored, not the whole source size) — all differential-tested in the sequential and `--threads` paths against rsync 3.4.1 for fresh-create, update and delta shapes. **Remaining divergences:** rsync's per-type breakdown on `Number of deleted files` is not reproduced; and `Total bytes sent`/`received` are FastSync wire bytes framed differently from rsync's, so they are not numerically comparable | | `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units. **A lone `-h` with no transfer arguments prints help instead** (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer remains human-readable | -| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an incremental re-run itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`) | +| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision; **every** non-root directory is rendered as created (`cd+++++++++`) because the sender never probes a directory's destination state, so a pre-existing destination directory that rsync reports as unchanged (`.d..t......`) is still itemized as created — this is not limited to re-runs; an incremental re-run additionally itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`); and directory attribute columns (`%M`/`%U`/`%G`) come from the source | | `--progress` | Show progress | ⚠️ Caveat | Protocol 2.25.0 prints rsync-style per-file progress blocks (percent, transferred/total bytes, rate, elapsed, `(xfr#N, to-chk=M/T)`) fed by the receiver's `STATUS_STATS`, in both the sequential and `--threads` send paths. FastSync also prints rsync's leading `./` transfer-root line and, when progress is requested (`--progress`/`-P`/`--info=progress`) and not `--quiet`, runs a **paths-only metadata pre-scan** (no file reads, no hashing) that supplies rsync's file-list total `T` for the `to-chk` denominator and the directory names; `--delete-during`/`--delete-delay` reuse their existing keep-set pre-scan instead of walking twice, and non-progress runs are untouched. Per-directory name lines are emitted (trailing `/`), and symlink (` -> target`) and special entries are named too, so a **fresh multi-directory tree's name set and `to-chk` denominator match rsync 3.4.1** (differential test, sequential and `--threads`) and a **single-file transfer's name lines and deterministic frames remain byte-identical** to rsync. **Order parity (parity-2.29):** the sequential scanner now emits entries in rsync's sorted depth-first flist order (non-directories ascending, then directories ascending), so the interleaving and the `to-chk` numerator match rsync for the default single-threaded transfer (differential `test_parity_order.py`; `--threads` has no rsync analogue and stays unordered). **Remaining divergences:** the leading `./` root line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an ancestor directory line is emitted whenever a child transfers (rsync suppresses it when the directory itself is unchanged); on a re-run, entries without a quick-check (symlinks, empty directories) are still named where rsync stays silent; and the rate/ETA are wall-clock dependent | | `-P` | Same as --partial --progress | ✅ Parity | Parses to `--partial` + `--progress`. The independent `--partial` retention semantics are rsync parity: an interrupted write retains the already-written temp at the destination (best-effort) so a later `--append`/`--append-verify` can resume. Progress presentation is owned by the `--progress` row; there is no separate `-P` divergence | -| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally and an incremental re-run may itemize directories/symlinks without a quick-check | +| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally, every non-root directory renders as created because the sender does not probe directory destination state (so a pre-existing unchanged directory still shows `cd+++++++++`), and directory attribute columns (`%M`/`%U`/`%G`) come from the source | | `--log-file=FILE` | Log to file | ✅ Parity | `log_file` config field | | `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` as the wire byte count) | | `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Parity | Applies to displayed paths and protocol debug output | diff --git a/src/client/client_report.c b/src/client/client_report.c index cb787c4..ff45260 100644 --- a/src/client/client_report.c +++ b/src/client/client_report.c @@ -597,7 +597,10 @@ void client_progress_begin(const Config* config) { } return; } - if (g_progress_active) + /* -i/--out-format alone do not print the header, but --progress always does + and --info=flist does under any output mode (rsync prints it for + `-i --info=flist` and `--out-format=... --info=flist` too). */ + if (g_progress_active || (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST))) printf("sending incremental file list\n"); /* rsync prints the transfer-root directory before the first entry. Under -i/--out-format it is the root change line (`.d..t...... ./`); otherwise it @@ -677,6 +680,125 @@ static bool progress_precount_add_dir(ProgressPrecount* p, const char* path) { return false; } +static int progress_path_compare(const void* left, const void* right) { + const char* const* a = (const char* const*)left; + const char* const* b = (const char* const*)right; + return strcmp(*a, *b); +} + +/* Sort the collected directory paths and drop duplicates so a large + * --files-from list (many entries sharing an implied ancestor) cannot grow the + * list without bound. */ +static void progress_precount_dedup_dirs(ArrayList* dir_paths) { + if (dir_paths == NULL || dir_paths->size < 2) + return; + qsort(dir_paths->items, (size_t)dir_paths->size, sizeof(char*), progress_path_compare); + int write = 0; + for (int read = 0; read < dir_paths->size; read++) { + char* current = (char*)dir_paths->items[read]; + if (write > 0 && strcmp((char*)dir_paths->items[write - 1], current) == 0) { + free(current); + continue; + } + dir_paths->items[write++] = current; + } + dir_paths->size = write; +} + +/* Create a metadata-bearing directory File for the transfer-relative directory + * `rel` ("" is the transfer root), stat'ing it below config->send_directory. + * The -d/--files-from dirs generator never traverses directories, so this + * synthesizes the metadata the recursive scanner captures through + * scanner_capture_dir_time, letting -i/--out-format render %M/%B/%U/%G and the + * transfer-root/ancestor lines identically on both paths. Returns NULL when + * the path cannot be stat'd as a directory or on allocation failure (the line + * is then simply omitted, exactly as before). */ +static File* progress_precount_make_dir(const Config* config, const char* rel) { + if (config == NULL || config->send_directory == NULL) + return NULL; + char* fs_path = (rel == NULL || rel[0] == '\0') ? str_dup(config->send_directory) + : path_cat(config->send_directory, rel); + if (fs_path == NULL) + return NULL; + struct stat st; + if (stat(fs_path, &st) != 0 || !S_ISDIR(st.st_mode)) { + free(fs_path); + return NULL; + } + File* file = file_create(fs_path); + if (file == NULL) { + free(fs_path); + return NULL; + } + file->is_dir = true; + file->metadata = + file_metadata_create(fs_path, &st, config->preserve_atimes, config->preserve_crtimes); + file->send_path = str_dup(rel != NULL ? rel : ""); + free(fs_path); + if (file->metadata == NULL || file->send_path == NULL) { + file_destroy(file); + return NULL; + } + return file; +} + +/* The -d/--files-from dirs generator neither traverses nor records directories, + * so its metadata walk captures no Files. Synthesize the transfer root and + * every listed/implied directory from `entry_rels` so -i/--out-format emits the + * same root and ancestor lines the recursive scan does. `root_emitted` is true + * when the generator itself emits the root entry (bare `-d `), whose + * data-pass line must not be duplicated. Returns false only on allocation + * failure. */ +static bool progress_precount_synthesize_dirs(const Config* config, ProgressPrecount* out, + const ArrayList* entry_rels, bool root_emitted) { + for (int i = 0; i < entry_rels->size; i++) { + const char* rel = (const char*)entry_rels->items[i]; + if (rel == NULL) + continue; + size_t len = strlen(rel); + for (size_t j = 1; j < len; j++) { + if (rel[j] != '/') + continue; + /* --no-implied-dirs: rsync neither creates nor itemizes an implied parent, + so only explicitly listed directories get a line. */ + if (config->no_implied_dirs) + break; + char* prefix = malloc(j + 1); + if (prefix == NULL) + return false; + memcpy(prefix, rel, j); + prefix[j] = '\0'; + if (!progress_precount_add_dir(out, prefix)) { + free(prefix); + return false; + } + free(prefix); + } + } + progress_precount_dedup_dirs(out->dir_paths); + for (int i = 0; i < out->dir_paths->size; i++) { + const char* rel = (const char*)out->dir_paths->items[i]; + File* dir = progress_precount_make_dir(config, rel); + if (dir == NULL) + continue; + if (!array_list_add(out->dir_files, dir)) { + file_destroy(dir); + return false; + } + } + /* Emit the transfer root only when the generator actually emitted an entry: + --prune-empty-dirs (or an empty --files-from list) transfers nothing, and + rsync prints no root line then either. */ + if (!root_emitted && entry_rels->size > 0) { + File* root = progress_precount_make_dir(config, ""); + if (root != NULL && !array_list_add(out->dir_files, root)) { + file_destroy(root); + return false; + } + } + return true; +} + /* Metadata-only walk collecting the full file-list total and every directory * name. It uses its own scanner (fresh filter compilation and hard-link table) * so the data pass's link-group state is never perturbed. */ @@ -689,9 +811,21 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out) return false; } out->total = 0; + /* The -d/--files-from dirs generator never calls scanner_capture_dir_time, so + the walk below captures no directory Files. Record every emitted entry's + transfer-relative name so the implied ancestors can be synthesized once the + walk is done. */ + bool synthesize = g_change_dirs_active && config->dirs; + ArrayList* entry_rels = synthesize ? array_list_create(free) : NULL; + if (synthesize && entry_rels == NULL) { + progress_precount_dispose(out); + return false; + } + bool root_emitted = false; PreparedScanner prepared; memset(&prepared, 0, sizeof(prepared)); if (!prepare_scanner(config, 0, &prepared)) { + array_list_delete(entry_rels); progress_precount_dispose(out); return false; } @@ -722,8 +856,23 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out) out->total += (unsigned long long)chunk->element_count; for (int i = 0; i < chunk->element_count && ok; i++) { const File* f = chunk->items[i]; - if (f != NULL && f->is_dir) - ok = progress_precount_add_dir(out, delete_display_path(config, file_wire_path(f))); + if (f == NULL) + continue; + const char* rel = delete_display_path(config, file_wire_path(f)); + if (f->is_dir) { + if (rel != NULL && rel[0] == '\0') + root_emitted = true; + ok = progress_precount_add_dir(out, rel); + if (!ok) + break; + } + if (synthesize && rel != NULL) { + char* dup = str_dup(rel); + if (dup == NULL || !array_list_add(entry_rels, dup)) { + free(dup); + ok = false; + } + } } chunk_destroy(chunk); } @@ -733,9 +882,18 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out) } prepared_scanner_destroy(&prepared); if (!ok) { + array_list_delete(entry_rels); progress_precount_dispose(out); return false; } + if (synthesize) { + bool synth_ok = progress_precount_synthesize_dirs(config, out, entry_rels, root_emitted); + array_list_delete(entry_rels); + if (!synth_ok) { + progress_precount_dispose(out); + return false; + } + } /* Build the name -> File lookup from the captured directory Files. */ for (int i = 0; i < out->dir_files->size; i++) { File* f = (File*)out->dir_files->items[i]; @@ -821,9 +979,14 @@ void client_progress_prepare(const Config* config, const ArrayList* plan_dirs, g_change_dirs_active = config->itemize_changes || config->out_format != NULL; if (!g_progress_active && !g_change_dirs_active) return; - bool ok = plan_dirs != NULL ? progress_precount_from_plan_dirs( - config, plan_dirs, plan_non_dir_count, &g_progress_precount) - : progress_precount_scan(config, &g_progress_precount); + /* -i/--out-format render directory metadata (%M/%B/%U/%G) that only the + metadata walk captures; the --delete-during/--delete-delay plan list has no + metadata, so prefer the walk whenever a change line is rendered. Pure + --progress keeps reusing the plan list and its cheaper path-only pass. */ + bool ok = (plan_dirs != NULL && !g_change_dirs_active) + ? progress_precount_from_plan_dirs(config, plan_dirs, plan_non_dir_count, + &g_progress_precount) + : progress_precount_scan(config, &g_progress_precount); if (!ok) { g_progress_total = 0; return; diff --git a/tests/integration/test_output_parity.py b/tests/integration/test_output_parity.py index b3b2f89..577a1ee 100644 --- a/tests/integration/test_output_parity.py +++ b/tests/integration/test_output_parity.py @@ -240,6 +240,75 @@ class TestItemizeParity: ) assert ".d..t...... ./" in fast, f"missing root line: {result.stdout!r}" + @requires_rsync + @pytest.mark.ci + def test_itemize_info_flist_header_matches_rsync(self, shared_server): + """`-i --info=flist` prints rsync's file-list header: the -i change + lines alone do not enable the flist category, but an explicit --info=flist + must not be suppressed when itemizing.""" + source = os.path.join(TEST_DATA_DIR, "out_itemfl_src") + dest = os.path.join(TEST_DATA_DIR, "out_itemfl_dst") + rdst = os.path.join(TEST_DATA_DIR, "out_itemfl_rdst") + _make_output_tree(source) + clean_dir(dest) + clean_dir(rdst) + flags = ["-a", "-i", "--info=flist"] + rsync_result = _rsync(flags + [source + "/", rdst + "/"]) + assert rsync_result.returncode == 0, rsync_result.stderr + result, _ = run_client(source, dest, flags=flags, port=shared_server.port) + assert result.returncode == 0, result.stderr[:300] + assert "sending incremental file list" in rsync_result.stdout + assert "sending incremental file list" in result.stdout, result.stdout + # -i alone (no explicit --info=flist) must stay silent like rsync. + clean_dir(dest) + clean_dir(rdst) + rsync_plain = _rsync(["-a", "-i", source + "/", rdst + "/"]) + plain, _ = run_client(source, dest, flags=["-a", "-i"], + port=shared_server.port) + assert "sending incremental file list" not in rsync_plain.stdout + assert "sending incremental file list" not in plain.stdout, plain.stdout + + @requires_rsync + @pytest.mark.ci + def test_itemize_files_from_dirs_root_and_ancestors(self, shared_server): + """The -d/--files-from dirs generator emits the transfer-root line and + rsync's implied ancestor directory lines. The generator traverses no + directories, so those must be synthesized from the listed entries.""" + source = os.path.join(TEST_DATA_DIR, "out_itemff_src") + dest = os.path.join(TEST_DATA_DIR, "out_itemff_dst") + rdst = os.path.join(TEST_DATA_DIR, "out_itemff_rdst") + clean_dir(source) + os.makedirs(os.path.join(source, "sub", "deep")) + with open(os.path.join(source, "sub", "deep", "d.txt"), "wb") as fh: + fh.write(b"deep\n") + clean_dir(dest) + clean_dir(rdst) + listing = os.path.join(TEST_DATA_DIR, "out_itemff.list") + with open(listing, "w") as fh: + fh.write("sub/deep/d.txt\n") + + flags = ["-d", "-i", "--files-from=" + listing] + rsync_result = _rsync(flags + [source + "/", rdst + "/"]) + assert rsync_result.returncode == 0, rsync_result.stderr + result, _ = run_client(source, dest, flags=flags, port=shared_server.port) + assert result.returncode == 0, result.stderr[:300] + + def dir_lines(text): + return sorted(line for line in text.splitlines() + if line.rsplit(" ", 1)[-1].endswith("/")) + + # rsync emits the implied parents (sub/, sub/deep/) but never the root + # here; FastSync emits the same set plus its unconditional root line. + expected = [line for line in dir_lines(rsync_result.stdout) + if not line.rsplit(" ", 1)[-1] == "./"] + fast = dir_lines(result.stdout) + assert [line for line in fast if not line.rsplit(" ", 1)[-1] == "./"] == expected, ( + f"rsync={rsync_result.stdout!r} fastsync={result.stdout!r}" + ) + assert "cd+++++++++ sub/" in fast, result.stdout + assert "cd+++++++++ sub/deep/" in fast, result.stdout + assert any(line.rsplit(" ", 1)[-1] == "./" for line in fast), result.stdout + @requires_rsync @pytest.mark.ci def test_itemize_modified_file_matches_rsync(self, shared_server): @@ -318,6 +387,47 @@ class TestOutFormatParity: if line: assert pattern.match(line), f"bad %M format: {line!r}" + @requires_rsync + @pytest.mark.ci + def test_out_format_directory_metadata_with_delete_during(self): + """--delete-during/--delete-delay reuse the per-directory plan pre-scan, + whose list carries no metadata. Directory %M/%B/%U/%G must still come + from the source, exactly as the plain recursive scan renders them.""" + source = os.path.join(TEST_DATA_DIR, "out_fmtmeta_src") + dest = os.path.join(TEST_DATA_DIR, "out_fmtmeta_dst") + rdst = os.path.join(TEST_DATA_DIR, "out_fmtmeta_rdst") + clean_dir(source) + os.makedirs(os.path.join(source, "sub", "deep")) + with open(os.path.join(source, "a.txt"), "wb") as fh: + fh.write(b"hello\n") + with open(os.path.join(source, "sub", "b.txt"), "wb") as fh: + fh.write(b"world\n") + clean_dir(dest) + clean_dir(rdst) + + def dir_lines(text): + # Directory names are the last whitespace-separated token. + return sorted(line for line in text.splitlines() + if line.rsplit(" ", 1)[-1].endswith("/") + and line.rsplit(" ", 1)[-1] != "./") + + for timing in ("--delete-during", "--delete-delay"): + for fmt in ("%M %n", "%B %n", "%U %G %n"): + clean_dir(dest) + clean_dir(rdst) + flags = ["-a", "--out-format=" + fmt, timing] + rsync_result = _rsync(flags + [source + "/", rdst + "/"]) + assert rsync_result.returncode == 0, rsync_result.stderr + with ServerManager() as server: + server.start(extra_args=["--allow-delete"]) + result, _ = run_client(source, dest, flags=flags, port=server.port) + assert result.returncode == 0, result.stderr[:300] + assert dir_lines(result.stdout) == dir_lines(rsync_result.stdout), ( + f"{timing} {fmt}: rsync={rsync_result.stdout!r} " + f"fastsync={result.stdout!r}" + ) + assert "1970/" not in result.stdout, result.stdout + class TestListOnlyParity: @requires_rsync