Merge branch 'fix/issues-f3' into fix/issues-triage

This commit is contained in:
2026-09-22 17:18:59 +02:00
3 changed files with 281 additions and 8 deletions
+2 -2
View File
@@ -123,10 +123,10 @@ Every one of those has an entry below with its remaining caveats.
|------|-------------------|-----------------|-------|
| `--stats` | Give transfer stats | ⚠️ Caveat | Prints transfer statistics. Protocol 2.25.0 populates the receiver-only counters the sender cannot observe (`Matched data`, `Number of deleted files`) from the receiver's `STATUS_STATS` report; the sender tracks the scanned file list per type so `Number of files` carries rsync's `(reg: X, dir: Y, link: Z, special: W)` breakdown (directories come from the scanner's captured directory list for `-a`/`-t`/`-p`, or from a lightweight traversed-directory counter on a plain `-r` run so the `dir:` category is present there too), `Number of regular files transferred` excludes symlinks/specials and up-to-date files, `Total file size` includes symlink target lengths, and `Total transferred file size` counts only transferred files. **Protocol 2.28.0 extends `STATUS_STATS`** with receiver-observed `literal_bytes` and the four `created_*` counters: `Number of created files` now carries rsync's `(reg/dir/link/special)` breakdown (the receiver reports which destination entries it newly created, including implicitly-created parent directories below the transfer root) and `Literal data` is exact for a delta transfer (the receiver counts the literal fragments it stored, not the whole source size) — all differential-tested in the sequential and `--threads` paths against rsync 3.4.1 for fresh-create, update and delta shapes. **Remaining divergences:** rsync's per-type breakdown on `Number of deleted files` is not reproduced; and `Total bytes sent`/`received` are FastSync wire bytes framed differently from rsync's, so they are not numerically comparable |
| `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units. **A lone `-h` with no transfer arguments prints help instead** (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer remains human-readable |
| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an incremental re-run itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`) |
| `-i`, `--itemize-changes` | Per-file change summary | ⚠️ Caveat | Prints rsync-style itemize lines to stdout for files actually sent (also under `-j`/`--threads`). Directory and transfer-root lines are now emitted too: a run produces rsync's `./` root line and per-directory `cd+++++++++`/`.d..t......` lines, rendered by the shared itemize code. **Residual:** the root `./` line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision; **every** non-root directory is rendered as created (`cd+++++++++`) because the sender never probes a directory's destination state, so a pre-existing destination directory that rsync reports as unchanged (`.d..t......`) is still itemized as created — this is not limited to re-runs; an incremental re-run additionally itemizes directories/symlinks that lack a quick-check where rsync stays silent (unchanged regular files still print nothing, matching single-`-i`); and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
| `--progress` | Show progress | ⚠️ Caveat | Protocol 2.25.0 prints rsync-style per-file progress blocks (percent, transferred/total bytes, rate, elapsed, `(xfr#N, to-chk=M/T)`) fed by the receiver's `STATUS_STATS`, in both the sequential and `--threads` send paths. FastSync also prints rsync's leading `./` transfer-root line and, when progress is requested (`--progress`/`-P`/`--info=progress`) and not `--quiet`, runs a **paths-only metadata pre-scan** (no file reads, no hashing) that supplies rsync's file-list total `T` for the `to-chk` denominator and the directory names; `--delete-during`/`--delete-delay` reuse their existing keep-set pre-scan instead of walking twice, and non-progress runs are untouched. Per-directory name lines are emitted (trailing `/`), and symlink (` -> target`) and special entries are named too, so a **fresh multi-directory tree's name set and `to-chk` denominator match rsync 3.4.1** (differential test, sequential and `--threads`) and a **single-file transfer's name lines and deterministic frames remain byte-identical** to rsync. **Order parity (parity-2.29):** the sequential scanner now emits entries in rsync's sorted depth-first flist order (non-directories ascending, then directories ascending), so the interleaving and the `to-chk` numerator match rsync for the default single-threaded transfer (differential `test_parity_order.py`; `--threads` has no rsync analogue and stays unordered). **Remaining divergences:** the leading `./` root line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an ancestor directory line is emitted whenever a child transfers (rsync suppresses it when the directory itself is unchanged); on a re-run, entries without a quick-check (symlinks, empty directories) are still named where rsync stays silent; and the rate/ETA are wall-clock dependent |
| `-P` | Same as --partial --progress | ✅ Parity | Parses to `--partial` + `--progress`. The independent `--partial` retention semantics are rsync parity: an interrupted write retains the already-written temp at the destination (best-effort) so a later `--append`/`--append-verify` can resume. Progress presentation is owned by the `--progress` row; there is no separate `-P` divergence |
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally and an incremental re-run may itemize directories/symlinks without a quick-check |
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible. Directory and transfer-root lines are now emitted (rsync's `./` root line and per-directory `cd...`/`.d..t...` lines), with the same residual as `-i`: the root line is emitted unconditionally, every non-root directory renders as created because the sender does not probe directory destination state (so a pre-existing unchanged directory still shows `cd+++++++++`), and directory attribute columns (`%M`/`%U`/`%G`) come from the source |
| `--log-file=FILE` | Log to file | ✅ Parity | `log_file` config field |
| `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` as the wire byte count) |
| `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Parity | Applies to displayed paths and protocol debug output |
+168 -5
View File
@@ -597,7 +597,10 @@ void client_progress_begin(const Config* config) {
}
return;
}
if (g_progress_active)
/* -i/--out-format alone do not print the header, but --progress always does
and --info=flist does under any output mode (rsync prints it for
`-i --info=flist` and `--out-format=... --info=flist` too). */
if (g_progress_active || (!config->quiet && info_flag_enabled(config, LOG_INFO_FLIST)))
printf("sending incremental file list\n");
/* rsync prints the transfer-root directory before the first entry. Under
-i/--out-format it is the root change line (`.d..t...... ./`); otherwise it
@@ -677,6 +680,125 @@ static bool progress_precount_add_dir(ProgressPrecount* p, const char* path) {
return false;
}
static int progress_path_compare(const void* left, const void* right) {
const char* const* a = (const char* const*)left;
const char* const* b = (const char* const*)right;
return strcmp(*a, *b);
}
/* Sort the collected directory paths and drop duplicates so a large
* --files-from list (many entries sharing an implied ancestor) cannot grow the
* list without bound. */
static void progress_precount_dedup_dirs(ArrayList* dir_paths) {
if (dir_paths == NULL || dir_paths->size < 2)
return;
qsort(dir_paths->items, (size_t)dir_paths->size, sizeof(char*), progress_path_compare);
int write = 0;
for (int read = 0; read < dir_paths->size; read++) {
char* current = (char*)dir_paths->items[read];
if (write > 0 && strcmp((char*)dir_paths->items[write - 1], current) == 0) {
free(current);
continue;
}
dir_paths->items[write++] = current;
}
dir_paths->size = write;
}
/* Create a metadata-bearing directory File for the transfer-relative directory
* `rel` ("" is the transfer root), stat'ing it below config->send_directory.
* The -d/--files-from dirs generator never traverses directories, so this
* synthesizes the metadata the recursive scanner captures through
* scanner_capture_dir_time, letting -i/--out-format render %M/%B/%U/%G and the
* transfer-root/ancestor lines identically on both paths. Returns NULL when
* the path cannot be stat'd as a directory or on allocation failure (the line
* is then simply omitted, exactly as before). */
static File* progress_precount_make_dir(const Config* config, const char* rel) {
if (config == NULL || config->send_directory == NULL)
return NULL;
char* fs_path = (rel == NULL || rel[0] == '\0') ? str_dup(config->send_directory)
: path_cat(config->send_directory, rel);
if (fs_path == NULL)
return NULL;
struct stat st;
if (stat(fs_path, &st) != 0 || !S_ISDIR(st.st_mode)) {
free(fs_path);
return NULL;
}
File* file = file_create(fs_path);
if (file == NULL) {
free(fs_path);
return NULL;
}
file->is_dir = true;
file->metadata =
file_metadata_create(fs_path, &st, config->preserve_atimes, config->preserve_crtimes);
file->send_path = str_dup(rel != NULL ? rel : "");
free(fs_path);
if (file->metadata == NULL || file->send_path == NULL) {
file_destroy(file);
return NULL;
}
return file;
}
/* The -d/--files-from dirs generator neither traverses nor records directories,
* so its metadata walk captures no Files. Synthesize the transfer root and
* every listed/implied directory from `entry_rels` so -i/--out-format emits the
* same root and ancestor lines the recursive scan does. `root_emitted` is true
* when the generator itself emits the root entry (bare `-d <dir>`), whose
* data-pass line must not be duplicated. Returns false only on allocation
* failure. */
static bool progress_precount_synthesize_dirs(const Config* config, ProgressPrecount* out,
const ArrayList* entry_rels, bool root_emitted) {
for (int i = 0; i < entry_rels->size; i++) {
const char* rel = (const char*)entry_rels->items[i];
if (rel == NULL)
continue;
size_t len = strlen(rel);
for (size_t j = 1; j < len; j++) {
if (rel[j] != '/')
continue;
/* --no-implied-dirs: rsync neither creates nor itemizes an implied parent,
so only explicitly listed directories get a line. */
if (config->no_implied_dirs)
break;
char* prefix = malloc(j + 1);
if (prefix == NULL)
return false;
memcpy(prefix, rel, j);
prefix[j] = '\0';
if (!progress_precount_add_dir(out, prefix)) {
free(prefix);
return false;
}
free(prefix);
}
}
progress_precount_dedup_dirs(out->dir_paths);
for (int i = 0; i < out->dir_paths->size; i++) {
const char* rel = (const char*)out->dir_paths->items[i];
File* dir = progress_precount_make_dir(config, rel);
if (dir == NULL)
continue;
if (!array_list_add(out->dir_files, dir)) {
file_destroy(dir);
return false;
}
}
/* Emit the transfer root only when the generator actually emitted an entry:
--prune-empty-dirs (or an empty --files-from list) transfers nothing, and
rsync prints no root line then either. */
if (!root_emitted && entry_rels->size > 0) {
File* root = progress_precount_make_dir(config, "");
if (root != NULL && !array_list_add(out->dir_files, root)) {
file_destroy(root);
return false;
}
}
return true;
}
/* Metadata-only walk collecting the full file-list total and every directory
* name. It uses its own scanner (fresh filter compilation and hard-link table)
* so the data pass's link-group state is never perturbed. */
@@ -689,9 +811,21 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
return false;
}
out->total = 0;
/* The -d/--files-from dirs generator never calls scanner_capture_dir_time, so
the walk below captures no directory Files. Record every emitted entry's
transfer-relative name so the implied ancestors can be synthesized once the
walk is done. */
bool synthesize = g_change_dirs_active && config->dirs;
ArrayList* entry_rels = synthesize ? array_list_create(free) : NULL;
if (synthesize && entry_rels == NULL) {
progress_precount_dispose(out);
return false;
}
bool root_emitted = false;
PreparedScanner prepared;
memset(&prepared, 0, sizeof(prepared));
if (!prepare_scanner(config, 0, &prepared)) {
array_list_delete(entry_rels);
progress_precount_dispose(out);
return false;
}
@@ -722,8 +856,23 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
out->total += (unsigned long long)chunk->element_count;
for (int i = 0; i < chunk->element_count && ok; i++) {
const File* f = chunk->items[i];
if (f != NULL && f->is_dir)
ok = progress_precount_add_dir(out, delete_display_path(config, file_wire_path(f)));
if (f == NULL)
continue;
const char* rel = delete_display_path(config, file_wire_path(f));
if (f->is_dir) {
if (rel != NULL && rel[0] == '\0')
root_emitted = true;
ok = progress_precount_add_dir(out, rel);
if (!ok)
break;
}
if (synthesize && rel != NULL) {
char* dup = str_dup(rel);
if (dup == NULL || !array_list_add(entry_rels, dup)) {
free(dup);
ok = false;
}
}
}
chunk_destroy(chunk);
}
@@ -733,9 +882,18 @@ static bool progress_precount_scan(const Config* config, ProgressPrecount* out)
}
prepared_scanner_destroy(&prepared);
if (!ok) {
array_list_delete(entry_rels);
progress_precount_dispose(out);
return false;
}
if (synthesize) {
bool synth_ok = progress_precount_synthesize_dirs(config, out, entry_rels, root_emitted);
array_list_delete(entry_rels);
if (!synth_ok) {
progress_precount_dispose(out);
return false;
}
}
/* Build the name -> File lookup from the captured directory Files. */
for (int i = 0; i < out->dir_files->size; i++) {
File* f = (File*)out->dir_files->items[i];
@@ -821,8 +979,13 @@ void client_progress_prepare(const Config* config, const ArrayList* plan_dirs,
g_change_dirs_active = config->itemize_changes || config->out_format != NULL;
if (!g_progress_active && !g_change_dirs_active)
return;
bool ok = plan_dirs != NULL ? progress_precount_from_plan_dirs(
config, plan_dirs, plan_non_dir_count, &g_progress_precount)
/* -i/--out-format render directory metadata (%M/%B/%U/%G) that only the
metadata walk captures; the --delete-during/--delete-delay plan list has no
metadata, so prefer the walk whenever a change line is rendered. Pure
--progress keeps reusing the plan list and its cheaper path-only pass. */
bool ok = (plan_dirs != NULL && !g_change_dirs_active)
? progress_precount_from_plan_dirs(config, plan_dirs, plan_non_dir_count,
&g_progress_precount)
: progress_precount_scan(config, &g_progress_precount);
if (!ok) {
g_progress_total = 0;
+110
View File
@@ -240,6 +240,75 @@ class TestItemizeParity:
)
assert ".d..t...... ./" in fast, f"missing root line: {result.stdout!r}"
@requires_rsync
@pytest.mark.ci
def test_itemize_info_flist_header_matches_rsync(self, shared_server):
"""`-i --info=flist` prints rsync's file-list header: the -i change
lines alone do not enable the flist category, but an explicit --info=flist
must not be suppressed when itemizing."""
source = os.path.join(TEST_DATA_DIR, "out_itemfl_src")
dest = os.path.join(TEST_DATA_DIR, "out_itemfl_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_itemfl_rdst")
_make_output_tree(source)
clean_dir(dest)
clean_dir(rdst)
flags = ["-a", "-i", "--info=flist"]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert "sending incremental file list" in rsync_result.stdout
assert "sending incremental file list" in result.stdout, result.stdout
# -i alone (no explicit --info=flist) must stay silent like rsync.
clean_dir(dest)
clean_dir(rdst)
rsync_plain = _rsync(["-a", "-i", source + "/", rdst + "/"])
plain, _ = run_client(source, dest, flags=["-a", "-i"],
port=shared_server.port)
assert "sending incremental file list" not in rsync_plain.stdout
assert "sending incremental file list" not in plain.stdout, plain.stdout
@requires_rsync
@pytest.mark.ci
def test_itemize_files_from_dirs_root_and_ancestors(self, shared_server):
"""The -d/--files-from dirs generator emits the transfer-root line and
rsync's implied ancestor directory lines. The generator traverses no
directories, so those must be synthesized from the listed entries."""
source = os.path.join(TEST_DATA_DIR, "out_itemff_src")
dest = os.path.join(TEST_DATA_DIR, "out_itemff_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_itemff_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub", "deep"))
with open(os.path.join(source, "sub", "deep", "d.txt"), "wb") as fh:
fh.write(b"deep\n")
clean_dir(dest)
clean_dir(rdst)
listing = os.path.join(TEST_DATA_DIR, "out_itemff.list")
with open(listing, "w") as fh:
fh.write("sub/deep/d.txt\n")
flags = ["-d", "-i", "--files-from=" + listing]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
def dir_lines(text):
return sorted(line for line in text.splitlines()
if line.rsplit(" ", 1)[-1].endswith("/"))
# rsync emits the implied parents (sub/, sub/deep/) but never the root
# here; FastSync emits the same set plus its unconditional root line.
expected = [line for line in dir_lines(rsync_result.stdout)
if not line.rsplit(" ", 1)[-1] == "./"]
fast = dir_lines(result.stdout)
assert [line for line in fast if not line.rsplit(" ", 1)[-1] == "./"] == expected, (
f"rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
assert "cd+++++++++ sub/" in fast, result.stdout
assert "cd+++++++++ sub/deep/" in fast, result.stdout
assert any(line.rsplit(" ", 1)[-1] == "./" for line in fast), result.stdout
@requires_rsync
@pytest.mark.ci
def test_itemize_modified_file_matches_rsync(self, shared_server):
@@ -318,6 +387,47 @@ class TestOutFormatParity:
if line:
assert pattern.match(line), f"bad %M format: {line!r}"
@requires_rsync
@pytest.mark.ci
def test_out_format_directory_metadata_with_delete_during(self):
"""--delete-during/--delete-delay reuse the per-directory plan pre-scan,
whose list carries no metadata. Directory %M/%B/%U/%G must still come
from the source, exactly as the plain recursive scan renders them."""
source = os.path.join(TEST_DATA_DIR, "out_fmtmeta_src")
dest = os.path.join(TEST_DATA_DIR, "out_fmtmeta_dst")
rdst = os.path.join(TEST_DATA_DIR, "out_fmtmeta_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub", "deep"))
with open(os.path.join(source, "a.txt"), "wb") as fh:
fh.write(b"hello\n")
with open(os.path.join(source, "sub", "b.txt"), "wb") as fh:
fh.write(b"world\n")
clean_dir(dest)
clean_dir(rdst)
def dir_lines(text):
# Directory names are the last whitespace-separated token.
return sorted(line for line in text.splitlines()
if line.rsplit(" ", 1)[-1].endswith("/")
and line.rsplit(" ", 1)[-1] != "./")
for timing in ("--delete-during", "--delete-delay"):
for fmt in ("%M %n", "%B %n", "%U %G %n"):
clean_dir(dest)
clean_dir(rdst)
flags = ["-a", "--out-format=" + fmt, timing]
rsync_result = _rsync(flags + [source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, result.stderr[:300]
assert dir_lines(result.stdout) == dir_lines(rsync_result.stdout), (
f"{timing} {fmt}: rsync={rsync_result.stdout!r} "
f"fastsync={result.stdout!r}"
)
assert "1970/" not in result.stdout, result.stdout
class TestListOnlyParity:
@requires_rsync