5 Commits
Author SHA1 Message Date
TapTap 919a729206 Release v2.21.0
CI / lint (push) Successful in 1m25s
CI / lint (pull_request) Successful in 1m25s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / sanitizers (address) (push) Successful in 1m6s
CI / sanitizers (undefined) (push) Successful in 1m0s
CI / fuzz-build (push) Successful in 34s
CI / coverage (push) Successful in 55s
CI / build-and-test (pull_request) Successful in 1m49s
CI / valgrind (push) Successful in 3m19s
CI / build-and-test (push) Successful in 5m23s
- Protocol 2.21.0: STATUS_ERROR_DETAIL rejection reasons and server-contacting --dry-run
- Daemon per-module/per-host caps and cross-process auth lockout
- Config X-macro serialization, authorized_root single-owner, Data charge ownership, receiver pipeline move
- Security audit hardening (SSH injection, FIFO/inplace, zstd DoS, TLS, dry-run oracle, bounds)
- Pre-auth basis_count NULL-deref fix; benchmark and nix-shell improvements
- Tested: unit, integration, ASan/UBSan, valgrind, fuzz, coverage (CI green)
2026-09-14 18:16:42 +02:00
TapTap 8cd2b550d9 Merge dev environment fix and push-only documentation
CI / lint (push) Successful in 1m25s
CI / sanitizers (undefined) (push) Successful in 1m2s
CI / sanitizers (address) (push) Successful in 1m9s
CI / fuzz-build (push) Successful in 36s
CI / coverage (push) Successful in 56s
CI / valgrind (push) Successful in 3m18s
CI / build-and-test (push) Successful in 5m25s
2026-09-14 18:08:51 +02:00
TapTap 99c0fd8016 Merge benchmark improvements: accurate data mix, transfer verification, warm mode 2026-09-14 18:08:51 +02:00
TapTap a2200f039a chore(dev): fix nix-shell environment; document push-only direction 2026-09-14 18:08:46 +02:00
TapTap 1437c6dc6b bench: fix data mix, verify transfers, robust netem, warm mode
- generate_bench_data now writes exactly (1-random_ratio)*target bytes of
  genuinely compressible repeated content instead of only the small fixed
  STRUCTURED_FILES set; measured composition is reported and --dry-run prints
  it for scaling checks
- verify each transfer against the source (paths/sizes/byte compare) before
  recording timing; add --no-verify; failed runs are counted as invalid
- correct p50/p95 with linear-interpolation percentile (was int(len*0.95))
- tc/netem: run tc directly as root, else sudo; clear error when tc/iproute2
  is missing or qdisc setup fails; netem_reset is always safe
- build into dedicated build-bench/ via --build-dir (Release), never reconfigure
  the user's build/
- parse --configs with shlex.split
- add MB/s throughput column and throughput_mbps JSON field
- add --warm incremental mode: untimed full seed then measure add/change deltas
2026-09-14 18:07:37 +02:00
5 changed files with 525 additions and 100 deletions

No files matched your search

+86 -35
View File
@@ -4,41 +4,7 @@ All notable changes to FastSync are documented here. Versions match
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must `PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
run the same version because the handshake is strict. run the same version because the handshake is strict.
## [Unreleased] ## [2.21.0] - 2026-09-14
### Added
- **Server-contacting `--dry-run` (protocol 2.21.0).** `--dry-run` now performs
a real handshake with a remote/daemon receiver and reports exactly what WOULD
change based on receiver state (existing destination files, mtimes, checksums,
basis dirs). The wire config carries the dry-run intent (`Config.dry_run`) and
the receiver answers each per-file check with `STATUS_DRY_RUN_TRANSFER` (would
transfer) or `STATUS_OK` (already up to date); the sender prints the
would-transfer set and its trailer without sending any file data. The receiver
performs the normal read-only incremental decision but mutates nothing: no temp
files, writes, renames, deletes, metadata/xattr/chown, or directory creation.
A plain local destination (no explicit `--server-port`/remote) keeps the
original client-side dry-run. Would-delete reporting for `--delete*` is
deferred to a follow-up; dry-run never deletes.
### Security
- Enforce the daemon's per-module `max connections` cap and add a global
`max connections per host` cap plus a cross-process `auth lockout`
(`auth lockout threshold` / `auth lockout duration`). Because the listener
forks one child per connection, the counters live in an anonymous shared
mapping created before the accept loop and reclaimed by the parent's
`SIGCHLD` handler, so the per-module, per-source and auth-failure state is
shared across every child (including after `SIGKILL`). The per-source table
now has a bounded lifetime (expired-lockout/idle entries are reclaimed, with a
rate-limited warning when it is genuinely full), and the occupancy counters are
re-derived from the shared slot table on every child exit so a child killed
mid-registration cannot leak a count. Trusted loopback peers are exempt from the
per-host cap and the auth lockout (they share one address); clients behind a
shared NAT/proxy still share a single per-host budget and lockout, which is
documented.
## [2.21.0] - 2026-09-13
### Added ### Added
@@ -51,12 +17,97 @@ run the same version because the handshake is strict.
the reason into a thread-local buffer exposed by `protocol_last_error()`. The the reason into a thread-local buffer exposed by `protocol_last_error()`. The
detail body is always consumed, so the stream cannot desynchronize, and detail body is always consumed, so the stream cannot desynchronize, and
messages are sliced to `MAX_ERROR_DETAIL_BYTES` (4096) on send. messages are sliced to `MAX_ERROR_DETAIL_BYTES` (4096) on send.
- **Server-contacting `--dry-run` (protocol 2.21.0).** `--dry-run` now performs
a real handshake with a remote/daemon receiver and reports exactly what WOULD
change based on receiver state (existing destination files, mtimes, checksums,
basis dirs). The wire config carries the dry-run intent (`Config.dry_run`) and
the receiver answers each per-file check with `STATUS_DRY_RUN_TRANSFER` (would
transfer) or `STATUS_OK` (already up to date); the sender prints the
would-transfer set and its trailer without sending any file data. The receiver
performs the normal read-only incremental decision but mutates nothing: no temp
files, writes, renames, deletes, metadata/xattr/chown, or directory creation.
A plain local destination (no explicit `--server-port`/remote) keeps the
original client-side dry-run. Would-delete reporting for `--delete*` is
deferred to a follow-up; dry-run never deletes.
- Daemon `max connections per host` (per-source-IP concurrent cap, default 0 =
unlimited), `auth lockout threshold` (default 10; 0 disables) and
`auth lockout duration` (default 300 s) config keys.
- `fastsync-server --allow-super` opt-in for a privileged standalone TCP server;
without it a root standalone receiver forces super-user activities off (device
nodes, `--write-devices`, ownership). The `--stdio` SSH argv is client-composed,
so super activities always stay off there.
### Changed ### Changed
- Config wire fields are now declared once in an X-macro table
(`CONFIG_WIRE_FIELDS` in `src/shared/config.h`) that generates the struct
members, defaults, and the send/receive sequence, removing the manual
six-site field sync. Wire bytes and `PROTOCOL_VERSION` are unchanged.
- `receive_incremental_check()` (the per-file `STATUS_CHECK` fast path) is split - `receive_incremental_check()` (the per-file `STATUS_CHECK` fast path) is split
into small static helpers with a short linear orchestrator. Pure refactor: the into small static helpers with a short linear orchestrator. Pure refactor: the
wire byte stream and all cleanup are unchanged. wire byte stream and all cleanup are unchanged.
- `authorized_root` state has a single owner (`utils.c`) with read accessors; the
duplicated statics in `file.c` and the server were removed.
- `Data` records its owning `ProtocolSession` so its memory charge is returned to
the session that reserved it, regardless of the destroying thread.
- The receiver pipeline moved out of `shared` into `server/receiver_pipeline.[ch]`;
the build now uses explicit `fastsync_shared` / `fastsync_client_core` /
`fastsync_server_core` targets instead of a GLOB, and the client no longer links
server code.
- The benchmark tool generates the requested random/compressible data mix
accurately, verifies each transfer before recording it, computes correct
percentiles, adds a MB/s column, handles `tc`/netem without requiring `sudo`
when already root, builds into a dedicated `build-bench/` directory, and adds a
`--warm` incremental-transfer mode.
- The `nix-shell` dev environment provides the full toolchain (clang-format,
cppcheck, pytest-xdist, OpenSSH, rsync, iproute2, valgrind, lcov) and no longer
builds on entry.
### Security
- Enforce the daemon's per-module `max connections` cap (0 = unlimited) and add
the shared per-source `max connections per host` cap plus a cross-process
`auth lockout`. Because the listener forks one child per connection, the
counters live in an anonymous shared mapping created before the accept loop and
reclaimed by the parent's `SIGCHLD` handler, so the per-module, per-source and
auth-failure state is shared across every child (including after `SIGKILL`). The
per-source table has a bounded lifetime (expired/idle entries are reclaimed,
with a rate-limited warning when genuinely full), and the occupancy counters are
re-derived from the shared slot table on every child exit. Trusted loopback
peers are exempt (they share one address); clients behind a shared NAT/proxy
share a single per-host budget and lockout, which is documented.
- Hardening from a full security audit:
- Fail a truncated zstd frame instead of spinning forever (remote DoS).
- Open receiver destination/basis/hard-link entries `O_NONBLOCK` so a
client-planted FIFO cannot block a worker indefinitely.
- Require a regular file before `--inplace` writes, closing a FIFO-hang and a
raw-device write that bypassed the `--write-devices` gate.
- Reject SSH destinations whose user/host begins with `-` and insert `--` before
the host token, closing `-o ProxyCommand=…` argument injection (RCE).
- Gate client `--force` recursive removal behind the server `--allow-delete`
policy.
- Reject empty `hosts allow`/`hosts deny`/`auth users` values instead of
silently meaning "unrestricted".
- Restrict TLS 1.2 to AEAD suites and set server cipher preference; load the
private key TOCTOU-safely from an `O_NOFOLLOW` fd; verify IP literals against
IP SANs; guard client-cert CN truncation.
- Make `--dry-run` content-blind: it neither reads destination files nor
hashes basis files, removing a 1-bit content oracle against `read only`
modules.
- Bound glob matching (iterative DP, no exponential backtracking) and bound
line reads for filter/`--files-from`/pattern files.
- Gate `system.posix_acl_*` xattrs on `--acls` and charge decompression/chunk
allocations against the per-connection memory budget.
### Fixed
- Pre-auth NULL dereference in `config_delete()` when an over-long
`basis_count` (and the analogous count fields) was received and then failed
validation; received counts are now validated before being published.
- Leaked inherited `Data` in the forked compression-truncation unit test
(valgrind definite leak).
- `receive_status()` no longer loses a captured rejection reason when owed
keepalives are drained.
## [2.20.0] - 2026-09-13 ## [2.20.0] - 2026-09-13
+5
View File
@@ -316,6 +316,11 @@ ssh user@host 'mkdir -p destination'
./build/client /path/to/source user@host:destination ./build/client /path/to/source user@host:destination
``` ```
FastSync is **push-only**: the source (first argument) is always a local
directory and only the destination may be remote. A remote source such as
`client user@host:src ./local` (a "pull") is intentionally not supported; see
[RSYNC_COMPAT.md](RSYNC_COMPAT.md#direction).
### TCP transfer ### TCP transfer
Start the FastSync server: Start the FastSync server:
+1
View File
@@ -641,6 +641,7 @@ now transmits targets (the prior behavior was broken/partial); its status moved
- **Module selection & confinement:** the client requests a module with an rsync-style `host::module[/path]` destination. The module name crosses the wire as a trailing string on the config frame (bumping `PROTOCOL_VERSION` 2.14.0 → 2.15.0; the bump is required because the config-frame layout changed and the strict same-version handshake is what prevents a peer from desynchronizing on the new trailing field). The daemon looks the module up in ITS OWN config and uses the module's `path` as the authorized root through the exact same `configure_authorization` confinement the standalone server applies to `--destination-root` (`file_open_secure_parent`, `has_path_traversal`, `path_is_within`); the client never supplies the root, every client-chosen-ownership/super-user request is refused unless the module declares `client owner = yes` (the daemon's per-module opt-in, see below), and the operator `--no-super` veto forces super-user activities off for every daemon connection. The client's `/path` part is relative inside the module and is rejected if absolute or if it contains `..`. Unknown modules are refused before any data moves (the run fails cleanly at the config handshake). An absolute destination and a module request against a non-daemon server are also refused. - **Module selection & confinement:** the client requests a module with an rsync-style `host::module[/path]` destination. The module name crosses the wire as a trailing string on the config frame (bumping `PROTOCOL_VERSION` 2.14.0 → 2.15.0; the bump is required because the config-frame layout changed and the strict same-version handshake is what prevents a peer from desynchronizing on the new trailing field). The daemon looks the module up in ITS OWN config and uses the module's `path` as the authorized root through the exact same `configure_authorization` confinement the standalone server applies to `--destination-root` (`file_open_secure_parent`, `has_path_traversal`, `path_is_within`); the client never supplies the root, every client-chosen-ownership/super-user request is refused unless the module declares `client owner = yes` (the daemon's per-module opt-in, see below), and the operator `--no-super` veto forces super-user activities off for every daemon connection. The client's `/path` part is relative inside the module and is rejected if absolute or if it contains `..`. Unknown modules are refused before any data moves (the run fails cleanly at the config handshake). An absolute destination and a module request against a non-daemon server are also refused.
- **`client owner` (client-chosen-ownership opt-in):** by default a daemon module refuses every request that would let the client pick an owner or ask for super-user activities — `--numeric-ids`, `--chown`, `--usermap`/`--groupmap`, `--fake-super`, `--copy-as`, and an explicit `--super` — at the config handshake (before `STATUS_OK`), because a daemon has no per-module opt-in for client-chosen ownership and any anonymous client could otherwise force arbitrary owner ids inside the module root. `client owner = yes` opts a single module in, allowing those requests within that module's root (a root standalone TCP listener honors them for its single operator-authorized root only when started with `--allow-super`; the flag is rejected with `--stdio`, whose client-composed remote argv must never opt back into super mode). Without the opt-in the daemon also forces super-user **device** activity off for that connection — char/block device-node creation (`--devices`) and `--write-devices` — even under the default `AUTO` mode, so a non-opted module can never be made to `mknod` or write a raw device; those entries are skipped (not refused) so an ordinary `-a` push still succeeds without device nodes. The opt-in does **not** lift the privilege requirement: `--copy-as` still needs a root receiver, and the operator `--no-super` veto still forces super-user activities off for every connection. The daemon logs a prominent startup warning for each `client owner = yes` module so the operator's deliberate choice is visible. - **`client owner` (client-chosen-ownership opt-in):** by default a daemon module refuses every request that would let the client pick an owner or ask for super-user activities — `--numeric-ids`, `--chown`, `--usermap`/`--groupmap`, `--fake-super`, `--copy-as`, and an explicit `--super` — at the config handshake (before `STATUS_OK`), because a daemon has no per-module opt-in for client-chosen ownership and any anonymous client could otherwise force arbitrary owner ids inside the module root. `client owner = yes` opts a single module in, allowing those requests within that module's root (a root standalone TCP listener honors them for its single operator-authorized root only when started with `--allow-super`; the flag is rejected with `--stdio`, whose client-composed remote argv must never opt back into super mode). Without the opt-in the daemon also forces super-user **device** activity off for that connection — char/block device-node creation (`--devices`) and `--write-devices` — even under the default `AUTO` mode, so a non-opted module can never be made to `mknod` or write a raw device; those entries are skipped (not refused) so an ordinary `-a` push still succeeds without device nodes. The opt-in does **not** lift the privilege requirement: `--copy-as` still needs a root receiver, and the operator `--no-super` veto still forces super-user activities off for every connection. The daemon logs a prominent startup warning for each `client owner = yes` module so the operator's deliberate choice is visible.
- **`read only` safe default:** every network transfer FastSync currently supports is a push that writes under the module root, so a `read only` module refuses the connection (clear server log "module is read only"; the client exits non-zero, nothing is transferred). A future pull/list operation can be opened up when it exists; the knob is already stored. - **`read only` safe default:** every network transfer FastSync currently supports is a push that writes under the module root, so a `read only` module refuses the connection (clear server log "module is read only"; the client exits non-zero, nothing is transferred). A future pull/list operation can be opened up when it exists; the knob is already stored.
- **Direction — remote source / pull is intentionally unsupported:** FastSync is push-only. The first positional argument is always a **local** source directory and the second is the destination; only the destination is parsed for remote syntax (`user@host:path` SSH, `host::module[/path]` daemon). A remote source such as `fastsync user@host:src ./local` is deliberately **not** implemented: rsync has no pull flag (direction is positional), so supporting a remote source is an optional feature rather than a compatibility requirement, and it would require a protocol role reversal (server as sender, client as receiver) across both transports. FastSync documents this as an intentional limitation rather than a missing rsync option. <a id="direction"></a>
- **`auth users` (A7 SCRAM-SHA-256 authentication):** a module that declares `auth users` requires the client to present credentials. The config frame carries ONLY the username; the daemon answers an auth-required module with `STATUS_AUTH_CHALLENGE` (PBKDF2 iteration count, 16-byte salt, 32-byte server nonce), the client answers with `STATUS_AUTH_RESPONSE` (fresh 32-byte client nonce + a 32-byte ClientProof), and the daemon accepts only when the proof verifies **and** the username is **on the module's `auth users` list** and has a store entry, replying `STATUS_AUTH_OK` with a 32-byte ServerSignature the client verifies before proceeding. Verification is constant-time over fixed 32-byte keys (the compare runs even for a miss), username membership uses a constant-time full-length scan, and an unknown/off-list user still receives a challenge and runs the same math against a dummy verifier: a deterministic per-username salt (`HMAC-SHA256(store dummy key, username)`), the store-wide uniform iteration count and dummy keys. Re-probing the same unknown username therefore yields an identical salt and iteration count while a different username yields a different salt, so there is no user-enumeration or timing oracle. The daemon logs the username but **never the password, proof or keys**. A module WITHOUT `auth users` stays open (legitimate rsync configuration); credentials sent to such a module are ignored. Read-only is orthogonal: even a correctly authenticated push to a `read only` module is still refused (all FastSync network transfers write). Fail-closed policy: a daemon whose config declares `auth users` on any module refuses to start unless a credential store was given (`--password-file` and/or `--early-input`); a missing or empty store is never silently treated as "open". A failed handshake (missing credentials, unknown/off-list user, wrong proof or malformed data) yields a single generic `STATUS_AUTH_FAILED` and the daemon closes before any data moves. The dummy key is persisted in an owner-only `<store_path>.dummykey` sidecar (auto-created on first load, mode 0600) so the dummy salt stays stable across daemon restarts, closing the restart-gated enumeration channel. The sidecar is secret material and must be protected like the credential store (owner-only 0600, included with the store in backups and rotation). It must be preserved across restarts for that guarantee; if it cannot be created (a process-substitution/FIFO store path such as `/dev/fd/N`, a read-only filesystem, a missing directory, or a create/write/fsync/link/fchmod failure), the daemon logs a warning and uses a transient per-run key, so unknown-user challenges change across restarts and the cross-restart guarantee does not hold for that deployment. One residual is accepted: the store iteration count is observable pre-auth by design, since the miss path must match a hit. **Transport policy (hardening A7-3/S1):** an auth-required module accepts credentials only when either (a) the connection is an encrypted, verified TLS connection whose client certificate matches `--client-cn`, or (b) the connection is plaintext from a loopback TCP peer **and** the operator explicitly passed `--allow-unauthenticated`. A remote plaintext peer, and a loopback plaintext peer without that flag, are refused at the config gate before any challenge is sent; `--allow-unauthenticated` never permits remote plaintext auth (remote peers still require verified TLS). Daemon modules are a `--daemon`-only feature — the SSH `--stdio` path never loads a daemon config and is not an auth transport for them. Because the loopback allowance trusts whichever peer the kernel reports as `127.0.0.1`, it assumes nothing relays remote connections to the daemon: a local TCP forwarder or TLS-terminating proxy in front of an auth-module listener makes remote clients appear as loopback and bypasses the mutual-TLS identity check, so do not front an auth-module listener with such a relay. - **`auth users` (A7 SCRAM-SHA-256 authentication):** a module that declares `auth users` requires the client to present credentials. The config frame carries ONLY the username; the daemon answers an auth-required module with `STATUS_AUTH_CHALLENGE` (PBKDF2 iteration count, 16-byte salt, 32-byte server nonce), the client answers with `STATUS_AUTH_RESPONSE` (fresh 32-byte client nonce + a 32-byte ClientProof), and the daemon accepts only when the proof verifies **and** the username is **on the module's `auth users` list** and has a store entry, replying `STATUS_AUTH_OK` with a 32-byte ServerSignature the client verifies before proceeding. Verification is constant-time over fixed 32-byte keys (the compare runs even for a miss), username membership uses a constant-time full-length scan, and an unknown/off-list user still receives a challenge and runs the same math against a dummy verifier: a deterministic per-username salt (`HMAC-SHA256(store dummy key, username)`), the store-wide uniform iteration count and dummy keys. Re-probing the same unknown username therefore yields an identical salt and iteration count while a different username yields a different salt, so there is no user-enumeration or timing oracle. The daemon logs the username but **never the password, proof or keys**. A module WITHOUT `auth users` stays open (legitimate rsync configuration); credentials sent to such a module are ignored. Read-only is orthogonal: even a correctly authenticated push to a `read only` module is still refused (all FastSync network transfers write). Fail-closed policy: a daemon whose config declares `auth users` on any module refuses to start unless a credential store was given (`--password-file` and/or `--early-input`); a missing or empty store is never silently treated as "open". A failed handshake (missing credentials, unknown/off-list user, wrong proof or malformed data) yields a single generic `STATUS_AUTH_FAILED` and the daemon closes before any data moves. The dummy key is persisted in an owner-only `<store_path>.dummykey` sidecar (auto-created on first load, mode 0600) so the dummy salt stays stable across daemon restarts, closing the restart-gated enumeration channel. The sidecar is secret material and must be protected like the credential store (owner-only 0600, included with the store in backups and rotation). It must be preserved across restarts for that guarantee; if it cannot be created (a process-substitution/FIFO store path such as `/dev/fd/N`, a read-only filesystem, a missing directory, or a create/write/fsync/link/fchmod failure), the daemon logs a warning and uses a transient per-run key, so unknown-user challenges change across restarts and the cross-restart guarantee does not hold for that deployment. One residual is accepted: the store iteration count is observable pre-auth by design, since the miss path must match a hit. **Transport policy (hardening A7-3/S1):** an auth-required module accepts credentials only when either (a) the connection is an encrypted, verified TLS connection whose client certificate matches `--client-cn`, or (b) the connection is plaintext from a loopback TCP peer **and** the operator explicitly passed `--allow-unauthenticated`. A remote plaintext peer, and a loopback plaintext peer without that flag, are refused at the config gate before any challenge is sent; `--allow-unauthenticated` never permits remote plaintext auth (remote peers still require verified TLS). Daemon modules are a `--daemon`-only feature — the SSH `--stdio` path never loads a daemon config and is not an auth transport for them. Because the loopback allowance trusts whichever peer the kernel reports as `127.0.0.1`, it assumes nothing relays remote connections to the daemon: a local TCP forwarder or TLS-terminating proxy in front of an auth-module listener makes remote clients appear as loopback and bypasses the mutual-TLS identity check, so do not front an auth-module listener with such a relay.
- **Credential store format:** server `--password-file`/`--early-input` files are line-based `user:$fastsync$1$pbkdf2-sha256$<iters>$<salt_b64>$<stored_key_b64>$<server_key_b64>`, one per line (standard base64; 16-byte salt, 32-byte keys; `iters` in `[100000, 10000000]`, default 600000). Every entry in the resulting store must agree on `iters` (a store whose entries disagree, or where a layered `--early-input` disagrees with `--password-file`, is rejected). Generate lines with `fastsync-server --hash-credentials FILE [--iterations N]`; the emitted lines are secret material, so redirect them to an owner-only (mode 0600) file (the tool warns on stderr if stdout is a group/other-accessible regular file). Blank lines and lines starting with `#`/`;` are comments; the parser is strict (a malformed line fails the whole load, so a typo can never let a different set of users in). **The legacy `user:SHA256HEX` form is hard-rejected** with an actionable "legacy" error; there is no auto-upgrade, so a replayable bearer digest can never be loaded by a 2.19.0 daemon. The client `--password-file` holds `user:password` on its first meaningful line (the literal password, used only for the handshake then burned); keep both files readable only by their owner (mode 0600). Per-username wire length is bounded (256 chars) and every decoded salt/key length is validated. Loading the store also maintains an owner-only `<store_path>.dummykey` sidecar (auto-created, mode 0600, exactly 32 bytes) holding the store-wide dummy key that shapes unknown-user challenges; persist it across daemon restarts so those challenges stay stable, and treat a sidecar with the wrong owner, a mode other than exactly 0600, the wrong size or the wrong type as a fatal load error (fail closed). If the sidecar cannot be created (e.g. a process-substitution store path such as `/dev/fd/N`, a read-only filesystem, a missing directory, or a create/write/fsync/link/fchmod failure), the daemon logs a warning and uses a transient per-run key, so the cross-restart stability guarantee does not hold there. - **Credential store format:** server `--password-file`/`--early-input` files are line-based `user:$fastsync$1$pbkdf2-sha256$<iters>$<salt_b64>$<stored_key_b64>$<server_key_b64>`, one per line (standard base64; 16-byte salt, 32-byte keys; `iters` in `[100000, 10000000]`, default 600000). Every entry in the resulting store must agree on `iters` (a store whose entries disagree, or where a layered `--early-input` disagrees with `--password-file`, is rejected). Generate lines with `fastsync-server --hash-credentials FILE [--iterations N]`; the emitted lines are secret material, so redirect them to an owner-only (mode 0600) file (the tool warns on stderr if stdout is a group/other-accessible regular file). Blank lines and lines starting with `#`/`;` are comments; the parser is strict (a malformed line fails the whole load, so a typo can never let a different set of users in). **The legacy `user:SHA256HEX` form is hard-rejected** with an actionable "legacy" error; there is no auto-upgrade, so a replayable bearer digest can never be loaded by a 2.19.0 daemon. The client `--password-file` holds `user:password` on its first meaningful line (the literal password, used only for the handshake then burned); keep both files readable only by their owner (mode 0600). Per-username wire length is bounded (256 chars) and every decoded salt/key length is validated. Loading the store also maintains an owner-only `<store_path>.dummykey` sidecar (auto-created, mode 0600, exactly 32 bytes) holding the store-wide dummy key that shapes unknown-user challenges; persist it across daemon restarts so those challenges stay stable, and treat a sidecar with the wrong owner, a mode other than exactly 0600, the wrong size or the wrong type as a fatal load error (fail closed). If the sidecar cannot be created (e.g. a process-substitution store path such as `/dev/fd/N`, a read-only filesystem, a missing directory, or a create/write/fsync/link/fchmod failure), the daemon logs a warning and uses a transient per-run key, so the cross-restart stability guarantee does not hold there.
- **Plaintext caveat:** an auth-required module is refused, **before any challenge is sent**, unless the connection is encrypted and verified TLS whose client certificate matches the server's `--client-cn`, or it is plaintext from a loopback TCP peer **and** the operator passed `--allow-unauthenticated`. A remote plaintext peer, and a loopback plaintext peer without that flag, never receive a challenge, and `--allow-unauthenticated` never permits remote plaintext auth (remote peers still require verified TLS). On the loopback plaintext transport that remains permitted, a local sniffer could still read the challenge and response and mount an **offline dictionary attack** against a weak password, so use `--tls` for any real deployment. `--client-cn` matches the certificate CN only (not a subjectAltName), which is acceptable for a private CA. Clients sending daemon credentials with `--password-file` to a non-loopback daemon must use `--tls`; the client rejects such a destination before any network I/O. Unlike the old challenge-less exchange there is **no replay**: the proof is bound to the fresh per-connection server nonce, so a captured `STATUS_AUTH_RESPONSE` cannot be reused on another connection (an integration test proxies the daemon and proves this). TLS client-CN (`--client-cn`) is an independent transport identity check and composes with password auth; because `--tls` already mandates `--client-cn`, a TLS auth connection always verifies the client CN, so both checks necessarily apply together on such a connection. - **Plaintext caveat:** an auth-required module is refused, **before any challenge is sent**, unless the connection is encrypted and verified TLS whose client certificate matches the server's `--client-cn`, or it is plaintext from a loopback TCP peer **and** the operator passed `--allow-unauthenticated`. A remote plaintext peer, and a loopback plaintext peer without that flag, never receive a challenge, and `--allow-unauthenticated` never permits remote plaintext auth (remote peers still require verified TLS). On the loopback plaintext transport that remains permitted, a local sniffer could still read the challenge and response and mount an **offline dictionary attack** against a weak password, so use `--tls` for any real deployment. `--client-cn` matches the certificate CN only (not a subjectAltName), which is acceptable for a private CA. Clients sending daemon credentials with `--password-file` to a non-loopback daemon must use `--tls`; the client rejects such a destination before any network I/O. Unlike the old challenge-less exchange there is **no replay**: the proof is bound to the fresh per-connection server nonce, so a captured `STATUS_AUTH_RESPONSE` cannot be reused on another connection (an integration test proxies the daemon and proves this). TLS client-CN (`--client-cn`) is an independent transport identity check and composes with password auth; because `--tls` already mandates `--client-cn`, a TLS auth connection always verifies the client CN, so both checks necessarily apply together on such a connection.
+399 -62
View File
@@ -3,19 +3,24 @@
Compares FastSync configs against rsync (no compression) and rsync+zstd. Compares FastSync configs against rsync (no compression) and rsync+zstd.
Data is ~75% random/incompressible and ~25% structured/compressible by default, Data is ~75% random/incompressible and ~25% structured/compressible by default,
controllable via --random-ratio. controllable via --random-ratio. Transfers are verified by default (source and
destination must match) so a fast-but-broken copy is never counted.
Usage: Usage:
python3 benchmark/bench.py python3 benchmark/bench.py
python3 benchmark/bench.py --runs 5 --profiles lan wan python3 benchmark/bench.py --runs 5 --profiles lan wan
python3 benchmark/bench.py --random-ratio 0.5 --size-mb 50 python3 benchmark/bench.py --random-ratio 0.5 --size-mb 50
python3 benchmark/bench.py --delay 50ms --jitter 10ms --throughput 100mbit python3 benchmark/bench.py --delay 50ms --jitter 10ms --throughput 100mbit
python3 benchmark/bench.py --warm --runs 3
python3 benchmark/bench.py --output json python3 benchmark/bench.py --output json
""" """
import argparse import argparse
import filecmp
import json import json
import math
import os import os
import random import random
import shlex
import shutil import shutil
import socket import socket
import statistics import statistics
@@ -25,7 +30,10 @@ import tempfile
import time import time
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
BUILD_DIR = os.path.join(PROJECT_ROOT, "build") DEFAULT_BUILD_DIR = "build-bench"
# Populated by configure_build_dirs(); default to the dedicated bench dir so
# importing this module never depends on the user's existing build/ tree.
BUILD_DIR = os.path.join(PROJECT_ROOT, DEFAULT_BUILD_DIR)
SERVER_CMD = [os.path.join(BUILD_DIR, "server"), "--allow-unauthenticated"] SERVER_CMD = [os.path.join(BUILD_DIR, "server"), "--allow-unauthenticated"]
CLIENT_CMD = [os.path.join(BUILD_DIR, "client")] CLIENT_CMD = [os.path.join(BUILD_DIR, "client")]
BENCH_DIR = os.path.join(PROJECT_ROOT, "bench_data") BENCH_DIR = os.path.join(PROJECT_ROOT, "bench_data")
@@ -57,6 +65,7 @@ RSYNC_CONFIGS = [
{"name": "rsync -z --zstd", "flags": ["-z", "--zc", "zstd"],"tool": "rsync"}, {"name": "rsync -z --zstd", "flags": ["-z", "--zc", "zstd"],"tool": "rsync"},
] ]
class RsyncDaemon: class RsyncDaemon:
"""Manages an rsync daemon for network-fair benchmarking.""" """Manages an rsync daemon for network-fair benchmarking."""
@@ -118,6 +127,12 @@ STRUCTURED_FILES = {
"nested/another.txt": b"another nested file\n" * 50, "nested/another.txt": b"another nested file\n" * 50,
} }
# Repeated text used to synthesize genuinely compressible filler of any size.
COMPRESSIBLE_TEXT = (
b"FastSync benchmark payload: the quick brown fox jumps over the lazy dog. "
b"0123456789 ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz\n"
)
class Progress: class Progress:
"""Simple progress bar with ETA.""" """Simple progress bar with ETA."""
@@ -152,35 +167,127 @@ class Progress:
sys.stderr.flush() sys.stderr.flush()
def write_compressible(path, nbytes):
"""Write exactly nbytes of highly compressible, repeated text content."""
if nbytes <= 0:
return
block = COMPRESSIBLE_TEXT * (max(1, 8192 // len(COMPRESSIBLE_TEXT)) + 1)
remaining = nbytes
with open(path, "wb") as f:
while remaining > 0:
piece = block if remaining >= len(block) else block[:remaining]
f.write(piece)
remaining -= len(piece)
def generate_bench_data(source_dir, size_mb=25, random_ratio=0.75): def generate_bench_data(source_dir, size_mb=25, random_ratio=0.75):
"""Generate test data. ~random_ratio is incompressible, rest is structured.""" """Generate test data honouring the requested random/compressible split.
Exactly ``random_ratio * target`` bytes are incompressible random data and
the remainder is genuinely compressible structured/repeated content. The
measured byte counts are returned so callers can report the real mix.
"""
if os.path.exists(source_dir): if os.path.exists(source_dir):
shutil.rmtree(source_dir) shutil.rmtree(source_dir)
os.makedirs(source_dir) os.makedirs(source_dir)
target = size_mb * 1024 * 1024 target = size_mb * 1024 * 1024
structured_budget = int(target * (1 - random_ratio)) random_budget = int(target * random_ratio)
written = 0 compressible_budget = target - random_budget
compressible_written = 0
random_written = 0
files = 0
# A handful of fixed, human-meaningful files (directories, small files, a
# binary blob) as long as they fit inside the compressible budget.
for rel_path, content in STRUCTURED_FILES.items(): for rel_path, content in STRUCTURED_FILES.items():
if written >= structured_budget: if compressible_written + len(content) > compressible_budget:
break break
full_path = os.path.join(source_dir, rel_path) full_path = os.path.join(source_dir, rel_path)
os.makedirs(os.path.dirname(full_path), exist_ok=True) os.makedirs(os.path.dirname(full_path), exist_ok=True)
with open(full_path, "wb") as f: with open(full_path, "wb") as f:
f.write(content) f.write(content)
written += len(content) compressible_written += len(content)
files += 1
os.makedirs(os.path.join(source_dir, "bulk"), exist_ok=True) # Fill the rest of the compressible share with generated repeated content.
i = 0 if compressible_written < compressible_budget:
while written < target: os.makedirs(os.path.join(source_dir, "compressible"), exist_ok=True)
chunk_size = min(5 * 1024 * 1024, target - written) i = 0
with open(os.path.join(source_dir, f"bulk/file_{i}.dat"), "wb") as f: while compressible_written < compressible_budget:
f.write(random.randbytes(chunk_size)) chunk = min(1024 * 1024, compressible_budget - compressible_written)
written += chunk_size write_compressible(os.path.join(source_dir, "compressible", f"text_{i}.dat"), chunk)
i += 1 compressible_written += chunk
files += 1
i += 1
return written # Incompressible share.
if random_written < random_budget:
os.makedirs(os.path.join(source_dir, "bulk"), exist_ok=True)
i = 0
while random_written < random_budget:
chunk = min(5 * 1024 * 1024, random_budget - random_written)
with open(os.path.join(source_dir, "bulk", f"file_{i}.dat"), "wb") as f:
f.write(random.randbytes(chunk))
random_written += chunk
files += 1
i += 1
return {
"total_bytes": compressible_written + random_written,
"compressible_bytes": compressible_written,
"random_bytes": random_written,
"files": files,
}
def list_relative_files(root):
"""Return the set of file paths (relative to root) under a directory."""
found = set()
for dirpath, _dirnames, filenames in os.walk(root):
for name in filenames:
full = os.path.join(dirpath, name)
found.add(os.path.relpath(full, root))
return found
def verify_transfer(source_dir, dest_dir):
"""Recursively check dest matches source (paths, sizes, content).
Returns (ok, detail). Content is compared byte-for-byte, never hashed, so
collisions are impossible. This is intentionally not part of the timing.
"""
if not os.path.isdir(dest_dir):
return False, "destination directory missing"
src_files = list_relative_files(source_dir)
dst_files = list_relative_files(dest_dir)
if src_files != dst_files:
missing = src_files - dst_files
extra = dst_files - src_files
return False, f"path set mismatch (missing {len(missing)}, extra {len(extra)})"
for rel in sorted(src_files):
src = os.path.join(source_dir, rel)
dst = os.path.join(dest_dir, rel)
if os.path.getsize(src) != os.path.getsize(dst):
return False, f"size mismatch: {rel}"
if not filecmp.cmp(src, dst, shallow=False):
return False, f"content mismatch: {rel}"
return True, ""
def percentile(values, pct):
"""Linear-interpolation percentile (matches numpy's default method)."""
if not values:
return None
ordered = sorted(values)
if len(ordered) == 1:
return ordered[0]
rank = (len(ordered) - 1) * (pct / 100.0)
low = math.floor(rank)
high = math.ceil(rank)
if low == high:
return ordered[int(rank)]
return ordered[low] + (ordered[high] - ordered[low]) * (rank - low)
def find_free_port(): def find_free_port():
@@ -208,18 +315,45 @@ def wait_proc(proc, timeout=5):
proc.wait() proc.wait()
def _tc_base_cmd():
"""Return the command prefix for tc, honouring root vs sudo."""
tc = shutil.which("tc")
if not tc:
raise RuntimeError(
"tc (iproute2) not found in PATH; install iproute2 to use network profiles")
if os.geteuid() == 0:
return [tc]
sudo = shutil.which("sudo")
if sudo:
return [sudo, tc]
raise RuntimeError(
"applying network limits requires root or sudo; "
"re-run as root or install sudo")
def _run_tc(args, check=True):
return subprocess.run(_tc_base_cmd() + args, check=check, capture_output=True)
def netem_apply(delay=None, jitter=None, throughput=None, loss=None): def netem_apply(delay=None, jitter=None, throughput=None, loss=None):
"""Apply tc/netem rules to loopback. Pass None to skip a parameter.""" """Apply tc/netem rules to loopback. Pass None to skip a parameter."""
netem_reset() netem_reset()
cmd = ["sudo", "tc", "qdisc", "add", "dev", "lo", "root", "netem"] params = []
if throughput: if throughput:
cmd += ["rate", throughput] params += ["rate", throughput]
if delay: if delay:
cmd += ["delay", delay, jitter or "0ms"] params += ["delay", delay, jitter or "0ms"]
if loss: if loss:
cmd += ["loss", loss] params += ["loss", loss]
if len(cmd) > 6: if not params:
subprocess.run(cmd, check=True, capture_output=True) return
try:
_run_tc(["qdisc", "add", "dev", "lo", "root", "netem"] + params)
except subprocess.CalledProcessError as exc:
detail = exc.stderr.decode(errors="replace").strip() if exc.stderr else str(exc)
raise RuntimeError(f"failed to apply network profile via tc/netem: {detail}") from exc
except RuntimeError:
raise
def netem_apply_profile(profile_name): def netem_apply_profile(profile_name):
@@ -236,7 +370,11 @@ def netem_apply_profile(profile_name):
def netem_reset(): def netem_reset():
subprocess.run("sudo tc qdisc del dev lo root".split(), capture_output=True) """Best-effort removal of any loopback qdisc. Always safe to call."""
try:
_run_tc(["qdisc", "del", "dev", "lo", "root"], check=False)
except Exception:
pass
def run_fastsync(source_dir, dest_dir, flags, port): def run_fastsync(source_dir, dest_dir, flags, port):
@@ -287,7 +425,81 @@ def run_transfer(config, source_dir, dest_dir, port=None, rsync_daemon=None):
return run_fastsync(source_dir, dest_dir, config["flags"], port) return run_fastsync(source_dir, dest_dir, config["flags"], port)
def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=None): def apply_incremental_changes(source_dir, target_bytes):
"""Add and modify a few files so a warm transfer has real work to do.
Returns a mutation record (changed byte count plus enough data to revert
and re-apply it) so every warm run can start from a pristine source.
"""
modified_n = 3
added_n = 2
per_file = max(4096, target_bytes // (modified_n + added_n))
modified = {}
added = {}
changed = 0
existing = sorted(list_relative_files(source_dir))
if existing:
step = max(1, len(existing) // modified_n)
for rel in existing[::step][:modified_n]:
path = os.path.join(source_dir, rel)
original_size = os.path.getsize(path)
with open(path, "ab") as f:
f.write(random.randbytes(per_file))
modified[rel] = (original_size, per_file)
changed += per_file
for i in range(added_n):
os.makedirs(os.path.join(source_dir, "incremental"), exist_ok=True)
rel = os.path.join("incremental", f"new_{i}.dat")
write_compressible(os.path.join(source_dir, rel), per_file)
added[rel] = per_file
changed += per_file
return {"changed": changed, "modified": modified, "added": added}
def revert_incremental_changes(source_dir, mutation):
"""Undo apply_incremental_changes so the source is pristine again."""
if not mutation:
return
for rel, (original_size, _appended) in mutation["modified"].items():
path = os.path.join(source_dir, rel)
if os.path.exists(path):
with open(path, "r+b") as f:
f.truncate(original_size)
for rel in mutation["added"]:
path = os.path.join(source_dir, rel)
if os.path.exists(path):
os.remove(path)
def reapply_incremental_changes(source_dir, mutation):
"""Re-apply a mutation after an untimed pristine seed transfer."""
if not mutation:
return
for rel, (_original_size, appended) in mutation["modified"].items():
with open(os.path.join(source_dir, rel), "ab") as f:
f.write(random.randbytes(appended))
for rel, size in mutation["added"].items():
write_compressible(os.path.join(source_dir, rel), size)
def expected_received_root(dest_dir, source_dir, tool):
"""Where a tool places transferred files inside dest_dir.
FastSync mirrors the absolute source path under dest_dir (see the
integration suite's get_dest_received_dir); rsync copies the source tree
contents directly into dest_dir.
"""
if tool == "rsync":
return dest_dir
return os.path.join(dest_dir, os.path.abspath(source_dir).lstrip(os.sep))
def run_benchmark(source_dir, dest_dir, configs, runs, profile_name,
measure_bytes, verify=True, warm=False, mutation=None,
progress=None):
"""Run benchmark for all configs, returns list of results.""" """Run benchmark for all configs, returns list of results."""
is_limited = profile_name != "unlimited" is_limited = profile_name != "unlimited"
has_rsync = any(c["tool"] == "rsync" for c in configs) has_rsync = any(c["tool"] == "rsync" for c in configs)
@@ -303,7 +515,10 @@ def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=No
results = [] results = []
for config in configs: for config in configs:
times = [] times = []
invalid = 0
for run_idx in range(runs): for run_idx in range(runs):
if warm:
revert_incremental_changes(source_dir, mutation)
if os.path.exists(dest_dir): if os.path.exists(dest_dir):
shutil.rmtree(dest_dir) shutil.rmtree(dest_dir)
os.makedirs(dest_dir, exist_ok=True) os.makedirs(dest_dir, exist_ok=True)
@@ -311,15 +526,33 @@ def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=No
port = find_free_port() port = find_free_port()
server = None server = None
try: try:
if config["tool"] == "fastsync": if config["tool"] == "fastsync" or warm:
server = subprocess.Popen( server = subprocess.Popen(
SERVER_CMD + ["-p", str(port)], SERVER_CMD + ["-p", str(port)],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
) )
wait_for_port(port) wait_for_port(port)
if warm:
seed = run_transfer(config, source_dir, dest_dir, port, rsync_daemon)
if seed is None:
invalid += 1
sys.stderr.write(" warm-mode seeding failed; run not counted\n")
continue
reapply_incremental_changes(source_dir, mutation)
t = run_transfer(config, source_dir, dest_dir, port, rsync_daemon) t = run_transfer(config, source_dir, dest_dir, port, rsync_daemon)
if t is not None: if t is None:
invalid += 1
elif verify:
root = expected_received_root(dest_dir, source_dir, config["tool"])
ok, detail = verify_transfer(source_dir, root)
if ok:
times.append(t)
else:
invalid += 1
sys.stderr.write(f" verification FAILED ({detail}); run not counted\n")
else:
times.append(t) times.append(t)
finally: finally:
if server: if server:
@@ -332,15 +565,22 @@ def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=No
"config": config["name"], "config": config["name"],
"tool": config["tool"], "tool": config["tool"],
"profile": profile_name, "profile": profile_name,
"warm": warm,
"runs": len(times), "runs": len(times),
"invalid": invalid,
"times": [round(t, 4) for t in times], "times": [round(t, 4) for t in times],
} }
if times: if times:
entry["p50"] = round(statistics.median(times), 4) p50 = percentile(times, 50)
entry["p95"] = round(sorted(times)[int(len(times) * 0.95)], 4) if len(times) > 1 else entry["p50"] p95 = percentile(times, 95)
entry["p50"] = round(p50, 4)
entry["p95"] = round(p95, 4)
entry["min"] = round(min(times), 4) entry["min"] = round(min(times), 4)
entry["max"] = round(max(times), 4) entry["max"] = round(max(times), 4)
entry["stdev"] = round(statistics.stdev(times), 4) if len(times) > 1 else 0.0 entry["stdev"] = round(statistics.stdev(times), 4) if len(times) > 1 else 0.0
if measure_bytes:
entry["throughput_mbps"] = round(
(measure_bytes / (1024 * 1024)) / p50, 3)
results.append(entry) results.append(entry)
return results return results
finally: finally:
@@ -350,44 +590,59 @@ def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=No
netem_reset() netem_reset()
def print_table(results, total_bytes, random_ratio): def print_table(results, measure_bytes, stats, warm):
"""Print results as a human-readable table grouped by profile.""" """Print results as a human-readable table grouped by profile."""
profiles = {} profiles = {}
for r in results: for r in results:
profiles.setdefault(r["profile"], []).append(r) profiles.setdefault(r["profile"], []).append(r)
total = stats["total_bytes"]
comp_pct = stats["compressible_bytes"] / total * 100 if total else 0
rand_pct = stats["random_bytes"] / total * 100 if total else 0
for profile, entries in profiles.items(): for profile, entries in profiles.items():
params = NETWORK_PROFILES.get(profile, {}) params = NETWORK_PROFILES.get(profile, {})
print(f"\n{'=' * 85}") print(f"\n{'=' * 95}")
print(f" Profile: {profile.upper()}") print(f" Profile: {profile.upper()}")
if params.get("rate"): if params.get("rate"):
print(f" Network: {params['rate']}, {params['delay']} +/- {params['jitter']}, loss {params['loss']}") print(f" Network: {params['rate']}, {params['delay']} +/- {params['jitter']}, loss {params['loss']}")
else: else:
print(f" Network: unlimited") print(f" Network: unlimited")
print(f" Data: {total_bytes / (1024*1024):.1f} MB ({random_ratio*100:.0f}% random, {(1-random_ratio)*100:.0f}% compressible)") print(f" Data: {total / (1024*1024):.1f} MB "
print(f"{'=' * 85}") f"({rand_pct:.0f}% random, {comp_pct:.0f}% compressible actual)")
if warm:
print(f" Mode: warm (incremental) — measured {measure_bytes / (1024*1024):.2f} MB "
f"changed after an untimed full seed")
else:
print(" Mode: cold (full copy)")
print(f"{'=' * 95}")
fs_entries = [e for e in entries if e.get("tool") == "fastsync"] fs_entries = [e for e in entries if e.get("tool") == "fastsync"]
rsync_entries = [e for e in entries if e.get("tool") == "rsync"] rsync_entries = [e for e in entries if e.get("tool") == "rsync"]
header = (f" {'Config':<38} {'p50':>8} {'p95':>8} {'min':>8} {'max':>8} "
f"{'stdev':>8} {'MB/s':>9} {'runs':>5} {'bad':>4}")
rule = (f" {'-' * 38} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} "
f"{'-' * 8} {'-' * 9} {'-' * 5} {'-' * 4}")
if fs_entries: if fs_entries:
print(f"\n FastSync:") print(f"\n FastSync:")
print(f" {'Config':<38} {'p50':>8} {'p95':>8} {'min':>8} {'max':>8} {'stdev':>8} {'runs':>5}") print(header)
print(f" {'-' * 38} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 5}") print(rule)
for e in sorted(fs_entries, key=lambda x: x.get("p50", 999)): for e in sorted(fs_entries, key=lambda x: x.get("p50", 999)):
_print_entry(e) _print_entry(e)
if rsync_entries: if rsync_entries:
print(f"\n rsync:") print(f"\n rsync:")
print(f" {'Config':<38} {'p50':>8} {'p95':>8} {'min':>8} {'max':>8} {'stdev':>8} {'runs':>5}") print(header)
print(f" {'-' * 38} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 5}") print(rule)
for e in sorted(rsync_entries, key=lambda x: x.get("p50", 999)): for e in sorted(rsync_entries, key=lambda x: x.get("p50", 999)):
_print_entry(e) _print_entry(e)
if params.get("rate_bps") and fs_entries and rsync_entries: if params.get("rate_bps") and fs_entries and rsync_entries:
fs_best = min((e["p50"] for e in fs_entries if "p50" in e), default=None) fs_best = min((e["p50"] for e in fs_entries if "p50" in e), default=None)
rsync_best = min((e["p50"] for e in rsync_entries if "p50" in e), default=None) rsync_best = min((e["p50"] for e in rsync_entries if "p50" in e), default=None)
theoretical = total_bytes / params["rate_bps"] theoretical = measure_bytes / params["rate_bps"]
if fs_best and rsync_best: if fs_best and rsync_best:
print(f"\n Theoretical max (line rate): {theoretical:.4f}s") print(f"\n Theoretical max (line rate): {theoretical:.4f}s")
print(f" FastSync best: {fs_best:.4f}s ({theoretical/fs_best:.2f}x vs line rate)") print(f" FastSync best: {fs_best:.4f}s ({theoretical/fs_best:.2f}x vs line rate)")
@@ -397,10 +652,43 @@ def print_table(results, total_bytes, random_ratio):
def _print_entry(e): def _print_entry(e):
if "p50" in e: if "p50" in e:
tp = f"{e['throughput_mbps']:.2f}" if "throughput_mbps" in e else "N/A"
print(f" {e['config']:<38} {e['p50']:>7.4f}s {e['p95']:>7.4f}s " print(f" {e['config']:<38} {e['p50']:>7.4f}s {e['p95']:>7.4f}s "
f"{e['min']:>7.4f}s {e['max']:>7.4f}s {e['stdev']:>7.4f} {e['runs']:>5}") f"{e['min']:>7.4f}s {e['max']:>7.4f}s {e['stdev']:>7.4f} "
f"{tp:>9} {e['runs']:>5} {e.get('invalid', 0):>4}")
else: else:
print(f" {e['config']:<38} {'N/A':>8} {'N/A':>8} {'N/A':>8} {'N/A':>8} {'N/A':>8} {e['runs']:>5}") print(f" {e['config']:<38} {'N/A':>8} {'N/A':>8} {'N/A':>8} {'N/A':>8} "
f"{'N/A':>8} {'N/A':>9} {e['runs']:>5} {e.get('invalid', 0):>4}")
def configure_build_dirs(build_dir):
"""Install the selected build directory and derived binary paths."""
global BUILD_DIR, SERVER_CMD, CLIENT_CMD
if not os.path.isabs(build_dir):
build_dir = os.path.join(PROJECT_ROOT, build_dir)
BUILD_DIR = os.path.abspath(build_dir)
SERVER_CMD = [os.path.join(BUILD_DIR, "server"), "--allow-unauthenticated"]
CLIENT_CMD = [os.path.join(BUILD_DIR, "client")]
def build_project():
"""Configure (Release) and build into the dedicated bench build dir."""
if shutil.which("cmake") is None:
sys.stderr.write("cmake not found in PATH; cannot build\n")
sys.exit(1)
os.makedirs(BUILD_DIR, exist_ok=True)
configure = ["cmake", "-B", BUILD_DIR, "-S", PROJECT_ROOT,
"-DCMAKE_BUILD_TYPE=Release"]
result = subprocess.run(configure, capture_output=True, text=True)
if result.returncode != 0:
sys.stderr.write("CMake configure failed:\n" + result.stdout + result.stderr + "\n")
sys.exit(1)
jobs = str(os.cpu_count() or 1)
result = subprocess.run(["cmake", "--build", BUILD_DIR, "-j", jobs],
capture_output=True, text=True)
if result.returncode != 0:
sys.stderr.write("Build failed:\n" + result.stdout + result.stderr + "\n")
sys.exit(1)
def main(): def main():
@@ -417,12 +705,20 @@ Custom network limits (--delay/--jitter/--throughput) override profiles.
Data mix: Data mix:
Default is ~75%% random/incompressible + ~25%% structured/compressible, Default is ~75%% random/incompressible + ~25%% structured/compressible,
reflecting typical real-world file sets. reflecting typical real-world file sets. The actual mix is measured and
reported. Transfers are verified (destination must match source) unless
--no-verify is given.
Warm mode:
--warm seeds the destination with an untimed full copy of a pristine base,
then measures only the incremental transfer after modifying a few files.
Examples: Examples:
%(prog)s --profiles wan --runs 5 %(prog)s --profiles wan --runs 5
%(prog)s --throughput 50mbit --delay 30ms --jitter 5ms %(prog)s --throughput 50mbit --delay 30ms --jitter 5ms
%(prog)s --random-ratio 0.5 --size-mb 100 %(prog)s --random-ratio 0.5 --size-mb 100
%(prog)s --warm --runs 3 --no-rsync
%(prog)s --dry-run --size-mb 4 --random-ratio 0.25
""") """)
parser.add_argument("--runs", type=int, default=3, parser.add_argument("--runs", type=int, default=3,
help="Number of runs per config (default: 3)") help="Number of runs per config (default: 3)")
@@ -430,7 +726,8 @@ Examples:
choices=list(NETWORK_PROFILES.keys()), choices=list(NETWORK_PROFILES.keys()),
help="Predefined network profiles (default: unlimited)") help="Predefined network profiles (default: unlimited)")
parser.add_argument("--configs", nargs="+", default=None, parser.add_argument("--configs", nargs="+", default=None,
help="Custom FastSync config flags") help="Custom FastSync config flags (shell-quoted, e.g. "
"\"-j -z --chunk-serialization\")")
parser.add_argument("--size-mb", type=int, default=25, parser.add_argument("--size-mb", type=int, default=25,
help="Test data size in MB (default: 25)") help="Test data size in MB (default: 25)")
parser.add_argument("--random-ratio", type=float, default=0.75, parser.add_argument("--random-ratio", type=float, default=0.75,
@@ -445,6 +742,14 @@ Examples:
help="Custom packet loss (e.g. 1%%)") help="Custom packet loss (e.g. 1%%)")
parser.add_argument("--no-rsync", action="store_true", parser.add_argument("--no-rsync", action="store_true",
help="Skip rsync comparison") help="Skip rsync comparison")
parser.add_argument("--no-verify", action="store_true",
help="Skip source/destination verification after each run")
parser.add_argument("--warm", action="store_true",
help="Incremental mode: seed dest first, measure only changes")
parser.add_argument("--build-dir", default=DEFAULT_BUILD_DIR,
help=f"Build directory (default: {DEFAULT_BUILD_DIR})")
parser.add_argument("--dry-run", action="store_true",
help="Only generate data and report its composition, then exit")
parser.add_argument("--progress", action="store_true", parser.add_argument("--progress", action="store_true",
help="Show progress bar with ETA") help="Show progress bar with ETA")
parser.add_argument("--output", choices=["table", "json"], default="table", parser.add_argument("--output", choices=["table", "json"], default="table",
@@ -453,14 +758,48 @@ Examples:
help="Don't clean up test data") help="Don't clean up test data")
args = parser.parse_args() args = parser.parse_args()
if not 0.0 <= args.random_ratio <= 1.0:
parser.error("--random-ratio must be between 0.0 and 1.0")
if args.size_mb <= 0:
parser.error("--size-mb must be positive")
configure_build_dirs(args.build_dir)
# Generate data
source_dir = os.path.join(BENCH_DIR, "source")
dest_dir = os.path.join(BENCH_DIR, "dest")
stats = generate_bench_data(source_dir, args.size_mb, args.random_ratio)
total_bytes = stats["total_bytes"]
comp_pct = stats["compressible_bytes"] / total_bytes * 100 if total_bytes else 0
rand_pct = stats["random_bytes"] / total_bytes * 100 if total_bytes else 0
print(f"Generated {total_bytes / (1024*1024):.1f} MB in {stats['files']} files "
f"({rand_pct:.0f}% random, {comp_pct:.0f}% compressible actual)",
file=sys.stderr)
if args.dry_run:
print(f"size_mb={args.size_mb} random_ratio={args.random_ratio:.4f} "
f"total_bytes={stats['total_bytes']} "
f"compressible_bytes={stats['compressible_bytes']} "
f"random_bytes={stats['random_bytes']} files={stats['files']}")
if not args.keep_data:
shutil.rmtree(BENCH_DIR, ignore_errors=True)
return
# Warm mode: keep a pristine base copy, then mutate the live source.
base_dir = None
measure_bytes = total_bytes
mutation = None
if args.warm:
change_target = max(64 * 1024, min(int(total_bytes * 0.01), 4 * 1024 * 1024))
mutation = apply_incremental_changes(source_dir, change_target)
measure_bytes = mutation["changed"]
revert_incremental_changes(source_dir, mutation)
print(f"Warm mode: each run seeds a full copy, then measures "
f"{measure_bytes / (1024*1024):.3f} MB of add/change deltas", file=sys.stderr)
# Build (Release: benchmarking a debug build is meaningless) # Build (Release: benchmarking a debug build is meaningless)
print("Building (Release)...", file=sys.stderr) print(f"Building (Release) into {BUILD_DIR}...", file=sys.stderr)
configure = (f"cmake -B {BUILD_DIR} -S {PROJECT_ROOT} " build_project()
f"-DCMAKE_BUILD_TYPE=Release > /dev/null 2>&1")
if os.system(configure) != 0:
print("CMake configure failed", file=sys.stderr); sys.exit(1)
if os.system(f"cmake --build {BUILD_DIR} -j$(nproc) > /dev/null 2>&1") != 0:
print("Build failed", file=sys.stderr); sys.exit(1)
# Determine active profile for display # Determine active profile for display
has_custom_net = args.delay or args.jitter or args.throughput or args.loss has_custom_net = args.delay or args.jitter or args.throughput or args.loss
@@ -480,19 +819,10 @@ Examples:
else: else:
profiles_to_run = args.profiles or ["unlimited"] profiles_to_run = args.profiles or ["unlimited"]
# Generate data # Build config list (shlex so quoted/space-separated flags survive)
source_dir = os.path.join(BENCH_DIR, "source")
dest_dir = os.path.join(BENCH_DIR, "dest")
total_bytes = generate_bench_data(source_dir, args.size_mb, args.random_ratio)
compressible_pct = (1 - args.random_ratio) * 100
random_pct = args.random_ratio * 100
print(f"Generated {total_bytes / (1024*1024):.1f} MB "
f"({random_pct:.0f}% random, {compressible_pct:.0f}% compressible)",
file=sys.stderr)
# Build config list
if args.configs: if args.configs:
fastsync_configs = [{"name": c, "flags": c.split(), "tool": "fastsync"} for c in args.configs] fastsync_configs = [{"name": c, "flags": shlex.split(c), "tool": "fastsync"}
for c in args.configs]
else: else:
fastsync_configs = list(FASTSYNC_CONFIGS) fastsync_configs = list(FASTSYNC_CONFIGS)
@@ -509,9 +839,16 @@ Examples:
all_results = [] all_results = []
try: try:
for profile in profiles_to_run: for profile in profiles_to_run:
results = run_benchmark(source_dir, dest_dir, configs, args.runs, profile, progress) results = run_benchmark(source_dir, dest_dir, configs, args.runs, profile,
measure_bytes, verify=not args.no_verify,
warm=args.warm, mutation=mutation,
progress=progress)
all_results.extend(results) all_results.extend(results)
except RuntimeError as exc:
sys.stderr.write(f"error: {exc}\n")
sys.exit(1)
finally: finally:
netem_reset()
if not args.keep_data: if not args.keep_data:
shutil.rmtree(BENCH_DIR, ignore_errors=True) shutil.rmtree(BENCH_DIR, ignore_errors=True)
@@ -519,7 +856,7 @@ Examples:
if args.output == "json": if args.output == "json":
print(json.dumps(all_results, indent=2)) print(json.dumps(all_results, indent=2))
else: else:
print_table(all_results, total_bytes, args.random_ratio) print_table(all_results, measure_bytes, stats, args.warm)
print() print()
+34 -3
View File
@@ -3,11 +3,35 @@
}: }:
pkgs.mkShell { pkgs.mkShell {
# Development shell for FastSync. Provides the host-side toolchain needed to
# build, lint, unit-test, integration-test and benchmark the project.
# It deliberately does NOT build on entry: run the CMake commands in README.md
# (or use the CI Docker image for exact CI parity).
nativeBuildInputs = with pkgs; [ nativeBuildInputs = with pkgs; [
# build
gcc gcc
cmake cmake
gnumake gnumake
pkg-config pkg-config
# lint / static analysis (matches CI)
clang-tools # clang-format
cppcheck
# tests
(python3.withPackages (ps: with ps; [ pytest pytest-xdist psutil ]))
openssh # SSH transport integration tests
# debugging
gdb
valgrind
# coverage
lcov
# benchmark tooling
rsync
iproute2 # tc/netem for network shaping
# misc
git
curl
nodejs
nixpkgs-fmt
docker docker
tea tea
]; ];
@@ -15,14 +39,21 @@ pkgs.mkShell {
buildInputs = with pkgs; [ buildInputs = with pkgs; [
zstd zstd
openssl openssl
(python3.withPackages (ps: with ps; [ pytest ]))
]; ];
# The CMake configure step fetches xxHash via FetchContent, which needs
# network access; NIX_ENFORCE_PURITY must be off so the sandbox does not block.
NIX_ENFORCE_PURITY = 0; NIX_ENFORCE_PURITY = 0;
shellHook = '' shellHook = ''
export NIX_ENFORCE_PURITY=0 export NIX_ENFORCE_PURITY=0
cmake -B build # Make an existing build tree available on PATH, but never build here.
export PATH="$PWD/build:$PATH" if [ -d "$PWD/build" ]; then
export PATH="$PWD/build:$PATH"
fi
echo "FastSync dev shell ready."
echo " Build: cmake -B build -S . && cmake --build build -j\$(nproc)"
echo " Unit: ./build/tests"
echo " CI parity: docker run --rm --user \"\$(id -u):\$(id -g)\" -v \"\$PWD:/workspace\" -w /workspace gitea.tap-tap.win/taptap/fastsync-ci:v10 ..."
''; '';
} }