diff --git a/README.md b/README.md index d6b7a42..39f1e8f 100644 --- a/README.md +++ b/README.md @@ -1,363 +1,377 @@ # FastSync -A high-performance file synchronization system with SSH and TCP transport, TLS encryption, streaming zstd compression, multithreaded transfer, incremental sync, metadata preservation, and rsync-compatible CLI flags. +FastSync is a high-performance file synchronization tool designed to become a +drop-in replacement for common `rsync` workflows. It keeps the familiar +source/destination model and rsync-style options while adding optional +multithreading, streaming zstd compression, chunking, zero-copy TCP transfers, +and native TCP/TLS transports. -## Technical Overview +The compatibility target is straightforward: -1. **Dual transport**: custom TCP client-server or SSH subprocess (rsync-style `user@host:/path`) -2. **TLS encryption**: OpenSSL-based TLS 1.2+ for encrypted TCP connections with optional CA verification -3. **Chunked file transfer**: files grouped into configurable-size chunks (default ~10 MB) -4. **Streaming zstd compression** (levels 1–22) using `ZSTD_compressStream2` -5. **Multithreading**: producer-consumer pipeline with thread-safe queues (scanner → loader → sender) -6. **Incremental sync**: skip files unchanged since last transfer (compares size + mtime) -7. **Batch incremental**: send incremental checks in batched groups for reduced round-trips -8. **Metadata preservation**: file mode and mtime are restored when enabled; ownership and atime are intentionally not restored -9. **`sendfile()` zero-copy** on TCP (~2× faster on loopback) -10. **SSH ControlMaster** for connection reuse across repeated invocations -11. **Bandwidth limiting**: token-bucket throttling (`--bwlimit`) -12. **`--delete`**: receiver removes files not present in sender manifest -13. **`--exclude` / `--include`**: glob-pattern filename filtering -14. **Path traversal protection**: `..` sequences in file paths are rejected automatically -15. **Connection limits**: server enforces maximum concurrent connections (default 100) -16. **Keep-alive**: periodic `STATUS_KEEPALIVE` messages detect stalled connections -17. **Abort handling**: `SIGINT` sends `STATUS_ABORT` for clean server-side teardown -18. **Atomic writes**: received files are written to a temporary name then atomically renamed -19. **Backup mode**: `--backup` preserves overwritten files with optional `--backup-dir` -20. **Log file**: `--log-file` redirects log output to a file instead of stderr -21. **Transfer statistics**: `--stats` prints summary of transferred bytes, files, and timing +- Existing rsync commands should keep the same meaning. +- FastSync-only performance options should be additive and optional. +- A normal compatibility-mode transfer should prioritize rsync filesystem + semantics over maximum throughput. -## System Architecture +FastSync currently speaks its own protocol to `fastsync-server`. SSH mode +starts that server remotely; it does not yet interoperate with an unmodified +rsync client or rsync daemon. See [Compatibility Status](#compatibility-status) +for the current boundary. -### Client -- Recursively scans source directories (BFS), supports exclude and include patterns -- Groups files into chunks (configurable size) -- Streaming zstd compression with configurable level -- Chunk serialization (compact binary format) or per-file transfer -- Incremental transfer: sends file metadata to server, skips unchanged files -- Batch incremental: groups incremental checks to minimize round-trips -- Manifests all sent paths when `--delete` is active -- Sends via TCP `sendfile()` or SSH pipe -- Optional progress display with throughput -- Bandwidth limiting via token-bucket algorithm -- Configurable I/O and connection timeouts (`--timeout`, `--contimeout`) -- Backup overwritten files (`--backup`) with optional directory (`--backup-dir`) -- Transfer statistics summary (`--stats`) -- Maximum directory depth control (`--max-depth`) -- Log file output (`--log-file`) -- Exclude patterns from file (`--exclude-from`) +## Why FastSync -### Server -- TCP mode: listens on configurable port (default 8080); SSH mode: runs via `--stdio` -- TLS mode: wraps TCP connections with OpenSSL with optional CA verification -- Receives and reassembles files -- Decompresses (streaming zstd), deserializes, restores metadata -- Handles incremental checks: compares size + mtime against destination files -- Handles batch incremental checks for reduced round-trips -- Processes `STATUS_MANIFEST` for `--delete`: walks destination tree, removes extras -- Per-connection concurrency via `fork()` with configurable connection limit (default 100) -- Thread pool for parallel processing -- Atomic writes: files written to `.tmp` path then atomically renamed on success -- Abort handling: cleanly shuts down on `STATUS_ABORT` from client -- Path traversal protection: rejects file paths containing `..` +FastSync uses a producer-consumer transfer pipeline and can combine several +optimizations for large or high-latency transfers: -## Protocol Details +- Multithreaded scanning, loading, and sending. +- Streaming zstd compression with levels 1 through 22. +- Configurable file chunking and compact chunk serialization. +- `sendfile()` zero-copy transfers over TCP. +- Batched incremental checks to reduce round trips. +- Optional block-level delta transfer for FastSync peers. +- Bandwidth limiting, progress reporting, statistics, and backups. +- TCP, SSH, and TLS transports. +- Atomic temporary-file writes by default. -### Status Codes -| Code | Meaning | -|------|---------| -| `STATUS_OK` | Operation successful | -| `STATUS_ERROR` | Error occurred | -| `STATUS_FINISHED` | Transfer complete | -| `STATUS_NEXT` | Ready for next file (per-file mode) | -| `STATUS_CHUNK` | Following data is a serialized chunk | -| `STATUS_MANIFEST` | Following data is a file manifest (for `--delete`) | -| `STATUS_CHECK` | Incremental check: client sends file path + size + mtime and, when negotiated, checksum; server responds with OK (skip) or NEXT (send) | -| `STATUS_CHECK_BATCH` | Batch incremental check: multiple file checks sent in one message | -| `STATUS_KEEPALIVE` | Keep-alive heartbeat to detect stalled connections | -| `STATUS_ABORT` | Abort signal: client interrupts, server cleans up and exits | -| `STATUS_DELTA_SIGNATURE` | Delta sync: following data is a file signature (rsync-style rolling hash) | -| `STATUS_DELTA_DATA` | Delta sync: following data is a delta patch for a file | +These optimizations are disabled or selected independently. Users can start +with rsync-style commands and add FastSync options when they are useful. -### Wire Format — Metadata +## Compatibility Status -When `use_metadata` is enabled (`-M`), each file entry carries a 4-byte `present` flag followed by five fields (`mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`). When disabled globally, no metadata bytes are sent — zero wire overhead. +FastSync is currently an rsync-compatible CLI in progress, not a complete +replacement for every rsync feature or protocol mode. -### Transfer Flow -``` -Config → (STATUS_NEXT | STATUS_CHUNK | STATUS_CHECK | STATUS_CHECK_BATCH)* → [STATUS_MANIFEST] → STATUS_FINISHED → STATUS_OK -``` +### Working today -Keep-alive (`STATUS_KEEPALIVE`) may be sent at any point during the transfer. The receiver resets its inactivity timer on receipt. If no data arrives within the receive timeout, the connection is aborted. +- Recursive directory scanning. +- Rsync-style source and destination arguments. +- SSH transport using `user@host:/path` destinations. +- TCP client/server transfers. +- Dry runs, excludes, includes, size filters, backups, statistics, and + bandwidth limiting. +- Incremental size/mtime checks and optional xxHash64 content checks. +- FastSync-native delta transfer for changed files. +- Optional mode and timestamp preservation. +- Delete manifests with server-side delete authorization. +- Temporary-file writes with atomic rename by default. +- Path traversal checks and destination-root confinement. -Abort (`STATUS_ABORT`) may be sent at any point. On receipt the server cleans up temporary files and exits the child process. +### Not yet equivalent to rsync -### Protocol Version +- The FastSync wire protocol is not the rsync wire protocol. +- SSH mode requires `fastsync-server` on the remote host. +- Archive mode does not yet provide all of rsync's `-rlptgoD` behavior. +- Symlink transfer is incomplete; link targets are not yet recreated in all + modes. +- Owner/group, ACL, xattr, hard-link, device, and special-file handling is + incomplete or unavailable. +- Sparse-file handling does not yet preserve all holes correctly. +- `--partial`, `--partial-dir`, `--append`, and `--append-verify` are not yet + full rsync-style resumable transfers. +- Several rsync short options currently have FastSync-specific meanings. Do + not assume every short option is interchangeable yet. -`2.2.0` — server and client must match. This version adds a 64-bit XXH64 checksum to checksum-enabled `STATUS_CHECK` messages and validates the negotiated compression choice (`zstd` or `none`). Older clients and servers must not be mixed with this version; mismatch results in `STATUS_ERROR`. +The detailed flag matrix is maintained in +[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It distinguishes implemented, +partial, alternate, and planned behavior. -Config negotiation is sender-driven: the client serializes transfer options and the server applies them while receiving and writing files. `--checksum` compares size and content checksum instead of timestamps. `--compress-choice zstd` enables zstd; `none` disables it. Unsupported choices are rejected during config exchange. +## Quick Start -## Command-Line Arguments +### Build -### Client - -| Argument | Description | -|----------|-------------| -| Positional | ` ` — automatic SSH detection if dest contains `:` | -| `-c [level]` | Compression with optional level (1–22, default 5) | -| `-z [level]` | Alias for `-c` | -| `-a, --archive` | Archive mode: enables `-c -m -M` (no `-s`) | -| `-m` | Multithreading mode | -| `-s` | Chunk serialization (batch all files per chunk) | -| `-f, --sendfile` | Sendfile zero-copy. Incompatible with `-c` / `-s`. TCP only. | -| `-M, --preserve` | Preserve supported file metadata (mode and mtime; ownership and atime are unsupported) | -| `-n, --dry-run` | Scan and print what would be transferred | -| `-p ` | SSH port (default: 22) | -| `-v, --verbose` | Enable debug logging | -| `--progress` | Show real-time transfer speed | -| `--delete` | Delete files on receiver not present in source | -| `--exclude ` | Exclude files matching glob pattern (repeatable) | -| `--exclude-from ` | Read exclude patterns from a file (one per line) | -| `--include ` | Only transfer files matching glob pattern (repeatable, whitelist) | -| `--max-size ` | Skip files larger than n bytes | -| `--min-size ` | Skip files smaller than n bytes | -| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `-s`. | -| `--bwlimit ` | Bandwidth limit in kilobytes per second | -| `--chunk-size ` | Chunk size in bytes (default: 10485760) | -| `--timeout ` | I/O timeout in seconds (default: 30) | -| `--contimeout ` | Connection timeout in seconds (default: 10) | -| `--backup` | Backup existing destination files before overwriting | -| `--backup-dir ` | Target directory for backups (requires `--backup`) | -| `--stats` | Print transfer statistics at end (bytes, files, timing) | -| `--max-depth ` | Maximum directory depth to recurse (0 = unlimited, default: 0) | -| `--log-file ` | Write log messages to file instead of stderr | -| `--source-dir ` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) | -| `--dest-dir ` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) | -| `--save-to-disk` | Write received files to disk | -| `--server-host ` | Server IP address (default: `127.0.0.1`) | -| `--server-port ` | Server port (default: `8080`) | -| `--tls` | Enable TLS encryption | -| `--cert ` | TLS certificate file (PEM) | -| `--key ` | TLS private key file (PEM) | -| `--ca ` | TLS CA certificate file for verification (PEM) | - -### Server - -| Argument | Description | -|----------|-------------| -| `--stdio` | Run in stdio mode (for SSH transport; single connection then exits) | -| `-p ` | TCP listen port (default: 8080, range: 1–65535) | -| `--tls` | Enable TLS encryption | -| `--cert ` | TLS certificate file (PEM) | -| `--key ` | TLS private key file (PEM) | -| `--ca ` | TLS CA certificate file for verification (PEM) | -| `-v, --verbose` | Enable debug logging | -| `--help` | Show help | - -## Environment Variables - -| Variable | Default | Description | -|----------|---------|-------------| -| `FASTSYNC_SOURCE_DIR` | — | Source directory fallback | -| `FASTSYNC_DEST_DIR` | — | Destination directory fallback | -| `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback | -| `FASTSYNC_SSH_PORT` | `22` | Default SSH port | -| `FASTSYNC_SERVER_HOST` | `127.0.0.1` | Default server host | -| `FASTSYNC_SERVER_PORT` | `8080` | Default server port | -| `FASTSYNC_TLS_CERT` | — | Default TLS certificate path | -| `FASTSYNC_TLS_KEY` | — | Default TLS private key path | -| `FASTSYNC_TLS_CA` | — | Default TLS CA certificate path | - -## Implementation Details - -### Data Structures -1. **Chunk** — collection of files (~10 MB total by default) -2. **File** — path, content (`Data`), optional `FileMetadata` pointer -3. **FileMetadata** — `mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`; uid/gid are advisory wire fields and are never applied by the receiver; atime is unsupported -4. **Config** — runtime parameters (transported over wire, TLS settings excluded). Includes `timeout`, `contimeout`, `quiet`, `backup`, `backup_dir`, `stats`, `max_depth`, `log_file`, `queue_size`. -5. **Queue** — thread-safe bounded queue with condition variables -6. **DirectoryScanner** — recursive BFS traversal with exclude and include pattern support, max-depth enforcement - -### Key Algorithms -1. **File scanning** — BFS directory traversal; entries matched against exclude and include patterns, max-depth enforced -2. **Chunking** — files accumulated until `chunk_size` threshold, then flushed -3. **Compression** — streaming zstd via `ZSTD_compressStream2` / `ZSTD_decompressStream` -4. **Network protocol** — status-code-driven exchange with metadata packing, keep-alive, and abort support -5. **Incremental check** — client sends `STATUS_CHECK` + path + size + mtime and, with `--checksum`, XXH64 content checksum; server compares against destination. Can be batched via `STATUS_CHECK_BATCH` for reduced round-trips. -6. **Bandwidth limiting** — token-bucket algorithm with `nanosleep` throttling on 64 KB write chunks -7. **Metadata restoration** — `chmod()`, `chown()`, `utimensat()` on the receiving side -8. **`--delete`** — sender tracks all sent paths; receiver walks destination tree and removes unlisted files/directories -9. **SSH transport** — `socketpair()` + `fork()` + `execvp("ssh", ...)` with `ControlMaster` and port support -10. **TLS transport** — OpenSSL `SSL_CTX` with TLS 1.2 minimum, optional CA verification, transparent `SSL_read`/`SSL_write` via `io_set_ssl()` -11. **Path traversal protection** — `has_path_traversal()` rejects any file path containing `..` components, preventing directory escape attacks -12. **Connection limiting** — server tracks active connections and rejects new ones beyond `max_connections` (default 100) -13. **Keep-alive** — idle connections receive periodic `STATUS_KEEPALIVE` to detect half-open TCP connections -14. **Abort handling** — `SIGINT` sets an abort flag; the next protocol operation sends `STATUS_ABORT` for clean server cleanup -15. **Atomic writes** — files are written to a `.tmp` suffix then atomically renamed via `rename()`, preventing partial files -16. **Backup** — before overwriting, existing files are moved to `--backup-dir` (or same directory with `~` suffix) preserving the original - -## Security Features - -### Path Traversal Protection -All received file paths are validated by `has_path_traversal()` before any disk operation. Any path containing `..` components is rejected with `STATUS_ERROR`, preventing directory escape attacks. - -### TLS Certificate Verification -When `--ca` is provided, the server performs mutual TLS verification (`SSL_VERIFY_PEER` with depth 4). Without `--ca`, TLS is still encrypted but peer certificates are not verified. - -### Connection Limits -The server enforces a maximum of 100 concurrent connections (configurable via `max_connections` in `Server`). When the limit is reached, new connections are immediately rejected and closed. - -### Abort Handling -If the client receives `SIGINT` (Ctrl+C) during a transfer, it sends `STATUS_ABORT` to the server. The server then cleans up temporary files and exits the child process, preventing incomplete files from remaining on disk. - -### Atomic Writes -Received files are written to a temporary path (suffixed with `.tmp`) and then atomically renamed to the final filename via `rename()`. This prevents partial or corrupted files from appearing at the destination if the transfer is interrupted. - -## Build Requirements - -- C11 compiler -- CMake >= 3.22 -- zstd library -- OpenSSL (development headers and libraries) -- pthreads -- SSH client (for SSH transport mode only) - -### Installing Dependencies - -**Ubuntu/Debian:** -```bash -sudo apt install cmake build-essential libzstd-dev libssl-dev openssh-client -``` - -**Nix:** -```bash -nix-shell # provides zstd, openssl, cmake, gcc -``` - -## Building +Requirements: C11 compiler, CMake 3.22 or newer, zstd, OpenSSL, pthreads, +and an SSH client for SSH transport. ```bash -cmake -B build -S . && cmake --build build -j$(nproc) +cmake -B build -S . +cmake --build build -j$(nproc) ``` -## Running +With Nix: -### Server (TCP mode) ```bash -./build/server +nix-shell +cmake -B build -S . +cmake --build build -j$(nproc) ``` -### Server with TLS +### SSH transfer + +The remote host must have `fastsync-server` available in `PATH`, or use +`--fastsync-server-path`. + ```bash -./build/server --tls --cert server.pem --key server-key.pem +./build/client /path/to/source user@host:/path/to/destination ``` -### Server via SSH -Place the `fastsync-server` binary in the remote `$PATH`. The client runs `ssh user@host fastsync-server --stdio` automatically when an SSH-style destination is given. +### TCP transfer + +Start the FastSync server: -### Client — SSH (rsync-style) ```bash -./build/client /path/to/send user@host:/path/to/receive +./build/server -p 8080 ``` -### Client — TCP +Then run the client: + ```bash -./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk +./build/client --server-host 127.0.0.1 --server-port 8080 \ + --source-dir /path/to/source --dest-dir /path/to/destination \ + --save-to-disk ``` -### Client — TCP with TLS +### TLS transfer + ```bash +./build/server --tls --cert server.pem --key server-key.pem -p 8443 ./build/client --tls --cert client.pem --key client-key.pem --ca ca.pem \ - --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk + --server-host example.com --server-port 8443 \ + --source-dir /path/to/source --dest-dir /path/to/destination \ + --save-to-disk ``` -### Common Options +## Common Workflows + +These examples show the intended rsync-style workflow. Options marked as +FastSync-native are optional performance or transport extensions. + ```bash -# Archive mode (compression + multithreading + metadata) -./build/client -a /path/to/send user@host:/path +# Basic synchronization +./build/client /source/ /destination/ -# Dry run -./build/client -n /path/to/send /path/to/receive +# Archive-style synchronization (current FastSync archive behavior) +./build/client -a /source/ user@host:/destination/ -# With progress and custom chunk size -./build/client --progress --chunk-size 2097152 /src user@host:/dst +# Preview a transfer without changing the destination +./build/client -n /source/ /destination/ -# Exclude temporary files + delete extras on receiver -./build/client --exclude "*.tmp" --exclude "*.o" --delete /src user@host:/dst +# Exclude temporary and object files +./build/client --exclude '*.tmp' --exclude '*.o' \ + /source/ user@host:/destination/ -# Incremental sync (skip unchanged files) -./build/client --incremental /src user@host:/dst +# Remove destination entries not present in the source +./build/client --delete /source/ user@host:/destination/ -# Bandwidth limit to 1 MB/s -./build/client --bwlimit 1024 /src user@host:/dst +# Skip unchanged files using size and modification time +./build/client --incremental /source/ user@host:/destination/ -# With timeouts and stats -./build/client --timeout 60 --contimeout 15 --stats /src user@host:/dst +# Verify content when size and time are not sufficient +./build/client --incremental --checksum /source/ user@host:/destination/ -# Backup overwritten files to a directory -./build/client --backup --backup-dir /backups /src user@host:/dst +# Preserve supported mode and timestamp metadata +./build/client -M /source/ user@host:/destination/ -# Exclude patterns from file, limit depth -./build/client --exclude-from ignore.txt --max-depth 3 /src user@host:/dst - -# Log to file -./build/client --log-file /tmp/fastsync.log /src user@host:/dst - -# All features -./build/client -a --progress --chunk-size 5242880 --exclude "*.log" --delete /src /dst +# Keep backups of overwritten destination files +./build/client --backup --backup-dir /backups \ + /source/ user@host:/destination/ ``` +## FastSync Extensions + +FastSync-native options are intended to add performance or operational +features without changing the meaning of ordinary compatibility options. + +| Option | Purpose | +|---|---| +| `-m` | Enable the multithreaded scanner/loader/sender pipeline. | +| `-c [level]`, `-z [level]` | Enable streaming zstd compression, levels 1-22. | +| `--compress-level ` | Set the zstd compression level. | +| `--chunk-size ` | Set the transfer chunk size. | +| `-s` | Enable FastSync chunk serialization. | +| `-f`, `--sendfile` | Use TCP `sendfile()` zero-copy transfer. Incompatible with compression and chunk serialization. | +| `--delta` | Use FastSync-native block delta transfer. Requires `--incremental`. | +| `--delta-block ` | Set the FastSync delta block size. | +| `--delta-max ` | Limit files eligible for FastSync delta transfer. | +| `--server-host ` | Select the TCP server host. | +| `--server-port ` | Select the TCP server port. | +| `--tls` | Enable TLS for TCP transport. | +| `--bwlimit ` | Apply token-bucket bandwidth limiting. | +| `--progress` | Show transfer progress and throughput. | +| `--stats` | Print transfer statistics. | +| `--timeout ` | Set I/O timeout. | +| `--contimeout ` | Set connection timeout. | + +Current short-option conflicts are tracked as compatibility work. In +particular, FastSync currently uses `-p` for SSH port, `-s` for chunk +serialization, and `-S` for sparse handling. These meanings must be reconciled +before FastSync can claim full rsync CLI compatibility. + +## Client Options + +### Selection and transfer + +| Option | Description | +|---|---| +| `-a`, `--archive` | Enable current archive preset. Full rsync archive semantics are planned. | +| `-n`, `--dry-run` | Scan and report without writing files. | +| `--delete` | Request removal of destination entries absent from the source. The server must allow deletion. | +| `--exclude ` | Exclude matching paths. Repeatable. | +| `--include ` | Include matching paths. Repeatable. | +| `--exclude-from ` | Read exclude patterns from a file. | +| `--include-from ` | Read include patterns from a file. | +| `--max-size ` | Skip files larger than the limit. | +| `--min-size ` | Skip files smaller than the limit. | +| `--max-depth ` | Limit recursive scanning depth; zero means unlimited. | +| `--incremental` | Skip files matching destination size and mtime. | +| `--checksum` | Include xxHash64 content checks in incremental comparisons. | +| `--backup` | Back up overwritten files. | +| `--backup-dir ` | Store backups under a separate directory. | +| `--suffix ` | Set the backup filename suffix. | +| `--partial` | Keep received data under the configured partial location. Full resume semantics are planned. | +| `--partial-dir ` | Set the partial transfer directory. | +| `--inplace` | Write directly to the destination instead of using a temporary file. | + +### Metadata and links + +| Option | Description | +|---|---| +| `-M`, `--preserve` | Preserve supported file metadata, currently mode and modification time. | +| `-l`, `--links` | Request symlink preservation; link-target transfer remains incomplete. | +| `--copy-links` | Copy symlink referents. | +| `--safe-links` | Skip symlinks that point outside the transfer tree. | +| `--copy-unsafe-links` | Copy unsafe symlink referents. | +| `-S`, `--sparse` | Request sparse-file handling; full hole preservation is planned. | + +### Output and logging + +| Option | Description | +|---|---| +| `-v`, `--verbose` | Enable debug logging. | +| `--progress` | Show live transfer progress. | +| `--stats` | Print transfer statistics. | +| `--log-file ` | Write log output to a file. | +| `-V`, `--version` | Print the FastSync protocol version. | +| `--help` | Print command usage. | + +### Paths and transport + +| Option | Description | +|---|---| +| `-p ` | SSH port in the current CLI. This conflicts with rsync's `-p` permissions option and is planned for correction. | +| `--fastsync-server-path ` | Remote FastSync server path for SSH mode. | +| `--source-dir ` | Set the source directory explicitly. | +| `--dest-dir ` | Set the destination directory explicitly. | +| `--save-to-disk` | Enable server-side disk persistence. | +| `--server-host ` | TCP server address. | +| `--server-port ` | TCP server port. | +| `--tls` | Enable TLS. Requires `--cert` and `--key`. | +| `--cert ` | TLS certificate file. | +| `--key ` | TLS private key file. | +| `--ca ` | CA file for peer verification. | + +## Server Options + +| Option | Description | +|---|---| +| `--stdio` | Serve one SSH connection over standard input/output. | +| `-p ` | TCP listen port. | +| `--tls` | Enable TLS. | +| `--cert ` | TLS certificate file. | +| `--key ` | TLS private key file. | +| `--ca ` | CA file for peer verification. | +| `--allow-delete` | Permit client delete manifests. Deletion is refused by default. | +| `-v`, `--verbose` | Enable debug logging. | +| `--help` | Print server usage. | + +## Architecture + +### Client + +- Recursively scans the source tree with include, exclude, size, and depth + filters. +- Sends individual files or serialized chunks. +- Performs incremental checks and optional content checksums. +- Uses a multithreaded producer-consumer pipeline when requested. +- Sends over TCP, TLS-wrapped TCP, or an SSH subprocess. +- Supports progress, statistics, backups, timeouts, and bandwidth limiting. + +### Server + +- Runs as a TCP listener or one-shot SSH `--stdio` server. +- Receives and reassembles files and decompresses streaming zstd data. +- Applies supported metadata and writes files through a confined destination + root. +- Uses temporary files and atomic rename by default. +- Handles delete manifests only when explicitly authorized. +- Enforces connection, message-size, and path-safety limits. + +## Protocol and Security + +FastSync protocol version `2.2.0` is shared by the client and server. The +current protocol is sender-driven and includes configuration negotiation, +incremental checks, checksums, manifests, keep-alives, abort handling, and +FastSync-native delta messages. Client and server versions must currently +match exactly. + +TLS provides encrypted TCP transport. Supplying `--ca` enables certificate +verification; without it, traffic is encrypted but peer identity is not +verified. Use certificate verification for deployments where authentication +matters. The default TCP transport is not encrypted. + +The receiver protects its destination root with path validation, `openat()` +directory traversal, `O_NOFOLLOW`, temporary files, and atomic renames. Delete +operations require the server's explicit `--allow-delete` policy. + +## Compatibility Roadmap + +The project will reach the drop-in replacement goal in stages: + +1. Correct rsync option meanings, including short options, combined options, + and `--option=value` syntax. +2. Add differential tests that compare FastSync and rsync contents, metadata, + links, deletes, filters, dry runs, and exit codes. +3. Make `-a` implement the expected recursive, links, permissions, times, + owner/group, and supported special-file behavior. +4. Complete symlink, sparse-file, metadata, delete-policy, and resumable-write + semantics. +5. Add rsync remote-shell and daemon protocol interoperability. +6. Keep FastSync performance options as negotiated, optional extensions. + +The exhaustive implementation matrix and compatibility notes are in +[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). + ## Testing -```bash -# Unit tests (18 suites — array_list, chunk, compression, config, data, delta, file, glob, -# metadata, property, protocol, queue, robustness, scanner, -# shared_utils, stress, transport_tcp, transport_ssh, transport_tls) -./build/tests +Run the unit test binary: -# Integration + benchmark suite -python3 test.py +```bash +./build/tests ``` -The benchmark prints throughput metrics, best configuration, and speedup vs rsync. +Run the Python integration suite: -## Performance Considerations +```bash +python3 -m pytest tests/ +``` -1. Chunk size (~10 MB default) balances memory and transfer efficiency -2. Compression level trades CPU for bandwidth -3. `sendfile()` bypasses userspace — ~2× faster on localhost for large files -4. Multithreading scales with core count and uses memory-based pipeline sizing -5. Metadata transfer adds negligible overhead (~24 bytes per file when enabled) -6. SSH socketpair buffer set to 1 MB for improved pipe throughput -7. SSH ControlMaster reuses connections across repeated invocations -8. Incremental sync eliminates redundant transfers entirely -9. Batch incremental reduces round-trips by grouping multiple checks into one message -10. Bandwidth limiting uses token-bucket with nanosleep for accurate throttling -11. Atomic writes add a single `rename()` per file — negligible overhead -12. Path traversal check is O(n) in path length with negligible cost +For stricter local validation: -## Benchmark Results +```bash +cmake -B build-strict -S . -DSTRICT_WARNINGS=ON +cmake --build build-strict -j$(nproc) +cmake -B build-asan -S . -DSANITIZER=address +cmake --build build-asan -j$(nproc) +``` -25 MB of mixed file sizes over `localhost` with disk I/O throttled (reads ≤ 15 MB/s, writes ≤ 10 MB/s) and network emulation via `tc netem`. Each test was run 3×; the median is reported below. +The benchmark tool compares FastSync configurations with rsync under +controlled local and network conditions: -### LAN (1000 Mbit, 20 ms ±1 ms, 0.1% loss) +```bash +python3 benchmark/bench.py --help +``` -| Configuration | Time | vs rsync (archive) | vs rsync (compress) | -|---|---|---|---| -| **Best: `-m -c`** | **0.20 s** | **11.2× faster** | **3.6× faster** | -| Compression (`-c`) | 0.31 s | 7.3× faster | 2.3× faster | -| Standard | 1.27 s | 1.8× faster | — | -| rsync (archive) | 2.27 s | — | — | -| rsync (archive + compress) | 0.72 s | — | — | +Benchmark results measure transfer performance only. They do not establish +rsync protocol or filesystem-semantic compatibility. -### WAN (100 Mbit, 50 ms ±10 ms, 1% loss) +## Performance Guidance -| Configuration | Time | vs rsync (archive) | vs rsync (compress) | -|---|---|---|---| -| **Best: `-m -c`** | **0.39 s** | **44.8× faster** | **3.8× faster** | -| Compression (`-c`) | 0.64 s | 27.3× faster | 2.3× faster | -| Standard | 7.12 s | 2.4× faster | — | -| rsync (archive) | 17.44 s | — | — | -| rsync (archive + compress) | 1.47 s | — | — | +- Use `-m` for workloads with many files or enough CPU parallelism. +- Use `-c` or `-z` when network bandwidth is more constrained than CPU. +- Tune `--chunk-size` for file sizes, memory limits, and network latency. +- Use `-f` for large uncompressed TCP transfers where zero-copy I/O helps. +- Use `--incremental` to avoid retransmitting unchanged files. +- Use `--delta` for changed files when both endpoints are FastSync peers. +- Use `--bwlimit` when sharing a link with other traffic. -Compression reduces the data on the wire enough that the transfer becomes latency-bound rather than bandwidth-bound. On WAN, the best configuration runs 10.8× faster than the theoretical limit for uncompressed data, since zstd shrinks the 25 MB payload to a fraction of its original size over the wire. +Always validate the compatibility behavior required by a deployment before +replacing an existing rsync job.