Merge pull request 'docs: update README with TLS, incremental sync, server CLI, and corrected build requirements' (#20) from update-readme into main

Reviewed-on: #20

README.md

@@ -1,35 +1,43 @@
 # FastSync
 
-A high-performance file synchronization system with SSH and TCP transport, streaming zstd compression, multithreaded transfer, metadata preservation, and rsync-compatible CLI flags.
+A high-performance file synchronization system with SSH and TCP transport, TLS encryption, streaming zstd compression, multithreaded transfer, incremental sync, metadata preservation, and rsync-compatible CLI flags.
 
 ## Technical Overview
 
 1. **Dual transport**: custom TCP client-server or SSH subprocess (rsync-style `user@host:/path`)
-2. **Chunked file transfer**: files grouped into configurable-size chunks (default ~10 MB)
+2. **TLS encryption**: OpenSSL-based TLS 1.2+ for encrypted TCP connections
-3. **Streaming zstd compression** (levels 1–22) using `ZSTD_compressStream2`
+3. **Chunked file transfer**: files grouped into configurable-size chunks (default ~10 MB)
-4. **Multithreading**: producer-consumer pipeline with thread-safe queues (scanner → loader → sender)
+4. **Streaming zstd compression** (levels 1–22) using `ZSTD_compressStream2`
-5. **Metadata preservation**: `mode`, `uid`, `gid`, `mtime` restored on disk when enabled
+5. **Multithreading**: producer-consumer pipeline with thread-safe queues (scanner → loader → sender)
-6. **`sendfile()` zero-copy** on TCP (~2× faster on loopback)
+6. **Incremental sync**: skip files unchanged since last transfer (compares size + mtime)
-7. **SSH ControlMaster** for connection reuse across repeated invocations
+7. **Metadata preservation**: `mode`, `uid`, `gid`, `mtime` restored on disk when enabled
-8. **`--delete`**: receiver removes files not present in sender manifest
+8. **`sendfile()` zero-copy** on TCP (~2× faster on loopback)
-9. **`--exclude`**: glob-pattern filename filtering (`*`, `?`, no `/` crossing)
+9. **SSH ControlMaster** for connection reuse across repeated invocations
+10. **Bandwidth limiting**: token-bucket throttling (`--bwlimit`)
+11. **`--delete`**: receiver removes files not present in sender manifest
+12. **`--exclude` / `--include`**: glob-pattern filename filtering
 
 ## System Architecture
 
 ### Client
-- Recursively scans source directories (BFS), supports exclude patterns
+- Recursively scans source directories (BFS), supports exclude and include patterns
 - Groups files into chunks (configurable size)
 - Streaming zstd compression with configurable level
 - Chunk serialization (compact binary format) or per-file transfer
+- Incremental transfer: sends file metadata to server, skips unchanged files
 - Manifests all sent paths when `--delete` is active
 - Sends via TCP `sendfile()` or SSH pipe
 - Optional progress display with throughput
+- Bandwidth limiting via token-bucket algorithm
 
 ### Server
-- TCP mode: listens on port 8080; SSH mode: runs via `--stdio`
+- TCP mode: listens on configurable port (default 8080); SSH mode: runs via `--stdio`
+- TLS mode: wraps TCP connections with OpenSSL
 - Receives and reassembles files
 - Decompresses (streaming zstd), deserializes, restores metadata
+- Handles incremental checks: compares size + mtime against destination files
 - Processes `STATUS_MANIFEST` for `--delete`: walks destination tree, removes extras
+- Per-connection concurrency via `fork()`
 - Thread pool for parallel processing
 
 ## Protocol Details
@@ -43,6 +51,7 @@ A high-performance file synchronization system with SSH and TCP transport, strea
 | `STATUS_NEXT` | Ready for next file (per-file mode) |
 | `STATUS_CHUNK` | Following data is a serialized chunk |
 | `STATUS_MANIFEST` | Following data is a file manifest (for `--delete`) |
+| `STATUS_CHECK` | Incremental check: client sends file path + size + mtime, server responds with OK (skip) or NEXT (send) |
 
 ### Wire Format — Metadata
 
@@ -50,11 +59,17 @@ When `use_metadata` is enabled (`-M`), each file entry carries a 4-byte `present
 
 ### Transfer Flow
 ```
-Config → (STATUS_NEXT | STATUS_CHUNK)* → [STATUS_MANIFEST] → STATUS_FINISHED → STATUS_OK
+Config → (STATUS_NEXT | STATUS_CHUNK | STATUS_CHECK)* → [STATUS_MANIFEST] → STATUS_FINISHED → STATUS_OK
 ```
 
+### Protocol Version
+
+`1.1.0` — server and client must match. Mismatch results in `STATUS_ERROR`.
+
 ## Command-Line Arguments
 
+### Client
+
 | Argument | Description |
 |----------|-------------|
 | Positional | `<source> <dest>` — automatic SSH detection if dest contains `:` |
@@ -63,20 +78,42 @@ Config → (STATUS_NEXT | STATUS_CHUNK)* → [STATUS_MANIFEST] → STATUS_FINISH
 | `-a, --archive` | Archive mode: enables `-c -m -M` (no `-s`) |
 | `-m` | Multithreading mode |
 | `-s` | Chunk serialization (batch all files per chunk) |
-| `-f` | Sendfile zero-copy. Incompatible with `-c` / `-s`. TCP only. |
+| `-f, --sendfile` | Sendfile zero-copy. Incompatible with `-c` / `-s`. TCP only. |
 | `-M, --preserve` | Preserve file metadata (mode, uid, gid, mtime) |
 | `-n, --dry-run` | Scan and print what would be transferred |
 | `-p <port>` | SSH port (default: 22) |
+| `-v, --verbose` | Enable debug logging |
 | `--progress` | Show real-time transfer speed |
 | `--delete` | Delete files on receiver not present in source |
 | `--exclude <pattern>` | Exclude files matching glob pattern (repeatable) |
+| `--include <pattern>` | Only transfer files matching glob pattern (repeatable, whitelist) |
+| `--max-size <n>` | Skip files larger than n bytes |
+| `--min-size <n>` | Skip files smaller than n bytes |
+| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `-s`. |
+| `--bwlimit <KB/s>` | Bandwidth limit in kilobytes per second |
 | `--chunk-size <n>` | Chunk size in bytes (default: 10485760) |
 | `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
 | `--dest-dir <path>` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) |
 | `--save-to-disk` | Write received files to disk |
 | `--server-host <ip>` | Server IP address (default: `127.0.0.1`) |
 | `--server-port <n>` | Server port (default: `8080`) |
+| `--tls` | Enable TLS encryption |
+| `--cert <path>` | TLS certificate file (PEM) |
+| `--key <path>` | TLS private key file (PEM) |
+| `--ca <path>` | TLS CA certificate file for verification (PEM) |
+
+### Server
+
+| Argument | Description |
+|----------|-------------|
+| `--stdio` | Run in stdio mode (for SSH transport; single connection then exits) |
+| `-p <port>` | TCP listen port (default: 8080, range: 1–65535) |
+| `--tls` | Enable TLS encryption |
+| `--cert <path>` | TLS certificate file (PEM) |
+| `--key <path>` | TLS private key file (PEM) |
+| `--ca <path>` | TLS CA certificate file for verification (PEM) |
 | `-v, --verbose` | Enable debug logging |
+| `--help` | Show help |
 
 ## Environment Variables
 
@@ -92,26 +129,42 @@ Config → (STATUS_NEXT | STATUS_CHUNK)* → [STATUS_MANIFEST] → STATUS_FINISH
 1. **Chunk** — collection of files (~10 MB total by default)
 2. **File** — path, content (`Data`), optional `FileMetadata` pointer
 3. **FileMetadata** — `mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`
-4. **Config** — runtime parameters (transported over wire)
+4. **Config** — runtime parameters (transported over wire, TLS settings excluded)
 5. **Queue** — thread-safe bounded queue with condition variables
-6. **DirectoryScanner** — recursive BFS traversal with exclude pattern support
+6. **DirectoryScanner** — recursive BFS traversal with exclude and include pattern support
 
 ### Key Algorithms
-1. **File scanning** — BFS directory traversal; each entry matched against exclude patterns
+1. **File scanning** — BFS directory traversal; entries matched against exclude and include patterns
 2. **Chunking** — files accumulated until `chunk_size` threshold, then flushed
 3. **Compression** — streaming zstd via `ZSTD_compressStream2` / `ZSTD_decompressStream`
 4. **Network protocol** — status-code-driven exchange with metadata packing
-5. **Metadata restoration** — `chmod()`, `chown()`, `utimensat()` on the receiving side
+5. **Incremental check** — client sends `STATUS_CHECK` + path + size + mtime; server compares against destination
-6. **`--delete`** — sender tracks all sent paths; receiver walks destination tree and removes unlisted files/directories
+6. **Bandwidth limiting** — token-bucket algorithm with `nanosleep` throttling on 64 KB write chunks
-7. **SSH transport** — `socketpair()` + `fork()` + `execvp("ssh", ...)` with `ControlMaster` and port support
+7. **Metadata restoration** — `chmod()`, `chown()`, `utimensat()` on the receiving side
+8. **`--delete`** — sender tracks all sent paths; receiver walks destination tree and removes unlisted files/directories
+9. **SSH transport** — `socketpair()` + `fork()` + `execvp("ssh", ...)` with `ControlMaster` and port support
+10. **TLS transport** — OpenSSL `SSL_CTX` with TLS 1.2 minimum, optional CA verification, transparent `SSL_read`/`SSL_write` via `io_set_ssl()`
 
 ## Build Requirements
 
 - C11 compiler
-- CMake 4.1+
+- CMake >= 3.22
-- zstd library (≥ 1.4.0 for streaming API)
+- zstd library
+- OpenSSL (development headers and libraries)
 - pthreads
-- SSH client (for SSH transport)
+- SSH client (for SSH transport mode only)
+
+### Installing Dependencies
+
+**Ubuntu/Debian:**
+```bash
+sudo apt install cmake build-essential libzstd-dev libssl-dev openssh-client
+```
+
+**Nix:**
+```bash
+nix-shell  # provides zstd, openssl, cmake, gcc
+```
 
 ## Building
 
@@ -126,6 +179,14 @@ cmake -B build -S . && cmake --build build -j$(nproc)
 ./build/server
 ```
 
+### Server with TLS
+```bash
+./build/server --tls --cert server.pem --key server-key.pem
+```
+
+### Server via SSH
+Place the `fastsync-server` binary in the remote `$PATH`. The client runs `ssh user@host fastsync-server --stdio` automatically when an SSH-style destination is given.
+
 ### Client — SSH (rsync-style)
 ```bash
 ./build/client /path/to/send user@host:/path/to/receive
@@ -136,6 +197,12 @@ cmake -B build -S . && cmake --build build -j$(nproc)
 ./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk
 ```
 
+### Client — TCP with TLS
+```bash
+./build/client --tls --cert client.pem --key client-key.pem --ca ca.pem \
+  --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk
+```
+
 ### Common Options
 ```bash
 # Archive mode (compression + multithreading + metadata)
@@ -150,13 +217,16 @@ cmake -B build -S . && cmake --build build -j$(nproc)
 # Exclude temporary files + delete extras on receiver
 ./build/client --exclude "*.tmp" --exclude "*.o" --delete /src user@host:/dst
 
+# Incremental sync (skip unchanged files)
+./build/client --incremental /src user@host:/dst
+
+# Bandwidth limit to 1 MB/s
+./build/client --bwlimit 1024 /src user@host:/dst
+
 # All features
 ./build/client -a --progress --chunk-size 5242880 --exclude "*.log" --delete /src /dst
 ```
 
-### Server via SSH
-Place the `fastsync-server` binary in the remote `$PATH`. The client runs `ssh user@host fastsync-server --stdio` automatically when an SSH-style destination is given.
-
 ## Testing
 
 ```bash
@@ -178,6 +248,8 @@ The benchmark prints throughput metrics, best configuration, and speedup vs rsyn
 5. Metadata transfer adds negligible overhead (~24 bytes per file when enabled)
 6. SSH socketpair buffer set to 1 MB for improved pipe throughput
 7. SSH ControlMaster reuses connections across repeated invocations
+8. Incremental sync eliminates redundant transfers entirely
+9. Bandwidth limiting uses token-bucket with nanosleep for accurate throttling
 
 ## Benchmark Results