5 Commits

Author SHA1 Message Date
TapTap 78876b110e fix: harden remaining review findings
CI / lint (pull_request) Successful in 34s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:44:31 +02:00
TapTap daf662cc2d fix: address remaining PR review findings
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:24:13 +02:00
TapTap 298bfe0d0f fix: address delegated review findings
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 35s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 14s
CI / build-and-test (pull_request) Successful in 1m14s
CI / coverage (pull_request) Successful in 30s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:13:57 +02:00
TapTap 6f8847899c fix: validate remaining wire booleans
CI / lint (pull_request) Successful in 34s
CI / sanitizers (address) (pull_request) Successful in 35s
CI / fuzz-build (pull_request) Successful in 14s
CI / sanitizers (undefined) (pull_request) Successful in 35s
CI / build-and-test (pull_request) Successful in 1m14s
CI / coverage (pull_request) Successful in 31s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 12:49:30 +02:00
TapTap ff189aeef2 fix: harden network security boundaries
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 13s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 32s
2026-08-15 12:34:41 +02:00
17 changed files with 725 additions and 391 deletions
+321 -333
View File
@@ -1,382 +1,370 @@
# FastSync # FastSync
FastSync is a high-performance file synchronization tool designed to become a A high-performance file synchronization system with SSH and TCP transport, TLS encryption, streaming zstd compression, multithreaded transfer, incremental sync, metadata preservation, and rsync-compatible CLI flags.
drop-in replacement for common `rsync` workflows. It keeps the familiar
source/destination model and rsync-style options while adding optional
multithreading, streaming zstd compression, chunking, zero-copy TCP transfers,
and native TCP/TLS transports.
The compatibility target is straightforward: ## Technical Overview
- Existing rsync commands should keep the same meaning. 1. **Dual transport**: custom TCP client-server or SSH subprocess (rsync-style `user@host:/path`)
- FastSync-only performance options should be additive and optional. 2. **TLS encryption**: OpenSSL-based TLS 1.2+ for encrypted TCP connections with optional CA verification
- A normal compatibility-mode transfer should prioritize rsync filesystem 3. **Chunked file transfer**: files grouped into configurable-size chunks (default ~10 MB)
semantics over maximum throughput. 4. **Streaming zstd compression** (levels 122) using `ZSTD_compressStream2`
5. **Multithreading**: producer-consumer pipeline with thread-safe queues (scanner → loader → sender)
6. **Incremental sync**: skip files unchanged since last transfer (compares size + mtime)
7. **Batch incremental**: send incremental checks in batched groups for reduced round-trips
8. **Metadata preservation**: file mode and mtime are restored when enabled; ownership and atime are intentionally not restored
9. **`sendfile()` zero-copy** on TCP (~2× faster on loopback)
10. **SSH ControlMaster** for connection reuse across repeated invocations
11. **Bandwidth limiting**: token-bucket throttling (`--bwlimit`)
12. **`--delete`**: receiver removes files not present in sender manifest
13. **`--exclude` / `--include`**: glob-pattern filename filtering
14. **Path traversal protection**: `..` sequences in file paths are rejected automatically
15. **Connection limits**: server enforces maximum concurrent connections (default 100)
16. **Keep-alive**: periodic `STATUS_KEEPALIVE` messages detect stalled connections
17. **Abort handling**: `SIGINT` sends `STATUS_ABORT` for clean server-side teardown
18. **Atomic writes**: received files are written to a temporary name then atomically renamed
19. **Backup mode**: `--backup` preserves overwritten files with optional `--backup-dir`
20. **Log file**: `--log-file` redirects log output to a file instead of stderr
21. **Transfer statistics**: `--stats` prints summary of transferred bytes, files, and timing
FastSync currently speaks its own protocol to `fastsync-server`. SSH mode ## System Architecture
starts that server remotely; it does not yet interoperate with an unmodified
rsync client or rsync daemon. See [Compatibility Status](#compatibility-status)
for the current boundary.
## Why FastSync ### Client
- Recursively scans source directories (BFS), supports exclude and include patterns
- Groups files into chunks (configurable size)
- Streaming zstd compression with configurable level
- Chunk serialization (compact binary format) or per-file transfer
- Incremental transfer: sends file metadata to server, skips unchanged files
- Batch incremental: groups incremental checks to minimize round-trips
- Manifests all sent paths when `--delete` is active
- Sends via TCP `sendfile()` or SSH pipe
- Optional progress display with throughput
- Bandwidth limiting via token-bucket algorithm
- Configurable I/O and connection timeouts (`--timeout`, `--contimeout`)
- Backup overwritten files (`--backup`) with optional directory (`--backup-dir`)
- Transfer statistics summary (`--stats`)
- Maximum directory depth control (`--max-depth`)
- Log file output (`--log-file`)
- Exclude patterns from file (`--exclude-from`)
FastSync uses a producer-consumer transfer pipeline and can combine several ### Server
optimizations for large or high-latency transfers: - TCP mode: listens on configurable port (default 8080); SSH mode: runs via `--stdio`
- TLS mode: wraps TCP connections with OpenSSL with optional CA verification
- Receives and reassembles files
- Decompresses (streaming zstd), deserializes, restores metadata
- Handles incremental checks: compares size + mtime against destination files
- Handles batch incremental checks for reduced round-trips
- Processes `STATUS_MANIFEST` for `--delete`: walks destination tree, removes extras
- Per-connection concurrency via `fork()` with configurable connection limit (default 100)
- Thread pool for parallel processing
- Atomic writes: files written to `.tmp` path then atomically renamed on success
- Abort handling: cleanly shuts down on `STATUS_ABORT` from client
- Path traversal protection: rejects file paths containing `..`
- Multithreaded scanning, loading, and sending. ## Protocol Details
- Streaming zstd compression with levels 1 through 22.
- Configurable file chunking and compact chunk serialization.
- `sendfile()` zero-copy transfers over TCP.
- Batched incremental checks to reduce round trips.
- Optional block-level delta transfer for FastSync peers.
- Bandwidth limiting, progress reporting, statistics, and backups.
- TCP, SSH, and TLS transports.
- Atomic temporary-file writes by default.
These optimizations are disabled or selected independently. Users can start ### Status Codes
with rsync-style commands and add FastSync options when they are useful. | Code | Meaning |
|------|---------|
| `STATUS_OK` | Operation successful |
| `STATUS_ERROR` | Error occurred |
| `STATUS_FINISHED` | Transfer complete |
| `STATUS_NEXT` | Ready for next file (per-file mode) |
| `STATUS_CHUNK` | Following data is a serialized chunk |
| `STATUS_MANIFEST` | Following data is a file manifest (for `--delete`) |
| `STATUS_CHECK` | Incremental check: client sends file path + size + mtime and, when negotiated, checksum; server responds with OK (skip) or NEXT (send) |
| `STATUS_CHECK_BATCH` | Batch incremental check: multiple file checks sent in one message |
| `STATUS_KEEPALIVE` | Keep-alive heartbeat to detect stalled connections |
| `STATUS_ABORT` | Abort signal: client interrupts, server cleans up and exits |
| `STATUS_DELTA_SIGNATURE` | Delta sync: following data is a file signature (rsync-style rolling hash) |
| `STATUS_DELTA_DATA` | Delta sync: following data is a delta patch for a file |
## Compatibility Status ### Wire Format — Metadata
FastSync is currently an rsync-compatible CLI in progress, not a complete When `use_metadata` is enabled (`-M`), each file entry carries a 4-byte `present` flag followed by five fields (`mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`). When disabled globally, no metadata bytes are sent — zero wire overhead.
replacement for every rsync feature or protocol mode.
### Working today ### Transfer Flow
```
- Recursive directory scanning. Config → (STATUS_NEXT | STATUS_CHUNK | STATUS_CHECK | STATUS_CHECK_BATCH)* → [STATUS_MANIFEST] → STATUS_FINISHED → STATUS_OK
- Rsync-style source and destination arguments.
- SSH transport using `user@host:destination` paths below the remote authorized root.
- TCP client/server transfers.
- Dry runs, excludes, includes, size filters, backups, statistics, and
bandwidth limiting.
- Incremental size/mtime checks and optional xxHash64 content checks.
- FastSync-native delta transfer for changed files.
- Optional mode and timestamp preservation.
- Delete manifests with server-side delete authorization.
- Temporary-file writes with atomic rename by default.
- Path traversal checks and destination-root confinement.
### Not yet equivalent to rsync
- The FastSync wire protocol is not the rsync wire protocol.
- SSH mode requires `fastsync-server` on the remote host.
- Archive mode does not yet provide all of rsync's `-rlptgoD` behavior.
- Symlink transfer is incomplete; link targets are not yet recreated in all
modes.
- Owner/group, ACL, xattr, hard-link, device, and special-file handling is
incomplete or unavailable.
- Sparse-file handling does not yet preserve all holes correctly.
- `--partial`, `--partial-dir`, `--append`, and `--append-verify` are not yet
full rsync-style resumable transfers.
- Several rsync short options currently have FastSync-specific meanings. Do
not assume every short option is interchangeable yet.
The detailed flag matrix is maintained in
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It distinguishes implemented,
partial, alternate, and planned behavior.
## Quick Start
### Build
Requirements: C11 compiler, CMake 3.22 or newer, xxHash, zstd, OpenSSL,
pthreads, and an SSH client for SSH transport. The first CMake configure fetches
xxHash from GitHub, so network access is required unless the dependency is
already cached.
```bash
cmake -B build -S .
cmake --build build -j$(nproc)
``` ```
With Nix: Keep-alive (`STATUS_KEEPALIVE`) may be sent at any point during the transfer. The receiver resets its inactivity timer on receipt. If no data arrives within the receive timeout, the connection is aborted.
```bash Abort (`STATUS_ABORT`) may be sent at any point. On receipt the server cleans up temporary files and exits the child process.
nix-shell
cmake -B build -S .
cmake --build build -j$(nproc)
```
### SSH transfer ### Protocol Version
The remote host must have `fastsync-server` available in `PATH`, or use `2.2.0` — server and client must match. This version adds a 64-bit XXH64 checksum to checksum-enabled `STATUS_CHECK` messages and validates the negotiated compression choice (`zstd` or `none`). Older clients and servers must not be mixed with this version; mismatch results in `STATUS_ERROR`.
`--fastsync-server-path`. SSH starts `fastsync-server --stdio` in its remote
working directory, so use a destination below that directory unless the
remote server is otherwise configured with a matching authorized root.
```bash Config negotiation is sender-driven: the client serializes transfer options and the server applies them while receiving and writing files. `--checksum` compares size and content checksum instead of timestamps. `--compress-choice zstd` enables zstd; `none` disables it. Unsupported choices are rejected during config exchange.
./build/client /path/to/source user@host:destination
```
### TCP transfer ## Command-Line Arguments
Start the FastSync server:
```bash
./build/server --destination-root /path/to -p 8080
```
Then run the client:
```bash
./build/client --server-host 127.0.0.1 --server-port 8080 \
--source-dir /path/to/source --dest-dir /path/to/destination \
--save-to-disk
```
### TLS transfer
```bash
./build/server --destination-root /path/to --tls --cert server.pem --key server-key.pem -p 8443
./build/client --tls --cert client.pem --key client-key.pem --ca ca.pem \
--server-host example.com --server-port 8443 \
--source-dir /path/to/source --dest-dir /path/to/destination \
--save-to-disk
```
## Common Workflows
These examples show the intended rsync-style workflow. Options marked as
FastSync-native are optional performance or transport extensions.
```bash
# Basic synchronization
./build/client /source/ /destination/
# Archive-style synchronization (current FastSync archive behavior)
./build/client -a /source/ user@host:destination/
# Preview a transfer without changing the destination
./build/client -n /source/ /destination/
# Exclude temporary and object files
./build/client --exclude '*.tmp' --exclude '*.o' \
/source/ user@host:destination/
# Remove destination entries not present in the source
./build/client --delete /source/ user@host:destination/
# Skip unchanged files using size and modification time
./build/client --incremental /source/ user@host:destination/
# Verify content when size and time are not sufficient
./build/client --incremental --checksum /source/ user@host:destination/
# Preserve supported mode and timestamp metadata
./build/client -M /source/ user@host:destination/
# Keep backups of overwritten destination files
./build/client --backup --backup-dir backups \
/source/ user@host:destination/
```
## FastSync Extensions
FastSync-native options are intended to add performance or operational
features without changing the meaning of ordinary compatibility options.
| Option | Purpose |
|---|---|
| `-m` | Enable the multithreaded scanner/loader/sender pipeline. |
| `-c [level]`, `-z [level]` | Enable streaming zstd compression, levels 1-22. |
| `--compress-level <n>` | Set the zstd compression level. |
| `--chunk-size <bytes>` | Set the transfer chunk size. |
| `-s` | Enable FastSync chunk serialization. |
| `-f`, `--sendfile` | Use TCP `sendfile()` zero-copy transfer. Incompatible with compression and chunk serialization. |
| `--delta` | Use FastSync-native block delta transfer. Requires `--incremental`. |
| `--delta-block <bytes>` | Set the FastSync delta block size. |
| `--delta-max <bytes>` | Limit files eligible for FastSync delta transfer. |
| `--server-host <host>` | Select the TCP server host. |
| `--server-port <port>` | Select the TCP server port. |
| `--tls` | Enable TLS for TCP transport. |
| `--bwlimit <KB/s>` | Apply token-bucket bandwidth limiting. |
| `--progress` | Show transfer progress and throughput. |
| `--stats` | Print transfer statistics. |
| `--timeout <seconds>` | Set I/O timeout. |
| `--contimeout <seconds>` | Set connection timeout. |
Current short-option conflicts are tracked as compatibility work. In
particular, FastSync currently uses `-p` for SSH port, `-s` for chunk
serialization, and `-S` for sparse handling. These meanings must be reconciled
before FastSync can claim full rsync CLI compatibility.
## Client Options
### Selection and transfer
| Option | Description |
|---|---|
| `-a`, `--archive` | Enable current archive preset. Full rsync archive semantics are planned. |
| `-n`, `--dry-run` | Scan and report without writing files. |
| `--delete` | Request removal of destination entries absent from the source. The server must allow deletion. |
| `--exclude <pattern>` | Exclude matching paths. Repeatable. |
| `--include <pattern>` | Include matching paths. Repeatable. |
| `--exclude-from <file>` | Read exclude patterns from a file. |
| `--include-from <file>` | Read include patterns from a file. |
| `--max-size <bytes>` | Skip files larger than the limit. |
| `--min-size <bytes>` | Skip files smaller than the limit. |
| `--max-depth <n>` | Limit recursive scanning depth; zero means unlimited. |
| `--incremental` | Skip files matching destination size and mtime. |
| `--checksum` | Include xxHash64 content checks in incremental comparisons. |
| `--backup` | Back up overwritten files. |
| `--backup-dir <dir>` | Store backups under a separate directory. |
| `--suffix <suffix>` | Set the backup filename suffix. |
| `--partial` | Select partial-transfer handling. With `--partial-dir`, completed files are written there; resumable transfers are not implemented. |
| `--partial-dir <dir>` | Set a relative partial-transfer directory below the server destination root; use with `--partial`. |
| `--inplace` | Write directly to the destination instead of using a temporary file. |
### Metadata and links
| Option | Description |
|---|---|
| `-M`, `--preserve` | Preserve supported file metadata, currently mode and modification time. |
| `-l`, `--links` | Request symlink preservation; link-target transfer remains incomplete. |
| `--copy-links` | Copy symlink referents. |
| `--safe-links` | Skip symlinks that point outside the transfer tree. |
| `--copy-unsafe-links` | Copy unsafe symlink referents. |
| `-S`, `--sparse` | Request sparse-file handling; full hole preservation is planned. |
### Output and logging
| Option | Description |
|---|---|
| `-v`, `--verbose` | Enable debug logging. |
| `--progress` | Show live transfer progress. |
| `--stats` | Print transfer statistics. |
| `--log-file <path>` | Write log output to a file. |
| `-V`, `--version` | Print the FastSync protocol version. |
| `--help` | Print command usage. |
### Paths and transport
| Option | Description |
|---|---|
| `-p <port>` | SSH port in the current CLI. This conflicts with rsync's `-p` permissions option and is planned for correction. |
| `--fastsync-server-path <path>` | Remote FastSync server path for SSH mode. |
| `--source-dir <path>` | Set the source directory explicitly. |
| `--dest-dir <path>` | Set the destination directory explicitly. |
| `--save-to-disk` | Enable server-side disk persistence. |
| `--server-host <host>` | TCP server address. |
| `--server-port <port>` | TCP server port. |
| `--tls` | Enable TLS. Requires `--cert` and `--key`. |
| `--cert <path>` | TLS certificate file. |
| `--key <path>` | TLS private key file. |
| `--ca <path>` | CA file for peer verification. |
## Server Options
| Option | Description |
|---|---|
| `--stdio` | Serve one SSH connection over standard input/output. |
| `-p <port>` | TCP listen port. |
| `--tls` | Enable TLS. |
| `--cert <path>` | TLS certificate file. |
| `--key <path>` | TLS private key file. |
| `--ca <path>` | CA file for peer verification. |
| `--destination-root <path>` | Confine received files to this server-side root; defaults to the current directory. |
| `--allow-delete` | Permit client delete manifests. Deletion is refused by default. |
| `-v`, `--verbose` | Enable debug logging. |
| `--help` | Print server usage. |
## Architecture
### Client ### Client
- Recursively scans the source tree with include, exclude, size, and depth | Argument | Description |
filters. |----------|-------------|
- Sends individual files or serialized chunks. | Positional | `<source> <dest>` — automatic SSH detection if dest contains `:` |
- Performs incremental checks and optional content checksums. | `-c [level]` | Compression with optional level (122, default 5) |
- Uses a multithreaded producer-consumer pipeline when requested. | `-z [level]` | Alias for `-c` |
- Sends over TCP, TLS-wrapped TCP, or an SSH subprocess. | `-a, --archive` | Archive mode: enables `-c -m -M` (no `-s`) |
- Supports progress, statistics, backups, timeouts, and bandwidth limiting. | `-m` | Multithreading mode |
| `-s` | Chunk serialization (batch all files per chunk) |
| `-f, --sendfile` | Sendfile zero-copy. Incompatible with `-c` / `-s`. TCP only. |
| `-M, --preserve` | Preserve supported file metadata (mode and mtime; ownership and atime are unsupported) |
| `-n, --dry-run` | Scan and print what would be transferred |
| `-p <port>` | SSH port (default: 22) |
| `-v, --verbose` | Enable debug logging |
| `--progress` | Show real-time transfer speed |
| `--delete` | Delete files on receiver not present in source |
| `--exclude <pattern>` | Exclude files matching glob pattern (repeatable) |
| `--exclude-from <file>` | Read exclude patterns from a file (one per line) |
| `--include <pattern>` | Only transfer files matching glob pattern (repeatable, whitelist) |
| `--max-size <n>` | Skip files larger than n bytes |
| `--min-size <n>` | Skip files smaller than n bytes |
| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `-s`. |
| `--bwlimit <KB/s>` | Bandwidth limit in kilobytes per second |
| `--chunk-size <n>` | Chunk size in bytes (default: 10485760) |
| `--timeout <sec>` | I/O timeout in seconds (default: 30) |
| `--contimeout <sec>` | Connection timeout in seconds (default: 10) |
| `--backup` | Backup existing destination files before overwriting |
| `--backup-dir <dir>` | Target directory for backups (requires `--backup`) |
| `--stats` | Print transfer statistics at end (bytes, files, timing) |
| `--max-depth <n>` | Maximum directory depth to recurse (0 = unlimited, default: 0) |
| `--log-file <path>` | Write log messages to file instead of stderr |
| `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
| `--dest-dir <path>` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) |
| `--save-to-disk` | Write received files to disk |
| `--server-host <ip>` | Server IP address (default: `127.0.0.1`) |
| `--server-port <n>` | Server port (default: `8080`) |
| `--tls` | Enable TLS encryption |
| `--cert <path>` | TLS certificate file (PEM) |
| `--key <path>` | TLS private key file (PEM) |
| `--ca <path>` | TLS CA certificate file for verification (PEM) |
| `--client-cn <name>` | Required TLS client certificate common name |
### Server ### Server
- Runs as a TCP listener or one-shot SSH `--stdio` server. | Argument | Description |
- Receives and reassembles files and decompresses streaming zstd data. |----------|-------------|
- Applies supported metadata and writes files through a confined destination | `--stdio` | Run in stdio mode (for SSH transport; single connection then exits) |
root. | `-p <port>` | TCP listen port (default: 8080, range: 165535) |
- Uses temporary files and atomic rename by default. | `--tls` | Enable TLS encryption |
- Handles delete manifests only when explicitly authorized. | `--cert <path>` | TLS certificate file (PEM) |
- Enforces connection, message-size, and path-safety limits. | `--key <path>` | TLS private key file (PEM) |
| `--ca <path>` | TLS CA certificate file for verification (PEM) |
| `--destination-root <path>` | Authorized destination root (default: `.`) |
| `--allow-delete` | Permit manifest deletion |
| `--allow-unauthenticated` | Permit plaintext TCP clients |
| `-v, --verbose` | Enable debug logging |
| `--help` | Show help |
## Protocol and Security ## Environment Variables
FastSync protocol version `2.2.0` is shared by the client and server. The | Variable | Default | Description |
current protocol is sender-driven and includes configuration negotiation, |----------|---------|-------------|
incremental checks, checksums, manifests, keep-alives, abort handling, and | `FASTSYNC_SOURCE_DIR` | — | Source directory fallback |
FastSync-native delta messages. Client and server versions must currently | `FASTSYNC_DEST_DIR` | — | Destination directory fallback |
match exactly. | `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback |
| `FASTSYNC_SSH_PORT` | `22` | Default SSH port |
| `FASTSYNC_SERVER_HOST` | `127.0.0.1` | Default server host |
| `FASTSYNC_SERVER_PORT` | `8080` | Default server port |
| `FASTSYNC_TLS_CERT` | — | Default TLS certificate path |
| `FASTSYNC_TLS_KEY` | — | Default TLS private key path |
| `FASTSYNC_TLS_CA` | — | Default TLS CA certificate path |
TLS provides encrypted TCP transport. Supplying `--ca` enables certificate ## Implementation Details
verification; without it, traffic is encrypted but peer identity is not
verified. Use certificate verification for deployments where authentication
matters. The default TCP transport is not encrypted.
The receiver protects its destination root with path validation, `openat()` ### Data Structures
directory traversal, `O_NOFOLLOW`, temporary files, and atomic renames. Delete 1. **Chunk** — collection of files (~10 MB total by default)
operations require the server's explicit `--allow-delete` policy. 2. **File** — path, content (`Data`), optional `FileMetadata` pointer
3. **FileMetadata**`mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`; uid/gid are advisory wire fields and are never applied by the receiver; atime is unsupported
4. **Config** — runtime parameters (transported over wire, TLS settings excluded). Includes `timeout`, `contimeout`, `quiet`, `backup`, `backup_dir`, `stats`, `max_depth`, `log_file`, `queue_size`.
5. **Queue** — thread-safe bounded queue with condition variables
6. **DirectoryScanner** — recursive BFS traversal with exclude and include pattern support, max-depth enforcement
## Compatibility Roadmap ### Key Algorithms
1. **File scanning** — BFS directory traversal; entries matched against exclude and include patterns, max-depth enforced
2. **Chunking** — files accumulated until `chunk_size` threshold, then flushed
3. **Compression** — streaming zstd via `ZSTD_compressStream2` / `ZSTD_decompressStream`
4. **Network protocol** — status-code-driven exchange with metadata packing, keep-alive, and abort support
5. **Incremental check** — client sends `STATUS_CHECK` + path + size + mtime and, with `--checksum`, XXH64 content checksum; server compares against destination. Can be batched via `STATUS_CHECK_BATCH` for reduced round-trips.
6. **Bandwidth limiting** — token-bucket algorithm with `nanosleep` throttling on 64 KB write chunks
7. **Metadata restoration**`chmod()`, `chown()`, `utimensat()` on the receiving side
8. **`--delete`** — sender tracks all sent paths; receiver walks destination tree and removes unlisted files/directories
9. **SSH transport**`socketpair()` + `fork()` + `execvp("ssh", ...)` with `ControlMaster` and port support
10. **TLS transport** — OpenSSL `SSL_CTX` with TLS 1.2 minimum, mutual CA verification, transparent `SSL_read`/`SSL_write` via `io_set_ssl()`
11. **Path traversal protection**`has_path_traversal()` rejects any file path containing `..` components, preventing directory escape attacks
12. **Connection limiting** — server tracks active connections and rejects new ones beyond `max_connections` (default 100)
13. **Keep-alive** — idle connections receive periodic `STATUS_KEEPALIVE` to detect half-open TCP connections
14. **Abort handling**`SIGINT` sets an abort flag; the next protocol operation sends `STATUS_ABORT` for clean server cleanup
15. **Atomic writes** — files are written to a `.tmp` suffix then atomically renamed via `rename()`, preventing partial files
16. **Backup** — before overwriting, existing files are moved to `--backup-dir` (or same directory with `~` suffix) preserving the original
The project will reach the drop-in replacement goal in stages: ## Security Features
1. Correct rsync option meanings, including short options, combined options, ### Path Traversal Protection
and `--option=value` syntax. All received file paths are validated by `has_path_traversal()` before any disk operation. Any path containing `..` components is rejected with `STATUS_ERROR`, preventing directory escape attacks.
2. Add differential tests that compare FastSync and rsync contents, metadata,
links, deletes, filters, dry runs, and exit codes.
3. Make `-a` implement the expected recursive, links, permissions, times,
owner/group, and supported special-file behavior.
4. Complete symlink, sparse-file, metadata, delete-policy, and resumable-write
semantics.
5. Add rsync remote-shell and daemon protocol interoperability.
6. Keep FastSync performance options as negotiated, optional extensions.
The exhaustive implementation matrix and compatibility notes are in ### TLS Certificate Verification
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). TLS requires `--ca` and performs mutual TLS verification (`SSL_VERIFY_PEER` with depth 4). Connections without certificate verification are rejected.
### Connection Limits
The server enforces a maximum of 100 concurrent connections (configurable via `max_connections` in `Server`). When the limit is reached, new connections are immediately rejected and closed.
### Abort Handling
If the client receives `SIGINT` (Ctrl+C) during a transfer, it sends `STATUS_ABORT` to the server. The server then cleans up temporary files and exits the child process, preventing incomplete files from remaining on disk.
### Atomic Writes
Received files are written to a temporary path (suffixed with `.tmp`) and then atomically renamed to the final filename via `rename()`. This prevents partial or corrupted files from appearing at the destination if the transfer is interrupted.
## Build Requirements
- C11 compiler
- CMake >= 3.22
- zstd library
- OpenSSL (development headers and libraries)
- pthreads
- SSH client (for SSH transport mode only)
### Installing Dependencies
**Ubuntu/Debian:**
```bash
sudo apt install cmake build-essential libzstd-dev libssl-dev openssh-client
```
**Nix:**
```bash
nix-shell # provides zstd, openssl, cmake, gcc
```
## Building
```bash
cmake -B build -S . && cmake --build build -j$(nproc)
```
## Running
### Server (TCP mode)
```bash
./build/server --allow-unauthenticated
```
### Server with TLS
```bash
./build/server --tls --cert server.pem --key server-key.pem --ca ca.pem --client-cn fastsync-client
```
### Server via SSH
Place the `fastsync-server` binary in the remote `$PATH`. The client runs `ssh user@host fastsync-server --stdio` automatically when an SSH-style destination is given.
### Client — SSH (rsync-style)
```bash
./build/client /path/to/send user@host:/path/to/receive
```
### Client — TCP
```bash
./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk
```
Plain TCP requires the explicit `--allow-unauthenticated` server option. Use TLS for
authenticated network connections.
### Client — TCP with TLS
```bash
./build/client --tls --cert client.pem --key client-key.pem --ca ca.pem \
--source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk
```
### Common Options
```bash
# Archive mode (compression + multithreading + metadata)
./build/client -a /path/to/send user@host:/path
# Dry run
./build/client -n /path/to/send /path/to/receive
# With progress and custom chunk size
./build/client --progress --chunk-size 2097152 /src user@host:/dst
# Exclude temporary files + delete extras on receiver
./build/client --exclude "*.tmp" --exclude "*.o" --delete /src user@host:/dst
# Incremental sync (skip unchanged files)
./build/client --incremental /src user@host:/dst
# Bandwidth limit to 1 MB/s
./build/client --bwlimit 1024 /src user@host:/dst
# With timeouts and stats
./build/client --timeout 60 --contimeout 15 --stats /src user@host:/dst
# Backup overwritten files to a directory
./build/client --backup --backup-dir /backups /src user@host:/dst
# Exclude patterns from file, limit depth
./build/client --exclude-from ignore.txt --max-depth 3 /src user@host:/dst
# Log to file
./build/client --log-file /tmp/fastsync.log /src user@host:/dst
# All features
./build/client -a --progress --chunk-size 5242880 --exclude "*.log" --delete /src /dst
```
## Testing ## Testing
Run the unit test binary:
```bash ```bash
# Unit tests (18 suites — array_list, chunk, compression, config, data, delta, file, glob,
# metadata, property, protocol, queue, robustness, scanner,
# shared_utils, stress, transport_tcp, transport_ssh, transport_tls)
./build/tests ./build/tests
# Integration + benchmark suite
python3 test.py
``` ```
Run the Python integration suite: The benchmark prints throughput metrics, best configuration, and speedup vs rsync.
```bash ## Performance Considerations
python3 -m pytest tests/
```
For stricter local validation: 1. Chunk size (~10 MB default) balances memory and transfer efficiency
2. Compression level trades CPU for bandwidth
3. `sendfile()` bypasses userspace — ~2× faster on localhost for large files
4. Multithreading scales with core count and uses memory-based pipeline sizing
5. Metadata transfer adds negligible overhead (~24 bytes per file when enabled)
6. SSH socketpair buffer set to 1 MB for improved pipe throughput
7. SSH ControlMaster reuses connections across repeated invocations
8. Incremental sync eliminates redundant transfers entirely
9. Batch incremental reduces round-trips by grouping multiple checks into one message
10. Bandwidth limiting uses token-bucket with nanosleep for accurate throttling
11. Atomic writes add a single `rename()` per file — negligible overhead
12. Path traversal check is O(n) in path length with negligible cost
```bash ## Benchmark Results
cmake -B build-strict -S . -DSTRICT_WARNINGS=ON
cmake --build build-strict -j$(nproc)
cmake -B build-asan -S . -DSANITIZER=address
cmake --build build-asan -j$(nproc)
```
The benchmark tool compares FastSync configurations with rsync under 25 MB of mixed file sizes over `localhost` with disk I/O throttled (reads ≤ 15 MB/s, writes ≤ 10 MB/s) and network emulation via `tc netem`. Each test was run 3×; the median is reported below.
controlled local and network conditions:
```bash ### LAN (1000 Mbit, 20 ms ±1 ms, 0.1% loss)
python3 benchmark/bench.py --help
```
Benchmark results measure transfer performance only. They do not establish | Configuration | Time | vs rsync (archive) | vs rsync (compress) |
rsync protocol or filesystem-semantic compatibility. |---|---|---|---|
| **Best: `-m -c`** | **0.20 s** | **11.2× faster** | **3.6× faster** |
| Compression (`-c`) | 0.31 s | 7.3× faster | 2.3× faster |
| Standard | 1.27 s | 1.8× faster | — |
| rsync (archive) | 2.27 s | — | — |
| rsync (archive + compress) | 0.72 s | — | — |
## Performance Guidance ### WAN (100 Mbit, 50 ms ±10 ms, 1% loss)
- Use `-m` for workloads with many files or enough CPU parallelism. | Configuration | Time | vs rsync (archive) | vs rsync (compress) |
- Use `-c` or `-z` when network bandwidth is more constrained than CPU. |---|---|---|---|
- Tune `--chunk-size` for file sizes, memory limits, and network latency. | **Best: `-m -c`** | **0.39 s** | **44.8× faster** | **3.8× faster** |
- Use `-f` for large uncompressed TCP transfers where zero-copy I/O helps. | Compression (`-c`) | 0.64 s | 27.3× faster | 2.3× faster |
- Use `--incremental` to avoid retransmitting unchanged files. | Standard | 7.12 s | 2.4× faster | — |
- Use `--delta` for changed files when both endpoints are FastSync peers. | rsync (archive) | 17.44 s | — | — |
- Use `--bwlimit` when sharing a link with other traffic. | rsync (archive + compress) | 1.47 s | — | — |
Always validate the compatibility behavior required by a deployment before Compression reduces the data on the wire enough that the transfer becomes latency-bound rather than bandwidth-bound. On WAN, the best configuration runs 10.8× faster than the theoretical limit for uncompressed data, since zstd shrinks the 25 MB payload to a fraction of its original size over the wire.
replacing an existing rsync job.
+2 -2
View File
@@ -152,14 +152,14 @@ This document maps rsync's full feature set to FastSync's current implementation
| Flag | Rsync Description | FastSync Status | Notes | | Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `-S`, `--sparse` | Sparse block handling | ⚠️ Partial | Flag is accepted, but full hole preservation is not implemented | | `-S`, `--sparse` | Sparse block handling | ✅ Implemented | `preserve_sparse` config field |
| `--preallocate` | Allocate dest files before writing | ❌ Not Implemented | | | `--preallocate` | Allocate dest files before writing | ❌ Not Implemented | |
## 11. Checksum & Comparison ## 11. Checksum & Comparison
| Flag | Rsync Description | FastSync Status | Notes | | Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `--checksum` | Skip based on checksum | Implemented | With `--incremental`, compares xxHash64 content checksums; `-c` remains compression | | `--checksum` | Skip based on checksum | ❌ Not Implemented | Removed because it had no effect; `-c` means compression |
| `--checksum-choice=STR` | Choose checksum algorithm | ❌ Not Implemented | xxHash used internally | | `--checksum-choice=STR` | Choose checksum algorithm | ❌ Not Implemented | xxHash used internally |
| `--compare-dest=DIR` | Compare dest files relative to DIR | ❌ Not Implemented | Removed because it had no effect | | `--compare-dest=DIR` | Compare dest files relative to DIR | ❌ Not Implemented | Removed because it had no effect |
| `--copy-dest=DIR` | Include copies of unchanged files | ❌ Not Implemented | Removed because it had no effect | | `--copy-dest=DIR` | Include copies of unchanged files | ❌ Not Implemented | Removed because it had no effect |
+2 -2
View File
@@ -400,8 +400,8 @@ static bool validate_config(const Config* config) {
return false; return false;
} }
if (config->use_tls) { if (config->use_tls) {
if (!config->tls_cert || !config->tls_key) { if (!config->tls_cert || !config->tls_key || !config->tls_ca) {
fprintf(stderr, "Error: --tls requires --cert and --key\n"); fprintf(stderr, "Error: --tls requires --cert, --key, and --ca\n");
return false; return false;
} }
} }
+44 -6
View File
@@ -19,10 +19,28 @@
#include <fcntl.h> #include <fcntl.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <unistd.h> #include <unistd.h>
#include <openssl/x509.h>
static char* authorized_root; static char* authorized_root;
static int authorized_root_fd = -1; static int authorized_root_fd = -1;
static bool allow_delete; static bool allow_delete;
static bool allow_unauthenticated;
static const char* required_client_cn;
static bool tls_client_identity_allowed(SSL* ssl) {
if (!ssl || !required_client_cn)
return false;
X509* certificate = SSL_get1_peer_certificate(ssl);
if (!certificate)
return false;
char common_name[256];
int length = X509_NAME_get_text_by_NID(X509_get_subject_name(certificate), NID_commonName,
common_name, sizeof(common_name));
bool allowed = length >= 0 && (size_t)length < sizeof(common_name) &&
strcmp(common_name, required_client_cn) == 0;
X509_free(certificate);
return allowed;
}
static bool path_is_within(const char* root, const char* path) { static bool path_is_within(const char* root, const char* path) {
size_t n = strlen(root); size_t n = strlen(root);
@@ -48,7 +66,7 @@ static bool __attribute__((unused)) configure_authorization(const char* root) {
return false; return false;
} }
file_set_authorized_root(authorized_root_fd, authorized_root); file_set_authorized_root(authorized_root_fd, authorized_root);
utils_set_authorized_root_fd(authorized_root_fd); utils_set_authorized_root(authorized_root_fd, authorized_root);
return true; return true;
} }
@@ -118,9 +136,9 @@ int receive_files(Config* config, int fd) {
send_status(fd, STATUS_ERROR); send_status(fd, STATUS_ERROR);
return -1; return -1;
} }
char* full_path = path_cat(config->receive_root_directory, check_path);
struct stat st; struct stat st;
bool has_old = full_path && lstat(full_path, &st) == 0; char* full_path = path_cat(config->receive_root_directory, check_path);
bool has_old = full_path && file_stat_secure(full_path, &st);
bool match = has_old && (unsigned long long)st.st_size == check_size && bool match = has_old && (unsigned long long)st.st_size == check_size &&
(long long)st.st_mtime == check_mtime; (long long)st.st_mtime == check_mtime;
bool sent = send_status(fd, match ? STATUS_OK : STATUS_NEXT); bool sent = send_status(fd, match ? STATUS_OK : STATUS_NEXT);
@@ -179,6 +197,18 @@ void handler(int file_descriptor) {
close(file_descriptor); close(file_descriptor);
return; return;
} }
if (!allow_unauthenticated && ssl == NULL) {
log_message(LOG_LEVEL_ERROR, "Rejected unauthenticated plaintext connection");
config_delete(config);
close(file_descriptor);
return;
}
if (ssl && required_client_cn && !tls_client_identity_allowed(ssl)) {
log_message(LOG_LEVEL_ERROR, "Rejected TLS client with unauthorized identity");
config_delete(config);
close(file_descriptor);
return;
}
char resolved_destination[PATH_MAX]; char resolved_destination[PATH_MAX];
char* canonical_destination = realpath(config->receive_root_directory, NULL); char* canonical_destination = realpath(config->receive_root_directory, NULL);
const char* destination = const char* destination =
@@ -281,8 +311,10 @@ static void print_server_usage(void) {
printf(" --cert <path> TLS certificate file (PEM)\n"); printf(" --cert <path> TLS certificate file (PEM)\n");
printf(" --key <path> TLS private key file (PEM)\n"); printf(" --key <path> TLS private key file (PEM)\n");
printf(" --ca <path> TLS CA certificate file (PEM)\n"); printf(" --ca <path> TLS CA certificate file (PEM)\n");
printf(" --client-cn <name> Required TLS client certificate CN\n");
printf(" --destination-root <path> Authorized destination root (default: .)\n"); printf(" --destination-root <path> Authorized destination root (default: .)\n");
printf(" --allow-delete Permit manifest deletion\n"); printf(" --allow-delete Permit manifest deletion\n");
printf(" --allow-unauthenticated Allow plaintext/anonymous network clients\n");
printf(" -v, --verbose Enable debug logging\n"); printf(" -v, --verbose Enable debug logging\n");
printf(" --help Show this help\n"); printf(" --help Show this help\n");
} }
@@ -313,10 +345,14 @@ int main(int argc, char* argv[]) {
tls_key = argv[++i]; tls_key = argv[++i];
} else if (strcmp(argv[i], "--ca") == 0 && i + 1 < argc) { } else if (strcmp(argv[i], "--ca") == 0 && i + 1 < argc) {
tls_ca = argv[++i]; tls_ca = argv[++i];
} else if (strcmp(argv[i], "--client-cn") == 0 && i + 1 < argc) {
required_client_cn = argv[++i];
} else if (strcmp(argv[i], "--destination-root") == 0 && i + 1 < argc) { } else if (strcmp(argv[i], "--destination-root") == 0 && i + 1 < argc) {
destination_root = argv[++i]; destination_root = argv[++i];
} else if (strcmp(argv[i], "--allow-delete") == 0) { } else if (strcmp(argv[i], "--allow-delete") == 0) {
allow_delete = true; allow_delete = true;
} else if (strcmp(argv[i], "--allow-unauthenticated") == 0) {
allow_unauthenticated = true;
} else if (strcmp(argv[i], "-p") == 0 && i + 1 < argc) { } else if (strcmp(argv[i], "-p") == 0 && i + 1 < argc) {
char* end; char* end;
long p = strtol(argv[++i], &end, 10); long p = strtol(argv[++i], &end, 10);
@@ -343,10 +379,12 @@ int main(int argc, char* argv[]) {
return 1; return 1;
} }
if (stdio_mode) { if (stdio_mode) {
/* SSH authenticates the stdio transport outside of FastSync. */
allow_unauthenticated = true;
io_set_fds(STDIN_FILENO, STDOUT_FILENO); io_set_fds(STDIN_FILENO, STDOUT_FILENO);
handler(STDIN_FILENO); handler(STDIN_FILENO);
file_set_authorized_root(-1, NULL); file_set_authorized_root(-1, NULL);
utils_set_authorized_root_fd(-1); utils_set_authorized_root(-1, NULL);
close(authorized_root_fd); close(authorized_root_fd);
free(authorized_root); free(authorized_root);
return 0; return 0;
@@ -357,8 +395,8 @@ int main(int argc, char* argv[]) {
return 1; return 1;
} }
if (use_tls) { if (use_tls) {
if (!tls_cert || !tls_key) { if (!tls_cert || !tls_key || !tls_ca || !required_client_cn) {
fprintf(stderr, "Error: --tls requires --cert and --key\n"); fprintf(stderr, "Error: --tls requires --cert, --key, --ca, and --client-cn\n");
server_delete(&g_server); server_delete(&g_server);
return 1; return 1;
} }
+92 -14
View File
@@ -1,4 +1,6 @@
#include <stddef.h> #include <stddef.h>
#include <stdint.h>
#include <limits.h>
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
@@ -14,18 +16,29 @@
/* Maximum individual file data size within a chunk (64 MB) */ /* Maximum individual file data size within a chunk (64 MB) */
#define MAX_FILE_DATA_SIZE (64ULL * 1024 * 1024) #define MAX_FILE_DATA_SIZE (64ULL * 1024 * 1024)
#define MAX_FILES_PER_CHUNK 65536U
Chunk* chunk_create(File** items, int element_count) { Chunk* chunk_create(File** items, int element_count) {
if (element_count < 0 || (element_count > 0 && items == NULL))
return NULL;
Chunk* chunk = (Chunk*)malloc(sizeof(Chunk)); Chunk* chunk = (Chunk*)malloc(sizeof(Chunk));
if (chunk == NULL) { if (chunk == NULL) {
perror("ERROR: Could not allocate memory for chunk structure"); perror("ERROR: Could not allocate memory for chunk structure");
return NULL; return NULL;
} }
chunk->items = (File**)malloc(element_count * sizeof(File*)); if (element_count == 0) {
if (chunk->items == NULL) { chunk->items = NULL;
free(chunk); } else {
return NULL; if ((size_t)element_count > SIZE_MAX / sizeof(File*)) {
free(chunk);
return NULL;
}
chunk->items = (File**)malloc((size_t)element_count * sizeof(File*));
if (chunk->items == NULL) {
free(chunk);
return NULL;
}
} }
for (int i = 0; i < element_count; i++) { for (int i = 0; i < element_count; i++) {
@@ -50,15 +63,31 @@ void chunk_destroy(void* item) {
} }
static unsigned long long per_file_serialize_size(File* file, bool use_metadata) { static unsigned long long per_file_serialize_size(File* file, bool use_metadata) {
return sizeof(size_t) + strlen(file->path) + unsigned long long size = sizeof(size_t);
(use_metadata ? sizeof(int) + (file->metadata ? FILE_METADATA_WIRE_SIZE : 0) : 0) + size_t path_len = strlen(file->path);
sizeof(size_t) + file->data->size; unsigned long long metadata_size =
use_metadata ? sizeof(int) + (file->metadata ? FILE_METADATA_WIRE_SIZE : 0) : 0;
if ((unsigned long long)path_len > ULLONG_MAX - size)
return 0;
size += path_len;
if (metadata_size > ULLONG_MAX - size)
return 0;
size += metadata_size;
if (sizeof(size_t) > ULLONG_MAX - size)
return 0;
size += sizeof(size_t);
if ((unsigned long long)file->data->size > ULLONG_MAX - size)
return 0;
return size + file->data->size;
} }
Data* chunk_serialize(Chunk* chunk, bool use_metadata) { Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
unsigned long long data_size = 0; unsigned long long data_size = 0;
for (int i = 0; i < chunk->element_count; i++) { for (int i = 0; i < chunk->element_count; i++) {
data_size += per_file_serialize_size(chunk->items[i], use_metadata); unsigned long long file_size = per_file_serialize_size(chunk->items[i], use_metadata);
if (file_size == 0 || file_size > ULLONG_MAX - data_size || data_size + file_size > SIZE_MAX)
return NULL;
data_size += file_size;
} }
Data* data = data_create_empty(data_size); Data* data = data_create_empty(data_size);
if (data == NULL) { if (data == NULL) {
@@ -88,10 +117,17 @@ Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
Chunk* chunk_deserialize(Data* data, bool use_metadata) { Chunk* chunk_deserialize(Data* data, bool use_metadata) {
ArrayList* files = array_list_create(file_destroy); ArrayList* files = array_list_create(file_destroy);
if (files == NULL)
return NULL;
char* data_pointer = data->data; char* data_pointer = data->data;
size_t remaining_size = data->size; size_t remaining_size = data->size;
while (remaining_size > 0) { while (remaining_size > 0) {
if ((unsigned int)files->size >= MAX_FILES_PER_CHUNK) {
log_message(LOG_LEVEL_ERROR, "Chunk contains too many files");
array_list_delete(files);
return NULL;
}
if (remaining_size < sizeof(size_t)) { if (remaining_size < sizeof(size_t)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for path length"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for path length");
array_list_delete(files); array_list_delete(files);
@@ -109,6 +145,10 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
return NULL; return NULL;
} }
if (path_len == SIZE_MAX) {
array_list_delete(files);
return NULL;
}
char* path = malloc(path_len + 1); char* path = malloc(path_len + 1);
if (path == NULL) { if (path == NULL) {
perror("Could not allocate memory for file path"); perror("Could not allocate memory for file path");
@@ -117,34 +157,53 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
} }
memcpy(path, data_pointer, path_len); memcpy(path, data_pointer, path_len);
path[path_len] = '\0'; path[path_len] = '\0';
if (memchr(path, '\0', path_len) != NULL) {
free(path);
array_list_delete(files);
return NULL;
}
data_pointer += path_len; data_pointer += path_len;
remaining_size -= path_len; remaining_size -= path_len;
File* file = file_create(path); File* file = file_create(path);
free(path); free(path);
if (file == NULL) {
array_list_delete(files);
return NULL;
}
if (use_metadata) { if (use_metadata) {
if (remaining_size < sizeof(int)) { if (remaining_size < sizeof(int)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
// Peek at present flag to determine total size needed before reading // Peek at present flag to determine total size needed before reading
int present_flag; int present_flag;
memcpy(&present_flag, data_pointer, sizeof(int)); memcpy(&present_flag, data_pointer, sizeof(int));
if (present_flag && remaining_size < sizeof(int) + FILE_METADATA_WIRE_SIZE) { if ((present_flag != 0 && present_flag != 1) ||
(present_flag == 1 && remaining_size < sizeof(int) + FILE_METADATA_WIRE_SIZE)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata body"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata body");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
file->metadata = metadata_from_buf(&data_pointer); file->metadata = metadata_from_buf(&data_pointer);
remaining_size -= sizeof(int); remaining_size -= sizeof(int);
if (file->metadata) if (present_flag == 1) {
if (file->metadata == NULL) {
file_destroy(file);
array_list_delete(files);
return NULL;
}
remaining_size -= FILE_METADATA_WIRE_SIZE; remaining_size -= FILE_METADATA_WIRE_SIZE;
}
} }
if (remaining_size < sizeof(size_t)) { if (remaining_size < sizeof(size_t)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for data size"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for data size");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
@@ -156,6 +215,7 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
if (remaining_size < file_data_size) { if (remaining_size < file_data_size) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for file content"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for file content");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
@@ -164,30 +224,48 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
if (file_data_size > MAX_FILE_DATA_SIZE) { if (file_data_size > MAX_FILE_DATA_SIZE) {
log_message(LOG_LEVEL_ERROR, "File data size %zu exceeds maximum %llu", file_data_size, log_message(LOG_LEVEL_ERROR, "File data size %zu exceeds maximum %llu", file_data_size,
(unsigned long long)MAX_FILE_DATA_SIZE); (unsigned long long)MAX_FILE_DATA_SIZE);
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
void* file_data = malloc(file_data_size); size_t allocation_size = file_data_size > 0 ? file_data_size : 1;
void* file_data = malloc(allocation_size);
if (file_data == NULL) { if (file_data == NULL) {
perror("Could not allocate memory for file data"); perror("Could not allocate memory for file data");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
memcpy(file_data, data_pointer, file_data_size); memcpy(file_data, data_pointer, file_data_size);
Data* replacement = data_create(file_data, file_data_size);
if (replacement == NULL) {
file_destroy(file);
array_list_delete(files);
return NULL;
}
data_destroy(file->data); data_destroy(file->data);
file->data = data_create(file_data, file_data_size); file->data = replacement;
data_pointer += file_data_size; data_pointer += file_data_size;
remaining_size -= file_data_size; remaining_size -= file_data_size;
array_list_add(files, file); if (!array_list_add(files, file)) {
file_destroy(file);
array_list_delete(files);
return NULL;
}
} }
File** file_array = (File**)array_list_to_array(files); File** file_array = (File**)array_list_to_array(files);
if (files->size > 0 && file_array == NULL) {
array_list_delete(files);
return NULL;
}
Chunk* chunk = chunk_create(file_array, files->size); Chunk* chunk = chunk_create(file_array, files->size);
free(file_array); free(file_array);
files->item_destroyer = NULL; if (chunk != NULL)
files->item_destroyer = NULL;
array_list_delete(files); array_list_delete(files);
return chunk; return chunk;
+50
View File
@@ -100,6 +100,35 @@ static void config_set_defaults(Config* config) {
config->compress_choice = NULL; config->compress_choice = NULL;
} }
static bool valid_wire_bool(int value) {
return value == 0 || value == 1;
}
static bool validate_received_config(const Config* config) {
return valid_wire_bool(config->save_to_disk) && valid_wire_bool(config->use_multithreading) &&
valid_wire_bool(config->use_chunk_serialization) &&
valid_wire_bool(config->use_compression) && valid_wire_bool(config->use_metadata) &&
valid_wire_bool(config->use_sendfile) && valid_wire_bool(config->use_delete) &&
valid_wire_bool(config->use_incremental) && valid_wire_bool(config->use_delta) &&
valid_wire_bool(config->backup) && valid_wire_bool(config->follow_symlinks) &&
valid_wire_bool(config->copy_links) && valid_wire_bool(config->safe_links) &&
valid_wire_bool(config->copy_unsafe_links) &&
valid_wire_bool(config->preserve_hard_links) && valid_wire_bool(config->preserve_acls) &&
valid_wire_bool(config->preserve_xattrs) && valid_wire_bool(config->preserve_devices) &&
valid_wire_bool(config->preserve_sparse) && valid_wire_bool(config->update) &&
valid_wire_bool(config->inplace) && valid_wire_bool(config->append) &&
valid_wire_bool(config->append_verify) && valid_wire_bool(config->delete_excluded) &&
valid_wire_bool(config->delete_after) && valid_wire_bool(config->relative) &&
valid_wire_bool(config->prune_empty_dirs) && valid_wire_bool(config->partial) &&
valid_wire_bool(config->delete_before) && valid_wire_bool(config->checksum) &&
(!config->use_compression ||
(config->compression_level >= 1 && config->compression_level <= 22)) &&
config->chunk_size > 0 && config->chunk_size <= MAX_CHUNK_SIZE &&
config->delta_block_size >= DELTA_BLOCK_SIZE_MIN &&
config->delta_block_size <= DELTA_BLOCK_SIZE_MAX &&
config->delta_max_file_size <= DELTA_MAX_FILE_SIZE && config->max_delete >= 0;
}
Config* config_create(void) { Config* config_create(void) {
Config* config = malloc(sizeof(Config)); Config* config = malloc(sizeof(Config));
if (!config) if (!config)
@@ -320,18 +349,26 @@ Config* config_receive(int file_descriptor) {
int tmp; int tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
if (!valid_wire_bool(tmp))
goto error;
config->save_to_disk = tmp; config->save_to_disk = tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
if (!valid_wire_bool(tmp))
goto error;
config->use_multithreading = tmp; config->use_multithreading = tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
config->use_chunk_serialization = tmp; config->use_chunk_serialization = tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
if (!valid_wire_bool(tmp))
goto error;
config->use_compression = tmp; config->use_compression = tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
if (!valid_wire_bool(tmp))
goto error;
config->use_metadata = tmp; config->use_metadata = tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
@@ -340,15 +377,23 @@ Config* config_receive(int file_descriptor) {
goto error; goto error;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
if (!valid_wire_bool(tmp))
goto error;
config->use_sendfile = tmp; config->use_sendfile = tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
if (!valid_wire_bool(tmp))
goto error;
config->use_delete = tmp; config->use_delete = tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
if (!valid_wire_bool(tmp))
goto error;
config->use_incremental = tmp; config->use_incremental = tmp;
if (!receive_int(file_descriptor, &tmp)) if (!receive_int(file_descriptor, &tmp))
goto error; goto error;
if (!valid_wire_bool(tmp))
goto error;
config->use_delta = tmp; config->use_delta = tmp;
if (!receive_n_data(file_descriptor, &config->delta_block_size, sizeof(config->delta_block_size))) if (!receive_n_data(file_descriptor, &config->delta_block_size, sizeof(config->delta_block_size)))
goto error; goto error;
@@ -440,6 +485,11 @@ Config* config_receive(int file_descriptor) {
send_status(file_descriptor, STATUS_ERROR); send_status(file_descriptor, STATUS_ERROR);
goto error; goto error;
} }
if (!validate_received_config(config)) {
fprintf(stderr, "Invalid configuration received from client\n");
send_status(file_descriptor, STATUS_ERROR);
goto error;
}
if (!send_status(file_descriptor, STATUS_OK)) if (!send_status(file_descriptor, STATUS_OK))
goto error; goto error;
return config; return config;
+12 -1
View File
@@ -36,6 +36,9 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
if (old_file_data == NULL || old_file_size == 0 || block_size == 0) if (old_file_data == NULL || old_file_size == 0 || block_size == 0)
return NULL; return NULL;
if (old_file_size > DELTA_MAX_FILE_SIZE || old_file_size > UINT32_MAX * (uint64_t)block_size)
return NULL;
uint32_t block_count = (uint32_t)((old_file_size + block_size - 1) / block_size); uint32_t block_count = (uint32_t)((old_file_size + block_size - 1) / block_size);
DeltaSignature* sig = malloc(sizeof(DeltaSignature)); DeltaSignature* sig = malloc(sizeof(DeltaSignature));
@@ -118,6 +121,13 @@ DeltaSignature* delta_signature_deserialize(const Data* data) {
return NULL; return NULL;
} }
if (sig->block_size == 0 || sig->block_size > DELTA_BLOCK_SIZE_MAX ||
sig->file_size > DELTA_MAX_FILE_SIZE || sig->file_size == 0 ||
(sig->file_size + sig->block_size - 1) / sig->block_size != sig->block_count) {
free(sig);
return NULL;
}
uint64_t expected = sizeof(uint64_t) + sizeof(uint32_t) + sizeof(uint32_t) + uint64_t expected = sizeof(uint64_t) + sizeof(uint32_t) + sizeof(uint32_t) +
(uint64_t)sig->block_count * (sizeof(uint32_t) + sizeof(uint32_t)); (uint64_t)sig->block_count * (sizeof(uint32_t) + sizeof(uint32_t));
if (data->size < expected) { if (data->size < expected) {
@@ -463,7 +473,8 @@ Delta* delta_deserialize(const Data* data) {
void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta, void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta,
uint32_t block_size) { uint32_t block_size) {
if (!old_data || !delta || (delta->new_file_size > 0 && delta->instructions == NULL) || if (!old_data || !delta || (delta->new_file_size > 0 && delta->instructions == NULL) ||
(delta->instruction_count > 0 && block_size == 0)) (delta->instruction_count > 0 && block_size == 0) ||
delta->new_file_size > DELTA_MAX_FILE_SIZE || delta->new_file_size > SIZE_MAX)
return NULL; return NULL;
void* output = malloc((size_t)delta->new_file_size); void* output = malloc((size_t)delta->new_file_size);
+61 -10
View File
@@ -23,6 +23,9 @@
#include "protocol.h" #include "protocol.h"
#include "utils.h" #include "utils.h"
#define MAX_SERVER_DELETE_COUNT 100000U
#define MAX_FILE_DATA_SIZE (64ULL * 1024 * 1024)
bool file_checksum(File* file, uint64_t* checksum) { bool file_checksum(File* file, uint64_t* checksum) {
if (!file || !checksum || !file->data) if (!file || !checksum || !file->data)
return false; return false;
@@ -155,7 +158,28 @@ bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
static bool to_disk_secure(const char* path, const void* data, unsigned long long data_size, static bool to_disk_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata); bool inplace, bool sparse, const FileMetadata* metadata);
static int open_secure_parent(const char* path, char** leaf_out); static int open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
bool file_path_exists_secure(const char* path) {
struct stat st;
return file_stat_secure(path, &st);
}
bool file_stat_secure(const char* path, struct stat* st) {
if (!path || !st)
return false;
char* leaf = NULL;
int parent_fd = open_secure_parent(path, &leaf, false);
if (parent_fd < 0)
return false;
int fd = openat(parent_fd, leaf, O_RDONLY | O_NONBLOCK | O_CLOEXEC | O_NOFOLLOW);
bool exists = fd >= 0 && fstat(fd, st) == 0 && S_ISREG(st->st_mode);
if (fd >= 0)
close(fd);
close(parent_fd);
free(leaf);
return exists;
}
static bool rename_secure(const char* old_path, const char* new_path); static bool rename_secure(const char* old_path, const char* new_path);
static int authorized_root_fd = -1; static int authorized_root_fd = -1;
static char* authorized_root_path; static char* authorized_root_path;
@@ -440,8 +464,16 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
} }
} }
Data* replacement = data_create(new_data, (size_t)new_size);
if (replacement == NULL) {
file_destroy(file);
free(old_data);
delta_signature_destroy(sig);
send_status(fd, STATUS_ERROR);
return NULL;
}
data_destroy(file->data); data_destroy(file->data);
file->data = data_create(new_data, (size_t)new_size); file->data = replacement;
free(old_data); free(old_data);
delta_signature_destroy(sig); delta_signature_destroy(sig);
@@ -483,6 +515,12 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
send_status(fd, STATUS_ERROR); send_status(fd, STATUS_ERROR);
return NULL; return NULL;
} }
if (uncompressed->size > MAX_FILE_DATA_SIZE) {
data_destroy(uncompressed);
file_destroy(file);
send_status(fd, STATUS_ERROR);
return NULL;
}
file_data = uncompressed; file_data = uncompressed;
} }
@@ -493,6 +531,7 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
delta_signature_destroy(sig); delta_signature_destroy(sig);
free(old_data); free(old_data);
send_status(fd, STATUS_ERROR);
return NULL; return NULL;
} }
@@ -532,7 +571,7 @@ File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
int old_fd = -1; int old_fd = -1;
if (full_path) { if (full_path) {
char* leaf = NULL; char* leaf = NULL;
int parent_fd = open_secure_parent(full_path, &leaf); int parent_fd = open_secure_parent(full_path, &leaf, false);
if (parent_fd >= 0) { if (parent_fd >= 0) {
old_fd = openat(parent_fd, leaf, O_RDONLY | O_CLOEXEC | O_NOFOLLOW); old_fd = openat(parent_fd, leaf, O_RDONLY | O_CLOEXEC | O_NOFOLLOW);
free(leaf); free(leaf);
@@ -542,7 +581,7 @@ File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
} }
unsigned long long old_size = has_old_file ? (unsigned long long)st.st_size : 0; unsigned long long old_size = has_old_file ? (unsigned long long)st.st_size : 0;
void* old_data = NULL; void* old_data = NULL;
if (has_old_file && old_size > 0) { if (has_old_file && old_size > 0 && old_size <= DELTA_MAX_FILE_SIZE && old_size <= SIZE_MAX) {
old_data = malloc((size_t)old_size); old_data = malloc((size_t)old_size);
if (old_data) { if (old_data) {
size_t got = 0; size_t got = 0;
@@ -643,6 +682,12 @@ File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
send_status(fd, STATUS_ERROR); send_status(fd, STATUS_ERROR);
return NULL; return NULL;
} }
if (uncompressed->size > MAX_FILE_DATA_SIZE) {
data_destroy(uncompressed);
file_destroy(file);
send_status(fd, STATUS_ERROR);
return NULL;
}
file_data = uncompressed; file_data = uncompressed;
} }
@@ -651,7 +696,7 @@ File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
return file; return file;
} }
static int open_secure_parent(const char* path, char** leaf_out) { static int open_secure_parent(const char* path, char** leaf_out, bool create_dirs) {
char* copy = str_dup(path); char* copy = str_dup(path);
if (!copy) if (!copy)
return -1; return -1;
@@ -691,7 +736,7 @@ static int open_secure_parent(const char* path, char** leaf_out) {
while (component) { while (component) {
if (strcmp(component, ".") != 0 && strcmp(component, "..") != 0) { if (strcmp(component, ".") != 0 && strcmp(component, "..") != 0) {
int next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); int next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (next < 0 && errno == ENOENT && mkdirat(fd, component, 0755) == 0) if (create_dirs && next < 0 && errno == ENOENT && mkdirat(fd, component, 0755) == 0)
next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (next < 0) { if (next < 0) {
close(fd); close(fd);
@@ -711,8 +756,8 @@ static int open_secure_parent(const char* path, char** leaf_out) {
static bool rename_secure(const char* old_path, const char* new_path) { static bool rename_secure(const char* old_path, const char* new_path) {
char *old_leaf = NULL, *new_leaf = NULL; char *old_leaf = NULL, *new_leaf = NULL;
int old_parent = open_secure_parent(old_path, &old_leaf); int old_parent = open_secure_parent(old_path, &old_leaf, true);
int new_parent = open_secure_parent(new_path, &new_leaf); int new_parent = open_secure_parent(new_path, &new_leaf, true);
bool ok = old_parent >= 0 && new_parent >= 0 && bool ok = old_parent >= 0 && new_parent >= 0 &&
renameat(old_parent, old_leaf, new_parent, new_leaf) == 0; renameat(old_parent, old_leaf, new_parent, new_leaf) == 0;
if (old_parent >= 0) if (old_parent >= 0)
@@ -741,7 +786,7 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
static bool to_disk_secure(const char* path, const void* data, unsigned long long data_size, static bool to_disk_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata) { bool inplace, bool sparse, const FileMetadata* metadata) {
char* leaf = NULL; char* leaf = NULL;
int dirfd = open_secure_parent(path, &leaf); int dirfd = open_secure_parent(path, &leaf, true);
if (dirfd < 0) if (dirfd < 0)
return false; return false;
int fd = -1; int fd = -1;
@@ -1011,6 +1056,11 @@ File* file_receive(const Config* config, int file_descriptor) {
file_destroy(file); file_destroy(file);
return NULL; return NULL;
} }
if (file_data_uncompressed->size > MAX_FILE_DATA_SIZE) {
data_destroy(file_data_uncompressed);
file_destroy(file);
return NULL;
}
file_data = file_data_uncompressed; file_data = file_data_uncompressed;
} }
data_destroy(file->data); data_destroy(file->data);
@@ -1068,7 +1118,8 @@ int receive_manifest(int fd, const Config* config, int* next_status) {
return *status_out == STATUS_FINISHED ? 0 : -1; return *status_out == STATUS_FINISHED ? 0 : -1;
} }
fprintf(stderr, "Deleting files not in manifest...\n"); fprintf(stderr, "Deleting files not in manifest...\n");
bool deletion_ok = delete_extras(config->receive_root_directory, manifest); bool deletion_ok =
delete_extras_limited(config->receive_root_directory, manifest, MAX_SERVER_DELETE_COUNT);
array_list_delete(manifest); array_list_delete(manifest);
return deletion_ok ? 0 : -1; return deletion_ok ? 0 : -1;
} }
+2
View File
@@ -40,6 +40,8 @@ bool to_disk(const char* path, const void* data, unsigned long long data_size, b
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config); bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
void file_set_authorized_root(int fd, const char* canonical_path); void file_set_authorized_root(int fd, const char* canonical_path);
File* receive_incremental_check(int fd, const Config* config, bool* skipped); File* receive_incremental_check(int fd, const Config* config, bool* skipped);
bool file_path_exists_secure(const char* path);
bool file_stat_secure(const char* path, struct stat* st);
int receive_manifest(int fd, const Config* config, int* next_status); int receive_manifest(int fd, const Config* config, int* next_status);
#endif #endif
+9 -2
View File
@@ -76,6 +76,11 @@ FileMetadata* metadata_from_buf(char** buf) {
memcpy(&mtime_nsec, *buf, sizeof(mtime_nsec)); memcpy(&mtime_nsec, *buf, sizeof(mtime_nsec));
*buf += sizeof(mtime_nsec); *buf += sizeof(mtime_nsec);
m->mtime_nsec = (long)mtime_nsec; m->mtime_nsec = (long)mtime_nsec;
if (present != 1 || mtime_nsec < 0 || mtime_nsec >= 1000000000LL || mode < 0 || uid < 0 ||
gid < 0) {
free(m);
return NULL;
}
return m; return m;
} }
@@ -175,7 +180,8 @@ FileMetadata* metadata_receive(int file_descriptor, int* ok) {
void file_restore_metadata(const char* path, const FileMetadata* metadata) { void file_restore_metadata(const char* path, const FileMetadata* metadata) {
if (metadata == NULL) if (metadata == NULL)
return; return;
if (chmod(path, metadata->mode & 07777 & ~(S_ISUID | S_ISGID)) != 0) mode_t safe_mode = metadata->mode & 0777 & ~(S_IWGRP | S_IWOTH);
if (chmod(path, safe_mode) != 0)
log_message(LOG_LEVEL_WARNING, "Failed to chmod %s: %s", path, strerror(errno)); log_message(LOG_LEVEL_WARNING, "Failed to chmod %s: %s", path, strerror(errno));
/* Never apply client-supplied ownership. The descriptor API below is the /* Never apply client-supplied ownership. The descriptor API below is the
receiver write path; retain this legacy API only for compatibility. */ receiver write path; retain this legacy API only for compatibility. */
@@ -192,7 +198,8 @@ bool file_restore_metadata_fd(int fd, const FileMetadata* metadata) {
if (fd < 0 || metadata == NULL) if (fd < 0 || metadata == NULL)
return metadata == NULL; return metadata == NULL;
bool ok = true; bool ok = true;
if (fchmod(fd, metadata->mode & 07777 & ~(S_ISUID | S_ISGID)) != 0) mode_t safe_mode = metadata->mode & 0777 & ~(S_IWGRP | S_IWOTH);
if (fchmod(fd, safe_mode) != 0)
ok = false; ok = false;
/* Client uid/gid values are deliberately not authoritative. */ /* Client uid/gid values are deliberately not authoritative. */
struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT}, struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
+2 -2
View File
@@ -234,9 +234,9 @@ int receive_thread(void* pipeline_context) {
} }
char* full_path = path_cat(config->receive_root_directory, check_path); char* full_path = path_cat(config->receive_root_directory, check_path);
struct stat st; struct stat st;
bool has_old = full_path && lstat(full_path, &st) == 0; bool has_old = full_path && file_stat_secure(full_path, &st);
bool match = has_old && (unsigned long long)st.st_size == check_size && bool match = has_old && (unsigned long long)st.st_size == check_size &&
(long long)st.st_mtime == check_mtime; (long long)st.st_mtime == check_mtime && S_ISREG(st.st_mode);
if (!send_status(file_descriptor, match ? STATUS_OK : STATUS_NEXT)) if (!send_status(file_descriptor, match ? STATUS_OK : STATUS_NEXT))
RECEIVE_THREAD_FAIL(); RECEIVE_THREAD_FAIL();
free(full_path); free(full_path);
+8 -5
View File
@@ -13,7 +13,7 @@
#define RECEIVE_TIMEOUT_SEC 60 /* 60 second per-message timeout */ #define RECEIVE_TIMEOUT_SEC 60 /* 60 second per-message timeout */
#define SEND_TIMEOUT_SEC 60 #define SEND_TIMEOUT_SEC 60
#define MAX_CONNECTION_MEMORY (1024ULL * 1024 * 1024) /* 1 GB total per connection */ #define MAX_CONNECTION_MEMORY (256ULL * 1024 * 1024) /* bounded cumulative receive budget */
static __thread int io_read_fd = -1; static __thread int io_read_fd = -1;
static __thread int io_write_fd = -1; static __thread int io_write_fd = -1;
@@ -25,7 +25,7 @@ static struct timespec bw_last_refill = {0, 0};
static mtx_t bw_mutex; static mtx_t bw_mutex;
static once_flag bw_mutex_once = ONCE_FLAG_INIT; static once_flag bw_mutex_once = ONCE_FLAG_INIT;
static __thread unsigned long long total_allocated_bytes = 0; static __thread unsigned long long total_allocated_bytes;
void io_set_fds(int read_fd, int write_fd) { void io_set_fds(int read_fd, int write_fd) {
io_read_fd = read_fd; io_read_fd = read_fd;
@@ -260,6 +260,11 @@ char* receive_str(int file_descriptor) {
free(data); free(data);
return NULL; return NULL;
} }
if (memchr(data, '\0', size) != NULL) {
free(data);
log_message(LOG_LEVEL_ERROR, "Received string contains an embedded NUL");
return NULL;
}
data[size] = '\0'; data[size] = '\0';
total_allocated_bytes += size + 1; total_allocated_bytes += size + 1;
log_message(LOG_LEVEL_DEBUG, "Received String: %s", data); log_message(LOG_LEVEL_DEBUG, "Received String: %s", data);
@@ -287,9 +292,7 @@ Data* receive_data(int file_descriptor) {
} }
size_t allocation_size = size == 0 ? 1 : (size_t)size; size_t allocation_size = size == 0 ? 1 : (size_t)size;
if (allocation_size > MAX_CONNECTION_MEMORY - total_allocated_bytes) { if (allocation_size > MAX_CONNECTION_MEMORY - total_allocated_bytes) {
log_message(LOG_LEVEL_ERROR, "Per-connection memory limit exceeded (%llu + %llu > %llu)", log_message(LOG_LEVEL_ERROR, "Per-connection memory limit exceeded");
(unsigned long long)total_allocated_bytes, size,
(unsigned long long)MAX_CONNECTION_MEMORY);
return NULL; return NULL;
} }
void* data = malloc(allocation_size); void* data = malloc(allocation_size);
+24 -2
View File
@@ -11,6 +11,7 @@
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <sys/socket.h> #include <sys/socket.h>
#include <sys/stat.h>
#include <sys/wait.h> #include <sys/wait.h>
#include <unistd.h> #include <unistd.h>
@@ -34,6 +35,10 @@ static void log_ssl_errors(void) {
static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key, static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key,
const char* ca_path) { const char* ca_path) {
if (!is_server && !ca_path) {
log_message(LOG_LEVEL_ERROR, "TLS clients require a CA certificate path");
return NULL;
}
const SSL_METHOD* method = is_server ? TLS_server_method() : TLS_client_method(); const SSL_METHOD* method = is_server ? TLS_server_method() : TLS_client_method();
SSL_CTX* ctx = SSL_CTX_new(method); SSL_CTX* ctx = SSL_CTX_new(method);
if (!ctx) { if (!ctx) {
@@ -42,9 +47,23 @@ static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key
return NULL; return NULL;
} }
SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION); if (SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION) != 1) {
SSL_CTX_free(ctx);
return NULL;
}
if (SSL_CTX_set_cipher_list(ctx, "HIGH:!aNULL:!eNULL:!MD5:!RC4:!3DES") != 1) {
SSL_CTX_free(ctx);
return NULL;
}
if (cert && key) { if (cert && key) {
struct stat key_stat;
if (stat(key, &key_stat) != 0 || !S_ISREG(key_stat.st_mode) || key_stat.st_uid != geteuid() ||
(key_stat.st_mode & (S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH))) {
log_message(LOG_LEVEL_ERROR, "TLS private key must be owned by the current user and private");
SSL_CTX_free(ctx);
return NULL;
}
if (SSL_CTX_use_certificate_file(ctx, cert, SSL_FILETYPE_PEM) <= 0) { if (SSL_CTX_use_certificate_file(ctx, cert, SSL_FILETYPE_PEM) <= 0) {
log_message(LOG_LEVEL_ERROR, "Failed to load certificate: %s", cert); log_message(LOG_LEVEL_ERROR, "Failed to load certificate: %s", cert);
log_ssl_errors(); log_ssl_errors();
@@ -91,7 +110,10 @@ static SSL* wrap_fd_with_ssl(int fd, SSL_CTX* ctx, bool is_server, const char* h
// Enable hostname verification for client connections when a hostname is provided. // Enable hostname verification for client connections when a hostname is provided.
// Must be done before SSL_connect to take effect during the handshake. // Must be done before SSL_connect to take effect during the handshake.
if (!is_server && hostname) { if (!is_server && hostname) {
SSL_set1_host(ssl, hostname); if (SSL_set1_host(ssl, hostname) != 1) {
SSL_free(ssl);
return NULL;
}
} }
// Retry SSL_accept/SSL_connect on WANT_READ/WANT_WRITE (non-blocking handshake) // Retry SSL_accept/SSL_connect on WANT_READ/WANT_WRITE (non-blocking handshake)
+87 -11
View File
@@ -7,13 +7,67 @@
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <stdint.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <unistd.h> #include <unistd.h>
static int authorized_root_fd = -1; static int authorized_root_fd = -1;
static char* authorized_root_path;
void utils_set_authorized_root(int fd, const char* canonical_path) {
authorized_root_fd = fd;
free(authorized_root_path);
authorized_root_path = canonical_path ? str_dup(canonical_path) : NULL;
}
void utils_set_authorized_root_fd(int fd) { void utils_set_authorized_root_fd(int fd) {
authorized_root_fd = fd; utils_set_authorized_root(fd, NULL);
}
static bool path_is_within_root(const char* root, const char* path) {
size_t root_len = strlen(root);
return strncmp(root, path, root_len) == 0 && (path[root_len] == '\0' || path[root_len] == '/');
}
static int open_authorized_destination(const char* dest_root) {
if (authorized_root_fd < 0 || !authorized_root_path || !dest_root ||
!path_is_within_root(authorized_root_path, dest_root))
return -1;
int dirfd = dup(authorized_root_fd);
if (dirfd < 0)
return -1;
const char* relative_path = dest_root + strlen(authorized_root_path);
while (*relative_path == '/')
relative_path++;
char* relative = str_dup(*relative_path ? relative_path : ".");
if (!relative) {
close(dirfd);
return -1;
}
char* saveptr = NULL;
char* component = strtok_r(relative, "/", &saveptr);
while (component) {
if (strcmp(component, ".") == 0 || strcmp(component, "..") == 0) {
free(relative);
close(dirfd);
return -1;
}
int next = openat(dirfd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (next < 0) {
free(relative);
close(dirfd);
return -1;
}
close(dirfd);
dirfd = next;
component = strtok_r(NULL, "/", &saveptr);
}
free(relative);
return dirfd;
} }
bool mkdir_r(const char* path) { bool mkdir_r(const char* path) {
@@ -143,7 +197,8 @@ static bool is_dir_in_manifest(const char* rel_path, ArrayList* manifest) {
return false; return false;
} }
static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifest) { static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifest,
size_t max_delete, size_t* deleted_count) {
int scanfd = dup(dirfd); int scanfd = dup(dirfd);
if (scanfd < 0) if (scanfd < 0)
return false; return false;
@@ -173,12 +228,17 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_removed = false; bool child_removed = false;
if (childfd >= 0) { if (childfd >= 0) {
child_removed = delete_extras_fd(childfd, child_rel, manifest); child_removed = delete_extras_fd(childfd, child_rel, manifest, max_delete, deleted_count);
close(childfd); close(childfd);
} }
if (child_removed && !is_dir_in_manifest(child_rel, manifest) && if (child_removed && !is_dir_in_manifest(child_rel, manifest)) {
unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0 && errno != ENOENT) { if (*deleted_count >= max_delete) {
operation_ok = false; operation_ok = false;
} else if (unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0 && errno != ENOENT) {
operation_ok = false;
} else {
(*deleted_count)++;
}
} else if (!child_removed) { } else if (!child_removed) {
all_removed = false; all_removed = false;
} }
@@ -192,8 +252,15 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
} }
} }
if (!found) { if (!found) {
if (*deleted_count >= max_delete) {
operation_ok = false;
free(child_rel);
continue;
}
if (unlinkat(dirfd, entry->d_name, 0) != 0 && errno != ENOENT) if (unlinkat(dirfd, entry->d_name, 0) != 0 && errno != ENOENT)
operation_ok = false; operation_ok = false;
else
(*deleted_count)++;
fprintf(stderr, " Deleted: %s\n", child_rel); fprintf(stderr, " Deleted: %s\n", child_rel);
} else { } else {
all_removed = false; all_removed = false;
@@ -206,18 +273,27 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
return operation_ok; return operation_ok;
} }
bool delete_extras(const char* dest_root, ArrayList* manifest) { bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete) {
int rootfd = authorized_root_fd >= 0 int rootfd;
? dup(authorized_root_fd) if (authorized_root_fd >= 0) {
: open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); rootfd =
authorized_root_path ? open_authorized_destination(dest_root) : dup(authorized_root_fd);
} else {
rootfd = open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
if (rootfd < 0) if (rootfd < 0)
return false; return false;
bool ok = delete_extras_fd(rootfd, "", manifest); size_t deleted_count = 0;
bool ok = delete_extras_fd(rootfd, "", manifest, max_delete, &deleted_count);
if (close(rootfd) != 0) if (close(rootfd) != 0)
ok = false; ok = false;
return ok; return ok;
} }
bool delete_extras(const char* dest_root, ArrayList* manifest) {
return delete_extras_limited(dest_root, manifest, SIZE_MAX);
}
bool has_path_traversal(const char* path) { bool has_path_traversal(const char* path) {
if (!path) if (!path)
return false; return false;
+3
View File
@@ -2,6 +2,7 @@
#define UTILS_H #define UTILS_H
#include "array_list.h" #include "array_list.h"
#include <stddef.h>
#include <stdbool.h> #include <stdbool.h>
bool mkdir_r(const char* path); bool mkdir_r(const char* path);
@@ -9,6 +10,8 @@ char* str_dup(const char* string);
char* path_cat(const char* path1, const char* path2); char* path_cat(const char* path1, const char* path2);
bool glob_match(const char* pattern, const char* str); bool glob_match(const char* pattern, const char* str);
bool delete_extras(const char* dest_root, ArrayList* manifest); bool delete_extras(const char* dest_root, ArrayList* manifest);
bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete);
void utils_set_authorized_root(int fd, const char* canonical_path);
void utils_set_authorized_root_fd(int fd); void utils_set_authorized_root_fd(int fd);
bool has_path_traversal(const char* path); bool has_path_traversal(const char* path);
+3 -1
View File
@@ -25,7 +25,9 @@ class ServerManager:
def start(self, extra_args=None): def start(self, extra_args=None):
self.stop() self.stop()
self._port = _find_free_port() self._port = _find_free_port()
cmd = SERVER_CMD + ["-p", str(self._port)] # Plain TCP is intentionally explicit in the server; integration tests
# exercise that opt-in mode rather than relying on the secure default.
cmd = SERVER_CMD + ["-p", str(self._port), "--allow-unauthenticated"]
if extra_args: if extra_args:
cmd += extra_args cmd += extra_args
self._proc = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) self._proc = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+3
View File
@@ -102,6 +102,7 @@ class TestTLSBasic:
with ServerManager() as server: with ServerManager() as server:
server.start(extra_args=[ server.start(extra_args=[
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"], "--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
"--ca", certs["ca"], "--client-cn", "fastsync-client",
]) ])
result, dur = run_client( result, dur = run_client(
SOURCE_DIR, DEST_DIR, SOURCE_DIR, DEST_DIR,
@@ -125,6 +126,7 @@ class TestTLSBasic:
with ServerManager() as server: with ServerManager() as server:
server.start(extra_args=[ server.start(extra_args=[
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"], "--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
"--ca", certs["ca"], "--client-cn", "fastsync-client",
]) ])
result, dur = run_client( result, dur = run_client(
SOURCE_DIR, DEST_DIR, SOURCE_DIR, DEST_DIR,
@@ -149,6 +151,7 @@ class TestTLSBasic:
with ServerManager() as server: with ServerManager() as server:
server.start(extra_args=[ server.start(extra_args=[
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"], "--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
"--ca", certs["ca"], "--client-cn", "fastsync-client",
]) ])
result, dur = run_client( result, dur = run_client(
SOURCE_DIR, DEST_DIR, SOURCE_DIR, DEST_DIR,