diff --git a/README.md b/README.md index bcf8e29..fcb2d9f 100644 --- a/README.md +++ b/README.md @@ -1,101 +1,98 @@ -# FastFileTransfer +# FastSync -A high-performance file synchronization system that implements a custom client-server protocol for efficient file transfer with compression and multithreading support. +A high-performance file synchronization system with a custom TCP-based protocol, optional metadata preservation, compression, multithreading, and zero-copy `sendfile()` support. ## Technical Overview -FastFileTransfer is a C implementation of a file synchronization system that: - -1. Uses a custom TCP-based protocol for client-server communication -2. Implements chunked file transfer (10MB chunks by default) -3. Supports zstd compression with configurable levels (1-22) -4. Utilizes multithreading for parallel file processing -5. Implements producer-consumer patterns with thread-safe queues -6. Provides both in-memory and disk-based storage options -7. Supports `sendfile()` for zero-copy file transfer +1. Custom TCP-based client-server protocol with status codes +2. Chunked file transfer (files grouped into ~10 MB chunks) +3. Optional zstd compression (levels 1–22) +4. Multithreading for parallel file processing (producer-consumer with thread-safe queues) +5. Optional file metadata preservation (`mode`, `uid`, `gid`, `mtime`) — restored on disk +6. In-memory and disk-based storage options +7. `sendfile()` zero-copy path (~2× faster on localhost) ## System Architecture -The system consists of two main components: - ### Client -- Scans source directories recursively -- Creates file chunks with configurable size (10MB default) -- Compresses data using zstd algorithm -- Serializes chunks into a compact binary format for batch transfer -- Sends files to server using custom protocol -- Supports sendfile for zero-copy file transfer (`-f`) -- Supports both single-threaded and multi-threaded operation +- Recursively scans source directories (BFS) +- Groups files into chunks (default ~10 MB total) +- Optionally compresses with zstd +- Optionally serializes chunks into a compact binary format +- Optionally attaches per-file metadata (mode, ownership, timestamps) +- Sends via custom protocol or `sendfile()` zero-copy path ### Server -- Listens for client connections on port 8080 -- Receives files using the custom protocol -- Decompresses received data -- Stores files either in memory or on disk -- Implements thread pool for parallel processing +- Listens on port 8080 +- Receives and reassembles files +- Decompresses, deserializes, restores metadata on disk +- Thread pool for parallel processing ## Protocol Details -The client-server communication uses the following status codes: -- `STATUS_OK`: Operation successful -- `STATUS_ERROR`: Error occurred -- `STATUS_FINISHED`: Transfer complete -- `STATUS_NEXT`: Ready for next file (per-file mode) -- `STATUS_CHUNK`: Following data is a serialized chunk (chunk mode) +Status codes: +| Code | Meaning | +|------|---------| +| `STATUS_OK` | Operation successful | +| `STATUS_ERROR` | Error occurred | +| `STATUS_FINISHED` | Transfer complete | +| `STATUS_NEXT` | Ready for next file (per-file mode) | +| `STATUS_CHUNK` | Following data is a serialized chunk | -## Configuration Options +### Wire Format — Metadata -### Command Line Arguments +When `use_metadata` is enabled (`-M`), each file entry carries a 4-byte `present` flag followed by five fields (`mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`). When disabled globally, no metadata bytes are sent — zero wire overhead. + +## Configuration + +### Command-Line Arguments | Argument | Description | |----------|-------------| -| `-m` | Enable multithreading mode | -| `-c [level]` | Enable compression with optional level (1-22, default: 5) | -| `-s` | Enable chunk serialization (batch-transfer all files per chunk) | -| `-f` | Enable sendfile (zero-copy file transfer, bypasses userspace memory). Can be combined with `-m`. Incompatible with `-c` and `-s`. | -| `--source-dir ` | Source directory to sync (overrides `FASTSYNC_SOURCE_DIR`) | -| `--dest-dir ` | Server-side destination directory (overrides `FASTSYNC_DEST_DIR`) | -| `--save-to-disk` | Persist received files to disk | +| `-m` | Multithreading mode | +| `-c [level]` | Compression with optional level (1–22, default 5) | +| `-s` | Chunk serialization (batch all files per chunk) | +| `-f` | Sendfile zero-copy. Incompatible with `-c` / `-s`. | +| `-M, --preserve` | Preserve file metadata (mode, uid, gid, mtime) | +| `--source-dir ` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) | +| `--dest-dir ` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) | +| `--save-to-disk` | Write received files to disk | ### Environment Variables -| Variable | Description | Default | -|----------|-------------|---------| -| `FASTSYNC_SOURCE_DIR` | Source directory for files (fallback, overridden by `--source-dir`) | Current user's documents directory | -| `FASTSYNC_DEST_DIR` | Destination directory (fallback, overridden by `--dest-dir`) | `./data_copied` | -| `FASTSYNC_SERVER_IP` | Server IP address | `127.0.0.1` | -| `FASTSYNC_SERVER_PORT` | Server port | `8080` | -| `FASTSYNC_SAVE_TO_DISK` | Save to disk (fallback, overridden by `--save-to-disk`) | `false` | +| Variable | Default | Description | +|----------|---------|-------------| +| `FASTSYNC_SOURCE_DIR` | User documents | Source directory fallback | +| `FASTSYNC_DEST_DIR` | `./data_copied` | Destination directory fallback | +| `FASTSYNC_SERVER_IP` | `127.0.0.1` | Server address | +| `FASTSYNC_SERVER_PORT` | `8080` | Server port | +| `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback | ## Implementation Details ### Data Structures - -1. **Chunk**: Collection of files (default 10MB total size) -2. **File**: File metadata with path and content -3. **FileReceive**: Received file data structure -4. **Config**: Configuration parameters structure -5. **Queue**: Thread-safe queue implementation using condition variables +1. **Chunk** — collection of files (~10 MB total) +2. **File** — path, content (`Data`), optional `FileMetadata` pointer +3. **FileMetadata** — `mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec` +4. **Config** — runtime parameters +5. **Queue** — thread-safe queue with condition variables ### Key Algorithms - -1. **File Scanning**: Recursive directory traversal with BFS -2. **Chunking**: Files grouped into chunks with size limit -3. **Compression**: zstd compression with configurable levels -4. **Network Protocol**: Custom TCP-based protocol with status codes -5. **Thread Synchronization**: Condition variables and mutexes for thread coordination +1. **File scanning** — recursive BFS directory traversal +2. **Chunking** — files grouped by size limit +3. **Compression** — zstd with configurable level +4. **Network protocol** — custom TCP with status codes and optional metadata packing +5. **Metadata restoration** — `chmod()`, `chown()`, `utimensat()` on the receiving side ## Build Requirements -- C11 compatible compiler -- CMake 4.1 or later +- C11 compiler +- CMake 4.1+ - zstd library -- pthread support +- pthreads ## Building ```bash -mkdir -p build && cd build -cmake .. -make +cmake -B build -S . && cmake --build build -j$(nproc) ``` ## Running @@ -107,66 +104,41 @@ make ### Client ```bash -# Basic usage with default settings (sends from ~/Documents/...) -./build/client -m -c 10 - -# Specify source and destination directories +# Basic ./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk -# Chunk serialization mode (batch per chunk) -./build/client -s +# With metadata preservation +./build/client -M --source-dir ... --dest-dir ... -# Compressed chunk serialization -./build/client -s -c 3 +# Multithreaded + compression +./build/client -m -c 10 -# Multithreaded with compressed chunk serialization -./build/client -m -s -c 3 - -# Sendfile (zero-copy, bypasses userspace for large files) +# Sendfile (zero-copy) ./build/client -f -# Sendfile with multithreading -./build/client -f -m +# All features +./build/client -m -c -s -M ``` ## Testing -The project includes comprehensive unit tests for core functionality: - ```bash +# Unit tests ./build/tests -``` -An integration test / benchmark script runs all configurations against ~50 MB of generated test data with byte-for-byte verification: - -```bash +# Integration benchmark (~50 MB data, 13 configurations + rsync comparison) python3 test.py -# Skip the throttled suite (disk I/O limits + 100ms network delay) if sudo is unavailable: -python3 test.py --no-throttled + +# Profiles: --wan (100 Mbit, 50 ms, 1% loss), --unlimited (no throttling) +python3 test.py --wan ``` -## Code Organization - -``` -src/ - client/ # Client implementation - server/ # Server implementation - shared/ # Shared data structures and utilities -tests/ # Unit tests -``` +The benchmark prints throughput metrics for the best configuration and speedup vs rsync. ## Performance Considerations -1. Chunk size (10MB default) affects memory usage and transfer efficiency -2. Compression level (1-22) trades CPU usage for space savings -3. `sendfile()` (`-f`) bypasses userspace memory, ~2x faster on localhost for large files -4. Multithreading improves performance on multi-core systems -5. Thread-safe queues minimize contention between producer/consumer threads - -## Extensibility - -The system is designed with clear interfaces that allow for: -1. Additional compression algorithms -2. Different transport protocols -3. Custom storage backends -4. Extended metadata support \ No newline at end of file +1. Chunk size (~10 MB) balances memory and transfer efficiency +2. Compression level trades CPU for bandwidth +3. `sendfile()` bypasses userspace — ~2× faster on localhost for large files +4. Multithreading scales with core count +5. Metadata transfer adds negligible overhead when disabled, ~24 bytes per file when enabled