TapTap/FastSync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update README for metadata transfer, merged File/FileReceive, test.py changes

Browse Source
This commit is contained in:
TapTap
2026-07-05 21:07:24 +02:00
parent 24f9a2afb0
commit c2ada295ca
1 changed files with 81 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+81 -109
View File
@@ -1,101 +1,98 @@
# FastFileTransfer
# FastSync
A high-performance file synchronization system that implements a custom client-server protocol for efficient file transfer with compression and multithreading support.
A high-performance file synchronization system with a custom TCP-based protocol, optional metadata preservation, compression, multithreading, and zero-copy `sendfile()` support.
## Technical Overview
FastFileTransfer is a C implementation of a file synchronization system that:
1. Uses a custom TCP-based protocol for client-server communication
2. Implements chunked file transfer (10MB chunks by default)
3. Supports zstd compression with configurable levels (1-22)
4. Utilizes multithreading for parallel file processing
5. Implements producer-consumer patterns with thread-safe queues
6. Provides both in-memory and disk-based storage options
7. Supports `sendfile()` for zero-copy file transfer
1. Custom TCP-based client-server protocol with status codes
2. Chunked file transfer (files grouped into ~10 MB chunks)
3. Optional zstd compression (levels 1–22)
4. Multithreading for parallel file processing (producer-consumer with thread-safe queues)
5. Optional file metadata preservation (`mode`, `uid`, `gid`, `mtime`) — restored on disk
6. In-memory and disk-based storage options
7. `sendfile()` zero-copy path (~2× faster on localhost)
## System Architecture
The system consists of two main components:
### Client
- Scans source directories recursively
- Creates file chunks with configurable size (10MB default)
- Compresses data using zstd algorithm
- Serializes chunks into a compact binary format for batch transfer
- Sends files to server using custom protocol
- Supports sendfile for zero-copy file transfer (`-f`)
- Supports both single-threaded and multi-threaded operation
- Recursively scans source directories (BFS)
- Groups files into chunks (default ~10 MB total)
- Optionally compresses with zstd
- Optionally serializes chunks into a compact binary format
- Optionally attaches per-file metadata (mode, ownership, timestamps)
- Sends via custom protocol or `sendfile()` zero-copy path
### Server
- Listens for client connections on port 8080
- Receives files using the custom protocol
- Decompresses received data
- Stores files either in memory or on disk
- Implements thread pool for parallel processing
- Listens on port 8080
- Receives and reassembles files
- Decompresses, deserializes, restores metadata on disk
- Thread pool for parallel processing
## Protocol Details
The client-server communication uses the following status codes:
- `STATUS_OK`: Operation successful
- `STATUS_ERROR`: Error occurred
- `STATUS_FINISHED`: Transfer complete
- `STATUS_NEXT`: Ready for next file (per-file mode)
- `STATUS_CHUNK`: Following data is a serialized chunk (chunk mode)
Status codes:
| Code | Meaning |
|------|---------|
| `STATUS_OK` | Operation successful |
| `STATUS_ERROR` | Error occurred |
| `STATUS_FINISHED` | Transfer complete |
| `STATUS_NEXT` | Ready for next file (per-file mode) |
| `STATUS_CHUNK` | Following data is a serialized chunk |
## Configuration Options
### Wire Format — Metadata
### Command Line Arguments
When `use_metadata` is enabled (`-M`), each file entry carries a 4-byte `present` flag followed by five fields (`mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`). When disabled globally, no metadata bytes are sent — zero wire overhead.
## Configuration
### Command-Line Arguments
| Argument | Description |
|----------|-------------|
| `-m` | Enable multithreading mode |
| `-c [level]` | Enable compression with optional level (1-22, default: 5) |
| `-s` | Enable chunk serialization (batch-transfer all files per chunk) |
| `-f` | Enable sendfile (zero-copy file transfer, bypasses userspace memory). Can be combined with `-m`. Incompatible with `-c` and `-s`. |
| `--source-dir <path>` | Source directory to sync (overrides `FASTSYNC_SOURCE_DIR`) |
| `--dest-dir <path>` | Server-side destination directory (overrides `FASTSYNC_DEST_DIR`) |
| `--save-to-disk` | Persist received files to disk |
| `-m` | Multithreading mode |
| `-c [level]` | Compression with optional level (1–22, default 5) |
| `-s` | Chunk serialization (batch all files per chunk) |
| `-f` | Sendfile zero-copy. Incompatible with `-c` / `-s`. |
| `-M, --preserve` | Preserve file metadata (mode, uid, gid, mtime) |
| `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
| `--dest-dir <path>` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) |
| `--save-to-disk` | Write received files to disk |
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `FASTSYNC_SOURCE_DIR` | Source directory for files (fallback, overridden by `--source-dir`) | Current user's documents directory |
| `FASTSYNC_DEST_DIR` | Destination directory (fallback, overridden by `--dest-dir`) | `./data_copied` |
| `FASTSYNC_SERVER_IP` | Server IP address | `127.0.0.1` |
| `FASTSYNC_SERVER_PORT` | Server port | `8080` |
| `FASTSYNC_SAVE_TO_DISK` | Save to disk (fallback, overridden by `--save-to-disk`) | `false` |
| Variable | Default | Description |
|----------|---------|-------------|
| `FASTSYNC_SOURCE_DIR` | User documents | Source directory fallback |
| `FASTSYNC_DEST_DIR` | `./data_copied` | Destination directory fallback |
| `FASTSYNC_SERVER_IP` | `127.0.0.1` | Server address |
| `FASTSYNC_SERVER_PORT` | `8080` | Server port |
| `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback |
## Implementation Details
### Data Structures
1. **Chunk**: Collection of files (default 10MB total size)
2. **File**: File metadata with path and content
3. **FileReceive**: Received file data structure
4. **Config**: Configuration parameters structure
5. **Queue**: Thread-safe queue implementation using condition variables
1. **Chunk** — collection of files (~10 MB total)
2. **File** — path, content (`Data`), optional `FileMetadata` pointer
3. **FileMetadata** — `mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`
4. **Config** — runtime parameters
5. **Queue** — thread-safe queue with condition variables
### Key Algorithms
1. **File Scanning**: Recursive directory traversal with BFS
2. **Chunking**: Files grouped into chunks with size limit
3. **Compression**: zstd compression with configurable levels
4. **Network Protocol**: Custom TCP-based protocol with status codes
5. **Thread Synchronization**: Condition variables and mutexes for thread coordination
1. **File scanning** — recursive BFS directory traversal
2. **Chunking** — files grouped by size limit
3. **Compression** — zstd with configurable level
4. **Network protocol** — custom TCP with status codes and optional metadata packing
5. **Metadata restoration** — `chmod()`, `chown()`, `utimensat()` on the receiving side
## Build Requirements
- C11 compatible compiler
- CMake 4.1 or later
- C11 compiler
- CMake 4.1+
- zstd library
- pthread support
- pthreads
## Building
```bash
mkdir -p build && cd build
cmake ..
make
cmake -B build -S . && cmake --build build -j$(nproc)
```
## Running
@@ -107,66 +104,41 @@ make
### Client
```bash
# Basic usage with default settings (sends from ~/Documents/...)
./build/client -m -c 10
# Specify source and destination directories
# Basic
./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk
# Chunk serialization mode (batch per chunk)
./build/client -s
# With metadata preservation
./build/client -M --source-dir ... --dest-dir ...
# Compressed chunk serialization
./build/client -s -c 3
# Multithreaded + compression
./build/client -m -c 10
# Multithreaded with compressed chunk serialization
./build/client -m -s -c 3
# Sendfile (zero-copy, bypasses userspace for large files)
# Sendfile (zero-copy)
./build/client -f
# Sendfile with multithreading
./build/client -f -m
# All features
./build/client -m -c -s -M
```
## Testing
The project includes comprehensive unit tests for core functionality:
```bash
# Unit tests
./build/tests
```
An integration test / benchmark script runs all configurations against ~50 MB of generated test data with byte-for-byte verification:
```bash
# Integration benchmark (~50 MB data, 13 configurations + rsync comparison)
python3 test.py
# Skip the throttled suite (disk I/O limits + 100ms network delay) if sudo is unavailable:
python3 test.py --no-throttled
# Profiles: --wan (100 Mbit, 50 ms, 1% loss), --unlimited (no throttling)
python3 test.py --wan
```
## Code Organization
```
src/
client/ # Client implementation
server/ # Server implementation
shared/ # Shared data structures and utilities
tests/ # Unit tests
```
The benchmark prints throughput metrics for the best configuration and speedup vs rsync.
## Performance Considerations
1. Chunk size (10MB default) affects memory usage and transfer efficiency
2. Compression level (1-22) trades CPU usage for space savings
3. `sendfile()` (`-f`) bypasses userspace memory, ~2x faster on localhost for large files
4. Multithreading improves performance on multi-core systems
5. Thread-safe queues minimize contention between producer/consumer threads
## Extensibility
The system is designed with clear interfaces that allow for:
1. Additional compression algorithms
2. Different transport protocols
3. Custom storage backends
4. Extended metadata support
1. Chunk size (~10 MB) balances memory and transfer efficiency
2. Compression level trades CPU for bandwidth
3. `sendfile()` bypasses userspace — ~2× faster on localhost for large files
4. Multithreading scales with core count
5. Metadata transfer adds negligible overhead when disabled, ~24 bytes per file when enabled