TapTap/FastSync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update README for metadata transfer, merged File/FileReceive, test.py changes

Browse Source
This commit is contained in:
TapTap
2026-07-05 21:07:24 +02:00
parent 24f9a2afb0
commit c2ada295ca
1 changed files with 81 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+81 -109
View File
@@ -1,101 +1,98 @@
# FastFileTransfer # FastSync
A high-performance file synchronization system that implements a custom client-server protocol for efficient file transfer with compression and multithreading support. A high-performance file synchronization system with a custom TCP-based protocol, optional metadata preservation, compression, multithreading, and zero-copy `sendfile()` support.
## Technical Overview ## Technical Overview
FastFileTransfer is a C implementation of a file synchronization system that: 1. Custom TCP-based client-server protocol with status codes
2. Chunked file transfer (files grouped into ~10 MB chunks)
1. Uses a custom TCP-based protocol for client-server communication 3. Optional zstd compression (levels 1–22)
2. Implements chunked file transfer (10MB chunks by default) 4. Multithreading for parallel file processing (producer-consumer with thread-safe queues)
3. Supports zstd compression with configurable levels (1-22) 5. Optional file metadata preservation (`mode`, `uid`, `gid`, `mtime`) — restored on disk
4. Utilizes multithreading for parallel file processing 6. In-memory and disk-based storage options
5. Implements producer-consumer patterns with thread-safe queues 7. `sendfile()` zero-copy path (~2× faster on localhost)
6. Provides both in-memory and disk-based storage options
7. Supports `sendfile()` for zero-copy file transfer
## System Architecture ## System Architecture
The system consists of two main components:
### Client ### Client
- Scans source directories recursively - Recursively scans source directories (BFS)
- Creates file chunks with configurable size (10MB default) - Groups files into chunks (default ~10 MB total)
- Compresses data using zstd algorithm - Optionally compresses with zstd
- Serializes chunks into a compact binary format for batch transfer - Optionally serializes chunks into a compact binary format
- Sends files to server using custom protocol - Optionally attaches per-file metadata (mode, ownership, timestamps)
- Supports sendfile for zero-copy file transfer (`-f`) - Sends via custom protocol or `sendfile()` zero-copy path
- Supports both single-threaded and multi-threaded operation
### Server ### Server
- Listens for client connections on port 8080 - Listens on port 8080
- Receives files using the custom protocol - Receives and reassembles files
- Decompresses received data - Decompresses, deserializes, restores metadata on disk
- Stores files either in memory or on disk - Thread pool for parallel processing
- Implements thread pool for parallel processing
## Protocol Details ## Protocol Details
The client-server communication uses the following status codes: Status codes:
- `STATUS_OK`: Operation successful | Code | Meaning |
- `STATUS_ERROR`: Error occurred |------|---------|
- `STATUS_FINISHED`: Transfer complete | `STATUS_OK` | Operation successful |
- `STATUS_NEXT`: Ready for next file (per-file mode) | `STATUS_ERROR` | Error occurred |
- `STATUS_CHUNK`: Following data is a serialized chunk (chunk mode) | `STATUS_FINISHED` | Transfer complete |
| `STATUS_NEXT` | Ready for next file (per-file mode) |
| `STATUS_CHUNK` | Following data is a serialized chunk |
## Configuration Options ### Wire Format — Metadata
### Command Line Arguments When `use_metadata` is enabled (`-M`), each file entry carries a 4-byte `present` flag followed by five fields (`mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`). When disabled globally, no metadata bytes are sent — zero wire overhead.
## Configuration
### Command-Line Arguments
| Argument | Description | | Argument | Description |
|----------|-------------| |----------|-------------|
| `-m` | Enable multithreading mode | | `-m` | Multithreading mode |
| `-c [level]` | Enable compression with optional level (1-22, default: 5) | | `-c [level]` | Compression with optional level (1–22, default 5) |
| `-s` | Enable chunk serialization (batch-transfer all files per chunk) | | `-s` | Chunk serialization (batch all files per chunk) |
| `-f` | Enable sendfile (zero-copy file transfer, bypasses userspace memory). Can be combined with `-m`. Incompatible with `-c` and `-s`. | | `-f` | Sendfile zero-copy. Incompatible with `-c` / `-s`. |
| `--source-dir <path>` | Source directory to sync (overrides `FASTSYNC_SOURCE_DIR`) | | `-M, --preserve` | Preserve file metadata (mode, uid, gid, mtime) |
| `--dest-dir <path>` | Server-side destination directory (overrides `FASTSYNC_DEST_DIR`) | | `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
| `--save-to-disk` | Persist received files to disk | | `--dest-dir <path>` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) |
| `--save-to-disk` | Write received files to disk |
### Environment Variables ### Environment Variables
| Variable | Description | Default | | Variable | Default | Description |
|----------|-------------|---------| |----------|---------|-------------|
| `FASTSYNC_SOURCE_DIR` | Source directory for files (fallback, overridden by `--source-dir`) | Current user's documents directory | | `FASTSYNC_SOURCE_DIR` | User documents | Source directory fallback |
| `FASTSYNC_DEST_DIR` | Destination directory (fallback, overridden by `--dest-dir`) | `./data_copied` | | `FASTSYNC_DEST_DIR` | `./data_copied` | Destination directory fallback |
| `FASTSYNC_SERVER_IP` | Server IP address | `127.0.0.1` | | `FASTSYNC_SERVER_IP` | `127.0.0.1` | Server address |
| `FASTSYNC_SERVER_PORT` | Server port | `8080` | | `FASTSYNC_SERVER_PORT` | `8080` | Server port |
| `FASTSYNC_SAVE_TO_DISK` | Save to disk (fallback, overridden by `--save-to-disk`) | `false` | | `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback |
## Implementation Details ## Implementation Details
### Data Structures ### Data Structures
1. **Chunk** — collection of files (~10 MB total)
1. **Chunk**: Collection of files (default 10MB total size) 2. **File** — path, content (`Data`), optional `FileMetadata` pointer
2. **File**: File metadata with path and content 3. **FileMetadata** — `mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`
3. **FileReceive**: Received file data structure 4. **Config** — runtime parameters
4. **Config**: Configuration parameters structure 5. **Queue** — thread-safe queue with condition variables
5. **Queue**: Thread-safe queue implementation using condition variables
### Key Algorithms ### Key Algorithms
1. **File scanning** — recursive BFS directory traversal
1. **File Scanning**: Recursive directory traversal with BFS 2. **Chunking** — files grouped by size limit
2. **Chunking**: Files grouped into chunks with size limit 3. **Compression** — zstd with configurable level
3. **Compression**: zstd compression with configurable levels 4. **Network protocol** — custom TCP with status codes and optional metadata packing
4. **Network Protocol**: Custom TCP-based protocol with status codes 5. **Metadata restoration** — `chmod()`, `chown()`, `utimensat()` on the receiving side
5. **Thread Synchronization**: Condition variables and mutexes for thread coordination
## Build Requirements ## Build Requirements
- C11 compatible compiler - C11 compiler
- CMake 4.1 or later - CMake 4.1+
- zstd library - zstd library
- pthread support - pthreads
## Building ## Building
```bash ```bash
mkdir -p build && cd build cmake -B build -S . && cmake --build build -j$(nproc)
cmake ..
make
``` ```
## Running ## Running
@@ -107,66 +104,41 @@ make
### Client ### Client
```bash ```bash
# Basic usage with default settings (sends from ~/Documents/...) # Basic
./build/client -m -c 10
# Specify source and destination directories
./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk ./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk
# Chunk serialization mode (batch per chunk) # With metadata preservation
./build/client -s ./build/client -M --source-dir ... --dest-dir ...
# Compressed chunk serialization # Multithreaded + compression
./build/client -s -c 3 ./build/client -m -c 10
# Multithreaded with compressed chunk serialization # Sendfile (zero-copy)
./build/client -m -s -c 3
# Sendfile (zero-copy, bypasses userspace for large files)
./build/client -f ./build/client -f
# Sendfile with multithreading # All features
./build/client -f -m ./build/client -m -c -s -M
``` ```
## Testing ## Testing
The project includes comprehensive unit tests for core functionality:
```bash ```bash
# Unit tests
./build/tests ./build/tests
```
An integration test / benchmark script runs all configurations against ~50 MB of generated test data with byte-for-byte verification: # Integration benchmark (~50 MB data, 13 configurations + rsync comparison)
```bash
python3 test.py python3 test.py
# Skip the throttled suite (disk I/O limits + 100ms network delay) if sudo is unavailable:
python3 test.py --no-throttled # Profiles: --wan (100 Mbit, 50 ms, 1% loss), --unlimited (no throttling)
python3 test.py --wan
``` ```
## Code Organization The benchmark prints throughput metrics for the best configuration and speedup vs rsync.
```
src/
client/ # Client implementation
server/ # Server implementation
shared/ # Shared data structures and utilities
tests/ # Unit tests
```
## Performance Considerations ## Performance Considerations
1. Chunk size (10MB default) affects memory usage and transfer efficiency 1. Chunk size (~10 MB) balances memory and transfer efficiency
2. Compression level (1-22) trades CPU usage for space savings 2. Compression level trades CPU for bandwidth
3. `sendfile()` (`-f`) bypasses userspace memory, ~2x faster on localhost for large files 3. `sendfile()` bypasses userspace — ~2× faster on localhost for large files
4. Multithreading improves performance on multi-core systems 4. Multithreading scales with core count
5. Thread-safe queues minimize contention between producer/consumer threads 5. Metadata transfer adds negligible overhead when disabled, ~24 bytes per file when enabled
## Extensibility
The system is designed with clear interfaces that allow for:
1. Additional compression algorithms
2. Different transport protocols
3. Custom storage backends
4. Extended metadata support