TapTap
01045d7d14
- Add file_send_sendfile() using sendfile() syscall to send file content directly from fd to socket, bypassing userspace memory - Add use_sendfile field to Config struct (default false) - Add -f / --sendfile flag parsing in client.c - sendfile path in send_chunk() used when -f is set without -c or -s - send_files_multithreaded() falls back to single-threaded when sendfile is enabled (loader pipeline becomes unnecessary) - Add Sendfile (-f) test case to test.py
166 lines
5.2 KiB
Markdown
166 lines
5.2 KiB
Markdown
# FastFileTransfer
|
|
|
|
A high-performance file synchronization system that implements a custom client-server protocol for efficient file transfer with compression and multithreading support.
|
|
|
|
## Technical Overview
|
|
|
|
FastFileTransfer is a C implementation of a file synchronization system that:
|
|
|
|
1. Uses a custom TCP-based protocol for client-server communication
|
|
2. Implements chunked file transfer (10MB chunks by default)
|
|
3. Supports zstd compression with configurable levels (1-22)
|
|
4. Utilizes multithreading for parallel file processing
|
|
5. Implements producer-consumer patterns with thread-safe queues
|
|
6. Provides both in-memory and disk-based storage options
|
|
|
|
## System Architecture
|
|
|
|
The system consists of two main components:
|
|
|
|
### Client
|
|
- Scans source directories recursively
|
|
- Creates file chunks with configurable size (10MB default)
|
|
- Compresses data using zstd algorithm
|
|
- Serializes chunks into a compact binary format for batch transfer
|
|
- Sends files to server using custom protocol
|
|
- Supports both single-threaded and multi-threaded operation
|
|
|
|
### Server
|
|
- Listens for client connections on port 8080
|
|
- Receives files using the custom protocol
|
|
- Decompresses received data
|
|
- Stores files either in memory or on disk
|
|
- Implements thread pool for parallel processing
|
|
|
|
## Protocol Details
|
|
|
|
The client-server communication uses the following status codes:
|
|
- `STATUS_OK`: Operation successful
|
|
- `STATUS_ERROR`: Error occurred
|
|
- `STATUS_FINISHED`: Transfer complete
|
|
- `STATUS_NEXT`: Ready for next file (per-file mode)
|
|
- `STATUS_CHUNK`: Following data is a serialized chunk (chunk mode)
|
|
|
|
## Configuration Options
|
|
|
|
### Command Line Arguments
|
|
| Argument | Description |
|
|
|----------|-------------|
|
|
| `-m` | Enable multithreading mode |
|
|
| `-c [level]` | Enable compression with optional level (1-22, default: 5) |
|
|
| `-s` | Enable chunk serialization (batch-transfer all files per chunk) |
|
|
| `-f` | Enable sendfile (zero-copy file transfer, bypasses userspace memory) |
|
|
| `--source-dir <path>` | Source directory to sync (overrides `FASTSYNC_SOURCE_DIR`) |
|
|
| `--dest-dir <path>` | Server-side destination directory (overrides `FASTSYNC_DEST_DIR`) |
|
|
| `--save-to-disk` | Persist received files to disk |
|
|
|
|
### Environment Variables
|
|
| Variable | Description | Default |
|
|
|----------|-------------|---------|
|
|
| `FASTSYNC_SOURCE_DIR` | Source directory for files (fallback, overridden by `--source-dir`) | Current user's documents directory |
|
|
| `FASTSYNC_DEST_DIR` | Destination directory (fallback, overridden by `--dest-dir`) | `./data_copied` |
|
|
| `FASTSYNC_SERVER_IP` | Server IP address | `127.0.0.1` |
|
|
| `FASTSYNC_SERVER_PORT` | Server port | `8080` |
|
|
| `FASTSYNC_SAVE_TO_DISK` | Save to disk (fallback, overridden by `--save-to-disk`) | `false` |
|
|
|
|
## Implementation Details
|
|
|
|
### Data Structures
|
|
|
|
1. **Chunk**: Collection of files (default 10MB total size)
|
|
2. **File**: File metadata with path and content
|
|
3. **FileReceive**: Received file data structure
|
|
4. **Config**: Configuration parameters structure
|
|
5. **Queue**: Thread-safe queue implementation using condition variables
|
|
|
|
### Key Algorithms
|
|
|
|
1. **File Scanning**: Recursive directory traversal with BFS
|
|
2. **Chunking**: Files grouped into chunks with size limit
|
|
3. **Compression**: zstd compression with configurable levels
|
|
4. **Network Protocol**: Custom TCP-based protocol with status codes
|
|
5. **Thread Synchronization**: Condition variables and mutexes for thread coordination
|
|
|
|
## Build Requirements
|
|
|
|
- C11 compatible compiler
|
|
- CMake 4.1 or later
|
|
- zstd library
|
|
- pthread support
|
|
|
|
## Building
|
|
|
|
```bash
|
|
mkdir -p build && cd build
|
|
cmake ..
|
|
make
|
|
```
|
|
|
|
## Running
|
|
|
|
### Server
|
|
```bash
|
|
./build/server
|
|
```
|
|
|
|
### Client
|
|
```bash
|
|
# Basic usage with default settings (sends from ~/Documents/...)
|
|
./build/client -m -c 10
|
|
|
|
# Specify source and destination directories
|
|
./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk
|
|
|
|
# Chunk serialization mode (batch per chunk)
|
|
./build/client -s
|
|
|
|
# Compressed chunk serialization
|
|
./build/client -s -c 3
|
|
|
|
# Multithreaded with compressed chunk serialization
|
|
./build/client -m -s -c 3
|
|
|
|
# Sendfile (zero-copy, bypasses userspace for large files)
|
|
./build/client -f
|
|
```
|
|
|
|
## Testing
|
|
|
|
The project includes comprehensive unit tests for core functionality:
|
|
|
|
```bash
|
|
./build/tests
|
|
```
|
|
|
|
An integration test / benchmark script runs all configurations against ~50 MB of generated test data with byte-for-byte verification:
|
|
|
|
```bash
|
|
python3 test.py
|
|
# Skip the throttled suite (disk I/O limits + 100ms network delay) if sudo is unavailable:
|
|
python3 test.py --no-throttled
|
|
```
|
|
|
|
## Code Organization
|
|
|
|
```
|
|
src/
|
|
client/ # Client implementation
|
|
server/ # Server implementation
|
|
shared/ # Shared data structures and utilities
|
|
tests/ # Unit tests
|
|
```
|
|
|
|
## Performance Considerations
|
|
|
|
1. Chunk size (10MB default) affects memory usage and transfer efficiency
|
|
2. Compression level (1-22) trades CPU usage for space savings
|
|
3. Multithreading improves performance on multi-core systems
|
|
4. Thread-safe queues minimize contention between producer/consumer threads
|
|
|
|
## Extensibility
|
|
|
|
The system is designed with clear interfaces that allow for:
|
|
1. Additional compression algorithms
|
|
2. Different transport protocols
|
|
3. Custom storage backends
|
|
4. Extended metadata support |