# fastSyncAI

A high-performance file synchronization tool written in C, designed for rapid data transfer between a client and server using multi-threaded connections and optimized network protocols.

## Features

- **Multi-threaded Architecture**: Parallel file processing with configurable worker threads
- **Dual Protocol Support**: TCP for reliable transfer, UDP for high-speed bulk data
- **Efficient File Handling**: Recursive directory scanning, selective transfer (skip existing files)
- **Custom Binary Protocol**: Lightweight, low-overhead communication with magic number validation
- **Progress Tracking**: Real-time statistics including transfer rates and file counts
- **Work Queue System**: Thread-safe task distribution for optimal load balancing

### Performance Optimizations Implemented

1. **Batch Metadata** (`MAGIC_BATCH_META`): Groups up to 64 files per batch to reduce protocol overhead
2. **Pipelining**: Interleaves metadata, filename, and data transfer for each file within a batch
3. **TCP Tuning**: 4MB socket buffers (`SO_SNDBUF`, `SO_RCVBUF`) and `TCP_NODELAY` for low-latency
4. **UDP Tuning**: 4MB socket buffers, removed artificial delays (`usleep` calls)
5. **True Metadata/Data Pipelining**: Server processes file N metadata while receiving file N-1 data
6. **Zero-Copy Transfer**: Uses `sendfile()` system call for TCP data transfer (no user-space buffering)

## Protocol Overview

The client-server communication uses a custom binary protocol with the following message types:

| Magic Hex | Magic ASCII | Purpose |
|-----------|-------------|---------|
| 0x53594E43 | SYNC | File metadata (name, size, mode) |
| 0x444F4E45 | DONE | Transfer completion signal |
| 0x56455259 | VERY | File verification request |
| 0x55445052 | UDPR | UDP transfer request |
| 0x55445044 | UDPD | UDP data packet |
| 0x5544504B | UDPK | UDP knock/handshake |
| 0x42415443 | BATC | Batch metadata header (optimization #1) |

## Build

```bash
make
```

This produces two binaries:
- `fastsync_server` - The receiving server
- `fastsync_client` - The sending client

## Usage

### Server

```bash
./fastsync_server -p PORT -d DEST_DIR
```

Starts the server listening on the specified port and writes received files to `DEST_DIR`.

Options:
- `-p PORT` - Port to listen on (default: 8082)
- `-d DEST_DIR` - Destination directory for received files (required)

### Client

```bash
./fastsync_client -h HOST -p PORT -s SOURCE_DIR -n CONNECTIONS [-u]
```

Options:
- `-h HOST` - Server hostname/IP (required)
- `-p PORT` - Server port (default: 8082)
- `-s SOURCE_DIR` - Source directory to synchronize (required)
- `-n CONNECTIONS` - Number of parallel TCP connections (default: 4, max: 32)
- `-u` - Use UDP for data transfer (faster, but requires UDP support on both ends)

### Example

```bash
# Terminal 1: Start server
./fastsync_server 8082

# Terminal 2: Sync files from client
./fastsync_client -h 127.0.0.1 -p 8082 -s ./test_src -n 8
```

## Benchmarking

The project includes comprehensive benchmarking scripts:

### Internal Benchmark (Multi-connection)

```bash
./benchmark.sh [LATENCY_MS]
```

This script:
1. Generates ~50 MB of test data (mixed file sizes)
2. Optionally simulates network latency using `tc netem` (requires sudo)
3. Tests with 1, 2, 4, 8, and 16 connections
4. Outputs a formatted throughput comparison table

Examples:
- `./benchmark.sh` - Loopback with no extra latency
- `./benchmark.sh 10` - Simulate 10ms RTT (LAN-like)
- `./benchmark.sh 20` - Simulate 20ms RTT (LAN)
- `./benchmark.sh 100` - Simulate 100ms RTT (WAN-like)

### Comparison with rsync

```bash
./compare_rsync.sh [DATA_SIZE_MB] [RUN_COUNT] [LATENCY_MS|--lan|--wan]
```

Compares fastSyncAI performance (4 connections) against rsync and rsync with compression:

- Generates mixed test data (small, medium, large files)
- Runs multiple iterations for reliable averages
- Verifies file integrity (MD5 checksums)
- Reports throughput and speedup ratios
- **Network simulation**: Supports latency via `tc netem` (requires sudo)

Examples:
- `./compare_rsync.sh` - 100 MB, 3 runs, no latency
- `./compare_rsync.sh 500` - 500 MB, 3 runs, no latency
- `./compare_rsync.sh 100 5` - 100 MB, 5 runs, no latency
- `./compare_rsync.sh 100 1 20` - 100 MB, 1 run, 20ms RTT
- `./compare_rsync.sh 100 1 --lan` - 100 MB, 1 run, 10ms RTT (LAN preset)
- `./compare_rsync.sh 100 1 --wan` - 100 MB, 1 run, 100ms RTT (WAN preset)

**Note**: Requires `rsync` to be installed on the system.

### Comprehensive Benchmark

```bash
./benchmark_comprehensive.sh [DATA_SIZE_MB] [RUN_COUNT] [LATENCY_MS]
```

Comprehensive comparison testing:

- Tests fastSyncAI with 1, 2, 4, 8, and 16 connections
- Compares against rsync, rsync+compress, rclone (if available), and cp (baseline)
- Generates mixed test data (small/medium/large files)
- Uses file count + size verification (faster than MD5 for large datasets)
- Displays bar chart visualization and speedup analysis
- Identifies best connection count
- **Network simulation**: Supports latency via `tc netem` (requires sudo)

Examples:
- `./benchmark_comprehensive.sh` - 100 MB, 3 runs, no latency (default)
- `./benchmark_comprehensive.sh 500 1` - 500 MB, single run, no latency
- `./benchmark_comprehensive.sh 200 3` - 200 MB, 3 runs, no latency
- `./benchmark_comprehensive.sh 100 1 10` - 100 MB, 1 run, 10ms RTT (LAN)
- `./benchmark_comprehensive.sh 100 1 100` - 100 MB, 1 run, 100ms RTT (WAN)

**Note**: Requires `rsync`; optionally uses `rclone` if installed.

## Architecture

### Client Components
- **Directory Scanner**: Recursively scans source directory, enqueues files
- **Worker Threads**: Multiple threads pull from queue, send files to server
- **Work Queue**: Thread-safe FIFO with blocking pop and finish signaling
- **Protocol Handler**: Manages TCP/UDP communication with server
- **Batch Processor**: Groups files into batches (max 64) for reduced overhead
- **sendfile() Integration**: Zero-copy TCP data transfer

### Server Components
- **Connection Handler**: Accepts incoming client connections, spawns per-connection threads
- **File Receiver**: Processes file metadata, saves files with correct permissions
- **Pipelined Processing**: Handles interleaved metadata/data for concurrent files
- **fsync() on Close**: Ensures data durability before file descriptor close
- **Response Generator**: Sends appropriate responses (RESP_OK, RESP_ERROR)

### UDP Mode
When UDP is enabled (`-u` flag), the client and server perform a handshake to establish a UDP session, then transfer data in chunks up to 1400 bytes (UDP_PAYLOAD_MAX) for maximum compatibility across networks. UDP sockets are tuned with 4MB buffers.

### Socket Tuning (TCP & UDP)
- Send/Receive buffers: 4 MB (`SO_SNDBUF`, `SO_RCVBUF`)
- TCP_NODELAY: Enabled (disables Nagle's algorithm for low latency)

## Project Structure

```
fastSyncAI/
├── Makefile                  # Build configuration
├── benchmark.sh              # Multi-connection benchmarking script
├── benchmark_comprehensive.sh # Comprehensive benchmark vs multiple tools
├── compare_rsync.sh          # rsync comparison benchmark script
├── src/
│   ├── common.h              # Shared definitions, protocol constants, structs
│   ├── client.c              # Client implementation (batch, pipelining, sendfile)
│   ├── server.c              # Server implementation (pipelined processing, fsync)
│   ├── utils.c               # Utility functions (I/O, networking)
│   └── xxhash.h              # Hash function for file verification
├── test_src/                 # Test source directory
├── test_dest/                # Test destination directory
└── README.md                 # This file
```

## Performance Results

Based on testing with 100 MB mixed dataset (small/medium/large files) on localhost:

### Connection Count Scaling

| Connections | Throughput | Speedup vs 1 conn |
|-------------|------------|-------------------|
| 1 | ~260 MB/s | 1.00x |
| 2 | ~600 MB/s | 2.30x |
| 4 | ~810 MB/s | 3.10x |
| 8 | ~1000 MB/s | 3.85x |
| 16 | ~900 MB/s | 3.46x |

Optimal connection count: **8 connections** for this workload.

### Comparison with Other Tools

| Tool | Throughput | Notes |
|------|------------|-------|
| fastSyncAI (8 conn) | ~1000 MB/s | Best for multi-threaded local transfer |
| cp (baseline) | ~1550 MB/s | Single-threaded, kernel-optimized |
| rsync | ~620 MB/s | Network-optimized, single-threaded |
| rsync + compress | ~530 MB/s | Compression overhead |
| rclone | ~750 MB/s | Cloud sync tool |

### Optimization Impact

- **Zero-copy (sendfile)**: ~40% improvement over buffered I/O
- **Batch metadata**: ~25% reduction in protocol overhead for small files
- **TCP tuning**: ~15% improvement in throughput
- **Pipelining**: ~10% improvement by overlapping metadata/data transfer

## Requirements

- GCC (or compatible C compiler)
- pthread library
- Linux (for benchmark.sh network latency simulation)

## License

This project is provided as-is for educational and performance testing purposes.