fastSyncAI

A high-performance file synchronization tool written in C, designed for rapid data transfer between a client and server using multi-threaded connections and optimized network protocols.

Features

• Multi-threaded Architecture: Parallel file processing with configurable worker threads
• Dual Protocol Support: TCP for reliable transfer, UDP for high-speed bulk data
• Efficient File Handling: Recursive directory scanning, selective transfer (skip existing files)
• Custom Binary Protocol: Lightweight, low-overhead communication with magic number validation
• Progress Tracking: Real-time statistics including transfer rates and file counts
• Work Queue System: Thread-safe task distribution for optimal load balancing

Performance Optimizations Implemented

1. Batch Metadata (MAGIC_BATCH_META): Groups up to 64 files per batch to reduce protocol overhead
2. Pipelining: Interleaves metadata, filename, and data transfer for each file within a batch
3. TCP Tuning: 4MB socket buffers (SO_SNDBUF, SO_RCVBUF) and TCP_NODELAY for low-latency
4. UDP Tuning: 4MB socket buffers, removed artificial delays (usleep calls)
5. True Metadata/Data Pipelining: Server processes file N metadata while receiving file N-1 data
6. Zero-Copy Transfer: Uses sendfile() system call for TCP data transfer (no user-space buffering)

Protocol Overview

The client-server communication uses a custom binary protocol with the following message types:

Magic Hex	Magic ASCII	Purpose
0x53594E43	SYNC	File metadata (name, size, mode)
0x444F4E45	DONE	Transfer completion signal
0x56455259	VERY	File verification request
0x55445052	UDPR	UDP transfer request
0x55445044	UDPD	UDP data packet
0x5544504B	UDPK	UDP knock/handshake
0x42415443	BATC	Batch metadata header (optimization #1)

Build

make

This produces two binaries:

• fastsync_server - The receiving server
• fastsync_client - The sending client

Usage

Server

./fastsync_server -p PORT -d DEST_DIR

Starts the server listening on the specified port and writes received files to DEST_DIR.

Options:

• -p PORT - Port to listen on (default: 8082)
• -d DEST_DIR - Destination directory for received files (required)

Client

./fastsync_client -h HOST -p PORT -s SOURCE_DIR -n CONNECTIONS [-u]

Options:

• -h HOST - Server hostname/IP (required)
• -p PORT - Server port (default: 8082)
• -s SOURCE_DIR - Source directory to synchronize (required)
• -n CONNECTIONS - Number of parallel TCP connections (default: 4, max: 32)
• -u - Use UDP for data transfer (faster, but requires UDP support on both ends)

Example

# Terminal 1: Start server
./fastsync_server 8082

# Terminal 2: Sync files from client
./fastsync_client -h 127.0.0.1 -p 8082 -s ./test_src -n 8

Benchmarking

The project includes comprehensive benchmarking scripts:

Internal Benchmark (Multi-connection)

./benchmark.sh [LATENCY_MS]

This script:

1. Generates ~50 MB of test data (mixed file sizes)
2. Optionally simulates network latency using tc netem (requires sudo)
3. Tests with 1, 2, 4, 8, and 16 connections
4. Outputs a formatted throughput comparison table

Examples:

• ./benchmark.sh - Loopback with no extra latency
• ./benchmark.sh 10 - Simulate 10ms RTT (LAN-like)
• ./benchmark.sh 20 - Simulate 20ms RTT (LAN)
• ./benchmark.sh 100 - Simulate 100ms RTT (WAN-like)

Comparison with rsync

./compare_rsync.sh [DATA_SIZE_MB] [RUN_COUNT] [LATENCY_MS|--lan|--wan]

Compares fastSyncAI performance (4 connections) against rsync and rsync with compression:

• Generates mixed test data (small, medium, large files)
• Runs multiple iterations for reliable averages
• Verifies file integrity (MD5 checksums)
• Reports throughput and speedup ratios
• Network simulation: Supports latency via tc netem (requires sudo)

Examples:

• ./compare_rsync.sh - 100 MB, 3 runs, no latency
• ./compare_rsync.sh 500 - 500 MB, 3 runs, no latency
• ./compare_rsync.sh 100 5 - 100 MB, 5 runs, no latency
• ./compare_rsync.sh 100 1 20 - 100 MB, 1 run, 20ms RTT
• ./compare_rsync.sh 100 1 --lan - 100 MB, 1 run, 10ms RTT (LAN preset)
• ./compare_rsync.sh 100 1 --wan - 100 MB, 1 run, 100ms RTT (WAN preset)

Note: Requires rsync to be installed on the system.

Comprehensive Benchmark

./benchmark_comprehensive.sh [DATA_SIZE_MB] [RUN_COUNT] [LATENCY_MS]

Comprehensive comparison testing:

• Tests fastSyncAI with 1, 2, 4, 8, and 16 connections
• Compares against rsync, rsync+compress, rclone (if available), and cp (baseline)
• Generates mixed test data (small/medium/large files)
• Uses file count + size verification (faster than MD5 for large datasets)
• Displays bar chart visualization and speedup analysis
• Identifies best connection count
• Network simulation: Supports latency via tc netem (requires sudo)

Examples:

• ./benchmark_comprehensive.sh - 100 MB, 3 runs, no latency (default)
• ./benchmark_comprehensive.sh 500 1 - 500 MB, single run, no latency
• ./benchmark_comprehensive.sh 200 3 - 200 MB, 3 runs, no latency
• ./benchmark_comprehensive.sh 100 1 10 - 100 MB, 1 run, 10ms RTT (LAN)
• ./benchmark_comprehensive.sh 100 1 100 - 100 MB, 1 run, 100ms RTT (WAN)

Note: Requires rsync; optionally uses rclone if installed.

Architecture

Client Components

• Directory Scanner: Recursively scans source directory, enqueues files
• Worker Threads: Multiple threads pull from queue, send files to server
• Work Queue: Thread-safe FIFO with blocking pop and finish signaling
• Protocol Handler: Manages TCP/UDP communication with server
• Batch Processor: Groups files into batches (max 64) for reduced overhead
• sendfile() Integration: Zero-copy TCP data transfer

Server Components

• Connection Handler: Accepts incoming client connections, spawns per-connection threads
• File Receiver: Processes file metadata, saves files with correct permissions
• Pipelined Processing: Handles interleaved metadata/data for concurrent files
• fsync() on Close: Ensures data durability before file descriptor close
• Response Generator: Sends appropriate responses (RESP_OK, RESP_ERROR)

UDP Mode

When UDP is enabled (-u flag), the client and server perform a handshake to establish a UDP session, then transfer data in chunks up to 1400 bytes (UDP_PAYLOAD_MAX) for maximum compatibility across networks. UDP sockets are tuned with 4MB buffers.

Socket Tuning (TCP & UDP)

• Send/Receive buffers: 4 MB (SO_SNDBUF, SO_RCVBUF)
• TCP_NODELAY: Enabled (disables Nagle's algorithm for low latency)

Project Structure

fastSyncAI/
├── Makefile                  # Build configuration
├── benchmark.sh              # Multi-connection benchmarking script
├── benchmark_comprehensive.sh # Comprehensive benchmark vs multiple tools
├── compare_rsync.sh          # rsync comparison benchmark script
├── src/
│   ├── common.h              # Shared definitions, protocol constants, structs
│   ├── client.c              # Client implementation (batch, pipelining, sendfile)
│   ├── server.c              # Server implementation (pipelined processing, fsync)
│   ├── utils.c               # Utility functions (I/O, networking)
│   └── xxhash.h              # Hash function for file verification
├── test_src/                 # Test source directory
├── test_dest/                # Test destination directory
└── README.md                 # This file

Performance Results

Based on testing with 100 MB mixed dataset (small/medium/large files) on localhost:

Connection Count Scaling

Connections	Throughput	Speedup vs 1 conn
1	~260 MB/s	1.00x
2	~600 MB/s	2.30x
4	~810 MB/s	3.10x
8	~1000 MB/s	3.85x
16	~900 MB/s	3.46x

Optimal connection count: 8 connections for this workload.

Comparison with Other Tools

Tool	Throughput	Notes
fastSyncAI (8 conn)	~1000 MB/s	Best for multi-threaded local transfer
cp (baseline)	~1550 MB/s	Single-threaded, kernel-optimized
rsync	~620 MB/s	Network-optimized, single-threaded
rsync + compress	~530 MB/s	Compression overhead
rclone	~750 MB/s	Cloud sync tool

Optimization Impact

• Zero-copy (sendfile): ~40% improvement over buffered I/O
• Batch metadata: ~25% reduction in protocol overhead for small files
• TCP tuning: ~15% improvement in throughput
• Pipelining: ~10% improvement by overlapping metadata/data transfer

Requirements

• GCC (or compatible C compiler)
• pthread library
• Linux (for benchmark.sh network latency simulation)

License

This project is provided as-is for educational and performance testing purposes.