fastSyncAI

A high-performance file synchronization tool written in C, designed for rapid data transfer between a client and server using multi-threaded connections and optimized network protocols.

Features

• Multi-threaded Architecture: Parallel file processing with configurable worker threads (default: 4, max: 32)
• TCP-based Transfer: Reliable, optimized TCP protocol with pipelining and batching
• LZ4 Compression: Built-in LZ4 compression for TCP transfers (enabled by default, use -c to disable)
• Experimental UDP Mode: UDP for bulk data transfer (disabled by default, use -u flag)
• Efficient File Handling: Recursive directory scanning, selective transfer (skip existing files)
• Custom Binary Protocol: Lightweight, low-overhead communication with magic number validation
• Progress Tracking: Real-time statistics including transfer rates and file counts
• Work Queue System: Thread-safe task distribution for optimal load balancing

Performance Optimizations Implemented

1. Batch Metadata (MAGIC_BATCH_META): Groups up to 64 files per batch to reduce protocol overhead
2. Pipelining: Interleaves metadata, filename, and data transfer for each file within a batch
3. TCP Tuning: 4MB socket buffers (SO_SNDBUF, SO_RCVBUF) and TCP_NODELAY for low-latency
4. True Metadata/Data Pipelining: Server processes file N metadata while receiving file N-1 data
5. Zero-Copy Transfer: Uses sendfile() system call for TCP data transfer (no user-space buffering)
6. LZ4 Compression: Optional LZ4 compression for compressible data (text, logs, etc.)

Protocol Overview

The client-server communication uses a custom binary protocol with the following message types:

Magic Hex	Magic ASCII	Purpose
0x53594E43	SYNC	File metadata (name, size, mode)
0x444F4E45	DONE	Transfer completion signal
0x56455259	VERY	File verification request
0x55445052	UDPR	UDP transfer request
0x55445044	UDPD	UDP data packet
0x5544504B	UDPK	UDP knock/handshake
0x55445041	UDPA	UDP acknowledgment
0x42415443	BATC	Batch metadata header

Build

make

This produces two binaries:

• fastsync_server - The receiving server
• fastsync_client - The sending client

Usage

Server

./fastsync_server -p PORT -d DEST_DIR

Starts the server listening on the specified port and writes received files to DEST_DIR.

Options:

• -p PORT - Port to listen on (default: 8082)
• -d DEST_DIR - Destination directory for received files (required)

Client

./fastsync_client -h HOST -p PORT -s SOURCE_DIR -n CONNECTIONS [-u]

Options:

• -h HOST - Server hostname/IP (required)
• -p PORT - Server port (default: 8082)
• -s SOURCE_DIR - Source directory to synchronize (required)
• -n CONNECTIONS - Number of parallel TCP connections (default: 4, max: 32)
• -u - EXPERIMENTAL: Use UDP for data transfer (requires -u on both client and server; disabled by default)
• -c - DISABLE compression (default: ON with LZ4)
• -l LEVEL - LZ4 compression level (1=fastest, 12=best, default: 1)

Example

# Terminal 1: Start server
./fastsync_server -p 8082 -d ./test_dest

# Terminal 2: Sync files from client (TCP, default)
./fastsync_client -h 127.0.0.1 -p 8082 -s ./test_src -n 8

# Terminal 2: Sync with experimental UDP (both ends must support -u)
./fastsync_server -p 8082 -d ./test_dest  # Server: UDP auto-detected
./fastsync_client -h 127.0.0.1 -p 8082 -s ./test_src -n 8 -u  # Client: enable UDP

UDP Mode (Experimental)

Status: UDP transfer is currently experimental and disabled by default.

• Works on localhost (verified: 45-74 MB/s)
• Known issue: Hangs with simulated packet loss (tc netem)
• Requires -u flag on the client; server auto-detects UDP requests
• Uses sliding window protocol with selective ACKs, window size 16, max 5 retries
• Not recommended for production use

Recommendation: Use TCP for all transfers. UDP code remains in codebase for future improvement.

Compression

fastSyncAI includes built-in LZ4 compression for TCP transfers, enabled by default.

• Default: Compression is ON (use -c flag to disable)
• Algorithm: LZ4 - fast compression with good ratio for compressible data
• Behavior: Automatically falls back to uncompressed transfer if compression doesn't reduce file size
• Overhead: Adds memory usage (file must be fully read into memory for compression)
• Best for: Text files, logs, databases, any compressible data

Adaptive Compression (NEW!)

fastSyncAI now features adaptive compression that automatically selects the optimal compression level based on file type:

File Type	Compression Level	Reasoning
Text files (.txt, .log, .csv, .json, .xml, .html, .js, .py, .c, .h, .cpp, .java, .sql, .sh)	Level 9	High compression ratio, CPU worth it
Config files (.cfg, .conf, .yaml, .ini)	Level 9	Typically text-based, good compression
Database files (.db, .sqlite, .mdb)	Level 9	Structured data compresses well
Already compressed (images, audio, video, archives, PDFs, binaries)	Level 1	Minimal CPU, won't compress much
Unknown types	Default level	User-configured or 1

This is enabled by default and works alongside the file type detection that skips compression entirely for already-compressed files.

Manual Compression Level Selection

You can manually override the compression level with the -l flag:

# Fastest compression (level 1 - default)
./fastsync_client -h 127.0.0.1 -s ./data -l 1

# Best compression (level 12 - slower but better ratio)
./fastsync_client -h 127.0.0.1 -s ./data -l 12

Additional Optimizations (NEW!)

1. Smart Skip: Automatically skips compression for 30+ already-compressed file extensions
2. Size Threshold: Skips compression for files < 1KB (overhead > benefit)
3. Memory Reuse: Allocates compression buffers once per batch instead of per-file
4. No Double-Read: Uses already-read buffer when compression doesn't help (eliminates redundant disk I/O)

When to disable compression (-c flag):

• Maximum throughput on localhost/LAN - uncompressed sendfile() is faster
• All files are already compressed

Benchmarking

The project includes a comprehensive network benchmarking script that compares fastSyncAI against rsync:

Network Benchmark

./benchmark_network.sh [SCENARIO] [SIZE_MB] [RUN_COUNT]

Scenarios (preset network conditions via tc netem):

• --lan - LAN simulation (10ms RTT, 0.1% loss, ±2ms jitter)
• --wan - WAN simulation (100ms RTT, 0.5% loss, ±10ms jitter)
• --wan-loss-1 - WAN with 1% packet loss
• --wan-loss-5 - WAN with 5% packet loss
• --wan-jitter - WAN with 50ms jitter
• --custom LATENCY Loss% JITTER - Custom network conditions

Size: Test data size in MB (default: 50) Run Count: Number of iterations (default: 1)

Examples:

• ./benchmark_network.sh --lan 50 1 - LAN, 50MB, 1 run
• ./benchmark_network.sh --wan 100 3 - WAN, 100MB, 3 runs
• ./benchmark_network.sh --wan-loss-1 200 1 - WAN with 1% loss, 200MB, 1 run
• ./benchmark_network.sh --custom 50 0.5% 5 100 1 - Custom: 50ms RTT, 0.5% loss, 5ms jitter, 100MB, 1 run

Note: Requires rsync and sudo (for tc netem network simulation).

Architecture

Client Components

• Directory Scanner: Recursively scans source directory, enqueues files
• Worker Threads: Multiple threads pull from queue, send files to server
• Work Queue: Thread-safe FIFO with blocking pop and finish signaling
• Protocol Handler: Manages TCP/UDP communication with server
• Batch Processor: Groups files into batches (max 64) for reduced overhead
• sendfile() Integration: Zero-copy TCP data transfer

Server Components

• Connection Handler: Accepts incoming client connections, spawns per-connection threads
• File Receiver: Processes file metadata, saves files with correct permissions
• Pipelined Processing: Handles interleaved metadata/data for concurrent files
• fsync() on Close: Ensures data durability before file descriptor close
• Response Generator: Sends appropriate responses (RESP_OK, RESP_ERROR)

Socket Tuning (TCP & UDP)

• Send/Receive buffers: 4 MB (SO_SNDBUF, SO_RCVBUF)
• TCP_NODELAY: Enabled (disables Nagle's algorithm for low latency)

Project Structure

fastSyncAI/
├── Makefile                      # Build configuration
├── LICENCE.md                    # PolyForm Noncommercial License 1.0.0
├── benchmark_network.sh          # Network condition benchmarking
├── src/
│   ├── common.h                  # Shared definitions, protocol constants, structs
│   ├── client.c                  # Client implementation (batch, pipelining, sendfile, LZ4, UDP)
│   ├── server.c                  # Server implementation (pipelined processing, fsync, LZ4, UDP)
│   ├── utils.c                   # Utility functions (I/O, networking)
│   ├── lz4.h                     # LZ4 compression library header
│   ├── lz4.c                     # LZ4 compression library implementation
│   └── xxhash.h                  # Hash function for file verification
├── test_src/                     # Test source directory
├── test_dest/                    # Test destination directory
└── README.md                     # This file

Performance Results

Based on recent testing with 50 MB mixed dataset on LAN simulation (10ms RTT, 0.1% loss, ±2ms jitter):

Connection Count Scaling (TCP)

Connections	Throughput	Time
1	193.00 MB/s	259.1ms
2	179.87 MB/s	278.0ms
4	167.84 MB/s	297.9ms
8	190.41 MB/s	262.6ms
16	198.14 MB/s	252.3ms

Optimal connection count: 16 connections for this LAN workload.

Comparison with rsync (LAN simulation)

Tool	Throughput	Relative Speed
fastSyncAI 16 conn	198.14 MB/s	3.47x
fastSyncAI 8 conn	190.41 MB/s	3.32x
fastSyncAI 4 conn	167.84 MB/s	2.93x
fastSyncAI 2 conn	179.87 MB/s	3.14x
fastSyncAI 1 conn	193.00 MB/s	3.37x
rsync (TCP)	57.34 MB/s	1.00x (baseline)
rsync + compress	19.12 MB/s	0.33x

Conclusion: fastSyncAI is 3-4x faster than rsync on LAN conditions, with 16 connections providing the best throughput.

UDP Performance (localhost, no loss)

Mode	Speed	Status
UDP (current)	45-74 MB/s	Experimental, slower than TCP
TCP (16 conn)	198 MB/s	Production, stable

Note: UDP is currently not production-ready and offers no speed advantage over TCP.

Compression Performance (localhost, text data)

Mode	File Size	Transfer Size	Speed	Ratio
TCP + LZ4	67 KB	~15 KB	86 MB/s	~4.5:1
TCP only	67 KB	67 KB	108 MB/s	1:1

Note: Compression reduces network transfer at the cost of CPU. On WAN, compression typically wins.

Optimization Impact

• Zero-copy (sendfile): ~40% improvement over buffered I/O
• Batch metadata: ~25% reduction in protocol overhead for small files
• TCP tuning: ~15% improvement in throughput
• Pipelining: ~10% improvement by overlapping metadata/data transfer
• LZ4 compression: 2-5x reduction in transfer size for compressible data
• Smart compression skip: 30-50% CPU reduction for mixed file sets (NEW!)
• Memory reuse: Reduces malloc/free overhead in batch processing (NEW!)
• No double-read: Eliminates redundant disk I/O for incompressible files (NEW!)
• Adaptive compression: Optimal level per file type, better ratio for text (NEW!)

Requirements

• GCC (or compatible C compiler)
• pthread library
• Linux (for tc netem network simulation in benchmarks)
• rsync (for benchmark comparisons)
• sudo access (for network simulation)

License

This project is licensed under the PolyForm Noncommercial License 1.0.0. See LICENCE.md for full license text.

Summary: Free for non-commercial use only. No commercial use permitted without separate licensing agreement.