Compare commits

..

38 Commits

Author SHA1 Message Date
TapTap fed77f6ce0 ci: use v5 Docker image to force runner to pull updated image with libssl-dev
CI / build-and-test (push) Successful in 33s
CI / build-and-test (pull_request) Successful in 27s
2026-07-18 17:53:08 +02:00
TapTap 6de0998f69 ci: retrigger CI with updated Docker image
CI / build-and-test (push) Failing after 3s
CI / build-and-test (pull_request) Failing after 3s
2026-07-18 17:50:55 +02:00
TapTap 43cf0bd81e ci: add libssl-dev to Docker image for TLS transport
CI / build-and-test (push) Failing after 2s
CI / build-and-test (pull_request) Failing after 2s
2026-07-18 17:43:46 +02:00
TapTap 907baf3379 fix: initialize ssl/ssl_ctx in client_connect_ssh to prevent use-after-free
CI / build-and-test (push) Failing after 3s
CI / build-and-test (pull_request) Failing after 3s
2026-07-18 17:39:37 +02:00
TapTap 8b6550f8c6 fix: io_set_ssl NULL in client_disconnect, SSL* type safety in protocol.h 2026-07-18 17:39:37 +02:00
TapTap 20ede4391c Fix re-review: remove __thread from io_ssl (breaks -m), strtol port parsing, --ca warning 2026-07-18 17:39:37 +02:00
TapTap 864eb1316d Fix TLS code review issues: SSL cleanup, TLS 1.2 min, CA verify, deprecation guards, server --help, port validation, __thread io_ssl, shared accept loop 2026-07-18 17:39:37 +02:00
TapTap eff50346fd TLS transport: OpenSSL-based encrypted TCP
- New transport_tls.h/c: TLS server (server_create_tls, server_listen_tls)
  and client (client_connect_tls) using OpenSSL
- protocol.c: io_set_ssl() + SSL_read/SSL_write in send_n_data/receive_n_data
- transport_tcp.h: ssl/ssl_ctx fields added to Server/Client structs
- config.h/c: use_tls, tls_cert, tls_key fields
- client_cli.c: --tls, --cert, --key flags
- server.c: --tls, --cert, --key, -p flags with TLS support
- CMakeLists.txt: OpenSSL::SSL + OpenSSL::Crypto linkage
- shell.nix: openssl added to buildInputs
2026-07-18 17:38:55 +02:00
TapTap 25383f893b Merge pull request 'Incremental sync: --incremental to skip unchanged files' (#14) from incremental-sync into main
CI / build-and-test (push) Successful in 27s
Reviewed-on: #14
2026-07-18 17:36:25 +02:00
TapTap 46c650e9ae refactor: consolidate _no_path file send variants
CI / build-and-test (push) Successful in 26s
CI / build-and-test (pull_request) Successful in 26s
Add bool send_path parameter to file_send_single_calls and
file_send_sendfile, remove the _no_path variants. Callers in
client_send.c pass true (send path) or false (skip path) based
on whether an incremental check already transmitted the path.
2026-07-18 17:27:33 +02:00
TapTap 7876721906 refactor: extract receive_chunk_data and receive_manifest shared helpers
- receive_chunk_data(fd, config) -> Chunk*: shared receive/decompress/deserialize
  (eliminates ~25 lines of duplication between server.c and multiprocessing.c)
- receive_manifest(fd, config, next_status): moved from server.c to file.c,
  fixes missing 'Deleting files not in manifest...' log in multiprocessing.c
- Removes redundant manual free loop in multiprocessing.c manifest handling
  (array_list_create(free) destructor already handles this)
- server.c and multiprocessing.c: remove local chunk/manifest helpers,
  remove compression.h include (no longer needed)
2026-07-18 17:19:12 +02:00
TapTap 5ed12f2bf0 refactor: deduplicate server receive logic, extract shared helpers
- Extract file_save_to_disk() helper (replaces 4x duplicated disk-save boilerplate)
- Extract receive_incremental_check() shared helper (deduplicates STATUS_CHECK
  handling between server.c single-threaded and multiprocessing.c multi-threaded paths)
- Break receive_files() into receive_chunk() and receive_manifest() sub-handlers
- Add incremental sync test case to test.py
- Rebase onto main
2026-07-18 17:19:12 +02:00
TapTap 03d6ba0da5 fix: address review #69 - STATUS_ERROR sends, metadata_receive error handling, extract incremental_check helper 2026-07-18 17:19:12 +02:00
TapTap f1af88a25c Fix incremental-sync review: sendfile_no_path, reject -s+incremental, no data mutation, auto -M, STATUS_ERROR handling, PROTOCOL_VERSION bump 2026-07-18 17:19:12 +02:00
TapTap 1905369c9f Incremental sync: --incremental flag to skip unchanged files
- New STATUS_CHECK protocol status (value 6)
- Client sends path + size + mtime; server replies OK (skip) or NEXT (send)
- config.h/c: use_incremental field sent/received over wire
- file.c/h: file_send_single_calls_no_path helper for incremental path
- client_cli.c: --incremental flag
- client_send.c: per-file check before send in both single and chunk paths
- server.c: STATUS_CHECK handling in receive_files()
- multiprocessing.c: STATUS_CHECK handling in receive_thread()
- test.py: incremental sync test case
2026-07-18 17:19:12 +02:00
TapTap a3acf4b8b6 Merge pull request 'test: add --full flag for lightweight default mode' (#18) from ci-test-light into main
CI / build-and-test (push) Successful in 26s
Reviewed-on: #18
2026-07-18 16:33:02 +02:00
TapTap 478bdd6fe6 add docker as dependency
CI / build-and-test (push) Successful in 27s
CI / build-and-test (pull_request) Successful in 26s
2026-07-18 16:31:55 +02:00
TapTap e408b08443 fix: skip netem_reset when no sudo (light mode)
CI / build-and-test (push) Successful in 26s
CI / build-and-test (pull_request) Successful in 27s
2026-07-18 16:10:52 +02:00
TapTap 2c904bfac5 fix: handle missing ssh in preflight checks
CI / build-and-test (push) Failing after 4s
CI / build-and-test (pull_request) Failing after 4s
2026-07-18 16:08:26 +02:00
TapTap e9554c1e69 ci: add integration tests (light mode)
CI / build-and-test (push) Failing after 4s
CI / build-and-test (pull_request) Failing after 4s
2026-07-18 16:06:22 +02:00
TapTap 8f2205dd30 test: add --full flag, default mode runs 5 fast tests (~0.3s)
CI / build-and-test (push) Successful in 4s
CI / build-and-test (pull_request) Successful in 4s
Default mode skips bulk data, network shaping, SSH/rsync
comparisons, and feature tests. --full restores original
31-test suite with LAN profiling and all benchmarks.
2026-07-18 16:02:36 +02:00
TapTap 70ab76fe90 Merge pull request 'ci: add build and unit test pipeline' (#17) from actions into main
CI / build-and-test (push) Successful in 4s
Reviewed-on: #17
2026-07-18 15:52:06 +02:00
TapTap ed7f8ee56e ci: add g++ and libc6-dev, use v4 image (verified locally)
CI / build-and-test (push) Successful in 4s
CI / build-and-test (pull_request) Successful in 4s
2026-07-18 15:31:41 +02:00
TapTap 8c7ebbfe32 ci: add make to Docker image, use v3 tag
CI / build-and-test (push) Failing after 10s
2026-07-18 15:27:35 +02:00
TapTap cc09bedc07 ci: use v2 image tag to bypass runner cache
CI / build-and-test (push) Failing after 8s
2026-07-18 15:20:57 +02:00
TapTap 5fe40a5242 ci: add node.js to Docker image for checkout action
CI / build-and-test (push) Failing after 1s
2026-07-18 15:19:41 +02:00
TapTap 9130ee4b03 ci: use custom Docker image with build tools pre-installed
CI / build-and-test (push) Failing after 1s
2026-07-18 15:15:26 +02:00
TapTap 36da0d6adb ci: restore apt-get update for package resolution
CI / build-and-test (push) Successful in 2m16s
2026-07-18 14:45:03 +02:00
TapTap e7087d518b ci: skip apt-get update for faster installs
CI / build-and-test (push) Failing after 2s
2026-07-18 14:43:02 +02:00
TapTap 3ea11b5984 ci: add build + unit tests workflow, lower cmake minimum to 3.22
CI / build-and-test (push) Successful in 2m17s
2026-07-18 14:41:27 +02:00
TapTap 33f70f7de3 add demo action
Gitea Actions Demo / Explore-Gitea-Actions (push) Successful in 59s
2026-07-18 14:22:58 +02:00
TapTap 45dd62f568 Merge pull request 'Bandwidth throttling: --bwlimit <KB/s> with token bucket' (#15) from bwlimit into main
Reviewed-on: #15
2026-07-17 10:22:46 +02:00
TapTap 4f8ae5bb70 fix: bw_tokens initialized to io_bwlimit, overflow check for --bwlimit 2026-07-17 10:07:18 +02:00
TapTap e2dac589a8 Merge pull request 'Add custom opencode agents for FastSync development' (#16) from add-custom-agents into main
Reviewed-on: #16
2026-07-17 10:02:04 +02:00
TapTap 1b87131e82 Add pr-review and pr-build skills
- pr-review: read-only code review of PR branches, outputs findings report
- pr-build: builds, tests, and fixes PR branches, commits fixes
2026-07-16 17:07:44 +02:00
TapTap 3dcaf0b79d Fix bwlimit review: remove __thread (breaks -m), validate >0, handle nanosleep EINTR 2026-07-16 17:00:14 +02:00
TapTap ef6aa143d0 Add custom opencode agents for FastSync development
- c-reviewer: memory/thread safety, style review for C11 code
- test-writer: unit test generation using custom test framework
- cmake-expert: CMake build system management
- perf-analyst: transfer pipeline performance analysis
- protocol-designer: wire protocol design and extension
- doc-generator: API docs, protocol specs, usage examples
2026-07-16 16:58:22 +02:00
TapTap 09a76e2a9a Bandwidth throttling: --bwlimit <KB/s> with token bucket
- protocol.h/c: io_set_bwlimit() + token bucket in send_n_data
  (64KB chunks, nanosleep-based deficit compensation)
- client_cli.c: --bwlimit <KB/s> flag
- test.py: bandwidth limit test case
2026-07-16 15:42:16 +02:00
28 changed files with 1397 additions and 343 deletions
+23
View File
@@ -0,0 +1,23 @@
name: CI
on: [push, pull_request]
jobs:
build-and-test:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v5
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Configure
run: cmake -B build -S .
- name: Build
run: cmake --build build -j$(nproc)
- name: Unit Tests
run: ./build/tests
- name: Integration Tests
run: python3 test.py
+75
View File
@@ -0,0 +1,75 @@
---
description: Reviews C code for memory safety, thread safety, null checks, buffer overflows, and style conventions specific to the FastSync codebase.
mode: subagent
---
You are a C code reviewer for the FastSync project — a high-performance file synchronization system written in C11.
## Your Role
Review C source files for correctness, safety, and style. You have deep knowledge of this codebase's patterns and conventions.
## Codebase Context
### Project Structure
- `src/shared/` — shared libraries (protocol, compression, queue, config, data, metadata, transport, etc.)
- `src/client/` — client CLI, file sending, scanner
- `src/server/` — TCP server
- `tests/` — unit tests with custom framework
### Key Data Types
- `Data` — generic buffer (`void *data`, `size_t size`). Always use `data_create()` / `data_destroy()`.
- `Queue` — thread-safe bounded queue with optional `item_destroyer` callback. Use `queue_create()` / `queue_destroy()`.
- `Config` — runtime configuration struct. Use `config_create()` / `config_delete()`.
- `Chunk` — collection of files for batch transfer.
- `FileMetadata` — mode, uid, gid, mtime fields.
- `Server` / `Client` — TCP transport structs.
### Threading
- Uses C11 `<threads.h>` (`thrd_t`, `mtx_t`, `cnd_t`), NOT pthreads directly.
- Producer-consumer pattern with `queue_enqueue_multithreaded()` / `queue_dequeue_multithreaded()`.
- Bounded queues use condition variables for signaling.
### Memory Conventions
- All heap allocations use `malloc`/`calloc`/`realloc` + `free`.
- Destroy functions (`data_destroy`, `queue_destroy`, `config_delete`, etc.) handle cleanup.
- Ownership is transferred at function boundaries — document who owns what.
## Review Checklist
### Memory Safety
- Every `malloc`/`calloc` has a corresponding `free` on all code paths (including error paths).
- No use-after-free: check that pointers aren't used after their destroy function is called.
- No double-free: ensure destroy functions aren't called twice on the same object.
- Null checks after allocation before use.
- Buffer sizes are correct — no off-by-one in string operations (`strlen` + 1 for null terminator).
- `Data` objects created with `data_create()` and freed with `data_destroy()`.
### Thread Safety
- Shared state accessed under proper mutex protection.
- No race conditions on queue operations — using `_multithreaded` variants when threads are involved.
- Condition variable signals happen under the lock.
- No deadlock potential — consistent lock ordering.
- `done` flags checked properly in consumer loops.
### Protocol Safety
- `send_n_data` / `receive_n_data` return values checked.
- Status codes validated before use.
- Config serialization/deserialization handles partial reads.
### Style
- Header guards: `#ifndef FILENAME_H` / `#define FILENAME_H` / `#endif`
- Function naming: `snake_case`, prefixed by module (`queue_create`, `data_compress`, `config_send`).
- `static` for file-local functions.
- Consistent pointer style: `Type *name` (space before asterisk).
- Error handling: return `false`/`NULL` on failure, log when appropriate.
## Output Format
For each issue found, report:
1. **File and line** — exact location
2. **Severity** — critical / warning / style
3. **Category** — memory / thread / protocol / style
4. **Description** — what's wrong and how to fix it
If the code is clean, say so explicitly. Be concise — don't pad with fluff.
+91
View File
@@ -0,0 +1,91 @@
---
description: Manages the CMake build system for FastSync — adding targets, source files, dependencies, compiler flags, and sanitizer configurations.
mode: subagent
---
You are a CMake expert for the FastSync project — a high-performance file synchronization system built with CMake 4.1+ and C11.
## Your Role
Manage the CMake build system: add new targets, configure dependencies, set compiler flags, and handle build configurations.
## Current Build Setup
### `CMakeLists.txt` (project root)
```cmake
cmake_minimum_required(VERSION 4.1)
project(FastFileTransfer)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_C_STANDARD 11)
set(CMAKE_C_STANDARD_REQUIRED ON)
add_compile_options(-Wall -g -O3)
set(THREADS_PREFER_PTHREAD_FLAG ON)
find_package(Threads REQUIRED)
find_library(ZSTD_LIBRARY zstd)
# ... error if not found
# Source file collection
file(GLOB SHARED_SRCS "src/shared/*.c")
file(GLOB SERVER_SRCS "src/server/*.c")
file(GLOB CLIENT_SRCS "src/client/*.c")
file(GLOB TEST_SRCS "tests/*.c")
# Targets
add_executable(server ${SERVER_SRCS} ${SHARED_SRCS})
target_include_directories(server PRIVATE src/shared src/server src/client)
target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY})
add_executable(client ${CLIENT_SRCS} ${SHARED_SRCS})
target_include_directories(client PRIVATE src/shared src/server src/client)
target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY})
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} src/client/scanner.c)
target_include_directories(tests PRIVATE tests src/shared src/server src/client)
target_link_libraries(tests PRIVATE Threads::Threads ${ZSTD_LIBRARY})
```
### Source Layout
```
src/shared/ — shared libraries (globbed as SHARED_SRCS)
src/client/ — client sources (globbed as CLIENT_SRCS)
src/server/ — server sources (globbed as SERVER_SRCS)
tests/ — test sources (globbed as TEST_SRCS)
```
### Dependencies
- **zstd** — found via `find_library(ZSTD_LIBRARY zstd)`
- **pthreads** — found via `find_package(Threads REQUIRED)`
- **C11 standard** — required
- **CMake 4.1+** — minimum version
## Conventions
- Use `file(GLOB ...)` for source collection (existing pattern).
- All targets link `Threads::Threads` and `${ZSTD_LIBRARY}`.
- Include directories: `src/shared`, `src/server`, `src/client`, `tests` (for test target).
- Sanitizer support is commented out but present (`-fsanitize=address`).
- Build with `cmake -B build -S . && cmake --build build -j$(nproc)`.
## When Making Changes
1. Preserve existing structure and conventions.
2. Use `file(GLOB)` for new source directories (match existing pattern).
3. Add new dependencies with `find_package` or `find_library`.
4. When adding a new executable target, follow the pattern of existing targets.
5. When adding a new library (static/shared), use `add_library` and follow the project's naming.
6. For sanitizer builds, use the commented-out `-fsanitize=address` lines as reference.
7. Always verify the build compiles after changes.
## Build Commands
```bash
cmake -B build -S .
cmake --build build -j$(nproc)
./build/server
./build/client
./build/tests
```
+91
View File
@@ -0,0 +1,91 @@
---
description: Generates and maintains API documentation, protocol specs, and usage examples from the FastSync C source code.
mode: subagent
---
You are a documentation generator for the FastSync project — a high-performance file synchronization system written in C11.
## Your Role
Generate accurate documentation from the actual source code. Maintain API references, protocol specifications, and usage examples.
## Project Structure
### Source Layout
```
src/shared/ — shared libraries (protocol, compression, queue, config, data, metadata, transport, etc.)
src/client/ — client CLI, file sending, scanner
src/server/ — TCP server
tests/ — unit tests
```
### Key Headers to Document
| Header | Purpose |
|--------|---------|
| `data.h` | Generic buffer type (`Data`) |
| `queue.h` | Thread-safe bounded queue |
| `chunk.h` | File chunking for batch transfer |
| `compression.h` | zstd streaming compression |
| `config.h` | Runtime configuration |
| `protocol.h` | Wire protocol (status codes, send/receive) |
| `metadata.h` | File metadata (mode, uid, gid, mtime) |
| `transport_tcp.h` | TCP client/server |
| `transport_ssh.h` | SSH transport with ControlMaster |
| `scanner.h` | Directory traversal and file scanning |
| `file.h` | File representation |
| `array_list.h` | Dynamic array |
| `log.h` | Logging utilities |
| `utils.h` | Shared utilities |
### README
The project README at `README.md` contains:
- Technical overview
- System architecture
- Protocol details
- Command-line arguments
- Environment variables
- Build instructions
- Benchmark results
## Documentation Types
### 1. API Reference (from headers)
For each public function:
- Signature (from the header)
- Brief description
- Parameters and return value
- Memory ownership rules
- Thread safety guarantees
### 2. Protocol Specification
- Wire format byte layouts
- Status code semantics
- Transfer flow diagrams
- Metadata encoding
### 3. Architecture Docs
- Data flow diagrams
- Component interactions
- Threading model
### 4. Usage Examples
- Command-line examples for common use cases
- Build instructions
- Integration scenarios
## Conventions
- Use `file:line` references when pointing to source locations
- Document actual behavior, not intended behavior
- Include error conditions and edge cases
- Keep docs close to the code they describe
- Use markdown formatting suitable for terminal rendering
## When Generating Documentation
1. Read the actual source files first — don't assume behavior
2. Cross-reference headers with implementations
3. Verify examples actually compile and work
4. Update README when adding/changing features
5. Keep protocol docs in sync with code changes
+74
View File
@@ -0,0 +1,74 @@
---
description: Analyzes performance bottlenecks in the FastSync transfer pipeline and suggests concrete optimizations for chunking, compression, threading, and network transport.
mode: subagent
---
You are a performance analyst for the FastSync project — a high-performance file synchronization system written in C11.
## Your Role
Analyze the transfer pipeline for performance bottlenecks and suggest concrete, actionable optimizations. You understand the full data flow from scanner to network.
## Architecture Overview
### Transfer Pipeline
```
DirectoryScanner → Queue(Scanner→Loader) → ChunkBuilder → Queue(Loader→Sender) → Network Send
```
1. **Scanner** — BFS traversal, builds file list, groups into chunks
2. **Loader** — reads file contents into memory
3. **Sender** — compresses + serializes + sends over TCP/SSH
### Key Components
| Component | File | Purpose |
|-----------|------|---------|
| Scanner | `src/client/scanner.c` | BFS directory traversal, exclude patterns, chunk building |
| Chunk | `src/shared/chunk.c` | File grouping (~10MB default), serialization |
| Compression | `src/shared/compression.c` | Streaming zstd (levels 122) |
| Queue | `src/shared/queue.c` | Thread-safe bounded queue with condition variables |
| Transport TCP | `src/shared/transport_tcp.c` | TCP with `sendfile()` zero-copy |
| Transport SSH | `src/shared/transport_ssh.c` | SSH with ControlMaster, socketpair |
| Protocol | `src/shared/protocol.c` | Status codes, data send/receive |
| Config | `src/shared/config.c` | Runtime parameters |
### Performance-Critical Paths
1. **Chunk size** (`DEFAULT_CHUNK_SIZE = 10MB`) — balances memory vs. transfer efficiency
2. **Compression level** (122) — trades CPU for bandwidth
3. **`sendfile()` zero-copy** — bypasses userspace, ~2× faster on loopback
4. **Multithreading** — producer-consumer with thread-safe queues
5. **SSH socketpair buffer** — set to 1MB for pipe throughput
6. **Streaming compression**`ZSTD_compressStream2` / `ZSTD_decompressStream`
## Analysis Framework
### When Analyzing, Consider
1. **CPU-bound vs I/O-bound** — Is the bottleneck CPU (compression) or I/O (disk/network)?
2. **Memory allocation** — Are there excessive malloc/free cycles in hot paths?
3. **Lock contention** — Are mutexes held too long? Is the queue the bottleneck?
4. **Syscall overhead** — Are there unnecessary read/write cycles?
5. **Pipeline stalls** — Is any stage starved or blocked?
6. **Data copying** — Are there unnecessary memcpy operations?
7. **Algorithmic** — Is the chunking/scanning algorithm optimal?
### Benchmark Context
From README benchmarks (25MB mixed files, localhost):
- Best config: `-m -c` (multithread + compression) → 0.20s, 11.2× faster than rsync
- `sendfile()` bypasses userspace → ~2× faster on localhost
- Compression reduces wire data enough that transfer becomes latency-bound on WAN
## Output Format
For each bottleneck found:
1. **Location** — file:line
2. **Impact** — high / medium / low
3. **Type** — CPU / IO / memory / lock / algorithmic
4. **Current behavior** — what's happening
5. **Suggested optimization** — concrete code change or approach
6. **Expected impact** — estimated speedup or resource savings
Also provide profiling guidance when asked (e.g., `perf`, `valgrind`, `gprof` commands).
+86
View File
@@ -0,0 +1,86 @@
---
description: Designs and extends the FastSync wire protocol — status codes, metadata format, chunk serialization, config serialization, and ensures backward compatibility.
mode: subagent
---
You are a protocol designer for the FastSync project — a high-performance file synchronization system with a custom binary wire protocol.
## Your Role
Design, extend, and document the wire protocol. Ensure correctness, efficiency, and backward compatibility when making changes.
## Current Protocol
### Status Codes (`src/shared/protocol.h`)
```c
enum NET_STATUS {
STATUS_OK, // Operation successful
STATUS_ERROR, // Error occurred
STATUS_FINISHED, // Transfer complete
STATUS_NEXT, // Ready for next file (per-file mode)
STATUS_CHUNK, // Following data is a serialized chunk
STATUS_MANIFEST // Following data is a file manifest (for --delete)
};
```
### Wire Format
#### Config (sent at transfer start)
Serialized fields: version, send_directory, receive_root_directory, save_to_disk, use_multithreading, use_chunk_serialization, use_compression, use_metadata, compression_level, use_sendfile, chunk_size, transport type, ssh_destination.
#### Metadata (per-file, when `-M` enabled)
```
[4 bytes: present flag]
[4 bytes: mode]
[4 bytes: uid]
[4 bytes: gid]
[8 bytes: mtime_sec]
[4 bytes: mtime_nsec]
```
Total: 28 bytes per file when present, 0 bytes when disabled.
#### Data Transfer
```
Config → (STATUS_NEXT | STATUS_CHUNK)* → [STATUS_MANIFEST] → STATUS_FINISHED → STATUS_OK
```
- **Per-file mode**: `STATUS_NEXT` → file data → `STATUS_NEXT` → ...
- **Chunk mode**: `STATUS_CHUNK` → serialized chunk data → ...
- **Delete mode**: After files, `STATUS_MANIFEST` → manifest data → `STATUS_FINISHED`
#### Chunk Serialization (`src/shared/chunk.c`)
Files grouped into chunks (~10MB default). Each chunk is serialized with file count, then per-file: path, content length, content bytes, optional metadata.
### Data Serialization (`src/shared/data.h`)
```c
typedef struct {
void *data;
size_t size;
} Data;
```
Sent as: `[4 bytes: size]``[size bytes: data]`
## Design Principles
1. **Efficiency** — minimize wire overhead; batch when possible
2. **Backward compatibility** — version field in config for negotiation
3. **Simplicity** — status-code-driven exchange, no complex state machines
4. **Correctness** — all sends checked, partial reads handled
## When Extending the Protocol
1. **Add new status codes** — append to enum, update protocol documentation
2. **Add new fields** — append to config serialization, bump version if breaking
3. **Add new metadata** — extend metadata format with new optional fields
4. **Wire format changes** — document exact byte layout
5. **Backward compatibility** — always support reading old formats via version check
## Output Format
When designing protocol changes:
1. **Motivation** — why the change is needed
2. **Wire format** — exact byte-level layout (hex offsets if complex)
3. **Status code changes** — new/modified codes
4. **Serialization code** — changes to `protocol.c`, `config.c`, `chunk.c`
5. **Compatibility notes** — how old clients/servers handle the change
6. **Testing strategy** — how to verify the protocol change works
+124
View File
@@ -0,0 +1,124 @@
---
description: Writes unit tests for the FastSync C codebase using the custom test framework. Creates test_*.c, test_*.h, and registers tests in runner.c.
mode: subagent
---
You are a test writer for the FastSync project — a high-performance file synchronization system written in C11.
## Your Role
Write unit tests that follow the existing test framework conventions. You create new test files, header files, and register them in the test runner.
## Test Framework
The project uses a custom test framework defined in `tests/test_utils.h`.
### Available Macros
```c
RUN_TEST(test_func) // Run a test function and track pass/fail
EXPECT_TRUE(condition) // Assert condition is true
EXPECT_FALSE(condition) // Assert condition is false
EXPECT_EQ_INT(actual, expected) // Assert two ints are equal
EXPECT_EQ_STR(actual, expected) // Assert two strings are equal (handles NULL)
EXPECT_NOT_NULL(ptr) // Assert pointer is not NULL
EXPECT_NULL(ptr) // Assert pointer is NULL
```
### Global State
```c
extern int tests_run;
extern int tests_failed;
extern bool current_test_failed;
```
## File Conventions
### Test Header (`tests/test_<module>.h`)
```c
#ifndef TEST_<MODULE>_H
#define TEST_<MODULE>_H
void test_<module>();
#endif
```
### Test Source (`tests/test_<module>.c`)
```c
#include "test_<module>.h"
#include "<module>.h" // The header being tested
#include "test_utils.h"
#include <stdlib.h>
#include <stdio.h>
static void test_<module>_<specific_case>() {
// Arrange
// Act
// Assert using EXPECT_* macros
// IMPORTANT: return immediately on failure (macros do this)
}
void test_<module>() {
test_<module>_<case1>();
test_<module>_<case2>();
// ...
}
```
### Registration in `tests/runner.c`
Add the `#include` and `RUN_TEST()` call:
```c
#include "test_<module>.h"
// ...
RUN_TEST(test_<module>);
```
## Patterns to Follow
### Memory Management in Tests
- `malloc` test data, `free` after assertions.
- Use destroy functions (`data_destroy`, `queue_destroy`, etc.) for framework objects.
- Don't leak — every allocation must be freed.
### Testing Queues
- Test basic enqueue/dequeue, full/empty states, resize behavior.
- Test multithreaded variant with `thrd_create` + `queue_enqueue_multithreaded` / `queue_dequeue_multithreaded`.
- Use `mtx_t` and `cnd_t` for thread synchronization in tests.
### Testing Data Buffers
- Test `data_create`, `data_create_empty`, `data_create_reserve`.
- Verify size and content after creation.
### Testing Compression
- Compress data, decompress, verify round-trip.
- Test with various compression levels.
### Testing Config
- Test `config_create` and `config_delete`.
- Test serialization round-trip (`config_send` + `config_receive`).
### Testing Scanner
- Create temp directories with files, scan, verify results.
- Test exclude pattern matching.
### Edge Cases to Always Cover
- NULL inputs
- Empty collections (size 0)
- Single element
- At capacity boundaries
- Invalid parameters
## Build & Run
```bash
cmake -B build -S . && cmake --build build -j$(nproc) && ./build/tests
```
## Output
When asked to write tests, produce:
1. The test header file content
2. The test source file content
3. The runner.c modification needed
4. Verify with a build and test run
+107
View File
@@ -0,0 +1,107 @@
---
name: pr-build
description: Builds and tests a pull request branch, fixing compilation errors and test failures. Use when the user says "build PR", "fix PR build", "run PR build", or wants to compile and test a PR branch.
---
# PR Build Skill
Builds, tests, and fixes a pull request branch. This skill CAN edit files, commit, and push.
## Workflow
### Step 1: Identify the PR branch
If the user specifies a PR number, check it out:
```bash
tea pr checkout <number>
```
If already on a PR branch, verify with:
```bash
git branch --show-current
git log main..HEAD --oneline
```
### Step 2: Clean build
```bash
rm -rf build
cmake -B build -S . 2>&1
cmake --build build -j$(nproc) 2>&1
```
Capture both stdout and stderr.
### Step 3: Handle build failures
If the build fails, read the error output carefully. Common issues:
**Missing include / undefined reference:**
- Check if the new `.c` file is in the right `file(GLOB ...)` directory
- Check if the new `.h` file is included properly
- Check if CMakeLists.txt needs updating (new target, new source file, new dependency)
**Type errors / implicit declarations:**
- Check function signatures match between `.h` and `.c`
- Check struct field names and types
**Linker errors:**
- Check if all required libraries are linked in CMakeLists.txt
- Check if all source files are included in the target
Use the cmake-expert agent to diagnose and fix CMake issues.
### Step 4: Run unit tests
If build succeeds:
```bash
./build/tests
```
### Step 5: Handle test failures
If tests fail:
- Read the test output carefully
- Check which test function failed and the assertion line
- Read the test source file and the module being tested
- Use the test-writer agent to investigate and fix
### Step 6: Run integration tests (optional)
```bash
python3 test.py
```
This runs the integration + benchmark suite. It takes longer — only run if the user asks or if unit tests pass.
### Step 7: Fix and commit
If fixes were needed:
```bash
git add -A
git commit -m "Fix build: <brief description of what was fixed>"
git push
```
### Step 8: Report results
Print a summary:
```
=== PR BUILD SUMMARY ===
Branch: <branch-name>
Build: [PASS/FAIL]
Unit tests: [PASS/FAIL] (<passed>/<total>)
Integration tests: [PASS/FAIL/SKIPPED]
Fixes applied: <count>
<list of fixes with commit hashes>
```
## Rules
- DO edit source files and CMakeLists.txt to fix issues
- DO commit and push fixes
- Always build from clean state (rm -rf build)
- Read error messages carefully before fixing
- Don't change functionality — only fix build/test issues
- Preserve existing code style when making fixes
+119
View File
@@ -0,0 +1,119 @@
---
name: pr-review
description: Reviews a pull request for bugs, memory safety, thread safety, and style issues. Use when the user says "review PR", "review this PR", "review pull request", or wants a code review of changes.
---
# PR Review Skill
Read-only code review of a pull request branch. Produces a report — does NOT edit files.
## Workflow
### Step 1: Identify the PR branch
If the user specifies a PR number, check it out:
```bash
tea pr checkout <number>
```
If already on a PR branch, verify with:
```bash
git branch --show-current
git log main..HEAD --oneline
```
### Step 2: Get changed files
```bash
git diff main --name-only -- '*.c' '*.h'
```
This gives the list of C source and header files changed in the PR.
### Step 3: Read all changed files
Use the Read tool to read every changed `.c` and `.h` file. Read full files — don't skip any.
### Step 4: Review each file
For each changed file, review for:
**Memory Safety**
- Every `malloc`/`calloc` has a matching `free` on all code paths (including error paths)
- No use-after-free (pointers used after `*_destroy()` is called)
- No double-free
- Null checks after allocation before use
- Correct buffer sizes (strlen + 1 for null terminators)
- `Data` objects created/destroyed properly
**Thread Safety**
- Shared state accessed under mutex
- No race conditions on queue operations
- Condition variable signals under lock
- No deadlock potential (consistent lock ordering)
- `done` flags checked properly in consumer loops
**Protocol Safety**
- `send_n_data` / `receive_n_data` return values checked
- Status codes validated before use
- Config serialization handles partial reads
**Logic Errors**
- Off-by-one in loops/buffers
- Incorrect size calculations
- Wrong enum values or comparisons
- Missing break statements in switch
**Error Handling**
- Resources freed on error paths (no leaks)
- Functions return appropriate error values
- Error messages are useful
### Step 5: Categorize findings
For each issue:
1. **File:line** — exact location
2. **Severity** — critical / warning / style
3. **Category** — memory / thread / protocol / logic / error
4. **Description** — what's wrong and how to fix it
### Step 6: Output report
Print a formatted summary:
```
=== PR REVIEW SUMMARY ===
Branch: <branch-name>
Files reviewed: <count>
Issues found: <count>
CRITICAL: <count>
WARNING: <count>
STYLE: <count>
=== ISSUES ===
[1] src/shared/compression.c:42 — CRITICAL (memory)
Potential leak: data returned from data_compress() not freed on error path
Fix: Add data_destroy(compressed) before return false
...
=== VERDICT ===
[PASS] No critical issues found
— or —
[FAIL] <N> critical issues must be fixed before merge
```
### Step 7: Optional PR comment
If the user wants to post the review as a PR comment:
```bash
tea pr comment <number> --comment "<review report>"
```
## Rules
- Do NOT edit any source files
- Do NOT run builds or tests
- Do NOT commit or push
- Report ALL issues — don't filter or minimize
- Be specific about line numbers and fix suggestions
+1 -1
View File
@@ -1,4 +1,4 @@
cmake_minimum_required(VERSION 4.1)
cmake_minimum_required(VERSION 3.22)
project(FastFileTransfer)
+6
View File
@@ -0,0 +1,6 @@
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc g++ make libc6-dev cmake libzstd-dev libssl-dev git ca-certificates curl && \
curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
apt-get install -y --no-install-recommends nodejs && \
rm -rf /var/lib/apt/lists/*
+1
View File
@@ -8,6 +8,7 @@ pkgs.mkShell {
cmake
gnumake
pkg-config
docker
tea
];
+31
View File
@@ -1,8 +1,11 @@
#include "client_send.h"
#include "config.h"
#include "log.h"
#include "protocol.h"
#include "transport_tls.h"
#include "utils.h"
#include <errno.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -32,6 +35,7 @@ static void print_usage(void) {
printf(" --include <pattern> Only include files matching pattern\n");
printf(" --max-size <n> Skip files larger than n bytes\n");
printf(" --min-size <n> Skip files smaller than n bytes\n");
printf(" --incremental Skip files unchanged since last transfer\n");
printf(" -m Enable multithreading\n");
printf(" -s Enable chunk serialization\n");
printf(" -f Enable sendfile (TCP only, not with -c or -s)\n");
@@ -43,6 +47,7 @@ static void print_usage(void) {
printf(" --save-to-disk Write received files to disk\n");
printf(" --server-host <ip> Server IP address (default: 127.0.0.1)\n");
printf(" --server-port <n> Server port (default: 8080)\n");
printf(" --bwlimit <KB/s> Bandwidth limit in kilobytes per second\n");
printf(" --tls Enable TLS encryption\n");
printf(" --cert <path> TLS certificate file (PEM)\n");
printf(" --key <path> TLS private key file (PEM)\n");
@@ -94,6 +99,8 @@ int main(int argc, char *argv[]) {
config->max_size = strtoull(argv[++i], NULL, 10);
} else if (strcmp(argv[i], "--min-size") == 0 && i + 1 < argc) {
config->min_size = strtoull(argv[++i], NULL, 10);
} else if (strcmp(argv[i], "--incremental") == 0) {
config->use_incremental = true;
} else if (strcmp(argv[i], "-c") == 0 || strcmp(argv[i], "-z") == 0) {
config->use_compression = true;
log_message(LOG_LEVEL_INFO, "Enabled Compression");
@@ -132,6 +139,20 @@ int main(int argc, char *argv[]) {
server_host = str_dup(argv[++i]);
} else if (strcmp(argv[i], "--server-port") == 0 && i + 1 < argc) {
server_port = atoi(argv[++i]);
} else if (strcmp(argv[i], "--bwlimit") == 0 && i + 1 < argc) {
char *end;
errno = 0;
unsigned long long kbps = strtoull(argv[++i], &end, 10);
if (errno != 0 || *end != '\0' || kbps == 0) {
fprintf(stderr, "Error: --bwlimit must be a positive integer\n");
return 1;
}
if (kbps > ULLONG_MAX / 1024) {
fprintf(stderr, "Error: --bwlimit value too large\n");
return 1;
}
io_set_bwlimit(kbps * 1024);
log_message(LOG_LEVEL_INFO, "Set bandwidth limit to %llu KB/s", kbps);
} else if (strcmp(argv[i], "--progress") == 0) {
config->show_progress = true;
} else if (strcmp(argv[i], "--chunk-size") == 0 && i + 1 < argc) {
@@ -200,6 +221,16 @@ int main(int argc, char *argv[]) {
return 1;
}
if (config->use_incremental && config->use_chunk_serialization) {
fprintf(stderr, "Error: --incremental is not supported with -s (chunk serialization)\n");
return 1;
}
if (config->use_incremental && !config->use_metadata) {
log_message(LOG_LEVEL_INFO, "Enabling metadata preservation for --incremental");
config->use_metadata = true;
}
if (config->use_tls) {
if (!config->tls_cert || !config->tls_key) {
fprintf(stderr, "Error: --tls requires --cert and --key\n");
+49 -8
View File
@@ -19,6 +19,27 @@
#include <threads.h>
#include <time.h>
static int incremental_check(Client *client, File *file) {
if (!send_status(client->file_descriptor, STATUS_CHECK)) return -1;
if (!send_str(client->file_descriptor, file->path)) return -1;
unsigned long long fsize = file->data->size;
long long mtime = file->metadata ? file->metadata->mtime_sec : 0;
if (!send_n_data(client->file_descriptor, &fsize, sizeof(fsize))) return -1;
if (!send_n_data(client->file_descriptor, &mtime, sizeof(mtime))) return -1;
Status s;
if (!receive_status(client->file_descriptor, &s)) return -1;
if (s == STATUS_ERROR) {
log_message(LOG_LEVEL_ERROR, "Server reported error for file");
return -1;
}
if (s == STATUS_OK) return 1;
if (s != STATUS_NEXT) {
log_message(LOG_LEVEL_ERROR, "Unexpected server status");
return -1;
}
return 0;
}
int send_chunk(Client *client, Chunk *chunk, Config *config) {
if (config->use_chunk_serialization) {
if (!send_status(client->file_descriptor, STATUS_CHUNK)) return -1;
@@ -33,17 +54,37 @@ int send_chunk(Client *client, Chunk *chunk, Config *config) {
data_destroy(data);
} else if (config->use_sendfile && !config->use_compression) {
for (int i = 0; i < chunk->element_count; i++) {
if (!send_status(client->file_descriptor, STATUS_NEXT)) return -1;
if (!file_send_sendfile(chunk->items[i], client->file_descriptor, config->use_metadata))
return -1;
if (config->use_incremental) {
int rc = incremental_check(client, chunk->items[i]);
if (rc < 0) return -1;
if (rc > 0) continue;
if (!file_send_sendfile(chunk->items[i], client->file_descriptor, config->use_metadata, false))
return -1;
} else {
if (!send_status(client->file_descriptor, STATUS_NEXT)) return -1;
if (!file_send_sendfile(chunk->items[i], client->file_descriptor, config->use_metadata, true))
return -1;
}
}
} else {
for (int i = 0; i < chunk->element_count; i++) {
if (!send_status(client->file_descriptor, STATUS_NEXT)) return -1;
if (!file_send_single_calls(chunk->items[i], client->file_descriptor,
config->use_metadata,
config->use_compression ? config->compression_level : 0))
return -1;
if (config->use_incremental) {
int rc = incremental_check(client, chunk->items[i]);
if (rc < 0) return -1;
if (rc > 0) continue;
if (!file_send_single_calls(chunk->items[i], client->file_descriptor,
config->use_metadata,
config->use_compression ? config->compression_level : 0,
false))
return -1;
} else {
if (!send_status(client->file_descriptor, STATUS_NEXT)) return -1;
if (!file_send_single_calls(chunk->items[i], client->file_descriptor,
config->use_metadata,
config->use_compression ? config->compression_level : 0,
true))
return -1;
}
}
}
return 0;
+30 -64
View File
@@ -1,11 +1,9 @@
#include "array_list.h"
#include "chunk.h"
#include "compression.h"
#include "config.h"
#include "data.h"
#include "file.h"
#include "log.h"
#include "metadata.h"
#include "multiprocessing.h"
#include "protocol.h"
#include "queue.h"
@@ -18,89 +16,57 @@
#include <stdlib.h>
#include <string.h>
int receive_files(Config *config, int file_descriptor) {
int receive_files(Config *config, int fd) {
Status status;
if (!receive_status(file_descriptor, &status)) return -1;
while (status == STATUS_NEXT || status == STATUS_CHUNK) {
if (status == STATUS_CHUNK) {
Data *chunk_data = receive_data(file_descriptor);
if (chunk_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to receive chunk data");
send_status(file_descriptor, STATUS_ERROR);
return -1;
}
Data *data_to_process = chunk_data;
if (config->use_compression) {
data_to_process = data_decompress(chunk_data);
data_destroy(chunk_data);
if (data_to_process == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk");
send_status(file_descriptor, STATUS_ERROR);
return -1;
}
}
Chunk *chunk = chunk_deserialize(data_to_process, config->use_metadata);
data_destroy(data_to_process);
if (chunk == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to deserialize chunk, skipping");
send_status(file_descriptor, STATUS_ERROR);
return -1;
}
if (!receive_status(fd, &status)) return -1;
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK) {
if (status == STATUS_CHECK) {
bool skipped;
File *file = receive_incremental_check(fd, config, &skipped);
if (skipped) goto next;
if (file == NULL && !skipped) return -1;
if (config->save_to_disk)
file_save_to_disk(config->receive_root_directory, file);
file_destroy(file);
} else if (status == STATUS_CHUNK) {
Chunk *chunk = receive_chunk_data(fd, config);
if (chunk == NULL) {
send_status(fd, STATUS_ERROR);
return -1;
}
for (int i = 0; i < chunk->element_count; i++) {
if (config->save_to_disk) {
char *disk_path = path_cat(config->receive_root_directory, chunk->items[i]->path);
if (disk_path) {
to_disk(disk_path, chunk->items[i]->data->data, chunk->items[i]->data->size);
file_restore_metadata(disk_path, chunk->items[i]->metadata);
free(disk_path);
}
}
if (config->save_to_disk)
file_save_to_disk(config->receive_root_directory, chunk->items[i]);
}
chunk_destroy(chunk);
} else {
File *file = file_receive(config, file_descriptor);
File *file = file_receive(config, fd);
if (file == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to receive file");
send_status(file_descriptor, STATUS_ERROR);
send_status(fd, STATUS_ERROR);
return -1;
}
if (config->save_to_disk) {
char *disk_path = path_cat(config->receive_root_directory, file->path);
if (disk_path) {
to_disk(disk_path, file->data->data, file->data->size);
file_restore_metadata(disk_path, file->metadata);
free(disk_path);
}
}
if (config->save_to_disk)
file_save_to_disk(config->receive_root_directory, file);
file_destroy(file);
}
if (!receive_status(file_descriptor, &status)) {
send_status(file_descriptor, STATUS_ERROR);
next:
if (!receive_status(fd, &status)) {
send_status(fd, STATUS_ERROR);
return -1;
}
}
if (status == STATUS_MANIFEST) {
int count;
if (!receive_int(file_descriptor, &count)) return -1;
ArrayList *manifest = array_list_create(free);
if (manifest) {
for (int i = 0; i < count; i++) {
char *s = receive_str(file_descriptor);
if (s) array_list_add(manifest, s);
}
fprintf(stderr, "Deleting files not in manifest...\n");
delete_extras(config->receive_root_directory, manifest);
array_list_delete(manifest);
}
if (!receive_status(file_descriptor, &status)) return -1;
if (receive_manifest(fd, config, &status) != 0) return -1;
}
if (status != STATUS_FINISHED) {
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
send_status(file_descriptor, STATUS_ERROR);
send_status(fd, STATUS_ERROR);
return -1;
}
send_status(file_descriptor, STATUS_OK);
send_status(fd, STATUS_OK);
return 0;
}
+23
View File
@@ -10,6 +10,7 @@
#include "file.h"
#include "log.h"
#include "metadata.h"
#include "protocol.h"
Chunk *chunk_create(File **items, int element_count) {
Chunk *chunk = (Chunk *)malloc(sizeof(Chunk));
@@ -178,5 +179,27 @@ Data *chunk_compress(Chunk *chunk, int compression_level, bool use_metadata) {
return compressed;
}
Chunk *receive_chunk_data(int fd, Config *config) {
Data *chunk_data = receive_data(fd);
if (chunk_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to receive chunk data");
return NULL;
}
Data *data_to_process = chunk_data;
if (config->use_compression) {
data_to_process = data_decompress(chunk_data);
data_destroy(chunk_data);
if (data_to_process == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk");
return NULL;
}
}
Chunk *chunk = chunk_deserialize(data_to_process, config->use_metadata);
data_destroy(data_to_process);
if (chunk == NULL)
log_message(LOG_LEVEL_ERROR, "Failed to deserialize chunk, skipping");
return chunk;
}
+2
View File
@@ -1,6 +1,7 @@
#ifndef CHUNK_H
#define CHUNK_H
#include "config.h"
#include "data.h"
#include "file.h"
#include <stdbool.h>
@@ -18,5 +19,6 @@ void chunk_destroy(void *chunk);
Data *chunk_serialize(Chunk *chunk, bool use_metadata);
Chunk *chunk_deserialize(Data *data, bool use_metadata);
Data *chunk_compress(Chunk *chunk, int compression_level, bool use_metadata);
Chunk *receive_chunk_data(int fd, Config *config);
#endif
+4
View File
@@ -38,6 +38,7 @@ Config *config_create(char *version, char *send_directory,
config->include_count = 0;
config->max_size = 0;
config->min_size = 0;
config->use_incremental = false;
config->use_tls = false;
config->tls_cert = NULL;
config->tls_key = NULL;
@@ -96,6 +97,7 @@ bool config_send(int file_descriptor, Config *config) {
if (!send_int(file_descriptor, (int)config->chunk_size)) return false;
if (!send_int(file_descriptor, config->use_sendfile)) return false;
if (!send_int(file_descriptor, config->use_delete)) return false;
if (!send_int(file_descriptor, config->use_incremental)) return false;
Status status;
if (!receive_status(file_descriptor, &status)) return false;
if (status != STATUS_OK) {
@@ -141,6 +143,8 @@ Config *config_receive(int file_descriptor) {
config->use_sendfile = tmp;
if (!receive_int(file_descriptor, &tmp)) goto error;
config->use_delete = tmp;
if (!receive_int(file_descriptor, &tmp)) goto error;
config->use_incremental = tmp;
config->show_progress = false;
config->dry_run = false;
config->ssh_port = 22;
+2 -1
View File
@@ -32,13 +32,14 @@ typedef struct Config {
int include_count;
unsigned long long max_size;
unsigned long long min_size;
bool use_incremental;
bool use_tls;
char *tls_cert;
char *tls_key;
char *tls_ca;
} Config;
#define PROTOCOL_VERSION "1.0.0"
#define PROTOCOL_VERSION "1.1.0"
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
Config *config_create(char *version, char *send_directory,
+110 -14
View File
@@ -96,26 +96,103 @@ bool file_load_data(File *file) {
return true;
}
bool file_send_single_calls(File *file, int file_descriptor, bool use_metadata, int compression_level) {
bool file_send_single_calls(File *file, int file_descriptor, bool use_metadata, int compression_level, bool send_path) {
Data *data_to_send = file->data;
Data *compressed_data = NULL;
if (compression_level > 0) {
Data *compressed_data = data_compress(file->data, compression_level);
compressed_data = data_compress(file->data, compression_level);
if (compressed_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to compress file data");
return false;
}
data_destroy(file->data);
if (compressed_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Compression failed in file_send_single_calls");
exit(EXIT_FAILURE);
}
file->data = compressed_data;
data_to_send = compressed_data;
}
if (!send_str(file_descriptor, file->path)) return false;
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) return false;
if (!send_data(file_descriptor, file->data)) return false;
if (send_path && !send_str(file_descriptor, file->path)) {
data_destroy(compressed_data);
return false;
}
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) {
data_destroy(compressed_data);
return false;
}
if (!send_data(file_descriptor, data_to_send)) {
data_destroy(compressed_data);
return false;
}
data_destroy(compressed_data);
return true;
}
bool file_save_to_disk(const char *root_directory, File *file) {
char *disk_path = path_cat((char *)root_directory, file->path);
if (disk_path == NULL) return false;
bool ok = to_disk(disk_path, file->data->data, file->data->size);
if (ok) file_restore_metadata(disk_path, file->metadata);
free(disk_path);
return ok;
}
File *receive_incremental_check(int fd, Config *config, bool *skipped) { *skipped = false;
char *check_path = receive_str(fd);
if (check_path == NULL) { send_status(fd, STATUS_ERROR); return NULL; }
unsigned long long check_size;
long long check_mtime;
if (!receive_n_data(fd, &check_size, sizeof(check_size)) ||
!receive_n_data(fd, &check_mtime, sizeof(check_mtime))) {
free(check_path);
send_status(fd, STATUS_ERROR);
return NULL;
}
char *full_path = path_cat(config->receive_root_directory, check_path);
struct stat st;
bool match = false;
if (full_path && stat(full_path, &st) == 0 &&
(unsigned long long)st.st_size == check_size &&
(long long)st.st_mtime == check_mtime) {
match = true;
}
free(full_path);
if (match) {
if (!send_status(fd, STATUS_OK)) { free(check_path); return NULL; }
free(check_path);
*skipped = true;
return NULL;
}
if (!send_status(fd, STATUS_NEXT)) { free(check_path); return NULL; }
File *file = file_create(check_path);
free(check_path);
if (file == NULL) { send_status(fd, STATUS_ERROR); return NULL; }
if (config->use_metadata) {
int meta_ok = 1;
file->metadata = metadata_receive(fd, &meta_ok);
if (!meta_ok) { file_destroy(file); send_status(fd, STATUS_ERROR); return NULL; }
}
Data *file_data = receive_data(fd);
if (file_data == NULL) {
file_destroy(file);
send_status(fd, STATUS_ERROR);
return NULL;
}
if (config->use_compression) {
Data *uncompressed = data_decompress(file_data);
data_destroy(file_data);
if (uncompressed == NULL) { file_destroy(file); send_status(fd, STATUS_ERROR); return NULL; }
file_data = uncompressed;
}
data_destroy(file->data);
file->data = file_data;
return file;
}
bool to_disk(const char *path, const void *data, unsigned long long data_size) {
char *directory = str_dup(path);
char *dir_to_free = directory;
@@ -141,8 +218,8 @@ bool to_disk(const char *path, const void *data, unsigned long long data_size) {
return true;
}
bool file_send_sendfile(File *file, int file_descriptor, bool use_metadata) {
if (!send_str(file_descriptor, file->path)) return false;
bool file_send_sendfile(File *file, int file_descriptor, bool use_metadata, bool send_path) {
if (send_path && !send_str(file_descriptor, file->path)) return false;
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) return false;
int fd = open(file->path, O_RDONLY);
@@ -178,7 +255,9 @@ File *file_receive(Config *config, int file_descriptor) {
free(path);
if (file == NULL) return NULL;
if (config->use_metadata) {
file->metadata = metadata_receive(file_descriptor);
int meta_ok = 1;
file->metadata = metadata_receive(file_descriptor, &meta_ok);
if (!meta_ok) { file_destroy(file); return NULL; }
}
Data *file_data = receive_data(file_descriptor);
if (file_data == NULL) {
@@ -216,4 +295,21 @@ size_t file_content_to_buffer(File *file) {
return bytes_read;
}
int receive_manifest(int fd, Config *config, int *next_status) {
int count;
if (!receive_int(fd, &count)) return -1;
ArrayList *manifest = array_list_create(free);
if (manifest) {
for (int i = 0; i < count; i++) {
char *s = receive_str(fd);
if (s) array_list_add(manifest, s);
}
fprintf(stderr, "Deleting files not in manifest...\n");
delete_extras(config->receive_root_directory, manifest);
array_list_delete(manifest);
}
if (!receive_status(fd, next_status)) return -1;
return 0;
}
+5 -2
View File
@@ -24,11 +24,14 @@ File *file_create(const char *path);
void file_destroy(void *item);
bool file_load_data(File *file);
File *file_receive(Config *config, int file_descriptor);
bool file_send_single_calls(File *file, int file_descriptor, bool use_metadata, int compression_level);
bool file_send_sendfile(File *file, int file_descriptor, bool use_metadata);
bool file_send_single_calls(File *file, int file_descriptor, bool use_metadata, int compression_level, bool send_path);
bool file_send_sendfile(File *file, int file_descriptor, bool use_metadata, bool send_path);
size_t file_content_to_buffer(File *file);
FileMetadata *file_metadata_create(struct stat *stats);
void file_metadata_destroy(void *metadata);
bool to_disk(const char *path, const void *data, unsigned long long data_size);
bool file_save_to_disk(const char *root_directory, File *file);
File *receive_incremental_check(int fd, Config *config, bool *skipped);
int receive_manifest(int fd, Config *config, int *next_status);
#endif
+10 -4
View File
@@ -50,22 +50,28 @@ bool metadata_send(int file_descriptor, FileMetadata *m) {
send_n_data(file_descriptor, &m->mtime_nsec, sizeof(long));
}
FileMetadata *metadata_receive(int file_descriptor) {
FileMetadata *metadata_receive(int file_descriptor, int *ok) {
int present;
if (!receive_n_data(file_descriptor, &present, sizeof(int)))
if (!receive_n_data(file_descriptor, &present, sizeof(int))) {
if (ok) *ok = 0;
return NULL;
if (!present)
}
if (!present) {
if (ok) *ok = 1;
return NULL;
}
FileMetadata *m = malloc(sizeof(FileMetadata));
if (m == NULL) return NULL;
if (m == NULL) { if (ok) *ok = 0; return NULL; }
if (!receive_n_data(file_descriptor, &m->mode, sizeof(mode_t)) ||
!receive_n_data(file_descriptor, &m->uid, sizeof(uid_t)) ||
!receive_n_data(file_descriptor, &m->gid, sizeof(gid_t)) ||
!receive_n_data(file_descriptor, &m->mtime_sec, sizeof(time_t)) ||
!receive_n_data(file_descriptor, &m->mtime_nsec, sizeof(long))) {
free(m);
if (ok) *ok = 0;
return NULL;
}
if (ok) *ok = 1;
return m;
}
+1 -1
View File
@@ -10,7 +10,7 @@
void metadata_to_buf(char **buf, FileMetadata *m);
FileMetadata *metadata_from_buf(char **buf);
bool metadata_send(int file_descriptor, FileMetadata *m);
FileMetadata *metadata_receive(int file_descriptor);
FileMetadata *metadata_receive(int file_descriptor, int *ok);
void file_restore_metadata(const char *path, FileMetadata *metadata);
#endif
+16 -48
View File
@@ -1,12 +1,10 @@
#include "multiprocessing.h"
#include "array_list.h"
#include "chunk.h"
#include "compression.h"
#include "config.h"
#include "data.h"
#include "file.h"
#include "log.h"
#include "metadata.h"
#include "protocol.h"
#include "queue.h"
#include "utils.h"
@@ -85,26 +83,8 @@ void pipeline_context_receiver_destroy(PipelineContextReceiver *context) {
static void receive_chunk_enqueue(int file_descriptor,
PipelineContextReceiver *context) {
Data *chunk_data = receive_data(file_descriptor);
if (chunk_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to receive chunk data");
return;
}
Data *data_to_process = chunk_data;
if (context->config->use_compression) {
data_to_process = data_decompress(chunk_data);
data_destroy(chunk_data);
if (data_to_process == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk");
return;
}
}
Chunk *chunk = chunk_deserialize(data_to_process, context->config->use_metadata);
data_destroy(data_to_process);
if (chunk == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to deserialize chunk, skipping");
return;
}
Chunk *chunk = receive_chunk_data(file_descriptor, context->config);
if (chunk == NULL) return;
for (int i = 0; i < chunk->element_count; i++) {
File *file = chunk->items[i];
@@ -126,8 +106,17 @@ int receive_thread(void *pipeline_context) {
Status status;
if (!receive_status(file_descriptor, &status)) return thrd_error;
while (status == STATUS_NEXT || status == STATUS_CHUNK) {
if (status == STATUS_CHUNK) {
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK) {
if (status == STATUS_CHECK) {
bool skipped;
File *file = receive_incremental_check(file_descriptor, config, &skipped);
if (!skipped) {
if (file == NULL) return thrd_error;
queue_enqueue_multithreaded(context->queue, file, &context->mutex,
&context->condition_not_empty,
&context->condition_not_full);
}
} else if (status == STATUS_CHUNK) {
receive_chunk_enqueue(file_descriptor, context);
} else {
File *file = file_receive(config, file_descriptor);
@@ -142,22 +131,7 @@ int receive_thread(void *pipeline_context) {
if (!receive_status(file_descriptor, &status)) return thrd_error;
}
if (status == STATUS_MANIFEST) {
int count;
if (!receive_int(file_descriptor, &count)) return thrd_error;
ArrayList *manifest = array_list_create(free);
if (manifest) {
for (int i = 0; i < count; i++) {
char *s = receive_str(file_descriptor);
if (s) {
array_list_add(manifest, s);
}
}
delete_extras(context->config->receive_root_directory, manifest);
for (int i = 0; i < manifest->size; i++)
free(manifest->items[i]);
array_list_delete(manifest);
}
if (!receive_status(file_descriptor, &status)) return thrd_error;
if (receive_manifest(file_descriptor, config, &status) != 0) return thrd_error;
}
mtx_lock(&context->mutex);
context->receiver_done = true;
@@ -182,14 +156,8 @@ int write_thread(void *pipeline_context) {
free(root_directory);
return thrd_success;
}
if (save_to_disk) {
char *disk_path = path_cat(root_directory, file->path);
if (disk_path) {
to_disk(disk_path, file->data->data, file->data->size);
file_restore_metadata(disk_path, file->metadata);
free(disk_path);
}
}
if (save_to_disk)
file_save_to_disk(root_directory, file);
file_destroy(file);
}
}
+49 -4
View File
@@ -1,20 +1,61 @@
#include "protocol.h"
#include "log.h"
#include <errno.h>
#include <openssl/ssl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
static __thread int io_read_fd = -1;
static __thread int io_write_fd = -1;
static SSL *io_ssl = NULL;
static unsigned long long io_bwlimit = 0;
static long long bw_tokens = 0;
static struct timespec bw_last_refill = {0, 0};
void io_set_fds(int read_fd, int write_fd) {
io_read_fd = read_fd;
io_write_fd = write_fd;
}
void io_set_bwlimit(unsigned long long bytes_per_sec) {
io_bwlimit = bytes_per_sec;
bw_tokens = (long long)io_bwlimit;
clock_gettime(CLOCK_MONOTONIC, &bw_last_refill);
}
static void bw_throttle(size_t bytes_written) {
if (io_bwlimit == 0) return;
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
long long elapsed_ns = (now.tv_sec - bw_last_refill.tv_sec) * 1000000000LL +
(now.tv_nsec - bw_last_refill.tv_nsec);
bw_last_refill = now;
long long tokens_to_add = (long long)((double)io_bwlimit * elapsed_ns / 1000000000.0);
bw_tokens += tokens_to_add;
if (bw_tokens > (long long)io_bwlimit)
bw_tokens = (long long)io_bwlimit;
bw_tokens -= (long long)bytes_written;
if (bw_tokens < 0) {
long long deficit_ns = (long long)((double)(-bw_tokens) / io_bwlimit * 1000000000.0);
struct timespec sleep_time, remaining;
sleep_time.tv_sec = deficit_ns / 1000000000LL;
sleep_time.tv_nsec = deficit_ns % 1000000000LL;
while (nanosleep(&sleep_time, &remaining) < 0 && errno == EINTR)
sleep_time = remaining;
bw_tokens = 0;
clock_gettime(CLOCK_MONOTONIC, &bw_last_refill);
}
}
void io_set_ssl(SSL *ssl) {
io_ssl = ssl;
}
@@ -28,17 +69,19 @@ bool send_n_data(int file_descriptor, void *data, size_t data_size) {
int fd = io_fd(io_write_fd, file_descriptor);
ssize_t total_bytes_send = 0;
while (total_bytes_send < data_size) {
size_t chunk = data_size - total_bytes_send;
if (io_bwlimit > 0 && chunk > 65536)
chunk = 65536;
ssize_t bytes_send;
if (io_ssl)
bytes_send = SSL_write(io_ssl, (char *)data + total_bytes_send,
data_size - total_bytes_send);
bytes_send = SSL_write(io_ssl, (char *)data + total_bytes_send, chunk);
else
bytes_send = write(fd, (char *)data + total_bytes_send,
data_size - total_bytes_send);
bytes_send = write(fd, (char *)data + total_bytes_send, chunk);
if (bytes_send <= 0) {
log_message(LOG_LEVEL_ERROR, "Could not send data");
return false;
}
bw_throttle((size_t)bytes_send);
total_bytes_send += bytes_send;
}
log_message(LOG_LEVEL_DEBUG, " Send n Data: %zu", total_bytes_send);
@@ -82,6 +125,8 @@ static const char *status_to_string(Status status) {
return "NEXT";
case STATUS_CHUNK:
return "CHUNK";
case STATUS_CHECK:
return "CHECK";
default:
return "UNKNOWN";
}
+3 -1
View File
@@ -8,9 +8,11 @@
typedef struct ssl_st SSL;
typedef int Status;
enum NET_STATUS { STATUS_OK, STATUS_ERROR, STATUS_FINISHED, STATUS_NEXT, STATUS_CHUNK, STATUS_MANIFEST };
enum NET_STATUS { STATUS_OK, STATUS_ERROR, STATUS_FINISHED, STATUS_NEXT, STATUS_CHUNK, STATUS_MANIFEST, STATUS_CHECK };
void io_set_fds(int read_fd, int write_fd);
void io_set_bwlimit(unsigned long long bytes_per_sec);
typedef struct ssl_st SSL;
void io_set_ssl(SSL *ssl);
bool send_n_data(int file_descriptor, void *data, size_t data_size);
bool receive_n_data(int file_descriptor, void *data, size_t data_size);
+2
View File
@@ -143,5 +143,7 @@ Client *client_connect_ssh(char *destination, int port) {
client->address.sin_family = AF_UNIX;
client->address_length = 0;
client->ssh_child_pid = pid;
client->ssl = NULL;
client->ssl_ctx = NULL;
return client;
}
+262 -195
View File
@@ -50,7 +50,7 @@ CLIENT_CMD_PREFIX = [
BASE_CLIENT_FLAGS = ["--save-to-disk"]
TEST_CASES = [
TEST_CASES_FULL = [
{"name": "Standard", "flags": []},
{"name": "Posix Args (no flags)", "flags": [], "posix": True},
{"name": "Standard (no metadata)", "flags": [], "use_metadata": False},
@@ -68,7 +68,14 @@ TEST_CASES = [
{"name": "Sendfile + Multithreading (-f -m)", "flags": ["-f", "-m"]},
]
SSH_CASES = [
TEST_CASES_LIGHT = [
{"name": "Standard", "flags": []},
{"name": "Compression (-c)", "flags": ["-c"]},
{"name": "Chunk Serialization (-s)", "flags": ["-s"]},
{"name": "Multithreading + Compression + Chunk Serialization (-m -c -s)", "flags": ["-m", "-c", "-s"]},
]
SSH_CASES_FULL = [
{"name": "SSH (localhost)", "flags": []},
{"name": "SSH Multithreading (-m)", "flags": ["-m"]},
{"name": "SSH Compression (-c)", "flags": ["-c"]},
@@ -79,11 +86,17 @@ SSH_CASES = [
{"name": "SSH Multithreading + Compression + Chunk Serialization (-m -c -s)", "flags": ["-m", "-c", "-s"]},
]
RSYNC_CASES = [
SSH_CASES_LIGHT = [
{"name": "SSH (localhost)", "flags": []},
]
RSYNC_CASES_FULL = [
{"name": "rsync (archive)", "args": ["-aH"]},
{"name": "rsync (archive + compress)", "args": ["-aHz"]},
]
RSYNC_CASES_LIGHT = []
def netem_apply(profile):
params = NETWORK_PROFILES[profile]
@@ -119,12 +132,12 @@ def find_free_port():
return s.getsockname()[1]
def generate_test_files(source_dir):
def generate_test_files(source_dir, full=False):
if os.path.exists(source_dir):
shutil.rmtree(source_dir)
os.makedirs(source_dir)
target_total = 25 * 1024 * 1024
target_total = 25 * 1024 * 1024 if full else 0
written = 0
files = {
@@ -141,14 +154,15 @@ def generate_test_files(source_dir):
f.write(content)
written += len(content)
os.makedirs(os.path.join(source_dir, "bulk"), exist_ok=True)
i = 0
while written < target_total:
chunk_size = min(5 * 1024 * 1024, target_total - written)
with open(os.path.join(source_dir, f"bulk/file_{i}.dat"), "wb") as f:
f.write(random.randbytes(chunk_size))
written += chunk_size
i += 1
if full:
os.makedirs(os.path.join(source_dir, "bulk"), exist_ok=True)
i = 0
while written < target_total:
chunk_size = min(5 * 1024 * 1024, target_total - written)
with open(os.path.join(source_dir, f"bulk/file_{i}.dat"), "wb") as f:
f.write(random.randbytes(chunk_size))
written += chunk_size
i += 1
total_mb = written / (1024 * 1024)
small_bytes = sum(len(c) for c in files.values())
@@ -254,19 +268,24 @@ def print_profile_header(profile_name):
print(" No limits applied")
def run_profile(profile_name, source_dir, dest_dir):
def run_profile(profile_name, source_dir, dest_dir, *, full=False, test_cases=None, ssh_cases=None, rsync_cases=None):
print_profile_header(profile_name)
is_limited = profile_name != "Unlimited"
client_prefix = CLIENT_CMD_PREFIX if is_limited else []
if test_cases is None:
test_cases = TEST_CASES_FULL if full else TEST_CASES_LIGHT
if ssh_cases is None:
ssh_cases = SSH_CASES_FULL if full else SSH_CASES_LIGHT
if rsync_cases is None:
rsync_cases = RSYNC_CASES_FULL if full else RSYNC_CASES_LIGHT
try:
if is_limited:
if is_limited and full:
netem_apply(profile_name)
else:
netem_reset()
results = []
for case in TEST_CASES:
for case in test_cases:
flags = BASE_CLIENT_FLAGS + (["-M"] if case.get("use_metadata", True) else []) + case["flags"]
if case.get("posix"):
cmd = client_prefix + BASE_CLIENT_CMD + [source_dir, dest_dir] + flags
@@ -281,7 +300,7 @@ def run_profile(profile_name, source_dir, dest_dir):
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
if SSH_AVAILABLE:
for case in SSH_CASES:
for case in ssh_cases:
flags = BASE_CLIENT_FLAGS + (["-M"] if case.get("use_metadata", True) else []) + case["flags"]
ssh_dest = f"localhost:{dest_dir}_ssh"
cmd = BASE_CLIENT_CMD + [source_dir, ssh_dest] + flags
@@ -293,182 +312,222 @@ def run_profile(profile_name, source_dir, dest_dir):
except Exception as e:
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
port, conf, daemon = start_rsync_daemon(source_dir)
try:
for case in RSYNC_CASES:
cmd = client_prefix + ["rsync"] + case["args"] + [f"rsync://localhost:{port}/source/", f"{dest_dir}/"]
print(f"\n --- {case['name']} ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, case["name"], source_dir, dest_dir, source_prefix="")
r["suite"] = profile_name
except subprocess.TimeoutExpired:
r = {"name": case["name"], "suite": profile_name, "status": "Timeout", "time": "N/A", "error": "Exceeded 120s"}
except Exception as e:
r = {"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)}
else:
if r["status"] != "Success" and client_prefix:
tmp = tempfile.mkdtemp()
try:
plain = subprocess.run(["rsync", "-aH", f"rsync://localhost:{port}/source/", f"{tmp}/"], capture_output=True, text=True, timeout=30)
if plain.returncode != 0:
errs = [l for l in (plain.stderr or "").split("\n") if l.strip()]
if errs:
r["error"] += f" | raw: {errs[-1][:150]}"
finally:
shutil.rmtree(tmp, ignore_errors=True)
results.append(r)
finally:
wait_proc(daemon)
if rsync_cases:
port, conf, daemon = start_rsync_daemon(source_dir)
try:
os.unlink(conf)
except Exception:
pass
for case in rsync_cases:
cmd = client_prefix + ["rsync"] + case["args"] + [f"rsync://localhost:{port}/source/", f"{dest_dir}/"]
print(f"\n --- {case['name']} ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, case["name"], source_dir, dest_dir, source_prefix="")
r["suite"] = profile_name
except subprocess.TimeoutExpired:
r = {"name": case["name"], "suite": profile_name, "status": "Timeout", "time": "N/A", "error": "Exceeded 120s"}
except Exception as e:
r = {"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)}
else:
if r["status"] != "Success" and client_prefix:
tmp = tempfile.mkdtemp()
try:
plain = subprocess.run(["rsync", "-aH", f"rsync://localhost:{port}/source/", f"{tmp}/"], capture_output=True, text=True, timeout=30)
if plain.returncode != 0:
errs = [l for l in (plain.stderr or "").split("\n") if l.strip()]
if errs:
r["error"] += f" | raw: {errs[-1][:150]}"
finally:
shutil.rmtree(tmp, ignore_errors=True)
results.append(r)
finally:
wait_proc(daemon)
try:
os.unlink(conf)
except Exception:
pass
# Feature-specific tests for rsync-compatible flags
print("\n " + "" * 56 + "\n Feature Tests\n " + "" * 56)
if full:
# Feature-specific tests for rsync-compatible flags
print("\n " + "" * 56 + "\n Feature Tests\n " + "" * 56)
# Dry run (-n) — no server needed
print("\n --- Dry run (-n) ---")
flags = BASE_CLIENT_FLAGS + ["-n"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
print(f" Running: {' '.join(cmd)}")
try:
start = time.monotonic()
result = subprocess.run(cmd, text=True, capture_output=True)
duration = time.monotonic() - start
r = {"name": "Dry run (-n)", "suite": profile_name}
if result.returncode == 0 and "Dry run:" in result.stdout:
r["status"] = "Success"
r["time"] = f"{duration:.4f}s"
r["error"] = ""
else:
r["status"] = "Failed"
r["time"] = "N/A"
r["error"] = f"Exit {result.returncode}: {(result.stderr or result.stdout)[:100]}"
results.append(r)
except Exception as e:
results.append({"name": "Dry run (-n)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Archive mode (-a)
feature_flags = BASE_CLIENT_FLAGS + ["-a"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Archive mode (-a) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Archive mode (-a)", source_dir, dest_dir)
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Archive mode (-a)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Exclude (--exclude small.txt)
feature_flags = BASE_CLIENT_FLAGS + ["--exclude", "small.txt"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Exclude (--exclude small.txt) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Exclude (--exclude small.txt)", source_dir, dest_dir,
expected_missing=["small.txt"])
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Exclude (--exclude small.txt)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Progress (--progress)
feature_flags = BASE_CLIENT_FLAGS + ["--progress"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Progress (--progress) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Progress (--progress)", source_dir, dest_dir)
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Progress (--progress)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Chunk size (--chunk-size 5242880)
feature_flags = BASE_CLIENT_FLAGS + ["--chunk-size", "5242880"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Chunk size (--chunk-size 5242880) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Chunk size (--chunk-size 5242880)", source_dir, dest_dir)
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Chunk size (--chunk-size 5242880)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Delete (--delete) — pre-populate dest, add extra files, then sync with --delete
# Note: server handles one client per launch, so we restart between syncs
print(f"\n --- Delete (--delete) ---")
try:
flags = BASE_CLIENT_FLAGS + ["-M"]
# First sync (no delete) to populate dest
s1 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
time.sleep(0.5)
first_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
r1 = subprocess.run(first_cmd, text=True, capture_output=True)
wait_proc(s1)
if r1.returncode != 0:
raise RuntimeError(f"First sync failed: {r1.stderr[:100]}")
# Add extra files to received dir
received = os.path.join(dest_dir, os.path.abspath(source_dir).lstrip(os.sep))
extra_path = os.path.join(received, "extra_file.txt")
with open(extra_path, "w") as f:
f.write("should be deleted")
extra_dir = os.path.join(received, "extra_dir")
os.makedirs(extra_dir, exist_ok=True)
with open(os.path.join(extra_dir, "nested.txt"), "w") as f:
f.write("nested extra")
# Second sync with --delete (fresh server)
s2 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
time.sleep(0.5)
second_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags + ["--delete"]
start = time.monotonic()
r2 = subprocess.run(second_cmd, text=True, capture_output=True)
duration = time.monotonic() - start
wait_proc(s2)
r = {"name": "Delete (--delete)", "suite": profile_name}
if r2.returncode == 0 and not os.path.exists(extra_path) and not os.path.exists(extra_dir):
mismatches, missing = verify_transfer(source_dir, received)
if not mismatches and not missing:
# Dry run (-n) — no server needed
print("\n --- Dry run (-n) ---")
flags = BASE_CLIENT_FLAGS + ["-n"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
print(f" Running: {' '.join(cmd)}")
try:
start = time.monotonic()
result = subprocess.run(cmd, text=True, capture_output=True)
duration = time.monotonic() - start
r = {"name": "Dry run (-n)", "suite": profile_name}
if result.returncode == 0 and "Dry run:" in result.stdout:
r["status"] = "Success"
r["time"] = f"{duration:.4f}s"
r["error"] = ""
else:
r["status"] = "Failed"
r["time"] = "N/A"
r["error"] = f"post-delete verify: mismatches={len(mismatches)}, missing={len(missing)}"
else:
r["status"] = "Failed"
r["time"] = "N/A"
errs = []
if r2.returncode != 0:
errs.append(f"Exit {r2.returncode}: {(r2.stderr or r2.stdout)[:60]}")
if os.path.exists(extra_path):
errs.append("extra_file.txt remains")
if os.path.exists(extra_dir):
errs.append("extra_dir remains")
r["error"] = " | ".join(errs)
results.append(r)
except Exception as e:
results.append({"name": "Delete (--delete)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
r["error"] = f"Exit {result.returncode}: {(result.stderr or result.stdout)[:100]}"
results.append(r)
except Exception as e:
results.append({"name": "Dry run (-n)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# SSH feature tests
if SSH_AVAILABLE:
ssh_dest = f"localhost:{dest_dir}_ssh"
ssh_feature_cases = [
{"name": "SSH Archive (-a)", "flags": ["-a"]},
{"name": "SSH Exclude (--exclude small.txt)", "flags": ["--exclude", "small.txt"], "expected_missing": ["small.txt"]},
]
for case in ssh_feature_cases:
flags = BASE_CLIENT_FLAGS + case["flags"]
cmd = BASE_CLIENT_CMD + [source_dir, ssh_dest] + flags
print(f"\n --- {case['name']} ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, case["name"], source_dir, f"{dest_dir}_ssh",
no_server=True,
expected_missing=case.get("expected_missing"))
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Archive mode (-a)
feature_flags = BASE_CLIENT_FLAGS + ["-a"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Archive mode (-a) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Archive mode (-a)", source_dir, dest_dir)
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Archive mode (-a)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Exclude (--exclude small.txt)
feature_flags = BASE_CLIENT_FLAGS + ["--exclude", "small.txt"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Exclude (--exclude small.txt) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Exclude (--exclude small.txt)", source_dir, dest_dir,
expected_missing=["small.txt"])
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Exclude (--exclude small.txt)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Progress (--progress)
feature_flags = BASE_CLIENT_FLAGS + ["--progress"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Progress (--progress) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Progress (--progress)", source_dir, dest_dir)
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Progress (--progress)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Bandwidth limit (--bwlimit 10240 = 10 MB/s)
feature_flags = BASE_CLIENT_FLAGS + ["--bwlimit", "10240"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Bandwidth limit (--bwlimit 10240 KB/s) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Bandwidth limit (--bwlimit 10240)", source_dir, dest_dir)
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Bandwidth limit (--bwlimit 10240)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Incremental sync (--incremental) — first sync, then second sync should skip all
print(f"\n --- Incremental (--incremental) ---")
try:
flags = BASE_CLIENT_FLAGS + ["-M"]
srv = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
time.sleep(0.5)
first_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
r1 = subprocess.run(first_cmd, text=True, capture_output=True)
wait_proc(srv)
if r1.returncode != 0:
raise RuntimeError(f"First sync failed: {r1.stderr[:100]}")
srv2 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
time.sleep(0.5)
second_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags + ["--incremental"]
start = time.monotonic()
r2 = subprocess.run(second_cmd, text=True, capture_output=True, timeout=30)
duration = time.monotonic() - start
wait_proc(srv2)
r = {"name": "Incremental (--incremental)", "suite": profile_name,
"status": "Success" if r2.returncode == 0 else "Failed",
"time": f"{duration:.4f}s" if r2.returncode == 0 else "N/A",
"error": "" if r2.returncode == 0 else f"Exit {r2.returncode}: {(r2.stderr or r2.stdout)[:60]}"}
results.append(r)
except Exception as e:
results.append({"name": "Incremental (--incremental)", "suite": profile_name,
"status": "Error", "time": "N/A", "error": str(e)})
# Chunk size (--chunk-size 5242880)
feature_flags = BASE_CLIENT_FLAGS + ["--chunk-size", "5242880"]
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
print(f"\n --- Chunk size (--chunk-size 5242880) ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, "Chunk size (--chunk-size 5242880)", source_dir, dest_dir)
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": "Chunk size (--chunk-size 5242880)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# Delete (--delete) — pre-populate dest, add extra files, then sync with --delete
# Note: server handles one client per launch, so we restart between syncs
print(f"\n --- Delete (--delete) ---")
try:
flags = BASE_CLIENT_FLAGS + ["-M"]
# First sync (no delete) to populate dest
s1 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
time.sleep(0.5)
first_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
r1 = subprocess.run(first_cmd, text=True, capture_output=True)
wait_proc(s1)
if r1.returncode != 0:
raise RuntimeError(f"First sync failed: {r1.stderr[:100]}")
# Add extra files to received dir
received = os.path.join(dest_dir, os.path.abspath(source_dir).lstrip(os.sep))
extra_path = os.path.join(received, "extra_file.txt")
with open(extra_path, "w") as f:
f.write("should be deleted")
extra_dir = os.path.join(received, "extra_dir")
os.makedirs(extra_dir, exist_ok=True)
with open(os.path.join(extra_dir, "nested.txt"), "w") as f:
f.write("nested extra")
# Second sync with --delete (fresh server)
s2 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
time.sleep(0.5)
second_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags + ["--delete"]
start = time.monotonic()
r2 = subprocess.run(second_cmd, text=True, capture_output=True)
duration = time.monotonic() - start
wait_proc(s2)
r = {"name": "Delete (--delete)", "suite": profile_name}
if r2.returncode == 0 and not os.path.exists(extra_path) and not os.path.exists(extra_dir):
mismatches, missing = verify_transfer(source_dir, received)
if not mismatches and not missing:
r["status"] = "Success"
r["time"] = f"{duration:.4f}s"
r["error"] = ""
else:
r["status"] = "Failed"
r["time"] = "N/A"
r["error"] = f"post-delete verify: mismatches={len(mismatches)}, missing={len(missing)}"
else:
r["status"] = "Failed"
r["time"] = "N/A"
errs = []
if r2.returncode != 0:
errs.append(f"Exit {r2.returncode}: {(r2.stderr or r2.stdout)[:60]}")
if os.path.exists(extra_path):
errs.append("extra_file.txt remains")
if os.path.exists(extra_dir):
errs.append("extra_dir remains")
r["error"] = " | ".join(errs)
results.append(r)
except Exception as e:
results.append({"name": "Delete (--delete)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
# SSH feature tests
if SSH_AVAILABLE:
ssh_dest = f"localhost:{dest_dir}_ssh"
ssh_feature_cases = [
{"name": "SSH Archive (-a)", "flags": ["-a"]},
{"name": "SSH Exclude (--exclude small.txt)", "flags": ["--exclude", "small.txt"], "expected_missing": ["small.txt"]},
]
for case in ssh_feature_cases:
flags = BASE_CLIENT_FLAGS + case["flags"]
cmd = BASE_CLIENT_CMD + [source_dir, ssh_dest] + flags
print(f"\n --- {case['name']} ---\n Running: {' '.join(cmd)}")
try:
r = run_single_test(cmd, case["name"], source_dir, f"{dest_dir}_ssh",
no_server=True,
expected_missing=case.get("expected_missing"))
r["suite"] = profile_name
results.append(r)
except Exception as e:
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
except (subprocess.CalledProcessError, RuntimeError) as e:
print(f" Error: {e}")
@@ -535,19 +594,26 @@ def check_ssh_localhost():
build_dir = os.path.abspath("build")
server_path = os.path.join(build_dir, "server")
r = subprocess.run(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=5",
"localhost", "which", "fastsync-server"],
capture_output=True, timeout=10)
try:
r = subprocess.run(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=5",
"localhost", "which", "fastsync-server"],
capture_output=True, timeout=10)
except FileNotFoundError:
SSH_AVAILABLE = False
return
if r.returncode == 0:
SSH_AVAILABLE = True
return
SSH_AVAILABLE = False
# Try each PATH dir: create symlink, then verify with which
r = subprocess.run(
["ssh", "-o", "BatchMode=yes", "localhost",
'echo "$PATH"'],
capture_output=True, timeout=10, text=True)
try:
r = subprocess.run(
["ssh", "-o", "BatchMode=yes", "localhost",
'echo "$PATH"'],
capture_output=True, timeout=10, text=True)
except FileNotFoundError:
return
if r.returncode != 0:
return
for d in r.stdout.strip().split(":"):
@@ -628,9 +694,10 @@ def main():
parser.add_argument("--keep-data", action="store_true")
parser.add_argument("--unlimited", action="store_true")
parser.add_argument("--wan", action="store_true")
parser.add_argument("--full", action="store_true", help="Run full test suite with network shaping, SSH, rsync benchmarks")
args = parser.parse_args()
total_bytes = generate_test_files(args.source_dir)
total_bytes = generate_test_files(args.source_dir, full=args.full)
if os.path.exists(args.dest_dir):
shutil.rmtree(args.dest_dir)
os.makedirs(args.dest_dir, exist_ok=True)
@@ -641,12 +708,12 @@ def main():
elif args.wan:
profiles.append("WAN")
else:
profiles.append("LAN")
profiles.append("LAN" if args.full else "Unlimited")
try:
all_results = []
for p in profiles:
all_results.extend(run_profile(p, args.source_dir, args.dest_dir))
all_results.extend(run_profile(p, args.source_dir, args.dest_dir, full=args.full))
print("\n" + "=" * 130)
print(f"{'RESULTS':^130}")