Compare commits
38 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| fed77f6ce0 | |||
| 6de0998f69 | |||
| 43cf0bd81e | |||
| 907baf3379 | |||
| 8b6550f8c6 | |||
| 20ede4391c | |||
| 864eb1316d | |||
| eff50346fd | |||
| 25383f893b | |||
| 46c650e9ae | |||
| 7876721906 | |||
| 5ed12f2bf0 | |||
| 03d6ba0da5 | |||
| f1af88a25c | |||
| 1905369c9f | |||
| a3acf4b8b6 | |||
| 478bdd6fe6 | |||
| e408b08443 | |||
| 2c904bfac5 | |||
| e9554c1e69 | |||
| 8f2205dd30 | |||
| 70ab76fe90 | |||
| ed7f8ee56e | |||
| 8c7ebbfe32 | |||
| cc09bedc07 | |||
| 5fe40a5242 | |||
| 9130ee4b03 | |||
| 36da0d6adb | |||
| e7087d518b | |||
| 3ea11b5984 | |||
| 33f70f7de3 | |||
| 45dd62f568 | |||
| 4f8ae5bb70 | |||
| e2dac589a8 | |||
| 1b87131e82 | |||
| 3dcaf0b79d | |||
| ef6aa143d0 | |||
| 09a76e2a9a |
@@ -0,0 +1,23 @@
|
|||||||
|
name: CI
|
||||||
|
|
||||||
|
on: [push, pull_request]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build-and-test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
container: gitea.tap-tap.win/taptap/fastsync-ci:v5
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Configure
|
||||||
|
run: cmake -B build -S .
|
||||||
|
|
||||||
|
- name: Build
|
||||||
|
run: cmake --build build -j$(nproc)
|
||||||
|
|
||||||
|
- name: Unit Tests
|
||||||
|
run: ./build/tests
|
||||||
|
|
||||||
|
- name: Integration Tests
|
||||||
|
run: python3 test.py
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
---
|
||||||
|
description: Reviews C code for memory safety, thread safety, null checks, buffer overflows, and style conventions specific to the FastSync codebase.
|
||||||
|
mode: subagent
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a C code reviewer for the FastSync project — a high-performance file synchronization system written in C11.
|
||||||
|
|
||||||
|
## Your Role
|
||||||
|
|
||||||
|
Review C source files for correctness, safety, and style. You have deep knowledge of this codebase's patterns and conventions.
|
||||||
|
|
||||||
|
## Codebase Context
|
||||||
|
|
||||||
|
### Project Structure
|
||||||
|
- `src/shared/` — shared libraries (protocol, compression, queue, config, data, metadata, transport, etc.)
|
||||||
|
- `src/client/` — client CLI, file sending, scanner
|
||||||
|
- `src/server/` — TCP server
|
||||||
|
- `tests/` — unit tests with custom framework
|
||||||
|
|
||||||
|
### Key Data Types
|
||||||
|
- `Data` — generic buffer (`void *data`, `size_t size`). Always use `data_create()` / `data_destroy()`.
|
||||||
|
- `Queue` — thread-safe bounded queue with optional `item_destroyer` callback. Use `queue_create()` / `queue_destroy()`.
|
||||||
|
- `Config` — runtime configuration struct. Use `config_create()` / `config_delete()`.
|
||||||
|
- `Chunk` — collection of files for batch transfer.
|
||||||
|
- `FileMetadata` — mode, uid, gid, mtime fields.
|
||||||
|
- `Server` / `Client` — TCP transport structs.
|
||||||
|
|
||||||
|
### Threading
|
||||||
|
- Uses C11 `<threads.h>` (`thrd_t`, `mtx_t`, `cnd_t`), NOT pthreads directly.
|
||||||
|
- Producer-consumer pattern with `queue_enqueue_multithreaded()` / `queue_dequeue_multithreaded()`.
|
||||||
|
- Bounded queues use condition variables for signaling.
|
||||||
|
|
||||||
|
### Memory Conventions
|
||||||
|
- All heap allocations use `malloc`/`calloc`/`realloc` + `free`.
|
||||||
|
- Destroy functions (`data_destroy`, `queue_destroy`, `config_delete`, etc.) handle cleanup.
|
||||||
|
- Ownership is transferred at function boundaries — document who owns what.
|
||||||
|
|
||||||
|
## Review Checklist
|
||||||
|
|
||||||
|
### Memory Safety
|
||||||
|
- Every `malloc`/`calloc` has a corresponding `free` on all code paths (including error paths).
|
||||||
|
- No use-after-free: check that pointers aren't used after their destroy function is called.
|
||||||
|
- No double-free: ensure destroy functions aren't called twice on the same object.
|
||||||
|
- Null checks after allocation before use.
|
||||||
|
- Buffer sizes are correct — no off-by-one in string operations (`strlen` + 1 for null terminator).
|
||||||
|
- `Data` objects created with `data_create()` and freed with `data_destroy()`.
|
||||||
|
|
||||||
|
### Thread Safety
|
||||||
|
- Shared state accessed under proper mutex protection.
|
||||||
|
- No race conditions on queue operations — using `_multithreaded` variants when threads are involved.
|
||||||
|
- Condition variable signals happen under the lock.
|
||||||
|
- No deadlock potential — consistent lock ordering.
|
||||||
|
- `done` flags checked properly in consumer loops.
|
||||||
|
|
||||||
|
### Protocol Safety
|
||||||
|
- `send_n_data` / `receive_n_data` return values checked.
|
||||||
|
- Status codes validated before use.
|
||||||
|
- Config serialization/deserialization handles partial reads.
|
||||||
|
|
||||||
|
### Style
|
||||||
|
- Header guards: `#ifndef FILENAME_H` / `#define FILENAME_H` / `#endif`
|
||||||
|
- Function naming: `snake_case`, prefixed by module (`queue_create`, `data_compress`, `config_send`).
|
||||||
|
- `static` for file-local functions.
|
||||||
|
- Consistent pointer style: `Type *name` (space before asterisk).
|
||||||
|
- Error handling: return `false`/`NULL` on failure, log when appropriate.
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
For each issue found, report:
|
||||||
|
1. **File and line** — exact location
|
||||||
|
2. **Severity** — critical / warning / style
|
||||||
|
3. **Category** — memory / thread / protocol / style
|
||||||
|
4. **Description** — what's wrong and how to fix it
|
||||||
|
|
||||||
|
If the code is clean, say so explicitly. Be concise — don't pad with fluff.
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
---
|
||||||
|
description: Manages the CMake build system for FastSync — adding targets, source files, dependencies, compiler flags, and sanitizer configurations.
|
||||||
|
mode: subagent
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a CMake expert for the FastSync project — a high-performance file synchronization system built with CMake 4.1+ and C11.
|
||||||
|
|
||||||
|
## Your Role
|
||||||
|
|
||||||
|
Manage the CMake build system: add new targets, configure dependencies, set compiler flags, and handle build configurations.
|
||||||
|
|
||||||
|
## Current Build Setup
|
||||||
|
|
||||||
|
### `CMakeLists.txt` (project root)
|
||||||
|
```cmake
|
||||||
|
cmake_minimum_required(VERSION 4.1)
|
||||||
|
project(FastFileTransfer)
|
||||||
|
|
||||||
|
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
|
||||||
|
set(CMAKE_C_STANDARD 11)
|
||||||
|
set(CMAKE_C_STANDARD_REQUIRED ON)
|
||||||
|
|
||||||
|
add_compile_options(-Wall -g -O3)
|
||||||
|
|
||||||
|
set(THREADS_PREFER_PTHREAD_FLAG ON)
|
||||||
|
find_package(Threads REQUIRED)
|
||||||
|
|
||||||
|
find_library(ZSTD_LIBRARY zstd)
|
||||||
|
# ... error if not found
|
||||||
|
|
||||||
|
# Source file collection
|
||||||
|
file(GLOB SHARED_SRCS "src/shared/*.c")
|
||||||
|
file(GLOB SERVER_SRCS "src/server/*.c")
|
||||||
|
file(GLOB CLIENT_SRCS "src/client/*.c")
|
||||||
|
file(GLOB TEST_SRCS "tests/*.c")
|
||||||
|
|
||||||
|
# Targets
|
||||||
|
add_executable(server ${SERVER_SRCS} ${SHARED_SRCS})
|
||||||
|
target_include_directories(server PRIVATE src/shared src/server src/client)
|
||||||
|
target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY})
|
||||||
|
|
||||||
|
add_executable(client ${CLIENT_SRCS} ${SHARED_SRCS})
|
||||||
|
target_include_directories(client PRIVATE src/shared src/server src/client)
|
||||||
|
target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY})
|
||||||
|
|
||||||
|
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} src/client/scanner.c)
|
||||||
|
target_include_directories(tests PRIVATE tests src/shared src/server src/client)
|
||||||
|
target_link_libraries(tests PRIVATE Threads::Threads ${ZSTD_LIBRARY})
|
||||||
|
```
|
||||||
|
|
||||||
|
### Source Layout
|
||||||
|
```
|
||||||
|
src/shared/ — shared libraries (globbed as SHARED_SRCS)
|
||||||
|
src/client/ — client sources (globbed as CLIENT_SRCS)
|
||||||
|
src/server/ — server sources (globbed as SERVER_SRCS)
|
||||||
|
tests/ — test sources (globbed as TEST_SRCS)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Dependencies
|
||||||
|
- **zstd** — found via `find_library(ZSTD_LIBRARY zstd)`
|
||||||
|
- **pthreads** — found via `find_package(Threads REQUIRED)`
|
||||||
|
- **C11 standard** — required
|
||||||
|
- **CMake 4.1+** — minimum version
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- Use `file(GLOB ...)` for source collection (existing pattern).
|
||||||
|
- All targets link `Threads::Threads` and `${ZSTD_LIBRARY}`.
|
||||||
|
- Include directories: `src/shared`, `src/server`, `src/client`, `tests` (for test target).
|
||||||
|
- Sanitizer support is commented out but present (`-fsanitize=address`).
|
||||||
|
- Build with `cmake -B build -S . && cmake --build build -j$(nproc)`.
|
||||||
|
|
||||||
|
## When Making Changes
|
||||||
|
|
||||||
|
1. Preserve existing structure and conventions.
|
||||||
|
2. Use `file(GLOB)` for new source directories (match existing pattern).
|
||||||
|
3. Add new dependencies with `find_package` or `find_library`.
|
||||||
|
4. When adding a new executable target, follow the pattern of existing targets.
|
||||||
|
5. When adding a new library (static/shared), use `add_library` and follow the project's naming.
|
||||||
|
6. For sanitizer builds, use the commented-out `-fsanitize=address` lines as reference.
|
||||||
|
7. Always verify the build compiles after changes.
|
||||||
|
|
||||||
|
## Build Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cmake -B build -S .
|
||||||
|
cmake --build build -j$(nproc)
|
||||||
|
./build/server
|
||||||
|
./build/client
|
||||||
|
./build/tests
|
||||||
|
```
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
---
|
||||||
|
description: Generates and maintains API documentation, protocol specs, and usage examples from the FastSync C source code.
|
||||||
|
mode: subagent
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a documentation generator for the FastSync project — a high-performance file synchronization system written in C11.
|
||||||
|
|
||||||
|
## Your Role
|
||||||
|
|
||||||
|
Generate accurate documentation from the actual source code. Maintain API references, protocol specifications, and usage examples.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
### Source Layout
|
||||||
|
```
|
||||||
|
src/shared/ — shared libraries (protocol, compression, queue, config, data, metadata, transport, etc.)
|
||||||
|
src/client/ — client CLI, file sending, scanner
|
||||||
|
src/server/ — TCP server
|
||||||
|
tests/ — unit tests
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Headers to Document
|
||||||
|
|
||||||
|
| Header | Purpose |
|
||||||
|
|--------|---------|
|
||||||
|
| `data.h` | Generic buffer type (`Data`) |
|
||||||
|
| `queue.h` | Thread-safe bounded queue |
|
||||||
|
| `chunk.h` | File chunking for batch transfer |
|
||||||
|
| `compression.h` | zstd streaming compression |
|
||||||
|
| `config.h` | Runtime configuration |
|
||||||
|
| `protocol.h` | Wire protocol (status codes, send/receive) |
|
||||||
|
| `metadata.h` | File metadata (mode, uid, gid, mtime) |
|
||||||
|
| `transport_tcp.h` | TCP client/server |
|
||||||
|
| `transport_ssh.h` | SSH transport with ControlMaster |
|
||||||
|
| `scanner.h` | Directory traversal and file scanning |
|
||||||
|
| `file.h` | File representation |
|
||||||
|
| `array_list.h` | Dynamic array |
|
||||||
|
| `log.h` | Logging utilities |
|
||||||
|
| `utils.h` | Shared utilities |
|
||||||
|
|
||||||
|
### README
|
||||||
|
The project README at `README.md` contains:
|
||||||
|
- Technical overview
|
||||||
|
- System architecture
|
||||||
|
- Protocol details
|
||||||
|
- Command-line arguments
|
||||||
|
- Environment variables
|
||||||
|
- Build instructions
|
||||||
|
- Benchmark results
|
||||||
|
|
||||||
|
## Documentation Types
|
||||||
|
|
||||||
|
### 1. API Reference (from headers)
|
||||||
|
For each public function:
|
||||||
|
- Signature (from the header)
|
||||||
|
- Brief description
|
||||||
|
- Parameters and return value
|
||||||
|
- Memory ownership rules
|
||||||
|
- Thread safety guarantees
|
||||||
|
|
||||||
|
### 2. Protocol Specification
|
||||||
|
- Wire format byte layouts
|
||||||
|
- Status code semantics
|
||||||
|
- Transfer flow diagrams
|
||||||
|
- Metadata encoding
|
||||||
|
|
||||||
|
### 3. Architecture Docs
|
||||||
|
- Data flow diagrams
|
||||||
|
- Component interactions
|
||||||
|
- Threading model
|
||||||
|
|
||||||
|
### 4. Usage Examples
|
||||||
|
- Command-line examples for common use cases
|
||||||
|
- Build instructions
|
||||||
|
- Integration scenarios
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- Use `file:line` references when pointing to source locations
|
||||||
|
- Document actual behavior, not intended behavior
|
||||||
|
- Include error conditions and edge cases
|
||||||
|
- Keep docs close to the code they describe
|
||||||
|
- Use markdown formatting suitable for terminal rendering
|
||||||
|
|
||||||
|
## When Generating Documentation
|
||||||
|
|
||||||
|
1. Read the actual source files first — don't assume behavior
|
||||||
|
2. Cross-reference headers with implementations
|
||||||
|
3. Verify examples actually compile and work
|
||||||
|
4. Update README when adding/changing features
|
||||||
|
5. Keep protocol docs in sync with code changes
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
---
|
||||||
|
description: Analyzes performance bottlenecks in the FastSync transfer pipeline and suggests concrete optimizations for chunking, compression, threading, and network transport.
|
||||||
|
mode: subagent
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a performance analyst for the FastSync project — a high-performance file synchronization system written in C11.
|
||||||
|
|
||||||
|
## Your Role
|
||||||
|
|
||||||
|
Analyze the transfer pipeline for performance bottlenecks and suggest concrete, actionable optimizations. You understand the full data flow from scanner to network.
|
||||||
|
|
||||||
|
## Architecture Overview
|
||||||
|
|
||||||
|
### Transfer Pipeline
|
||||||
|
```
|
||||||
|
DirectoryScanner → Queue(Scanner→Loader) → ChunkBuilder → Queue(Loader→Sender) → Network Send
|
||||||
|
```
|
||||||
|
|
||||||
|
1. **Scanner** — BFS traversal, builds file list, groups into chunks
|
||||||
|
2. **Loader** — reads file contents into memory
|
||||||
|
3. **Sender** — compresses + serializes + sends over TCP/SSH
|
||||||
|
|
||||||
|
### Key Components
|
||||||
|
|
||||||
|
| Component | File | Purpose |
|
||||||
|
|-----------|------|---------|
|
||||||
|
| Scanner | `src/client/scanner.c` | BFS directory traversal, exclude patterns, chunk building |
|
||||||
|
| Chunk | `src/shared/chunk.c` | File grouping (~10MB default), serialization |
|
||||||
|
| Compression | `src/shared/compression.c` | Streaming zstd (levels 1–22) |
|
||||||
|
| Queue | `src/shared/queue.c` | Thread-safe bounded queue with condition variables |
|
||||||
|
| Transport TCP | `src/shared/transport_tcp.c` | TCP with `sendfile()` zero-copy |
|
||||||
|
| Transport SSH | `src/shared/transport_ssh.c` | SSH with ControlMaster, socketpair |
|
||||||
|
| Protocol | `src/shared/protocol.c` | Status codes, data send/receive |
|
||||||
|
| Config | `src/shared/config.c` | Runtime parameters |
|
||||||
|
|
||||||
|
### Performance-Critical Paths
|
||||||
|
|
||||||
|
1. **Chunk size** (`DEFAULT_CHUNK_SIZE = 10MB`) — balances memory vs. transfer efficiency
|
||||||
|
2. **Compression level** (1–22) — trades CPU for bandwidth
|
||||||
|
3. **`sendfile()` zero-copy** — bypasses userspace, ~2× faster on loopback
|
||||||
|
4. **Multithreading** — producer-consumer with thread-safe queues
|
||||||
|
5. **SSH socketpair buffer** — set to 1MB for pipe throughput
|
||||||
|
6. **Streaming compression** — `ZSTD_compressStream2` / `ZSTD_decompressStream`
|
||||||
|
|
||||||
|
## Analysis Framework
|
||||||
|
|
||||||
|
### When Analyzing, Consider
|
||||||
|
|
||||||
|
1. **CPU-bound vs I/O-bound** — Is the bottleneck CPU (compression) or I/O (disk/network)?
|
||||||
|
2. **Memory allocation** — Are there excessive malloc/free cycles in hot paths?
|
||||||
|
3. **Lock contention** — Are mutexes held too long? Is the queue the bottleneck?
|
||||||
|
4. **Syscall overhead** — Are there unnecessary read/write cycles?
|
||||||
|
5. **Pipeline stalls** — Is any stage starved or blocked?
|
||||||
|
6. **Data copying** — Are there unnecessary memcpy operations?
|
||||||
|
7. **Algorithmic** — Is the chunking/scanning algorithm optimal?
|
||||||
|
|
||||||
|
### Benchmark Context
|
||||||
|
|
||||||
|
From README benchmarks (25MB mixed files, localhost):
|
||||||
|
- Best config: `-m -c` (multithread + compression) → 0.20s, 11.2× faster than rsync
|
||||||
|
- `sendfile()` bypasses userspace → ~2× faster on localhost
|
||||||
|
- Compression reduces wire data enough that transfer becomes latency-bound on WAN
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
For each bottleneck found:
|
||||||
|
1. **Location** — file:line
|
||||||
|
2. **Impact** — high / medium / low
|
||||||
|
3. **Type** — CPU / IO / memory / lock / algorithmic
|
||||||
|
4. **Current behavior** — what's happening
|
||||||
|
5. **Suggested optimization** — concrete code change or approach
|
||||||
|
6. **Expected impact** — estimated speedup or resource savings
|
||||||
|
|
||||||
|
Also provide profiling guidance when asked (e.g., `perf`, `valgrind`, `gprof` commands).
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
---
|
||||||
|
description: Designs and extends the FastSync wire protocol — status codes, metadata format, chunk serialization, config serialization, and ensures backward compatibility.
|
||||||
|
mode: subagent
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a protocol designer for the FastSync project — a high-performance file synchronization system with a custom binary wire protocol.
|
||||||
|
|
||||||
|
## Your Role
|
||||||
|
|
||||||
|
Design, extend, and document the wire protocol. Ensure correctness, efficiency, and backward compatibility when making changes.
|
||||||
|
|
||||||
|
## Current Protocol
|
||||||
|
|
||||||
|
### Status Codes (`src/shared/protocol.h`)
|
||||||
|
```c
|
||||||
|
enum NET_STATUS {
|
||||||
|
STATUS_OK, // Operation successful
|
||||||
|
STATUS_ERROR, // Error occurred
|
||||||
|
STATUS_FINISHED, // Transfer complete
|
||||||
|
STATUS_NEXT, // Ready for next file (per-file mode)
|
||||||
|
STATUS_CHUNK, // Following data is a serialized chunk
|
||||||
|
STATUS_MANIFEST // Following data is a file manifest (for --delete)
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### Wire Format
|
||||||
|
|
||||||
|
#### Config (sent at transfer start)
|
||||||
|
Serialized fields: version, send_directory, receive_root_directory, save_to_disk, use_multithreading, use_chunk_serialization, use_compression, use_metadata, compression_level, use_sendfile, chunk_size, transport type, ssh_destination.
|
||||||
|
|
||||||
|
#### Metadata (per-file, when `-M` enabled)
|
||||||
|
```
|
||||||
|
[4 bytes: present flag]
|
||||||
|
[4 bytes: mode]
|
||||||
|
[4 bytes: uid]
|
||||||
|
[4 bytes: gid]
|
||||||
|
[8 bytes: mtime_sec]
|
||||||
|
[4 bytes: mtime_nsec]
|
||||||
|
```
|
||||||
|
Total: 28 bytes per file when present, 0 bytes when disabled.
|
||||||
|
|
||||||
|
#### Data Transfer
|
||||||
|
```
|
||||||
|
Config → (STATUS_NEXT | STATUS_CHUNK)* → [STATUS_MANIFEST] → STATUS_FINISHED → STATUS_OK
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Per-file mode**: `STATUS_NEXT` → file data → `STATUS_NEXT` → ...
|
||||||
|
- **Chunk mode**: `STATUS_CHUNK` → serialized chunk data → ...
|
||||||
|
- **Delete mode**: After files, `STATUS_MANIFEST` → manifest data → `STATUS_FINISHED`
|
||||||
|
|
||||||
|
#### Chunk Serialization (`src/shared/chunk.c`)
|
||||||
|
Files grouped into chunks (~10MB default). Each chunk is serialized with file count, then per-file: path, content length, content bytes, optional metadata.
|
||||||
|
|
||||||
|
### Data Serialization (`src/shared/data.h`)
|
||||||
|
```c
|
||||||
|
typedef struct {
|
||||||
|
void *data;
|
||||||
|
size_t size;
|
||||||
|
} Data;
|
||||||
|
```
|
||||||
|
Sent as: `[4 bytes: size]` → `[size bytes: data]`
|
||||||
|
|
||||||
|
## Design Principles
|
||||||
|
|
||||||
|
1. **Efficiency** — minimize wire overhead; batch when possible
|
||||||
|
2. **Backward compatibility** — version field in config for negotiation
|
||||||
|
3. **Simplicity** — status-code-driven exchange, no complex state machines
|
||||||
|
4. **Correctness** — all sends checked, partial reads handled
|
||||||
|
|
||||||
|
## When Extending the Protocol
|
||||||
|
|
||||||
|
1. **Add new status codes** — append to enum, update protocol documentation
|
||||||
|
2. **Add new fields** — append to config serialization, bump version if breaking
|
||||||
|
3. **Add new metadata** — extend metadata format with new optional fields
|
||||||
|
4. **Wire format changes** — document exact byte layout
|
||||||
|
5. **Backward compatibility** — always support reading old formats via version check
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
When designing protocol changes:
|
||||||
|
1. **Motivation** — why the change is needed
|
||||||
|
2. **Wire format** — exact byte-level layout (hex offsets if complex)
|
||||||
|
3. **Status code changes** — new/modified codes
|
||||||
|
4. **Serialization code** — changes to `protocol.c`, `config.c`, `chunk.c`
|
||||||
|
5. **Compatibility notes** — how old clients/servers handle the change
|
||||||
|
6. **Testing strategy** — how to verify the protocol change works
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
---
|
||||||
|
description: Writes unit tests for the FastSync C codebase using the custom test framework. Creates test_*.c, test_*.h, and registers tests in runner.c.
|
||||||
|
mode: subagent
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a test writer for the FastSync project — a high-performance file synchronization system written in C11.
|
||||||
|
|
||||||
|
## Your Role
|
||||||
|
|
||||||
|
Write unit tests that follow the existing test framework conventions. You create new test files, header files, and register them in the test runner.
|
||||||
|
|
||||||
|
## Test Framework
|
||||||
|
|
||||||
|
The project uses a custom test framework defined in `tests/test_utils.h`.
|
||||||
|
|
||||||
|
### Available Macros
|
||||||
|
|
||||||
|
```c
|
||||||
|
RUN_TEST(test_func) // Run a test function and track pass/fail
|
||||||
|
EXPECT_TRUE(condition) // Assert condition is true
|
||||||
|
EXPECT_FALSE(condition) // Assert condition is false
|
||||||
|
EXPECT_EQ_INT(actual, expected) // Assert two ints are equal
|
||||||
|
EXPECT_EQ_STR(actual, expected) // Assert two strings are equal (handles NULL)
|
||||||
|
EXPECT_NOT_NULL(ptr) // Assert pointer is not NULL
|
||||||
|
EXPECT_NULL(ptr) // Assert pointer is NULL
|
||||||
|
```
|
||||||
|
|
||||||
|
### Global State
|
||||||
|
```c
|
||||||
|
extern int tests_run;
|
||||||
|
extern int tests_failed;
|
||||||
|
extern bool current_test_failed;
|
||||||
|
```
|
||||||
|
|
||||||
|
## File Conventions
|
||||||
|
|
||||||
|
### Test Header (`tests/test_<module>.h`)
|
||||||
|
```c
|
||||||
|
#ifndef TEST_<MODULE>_H
|
||||||
|
#define TEST_<MODULE>_H
|
||||||
|
|
||||||
|
void test_<module>();
|
||||||
|
|
||||||
|
#endif
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test Source (`tests/test_<module>.c`)
|
||||||
|
```c
|
||||||
|
#include "test_<module>.h"
|
||||||
|
#include "<module>.h" // The header being tested
|
||||||
|
#include "test_utils.h"
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
|
||||||
|
static void test_<module>_<specific_case>() {
|
||||||
|
// Arrange
|
||||||
|
// Act
|
||||||
|
// Assert using EXPECT_* macros
|
||||||
|
// IMPORTANT: return immediately on failure (macros do this)
|
||||||
|
}
|
||||||
|
|
||||||
|
void test_<module>() {
|
||||||
|
test_<module>_<case1>();
|
||||||
|
test_<module>_<case2>();
|
||||||
|
// ...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Registration in `tests/runner.c`
|
||||||
|
Add the `#include` and `RUN_TEST()` call:
|
||||||
|
```c
|
||||||
|
#include "test_<module>.h"
|
||||||
|
// ...
|
||||||
|
RUN_TEST(test_<module>);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Patterns to Follow
|
||||||
|
|
||||||
|
### Memory Management in Tests
|
||||||
|
- `malloc` test data, `free` after assertions.
|
||||||
|
- Use destroy functions (`data_destroy`, `queue_destroy`, etc.) for framework objects.
|
||||||
|
- Don't leak — every allocation must be freed.
|
||||||
|
|
||||||
|
### Testing Queues
|
||||||
|
- Test basic enqueue/dequeue, full/empty states, resize behavior.
|
||||||
|
- Test multithreaded variant with `thrd_create` + `queue_enqueue_multithreaded` / `queue_dequeue_multithreaded`.
|
||||||
|
- Use `mtx_t` and `cnd_t` for thread synchronization in tests.
|
||||||
|
|
||||||
|
### Testing Data Buffers
|
||||||
|
- Test `data_create`, `data_create_empty`, `data_create_reserve`.
|
||||||
|
- Verify size and content after creation.
|
||||||
|
|
||||||
|
### Testing Compression
|
||||||
|
- Compress data, decompress, verify round-trip.
|
||||||
|
- Test with various compression levels.
|
||||||
|
|
||||||
|
### Testing Config
|
||||||
|
- Test `config_create` and `config_delete`.
|
||||||
|
- Test serialization round-trip (`config_send` + `config_receive`).
|
||||||
|
|
||||||
|
### Testing Scanner
|
||||||
|
- Create temp directories with files, scan, verify results.
|
||||||
|
- Test exclude pattern matching.
|
||||||
|
|
||||||
|
### Edge Cases to Always Cover
|
||||||
|
- NULL inputs
|
||||||
|
- Empty collections (size 0)
|
||||||
|
- Single element
|
||||||
|
- At capacity boundaries
|
||||||
|
- Invalid parameters
|
||||||
|
|
||||||
|
## Build & Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cmake -B build -S . && cmake --build build -j$(nproc) && ./build/tests
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
When asked to write tests, produce:
|
||||||
|
1. The test header file content
|
||||||
|
2. The test source file content
|
||||||
|
3. The runner.c modification needed
|
||||||
|
4. Verify with a build and test run
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
---
|
||||||
|
name: pr-build
|
||||||
|
description: Builds and tests a pull request branch, fixing compilation errors and test failures. Use when the user says "build PR", "fix PR build", "run PR build", or wants to compile and test a PR branch.
|
||||||
|
---
|
||||||
|
|
||||||
|
# PR Build Skill
|
||||||
|
|
||||||
|
Builds, tests, and fixes a pull request branch. This skill CAN edit files, commit, and push.
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
### Step 1: Identify the PR branch
|
||||||
|
|
||||||
|
If the user specifies a PR number, check it out:
|
||||||
|
```bash
|
||||||
|
tea pr checkout <number>
|
||||||
|
```
|
||||||
|
|
||||||
|
If already on a PR branch, verify with:
|
||||||
|
```bash
|
||||||
|
git branch --show-current
|
||||||
|
git log main..HEAD --oneline
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Clean build
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rm -rf build
|
||||||
|
cmake -B build -S . 2>&1
|
||||||
|
cmake --build build -j$(nproc) 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
Capture both stdout and stderr.
|
||||||
|
|
||||||
|
### Step 3: Handle build failures
|
||||||
|
|
||||||
|
If the build fails, read the error output carefully. Common issues:
|
||||||
|
|
||||||
|
**Missing include / undefined reference:**
|
||||||
|
- Check if the new `.c` file is in the right `file(GLOB ...)` directory
|
||||||
|
- Check if the new `.h` file is included properly
|
||||||
|
- Check if CMakeLists.txt needs updating (new target, new source file, new dependency)
|
||||||
|
|
||||||
|
**Type errors / implicit declarations:**
|
||||||
|
- Check function signatures match between `.h` and `.c`
|
||||||
|
- Check struct field names and types
|
||||||
|
|
||||||
|
**Linker errors:**
|
||||||
|
- Check if all required libraries are linked in CMakeLists.txt
|
||||||
|
- Check if all source files are included in the target
|
||||||
|
|
||||||
|
Use the cmake-expert agent to diagnose and fix CMake issues.
|
||||||
|
|
||||||
|
### Step 4: Run unit tests
|
||||||
|
|
||||||
|
If build succeeds:
|
||||||
|
```bash
|
||||||
|
./build/tests
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 5: Handle test failures
|
||||||
|
|
||||||
|
If tests fail:
|
||||||
|
- Read the test output carefully
|
||||||
|
- Check which test function failed and the assertion line
|
||||||
|
- Read the test source file and the module being tested
|
||||||
|
- Use the test-writer agent to investigate and fix
|
||||||
|
|
||||||
|
### Step 6: Run integration tests (optional)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 test.py
|
||||||
|
```
|
||||||
|
|
||||||
|
This runs the integration + benchmark suite. It takes longer — only run if the user asks or if unit tests pass.
|
||||||
|
|
||||||
|
### Step 7: Fix and commit
|
||||||
|
|
||||||
|
If fixes were needed:
|
||||||
|
```bash
|
||||||
|
git add -A
|
||||||
|
git commit -m "Fix build: <brief description of what was fixed>"
|
||||||
|
git push
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 8: Report results
|
||||||
|
|
||||||
|
Print a summary:
|
||||||
|
|
||||||
|
```
|
||||||
|
=== PR BUILD SUMMARY ===
|
||||||
|
Branch: <branch-name>
|
||||||
|
Build: [PASS/FAIL]
|
||||||
|
Unit tests: [PASS/FAIL] (<passed>/<total>)
|
||||||
|
Integration tests: [PASS/FAIL/SKIPPED]
|
||||||
|
|
||||||
|
Fixes applied: <count>
|
||||||
|
<list of fixes with commit hashes>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- DO edit source files and CMakeLists.txt to fix issues
|
||||||
|
- DO commit and push fixes
|
||||||
|
- Always build from clean state (rm -rf build)
|
||||||
|
- Read error messages carefully before fixing
|
||||||
|
- Don't change functionality — only fix build/test issues
|
||||||
|
- Preserve existing code style when making fixes
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
---
|
||||||
|
name: pr-review
|
||||||
|
description: Reviews a pull request for bugs, memory safety, thread safety, and style issues. Use when the user says "review PR", "review this PR", "review pull request", or wants a code review of changes.
|
||||||
|
---
|
||||||
|
|
||||||
|
# PR Review Skill
|
||||||
|
|
||||||
|
Read-only code review of a pull request branch. Produces a report — does NOT edit files.
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
### Step 1: Identify the PR branch
|
||||||
|
|
||||||
|
If the user specifies a PR number, check it out:
|
||||||
|
```bash
|
||||||
|
tea pr checkout <number>
|
||||||
|
```
|
||||||
|
|
||||||
|
If already on a PR branch, verify with:
|
||||||
|
```bash
|
||||||
|
git branch --show-current
|
||||||
|
git log main..HEAD --oneline
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Get changed files
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git diff main --name-only -- '*.c' '*.h'
|
||||||
|
```
|
||||||
|
|
||||||
|
This gives the list of C source and header files changed in the PR.
|
||||||
|
|
||||||
|
### Step 3: Read all changed files
|
||||||
|
|
||||||
|
Use the Read tool to read every changed `.c` and `.h` file. Read full files — don't skip any.
|
||||||
|
|
||||||
|
### Step 4: Review each file
|
||||||
|
|
||||||
|
For each changed file, review for:
|
||||||
|
|
||||||
|
**Memory Safety**
|
||||||
|
- Every `malloc`/`calloc` has a matching `free` on all code paths (including error paths)
|
||||||
|
- No use-after-free (pointers used after `*_destroy()` is called)
|
||||||
|
- No double-free
|
||||||
|
- Null checks after allocation before use
|
||||||
|
- Correct buffer sizes (strlen + 1 for null terminators)
|
||||||
|
- `Data` objects created/destroyed properly
|
||||||
|
|
||||||
|
**Thread Safety**
|
||||||
|
- Shared state accessed under mutex
|
||||||
|
- No race conditions on queue operations
|
||||||
|
- Condition variable signals under lock
|
||||||
|
- No deadlock potential (consistent lock ordering)
|
||||||
|
- `done` flags checked properly in consumer loops
|
||||||
|
|
||||||
|
**Protocol Safety**
|
||||||
|
- `send_n_data` / `receive_n_data` return values checked
|
||||||
|
- Status codes validated before use
|
||||||
|
- Config serialization handles partial reads
|
||||||
|
|
||||||
|
**Logic Errors**
|
||||||
|
- Off-by-one in loops/buffers
|
||||||
|
- Incorrect size calculations
|
||||||
|
- Wrong enum values or comparisons
|
||||||
|
- Missing break statements in switch
|
||||||
|
|
||||||
|
**Error Handling**
|
||||||
|
- Resources freed on error paths (no leaks)
|
||||||
|
- Functions return appropriate error values
|
||||||
|
- Error messages are useful
|
||||||
|
|
||||||
|
### Step 5: Categorize findings
|
||||||
|
|
||||||
|
For each issue:
|
||||||
|
1. **File:line** — exact location
|
||||||
|
2. **Severity** — critical / warning / style
|
||||||
|
3. **Category** — memory / thread / protocol / logic / error
|
||||||
|
4. **Description** — what's wrong and how to fix it
|
||||||
|
|
||||||
|
### Step 6: Output report
|
||||||
|
|
||||||
|
Print a formatted summary:
|
||||||
|
|
||||||
|
```
|
||||||
|
=== PR REVIEW SUMMARY ===
|
||||||
|
Branch: <branch-name>
|
||||||
|
Files reviewed: <count>
|
||||||
|
Issues found: <count>
|
||||||
|
|
||||||
|
CRITICAL: <count>
|
||||||
|
WARNING: <count>
|
||||||
|
STYLE: <count>
|
||||||
|
|
||||||
|
=== ISSUES ===
|
||||||
|
[1] src/shared/compression.c:42 — CRITICAL (memory)
|
||||||
|
Potential leak: data returned from data_compress() not freed on error path
|
||||||
|
Fix: Add data_destroy(compressed) before return false
|
||||||
|
|
||||||
|
...
|
||||||
|
|
||||||
|
=== VERDICT ===
|
||||||
|
[PASS] No critical issues found
|
||||||
|
— or —
|
||||||
|
[FAIL] <N> critical issues must be fixed before merge
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 7: Optional PR comment
|
||||||
|
|
||||||
|
If the user wants to post the review as a PR comment:
|
||||||
|
```bash
|
||||||
|
tea pr comment <number> --comment "<review report>"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- Do NOT edit any source files
|
||||||
|
- Do NOT run builds or tests
|
||||||
|
- Do NOT commit or push
|
||||||
|
- Report ALL issues — don't filter or minimize
|
||||||
|
- Be specific about line numbers and fix suggestions
|
||||||
+6
-4
@@ -1,4 +1,4 @@
|
|||||||
cmake_minimum_required(VERSION 4.1)
|
cmake_minimum_required(VERSION 3.22)
|
||||||
|
|
||||||
project(FastFileTransfer)
|
project(FastFileTransfer)
|
||||||
|
|
||||||
@@ -19,6 +19,8 @@ if(NOT ZSTD_LIBRARY)
|
|||||||
message(FATAL_ERROR "zstd library not found. Ensure it is in your nix-shell!")
|
message(FATAL_ERROR "zstd library not found. Ensure it is in your nix-shell!")
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
|
find_package(OpenSSL REQUIRED)
|
||||||
|
|
||||||
file(GLOB SHARED_SRCS "src/shared/*.c")
|
file(GLOB SHARED_SRCS "src/shared/*.c")
|
||||||
file(GLOB SERVER_SRCS "src/server/*.c")
|
file(GLOB SERVER_SRCS "src/server/*.c")
|
||||||
file(GLOB CLIENT_SRCS "src/client/*.c")
|
file(GLOB CLIENT_SRCS "src/client/*.c")
|
||||||
@@ -26,13 +28,13 @@ file(GLOB TEST_SRCS "tests/*.c")
|
|||||||
|
|
||||||
add_executable(server ${SERVER_SRCS} ${SHARED_SRCS})
|
add_executable(server ${SERVER_SRCS} ${SHARED_SRCS})
|
||||||
target_include_directories(server PRIVATE src/shared src/server src/client)
|
target_include_directories(server PRIVATE src/shared src/server src/client)
|
||||||
target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY})
|
target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto)
|
||||||
|
|
||||||
add_executable(client ${CLIENT_SRCS} ${SHARED_SRCS})
|
add_executable(client ${CLIENT_SRCS} ${SHARED_SRCS})
|
||||||
target_include_directories(client PRIVATE src/shared src/server src/client)
|
target_include_directories(client PRIVATE src/shared src/server src/client)
|
||||||
target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY})
|
target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto)
|
||||||
|
|
||||||
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} src/client/scanner.c)
|
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} src/client/scanner.c)
|
||||||
target_include_directories(tests PRIVATE tests src/shared src/server src/client)
|
target_include_directories(tests PRIVATE tests src/shared src/server src/client)
|
||||||
target_link_libraries(tests PRIVATE Threads::Threads ${ZSTD_LIBRARY})
|
target_link_libraries(tests PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
FROM ubuntu:24.04
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
gcc g++ make libc6-dev cmake libzstd-dev libssl-dev git ca-certificates curl && \
|
||||||
|
curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
|
||||||
|
apt-get install -y --no-install-recommends nodejs && \
|
||||||
|
rm -rf /var/lib/apt/lists/*
|
||||||
@@ -8,11 +8,13 @@ pkgs.mkShell {
|
|||||||
cmake
|
cmake
|
||||||
gnumake
|
gnumake
|
||||||
pkg-config
|
pkg-config
|
||||||
|
docker
|
||||||
tea
|
tea
|
||||||
];
|
];
|
||||||
|
|
||||||
buildInputs = with pkgs; [
|
buildInputs = with pkgs; [
|
||||||
zstd
|
zstd
|
||||||
|
openssl
|
||||||
];
|
];
|
||||||
|
|
||||||
NIX_ENFORCE_PURITY = 0;
|
NIX_ENFORCE_PURITY = 0;
|
||||||
|
|||||||
@@ -1,7 +1,11 @@
|
|||||||
#include "client_send.h"
|
#include "client_send.h"
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
|
#include "protocol.h"
|
||||||
|
#include "transport_tls.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
|
#include <errno.h>
|
||||||
|
#include <limits.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
@@ -43,6 +47,11 @@ static void print_usage(void) {
|
|||||||
printf(" --save-to-disk Write received files to disk\n");
|
printf(" --save-to-disk Write received files to disk\n");
|
||||||
printf(" --server-host <ip> Server IP address (default: 127.0.0.1)\n");
|
printf(" --server-host <ip> Server IP address (default: 127.0.0.1)\n");
|
||||||
printf(" --server-port <n> Server port (default: 8080)\n");
|
printf(" --server-port <n> Server port (default: 8080)\n");
|
||||||
|
printf(" --bwlimit <KB/s> Bandwidth limit in kilobytes per second\n");
|
||||||
|
printf(" --tls Enable TLS encryption\n");
|
||||||
|
printf(" --cert <path> TLS certificate file (PEM)\n");
|
||||||
|
printf(" --key <path> TLS private key file (PEM)\n");
|
||||||
|
printf(" --ca <path> TLS CA certificate file (PEM)\n");
|
||||||
printf(" --help Show this help\n");
|
printf(" --help Show this help\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -130,12 +139,37 @@ int main(int argc, char *argv[]) {
|
|||||||
server_host = str_dup(argv[++i]);
|
server_host = str_dup(argv[++i]);
|
||||||
} else if (strcmp(argv[i], "--server-port") == 0 && i + 1 < argc) {
|
} else if (strcmp(argv[i], "--server-port") == 0 && i + 1 < argc) {
|
||||||
server_port = atoi(argv[++i]);
|
server_port = atoi(argv[++i]);
|
||||||
|
} else if (strcmp(argv[i], "--bwlimit") == 0 && i + 1 < argc) {
|
||||||
|
char *end;
|
||||||
|
errno = 0;
|
||||||
|
unsigned long long kbps = strtoull(argv[++i], &end, 10);
|
||||||
|
if (errno != 0 || *end != '\0' || kbps == 0) {
|
||||||
|
fprintf(stderr, "Error: --bwlimit must be a positive integer\n");
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
if (kbps > ULLONG_MAX / 1024) {
|
||||||
|
fprintf(stderr, "Error: --bwlimit value too large\n");
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
io_set_bwlimit(kbps * 1024);
|
||||||
|
log_message(LOG_LEVEL_INFO, "Set bandwidth limit to %llu KB/s", kbps);
|
||||||
} else if (strcmp(argv[i], "--progress") == 0) {
|
} else if (strcmp(argv[i], "--progress") == 0) {
|
||||||
config->show_progress = true;
|
config->show_progress = true;
|
||||||
} else if (strcmp(argv[i], "--chunk-size") == 0 && i + 1 < argc) {
|
} else if (strcmp(argv[i], "--chunk-size") == 0 && i + 1 < argc) {
|
||||||
unsigned long long val = strtoull(argv[++i], NULL, 10);
|
unsigned long long val = strtoull(argv[++i], NULL, 10);
|
||||||
if (val > 0)
|
if (val > 0)
|
||||||
config->chunk_size = val;
|
config->chunk_size = val;
|
||||||
|
} else if (strcmp(argv[i], "--tls") == 0) {
|
||||||
|
config->use_tls = true;
|
||||||
|
} else if (strcmp(argv[i], "--cert") == 0 && i + 1 < argc) {
|
||||||
|
free(config->tls_cert);
|
||||||
|
config->tls_cert = str_dup(argv[++i]);
|
||||||
|
} else if (strcmp(argv[i], "--key") == 0 && i + 1 < argc) {
|
||||||
|
free(config->tls_key);
|
||||||
|
config->tls_key = str_dup(argv[++i]);
|
||||||
|
} else if (strcmp(argv[i], "--ca") == 0 && i + 1 < argc) {
|
||||||
|
free(config->tls_ca);
|
||||||
|
config->tls_ca = str_dup(argv[++i]);
|
||||||
} else if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) {
|
} else if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) {
|
||||||
set_log_level(LOG_LEVEL_DEBUG);
|
set_log_level(LOG_LEVEL_DEBUG);
|
||||||
} else if (argv[i][0] == '-') {
|
} else if (argv[i][0] == '-') {
|
||||||
@@ -197,6 +231,14 @@ int main(int argc, char *argv[]) {
|
|||||||
config->use_metadata = true;
|
config->use_metadata = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (config->use_tls) {
|
||||||
|
if (!config->tls_cert || !config->tls_key) {
|
||||||
|
fprintf(stderr, "Error: --tls requires --cert and --key\n");
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
tls_global_init();
|
||||||
|
}
|
||||||
|
|
||||||
if (config->use_multithreading)
|
if (config->use_multithreading)
|
||||||
return send_files_multithreaded(config);
|
return send_files_multithreaded(config);
|
||||||
return send_files(config);
|
return send_files(config);
|
||||||
|
|||||||
@@ -11,6 +11,7 @@
|
|||||||
#include "scanner.h"
|
#include "scanner.h"
|
||||||
#include "transport_tcp.h"
|
#include "transport_tcp.h"
|
||||||
#include "transport_ssh.h"
|
#include "transport_ssh.h"
|
||||||
|
#include "transport_tls.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
@@ -57,11 +58,11 @@ int send_chunk(Client *client, Chunk *chunk, Config *config) {
|
|||||||
int rc = incremental_check(client, chunk->items[i]);
|
int rc = incremental_check(client, chunk->items[i]);
|
||||||
if (rc < 0) return -1;
|
if (rc < 0) return -1;
|
||||||
if (rc > 0) continue;
|
if (rc > 0) continue;
|
||||||
if (!file_send_sendfile_no_path(chunk->items[i], client->file_descriptor, config->use_metadata))
|
if (!file_send_sendfile(chunk->items[i], client->file_descriptor, config->use_metadata, false))
|
||||||
return -1;
|
return -1;
|
||||||
} else {
|
} else {
|
||||||
if (!send_status(client->file_descriptor, STATUS_NEXT)) return -1;
|
if (!send_status(client->file_descriptor, STATUS_NEXT)) return -1;
|
||||||
if (!file_send_sendfile(chunk->items[i], client->file_descriptor, config->use_metadata))
|
if (!file_send_sendfile(chunk->items[i], client->file_descriptor, config->use_metadata, true))
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -71,15 +72,17 @@ int send_chunk(Client *client, Chunk *chunk, Config *config) {
|
|||||||
int rc = incremental_check(client, chunk->items[i]);
|
int rc = incremental_check(client, chunk->items[i]);
|
||||||
if (rc < 0) return -1;
|
if (rc < 0) return -1;
|
||||||
if (rc > 0) continue;
|
if (rc > 0) continue;
|
||||||
if (!file_send_single_calls_no_path(chunk->items[i], client->file_descriptor,
|
if (!file_send_single_calls(chunk->items[i], client->file_descriptor,
|
||||||
config->use_metadata,
|
config->use_metadata,
|
||||||
config->use_compression ? config->compression_level : 0))
|
config->use_compression ? config->compression_level : 0,
|
||||||
|
false))
|
||||||
return -1;
|
return -1;
|
||||||
} else {
|
} else {
|
||||||
if (!send_status(client->file_descriptor, STATUS_NEXT)) return -1;
|
if (!send_status(client->file_descriptor, STATUS_NEXT)) return -1;
|
||||||
if (!file_send_single_calls(chunk->items[i], client->file_descriptor,
|
if (!file_send_single_calls(chunk->items[i], client->file_descriptor,
|
||||||
config->use_metadata,
|
config->use_metadata,
|
||||||
config->use_compression ? config->compression_level : 0))
|
config->use_compression ? config->compression_level : 0,
|
||||||
|
true))
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -96,6 +99,16 @@ static int send_chunks_multithreaded(void *pipeline_context) {
|
|||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
client = client_connect_ssh(context->config->ssh_destination, context->config->ssh_port);
|
client = client_connect_ssh(context->config->ssh_destination, context->config->ssh_port);
|
||||||
|
} else if (context->config->use_tls) {
|
||||||
|
client = client_create();
|
||||||
|
if (!client || !client_connect_tls(client, server_host, server_port,
|
||||||
|
context->config->tls_cert,
|
||||||
|
context->config->tls_key,
|
||||||
|
context->config->tls_ca)) {
|
||||||
|
if (client) client_delete(client);
|
||||||
|
fprintf(stderr, "Error: could not connect to server via TLS\n");
|
||||||
|
return thrd_error;
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
client = client_create();
|
client = client_create();
|
||||||
if (!client || !client_connect(client, server_host, server_port)) {
|
if (!client || !client_connect(client, server_host, server_port)) {
|
||||||
@@ -240,6 +253,15 @@ int send_files(Config *config) {
|
|||||||
}
|
}
|
||||||
client = client_connect_ssh(config->ssh_destination, config->ssh_port);
|
client = client_connect_ssh(config->ssh_destination, config->ssh_port);
|
||||||
if (!client) return 1;
|
if (!client) return 1;
|
||||||
|
} else if (config->use_tls) {
|
||||||
|
client = client_create();
|
||||||
|
if (!client || !client_connect_tls(client, server_host, server_port,
|
||||||
|
config->tls_cert, config->tls_key,
|
||||||
|
config->tls_ca)) {
|
||||||
|
if (client) client_delete(client);
|
||||||
|
fprintf(stderr, "Error: could not connect to server via TLS\n");
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
client = client_create();
|
client = client_create();
|
||||||
if (!client || !client_connect(client, server_host, server_port)) {
|
if (!client || !client_connect(client, server_host, server_port)) {
|
||||||
|
|||||||
+93
-120
@@ -1,163 +1,72 @@
|
|||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
#include "chunk.h"
|
#include "chunk.h"
|
||||||
#include "compression.h"
|
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include "metadata.h"
|
|
||||||
#include "multiprocessing.h"
|
#include "multiprocessing.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "queue.h"
|
#include "queue.h"
|
||||||
#include "transport_tcp.h"
|
#include "transport_tcp.h"
|
||||||
|
#include "transport_tls.h"
|
||||||
#include "unistd.h"
|
#include "unistd.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include <signal.h>
|
#include <signal.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
#include <sys/stat.h>
|
|
||||||
|
|
||||||
int receive_files(Config *config, int file_descriptor) {
|
int receive_files(Config *config, int fd) {
|
||||||
Status status;
|
Status status;
|
||||||
if (!receive_status(file_descriptor, &status)) return -1;
|
if (!receive_status(fd, &status)) return -1;
|
||||||
|
|
||||||
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK) {
|
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK) {
|
||||||
if (status == STATUS_CHECK) {
|
if (status == STATUS_CHECK) {
|
||||||
char *check_path = receive_str(file_descriptor);
|
bool skipped;
|
||||||
if (check_path == NULL) { send_status(file_descriptor, STATUS_ERROR); return -1; }
|
File *file = receive_incremental_check(fd, config, &skipped);
|
||||||
unsigned long long check_size;
|
if (skipped) goto next;
|
||||||
long long check_mtime;
|
if (file == NULL && !skipped) return -1;
|
||||||
if (!receive_n_data(file_descriptor, &check_size, sizeof(check_size)) ||
|
if (config->save_to_disk)
|
||||||
!receive_n_data(file_descriptor, &check_mtime, sizeof(check_mtime))) {
|
file_save_to_disk(config->receive_root_directory, file);
|
||||||
free(check_path);
|
file_destroy(file);
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
char *full_path = path_cat(config->receive_root_directory, check_path);
|
|
||||||
struct stat st;
|
|
||||||
bool match = false;
|
|
||||||
if (full_path && stat(full_path, &st) == 0 &&
|
|
||||||
(unsigned long long)st.st_size == check_size &&
|
|
||||||
(long long)st.st_mtime == check_mtime) {
|
|
||||||
match = true;
|
|
||||||
}
|
|
||||||
free(full_path);
|
|
||||||
if (match) {
|
|
||||||
if (!send_status(file_descriptor, STATUS_OK)) { free(check_path); return -1; }
|
|
||||||
free(check_path);
|
|
||||||
} else {
|
|
||||||
if (!send_status(file_descriptor, STATUS_NEXT)) { free(check_path); return -1; }
|
|
||||||
File *file = file_create(check_path);
|
|
||||||
free(check_path);
|
|
||||||
if (file == NULL) { send_status(file_descriptor, STATUS_ERROR); return -1; }
|
|
||||||
if (config->use_metadata) {
|
|
||||||
int meta_ok = 1;
|
|
||||||
file->metadata = metadata_receive(file_descriptor, &meta_ok);
|
|
||||||
if (!meta_ok) { file_destroy(file); send_status(file_descriptor, STATUS_ERROR); return -1; }
|
|
||||||
}
|
|
||||||
Data *file_data = receive_data(file_descriptor);
|
|
||||||
if (file_data == NULL) {
|
|
||||||
file_destroy(file);
|
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
if (config->use_compression) {
|
|
||||||
Data *uncompressed = data_decompress(file_data);
|
|
||||||
data_destroy(file_data);
|
|
||||||
if (uncompressed == NULL) { file_destroy(file); send_status(file_descriptor, STATUS_ERROR); return -1; }
|
|
||||||
file_data = uncompressed;
|
|
||||||
}
|
|
||||||
data_destroy(file->data);
|
|
||||||
file->data = file_data;
|
|
||||||
if (config->save_to_disk) {
|
|
||||||
char *disk_path = path_cat(config->receive_root_directory, file->path);
|
|
||||||
if (disk_path) {
|
|
||||||
to_disk(disk_path, file->data->data, file->data->size);
|
|
||||||
file_restore_metadata(disk_path, file->metadata);
|
|
||||||
free(disk_path);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
file_destroy(file);
|
|
||||||
}
|
|
||||||
} else if (status == STATUS_CHUNK) {
|
} else if (status == STATUS_CHUNK) {
|
||||||
Data *chunk_data = receive_data(file_descriptor);
|
Chunk *chunk = receive_chunk_data(fd, config);
|
||||||
if (chunk_data == NULL) {
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to receive chunk data");
|
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
Data *data_to_process = chunk_data;
|
|
||||||
if (config->use_compression) {
|
|
||||||
data_to_process = data_decompress(chunk_data);
|
|
||||||
data_destroy(chunk_data);
|
|
||||||
if (data_to_process == NULL) {
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk");
|
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Chunk *chunk = chunk_deserialize(data_to_process, config->use_metadata);
|
|
||||||
data_destroy(data_to_process);
|
|
||||||
if (chunk == NULL) {
|
if (chunk == NULL) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to deserialize chunk, skipping");
|
send_status(fd, STATUS_ERROR);
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
|
|
||||||
for (int i = 0; i < chunk->element_count; i++) {
|
for (int i = 0; i < chunk->element_count; i++) {
|
||||||
if (config->save_to_disk) {
|
if (config->save_to_disk)
|
||||||
char *disk_path = path_cat(config->receive_root_directory, chunk->items[i]->path);
|
file_save_to_disk(config->receive_root_directory, chunk->items[i]);
|
||||||
if (disk_path) {
|
|
||||||
to_disk(disk_path, chunk->items[i]->data->data, chunk->items[i]->data->size);
|
|
||||||
file_restore_metadata(disk_path, chunk->items[i]->metadata);
|
|
||||||
free(disk_path);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
chunk_destroy(chunk);
|
chunk_destroy(chunk);
|
||||||
} else {
|
} else {
|
||||||
File *file = file_receive(config, file_descriptor);
|
File *file = file_receive(config, fd);
|
||||||
if (file == NULL) {
|
if (file == NULL) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to receive file");
|
log_message(LOG_LEVEL_ERROR, "Failed to receive file");
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
send_status(fd, STATUS_ERROR);
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
if (config->save_to_disk) {
|
if (config->save_to_disk)
|
||||||
char *disk_path = path_cat(config->receive_root_directory, file->path);
|
file_save_to_disk(config->receive_root_directory, file);
|
||||||
if (disk_path) {
|
|
||||||
to_disk(disk_path, file->data->data, file->data->size);
|
|
||||||
file_restore_metadata(disk_path, file->metadata);
|
|
||||||
free(disk_path);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
}
|
}
|
||||||
if (!receive_status(file_descriptor, &status)) {
|
next:
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
if (!receive_status(fd, &status)) {
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (status == STATUS_MANIFEST) {
|
if (status == STATUS_MANIFEST) {
|
||||||
int count;
|
if (receive_manifest(fd, config, &status) != 0) return -1;
|
||||||
if (!receive_int(file_descriptor, &count)) return -1;
|
|
||||||
ArrayList *manifest = array_list_create(free);
|
|
||||||
if (manifest) {
|
|
||||||
for (int i = 0; i < count; i++) {
|
|
||||||
char *s = receive_str(file_descriptor);
|
|
||||||
if (s) array_list_add(manifest, s);
|
|
||||||
}
|
|
||||||
fprintf(stderr, "Deleting files not in manifest...\n");
|
|
||||||
delete_extras(config->receive_root_directory, manifest);
|
|
||||||
array_list_delete(manifest);
|
|
||||||
}
|
|
||||||
if (!receive_status(file_descriptor, &status)) return -1;
|
|
||||||
}
|
}
|
||||||
if (status != STATUS_FINISHED) {
|
if (status != STATUS_FINISHED) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
|
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
send_status(fd, STATUS_ERROR);
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
send_status(file_descriptor, STATUS_OK);
|
send_status(fd, STATUS_OK);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -210,24 +119,88 @@ static void cleanup(int sig) {
|
|||||||
_exit(0);
|
_exit(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static void print_server_usage(void) {
|
||||||
|
printf("FastSync Server\n");
|
||||||
|
printf("Usage: fastsync-server [options]\n");
|
||||||
|
printf("\n");
|
||||||
|
printf("Options:\n");
|
||||||
|
printf(" --stdio Run in stdio mode (SSH transport)\n");
|
||||||
|
printf(" -p <port> TCP port (default: 8080, range: 1-65535)\n");
|
||||||
|
printf(" --tls Enable TLS encryption\n");
|
||||||
|
printf(" --cert <path> TLS certificate file (PEM)\n");
|
||||||
|
printf(" --key <path> TLS private key file (PEM)\n");
|
||||||
|
printf(" --ca <path> TLS CA certificate file (PEM)\n");
|
||||||
|
printf(" -v, --verbose Enable debug logging\n");
|
||||||
|
printf(" --help Show this help\n");
|
||||||
|
}
|
||||||
|
|
||||||
int main(int argc, char *argv[]) {
|
int main(int argc, char *argv[]) {
|
||||||
|
bool use_tls = false;
|
||||||
|
char *tls_cert = NULL;
|
||||||
|
char *tls_key = NULL;
|
||||||
|
char *tls_ca = NULL;
|
||||||
|
int port = 8080;
|
||||||
|
|
||||||
signal(SIGPIPE, SIG_IGN);
|
signal(SIGPIPE, SIG_IGN);
|
||||||
for (int i = 1; i < argc; i++) {
|
for (int i = 1; i < argc; i++) {
|
||||||
if (strcmp(argv[i], "--stdio") == 0) {
|
if (strcmp(argv[i], "--help") == 0) {
|
||||||
|
print_server_usage();
|
||||||
|
return 0;
|
||||||
|
} else if (strcmp(argv[i], "--stdio") == 0) {
|
||||||
io_set_fds(STDIN_FILENO, STDOUT_FILENO);
|
io_set_fds(STDIN_FILENO, STDOUT_FILENO);
|
||||||
handler(STDIN_FILENO);
|
handler(STDIN_FILENO);
|
||||||
return 0;
|
return 0;
|
||||||
} else if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) {
|
} else if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) {
|
||||||
set_log_level(LOG_LEVEL_DEBUG);
|
set_log_level(LOG_LEVEL_DEBUG);
|
||||||
|
} else if (strcmp(argv[i], "--tls") == 0) {
|
||||||
|
use_tls = true;
|
||||||
|
} else if (strcmp(argv[i], "--cert") == 0 && i + 1 < argc) {
|
||||||
|
tls_cert = argv[++i];
|
||||||
|
} else if (strcmp(argv[i], "--key") == 0 && i + 1 < argc) {
|
||||||
|
tls_key = argv[++i];
|
||||||
|
} else if (strcmp(argv[i], "--ca") == 0 && i + 1 < argc) {
|
||||||
|
tls_ca = argv[++i];
|
||||||
|
} else if (strcmp(argv[i], "-p") == 0 && i + 1 < argc) {
|
||||||
|
char *end;
|
||||||
|
long p = strtol(argv[++i], &end, 10);
|
||||||
|
if (*end || p <= 0 || p > 65535) {
|
||||||
|
fprintf(stderr, "Error: invalid port '%s' (must be 1-65535)\n", argv[i]);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
port = (int)p;
|
||||||
|
} else if (argv[i][0] == '-') {
|
||||||
|
fprintf(stderr, "Unknown option: %s\n", argv[i]);
|
||||||
|
print_server_usage();
|
||||||
|
return 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (tls_ca && !use_tls) {
|
||||||
|
log_message(LOG_LEVEL_WARNING, "--ca has no effect without --tls");
|
||||||
|
}
|
||||||
|
|
||||||
signal(SIGINT, cleanup);
|
signal(SIGINT, cleanup);
|
||||||
signal(SIGTERM, cleanup);
|
signal(SIGTERM, cleanup);
|
||||||
g_server = server_create(8080);
|
g_server = server_create(port);
|
||||||
if (g_server == NULL) {
|
if (g_server == NULL) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to create server");
|
log_message(LOG_LEVEL_ERROR, "Failed to create server");
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
server_listen(g_server, handler);
|
if (use_tls) {
|
||||||
|
if (!tls_cert || !tls_key) {
|
||||||
|
fprintf(stderr, "Error: --tls requires --cert and --key\n");
|
||||||
|
server_delete(&g_server);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
tls_global_init();
|
||||||
|
if (!server_create_tls(g_server, tls_cert, tls_key, tls_ca)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Failed to set up TLS");
|
||||||
|
server_delete(&g_server);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
server_listen_tls(g_server, handler);
|
||||||
|
} else {
|
||||||
|
server_listen(g_server, handler);
|
||||||
|
}
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,6 +10,7 @@
|
|||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include "metadata.h"
|
#include "metadata.h"
|
||||||
|
#include "protocol.h"
|
||||||
|
|
||||||
Chunk *chunk_create(File **items, int element_count) {
|
Chunk *chunk_create(File **items, int element_count) {
|
||||||
Chunk *chunk = (Chunk *)malloc(sizeof(Chunk));
|
Chunk *chunk = (Chunk *)malloc(sizeof(Chunk));
|
||||||
@@ -178,5 +179,27 @@ Data *chunk_compress(Chunk *chunk, int compression_level, bool use_metadata) {
|
|||||||
return compressed;
|
return compressed;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
Chunk *receive_chunk_data(int fd, Config *config) {
|
||||||
|
Data *chunk_data = receive_data(fd);
|
||||||
|
if (chunk_data == NULL) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Failed to receive chunk data");
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
Data *data_to_process = chunk_data;
|
||||||
|
if (config->use_compression) {
|
||||||
|
data_to_process = data_decompress(chunk_data);
|
||||||
|
data_destroy(chunk_data);
|
||||||
|
if (data_to_process == NULL) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk");
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Chunk *chunk = chunk_deserialize(data_to_process, config->use_metadata);
|
||||||
|
data_destroy(data_to_process);
|
||||||
|
if (chunk == NULL)
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Failed to deserialize chunk, skipping");
|
||||||
|
return chunk;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
#ifndef CHUNK_H
|
#ifndef CHUNK_H
|
||||||
#define CHUNK_H
|
#define CHUNK_H
|
||||||
|
|
||||||
|
#include "config.h"
|
||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
@@ -18,5 +19,6 @@ void chunk_destroy(void *chunk);
|
|||||||
Data *chunk_serialize(Chunk *chunk, bool use_metadata);
|
Data *chunk_serialize(Chunk *chunk, bool use_metadata);
|
||||||
Chunk *chunk_deserialize(Data *data, bool use_metadata);
|
Chunk *chunk_deserialize(Data *data, bool use_metadata);
|
||||||
Data *chunk_compress(Chunk *chunk, int compression_level, bool use_metadata);
|
Data *chunk_compress(Chunk *chunk, int compression_level, bool use_metadata);
|
||||||
|
Chunk *receive_chunk_data(int fd, Config *config);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|||||||
@@ -39,6 +39,10 @@ Config *config_create(char *version, char *send_directory,
|
|||||||
config->max_size = 0;
|
config->max_size = 0;
|
||||||
config->min_size = 0;
|
config->min_size = 0;
|
||||||
config->use_incremental = false;
|
config->use_incremental = false;
|
||||||
|
config->use_tls = false;
|
||||||
|
config->tls_cert = NULL;
|
||||||
|
config->tls_key = NULL;
|
||||||
|
config->tls_ca = NULL;
|
||||||
return config;
|
return config;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -74,6 +78,9 @@ void config_delete(Config *config) {
|
|||||||
for (int i = 0; i < config->include_count; i++)
|
for (int i = 0; i < config->include_count; i++)
|
||||||
free(config->include_patterns[i]);
|
free(config->include_patterns[i]);
|
||||||
free(config->include_patterns);
|
free(config->include_patterns);
|
||||||
|
free(config->tls_cert);
|
||||||
|
free(config->tls_key);
|
||||||
|
free(config->tls_ca);
|
||||||
free(config);
|
free(config);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,6 +156,10 @@ Config *config_receive(int file_descriptor) {
|
|||||||
config->include_count = 0;
|
config->include_count = 0;
|
||||||
config->max_size = 0;
|
config->max_size = 0;
|
||||||
config->min_size = 0;
|
config->min_size = 0;
|
||||||
|
config->use_tls = false;
|
||||||
|
config->tls_cert = NULL;
|
||||||
|
config->tls_key = NULL;
|
||||||
|
config->tls_ca = NULL;
|
||||||
if (!send_status(file_descriptor, STATUS_OK)) goto error;
|
if (!send_status(file_descriptor, STATUS_OK)) goto error;
|
||||||
return config;
|
return config;
|
||||||
|
|
||||||
|
|||||||
@@ -33,6 +33,10 @@ typedef struct Config {
|
|||||||
unsigned long long max_size;
|
unsigned long long max_size;
|
||||||
unsigned long long min_size;
|
unsigned long long min_size;
|
||||||
bool use_incremental;
|
bool use_incremental;
|
||||||
|
bool use_tls;
|
||||||
|
char *tls_cert;
|
||||||
|
char *tls_key;
|
||||||
|
char *tls_ca;
|
||||||
} Config;
|
} Config;
|
||||||
|
|
||||||
#define PROTOCOL_VERSION "1.1.0"
|
#define PROTOCOL_VERSION "1.1.0"
|
||||||
|
|||||||
+86
-51
@@ -96,7 +96,7 @@ bool file_load_data(File *file) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_send_single_calls_no_path(File *file, int file_descriptor, bool use_metadata, int compression_level) {
|
bool file_send_single_calls(File *file, int file_descriptor, bool use_metadata, int compression_level, bool send_path) {
|
||||||
Data *data_to_send = file->data;
|
Data *data_to_send = file->data;
|
||||||
Data *compressed_data = NULL;
|
Data *compressed_data = NULL;
|
||||||
if (compression_level > 0) {
|
if (compression_level > 0) {
|
||||||
@@ -107,6 +107,10 @@ bool file_send_single_calls_no_path(File *file, int file_descriptor, bool use_me
|
|||||||
}
|
}
|
||||||
data_to_send = compressed_data;
|
data_to_send = compressed_data;
|
||||||
}
|
}
|
||||||
|
if (send_path && !send_str(file_descriptor, file->path)) {
|
||||||
|
data_destroy(compressed_data);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) {
|
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) {
|
||||||
data_destroy(compressed_data);
|
data_destroy(compressed_data);
|
||||||
return false;
|
return false;
|
||||||
@@ -119,60 +123,74 @@ bool file_send_single_calls_no_path(File *file, int file_descriptor, bool use_me
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_send_sendfile_no_path(File *file, int file_descriptor, bool use_metadata) {
|
bool file_save_to_disk(const char *root_directory, File *file) {
|
||||||
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) return false;
|
char *disk_path = path_cat((char *)root_directory, file->path);
|
||||||
|
if (disk_path == NULL) return false;
|
||||||
int fd = open(file->path, O_RDONLY);
|
bool ok = to_disk(disk_path, file->data->data, file->data->size);
|
||||||
if (fd == -1) {
|
if (ok) file_restore_metadata(disk_path, file->metadata);
|
||||||
perror("Could not open file for sendfile");
|
free(disk_path);
|
||||||
return false;
|
return ok;
|
||||||
}
|
|
||||||
|
|
||||||
unsigned long long file_size = file->data->size;
|
|
||||||
if (!send_n_data(file_descriptor, &file_size, sizeof(unsigned long long))) {
|
|
||||||
close(fd);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
off_t offset = 0;
|
|
||||||
while (offset < file_size) {
|
|
||||||
ssize_t sent = sendfile(file_descriptor, fd, &offset, file_size - offset);
|
|
||||||
if (sent == -1) {
|
|
||||||
perror("sendfile failed");
|
|
||||||
close(fd);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
close(fd);
|
|
||||||
return true;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_send_single_calls(File *file, int file_descriptor, bool use_metadata, int compression_level) {
|
File *receive_incremental_check(int fd, Config *config, bool *skipped) { *skipped = false;
|
||||||
Data *data_to_send = file->data;
|
char *check_path = receive_str(fd);
|
||||||
Data *compressed_data = NULL;
|
if (check_path == NULL) { send_status(fd, STATUS_ERROR); return NULL; }
|
||||||
if (compression_level > 0) {
|
|
||||||
compressed_data = data_compress(file->data, compression_level);
|
unsigned long long check_size;
|
||||||
if (compressed_data == NULL) {
|
long long check_mtime;
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to compress file data");
|
if (!receive_n_data(fd, &check_size, sizeof(check_size)) ||
|
||||||
return false;
|
!receive_n_data(fd, &check_mtime, sizeof(check_mtime))) {
|
||||||
}
|
free(check_path);
|
||||||
data_to_send = compressed_data;
|
send_status(fd, STATUS_ERROR);
|
||||||
|
return NULL;
|
||||||
}
|
}
|
||||||
if (!send_str(file_descriptor, file->path)) {
|
|
||||||
data_destroy(compressed_data);
|
char *full_path = path_cat(config->receive_root_directory, check_path);
|
||||||
return false;
|
struct stat st;
|
||||||
|
bool match = false;
|
||||||
|
if (full_path && stat(full_path, &st) == 0 &&
|
||||||
|
(unsigned long long)st.st_size == check_size &&
|
||||||
|
(long long)st.st_mtime == check_mtime) {
|
||||||
|
match = true;
|
||||||
}
|
}
|
||||||
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) {
|
free(full_path);
|
||||||
data_destroy(compressed_data);
|
|
||||||
return false;
|
if (match) {
|
||||||
|
if (!send_status(fd, STATUS_OK)) { free(check_path); return NULL; }
|
||||||
|
free(check_path);
|
||||||
|
*skipped = true;
|
||||||
|
return NULL;
|
||||||
}
|
}
|
||||||
if (!send_data(file_descriptor, data_to_send)) {
|
|
||||||
data_destroy(compressed_data);
|
if (!send_status(fd, STATUS_NEXT)) { free(check_path); return NULL; }
|
||||||
return false;
|
|
||||||
|
File *file = file_create(check_path);
|
||||||
|
free(check_path);
|
||||||
|
if (file == NULL) { send_status(fd, STATUS_ERROR); return NULL; }
|
||||||
|
|
||||||
|
if (config->use_metadata) {
|
||||||
|
int meta_ok = 1;
|
||||||
|
file->metadata = metadata_receive(fd, &meta_ok);
|
||||||
|
if (!meta_ok) { file_destroy(file); send_status(fd, STATUS_ERROR); return NULL; }
|
||||||
}
|
}
|
||||||
data_destroy(compressed_data);
|
|
||||||
return true;
|
Data *file_data = receive_data(fd);
|
||||||
|
if (file_data == NULL) {
|
||||||
|
file_destroy(file);
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (config->use_compression) {
|
||||||
|
Data *uncompressed = data_decompress(file_data);
|
||||||
|
data_destroy(file_data);
|
||||||
|
if (uncompressed == NULL) { file_destroy(file); send_status(fd, STATUS_ERROR); return NULL; }
|
||||||
|
file_data = uncompressed;
|
||||||
|
}
|
||||||
|
|
||||||
|
data_destroy(file->data);
|
||||||
|
file->data = file_data;
|
||||||
|
return file;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool to_disk(const char *path, const void *data, unsigned long long data_size) {
|
bool to_disk(const char *path, const void *data, unsigned long long data_size) {
|
||||||
@@ -200,8 +218,8 @@ bool to_disk(const char *path, const void *data, unsigned long long data_size) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool file_send_sendfile(File *file, int file_descriptor, bool use_metadata) {
|
bool file_send_sendfile(File *file, int file_descriptor, bool use_metadata, bool send_path) {
|
||||||
if (!send_str(file_descriptor, file->path)) return false;
|
if (send_path && !send_str(file_descriptor, file->path)) return false;
|
||||||
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) return false;
|
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) return false;
|
||||||
|
|
||||||
int fd = open(file->path, O_RDONLY);
|
int fd = open(file->path, O_RDONLY);
|
||||||
@@ -277,4 +295,21 @@ size_t file_content_to_buffer(File *file) {
|
|||||||
return bytes_read;
|
return bytes_read;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int receive_manifest(int fd, Config *config, int *next_status) {
|
||||||
|
int count;
|
||||||
|
if (!receive_int(fd, &count)) return -1;
|
||||||
|
ArrayList *manifest = array_list_create(free);
|
||||||
|
if (manifest) {
|
||||||
|
for (int i = 0; i < count; i++) {
|
||||||
|
char *s = receive_str(fd);
|
||||||
|
if (s) array_list_add(manifest, s);
|
||||||
|
}
|
||||||
|
fprintf(stderr, "Deleting files not in manifest...\n");
|
||||||
|
delete_extras(config->receive_root_directory, manifest);
|
||||||
|
array_list_delete(manifest);
|
||||||
|
}
|
||||||
|
if (!receive_status(fd, next_status)) return -1;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+5
-4
@@ -24,13 +24,14 @@ File *file_create(const char *path);
|
|||||||
void file_destroy(void *item);
|
void file_destroy(void *item);
|
||||||
bool file_load_data(File *file);
|
bool file_load_data(File *file);
|
||||||
File *file_receive(Config *config, int file_descriptor);
|
File *file_receive(Config *config, int file_descriptor);
|
||||||
bool file_send_single_calls(File *file, int file_descriptor, bool use_metadata, int compression_level);
|
bool file_send_single_calls(File *file, int file_descriptor, bool use_metadata, int compression_level, bool send_path);
|
||||||
bool file_send_single_calls_no_path(File *file, int file_descriptor, bool use_metadata, int compression_level);
|
bool file_send_sendfile(File *file, int file_descriptor, bool use_metadata, bool send_path);
|
||||||
bool file_send_sendfile(File *file, int file_descriptor, bool use_metadata);
|
|
||||||
bool file_send_sendfile_no_path(File *file, int file_descriptor, bool use_metadata);
|
|
||||||
size_t file_content_to_buffer(File *file);
|
size_t file_content_to_buffer(File *file);
|
||||||
FileMetadata *file_metadata_create(struct stat *stats);
|
FileMetadata *file_metadata_create(struct stat *stats);
|
||||||
void file_metadata_destroy(void *metadata);
|
void file_metadata_destroy(void *metadata);
|
||||||
bool to_disk(const char *path, const void *data, unsigned long long data_size);
|
bool to_disk(const char *path, const void *data, unsigned long long data_size);
|
||||||
|
bool file_save_to_disk(const char *root_directory, File *file);
|
||||||
|
File *receive_incremental_check(int fd, Config *config, bool *skipped);
|
||||||
|
int receive_manifest(int fd, Config *config, int *next_status);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|||||||
@@ -1,19 +1,16 @@
|
|||||||
#include "multiprocessing.h"
|
#include "multiprocessing.h"
|
||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
#include "chunk.h"
|
#include "chunk.h"
|
||||||
#include "compression.h"
|
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
#include "metadata.h"
|
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "queue.h"
|
#include "queue.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
#include <sys/stat.h>
|
|
||||||
#include <threads.h>
|
#include <threads.h>
|
||||||
|
|
||||||
PipelineContextSender *pipeline_context_sender_create(Config *config,
|
PipelineContextSender *pipeline_context_sender_create(Config *config,
|
||||||
@@ -86,26 +83,8 @@ void pipeline_context_receiver_destroy(PipelineContextReceiver *context) {
|
|||||||
|
|
||||||
static void receive_chunk_enqueue(int file_descriptor,
|
static void receive_chunk_enqueue(int file_descriptor,
|
||||||
PipelineContextReceiver *context) {
|
PipelineContextReceiver *context) {
|
||||||
Data *chunk_data = receive_data(file_descriptor);
|
Chunk *chunk = receive_chunk_data(file_descriptor, context->config);
|
||||||
if (chunk_data == NULL) {
|
if (chunk == NULL) return;
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to receive chunk data");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
Data *data_to_process = chunk_data;
|
|
||||||
if (context->config->use_compression) {
|
|
||||||
data_to_process = data_decompress(chunk_data);
|
|
||||||
data_destroy(chunk_data);
|
|
||||||
if (data_to_process == NULL) {
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Chunk *chunk = chunk_deserialize(data_to_process, context->config->use_metadata);
|
|
||||||
data_destroy(data_to_process);
|
|
||||||
if (chunk == NULL) {
|
|
||||||
log_message(LOG_LEVEL_ERROR, "Failed to deserialize chunk, skipping");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
for (int i = 0; i < chunk->element_count; i++) {
|
for (int i = 0; i < chunk->element_count; i++) {
|
||||||
File *file = chunk->items[i];
|
File *file = chunk->items[i];
|
||||||
@@ -129,52 +108,10 @@ int receive_thread(void *pipeline_context) {
|
|||||||
if (!receive_status(file_descriptor, &status)) return thrd_error;
|
if (!receive_status(file_descriptor, &status)) return thrd_error;
|
||||||
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK) {
|
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK) {
|
||||||
if (status == STATUS_CHECK) {
|
if (status == STATUS_CHECK) {
|
||||||
char *check_path = receive_str(file_descriptor);
|
bool skipped;
|
||||||
if (check_path == NULL) { send_status(file_descriptor, STATUS_ERROR); return thrd_error; }
|
File *file = receive_incremental_check(file_descriptor, config, &skipped);
|
||||||
unsigned long long check_size;
|
if (!skipped) {
|
||||||
long long check_mtime;
|
if (file == NULL) return thrd_error;
|
||||||
if (!receive_n_data(file_descriptor, &check_size, sizeof(check_size)) ||
|
|
||||||
!receive_n_data(file_descriptor, &check_mtime, sizeof(check_mtime))) {
|
|
||||||
free(check_path);
|
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
|
||||||
return thrd_error;
|
|
||||||
}
|
|
||||||
char *full_path = path_cat(config->receive_root_directory, check_path);
|
|
||||||
struct stat st;
|
|
||||||
bool match = false;
|
|
||||||
if (full_path && stat(full_path, &st) == 0 &&
|
|
||||||
(unsigned long long)st.st_size == check_size &&
|
|
||||||
(long long)st.st_mtime == check_mtime) {
|
|
||||||
match = true;
|
|
||||||
}
|
|
||||||
free(full_path);
|
|
||||||
if (match) {
|
|
||||||
if (!send_status(file_descriptor, STATUS_OK)) { free(check_path); return thrd_error; }
|
|
||||||
free(check_path);
|
|
||||||
} else {
|
|
||||||
if (!send_status(file_descriptor, STATUS_NEXT)) { free(check_path); return thrd_error; }
|
|
||||||
File *file = file_create(check_path);
|
|
||||||
free(check_path);
|
|
||||||
if (file == NULL) { send_status(file_descriptor, STATUS_ERROR); return thrd_error; }
|
|
||||||
if (config->use_metadata) {
|
|
||||||
int meta_ok = 1;
|
|
||||||
file->metadata = metadata_receive(file_descriptor, &meta_ok);
|
|
||||||
if (!meta_ok) { file_destroy(file); send_status(file_descriptor, STATUS_ERROR); return thrd_error; }
|
|
||||||
}
|
|
||||||
Data *file_data = receive_data(file_descriptor);
|
|
||||||
if (file_data == NULL) {
|
|
||||||
file_destroy(file);
|
|
||||||
send_status(file_descriptor, STATUS_ERROR);
|
|
||||||
return thrd_error;
|
|
||||||
}
|
|
||||||
if (config->use_compression) {
|
|
||||||
Data *uncompressed = data_decompress(file_data);
|
|
||||||
data_destroy(file_data);
|
|
||||||
if (uncompressed == NULL) { file_destroy(file); send_status(file_descriptor, STATUS_ERROR); return thrd_error; }
|
|
||||||
file_data = uncompressed;
|
|
||||||
}
|
|
||||||
data_destroy(file->data);
|
|
||||||
file->data = file_data;
|
|
||||||
queue_enqueue_multithreaded(context->queue, file, &context->mutex,
|
queue_enqueue_multithreaded(context->queue, file, &context->mutex,
|
||||||
&context->condition_not_empty,
|
&context->condition_not_empty,
|
||||||
&context->condition_not_full);
|
&context->condition_not_full);
|
||||||
@@ -194,22 +131,7 @@ int receive_thread(void *pipeline_context) {
|
|||||||
if (!receive_status(file_descriptor, &status)) return thrd_error;
|
if (!receive_status(file_descriptor, &status)) return thrd_error;
|
||||||
}
|
}
|
||||||
if (status == STATUS_MANIFEST) {
|
if (status == STATUS_MANIFEST) {
|
||||||
int count;
|
if (receive_manifest(file_descriptor, config, &status) != 0) return thrd_error;
|
||||||
if (!receive_int(file_descriptor, &count)) return thrd_error;
|
|
||||||
ArrayList *manifest = array_list_create(free);
|
|
||||||
if (manifest) {
|
|
||||||
for (int i = 0; i < count; i++) {
|
|
||||||
char *s = receive_str(file_descriptor);
|
|
||||||
if (s) {
|
|
||||||
array_list_add(manifest, s);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
delete_extras(context->config->receive_root_directory, manifest);
|
|
||||||
for (int i = 0; i < manifest->size; i++)
|
|
||||||
free(manifest->items[i]);
|
|
||||||
array_list_delete(manifest);
|
|
||||||
}
|
|
||||||
if (!receive_status(file_descriptor, &status)) return thrd_error;
|
|
||||||
}
|
}
|
||||||
mtx_lock(&context->mutex);
|
mtx_lock(&context->mutex);
|
||||||
context->receiver_done = true;
|
context->receiver_done = true;
|
||||||
@@ -234,14 +156,8 @@ int write_thread(void *pipeline_context) {
|
|||||||
free(root_directory);
|
free(root_directory);
|
||||||
return thrd_success;
|
return thrd_success;
|
||||||
}
|
}
|
||||||
if (save_to_disk) {
|
if (save_to_disk)
|
||||||
char *disk_path = path_cat(root_directory, file->path);
|
file_save_to_disk(root_directory, file);
|
||||||
if (disk_path) {
|
|
||||||
to_disk(disk_path, file->data->data, file->data->size);
|
|
||||||
file_restore_metadata(disk_path, file->metadata);
|
|
||||||
free(disk_path);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+63
-4
@@ -1,18 +1,65 @@
|
|||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
|
#include <errno.h>
|
||||||
|
#include <openssl/ssl.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
|
#include <time.h>
|
||||||
#include <unistd.h>
|
#include <unistd.h>
|
||||||
|
|
||||||
static __thread int io_read_fd = -1;
|
static __thread int io_read_fd = -1;
|
||||||
static __thread int io_write_fd = -1;
|
static __thread int io_write_fd = -1;
|
||||||
|
static SSL *io_ssl = NULL;
|
||||||
|
|
||||||
|
static unsigned long long io_bwlimit = 0;
|
||||||
|
static long long bw_tokens = 0;
|
||||||
|
static struct timespec bw_last_refill = {0, 0};
|
||||||
|
|
||||||
void io_set_fds(int read_fd, int write_fd) {
|
void io_set_fds(int read_fd, int write_fd) {
|
||||||
io_read_fd = read_fd;
|
io_read_fd = read_fd;
|
||||||
io_write_fd = write_fd;
|
io_write_fd = write_fd;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void io_set_bwlimit(unsigned long long bytes_per_sec) {
|
||||||
|
io_bwlimit = bytes_per_sec;
|
||||||
|
bw_tokens = (long long)io_bwlimit;
|
||||||
|
clock_gettime(CLOCK_MONOTONIC, &bw_last_refill);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void bw_throttle(size_t bytes_written) {
|
||||||
|
if (io_bwlimit == 0) return;
|
||||||
|
|
||||||
|
struct timespec now;
|
||||||
|
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||||
|
|
||||||
|
long long elapsed_ns = (now.tv_sec - bw_last_refill.tv_sec) * 1000000000LL +
|
||||||
|
(now.tv_nsec - bw_last_refill.tv_nsec);
|
||||||
|
bw_last_refill = now;
|
||||||
|
|
||||||
|
long long tokens_to_add = (long long)((double)io_bwlimit * elapsed_ns / 1000000000.0);
|
||||||
|
bw_tokens += tokens_to_add;
|
||||||
|
if (bw_tokens > (long long)io_bwlimit)
|
||||||
|
bw_tokens = (long long)io_bwlimit;
|
||||||
|
|
||||||
|
bw_tokens -= (long long)bytes_written;
|
||||||
|
|
||||||
|
if (bw_tokens < 0) {
|
||||||
|
long long deficit_ns = (long long)((double)(-bw_tokens) / io_bwlimit * 1000000000.0);
|
||||||
|
struct timespec sleep_time, remaining;
|
||||||
|
sleep_time.tv_sec = deficit_ns / 1000000000LL;
|
||||||
|
sleep_time.tv_nsec = deficit_ns % 1000000000LL;
|
||||||
|
while (nanosleep(&sleep_time, &remaining) < 0 && errno == EINTR)
|
||||||
|
sleep_time = remaining;
|
||||||
|
bw_tokens = 0;
|
||||||
|
clock_gettime(CLOCK_MONOTONIC, &bw_last_refill);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void io_set_ssl(SSL *ssl) {
|
||||||
|
io_ssl = ssl;
|
||||||
|
}
|
||||||
|
|
||||||
static int io_fd(int dir_fd, int file_descriptor) {
|
static int io_fd(int dir_fd, int file_descriptor) {
|
||||||
return (dir_fd != -1) ? dir_fd : file_descriptor;
|
return (dir_fd != -1) ? dir_fd : file_descriptor;
|
||||||
}
|
}
|
||||||
@@ -22,12 +69,19 @@ bool send_n_data(int file_descriptor, void *data, size_t data_size) {
|
|||||||
int fd = io_fd(io_write_fd, file_descriptor);
|
int fd = io_fd(io_write_fd, file_descriptor);
|
||||||
ssize_t total_bytes_send = 0;
|
ssize_t total_bytes_send = 0;
|
||||||
while (total_bytes_send < data_size) {
|
while (total_bytes_send < data_size) {
|
||||||
ssize_t bytes_send =
|
size_t chunk = data_size - total_bytes_send;
|
||||||
write(fd, (char *)data + total_bytes_send, data_size - total_bytes_send);
|
if (io_bwlimit > 0 && chunk > 65536)
|
||||||
|
chunk = 65536;
|
||||||
|
ssize_t bytes_send;
|
||||||
|
if (io_ssl)
|
||||||
|
bytes_send = SSL_write(io_ssl, (char *)data + total_bytes_send, chunk);
|
||||||
|
else
|
||||||
|
bytes_send = write(fd, (char *)data + total_bytes_send, chunk);
|
||||||
if (bytes_send <= 0) {
|
if (bytes_send <= 0) {
|
||||||
log_message(LOG_LEVEL_ERROR, "Could not send data");
|
log_message(LOG_LEVEL_ERROR, "Could not send data");
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
bw_throttle((size_t)bytes_send);
|
||||||
total_bytes_send += bytes_send;
|
total_bytes_send += bytes_send;
|
||||||
}
|
}
|
||||||
log_message(LOG_LEVEL_DEBUG, " Send n Data: %zu", total_bytes_send);
|
log_message(LOG_LEVEL_DEBUG, " Send n Data: %zu", total_bytes_send);
|
||||||
@@ -39,8 +93,13 @@ bool receive_n_data(int file_descriptor, void *data, size_t data_size) {
|
|||||||
int fd = io_fd(io_read_fd, file_descriptor);
|
int fd = io_fd(io_read_fd, file_descriptor);
|
||||||
size_t total_bytes_received = 0;
|
size_t total_bytes_received = 0;
|
||||||
while (total_bytes_received < data_size) {
|
while (total_bytes_received < data_size) {
|
||||||
ssize_t bytes_received =
|
ssize_t bytes_received;
|
||||||
read(fd, (char *)data + total_bytes_received, data_size - total_bytes_received);
|
if (io_ssl)
|
||||||
|
bytes_received = SSL_read(io_ssl, (char *)data + total_bytes_received,
|
||||||
|
data_size - total_bytes_received);
|
||||||
|
else
|
||||||
|
bytes_received = read(fd, (char *)data + total_bytes_received,
|
||||||
|
data_size - total_bytes_received);
|
||||||
if (bytes_received <= 0) {
|
if (bytes_received <= 0) {
|
||||||
if (bytes_received == 0)
|
if (bytes_received == 0)
|
||||||
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
|
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
|
||||||
|
|||||||
@@ -5,10 +5,15 @@
|
|||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
#include <stddef.h>
|
#include <stddef.h>
|
||||||
|
|
||||||
|
typedef struct ssl_st SSL;
|
||||||
|
|
||||||
typedef int Status;
|
typedef int Status;
|
||||||
enum NET_STATUS { STATUS_OK, STATUS_ERROR, STATUS_FINISHED, STATUS_NEXT, STATUS_CHUNK, STATUS_MANIFEST, STATUS_CHECK };
|
enum NET_STATUS { STATUS_OK, STATUS_ERROR, STATUS_FINISHED, STATUS_NEXT, STATUS_CHUNK, STATUS_MANIFEST, STATUS_CHECK };
|
||||||
|
|
||||||
void io_set_fds(int read_fd, int write_fd);
|
void io_set_fds(int read_fd, int write_fd);
|
||||||
|
void io_set_bwlimit(unsigned long long bytes_per_sec);
|
||||||
|
typedef struct ssl_st SSL;
|
||||||
|
void io_set_ssl(SSL *ssl);
|
||||||
bool send_n_data(int file_descriptor, void *data, size_t data_size);
|
bool send_n_data(int file_descriptor, void *data, size_t data_size);
|
||||||
bool receive_n_data(int file_descriptor, void *data, size_t data_size);
|
bool receive_n_data(int file_descriptor, void *data, size_t data_size);
|
||||||
|
|
||||||
|
|||||||
@@ -143,5 +143,7 @@ Client *client_connect_ssh(char *destination, int port) {
|
|||||||
client->address.sin_family = AF_UNIX;
|
client->address.sin_family = AF_UNIX;
|
||||||
client->address_length = 0;
|
client->address_length = 0;
|
||||||
client->ssh_child_pid = pid;
|
client->ssh_child_pid = pid;
|
||||||
|
client->ssl = NULL;
|
||||||
|
client->ssl_ctx = NULL;
|
||||||
return client;
|
return client;
|
||||||
}
|
}
|
||||||
|
|||||||
+50
-16
@@ -1,6 +1,8 @@
|
|||||||
#include "transport_tcp.h"
|
#include "transport_tcp.h"
|
||||||
#include "log.h"
|
#include "log.h"
|
||||||
|
#include "protocol.h"
|
||||||
#include <arpa/inet.h>
|
#include <arpa/inet.h>
|
||||||
|
#include <openssl/ssl.h>
|
||||||
#include <signal.h>
|
#include <signal.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
@@ -36,6 +38,7 @@ Server *server_create(int port) {
|
|||||||
server->address.sin_addr.s_addr = INADDR_ANY;
|
server->address.sin_addr.s_addr = INADDR_ANY;
|
||||||
server->address.sin_port = htons(port);
|
server->address.sin_port = htons(port);
|
||||||
server->address_length = sizeof(server->address);
|
server->address_length = sizeof(server->address);
|
||||||
|
server->ssl_ctx = NULL;
|
||||||
|
|
||||||
if (bind(server->file_descriptor, (struct sockaddr *)&server->address,
|
if (bind(server->file_descriptor, (struct sockaddr *)&server->address,
|
||||||
server->address_length) < 0) {
|
server->address_length) < 0) {
|
||||||
@@ -51,43 +54,63 @@ Server *server_create(int port) {
|
|||||||
void server_delete(Server **server) {
|
void server_delete(Server **server) {
|
||||||
if (server == NULL || *server == NULL) return;
|
if (server == NULL || *server == NULL) return;
|
||||||
close((*server)->file_descriptor);
|
close((*server)->file_descriptor);
|
||||||
|
if ((*server)->ssl_ctx) {
|
||||||
|
SSL_CTX_free((*server)->ssl_ctx);
|
||||||
|
(*server)->ssl_ctx = NULL;
|
||||||
|
}
|
||||||
free(*server);
|
free(*server);
|
||||||
*server = NULL;
|
*server = NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool server_listen(Server *server, void (*handler)(int file_descriptor)) {
|
static void accept_loop(Server *server, void (*child_fn)(int, void *),
|
||||||
log_message(LOG_LEVEL_INFO, "Start Listening on Port: %d",
|
void *child_ctx, const char *log_fmt) {
|
||||||
server->address.sin_port);
|
|
||||||
if (listen(server->file_descriptor, SOMAXCONN) < 0) {
|
if (listen(server->file_descriptor, SOMAXCONN) < 0) {
|
||||||
perror("Could not listen on port!");
|
perror("Could not listen on port!");
|
||||||
return false;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
signal(SIGCHLD, SIG_IGN);
|
signal(SIGCHLD, SIG_IGN);
|
||||||
|
|
||||||
while (1) {
|
while (1) {
|
||||||
struct sockaddr_in client_addr;
|
struct sockaddr_in client_addr;
|
||||||
socklen_t client_len = sizeof(client_addr);
|
socklen_t client_len = sizeof(client_addr);
|
||||||
int file_descriptor =
|
int fd = accept(server->file_descriptor, (struct sockaddr *)&client_addr,
|
||||||
accept(server->file_descriptor, (struct sockaddr *)&client_addr,
|
&client_len);
|
||||||
&client_len);
|
if (fd < 0) {
|
||||||
if (file_descriptor < 0) {
|
|
||||||
perror("Could not accept the connection");
|
perror("Could not accept the connection");
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
log_message(LOG_LEVEL_INFO, "Received Connection");
|
log_message(LOG_LEVEL_INFO, "%s", log_fmt);
|
||||||
pid_t pid = fork();
|
pid_t pid = fork();
|
||||||
if (pid == 0) {
|
if (pid == 0) {
|
||||||
close(server->file_descriptor);
|
close(server->file_descriptor);
|
||||||
handler(file_descriptor);
|
child_fn(fd, child_ctx);
|
||||||
close(file_descriptor);
|
close(fd);
|
||||||
_exit(0);
|
_exit(0);
|
||||||
}
|
}
|
||||||
close(file_descriptor);
|
close(fd);
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct plain_ctx { void (*handler)(int); };
|
||||||
|
|
||||||
|
static void plain_child_fn(int fd, void *ctx) {
|
||||||
|
((struct plain_ctx *)ctx)->handler(fd);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool server_listen(Server *server, void (*handler)(int file_descriptor)) {
|
||||||
|
log_message(LOG_LEVEL_INFO, "Start Listening on Port: %d",
|
||||||
|
ntohs(server->address.sin_port));
|
||||||
|
struct plain_ctx ctx = {handler};
|
||||||
|
accept_loop(server, plain_child_fn, &ctx, "Received Connection");
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void server_accept_loop(Server *server, void (*child_fn)(int, void *),
|
||||||
|
void *child_ctx, const char *log_fmt) {
|
||||||
|
log_message(LOG_LEVEL_INFO, "Start TLS Listening on Port: %d",
|
||||||
|
ntohs(server->address.sin_port));
|
||||||
|
accept_loop(server, child_fn, child_ctx, log_fmt);
|
||||||
|
}
|
||||||
|
|
||||||
Client *client_create() {
|
Client *client_create() {
|
||||||
int file_descriptor = socket(AF_INET, SOCK_STREAM, 0);
|
int file_descriptor = socket(AF_INET, SOCK_STREAM, 0);
|
||||||
if (file_descriptor < 0) {
|
if (file_descriptor < 0) {
|
||||||
@@ -104,6 +127,8 @@ Client *client_create() {
|
|||||||
client->address.sin_family = AF_INET;
|
client->address.sin_family = AF_INET;
|
||||||
client->address_length = sizeof(client->address);
|
client->address_length = sizeof(client->address);
|
||||||
client->ssh_child_pid = -1;
|
client->ssh_child_pid = -1;
|
||||||
|
client->ssl = NULL;
|
||||||
|
client->ssl_ctx = NULL;
|
||||||
return client;
|
return client;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -124,6 +149,12 @@ bool client_connect(Client *client, char *host, int port) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
void client_disconnect(Client *client) {
|
void client_disconnect(Client *client) {
|
||||||
|
if (client->ssl) {
|
||||||
|
SSL_shutdown(client->ssl);
|
||||||
|
SSL_free(client->ssl);
|
||||||
|
client->ssl = NULL;
|
||||||
|
io_set_ssl(NULL);
|
||||||
|
}
|
||||||
close(client->file_descriptor);
|
close(client->file_descriptor);
|
||||||
if (client->ssh_child_pid > 0) {
|
if (client->ssh_child_pid > 0) {
|
||||||
int status;
|
int status;
|
||||||
@@ -133,7 +164,10 @@ void client_disconnect(Client *client) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
void client_delete(Client *client) {
|
void client_delete(Client *client) {
|
||||||
if (client == NULL)
|
if (client == NULL) return;
|
||||||
return;
|
if (client->ssl_ctx) {
|
||||||
|
SSL_CTX_free(client->ssl_ctx);
|
||||||
|
client->ssl_ctx = NULL;
|
||||||
|
}
|
||||||
free(client);
|
free(client);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ typedef struct Server {
|
|||||||
struct sockaddr_in address;
|
struct sockaddr_in address;
|
||||||
unsigned int address_length;
|
unsigned int address_length;
|
||||||
int file_descriptor;
|
int file_descriptor;
|
||||||
|
void *ssl_ctx;
|
||||||
} Server;
|
} Server;
|
||||||
|
|
||||||
typedef struct Client {
|
typedef struct Client {
|
||||||
@@ -16,10 +17,14 @@ typedef struct Client {
|
|||||||
unsigned int address_length;
|
unsigned int address_length;
|
||||||
int file_descriptor;
|
int file_descriptor;
|
||||||
pid_t ssh_child_pid;
|
pid_t ssh_child_pid;
|
||||||
|
void *ssl;
|
||||||
|
void *ssl_ctx;
|
||||||
} Client;
|
} Client;
|
||||||
|
|
||||||
Server *server_create(int port);
|
Server *server_create(int port);
|
||||||
bool server_listen(Server *server, void (*handler)(int file_descriptor));
|
bool server_listen(Server *server, void (*handler)(int file_descriptor));
|
||||||
|
void server_accept_loop(Server *server, void (*child_fn)(int, void *),
|
||||||
|
void *child_ctx, const char *log_fmt);
|
||||||
void server_delete(Server **server);
|
void server_delete(Server **server);
|
||||||
Client *client_create();
|
Client *client_create();
|
||||||
bool client_connect(Client *client, char *host, int port);
|
bool client_connect(Client *client, char *host, int port);
|
||||||
|
|||||||
@@ -0,0 +1,162 @@
|
|||||||
|
#include "transport_tls.h"
|
||||||
|
#include "log.h"
|
||||||
|
#include "protocol.h"
|
||||||
|
#include "transport_tcp.h"
|
||||||
|
#include <arpa/inet.h>
|
||||||
|
#include <openssl/err.h>
|
||||||
|
#include <openssl/ssl.h>
|
||||||
|
#include <signal.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <string.h>
|
||||||
|
#include <sys/socket.h>
|
||||||
|
#include <sys/wait.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
bool tls_global_init(void) {
|
||||||
|
#if OPENSSL_VERSION_NUMBER < 0x10100000L
|
||||||
|
SSL_library_init();
|
||||||
|
OpenSSL_add_all_algorithms();
|
||||||
|
SSL_load_error_strings();
|
||||||
|
#endif
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void log_ssl_errors(void) {
|
||||||
|
unsigned long err;
|
||||||
|
char buf[256];
|
||||||
|
while ((err = ERR_get_error()) != 0) {
|
||||||
|
ERR_error_string_n(err, buf, sizeof(buf));
|
||||||
|
log_message(LOG_LEVEL_ERROR, "SSL error: %s", buf);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static SSL_CTX *create_ssl_ctx(bool is_server, const char *cert,
|
||||||
|
const char *key, const char *ca_path) {
|
||||||
|
const SSL_METHOD *method =
|
||||||
|
is_server ? TLS_server_method() : TLS_client_method();
|
||||||
|
SSL_CTX *ctx = SSL_CTX_new(method);
|
||||||
|
if (!ctx) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Unable to create SSL context");
|
||||||
|
log_ssl_errors();
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION);
|
||||||
|
|
||||||
|
if (cert && key) {
|
||||||
|
if (SSL_CTX_use_certificate_file(ctx, cert, SSL_FILETYPE_PEM) <= 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Failed to load certificate: %s", cert);
|
||||||
|
log_ssl_errors();
|
||||||
|
SSL_CTX_free(ctx);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
if (SSL_CTX_use_PrivateKey_file(ctx, key, SSL_FILETYPE_PEM) <= 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Failed to load private key: %s", key);
|
||||||
|
log_ssl_errors();
|
||||||
|
SSL_CTX_free(ctx);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
if (!SSL_CTX_check_private_key(ctx)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"Private key does not match certificate");
|
||||||
|
SSL_CTX_free(ctx);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (ca_path) {
|
||||||
|
if (!SSL_CTX_load_verify_locations(ctx, ca_path, NULL)) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Failed to load CA: %s", ca_path);
|
||||||
|
log_ssl_errors();
|
||||||
|
SSL_CTX_free(ctx);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
|
||||||
|
SSL_CTX_set_verify_depth(ctx, 4);
|
||||||
|
}
|
||||||
|
|
||||||
|
return ctx;
|
||||||
|
}
|
||||||
|
|
||||||
|
static SSL *wrap_fd_with_ssl(int fd, SSL_CTX *ctx, bool is_server) {
|
||||||
|
SSL *ssl = SSL_new(ctx);
|
||||||
|
if (!ssl) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "Failed to create SSL object");
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
SSL_set_fd(ssl, fd);
|
||||||
|
int ret;
|
||||||
|
if (is_server)
|
||||||
|
ret = SSL_accept(ssl);
|
||||||
|
else
|
||||||
|
ret = SSL_connect(ssl);
|
||||||
|
|
||||||
|
if (ret <= 0) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "SSL %s failed",
|
||||||
|
is_server ? "accept" : "connect");
|
||||||
|
log_ssl_errors();
|
||||||
|
SSL_free(ssl);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
return ssl;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool server_create_tls(Server *server, const char *cert_path,
|
||||||
|
const char *key_path, const char *ca_path) {
|
||||||
|
SSL_CTX *ctx = create_ssl_ctx(true, cert_path, key_path, ca_path);
|
||||||
|
if (!ctx) return false;
|
||||||
|
server->ssl_ctx = ctx;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct tls_child_ctx {
|
||||||
|
void (*handler)(int);
|
||||||
|
SSL_CTX *ssl_ctx;
|
||||||
|
};
|
||||||
|
|
||||||
|
static void tls_child_fn(int fd, void *arg) {
|
||||||
|
struct tls_child_ctx *ctx = (struct tls_child_ctx *)arg;
|
||||||
|
SSL *ssl = wrap_fd_with_ssl(fd, ctx->ssl_ctx, true);
|
||||||
|
if (!ssl) return;
|
||||||
|
io_set_ssl(ssl);
|
||||||
|
ctx->handler(fd);
|
||||||
|
SSL_shutdown(ssl);
|
||||||
|
SSL_free(ssl);
|
||||||
|
io_set_ssl(NULL);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool server_listen_tls(Server *server, void (*handler)(int file_descriptor)) {
|
||||||
|
struct tls_child_ctx ctx = {handler, (SSL_CTX *)server->ssl_ctx};
|
||||||
|
server_accept_loop(server, tls_child_fn, &ctx, "Received TLS Connection");
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool client_connect_tls(Client *client, char *host, int port,
|
||||||
|
const char *cert_path, const char *key_path,
|
||||||
|
const char *ca_path) {
|
||||||
|
client->address.sin_port = htons(port);
|
||||||
|
if (inet_pton(AF_INET, host, &client->address.sin_addr) <= 0) {
|
||||||
|
perror("Could not convert host address!");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (connect(client->file_descriptor, (struct sockaddr *)&client->address,
|
||||||
|
client->address_length) < 0) {
|
||||||
|
perror("Could not connect to Server!");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
SSL_CTX *ctx = create_ssl_ctx(false, cert_path, key_path, ca_path);
|
||||||
|
if (!ctx) return false;
|
||||||
|
client->ssl_ctx = ctx;
|
||||||
|
|
||||||
|
SSL *ssl = wrap_fd_with_ssl(client->file_descriptor, ctx, false);
|
||||||
|
if (!ssl) {
|
||||||
|
SSL_CTX_free(ctx);
|
||||||
|
client->ssl_ctx = NULL;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
client->ssl = ssl;
|
||||||
|
io_set_ssl(ssl);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
#ifndef TRANSPORT_TLS_H
|
||||||
|
#define TRANSPORT_TLS_H
|
||||||
|
|
||||||
|
#include "transport_tcp.h"
|
||||||
|
#include <stdbool.h>
|
||||||
|
|
||||||
|
bool tls_global_init(void);
|
||||||
|
|
||||||
|
bool server_create_tls(Server *server, const char *cert_path,
|
||||||
|
const char *key_path, const char *ca_path);
|
||||||
|
bool server_listen_tls(Server *server, void (*handler)(int file_descriptor));
|
||||||
|
bool client_connect_tls(Client *client, char *host, int port,
|
||||||
|
const char *cert_path, const char *key_path,
|
||||||
|
const char *ca_path);
|
||||||
|
|
||||||
|
#endif
|
||||||
@@ -50,7 +50,7 @@ CLIENT_CMD_PREFIX = [
|
|||||||
|
|
||||||
BASE_CLIENT_FLAGS = ["--save-to-disk"]
|
BASE_CLIENT_FLAGS = ["--save-to-disk"]
|
||||||
|
|
||||||
TEST_CASES = [
|
TEST_CASES_FULL = [
|
||||||
{"name": "Standard", "flags": []},
|
{"name": "Standard", "flags": []},
|
||||||
{"name": "Posix Args (no flags)", "flags": [], "posix": True},
|
{"name": "Posix Args (no flags)", "flags": [], "posix": True},
|
||||||
{"name": "Standard (no metadata)", "flags": [], "use_metadata": False},
|
{"name": "Standard (no metadata)", "flags": [], "use_metadata": False},
|
||||||
@@ -68,7 +68,14 @@ TEST_CASES = [
|
|||||||
{"name": "Sendfile + Multithreading (-f -m)", "flags": ["-f", "-m"]},
|
{"name": "Sendfile + Multithreading (-f -m)", "flags": ["-f", "-m"]},
|
||||||
]
|
]
|
||||||
|
|
||||||
SSH_CASES = [
|
TEST_CASES_LIGHT = [
|
||||||
|
{"name": "Standard", "flags": []},
|
||||||
|
{"name": "Compression (-c)", "flags": ["-c"]},
|
||||||
|
{"name": "Chunk Serialization (-s)", "flags": ["-s"]},
|
||||||
|
{"name": "Multithreading + Compression + Chunk Serialization (-m -c -s)", "flags": ["-m", "-c", "-s"]},
|
||||||
|
]
|
||||||
|
|
||||||
|
SSH_CASES_FULL = [
|
||||||
{"name": "SSH (localhost)", "flags": []},
|
{"name": "SSH (localhost)", "flags": []},
|
||||||
{"name": "SSH Multithreading (-m)", "flags": ["-m"]},
|
{"name": "SSH Multithreading (-m)", "flags": ["-m"]},
|
||||||
{"name": "SSH Compression (-c)", "flags": ["-c"]},
|
{"name": "SSH Compression (-c)", "flags": ["-c"]},
|
||||||
@@ -79,11 +86,17 @@ SSH_CASES = [
|
|||||||
{"name": "SSH Multithreading + Compression + Chunk Serialization (-m -c -s)", "flags": ["-m", "-c", "-s"]},
|
{"name": "SSH Multithreading + Compression + Chunk Serialization (-m -c -s)", "flags": ["-m", "-c", "-s"]},
|
||||||
]
|
]
|
||||||
|
|
||||||
RSYNC_CASES = [
|
SSH_CASES_LIGHT = [
|
||||||
|
{"name": "SSH (localhost)", "flags": []},
|
||||||
|
]
|
||||||
|
|
||||||
|
RSYNC_CASES_FULL = [
|
||||||
{"name": "rsync (archive)", "args": ["-aH"]},
|
{"name": "rsync (archive)", "args": ["-aH"]},
|
||||||
{"name": "rsync (archive + compress)", "args": ["-aHz"]},
|
{"name": "rsync (archive + compress)", "args": ["-aHz"]},
|
||||||
]
|
]
|
||||||
|
|
||||||
|
RSYNC_CASES_LIGHT = []
|
||||||
|
|
||||||
|
|
||||||
def netem_apply(profile):
|
def netem_apply(profile):
|
||||||
params = NETWORK_PROFILES[profile]
|
params = NETWORK_PROFILES[profile]
|
||||||
@@ -119,12 +132,12 @@ def find_free_port():
|
|||||||
return s.getsockname()[1]
|
return s.getsockname()[1]
|
||||||
|
|
||||||
|
|
||||||
def generate_test_files(source_dir):
|
def generate_test_files(source_dir, full=False):
|
||||||
if os.path.exists(source_dir):
|
if os.path.exists(source_dir):
|
||||||
shutil.rmtree(source_dir)
|
shutil.rmtree(source_dir)
|
||||||
os.makedirs(source_dir)
|
os.makedirs(source_dir)
|
||||||
|
|
||||||
target_total = 25 * 1024 * 1024
|
target_total = 25 * 1024 * 1024 if full else 0
|
||||||
written = 0
|
written = 0
|
||||||
|
|
||||||
files = {
|
files = {
|
||||||
@@ -141,14 +154,15 @@ def generate_test_files(source_dir):
|
|||||||
f.write(content)
|
f.write(content)
|
||||||
written += len(content)
|
written += len(content)
|
||||||
|
|
||||||
os.makedirs(os.path.join(source_dir, "bulk"), exist_ok=True)
|
if full:
|
||||||
i = 0
|
os.makedirs(os.path.join(source_dir, "bulk"), exist_ok=True)
|
||||||
while written < target_total:
|
i = 0
|
||||||
chunk_size = min(5 * 1024 * 1024, target_total - written)
|
while written < target_total:
|
||||||
with open(os.path.join(source_dir, f"bulk/file_{i}.dat"), "wb") as f:
|
chunk_size = min(5 * 1024 * 1024, target_total - written)
|
||||||
f.write(random.randbytes(chunk_size))
|
with open(os.path.join(source_dir, f"bulk/file_{i}.dat"), "wb") as f:
|
||||||
written += chunk_size
|
f.write(random.randbytes(chunk_size))
|
||||||
i += 1
|
written += chunk_size
|
||||||
|
i += 1
|
||||||
|
|
||||||
total_mb = written / (1024 * 1024)
|
total_mb = written / (1024 * 1024)
|
||||||
small_bytes = sum(len(c) for c in files.values())
|
small_bytes = sum(len(c) for c in files.values())
|
||||||
@@ -254,19 +268,24 @@ def print_profile_header(profile_name):
|
|||||||
print(" No limits applied")
|
print(" No limits applied")
|
||||||
|
|
||||||
|
|
||||||
def run_profile(profile_name, source_dir, dest_dir):
|
def run_profile(profile_name, source_dir, dest_dir, *, full=False, test_cases=None, ssh_cases=None, rsync_cases=None):
|
||||||
print_profile_header(profile_name)
|
print_profile_header(profile_name)
|
||||||
is_limited = profile_name != "Unlimited"
|
is_limited = profile_name != "Unlimited"
|
||||||
client_prefix = CLIENT_CMD_PREFIX if is_limited else []
|
client_prefix = CLIENT_CMD_PREFIX if is_limited else []
|
||||||
|
|
||||||
|
if test_cases is None:
|
||||||
|
test_cases = TEST_CASES_FULL if full else TEST_CASES_LIGHT
|
||||||
|
if ssh_cases is None:
|
||||||
|
ssh_cases = SSH_CASES_FULL if full else SSH_CASES_LIGHT
|
||||||
|
if rsync_cases is None:
|
||||||
|
rsync_cases = RSYNC_CASES_FULL if full else RSYNC_CASES_LIGHT
|
||||||
|
|
||||||
try:
|
try:
|
||||||
if is_limited:
|
if is_limited and full:
|
||||||
netem_apply(profile_name)
|
netem_apply(profile_name)
|
||||||
else:
|
|
||||||
netem_reset()
|
|
||||||
|
|
||||||
results = []
|
results = []
|
||||||
for case in TEST_CASES:
|
for case in test_cases:
|
||||||
flags = BASE_CLIENT_FLAGS + (["-M"] if case.get("use_metadata", True) else []) + case["flags"]
|
flags = BASE_CLIENT_FLAGS + (["-M"] if case.get("use_metadata", True) else []) + case["flags"]
|
||||||
if case.get("posix"):
|
if case.get("posix"):
|
||||||
cmd = client_prefix + BASE_CLIENT_CMD + [source_dir, dest_dir] + flags
|
cmd = client_prefix + BASE_CLIENT_CMD + [source_dir, dest_dir] + flags
|
||||||
@@ -281,7 +300,7 @@ def run_profile(profile_name, source_dir, dest_dir):
|
|||||||
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
if SSH_AVAILABLE:
|
if SSH_AVAILABLE:
|
||||||
for case in SSH_CASES:
|
for case in ssh_cases:
|
||||||
flags = BASE_CLIENT_FLAGS + (["-M"] if case.get("use_metadata", True) else []) + case["flags"]
|
flags = BASE_CLIENT_FLAGS + (["-M"] if case.get("use_metadata", True) else []) + case["flags"]
|
||||||
ssh_dest = f"localhost:{dest_dir}_ssh"
|
ssh_dest = f"localhost:{dest_dir}_ssh"
|
||||||
cmd = BASE_CLIENT_CMD + [source_dir, ssh_dest] + flags
|
cmd = BASE_CLIENT_CMD + [source_dir, ssh_dest] + flags
|
||||||
@@ -293,210 +312,222 @@ def run_profile(profile_name, source_dir, dest_dir):
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
port, conf, daemon = start_rsync_daemon(source_dir)
|
if rsync_cases:
|
||||||
try:
|
port, conf, daemon = start_rsync_daemon(source_dir)
|
||||||
for case in RSYNC_CASES:
|
|
||||||
cmd = client_prefix + ["rsync"] + case["args"] + [f"rsync://localhost:{port}/source/", f"{dest_dir}/"]
|
|
||||||
print(f"\n --- {case['name']} ---\n Running: {' '.join(cmd)}")
|
|
||||||
try:
|
|
||||||
r = run_single_test(cmd, case["name"], source_dir, dest_dir, source_prefix="")
|
|
||||||
r["suite"] = profile_name
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
r = {"name": case["name"], "suite": profile_name, "status": "Timeout", "time": "N/A", "error": "Exceeded 120s"}
|
|
||||||
except Exception as e:
|
|
||||||
r = {"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)}
|
|
||||||
else:
|
|
||||||
if r["status"] != "Success" and client_prefix:
|
|
||||||
tmp = tempfile.mkdtemp()
|
|
||||||
try:
|
|
||||||
plain = subprocess.run(["rsync", "-aH", f"rsync://localhost:{port}/source/", f"{tmp}/"], capture_output=True, text=True, timeout=30)
|
|
||||||
if plain.returncode != 0:
|
|
||||||
errs = [l for l in (plain.stderr or "").split("\n") if l.strip()]
|
|
||||||
if errs:
|
|
||||||
r["error"] += f" | raw: {errs[-1][:150]}"
|
|
||||||
finally:
|
|
||||||
shutil.rmtree(tmp, ignore_errors=True)
|
|
||||||
results.append(r)
|
|
||||||
finally:
|
|
||||||
wait_proc(daemon)
|
|
||||||
try:
|
try:
|
||||||
os.unlink(conf)
|
for case in rsync_cases:
|
||||||
except Exception:
|
cmd = client_prefix + ["rsync"] + case["args"] + [f"rsync://localhost:{port}/source/", f"{dest_dir}/"]
|
||||||
pass
|
print(f"\n --- {case['name']} ---\n Running: {' '.join(cmd)}")
|
||||||
|
try:
|
||||||
|
r = run_single_test(cmd, case["name"], source_dir, dest_dir, source_prefix="")
|
||||||
|
r["suite"] = profile_name
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
r = {"name": case["name"], "suite": profile_name, "status": "Timeout", "time": "N/A", "error": "Exceeded 120s"}
|
||||||
|
except Exception as e:
|
||||||
|
r = {"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)}
|
||||||
|
else:
|
||||||
|
if r["status"] != "Success" and client_prefix:
|
||||||
|
tmp = tempfile.mkdtemp()
|
||||||
|
try:
|
||||||
|
plain = subprocess.run(["rsync", "-aH", f"rsync://localhost:{port}/source/", f"{tmp}/"], capture_output=True, text=True, timeout=30)
|
||||||
|
if plain.returncode != 0:
|
||||||
|
errs = [l for l in (plain.stderr or "").split("\n") if l.strip()]
|
||||||
|
if errs:
|
||||||
|
r["error"] += f" | raw: {errs[-1][:150]}"
|
||||||
|
finally:
|
||||||
|
shutil.rmtree(tmp, ignore_errors=True)
|
||||||
|
results.append(r)
|
||||||
|
finally:
|
||||||
|
wait_proc(daemon)
|
||||||
|
try:
|
||||||
|
os.unlink(conf)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
# Feature-specific tests for rsync-compatible flags
|
if full:
|
||||||
print("\n " + "─" * 56 + "\n Feature Tests\n " + "─" * 56)
|
# Feature-specific tests for rsync-compatible flags
|
||||||
|
print("\n " + "─" * 56 + "\n Feature Tests\n " + "─" * 56)
|
||||||
|
|
||||||
# Dry run (-n) — no server needed
|
# Dry run (-n) — no server needed
|
||||||
print("\n --- Dry run (-n) ---")
|
print("\n --- Dry run (-n) ---")
|
||||||
flags = BASE_CLIENT_FLAGS + ["-n"]
|
flags = BASE_CLIENT_FLAGS + ["-n"]
|
||||||
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
|
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
|
||||||
print(f" Running: {' '.join(cmd)}")
|
print(f" Running: {' '.join(cmd)}")
|
||||||
try:
|
try:
|
||||||
start = time.monotonic()
|
start = time.monotonic()
|
||||||
result = subprocess.run(cmd, text=True, capture_output=True)
|
result = subprocess.run(cmd, text=True, capture_output=True)
|
||||||
duration = time.monotonic() - start
|
duration = time.monotonic() - start
|
||||||
r = {"name": "Dry run (-n)", "suite": profile_name}
|
r = {"name": "Dry run (-n)", "suite": profile_name}
|
||||||
if result.returncode == 0 and "Dry run:" in result.stdout:
|
if result.returncode == 0 and "Dry run:" in result.stdout:
|
||||||
r["status"] = "Success"
|
|
||||||
r["time"] = f"{duration:.4f}s"
|
|
||||||
r["error"] = ""
|
|
||||||
else:
|
|
||||||
r["status"] = "Failed"
|
|
||||||
r["time"] = "N/A"
|
|
||||||
r["error"] = f"Exit {result.returncode}: {(result.stderr or result.stdout)[:100]}"
|
|
||||||
results.append(r)
|
|
||||||
except Exception as e:
|
|
||||||
results.append({"name": "Dry run (-n)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
|
||||||
|
|
||||||
# Archive mode (-a)
|
|
||||||
feature_flags = BASE_CLIENT_FLAGS + ["-a"]
|
|
||||||
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
|
||||||
print(f"\n --- Archive mode (-a) ---\n Running: {' '.join(cmd)}")
|
|
||||||
try:
|
|
||||||
r = run_single_test(cmd, "Archive mode (-a)", source_dir, dest_dir)
|
|
||||||
r["suite"] = profile_name
|
|
||||||
results.append(r)
|
|
||||||
except Exception as e:
|
|
||||||
results.append({"name": "Archive mode (-a)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
|
||||||
|
|
||||||
# Exclude (--exclude small.txt)
|
|
||||||
feature_flags = BASE_CLIENT_FLAGS + ["--exclude", "small.txt"]
|
|
||||||
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
|
||||||
print(f"\n --- Exclude (--exclude small.txt) ---\n Running: {' '.join(cmd)}")
|
|
||||||
try:
|
|
||||||
r = run_single_test(cmd, "Exclude (--exclude small.txt)", source_dir, dest_dir,
|
|
||||||
expected_missing=["small.txt"])
|
|
||||||
r["suite"] = profile_name
|
|
||||||
results.append(r)
|
|
||||||
except Exception as e:
|
|
||||||
results.append({"name": "Exclude (--exclude small.txt)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
|
||||||
|
|
||||||
# Progress (--progress)
|
|
||||||
feature_flags = BASE_CLIENT_FLAGS + ["--progress"]
|
|
||||||
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
|
||||||
print(f"\n --- Progress (--progress) ---\n Running: {' '.join(cmd)}")
|
|
||||||
try:
|
|
||||||
r = run_single_test(cmd, "Progress (--progress)", source_dir, dest_dir)
|
|
||||||
r["suite"] = profile_name
|
|
||||||
results.append(r)
|
|
||||||
except Exception as e:
|
|
||||||
results.append({"name": "Progress (--progress)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
|
||||||
|
|
||||||
# Incremental sync (--incremental) — first sync, then second sync should skip all
|
|
||||||
print(f"\n --- Incremental (--incremental) ---")
|
|
||||||
try:
|
|
||||||
flags = BASE_CLIENT_FLAGS + ["-M"]
|
|
||||||
srv = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
|
|
||||||
time.sleep(0.5)
|
|
||||||
first_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
|
|
||||||
r1 = subprocess.run(first_cmd, text=True, capture_output=True)
|
|
||||||
wait_proc(srv)
|
|
||||||
if r1.returncode != 0:
|
|
||||||
raise RuntimeError(f"First sync failed: {r1.stderr[:100]}")
|
|
||||||
# Second sync with --incremental should take less time
|
|
||||||
srv2 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
|
|
||||||
time.sleep(0.5)
|
|
||||||
second_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags + ["--incremental"]
|
|
||||||
start = time.monotonic()
|
|
||||||
r2 = subprocess.run(second_cmd, text=True, capture_output=True, timeout=30)
|
|
||||||
duration = time.monotonic() - start
|
|
||||||
wait_proc(srv2)
|
|
||||||
r = {"name": "Incremental (--incremental)", "suite": profile_name,
|
|
||||||
"status": "Success" if r2.returncode == 0 else "Failed",
|
|
||||||
"time": f"{duration:.4f}s" if r2.returncode == 0 else "N/A",
|
|
||||||
"error": "" if r2.returncode == 0 else f"Exit {r2.returncode}: {(r2.stderr or r2.stdout)[:60]}"}
|
|
||||||
results.append(r)
|
|
||||||
except Exception as e:
|
|
||||||
results.append({"name": "Incremental (--incremental)", "suite": profile_name,
|
|
||||||
"status": "Error", "time": "N/A", "error": str(e)})
|
|
||||||
|
|
||||||
# Chunk size (--chunk-size 5242880)
|
|
||||||
feature_flags = BASE_CLIENT_FLAGS + ["--chunk-size", "5242880"]
|
|
||||||
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
|
||||||
print(f"\n --- Chunk size (--chunk-size 5242880) ---\n Running: {' '.join(cmd)}")
|
|
||||||
try:
|
|
||||||
r = run_single_test(cmd, "Chunk size (--chunk-size 5242880)", source_dir, dest_dir)
|
|
||||||
r["suite"] = profile_name
|
|
||||||
results.append(r)
|
|
||||||
except Exception as e:
|
|
||||||
results.append({"name": "Chunk size (--chunk-size 5242880)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
|
||||||
|
|
||||||
# Delete (--delete) — pre-populate dest, add extra files, then sync with --delete
|
|
||||||
# Note: server handles one client per launch, so we restart between syncs
|
|
||||||
print(f"\n --- Delete (--delete) ---")
|
|
||||||
try:
|
|
||||||
flags = BASE_CLIENT_FLAGS + ["-M"]
|
|
||||||
# First sync (no delete) to populate dest
|
|
||||||
s1 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
|
|
||||||
time.sleep(0.5)
|
|
||||||
first_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
|
|
||||||
r1 = subprocess.run(first_cmd, text=True, capture_output=True)
|
|
||||||
wait_proc(s1)
|
|
||||||
if r1.returncode != 0:
|
|
||||||
raise RuntimeError(f"First sync failed: {r1.stderr[:100]}")
|
|
||||||
# Add extra files to received dir
|
|
||||||
received = os.path.join(dest_dir, os.path.abspath(source_dir).lstrip(os.sep))
|
|
||||||
extra_path = os.path.join(received, "extra_file.txt")
|
|
||||||
with open(extra_path, "w") as f:
|
|
||||||
f.write("should be deleted")
|
|
||||||
extra_dir = os.path.join(received, "extra_dir")
|
|
||||||
os.makedirs(extra_dir, exist_ok=True)
|
|
||||||
with open(os.path.join(extra_dir, "nested.txt"), "w") as f:
|
|
||||||
f.write("nested extra")
|
|
||||||
# Second sync with --delete (fresh server)
|
|
||||||
s2 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
|
|
||||||
time.sleep(0.5)
|
|
||||||
second_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags + ["--delete"]
|
|
||||||
start = time.monotonic()
|
|
||||||
r2 = subprocess.run(second_cmd, text=True, capture_output=True)
|
|
||||||
duration = time.monotonic() - start
|
|
||||||
wait_proc(s2)
|
|
||||||
r = {"name": "Delete (--delete)", "suite": profile_name}
|
|
||||||
if r2.returncode == 0 and not os.path.exists(extra_path) and not os.path.exists(extra_dir):
|
|
||||||
mismatches, missing = verify_transfer(source_dir, received)
|
|
||||||
if not mismatches and not missing:
|
|
||||||
r["status"] = "Success"
|
r["status"] = "Success"
|
||||||
r["time"] = f"{duration:.4f}s"
|
r["time"] = f"{duration:.4f}s"
|
||||||
r["error"] = ""
|
r["error"] = ""
|
||||||
else:
|
else:
|
||||||
r["status"] = "Failed"
|
r["status"] = "Failed"
|
||||||
r["time"] = "N/A"
|
r["time"] = "N/A"
|
||||||
r["error"] = f"post-delete verify: mismatches={len(mismatches)}, missing={len(missing)}"
|
r["error"] = f"Exit {result.returncode}: {(result.stderr or result.stdout)[:100]}"
|
||||||
else:
|
results.append(r)
|
||||||
r["status"] = "Failed"
|
except Exception as e:
|
||||||
r["time"] = "N/A"
|
results.append({"name": "Dry run (-n)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
errs = []
|
|
||||||
if r2.returncode != 0:
|
|
||||||
errs.append(f"Exit {r2.returncode}: {(r2.stderr or r2.stdout)[:60]}")
|
|
||||||
if os.path.exists(extra_path):
|
|
||||||
errs.append("extra_file.txt remains")
|
|
||||||
if os.path.exists(extra_dir):
|
|
||||||
errs.append("extra_dir remains")
|
|
||||||
r["error"] = " | ".join(errs)
|
|
||||||
results.append(r)
|
|
||||||
except Exception as e:
|
|
||||||
results.append({"name": "Delete (--delete)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
|
||||||
|
|
||||||
# SSH feature tests
|
# Archive mode (-a)
|
||||||
if SSH_AVAILABLE:
|
feature_flags = BASE_CLIENT_FLAGS + ["-a"]
|
||||||
ssh_dest = f"localhost:{dest_dir}_ssh"
|
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
||||||
ssh_feature_cases = [
|
print(f"\n --- Archive mode (-a) ---\n Running: {' '.join(cmd)}")
|
||||||
{"name": "SSH Archive (-a)", "flags": ["-a"]},
|
try:
|
||||||
{"name": "SSH Exclude (--exclude small.txt)", "flags": ["--exclude", "small.txt"], "expected_missing": ["small.txt"]},
|
r = run_single_test(cmd, "Archive mode (-a)", source_dir, dest_dir)
|
||||||
]
|
r["suite"] = profile_name
|
||||||
for case in ssh_feature_cases:
|
results.append(r)
|
||||||
flags = BASE_CLIENT_FLAGS + case["flags"]
|
except Exception as e:
|
||||||
cmd = BASE_CLIENT_CMD + [source_dir, ssh_dest] + flags
|
results.append({"name": "Archive mode (-a)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
print(f"\n --- {case['name']} ---\n Running: {' '.join(cmd)}")
|
|
||||||
try:
|
# Exclude (--exclude small.txt)
|
||||||
r = run_single_test(cmd, case["name"], source_dir, f"{dest_dir}_ssh",
|
feature_flags = BASE_CLIENT_FLAGS + ["--exclude", "small.txt"]
|
||||||
no_server=True,
|
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
||||||
expected_missing=case.get("expected_missing"))
|
print(f"\n --- Exclude (--exclude small.txt) ---\n Running: {' '.join(cmd)}")
|
||||||
r["suite"] = profile_name
|
try:
|
||||||
results.append(r)
|
r = run_single_test(cmd, "Exclude (--exclude small.txt)", source_dir, dest_dir,
|
||||||
except Exception as e:
|
expected_missing=["small.txt"])
|
||||||
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
r["suite"] = profile_name
|
||||||
|
results.append(r)
|
||||||
|
except Exception as e:
|
||||||
|
results.append({"name": "Exclude (--exclude small.txt)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
|
# Progress (--progress)
|
||||||
|
feature_flags = BASE_CLIENT_FLAGS + ["--progress"]
|
||||||
|
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
||||||
|
print(f"\n --- Progress (--progress) ---\n Running: {' '.join(cmd)}")
|
||||||
|
try:
|
||||||
|
r = run_single_test(cmd, "Progress (--progress)", source_dir, dest_dir)
|
||||||
|
r["suite"] = profile_name
|
||||||
|
results.append(r)
|
||||||
|
except Exception as e:
|
||||||
|
results.append({"name": "Progress (--progress)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
|
# Bandwidth limit (--bwlimit 10240 = 10 MB/s)
|
||||||
|
feature_flags = BASE_CLIENT_FLAGS + ["--bwlimit", "10240"]
|
||||||
|
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
||||||
|
print(f"\n --- Bandwidth limit (--bwlimit 10240 KB/s) ---\n Running: {' '.join(cmd)}")
|
||||||
|
try:
|
||||||
|
r = run_single_test(cmd, "Bandwidth limit (--bwlimit 10240)", source_dir, dest_dir)
|
||||||
|
r["suite"] = profile_name
|
||||||
|
results.append(r)
|
||||||
|
except Exception as e:
|
||||||
|
results.append({"name": "Bandwidth limit (--bwlimit 10240)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
|
# Incremental sync (--incremental) — first sync, then second sync should skip all
|
||||||
|
print(f"\n --- Incremental (--incremental) ---")
|
||||||
|
try:
|
||||||
|
flags = BASE_CLIENT_FLAGS + ["-M"]
|
||||||
|
srv = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
|
||||||
|
time.sleep(0.5)
|
||||||
|
first_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
|
||||||
|
r1 = subprocess.run(first_cmd, text=True, capture_output=True)
|
||||||
|
wait_proc(srv)
|
||||||
|
if r1.returncode != 0:
|
||||||
|
raise RuntimeError(f"First sync failed: {r1.stderr[:100]}")
|
||||||
|
srv2 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
|
||||||
|
time.sleep(0.5)
|
||||||
|
second_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags + ["--incremental"]
|
||||||
|
start = time.monotonic()
|
||||||
|
r2 = subprocess.run(second_cmd, text=True, capture_output=True, timeout=30)
|
||||||
|
duration = time.monotonic() - start
|
||||||
|
wait_proc(srv2)
|
||||||
|
r = {"name": "Incremental (--incremental)", "suite": profile_name,
|
||||||
|
"status": "Success" if r2.returncode == 0 else "Failed",
|
||||||
|
"time": f"{duration:.4f}s" if r2.returncode == 0 else "N/A",
|
||||||
|
"error": "" if r2.returncode == 0 else f"Exit {r2.returncode}: {(r2.stderr or r2.stdout)[:60]}"}
|
||||||
|
results.append(r)
|
||||||
|
except Exception as e:
|
||||||
|
results.append({"name": "Incremental (--incremental)", "suite": profile_name,
|
||||||
|
"status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
|
# Chunk size (--chunk-size 5242880)
|
||||||
|
feature_flags = BASE_CLIENT_FLAGS + ["--chunk-size", "5242880"]
|
||||||
|
cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + feature_flags
|
||||||
|
print(f"\n --- Chunk size (--chunk-size 5242880) ---\n Running: {' '.join(cmd)}")
|
||||||
|
try:
|
||||||
|
r = run_single_test(cmd, "Chunk size (--chunk-size 5242880)", source_dir, dest_dir)
|
||||||
|
r["suite"] = profile_name
|
||||||
|
results.append(r)
|
||||||
|
except Exception as e:
|
||||||
|
results.append({"name": "Chunk size (--chunk-size 5242880)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
|
# Delete (--delete) — pre-populate dest, add extra files, then sync with --delete
|
||||||
|
# Note: server handles one client per launch, so we restart between syncs
|
||||||
|
print(f"\n --- Delete (--delete) ---")
|
||||||
|
try:
|
||||||
|
flags = BASE_CLIENT_FLAGS + ["-M"]
|
||||||
|
# First sync (no delete) to populate dest
|
||||||
|
s1 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
|
||||||
|
time.sleep(0.5)
|
||||||
|
first_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags
|
||||||
|
r1 = subprocess.run(first_cmd, text=True, capture_output=True)
|
||||||
|
wait_proc(s1)
|
||||||
|
if r1.returncode != 0:
|
||||||
|
raise RuntimeError(f"First sync failed: {r1.stderr[:100]}")
|
||||||
|
# Add extra files to received dir
|
||||||
|
received = os.path.join(dest_dir, os.path.abspath(source_dir).lstrip(os.sep))
|
||||||
|
extra_path = os.path.join(received, "extra_file.txt")
|
||||||
|
with open(extra_path, "w") as f:
|
||||||
|
f.write("should be deleted")
|
||||||
|
extra_dir = os.path.join(received, "extra_dir")
|
||||||
|
os.makedirs(extra_dir, exist_ok=True)
|
||||||
|
with open(os.path.join(extra_dir, "nested.txt"), "w") as f:
|
||||||
|
f.write("nested extra")
|
||||||
|
# Second sync with --delete (fresh server)
|
||||||
|
s2 = subprocess.Popen(SERVER_CMD, stdout=subprocess.DEVNULL, stderr=None)
|
||||||
|
time.sleep(0.5)
|
||||||
|
second_cmd = client_prefix + BASE_CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir] + flags + ["--delete"]
|
||||||
|
start = time.monotonic()
|
||||||
|
r2 = subprocess.run(second_cmd, text=True, capture_output=True)
|
||||||
|
duration = time.monotonic() - start
|
||||||
|
wait_proc(s2)
|
||||||
|
r = {"name": "Delete (--delete)", "suite": profile_name}
|
||||||
|
if r2.returncode == 0 and not os.path.exists(extra_path) and not os.path.exists(extra_dir):
|
||||||
|
mismatches, missing = verify_transfer(source_dir, received)
|
||||||
|
if not mismatches and not missing:
|
||||||
|
r["status"] = "Success"
|
||||||
|
r["time"] = f"{duration:.4f}s"
|
||||||
|
r["error"] = ""
|
||||||
|
else:
|
||||||
|
r["status"] = "Failed"
|
||||||
|
r["time"] = "N/A"
|
||||||
|
r["error"] = f"post-delete verify: mismatches={len(mismatches)}, missing={len(missing)}"
|
||||||
|
else:
|
||||||
|
r["status"] = "Failed"
|
||||||
|
r["time"] = "N/A"
|
||||||
|
errs = []
|
||||||
|
if r2.returncode != 0:
|
||||||
|
errs.append(f"Exit {r2.returncode}: {(r2.stderr or r2.stdout)[:60]}")
|
||||||
|
if os.path.exists(extra_path):
|
||||||
|
errs.append("extra_file.txt remains")
|
||||||
|
if os.path.exists(extra_dir):
|
||||||
|
errs.append("extra_dir remains")
|
||||||
|
r["error"] = " | ".join(errs)
|
||||||
|
results.append(r)
|
||||||
|
except Exception as e:
|
||||||
|
results.append({"name": "Delete (--delete)", "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
|
# SSH feature tests
|
||||||
|
if SSH_AVAILABLE:
|
||||||
|
ssh_dest = f"localhost:{dest_dir}_ssh"
|
||||||
|
ssh_feature_cases = [
|
||||||
|
{"name": "SSH Archive (-a)", "flags": ["-a"]},
|
||||||
|
{"name": "SSH Exclude (--exclude small.txt)", "flags": ["--exclude", "small.txt"], "expected_missing": ["small.txt"]},
|
||||||
|
]
|
||||||
|
for case in ssh_feature_cases:
|
||||||
|
flags = BASE_CLIENT_FLAGS + case["flags"]
|
||||||
|
cmd = BASE_CLIENT_CMD + [source_dir, ssh_dest] + flags
|
||||||
|
print(f"\n --- {case['name']} ---\n Running: {' '.join(cmd)}")
|
||||||
|
try:
|
||||||
|
r = run_single_test(cmd, case["name"], source_dir, f"{dest_dir}_ssh",
|
||||||
|
no_server=True,
|
||||||
|
expected_missing=case.get("expected_missing"))
|
||||||
|
r["suite"] = profile_name
|
||||||
|
results.append(r)
|
||||||
|
except Exception as e:
|
||||||
|
results.append({"name": case["name"], "suite": profile_name, "status": "Error", "time": "N/A", "error": str(e)})
|
||||||
|
|
||||||
except (subprocess.CalledProcessError, RuntimeError) as e:
|
except (subprocess.CalledProcessError, RuntimeError) as e:
|
||||||
print(f" Error: {e}")
|
print(f" Error: {e}")
|
||||||
@@ -563,19 +594,26 @@ def check_ssh_localhost():
|
|||||||
build_dir = os.path.abspath("build")
|
build_dir = os.path.abspath("build")
|
||||||
server_path = os.path.join(build_dir, "server")
|
server_path = os.path.join(build_dir, "server")
|
||||||
|
|
||||||
r = subprocess.run(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=5",
|
try:
|
||||||
"localhost", "which", "fastsync-server"],
|
r = subprocess.run(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=5",
|
||||||
capture_output=True, timeout=10)
|
"localhost", "which", "fastsync-server"],
|
||||||
|
capture_output=True, timeout=10)
|
||||||
|
except FileNotFoundError:
|
||||||
|
SSH_AVAILABLE = False
|
||||||
|
return
|
||||||
if r.returncode == 0:
|
if r.returncode == 0:
|
||||||
SSH_AVAILABLE = True
|
SSH_AVAILABLE = True
|
||||||
return
|
return
|
||||||
|
|
||||||
SSH_AVAILABLE = False
|
SSH_AVAILABLE = False
|
||||||
# Try each PATH dir: create symlink, then verify with which
|
# Try each PATH dir: create symlink, then verify with which
|
||||||
r = subprocess.run(
|
try:
|
||||||
["ssh", "-o", "BatchMode=yes", "localhost",
|
r = subprocess.run(
|
||||||
'echo "$PATH"'],
|
["ssh", "-o", "BatchMode=yes", "localhost",
|
||||||
capture_output=True, timeout=10, text=True)
|
'echo "$PATH"'],
|
||||||
|
capture_output=True, timeout=10, text=True)
|
||||||
|
except FileNotFoundError:
|
||||||
|
return
|
||||||
if r.returncode != 0:
|
if r.returncode != 0:
|
||||||
return
|
return
|
||||||
for d in r.stdout.strip().split(":"):
|
for d in r.stdout.strip().split(":"):
|
||||||
@@ -656,9 +694,10 @@ def main():
|
|||||||
parser.add_argument("--keep-data", action="store_true")
|
parser.add_argument("--keep-data", action="store_true")
|
||||||
parser.add_argument("--unlimited", action="store_true")
|
parser.add_argument("--unlimited", action="store_true")
|
||||||
parser.add_argument("--wan", action="store_true")
|
parser.add_argument("--wan", action="store_true")
|
||||||
|
parser.add_argument("--full", action="store_true", help="Run full test suite with network shaping, SSH, rsync benchmarks")
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
total_bytes = generate_test_files(args.source_dir)
|
total_bytes = generate_test_files(args.source_dir, full=args.full)
|
||||||
if os.path.exists(args.dest_dir):
|
if os.path.exists(args.dest_dir):
|
||||||
shutil.rmtree(args.dest_dir)
|
shutil.rmtree(args.dest_dir)
|
||||||
os.makedirs(args.dest_dir, exist_ok=True)
|
os.makedirs(args.dest_dir, exist_ok=True)
|
||||||
@@ -669,12 +708,12 @@ def main():
|
|||||||
elif args.wan:
|
elif args.wan:
|
||||||
profiles.append("WAN")
|
profiles.append("WAN")
|
||||||
else:
|
else:
|
||||||
profiles.append("LAN")
|
profiles.append("LAN" if args.full else "Unlimited")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
all_results = []
|
all_results = []
|
||||||
for p in profiles:
|
for p in profiles:
|
||||||
all_results.extend(run_profile(p, args.source_dir, args.dest_dir))
|
all_results.extend(run_profile(p, args.source_dir, args.dest_dir, full=args.full))
|
||||||
|
|
||||||
print("\n" + "=" * 130)
|
print("\n" + "=" * 130)
|
||||||
print(f"{'RESULTS':^130}")
|
print(f"{'RESULTS':^130}")
|
||||||
|
|||||||
Reference in New Issue
Block a user