Compare commits

..

13 Commits

Author SHA1 Message Date
TapTap 02ec38c8a7 fix: address re-review — coverage build, chunk OOB guard, UBSan loads, srand scope
CI / lint (push) Successful in 8s
CI / lint (pull_request) Successful in 8s
CI / sanitizers (address) (push) Successful in 16s
CI / sanitizers (undefined) (push) Successful in 14s
CI / fuzz-build (push) Successful in 12s
CI / coverage (push) Successful in 10s
CI / build-and-test (push) Successful in 54s
CI / valgrind (push) Successful in 14s
CI / sanitizers (address) (pull_request) Successful in 14s
CI / sanitizers (undefined) (pull_request) Successful in 14s
CI / fuzz-build (pull_request) Successful in 12s
CI / build-and-test (pull_request) Successful in 55s
CI / coverage (pull_request) Successful in 10s
CI / valgrind (pull_request) Successful in 12s
- Remove -DSTRICT_WARNINGS=ON from coverage job (_FORTIFY_SOURCE + -O0 + -Werror fatal)
- Fix chunk.c metadata guard: peek at present flag before calling metadata_from_buf
  so the remaining_size check covers the full sizeof(int)+FILE_METADATA_WIRE_SIZE
- Fix chunk.c UBSan misaligned loads: use memcpy instead of *(size_t*)deref
- Move srand(42) to test_property() top level so all property tests are seeded
2026-07-20 14:38:10 +02:00
TapTap cda61f5153 ci: bump CI image to v9 with libclang-rt-18-dev
v8 had cached image without libclang-rt-18-dev on CI runner.
v9 forces a fresh pull and includes all fuzzer runtime libraries.
2026-07-20 14:38:10 +02:00
TapTap 62d4c10168 fix: bake CI deps into Docker image, fix chunk use_metadata OOB
- Build and push fastsync-ci:v8 with lcov, valgrind, clang baked in
- Remove all apt-get install steps from CI (v8 has them pre-installed)
- Fix chunk.c use_metadata=true OOB: add remaining_size guard before
  metadata_from_buf reads past the buffer. The bug allowed network-facing
  chunk_deserialize to heap-buffer-overflow on crafted inputs.
- Also guard against unsigned underflow on remaining_size - sizeof(int)
2026-07-20 14:38:10 +02:00
TapTap c9369b7118 fix: address PR review — fuzz target fixes, UBSan, CI improvements
Critical fixes:
- Add missing #include <string.h> to 5 fuzz targets (wouldn't compile with GCC14+/Clang16+)
- Fix fuzz_metadata_from_buf.c OOB read: guard size >= sizeof(int) + FILE_METADATA_WIRE_SIZE
- Add Clang compiler check for ENABLE_FUZZ in CMakeLists.txt (fail fast at configure time)
- Add fuzz-build CI job (install clang, build all 6 fuzz targets with ENABLE_FUZZ=ON)

Warning fixes:
- Add UBSan to sanitizer CI matrix (address + undefined)
- Remove tautological test_property_glob_consistency (pure/deterministic function)
- Use fixed seed srand(42) instead of srand(time(NULL)) for reproducible property tests
- Add valid-header + truncated-instructions delta robustness test (exercises instruction-loop error paths)
- Fix lcov --remove to exclude '*/_deps/*' (xxhash coverage pollution)
- Add comment explaining FASTSYNC_UNDER_VALGRIND skip in test_file.c
- Update .gitignore for build-*/ directories
2026-07-20 14:37:40 +02:00
TapTap 20cfc5f438 ci: fix coverage and valgrind jobs, skip fork tests under valgrind
- Install lcov in coverage job (not in fastsync-ci:v7)
- Use lcov 2.x compatible flags (--branch-coverage instead of --rc)
- Remove unused xxhash exclude pattern that lcov 2.x rejects
- Install valgrind in valgrind job
- Skip fork-based file tests under valgrind (pipe timing issues)
- Add FASTSYNC_UNDER_VALGRIND env var for test skip detection
- Move cleanup before assertions in fork tests to prevent leaks
- Add coverage.info to .gitignore
2026-07-20 14:37:40 +02:00
TapTap 68603d1395 ci: install lcov and valgrind in CI jobs
The fastsync-ci:v7 container does not include lcov or valgrind.
Add apt-get install steps to the coverage and valgrind jobs.

Also update lcov flags for lcov 2.x compatibility:
--rc lcov_branch_coverage=1 -> --branch-coverage
2026-07-20 14:37:40 +02:00
TapTap d4dc8f7c51 fix: resolve cppcheck warnings in new test files
- Remove unused ConsumerCtx struct in test_stress.c
- Add const qualifiers to variables only checked for NULL
  in test_metadata.c, test_protocol.c, test_robustness.c
2026-07-20 14:37:40 +02:00
TapTap 9949597cbf style: fix clang-format violations in test files
Fix line wrapping and designated initializer alignment in
test_file.c, test_protocol.c, and test_stress.c to pass
CI clang-format check.
2026-07-20 14:37:40 +02:00
TapTap dbc0897fc6 testing: add comprehensive test suite for test-driven development
Add unit tests for previously untested modules (protocol, data, metadata,
file, glob), robustness tests for deserialization of malformed inputs,
thread safety stress tests, property-based roundtrip tests, and 6 fuzz
targets. Update CI with CTest integration, coverage reporting, and
valgrind memory checking.

New test files:
- test_data.c: Data type lifecycle (5 tests)
- test_protocol.c: Protocol I/O roundtrips and error paths (9 tests)
- test_metadata.c: Metadata serialization roundtrips (6 tests)
- test_file.c: File operations and send/receive (12 tests)
- test_glob.c: Glob pattern matching (10 tests)
- test_robustness.c: Malformed input handling for chunk/delta/protocol (11 tests)
- test_stress.c: MPMC queue stress, backpressure, rapid create/destroy (3 tests)
- test_property.c: Compress, delta, chunk, glob roundtrip properties (4 tests)
- tests/fuzz/: 6 libFuzzer targets for deserialization functions

Extended existing tests:
- test_config.c: config_send/config_receive roundtrip via fork+pipe
- test_shared_utils.c: mkdir_r, glob_match, delete_extras

Infrastructure:
- CTest integration in CMakeLists.txt
- Coverage support (-DENABLE_COVERAGE=ON)
- Fuzz target support (-DENABLE_FUZZ=ON)
- CI: coverage, valgrind, address sanitizer jobs
- Fixed MPMC stress test race condition (cnd_signal -> cnd_broadcast)
2026-07-20 14:37:40 +02:00
TapTap a4b35e136b Merge pull request 'docs: add AGENTS.md with custom-image dependency install rule' (#27) from feat/custom-image-deps-rule into main
CI / lint (push) Successful in 12s
CI / sanitizers (address) (push) Successful in 14s
CI / build-and-test (push) Successful in 53s
Reviewed-on: #27
2026-07-19 22:57:15 +02:00
TapTap 02ceb6828b fix: re-review cleanup — deps/conventions/sanitizer sections in cmake-expert, AGENTS.md notes
CI / lint (push) Successful in 8s
CI / lint (pull_request) Successful in 7s
CI / sanitizers (address) (push) Successful in 14s
CI / build-and-test (push) Successful in 54s
CI / build-and-test (pull_request) Successful in 53s
CI / sanitizers (address) (pull_request) Successful in 14s
2026-07-19 22:54:02 +02:00
TapTap 0f9e689e39 fix: address PR review — lcov/valgrind in Dockerfile, correct cmake-expert.md, expand AGENTS.md
CI / lint (push) Successful in 8s
CI / lint (pull_request) Successful in 7s
CI / build-and-test (push) Successful in 53s
CI / sanitizers (address) (push) Successful in 14s
CI / build-and-test (pull_request) Successful in 53s
CI / sanitizers (address) (pull_request) Successful in 14s
2026-07-19 22:37:26 +02:00
TapTap a7bb6454a4 docs: add AGENTS.md with custom-image dependency install rule
CI / lint (push) Successful in 8s
CI / build-and-test (push) Successful in 52s
CI / sanitizers (address) (push) Successful in 13s
CI / lint (pull_request) Successful in 7s
CI / build-and-test (pull_request) Successful in 53s
CI / sanitizers (address) (pull_request) Successful in 14s
2026-07-19 20:43:54 +02:00
3 changed files with 103 additions and 33 deletions
+2
View File
@@ -9,6 +9,8 @@ You are a system architect for the FastSync project — a high-performance file
Make high-level design decisions. Evaluate trade-offs, plan module interactions, design data flow, and ensure architectural coherence across the codebase. Make high-level design decisions. Evaluate trade-offs, plan module interactions, design data flow, and ensure architectural coherence across the codebase.
> **Environment rule:** dependency installation must always use the project's custom Docker image (repo-root `Dockerfile`, same as CI) — never ad-hoc host package installs. See `AGENTS.md`.
## Project Architecture ## Project Architecture
### Module Map ### Module Map
+44 -33
View File
@@ -3,7 +3,7 @@ description: Manages the CMake build system for FastSync — adding targets, sou
mode: subagent mode: subagent
--- ---
You are a CMake expert for the FastSync project — a high-performance file synchronization system built with CMake 4.1+ and C11. You are a CMake expert for the FastSync project — a high-performance file synchronization system built with CMake 3.22+ and C11.
## Your Role ## Your Role
@@ -13,7 +13,7 @@ Manage the CMake build system: add new targets, configure dependencies, set comp
### `CMakeLists.txt` (project root) ### `CMakeLists.txt` (project root)
```cmake ```cmake
cmake_minimum_required(VERSION 4.1) cmake_minimum_required(VERSION 3.22)
project(FastFileTransfer) project(FastFileTransfer)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON) set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
@@ -22,30 +22,53 @@ set(CMAKE_C_STANDARD_REQUIRED ON)
add_compile_options(-Wall -g -O3) add_compile_options(-Wall -g -O3)
# Sanitizer option
set(SANITIZER "none" CACHE STRING "Sanitizer to enable (address, thread, none)")
set_property(CACHE SANITIZER PROPERTY STRINGS address thread none)
if(SANITIZER STREQUAL "address")
add_compile_options(-fsanitize=address -fno-omit-frame-pointer -g)
add_link_options(-fsanitize=address)
elseif(SANITIZER STREQUAL "thread")
add_compile_options(-fsanitize=thread -fno-omit-frame-pointer -g)
add_link_options(-fsanitize=thread)
elseif(NOT SANITIZER STREQUAL "none")
message(FATAL_ERROR "Unknown sanitizer: ${SANITIZER}. Supported values: address, thread, none")
endif()
option(STRICT_WARNINGS "Enable strict warnings" OFF)
if(STRICT_WARNINGS)
add_compile_options(-Wextra -Wpedantic -Werror)
endif()
include(FetchContent)
FetchContent_Declare(xxhash GIT_REPOSITORY https://github.com/Cyan4973/xxHash GIT_TAG v0.8.3 SOURCE_SUBDIR cmake_unofficial)
FetchContent_MakeAvailable(xxhash)
set(THREADS_PREFER_PTHREAD_FLAG ON) set(THREADS_PREFER_PTHREAD_FLAG ON)
find_package(Threads REQUIRED) find_package(Threads REQUIRED)
find_library(ZSTD_LIBRARY zstd) find_library(ZSTD_LIBRARY zstd)
# ... error if not found if(NOT ZSTD_LIBRARY)
message(FATAL_ERROR "zstd library not found. Ensure it is in your nix-shell!")
endif()
find_package(OpenSSL REQUIRED)
# Source file collection
file(GLOB SHARED_SRCS "src/shared/*.c") file(GLOB SHARED_SRCS "src/shared/*.c")
file(GLOB SERVER_SRCS "src/server/*.c") file(GLOB SERVER_SRCS "src/server/*.c")
file(GLOB CLIENT_SRCS "src/client/*.c") file(GLOB CLIENT_SRCS "src/client/*.c")
file(GLOB TEST_SRCS "tests/*.c") file(GLOB TEST_SRCS "tests/*.c")
# Targets
add_executable(server ${SERVER_SRCS} ${SHARED_SRCS}) add_executable(server ${SERVER_SRCS} ${SHARED_SRCS})
target_include_directories(server PRIVATE src/shared src/server src/client) target_include_directories(server PRIVATE src/shared src/server src/client)
target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY}) target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
add_executable(client ${CLIENT_SRCS} ${SHARED_SRCS}) add_executable(client ${CLIENT_SRCS} ${SHARED_SRCS})
target_include_directories(client PRIVATE src/shared src/server src/client) target_include_directories(client PRIVATE src/shared src/server src/client)
target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY}) target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} src/client/scanner.c) add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} src/client/scanner.c)
target_include_directories(tests PRIVATE tests src/shared src/server src/client) target_include_directories(tests PRIVATE tests src/shared src/server src/client)
target_link_libraries(tests PRIVATE Threads::Threads ${ZSTD_LIBRARY}) target_link_libraries(tests PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
``` ```
### Source Layout ### Source Layout
@@ -59,16 +82,19 @@ tests/ — test sources (globbed as TEST_SRCS)
### Dependencies ### Dependencies
- **zstd** — found via `find_library(ZSTD_LIBRARY zstd)` - **zstd** — found via `find_library(ZSTD_LIBRARY zstd)`
- **pthreads** — found via `find_package(Threads REQUIRED)` - **pthreads** — found via `find_package(Threads REQUIRED)`
- **OpenSSL** — found via `find_package(OpenSSL REQUIRED)`
- **xxhash** — fetched via `FetchContent` from GitHub (v0.8.3)
- **C11 standard** — required - **C11 standard** — required
- **CMake 4.1+** — minimum version - **CMake 3.22+** — minimum version
## Conventions ## Conventions
- Use `file(GLOB ...)` for source collection (existing pattern). - Use `file(GLOB ...)` for source collection (existing pattern).
- All targets link `Threads::Threads` and `${ZSTD_LIBRARY}`. - All targets link `Threads::Threads`, `${ZSTD_LIBRARY}`, `OpenSSL::SSL`, `OpenSSL::Crypto`, and `xxhash`.
- Include directories: `src/shared`, `src/server`, `src/client`, `tests` (for test target). - Include directories: `src/shared`, `src/server`, `src/client`, `tests` (for test target).
- Sanitizer support is commented out but present (`-fsanitize=address`). - Sanitizer support: pass `-DSANITIZER=address` or `-DSANITIZER=thread` to cmake (live option in CMakeLists.txt).
- Build with `cmake -B build -S . && cmake --build build -j$(nproc)`. - Build with `cmake -B build -S . && cmake --build build -j$(nproc)`.
- Install dependencies only via the project's custom Docker image (repo-root `Dockerfile`, same image CI uses) — never via host package installs; see `AGENTS.md`.
## When Making Changes ## When Making Changes
@@ -77,28 +103,21 @@ tests/ — test sources (globbed as TEST_SRCS)
3. Add new dependencies with `find_package` or `find_library`. 3. Add new dependencies with `find_package` or `find_library`.
4. When adding a new executable target, follow the pattern of existing targets. 4. When adding a new executable target, follow the pattern of existing targets.
5. When adding a new library (static/shared), use `add_library` and follow the project's naming. 5. When adding a new library (static/shared), use `add_library` and follow the project's naming.
6. For sanitizer builds, use the commented-out `-fsanitize=address` lines as reference. 6. For sanitizer builds, pass `-DSANITIZER=address` or `-DSANITIZER=thread` to cmake (matching CI's matrix strategy).
7. Always verify the build compiles after changes. 7. Always verify the build compiles after changes.
## Sanitizer Configurations ## Sanitizer Configurations
### AddressSanitizer (memory errors) Use the project's built-in `-DSANITIZER=` option (matching the CI matrix):
```bash ```bash
cmake -B build -S . \ cmake -B build -S . -DSANITIZER=address # AddressSanitizer (memory errors)
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -g" \ cmake --build build -j$(nproc)
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
cmake -B build -S . -DSANITIZER=thread # ThreadSanitizer (race conditions)
cmake --build build -j$(nproc) cmake --build build -j$(nproc)
``` ```
### ThreadSanitizer (race conditions) For UndefinedBehaviorSanitizer (no `-DSANITIZER=undefined` option in CMakeLists.txt yet), use the manual flag approach:
```bash
cmake -B build -S . \
-DCMAKE_C_FLAGS="-fsanitize=thread -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread"
cmake --build build -j$(nproc)
```
### UndefinedBehaviorSanitizer
```bash ```bash
cmake -B build -S . \ cmake -B build -S . \
-DCMAKE_C_FLAGS="-fsanitize=undefined -fno-omit-frame-pointer -g" \ -DCMAKE_C_FLAGS="-fsanitize=undefined -fno-omit-frame-pointer -g" \
@@ -106,14 +125,6 @@ cmake -B build -S . \
cmake --build build -j$(nproc) cmake --build build -j$(nproc)
``` ```
### Combined Sanitizers
```bash
cmake -B build -S . \
-DCMAKE_C_FLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined"
cmake --build build -j$(nproc)
```
### Using ccache (faster rebuilds) ### Using ccache (faster rebuilds)
```bash ```bash
cmake -B build -S . -DCMAKE_C_COMPILER_LAUNCHER=ccache cmake -B build -S . -DCMAKE_C_COMPILER_LAUNCHER=ccache
+57
View File
@@ -0,0 +1,57 @@
# AGENTS.md
FastSync is a high-performance file synchronization system written in C11. It supports TCP and SSH transports, TLS encryption (OpenSSL), streaming zstd compression, multithreaded transfers, and incremental sync. The build uses CMake; CI runs on Gitea Actions (`.gitea/workflows/ci.yaml`).
## Dependency installation
**Rule: always install dependencies using the project's custom Docker image — never via ad-hoc system package installs on the host** (no `apt-get install` / `pip install` on the host machine).
The image is built from the repo-root `Dockerfile` and is the same image CI uses: `gitea.tap-tap.win/taptap/fastsync-ci:v7`. It contains the full toolchain: gcc/g++, CMake, libzstd-dev, libssl-dev, make, git, cppcheck, clang-format, python3 + pytest, openssh-client, and Node.js.
```bash
# Use the prebuilt CI image directly (faster, guaranteed CI parity)
docker pull gitea.tap-tap.win/taptap/fastsync-ci:v7
docker tag gitea.tap-tap.win/taptap/fastsync-ci:v7 fastsync-ci:local
# Or build the image from the repo-root Dockerfile
# (Note: the prebuilt :v7 image reflects the previous Dockerfile state;
# rebuild from source to pick up any newly added packages like lcov/valgrind.)
docker build -t fastsync-ci:local .
# Build, run unit tests, and run integration tests inside the container
docker run --rm -v "$PWD:/workspace" -w /workspace fastsync-ci:local \
sh -c 'cmake -B build -S . && cmake --build build -j$(nproc) && ./build/tests && python3 -m pytest tests/'
# Avoid root-owned build/ artifacts by matching your host UID/GID
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/workspace" \
-w /workspace fastsync-ci:local \
sh -c 'cmake -B build -S . && cmake --build build -j$(nproc) && ./build/tests && python3 -m pytest tests/'
```
> **Note:** The first `cmake configure` (`cmake -B build -S .`) fetches xxHash from GitHub via `FetchContent` — network access is required. Subsequent reconfigures reuse the cached source.
If a dependency is missing from the image, add it to the `Dockerfile` (and rebuild) rather than installing it on the host.
## CI Conventions
When configuring for CI parity, use:
```bash
cmake -B build -S . -DSTRICT_WARNINGS=ON # -Wextra -Wpedantic -Werror
cmake -B build -S . -DSANITIZER=address # AddressSanitizer (ASan)
cmake -B build -S . -DSANITIZER=thread # ThreadSanitizer (TSan)
```
The CI workflow (`.gitea/workflows/ci.yaml`) runs lint (clang-format, cppcheck), build + test (unit + integration), and sanitizer (currently only `address`) jobs sequentially.
## Build
```bash
cmake -B build -S . && cmake --build build -j$(nproc)
```
## Test
```bash
./build/tests # unit tests
python3 -m pytest tests/ # integration tests
```