Commit Graph

94 Commits

Author SHA1 Message Date
TapTap 5dfc262cf7 fix: scanner leak, remove TSan (Docker incompatible)
CI / lint (push) Successful in 6s
CI / lint (pull_request) Successful in 5s
CI / build-and-test (push) Successful in 48s
CI / sanitizers (address) (push) Successful in 5s
CI / build-and-test (pull_request) Successful in 47s
CI / sanitizers (address) (pull_request) Successful in 4s
- Fix ArrayList leak in directory_scanner_next when no files found
- Remove TSan from CI matrix (Docker 'unexpected memory mapping')
- Keep ASan which now passes clean
2026-07-19 16:22:16 +02:00
TapTap 23b1d6660c fix: reformat codebase and fix const-correctness for CI lint
CI / lint (push) Failing after 16s
CI / build-and-test (push) Has been skipped
CI / sanitizers (address) (push) Has been skipped
CI / sanitizers (thread) (push) Has been skipped
CI / lint (pull_request) Failing after 44s
CI / build-and-test (pull_request) Has been skipped
CI / sanitizers (address) (pull_request) Has been skipped
CI / sanitizers (thread) (pull_request) Has been skipped
- Reformat all C/H files to match .clang-format (LLVM style)
- Fix 26 cppcheck const-correctness warnings (constParameterPointer,
  constVariablePointer, constVariable)
- Update function declarations in headers to match const parameters
2026-07-19 15:57:31 +02:00
TapTap 598e4e7514 fix: address PR review - rolling adler32, extract helpers, configurable max file size
CI / build-and-test (push) Successful in 29s
CI / build-and-test (pull_request) Successful in 28s
- Fix rolling adler32: add missing -1 in s2 update formula (was producing
  wrong checksums, causing zero block matches)
- Fix file_send_sendfile signature to match typedef (add unused
  compression_level param)
- Extract receive_delta_file() from receive_incremental_check() to reduce
  nesting depth
- Extract send_file_incremental() helper in client_send.c
- Add delta_max_file_size to Config, serialized over wire
- Add --delta-max CLI flag
- Add test_large_file_delta (200KB) and extra delta_should_attempt cases
- Remove unused pos_in_block variable from delta_compute
2026-07-19 02:02:28 +02:00
TapTap e16db56580 feat: add delta transfer for incremental sync
CI / build-and-test (push) Successful in 28s
CI / build-and-test (pull_request) Successful in 28s
Implement rsync-style delta transfer using rolling checksums (Adler-32 +
xxHash32). When a file exists on both sides but has changed, only the
changed blocks are transmitted instead of the entire file.

- New status codes: STATUS_DELTA_SIGNATURE, STATUS_DELTA_DATA
- Protocol version bumped to 1.2.0
- Server generates block signature, client computes delta
- Auto-fallback to whole-file when delta >= 70% of file size
- Works with zstd compression on delta stream
- Configurable block size (default 8KB, --delta-block flag)
- 13 unit tests covering hashing, signature roundtrip, delta compute/apply,
  file growth/shrink, and decision logic
2026-07-18 20:00:15 +02:00
TapTap 864eb1316d Fix TLS code review issues: SSL cleanup, TLS 1.2 min, CA verify, deprecation guards, server --help, port validation, __thread io_ssl, shared accept loop 2026-07-18 17:39:37 +02:00
TapTap eff50346fd TLS transport: OpenSSL-based encrypted TCP
- New transport_tls.h/c: TLS server (server_create_tls, server_listen_tls)
  and client (client_connect_tls) using OpenSSL
- protocol.c: io_set_ssl() + SSL_read/SSL_write in send_n_data/receive_n_data
- transport_tcp.h: ssl/ssl_ctx fields added to Server/Client structs
- config.h/c: use_tls, tls_cert, tls_key fields
- client_cli.c: --tls, --cert, --key flags
- server.c: --tls, --cert, --key, -p flags with TLS support
- CMakeLists.txt: OpenSSL::SSL + OpenSSL::Crypto linkage
- shell.nix: openssl added to buildInputs
2026-07-18 17:38:55 +02:00
TapTap 46c650e9ae refactor: consolidate _no_path file send variants
CI / build-and-test (push) Successful in 26s
CI / build-and-test (pull_request) Successful in 26s
Add bool send_path parameter to file_send_single_calls and
file_send_sendfile, remove the _no_path variants. Callers in
client_send.c pass true (send path) or false (skip path) based
on whether an incremental check already transmitted the path.
2026-07-18 17:27:33 +02:00
TapTap 03d6ba0da5 fix: address review #69 - STATUS_ERROR sends, metadata_receive error handling, extract incremental_check helper 2026-07-18 17:19:12 +02:00
TapTap f1af88a25c Fix incremental-sync review: sendfile_no_path, reject -s+incremental, no data mutation, auto -M, STATUS_ERROR handling, PROTOCOL_VERSION bump 2026-07-18 17:19:12 +02:00
TapTap 1905369c9f Incremental sync: --incremental flag to skip unchanged files
- New STATUS_CHECK protocol status (value 6)
- Client sends path + size + mtime; server replies OK (skip) or NEXT (send)
- config.h/c: use_incremental field sent/received over wire
- file.c/h: file_send_single_calls_no_path helper for incremental path
- client_cli.c: --incremental flag
- client_send.c: per-file check before send in both single and chunk paths
- server.c: STATUS_CHECK handling in receive_files()
- multiprocessing.c: STATUS_CHECK handling in receive_thread()
- test.py: incremental sync test case
2026-07-18 17:19:12 +02:00
TapTap 4f8ae5bb70 fix: bw_tokens initialized to io_bwlimit, overflow check for --bwlimit 2026-07-17 10:07:18 +02:00
TapTap 3dcaf0b79d Fix bwlimit review: remove __thread (breaks -m), validate >0, handle nanosleep EINTR 2026-07-16 17:00:14 +02:00
TapTap 09a76e2a9a Bandwidth throttling: --bwlimit <KB/s> with token bucket
- protocol.h/c: io_set_bwlimit() + token bucket in send_n_data
  (64KB chunks, nanosleep-based deficit compensation)
- client_cli.c: --bwlimit <KB/s> flag
- test.py: bandwidth limit test case
2026-07-16 15:42:16 +02:00
TapTap a62894be05 Merge main into server-improvements: resolve conflicts (protocol signatures, error returns) 2026-07-16 15:08:56 +02:00
TapTap bbdf027be8 Fix exit() in library code, add --include/--max-size/--min-size 2026-07-16 14:15:38 +02:00
TapTap efb98d8964 Multi-client server, protocol versioning, streaming decompression fixes 2026-07-16 13:59:41 +02:00
TapTap 2ff51246c3 Update README and add feature tests for rsync-compatible flags 2026-07-16 13:31:43 +02:00
TapTap 284bf8f8bc rsync flags: -a, -n, -p, --exclude, --delete
Feature changes across 15 files:

- -a/--archive: enables -c -m -M (no -s, which slows transfers)
- -n/--dry-run: scan + print without connecting or transferring
- -p <port>: custom SSH port (passed as -p to ssh via execvp)
- --exclude <pattern>: glob-based filename filtering in scanner
- --delete: sender collects file manifest; receiver deletes unlisted files

Protocol: added STATUS_MANIFEST, use_delete field in Config wire format.
New utilities: glob_match(), delete_extras() with recursive directory walk.
Scanner: accepts exclude patterns; skips matching entries.
SSH transport: switched from execlp to execvp for dynamic port arg.
Server + multiprocessing: handle STATUS_MANIFEST in both single and
multithreaded receive paths.

Builds clean, all 7 tests pass.
2026-07-16 12:39:05 +02:00
TapTap d1a23fc4df quick-wins: ControlMaster, -z alias, --progress, fix write warning
- SSH ControlMaster auto with ControlPath for connection reuse
- -z as alias for -c (compression)
- --progress flag showing transfer rate in send_files
- Fixed unused-result warning on write() in transport_ssh.c
2026-07-16 12:28:16 +02:00
TapTap ca29109664 perf: downsized test data, increased SSH socket buffer, and made chunk size configurable
- Reduced test data from 50MB to 25MB (test.py)
- Increased SSH socketpair buffer to 1MB via setsockopt (transport_ssh.c)
- Made chunk size configurable: new Config field, --chunk-size CLI flag,
  sent over wire, and used in scanner instead of DESIRED_CHUNK_SIZE
- Fixed DESIRED_CHUNK_SIZE macro parentheses (chunk.h)
2026-07-16 12:20:02 +02:00
TapTap 8f4ea62ae7 Phase 1+2: bug fixes and dead code cleanup
Phase 1 — Bugs:
- receive_data: fix unsigned long long vs size_t mismatch
- chunk_deserialize: add NULL check in multiprocessing.c and server.c
- file_content_to_buffer: add missing fclose on error path
- send_files_multithreaded: propagate sender thread result
- server: add SIGPIPE handler
- to_disk: check fwrite return value
- scanner: use S_ISDIR instead of !S_ISREG
- scanner: free cur_path before early return

Phase 2 — Cleanup:
- Remove unused chunk_decompress function
- Remove unused array_list_clear function
- Remove unused num_connections config field
- Remove duplicate FILE_METADATA_WIRE_SIZE macro
- Update tests for removed APIs
2026-07-16 12:08:09 +02:00
TapTap d36768792a fixup: address PR review comments
- Merge io.h/c back into protocol.h/c
- Change send_data to take Data* argument
- Remove redundant file_load_data from send_chunk
- Move compression into file_send_single_calls
- Move file_receive from multiprocessing.c to file.c
2026-07-16 11:54:20 +02:00
TapTap 02619cca5c refactor: split monolithic modules and extract shared components
- Remove dead socket.c/socket.h (accumulated duplicate symbols)
- Extract compression.h/c (data_compress/decompress from data.c)
- Extract io.h/c (low-level I/O from protocol.c)
- Extract metadata.h/c (unified metadata wire format from file.c, chunk.c, utils.c)
- Split client.c into client_cli.c (CLI parsing) + client_send.c (send logic)
- Move receiver pipeline threads from server.c to multiprocessing.c
- Move to_disk/file_restore_metadata from utils.c to file.c/metadata.c
- Fix const qualifiers on to_disk and str_dup signatures
- Suppress chown unused-result warning
2026-07-15 19:36:26 +02:00
TapTap b3f69208ce cleanup: remove dead code, fix bugs, refactor SSH dest parsing
- Remove debug printf calls from file.c
- Remove dead code: chunk_print, chunk_format, chunk_data_create/delete
- Remove unused config.use_single_send_per_file field
- Fix server_delete() no-op (double-pointer)
- Fix double cast (long)(long) -> ptrdiff_t in chunk.c
- Fix thread return value: thrd_error instead of 1
- Remove unused config param from receive_chunk_enqueue()
- Make queue_double_capacity() static
- Move SSH dest parsing into config.c as config_parse_ssh_dest()
- Fix test_config_ssh_dest to test parsing round-trip
- Remove chunk_format test (obsolete format)
- Replace chunk_data_delete with data_destroy in tests
2026-07-07 19:07:24 +02:00
TapTap a96d48f0ed feat: default log level to WARNING, add -v/--verbose flag 2026-07-06 20:26:20 +02:00
TapTap 372c559ef2 fix: SSH transport - log to stderr, use-after-free, systemd scope conflict 2026-07-06 20:15:36 +02:00
TapTap 1012c5c151 refactor: address PR review feedback
- Replace FASTSYNC_SERVER_IP/FASTSYNC_SERVER_PORT env vars with
  --server-host and --server-port CLI flags
- Simplify STATUS_OK handshake: single disconnect/delete path
- Remove hardcoded default directory (require explicit source/dest)
- Clean up free(NULL) on positional args path
- Fix send_files() resource leak on STATUS_OK failure
2026-07-06 19:54:22 +02:00
TapTap fbd7ccf4dd feat: add SSH transport, rsync-style CLI, and --stdio server mode
- Replace send()/recv() with read()/write() + io_set_fds() for fd abstraction
- Add client_connect_ssh() via socketpair + fork/exec; reap child in client_disconnect()
- Add server --stdio flag for SSH invocation
- Replace handle_arg() with rsync-style positional args (fastsync <src> <dest>)
- Add SSH vs TCP auto-detection via is_remote_dest()
- Add TransportType and ssh_destination to Config, fix uninitialized fields in
  config_receive() (pre-existing multithreaded crash)
- Sender threads now wait for server STATUS_OK before disconnecting
- Add --help, error handling for sendfile+SSH combos
- Add pre-flight checks and Posix Args test case to test.py
- Add test_config_ssh_dest() unit test
2026-07-06 18:42:33 +02:00
TapTap c55d89c40f fix: sendfile path now conditionally sends metadata 2026-07-05 20:52:18 +02:00
TapTap 1d90d3230e fix: make file_send_single_calls metadata conditional, add missing stdbool.h 2026-07-05 20:49:57 +02:00
TapTap 5fa0dc11e1 fix: address PR review comments
- Rename receive_file_receive -> file_receive (comment 62)
- Guard metadata receive with config->use_metadata in server.c (comment 63)
- Add NULL check after malloc in file_load_data (comment 66)
- Make chunk_serialize/chunk_deserialize metadata conditional on use_metadata param,
  thread through chunk_compress/chunk_decompress (comment 65)
- Add missing stdbool.h include to chunk.h
2026-07-05 20:45:34 +02:00
TapTap 660834d453 Merge branch 'main' into metadata-transfer, resolve conflicts
Conflicts resolved:
- config_create signature: added both use_metadata and use_sendfile params
- config_send/config_receive: wire protocol includes both use_metadata and use_sendfile
- client.c: config_create call updated, arg parsing includes both -M/--preserve and -f/--sendfile
- test_config.c: all three config_create calls updated
- file_send_sendfile: use file->data->size instead of removed struct stat
2026-07-05 20:29:04 +02:00
TapTap 4a1e666365 refactor: merge File/FileReceive, add optional metadata transfer with -M flag
- Remove FileReceive struct; use File everywhere with nullable FileMetadata*
- Remove struct stat from File; file size lives in Data->size (data_create_reserve)
- Add FileMetadata struct (mode, uid, gid, mtime) sent conditionally over wire
- Add -M / --preserve flag to client
- Restore permissions, ownership, timestamps on disk write
- Wire format uses per-file present flag for metadata (zero overhead when off)
2026-07-05 20:21:46 +02:00
TapTap a48b3e3a17 refactor: pass use_sendfile to config_create 2026-07-05 16:38:13 +02:00
TapTap 2fb7203c82 feat: sendfile works with -m, error on -f + -c/-s
- load_files_multithreaded skips file_load_data when sendfile is active
- -f + -m works: sendfile bypasses loader, sender uses sendfile directly
- -f combined with -c or -s prints error and exits with rc=1
- Add Sendfile + Multithreading test case
2026-07-05 16:31:08 +02:00
TapTap 01045d7d14 feat: add -f/--sendfile for zero-copy file transfer
- Add file_send_sendfile() using sendfile() syscall to send file
  content directly from fd to socket, bypassing userspace memory
- Add use_sendfile field to Config struct (default false)
- Add -f / --sendfile flag parsing in client.c
- sendfile path in send_chunk() used when -f is set without -c or -s
- send_files_multithreaded() falls back to single-threaded when
  sendfile is enabled (loader pipeline becomes unnecessary)
- Add Sendfile (-f) test case to test.py
2026-07-05 16:25:20 +02:00
TapTap f65cd02627 refactor: --source-dir, --dest-dir, --save-to-disk as CLI args
- Replace env var control (FASTSYNC_SOURCE_DIR, FASTSYNC_DEST_DIR,
  FASTSYNC_SAVE_TO_DISK) with explicit --source-dir, --dest-dir,
  --save-to-disk CLI arguments
- Keep env vars as fallback for backward compatibility
- Fix test.py: split '-c 0' into separate args ['-c', '0']
- Fix test.py: throttled suite failed because systemd-run didn't
  inherit env vars; now all settings are passed as CLI args
- Add env_name to exception/timeout result entries for correct display
2026-07-04 22:51:34 +02:00
TapTap 4867e04677 fix: scanner mid-directory resume bug, -c -s protocol fix, 50 MB test data
- Fix scanner: when chunk fills up mid-directory, save DIR handle so
  remaining files in that directory are not skipped (pre-existing bug)
- Fix -c -s: add STATUS_CHUNK to protocol, send it before chunk data,
  handle it on the server side for both single and multithreaded paths
- Extract chunk_serialize/chunk_deserialize from chunk_compress/decompress
- Remove dead declarations (file_receive_from_buffer, etc.)
- Fix memory leak in receive_file_receive (free -> data_destroy)
- test.py generates ~50 MB of test data across bulk files
- All 8 integration tests + 7 unit tests pass
2026-07-04 22:51:31 +02:00
TapTap c4aeed9f4d feat: implement chunk serialization protocol with -s flag
- Extract chunk_serialize/chunk_deserialize from chunk_compress/chunk_decompress
- Add STATUS_CHUNK to wire protocol for chunk-mode transfers
- Client: -s sends chunk via STATUS_CHUNK (serialized or compressed+serialized)
- Server: handle STATUS_CHUNK in both single-threaded and multithreaded paths
- Fix memory leak in receive_file_receive (free->data_destroy)
- Remove dead declarations: file_receive_from_buffer, file_receive_decompress,
  file_compress, chunk_data_to_disk, unused #defines
2026-07-04 21:38:46 +02:00
Theo Tappe 4d6171d6bf fix: bring project back to a working state 2026-07-04 17:55:03 +02:00
Theo Tappe 75d5f91c6a add data compression and decompression 2026-06-14 16:52:07 +00:00
Theo Tappe 7478d050e3 Improved testing capabilities and fixed missing finished status in multithreading 2026-06-11 15:41:37 +02:00
Theo Tappe f3e93d3fd9 improved testing capabilities and fixed bug that pre ended chunk sending 2026-06-10 22:11:12 +02:00
Theo Tappe 1e5eeb704f First Commit 2026-06-10 16:58:35 +02:00