100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
TapTap	293ba5191e	docs: update README with TLS, incremental sync, bandwidth limiting, server CLI, and corrected build requirements	2026-07-18 18:20:54 +02:00
TapTap	0642a43c2c	Merge pull request 'TLS transport: OpenSSL-based encrypted TCP' (#13) from tls-transport into main	2026-07-18 17:54:25 +02:00
TapTap	fed77f6ce0	ci: use v5 Docker image to force runner to pull updated image with libssl-dev	2026-07-18 17:53:08 +02:00
TapTap	6de0998f69	ci: retrigger CI with updated Docker image	2026-07-18 17:50:55 +02:00
TapTap	43cf0bd81e	ci: add libssl-dev to Docker image for TLS transport	2026-07-18 17:43:46 +02:00
TapTap	907baf3379	fix: initialize ssl/ssl_ctx in client_connect_ssh to prevent use-after-free	2026-07-18 17:39:37 +02:00
TapTap	8b6550f8c6	fix: io_set_ssl NULL in client_disconnect, SSL* type safety in protocol.h	2026-07-18 17:39:37 +02:00
TapTap	20ede4391c	Fix re-review: remove __thread from io_ssl (breaks -m), strtol port parsing, --ca warning	2026-07-18 17:39:37 +02:00
TapTap	864eb1316d	Fix TLS code review issues: SSL cleanup, TLS 1.2 min, CA verify, deprecation guards, server --help, port validation, __thread io_ssl, shared accept loop	2026-07-18 17:39:37 +02:00
TapTap	eff50346fd	TLS transport: OpenSSL-based encrypted TCP ... - New transport_tls.h/c: TLS server (server_create_tls, server_listen_tls) and client (client_connect_tls) using OpenSSL - protocol.c: io_set_ssl() + SSL_read/SSL_write in send_n_data/receive_n_data - transport_tcp.h: ssl/ssl_ctx fields added to Server/Client structs - config.h/c: use_tls, tls_cert, tls_key fields - client_cli.c: --tls, --cert, --key flags - server.c: --tls, --cert, --key, -p flags with TLS support - CMakeLists.txt: OpenSSL::SSL + OpenSSL::Crypto linkage - shell.nix: openssl added to buildInputs	2026-07-18 17:38:55 +02:00
TapTap	25383f893b	Merge pull request 'Incremental sync: --incremental to skip unchanged files' (#14) from incremental-sync into main ... Reviewed-on: #14	2026-07-18 17:36:25 +02:00
TapTap	46c650e9ae	refactor: consolidate _no_path file send variants ... Add bool send_path parameter to file_send_single_calls and file_send_sendfile, remove the _no_path variants. Callers in client_send.c pass true (send path) or false (skip path) based on whether an incremental check already transmitted the path.	2026-07-18 17:27:33 +02:00
TapTap	7876721906	refactor: extract receive_chunk_data and receive_manifest shared helpers ... - receive_chunk_data(fd, config) -> Chunk*: shared receive/decompress/deserialize (eliminates ~25 lines of duplication between server.c and multiprocessing.c) - receive_manifest(fd, config, next_status): moved from server.c to file.c, fixes missing 'Deleting files not in manifest...' log in multiprocessing.c - Removes redundant manual free loop in multiprocessing.c manifest handling (array_list_create(free) destructor already handles this) - server.c and multiprocessing.c: remove local chunk/manifest helpers, remove compression.h include (no longer needed)	2026-07-18 17:19:12 +02:00
TapTap	5ed12f2bf0	refactor: deduplicate server receive logic, extract shared helpers ... - Extract file_save_to_disk() helper (replaces 4x duplicated disk-save boilerplate) - Extract receive_incremental_check() shared helper (deduplicates STATUS_CHECK handling between server.c single-threaded and multiprocessing.c multi-threaded paths) - Break receive_files() into receive_chunk() and receive_manifest() sub-handlers - Add incremental sync test case to test.py - Rebase onto main	2026-07-18 17:19:12 +02:00
TapTap	03d6ba0da5	fix: address review #69 - STATUS_ERROR sends, metadata_receive error handling, extract incremental_check helper	2026-07-18 17:19:12 +02:00
TapTap	f1af88a25c	Fix incremental-sync review: sendfile_no_path, reject -s+incremental, no data mutation, auto -M, STATUS_ERROR handling, PROTOCOL_VERSION bump	2026-07-18 17:19:12 +02:00
TapTap	1905369c9f	Incremental sync: --incremental flag to skip unchanged files ... - New STATUS_CHECK protocol status (value 6) - Client sends path + size + mtime; server replies OK (skip) or NEXT (send) - config.h/c: use_incremental field sent/received over wire - file.c/h: file_send_single_calls_no_path helper for incremental path - client_cli.c: --incremental flag - client_send.c: per-file check before send in both single and chunk paths - server.c: STATUS_CHECK handling in receive_files() - multiprocessing.c: STATUS_CHECK handling in receive_thread() - test.py: incremental sync test case	2026-07-18 17:19:12 +02:00
TapTap	a3acf4b8b6	Merge pull request 'test: add --full flag for lightweight default mode' (#18) from ci-test-light into main ... Reviewed-on: #18	2026-07-18 16:33:02 +02:00
TapTap	478bdd6fe6	add docker as dependency	2026-07-18 16:31:55 +02:00
TapTap	e408b08443	fix: skip netem_reset when no sudo (light mode)	2026-07-18 16:10:52 +02:00
TapTap	2c904bfac5	fix: handle missing ssh in preflight checks	2026-07-18 16:08:26 +02:00
TapTap	e9554c1e69	ci: add integration tests (light mode)	2026-07-18 16:06:22 +02:00
TapTap	8f2205dd30	test: add --full flag, default mode runs 5 fast tests (~0.3s) ... Default mode skips bulk data, network shaping, SSH/rsync comparisons, and feature tests. --full restores original 31-test suite with LAN profiling and all benchmarks.	2026-07-18 16:02:36 +02:00
TapTap	70ab76fe90	Merge pull request 'ci: add build and unit test pipeline' (#17) from actions into main ... Reviewed-on: #17	2026-07-18 15:52:06 +02:00
TapTap	ed7f8ee56e	ci: add g++ and libc6-dev, use v4 image (verified locally)	2026-07-18 15:31:41 +02:00
TapTap	8c7ebbfe32	ci: add make to Docker image, use v3 tag	2026-07-18 15:27:35 +02:00
TapTap	cc09bedc07	ci: use v2 image tag to bypass runner cache	2026-07-18 15:20:57 +02:00
TapTap	5fe40a5242	ci: add node.js to Docker image for checkout action	2026-07-18 15:19:41 +02:00
TapTap	9130ee4b03	ci: use custom Docker image with build tools pre-installed	2026-07-18 15:15:26 +02:00
TapTap	36da0d6adb	ci: restore apt-get update for package resolution	2026-07-18 14:45:03 +02:00
TapTap	e7087d518b	ci: skip apt-get update for faster installs	2026-07-18 14:43:02 +02:00
TapTap	3ea11b5984	ci: add build + unit tests workflow, lower cmake minimum to 3.22	2026-07-18 14:41:27 +02:00
TapTap	33f70f7de3	add demo action	2026-07-18 14:22:58 +02:00
TapTap	45dd62f568	Merge pull request 'Bandwidth throttling: --bwlimit <KB/s> with token bucket' (#15) from bwlimit into main ... Reviewed-on: #15	2026-07-17 10:22:46 +02:00
TapTap	4f8ae5bb70	fix: bw_tokens initialized to io_bwlimit, overflow check for --bwlimit	2026-07-17 10:07:18 +02:00
TapTap	e2dac589a8	Merge pull request 'Add custom opencode agents for FastSync development' (#16) from add-custom-agents into main ... Reviewed-on: #16	2026-07-17 10:02:04 +02:00
TapTap	1b87131e82	Add pr-review and pr-build skills ... - pr-review: read-only code review of PR branches, outputs findings report - pr-build: builds, tests, and fixes PR branches, commits fixes	2026-07-16 17:07:44 +02:00
TapTap	3dcaf0b79d	Fix bwlimit review: remove __thread (breaks -m), validate >0, handle nanosleep EINTR	2026-07-16 17:00:14 +02:00
TapTap	ef6aa143d0	Add custom opencode agents for FastSync development ... - c-reviewer: memory/thread safety, style review for C11 code - test-writer: unit test generation using custom test framework - cmake-expert: CMake build system management - perf-analyst: transfer pipeline performance analysis - protocol-designer: wire protocol design and extension - doc-generator: API docs, protocol specs, usage examples	2026-07-16 16:58:22 +02:00
TapTap	09a76e2a9a	Bandwidth throttling: --bwlimit <KB/s> with token bucket ... - protocol.h/c: io_set_bwlimit() + token bucket in send_n_data (64KB chunks, nanosleep-based deficit compensation) - client_cli.c: --bwlimit <KB/s> flag - test.py: bandwidth limit test case	2026-07-16 15:42:16 +02:00
TapTap	c2436bef59	Merge pull request 'Multi-client server, protocol versioning, streaming decompression' (#11) from server-improvements into main ... Reviewed-on: #11	2026-07-16 15:15:20 +02:00
TapTap	a62894be05	Merge main into server-improvements: resolve conflicts (protocol signatures, error returns)	2026-07-16 15:08:56 +02:00
TapTap	93a5768bf0	Merge pull request 'Fix exit() in library code, add --include/--max-size/--min-size' (#12) from exit-and-select into main ... Reviewed-on: #12	2026-07-16 15:03:38 +02:00
TapTap	bbdf027be8	Fix exit() in library code, add --include/--max-size/--min-size	2026-07-16 14:15:38 +02:00
TapTap	efb98d8964	Multi-client server, protocol versioning, streaming decompression fixes	2026-07-16 13:59:41 +02:00
TapTap	07207cd71d	Merge pull request 'Fair benchmark: random data, exclude dry run from metrics' (#10) from fair-benchmark into main ... Reviewed-on: #10	2026-07-16 13:46:27 +02:00
TapTap	ea551c4a1d	Fair benchmark: random data instead of zeros, exclude dry run from metrics	2026-07-16 13:43:33 +02:00
TapTap	eff00fa021	Merge pull request 'Rsync-compatible flags: -a, -n, -p, --exclude, --delete' (#8) from rsync-flags into main ... Reviewed-on: #8	2026-07-16 13:38:19 +02:00
TapTap	ab24724cb0	Merge pull request 'Update README and add feature tests for rsync-compatible flags' (#9) from update-readme-and-tests into rsync-flags ... Reviewed-on: #9	2026-07-16 13:38:13 +02:00
TapTap	2ff51246c3	Update README and add feature tests for rsync-compatible flags	2026-07-16 13:31:43 +02:00
TapTap	284bf8f8bc	rsync flags: -a, -n, -p, --exclude, --delete ... Feature changes across 15 files: - -a/--archive: enables -c -m -M (no -s, which slows transfers) - -n/--dry-run: scan + print without connecting or transferring - -p <port>: custom SSH port (passed as -p to ssh via execvp) - --exclude <pattern>: glob-based filename filtering in scanner - --delete: sender collects file manifest; receiver deletes unlisted files Protocol: added STATUS_MANIFEST, use_delete field in Config wire format. New utilities: glob_match(), delete_extras() with recursive directory walk. Scanner: accepts exclude patterns; skips matching entries. SSH transport: switched from execlp to execvp for dynamic port arg. Server + multiprocessing: handle STATUS_MANIFEST in both single and multithreaded receive paths. Builds clean, all 7 tests pass.	2026-07-16 12:39:05 +02:00
TapTap	f778d1903c	Merge pull request 'Streaming zstd: switch to ZSTD_compressStream2 / ZSTD_decompressStream' (#7) from streaming-zstd into main ... Reviewed-on: #7	2026-07-16 12:31:49 +02:00
TapTap	04aad76481	Merge pull request 'Quick wins: ControlMaster, -z alias, --progress, fix write warning' (#6) from quick-wins into main ... Reviewed-on: #6	2026-07-16 12:30:13 +02:00
TapTap	52571bc960	streaming zstd: switch to ZSTD_compressStream2 / ZSTD_decompressStream ... Replaced one-shot ZSTD_compress / ZSTD_decompress with streaming API (ZSTD_compressStream2 + ZSTD_decompressStream) in both data_compress and data_decompress. Interface unchanged. Prepares for true streaming transfer where compression can be interleaved with transmission.	2026-07-16 12:29:31 +02:00
TapTap	d1a23fc4df	quick-wins: ControlMaster, -z alias, --progress, fix write warning ... - SSH ControlMaster auto with ControlPath for connection reuse - -z as alias for -c (compression) - --progress flag showing transfer rate in send_files - Fixed unused-result warning on write() in transport_ssh.c	2026-07-16 12:28:16 +02:00
TapTap	78cc735cff	Merge pull request 'Performance tuning: test data, SSH buffer, and configurable chunk size' (#5) from perf-and-tuning into main ... Reviewed-on: #5	2026-07-16 12:24:44 +02:00
TapTap	ca29109664	perf: downsized test data, increased SSH socket buffer, and made chunk size configurable ... - Reduced test data from 50MB to 25MB (test.py) - Increased SSH socketpair buffer to 1MB via setsockopt (transport_ssh.c) - Made chunk size configurable: new Config field, --chunk-size CLI flag, sent over wire, and used in scanner instead of DESIRED_CHUNK_SIZE - Fixed DESIRED_CHUNK_SIZE macro parentheses (chunk.h)	2026-07-16 12:20:02 +02:00
TapTap	270623b90e	Merge pull request 'Phase 1+2: bug fixes and dead code cleanup' (#4) from bugfix-cleanup into main ... Reviewed-on: #4	2026-07-16 12:11:31 +02:00
TapTap	8f4ea62ae7	Phase 1+2: bug fixes and dead code cleanup ... Phase 1 — Bugs: - receive_data: fix unsigned long long vs size_t mismatch - chunk_deserialize: add NULL check in multiprocessing.c and server.c - file_content_to_buffer: add missing fclose on error path - send_files_multithreaded: propagate sender thread result - server: add SIGPIPE handler - to_disk: check fwrite return value - scanner: use S_ISDIR instead of !S_ISREG - scanner: free cur_path before early return Phase 2 — Cleanup: - Remove unused chunk_decompress function - Remove unused array_list_clear function - Remove unused num_connections config field - Remove duplicate FILE_METADATA_WIRE_SIZE macro - Update tests for removed APIs	2026-07-16 12:08:09 +02:00
TapTap	81800fa661	Merge pull request 'refactor: split monolithic modules and extract shared components' (#1) from cleanup into main ... Reviewed-on: #1	2026-07-16 11:58:37 +02:00
TapTap	56d26f083c	Merge pull request 'fixup: address PR review comments' (#3) from fixup-pr-comments into cleanup ... Reviewed-on: #3	2026-07-16 11:58:26 +02:00
TapTap	d36768792a	fixup: address PR review comments ... - Merge io.h/c back into protocol.h/c - Change send_data to take Data* argument - Remove redundant file_load_data from send_chunk - Move compression into file_send_single_calls - Move file_receive from multiprocessing.c to file.c	2026-07-16 11:54:20 +02:00
TapTap	02619cca5c	refactor: split monolithic modules and extract shared components ... - Remove dead socket.c/socket.h (accumulated duplicate symbols) - Extract compression.h/c (data_compress/decompress from data.c) - Extract io.h/c (low-level I/O from protocol.c) - Extract metadata.h/c (unified metadata wire format from file.c, chunk.c, utils.c) - Split client.c into client_cli.c (CLI parsing) + client_send.c (send logic) - Move receiver pipeline threads from server.c to multiprocessing.c - Move to_disk/file_restore_metadata from utils.c to file.c/metadata.c - Fix const qualifiers on to_disk and str_dup signatures - Suppress chown unused-result warning	2026-07-15 19:36:26 +02:00
TapTap	b3f69208ce	cleanup: remove dead code, fix bugs, refactor SSH dest parsing ... - Remove debug printf calls from file.c - Remove dead code: chunk_print, chunk_format, chunk_data_create/delete - Remove unused config.use_single_send_per_file field - Fix server_delete() no-op (double-pointer) - Fix double cast (long)(long) -> ptrdiff_t in chunk.c - Fix thread return value: thrd_error instead of 1 - Remove unused config param from receive_chunk_enqueue() - Make queue_double_capacity() static - Move SSH dest parsing into config.c as config_parse_ssh_dest() - Fix test_config_ssh_dest to test parsing round-trip - Remove chunk_format test (obsolete format) - Replace chunk_data_delete with data_destroy in tests	2026-07-07 19:07:24 +02:00
TapTap	783400d7e9	Merge pull request 'feat: add SSH transport, rsync-style CLI, and --stdio server mode' (#16) from transport-abstraction into main ... Reviewed-on: #16	2026-07-07 18:59:45 +02:00
TapTap	afca5e4f69	test: expand SSH tests to all valid flag combinations	2026-07-07 18:55:47 +02:00
TapTap	e099dc4843	fix: skip wrappers dirs in SSH PATH scan; verify symlink with which	2026-07-06 20:39:59 +02:00
TapTap	c2bdf68429	fix: check SSH symlink by file existence instead of PATH lookup	2026-07-06 20:33:51 +02:00
TapTap	1a12484469	fix: add build dir to PATH in nix-shell; robust SSH PATH detection in check_ssh_localhost	2026-07-06 20:30:38 +02:00
TapTap	a96d48f0ed	feat: default log level to WARNING, add -v/--verbose flag	2026-07-06 20:26:20 +02:00
TapTap	372c559ef2	fix: SSH transport - log to stderr, use-after-free, systemd scope conflict	2026-07-06 20:15:36 +02:00
TapTap	3b92501dc7	feat: add SSH test cases to test.py ... - Add check_ssh_localhost() that tests SSH connectivity and symlinks fastsync-server into PATH if needed - Add SSH_CASES with single-threaded, multithreaded, and compression tests - Support no_server mode in run_single_test for SSH tests - Skip SSH tests gracefully when SSH/localhost/fastsync-server unavailable	2026-07-06 20:03:57 +02:00
TapTap	1012c5c151	refactor: address PR review feedback ... - Replace FASTSYNC_SERVER_IP/FASTSYNC_SERVER_PORT env vars with --server-host and --server-port CLI flags - Simplify STATUS_OK handshake: single disconnect/delete path - Remove hardcoded default directory (require explicit source/dest) - Clean up free(NULL) on positional args path - Fix send_files() resource leak on STATUS_OK failure	2026-07-06 19:54:22 +02:00
TapTap	5b4395ada1	fix: build binaries before pre-flight checks in test.py	2026-07-06 18:52:56 +02:00
TapTap	c0dfa52f67	fix: handle TimeoutExpired in server --stdio pre-flight check	2026-07-06 18:49:02 +02:00
TapTap	fbd7ccf4dd	feat: add SSH transport, rsync-style CLI, and --stdio server mode ... - Replace send()/recv() with read()/write() + io_set_fds() for fd abstraction - Add client_connect_ssh() via socketpair + fork/exec; reap child in client_disconnect() - Add server --stdio flag for SSH invocation - Replace handle_arg() with rsync-style positional args (fastsync <src> <dest>) - Add SSH vs TCP auto-detection via is_remote_dest() - Add TransportType and ssh_destination to Config, fix uninitialized fields in config_receive() (pre-existing multithreaded crash) - Sender threads now wait for server STATUS_OK before disconnecting - Add --help, error handling for sendfile+SSH combos - Add pre-flight checks and Posix Args test case to test.py - Add test_config_ssh_dest() unit test	2026-07-06 18:42:33 +02:00
TapTap	29ad48e5d6	Merge pull request 'docs: update README for metadata transfer, merged File/FileReceive, test.py' (#15) from readme-update into main ... Reviewed-on: #15	2026-07-05 21:11:55 +02:00
TapTap	e4ab2a414c	docs: add LAN/WAN benchmark results table to README	2026-07-05 21:11:33 +02:00
TapTap	c2ada295ca	docs: update README for metadata transfer, merged File/FileReceive, test.py changes	2026-07-05 21:07:58 +02:00
TapTap	24f9a2afb0	Merge pull request 'feat: optional metadata transfer, merged File/FileReceive, sendfile fix, test.py refactor' (#14) from metadata-transfer into main ... Reviewed-on: #14	2026-07-05 21:05:40 +02:00
TapTap	1a6879530e	refactor: shrink test.py to 401 lines, extract helpers, flatten profile/rsync into run_profile	2026-07-05 21:04:27 +02:00
TapTap	2ec8d4202e	refactor: deduplicate test.py logic, fold no-metadata into TEST_CASES, fix sendfile metadata bug	2026-07-05 20:57:21 +02:00
TapTap	b650911945	Merge pull request 'refactor: merge File/FileReceive, add optional metadata transfer (-M)' (#13) from metadata-transfer into main ... Reviewed-on: #13	2026-07-05 20:53:31 +02:00
TapTap	c55d89c40f	fix: sendfile path now conditionally sends metadata	2026-07-05 20:52:18 +02:00
TapTap	1d90d3230e	fix: make file_send_single_calls metadata conditional, add missing stdbool.h	2026-07-05 20:49:57 +02:00
TapTap	12ae8c7f7a	test: add Standard (no metadata) case to test.py	2026-07-05 20:47:32 +02:00
TapTap	5fa0dc11e1	fix: address PR review comments ... - Rename receive_file_receive -> file_receive (comment 62) - Guard metadata receive with config->use_metadata in server.c (comment 63) - Add NULL check after malloc in file_load_data (comment 66) - Make chunk_serialize/chunk_deserialize metadata conditional on use_metadata param, thread through chunk_compress/chunk_decompress (comment 65) - Add missing stdbool.h include to chunk.h	2026-07-05 20:45:34 +02:00
TapTap	660834d453	Merge branch 'main' into metadata-transfer, resolve conflicts ... Conflicts resolved: - config_create signature: added both use_metadata and use_sendfile params - config_send/config_receive: wire protocol includes both use_metadata and use_sendfile - client.c: config_create call updated, arg parsing includes both -M/--preserve and -f/--sendfile - test_config.c: all three config_create calls updated - file_send_sendfile: use file->data->size instead of removed struct stat	2026-07-05 20:29:04 +02:00
TapTap	1f75522d21	Merge pull request 'bench: add speedup metrics vs rsync and theoretical uncompressed throughput' (#11) from benchmark-metrics into main ... Reviewed-on: #11	2026-07-05 20:26:18 +02:00
TapTap	d4fdc16c9e	bench: enable -M (metadata) in benchmark for fair rsync comparison	2026-07-05 20:23:31 +02:00
TapTap	1444eb0f59	Merge pull request 'Update opencode.json' (#12) from update-opencode.json into main ... Reviewed-on: #12	2026-07-05 20:23:20 +02:00
TapTap	11b1765719	Update opencode.json	2026-07-05 20:23:04 +02:00
TapTap	4a1e666365	refactor: merge File/FileReceive, add optional metadata transfer with -M flag ... - Remove FileReceive struct; use File everywhere with nullable FileMetadata* - Remove struct stat from File; file size lives in Data->size (data_create_reserve) - Add FileMetadata struct (mode, uid, gid, mtime) sent conditionally over wire - Add -M / --preserve flag to client - Restore permissions, ownership, timestamps on disk write - Wire format uses per-file present flag for metadata (zero overhead when off)	2026-07-05 20:21:46 +02:00
TapTap	b99b733025	bench: add speedup metrics vs rsync and theoretical uncompressed throughput	2026-07-05 19:57:53 +02:00
TapTap	a1189e51eb	Merge pull request 'bench: run rsync over localhost network for fair comparison' (#10) from networked-rsync into main ... Reviewed-on: #10	2026-07-05 19:50:41 +02:00
TapTap	7d528b1b4d	test: use temp dir for rsync error retry	2026-07-05 19:06:25 +02:00
TapTap	251d2a3407	test: simplify rsync daemon startup, capture stderr	2026-07-05 19:06:12 +02:00
TapTap	266c1e37c7	test: skip systemd-run boilerplate in rsync errors, add raw retry	2026-07-05 18:57:37 +02:00
TapTap	b680a8dff4	test: fix indentation in rsync error handling	2026-07-05 18:46:45 +02:00
TapTap	895927fbad	test: include stdout in rsync error output	2026-07-05 18:44:36 +02:00