Commit Graph

188 Commits

Author SHA1 Message Date
TapTap d36768792a fixup: address PR review comments
- Merge io.h/c back into protocol.h/c
- Change send_data to take Data* argument
- Remove redundant file_load_data from send_chunk
- Move compression into file_send_single_calls
- Move file_receive from multiprocessing.c to file.c
2026-07-16 11:54:20 +02:00
TapTap 02619cca5c refactor: split monolithic modules and extract shared components
- Remove dead socket.c/socket.h (accumulated duplicate symbols)
- Extract compression.h/c (data_compress/decompress from data.c)
- Extract io.h/c (low-level I/O from protocol.c)
- Extract metadata.h/c (unified metadata wire format from file.c, chunk.c, utils.c)
- Split client.c into client_cli.c (CLI parsing) + client_send.c (send logic)
- Move receiver pipeline threads from server.c to multiprocessing.c
- Move to_disk/file_restore_metadata from utils.c to file.c/metadata.c
- Fix const qualifiers on to_disk and str_dup signatures
- Suppress chown unused-result warning
2026-07-15 19:36:26 +02:00
TapTap b3f69208ce cleanup: remove dead code, fix bugs, refactor SSH dest parsing
- Remove debug printf calls from file.c
- Remove dead code: chunk_print, chunk_format, chunk_data_create/delete
- Remove unused config.use_single_send_per_file field
- Fix server_delete() no-op (double-pointer)
- Fix double cast (long)(long) -> ptrdiff_t in chunk.c
- Fix thread return value: thrd_error instead of 1
- Remove unused config param from receive_chunk_enqueue()
- Make queue_double_capacity() static
- Move SSH dest parsing into config.c as config_parse_ssh_dest()
- Fix test_config_ssh_dest to test parsing round-trip
- Remove chunk_format test (obsolete format)
- Replace chunk_data_delete with data_destroy in tests
2026-07-07 19:07:24 +02:00
TapTap 783400d7e9 Merge pull request 'feat: add SSH transport, rsync-style CLI, and --stdio server mode' (#16) from transport-abstraction into main
Reviewed-on: #16
2026-07-07 18:59:45 +02:00
TapTap afca5e4f69 test: expand SSH tests to all valid flag combinations 2026-07-07 18:55:47 +02:00
TapTap e099dc4843 fix: skip wrappers dirs in SSH PATH scan; verify symlink with which 2026-07-06 20:39:59 +02:00
TapTap c2bdf68429 fix: check SSH symlink by file existence instead of PATH lookup 2026-07-06 20:33:51 +02:00
TapTap 1a12484469 fix: add build dir to PATH in nix-shell; robust SSH PATH detection in check_ssh_localhost 2026-07-06 20:30:38 +02:00
TapTap a96d48f0ed feat: default log level to WARNING, add -v/--verbose flag 2026-07-06 20:26:20 +02:00
TapTap 372c559ef2 fix: SSH transport - log to stderr, use-after-free, systemd scope conflict 2026-07-06 20:15:36 +02:00
TapTap 3b92501dc7 feat: add SSH test cases to test.py
- Add check_ssh_localhost() that tests SSH connectivity and symlinks
  fastsync-server into PATH if needed
- Add SSH_CASES with single-threaded, multithreaded, and compression tests
- Support no_server mode in run_single_test for SSH tests
- Skip SSH tests gracefully when SSH/localhost/fastsync-server unavailable
2026-07-06 20:03:57 +02:00
TapTap 1012c5c151 refactor: address PR review feedback
- Replace FASTSYNC_SERVER_IP/FASTSYNC_SERVER_PORT env vars with
  --server-host and --server-port CLI flags
- Simplify STATUS_OK handshake: single disconnect/delete path
- Remove hardcoded default directory (require explicit source/dest)
- Clean up free(NULL) on positional args path
- Fix send_files() resource leak on STATUS_OK failure
2026-07-06 19:54:22 +02:00
TapTap 5b4395ada1 fix: build binaries before pre-flight checks in test.py 2026-07-06 18:52:56 +02:00
TapTap c0dfa52f67 fix: handle TimeoutExpired in server --stdio pre-flight check 2026-07-06 18:49:02 +02:00
TapTap fbd7ccf4dd feat: add SSH transport, rsync-style CLI, and --stdio server mode
- Replace send()/recv() with read()/write() + io_set_fds() for fd abstraction
- Add client_connect_ssh() via socketpair + fork/exec; reap child in client_disconnect()
- Add server --stdio flag for SSH invocation
- Replace handle_arg() with rsync-style positional args (fastsync <src> <dest>)
- Add SSH vs TCP auto-detection via is_remote_dest()
- Add TransportType and ssh_destination to Config, fix uninitialized fields in
  config_receive() (pre-existing multithreaded crash)
- Sender threads now wait for server STATUS_OK before disconnecting
- Add --help, error handling for sendfile+SSH combos
- Add pre-flight checks and Posix Args test case to test.py
- Add test_config_ssh_dest() unit test
2026-07-06 18:42:33 +02:00
TapTap 29ad48e5d6 Merge pull request 'docs: update README for metadata transfer, merged File/FileReceive, test.py' (#15) from readme-update into main
Reviewed-on: #15
2026-07-05 21:11:55 +02:00
TapTap e4ab2a414c docs: add LAN/WAN benchmark results table to README 2026-07-05 21:11:33 +02:00
TapTap c2ada295ca docs: update README for metadata transfer, merged File/FileReceive, test.py changes 2026-07-05 21:07:58 +02:00
TapTap 24f9a2afb0 Merge pull request 'feat: optional metadata transfer, merged File/FileReceive, sendfile fix, test.py refactor' (#14) from metadata-transfer into main
Reviewed-on: #14
2026-07-05 21:05:40 +02:00
TapTap 1a6879530e refactor: shrink test.py to 401 lines, extract helpers, flatten profile/rsync into run_profile 2026-07-05 21:04:27 +02:00
TapTap 2ec8d4202e refactor: deduplicate test.py logic, fold no-metadata into TEST_CASES, fix sendfile metadata bug 2026-07-05 20:57:21 +02:00
TapTap b650911945 Merge pull request 'refactor: merge File/FileReceive, add optional metadata transfer (-M)' (#13) from metadata-transfer into main
Reviewed-on: #13
2026-07-05 20:53:31 +02:00
TapTap c55d89c40f fix: sendfile path now conditionally sends metadata 2026-07-05 20:52:18 +02:00
TapTap 1d90d3230e fix: make file_send_single_calls metadata conditional, add missing stdbool.h 2026-07-05 20:49:57 +02:00
TapTap 12ae8c7f7a test: add Standard (no metadata) case to test.py 2026-07-05 20:47:32 +02:00
TapTap 5fa0dc11e1 fix: address PR review comments
- Rename receive_file_receive -> file_receive (comment 62)
- Guard metadata receive with config->use_metadata in server.c (comment 63)
- Add NULL check after malloc in file_load_data (comment 66)
- Make chunk_serialize/chunk_deserialize metadata conditional on use_metadata param,
  thread through chunk_compress/chunk_decompress (comment 65)
- Add missing stdbool.h include to chunk.h
2026-07-05 20:45:34 +02:00
TapTap 660834d453 Merge branch 'main' into metadata-transfer, resolve conflicts
Conflicts resolved:
- config_create signature: added both use_metadata and use_sendfile params
- config_send/config_receive: wire protocol includes both use_metadata and use_sendfile
- client.c: config_create call updated, arg parsing includes both -M/--preserve and -f/--sendfile
- test_config.c: all three config_create calls updated
- file_send_sendfile: use file->data->size instead of removed struct stat
2026-07-05 20:29:04 +02:00
TapTap 1f75522d21 Merge pull request 'bench: add speedup metrics vs rsync and theoretical uncompressed throughput' (#11) from benchmark-metrics into main
Reviewed-on: #11
2026-07-05 20:26:18 +02:00
TapTap d4fdc16c9e bench: enable -M (metadata) in benchmark for fair rsync comparison 2026-07-05 20:23:31 +02:00
TapTap 1444eb0f59 Merge pull request 'Update opencode.json' (#12) from update-opencode.json into main
Reviewed-on: #12
2026-07-05 20:23:20 +02:00
TapTap 11b1765719 Update opencode.json 2026-07-05 20:23:04 +02:00
TapTap 4a1e666365 refactor: merge File/FileReceive, add optional metadata transfer with -M flag
- Remove FileReceive struct; use File everywhere with nullable FileMetadata*
- Remove struct stat from File; file size lives in Data->size (data_create_reserve)
- Add FileMetadata struct (mode, uid, gid, mtime) sent conditionally over wire
- Add -M / --preserve flag to client
- Restore permissions, ownership, timestamps on disk write
- Wire format uses per-file present flag for metadata (zero overhead when off)
2026-07-05 20:21:46 +02:00
TapTap b99b733025 bench: add speedup metrics vs rsync and theoretical uncompressed throughput 2026-07-05 19:57:53 +02:00
TapTap a1189e51eb Merge pull request 'bench: run rsync over localhost network for fair comparison' (#10) from networked-rsync into main
Reviewed-on: #10
2026-07-05 19:50:41 +02:00
TapTap 7d528b1b4d test: use temp dir for rsync error retry 2026-07-05 19:06:25 +02:00
TapTap 251d2a3407 test: simplify rsync daemon startup, capture stderr 2026-07-05 19:06:12 +02:00
TapTap 266c1e37c7 test: skip systemd-run boilerplate in rsync errors, add raw retry 2026-07-05 18:57:37 +02:00
TapTap b680a8dff4 test: fix indentation in rsync error handling 2026-07-05 18:46:45 +02:00
TapTap 895927fbad test: include stdout in rsync error output 2026-07-05 18:44:36 +02:00
TapTap 6a609c1b07 test: add text=True to rsync subprocess.run
Without text=True, stderr is bytes, and bytes.split('\n') throws
TypeError: a bytes-like object is required, not 'str'.
2026-07-05 18:42:35 +02:00
TapTap 1c26a7a712 test: robust netem setup and rsync daemon startup
- netem_apply now calls netem_reset() first to clear any leftover qdisc
- Rsync daemon startup polls TCP port instead of fixed sleep
- Dest dir cleared at start of run
2026-07-05 18:33:04 +02:00
TapTap 0c2364bd62 test: clear dest dir at start of run 2026-07-05 18:11:23 +02:00
TapTap 5170155112 bench: run rsync over localhost network for fair comparison
Previously rsync ran as a direct local copy (rsync -aH source/ dest/), bypassing the
loopback interface entirely. This meant tc netem latency/loss/rate limits were never
applied to rsync, making the comparison fundamentally unfair.

Now rsync goes through the network via an rsync daemon on localhost:
  rsync --daemon --no-detach --config=rsyncd.conf
  rsync rsync://localhost:PORT/source/ dest/

The daemon is started before rsync tests and killed afterward. Both the rsync client
and FastSync client run under the same systemd-run disk I/O limits when applicable.
2026-07-05 16:58:13 +02:00
TapTap f77e6bd16a Merge pull request 'bench: network profiles (LAN/WAN), jitter, packet loss, rsync comparison' (#9) from improved-benchmark into main
Reviewed-on: #9
2026-07-05 16:52:26 +02:00
TapTap 7a1c491f7f bench: default is LAN-only, --unlimited and --wan for exclusive profiles 2026-07-05 16:51:11 +02:00
TapTap a90e510424 bench: --wan runs only WAN, not also Unlimited 2026-07-05 16:48:45 +02:00
TapTap 39faff1c4c bench: LAN 20ms ±1ms, 0.1% loss 2026-07-05 16:47:35 +02:00
TapTap 89274c0fe2 bench: network profiles (LAN/WAN), jitter, packet loss, rsync comparison
- Replace single 'Throttled' suite with profiles: Unlimited, LAN (default),
  WAN (--wan flag)
- LAN: 1000mbit, 1ms ±0.1ms delay, 0% loss
- WAN: 100mbit, 50ms ±10ms delay, 1% loss (with --wan)
- Add jitter and packet loss to tc netem config
- Add rsync -aH (archive) and rsync -aHz (archive+compress) as comparison
- Fix verify_transfer to take the received directory directly
- --no-unlimited, --no-lan to skip profiles
- 36 tests total (10 FastSync + 2 rsync across 3 profiles)
2026-07-05 16:45:08 +02:00
TapTap 433a6952e5 Merge pull request 'feat: add -f/--sendfile for zero-copy file transfer' (#8) from sendfile-support into main
Reviewed-on: #8
2026-07-05 16:39:19 +02:00
TapTap a48b3e3a17 refactor: pass use_sendfile to config_create 2026-07-05 16:38:13 +02:00