Commit Graph

85 Commits

Author SHA1 Message Date
TapTap a62894be05 Merge main into server-improvements: resolve conflicts (protocol signatures, error returns) 2026-07-16 15:08:56 +02:00
TapTap bbdf027be8 Fix exit() in library code, add --include/--max-size/--min-size 2026-07-16 14:15:38 +02:00
TapTap efb98d8964 Multi-client server, protocol versioning, streaming decompression fixes 2026-07-16 13:59:41 +02:00
TapTap 2ff51246c3 Update README and add feature tests for rsync-compatible flags 2026-07-16 13:31:43 +02:00
TapTap 284bf8f8bc rsync flags: -a, -n, -p, --exclude, --delete
Feature changes across 15 files:

- -a/--archive: enables -c -m -M (no -s, which slows transfers)
- -n/--dry-run: scan + print without connecting or transferring
- -p <port>: custom SSH port (passed as -p to ssh via execvp)
- --exclude <pattern>: glob-based filename filtering in scanner
- --delete: sender collects file manifest; receiver deletes unlisted files

Protocol: added STATUS_MANIFEST, use_delete field in Config wire format.
New utilities: glob_match(), delete_extras() with recursive directory walk.
Scanner: accepts exclude patterns; skips matching entries.
SSH transport: switched from execlp to execvp for dynamic port arg.
Server + multiprocessing: handle STATUS_MANIFEST in both single and
multithreaded receive paths.

Builds clean, all 7 tests pass.
2026-07-16 12:39:05 +02:00
TapTap f778d1903c Merge pull request 'Streaming zstd: switch to ZSTD_compressStream2 / ZSTD_decompressStream' (#7) from streaming-zstd into main
Reviewed-on: #7
2026-07-16 12:31:49 +02:00
TapTap 52571bc960 streaming zstd: switch to ZSTD_compressStream2 / ZSTD_decompressStream
Replaced one-shot ZSTD_compress / ZSTD_decompress with streaming
API (ZSTD_compressStream2 + ZSTD_decompressStream) in both
data_compress and data_decompress. Interface unchanged.

Prepares for true streaming transfer where compression can be
interleaved with transmission.
2026-07-16 12:29:31 +02:00
TapTap d1a23fc4df quick-wins: ControlMaster, -z alias, --progress, fix write warning
- SSH ControlMaster auto with ControlPath for connection reuse
- -z as alias for -c (compression)
- --progress flag showing transfer rate in send_files
- Fixed unused-result warning on write() in transport_ssh.c
2026-07-16 12:28:16 +02:00
TapTap ca29109664 perf: downsized test data, increased SSH socket buffer, and made chunk size configurable
- Reduced test data from 50MB to 25MB (test.py)
- Increased SSH socketpair buffer to 1MB via setsockopt (transport_ssh.c)
- Made chunk size configurable: new Config field, --chunk-size CLI flag,
  sent over wire, and used in scanner instead of DESIRED_CHUNK_SIZE
- Fixed DESIRED_CHUNK_SIZE macro parentheses (chunk.h)
2026-07-16 12:20:02 +02:00
TapTap 8f4ea62ae7 Phase 1+2: bug fixes and dead code cleanup
Phase 1 — Bugs:
- receive_data: fix unsigned long long vs size_t mismatch
- chunk_deserialize: add NULL check in multiprocessing.c and server.c
- file_content_to_buffer: add missing fclose on error path
- send_files_multithreaded: propagate sender thread result
- server: add SIGPIPE handler
- to_disk: check fwrite return value
- scanner: use S_ISDIR instead of !S_ISREG
- scanner: free cur_path before early return

Phase 2 — Cleanup:
- Remove unused chunk_decompress function
- Remove unused array_list_clear function
- Remove unused num_connections config field
- Remove duplicate FILE_METADATA_WIRE_SIZE macro
- Update tests for removed APIs
2026-07-16 12:08:09 +02:00
TapTap d36768792a fixup: address PR review comments
- Merge io.h/c back into protocol.h/c
- Change send_data to take Data* argument
- Remove redundant file_load_data from send_chunk
- Move compression into file_send_single_calls
- Move file_receive from multiprocessing.c to file.c
2026-07-16 11:54:20 +02:00
TapTap 02619cca5c refactor: split monolithic modules and extract shared components
- Remove dead socket.c/socket.h (accumulated duplicate symbols)
- Extract compression.h/c (data_compress/decompress from data.c)
- Extract io.h/c (low-level I/O from protocol.c)
- Extract metadata.h/c (unified metadata wire format from file.c, chunk.c, utils.c)
- Split client.c into client_cli.c (CLI parsing) + client_send.c (send logic)
- Move receiver pipeline threads from server.c to multiprocessing.c
- Move to_disk/file_restore_metadata from utils.c to file.c/metadata.c
- Fix const qualifiers on to_disk and str_dup signatures
- Suppress chown unused-result warning
2026-07-15 19:36:26 +02:00
TapTap b3f69208ce cleanup: remove dead code, fix bugs, refactor SSH dest parsing
- Remove debug printf calls from file.c
- Remove dead code: chunk_print, chunk_format, chunk_data_create/delete
- Remove unused config.use_single_send_per_file field
- Fix server_delete() no-op (double-pointer)
- Fix double cast (long)(long) -> ptrdiff_t in chunk.c
- Fix thread return value: thrd_error instead of 1
- Remove unused config param from receive_chunk_enqueue()
- Make queue_double_capacity() static
- Move SSH dest parsing into config.c as config_parse_ssh_dest()
- Fix test_config_ssh_dest to test parsing round-trip
- Remove chunk_format test (obsolete format)
- Replace chunk_data_delete with data_destroy in tests
2026-07-07 19:07:24 +02:00
TapTap a96d48f0ed feat: default log level to WARNING, add -v/--verbose flag 2026-07-06 20:26:20 +02:00
TapTap 372c559ef2 fix: SSH transport - log to stderr, use-after-free, systemd scope conflict 2026-07-06 20:15:36 +02:00
TapTap fbd7ccf4dd feat: add SSH transport, rsync-style CLI, and --stdio server mode
- Replace send()/recv() with read()/write() + io_set_fds() for fd abstraction
- Add client_connect_ssh() via socketpair + fork/exec; reap child in client_disconnect()
- Add server --stdio flag for SSH invocation
- Replace handle_arg() with rsync-style positional args (fastsync <src> <dest>)
- Add SSH vs TCP auto-detection via is_remote_dest()
- Add TransportType and ssh_destination to Config, fix uninitialized fields in
  config_receive() (pre-existing multithreaded crash)
- Sender threads now wait for server STATUS_OK before disconnecting
- Add --help, error handling for sendfile+SSH combos
- Add pre-flight checks and Posix Args test case to test.py
- Add test_config_ssh_dest() unit test
2026-07-06 18:42:33 +02:00
TapTap c55d89c40f fix: sendfile path now conditionally sends metadata 2026-07-05 20:52:18 +02:00
TapTap 1d90d3230e fix: make file_send_single_calls metadata conditional, add missing stdbool.h 2026-07-05 20:49:57 +02:00
TapTap 5fa0dc11e1 fix: address PR review comments
- Rename receive_file_receive -> file_receive (comment 62)
- Guard metadata receive with config->use_metadata in server.c (comment 63)
- Add NULL check after malloc in file_load_data (comment 66)
- Make chunk_serialize/chunk_deserialize metadata conditional on use_metadata param,
  thread through chunk_compress/chunk_decompress (comment 65)
- Add missing stdbool.h include to chunk.h
2026-07-05 20:45:34 +02:00
TapTap 660834d453 Merge branch 'main' into metadata-transfer, resolve conflicts
Conflicts resolved:
- config_create signature: added both use_metadata and use_sendfile params
- config_send/config_receive: wire protocol includes both use_metadata and use_sendfile
- client.c: config_create call updated, arg parsing includes both -M/--preserve and -f/--sendfile
- test_config.c: all three config_create calls updated
- file_send_sendfile: use file->data->size instead of removed struct stat
2026-07-05 20:29:04 +02:00
TapTap 4a1e666365 refactor: merge File/FileReceive, add optional metadata transfer with -M flag
- Remove FileReceive struct; use File everywhere with nullable FileMetadata*
- Remove struct stat from File; file size lives in Data->size (data_create_reserve)
- Add FileMetadata struct (mode, uid, gid, mtime) sent conditionally over wire
- Add -M / --preserve flag to client
- Restore permissions, ownership, timestamps on disk write
- Wire format uses per-file present flag for metadata (zero overhead when off)
2026-07-05 20:21:46 +02:00
TapTap a48b3e3a17 refactor: pass use_sendfile to config_create 2026-07-05 16:38:13 +02:00
TapTap 01045d7d14 feat: add -f/--sendfile for zero-copy file transfer
- Add file_send_sendfile() using sendfile() syscall to send file
  content directly from fd to socket, bypassing userspace memory
- Add use_sendfile field to Config struct (default false)
- Add -f / --sendfile flag parsing in client.c
- sendfile path in send_chunk() used when -f is set without -c or -s
- send_files_multithreaded() falls back to single-threaded when
  sendfile is enabled (loader pipeline becomes unnecessary)
- Add Sendfile (-f) test case to test.py
2026-07-05 16:25:20 +02:00
TapTap c4aeed9f4d feat: implement chunk serialization protocol with -s flag
- Extract chunk_serialize/chunk_deserialize from chunk_compress/chunk_decompress
- Add STATUS_CHUNK to wire protocol for chunk-mode transfers
- Client: -s sends chunk via STATUS_CHUNK (serialized or compressed+serialized)
- Server: handle STATUS_CHUNK in both single-threaded and multithreaded paths
- Fix memory leak in receive_file_receive (free->data_destroy)
- Remove dead declarations: file_receive_from_buffer, file_receive_decompress,
  file_compress, chunk_data_to_disk, unused #defines
2026-07-04 21:38:46 +02:00
TapTap 43c0a1395d Add compression and scanner tests; fix chunk_decompress data ownership bug, data_destroy NULL safety, chunk_compress memory leak 2026-07-04 21:25:34 +02:00
TapTap f189adcbff fix: config_send proto mismatch (use_compression→compression_level) and format string %d→%zu for size_t 2026-07-04 21:11:34 +02:00
TapTap fe8fca47b4 fix: re-apply socket.c file descriptor check after merge 2026-07-04 21:08:30 +02:00
TapTap fefa968945 Merge remote-tracking branch 'origin/main' into fix-critical-issues
# Conflicts:
#	src/shared/chunk.c
2026-07-04 21:07:45 +02:00
TapTap 1486ac7088 fix: chunk_decompress compilation, chunk_compress data ptr, chunk_format serialization, file_destroy leak, test_chunk memcmp 2026-07-04 21:02:59 +02:00
TapTap 7c02b16325 Revert "Merge pull request 'Fix critical issues: implement chunk_decompress and fix socket file descriptor check' (#1) from fix-critical-issues into main"
This reverts commit d176507047, reversing
changes made to ae4c8ff9f0.
2026-07-04 20:51:35 +02:00
TapTap e6c55356c8 Fix critical issues: implement chunk_decompress and fix file descriptor check in server_listen 2026-07-04 20:24:16 +02:00
Theo Tappe 4d6171d6bf fix: bring project back to a working state 2026-07-04 17:55:03 +02:00
Theo Tappe 75d5f91c6a add data compression and decompression 2026-06-14 16:52:07 +00:00
Theo Tappe f3e93d3fd9 improved testing capabilities and fixed bug that pre ended chunk sending 2026-06-10 22:11:12 +02:00
Theo Tappe 1e5eeb704f First Commit 2026-06-10 16:58:35 +02:00