Author SHA1 Message Date
TapTap f7c6c91e13 fix: set *failed on delta oversize branch; add -m RSF skip test
CI / lint (push) Successful in 20s
CI / sanitizers (address) (push) Successful in 41s
CI / sanitizers (undefined) (push) Successful in 39s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / build-and-test (push) Successful in 1m23s
CI / valgrind (push) Successful in 35s
Review nits from independent review of the four fix branches:
- receive_delta_file STATUS_NEXT oversize branch now sets *failed=true
- receive_incremental_check oversize branch returns NULL (receiver sends the
  single STATUS_ERROR) instead of double-sending
- add multithreaded -m --ignore-existing --remove-source-files integration
  coverage so the writer-thread outcome path is exercised
2026-09-05 13:16:42 +02:00
TapTap 39805e4eee Merge fix/quality-cleanup into dev
fixes #260 (dead code, CLI =form/missing-arg diagnostics)
2026-09-05 13:12:33 +02:00
TapTap 9ef78ef48b Merge fix/perf-delta into dev
fixes #259 (delta_compute hash index)
2026-09-05 13:12:33 +02:00
TapTap 68e7ed57d0 Merge fix/security-hardening into dev
fixes #254 (receiver queue byte-budget) #258 (inplace setuid/truncate)

# Conflicts:
#	src/shared/multiprocessing.c
2026-09-05 13:12:31 +02:00
TapTap 11c591c5aa Merge fix/receiver-correctness into dev
fixes #251 #252 #253 #255 #256 #257 (remove-source-files skips, backup NULL-empty,
partial-dir install, lazy STATUS_CHECK, delta *failed, >64MiB files)
2026-09-05 13:11:41 +02:00
TapTap 1d61a1426e fix: #260 accept --opt=value uniformly and report missing arguments
- Correct misleading doc comments on set_string_option /
  set_positive_int_option / set_nonneg_int_option (they return 0/-1,
  not true/false).
- find_table_option_with_equals() now matches every OPTION_TABLE value
  option (OPT_STRING/OPT_POS_INT/OPT_NONNEG_INT/OPT_ULL), so forms such
  as --max-size=2G, --min-size=1K, --suffix=.bak, --timeout=30,
  --max-depth=5, --backup-dir=X parse instead of dying as 'Unknown option'.
  --max-size/--min-size now accept rsync-style binary suffixes (0 remains a
  valid 'no limit' byte count). Existing special handling for
  --compress-choice, --compress-level, --modify-window=, --chmod=,
  --skip-compress=, --compress-threads= is preserved.
- Options that require a separate value (-p, --exclude, --include,
  --delta-block, --delta-max, --server-port, --bwlimit, --chunk-size,
  --log-file, --exclude-from, --include-from, -T, --skip-compress,
  --compress-threads) now emit an explicit 'missing argument' diagnostic
  instead of falling through to the generic 'Unknown option' branch when
  given as the final argv entry.
- Add unit tests covering the = forms (--max-size=2G, --min-size=1K,
  --suffix=.bak, --timeout=30, --max-depth=5, --backup-dir=X) and a clean
  'missing argument' (not 'Unknown option') diagnostic for trailing
  --exclude/--server-port/--skip-compress/-T.
2026-09-05 12:58:30 +02:00
TapTap beb681c2dc fix: #260 remove dead receive_files() and mkdir_r()
receive_files() in src/server/server.c was a non-static, unprototyped,
zero-caller duplicate of receiver_process()/receiver_receive_files() in
src/server/receiver.c. Delete it along with the includes it uniquely pulled
(chunk.h, metadata.h, sys/stat.h, duplicate quoted unistd.h); remaining code
still uses file.h/config.h/protocol.h (via multiprocessing.h) directly.

mkdir_r() in src/shared/utils.c had zero callers in src/ and tests/; remove
the function and its declaration in utils.h, plus the unused libgen.h include.
2026-09-05 12:58:25 +02:00
TapTap 3704a6adaa test: integration coverage for #251 #252 #253 #257
- remove-source-files keeps sources skipped by --existing/--ignore-existing/
  --update (rsync reference behavior)
- --backup keeps <file>~, --suffix .bak, and --backup-dir backups
- --partial --partial-dir installs completed files in the destination
- a 100 MB file transfers end to end (>64 MiB whole-file cap regression)
2026-09-05 12:46:47 +02:00
TapTap 14c064a093 fix: #251 #252 #253 #255 #256 #257 receiver & remove-source correctness
#251 --remove-source-files deletes sources that were skipped receiver-side
     (--existing/--ignore-existing/--update). The receiver now reports a
     per-file outcome for every processed data file when the sender requests
     removal; the client only unlinks sources the receiver actually wrote.
     add remove_source_files to the wire config and bump the protocol to 2.5.0.
#252 --backup/--suffix/--backup-dir broken by NULL-vs-empty wire loss. Receivers
     canonicalize the empty wire string back to NULL for backup_dir, temp_dir,
     partial_dir and suffix, and --suffix is received unconditionally.
#253 --partial --partial-dir never installed completed files. file_save_to_disk
     now renames a fully written partial-dir file into the real destination.
#255 STATUS_CHECK read the entire old file before the size/mtime quick check.
     Old contents are only read when a checksum compare or delta needs them.
#256 receive_delta_file failure paths did not set *failed, so the caller sent
     STATUS_NEXT and waited for a body that never came. Every NULL return now
     marks the transfer failed.
#257 files >64 MiB could not transfer. Whole-file receive caps raised to the
     256 MiB connection/allocation ceiling (chunk caps stay 64 MiB) and the
     client ignores SIGPIPE so a server-side close surfaces as a clean error.

Unit tests added: config NULL-vs-empty round trip, incremental quick-check
skip/NEXT paths, delta oversize failure, partial-dir install, save-result
skip reporting.
2026-09-05 12:46:44 +02:00
TapTap 2234879b2f fix: #258 normalize mode and truncate on --inplace overwrite
The --inplace branch opened the destination with O_WRONLY|O_CREAT (no
O_TRUNC) and only restored metadata when the sender supplied it.  Two
flaws resulted:
1. An existing destination file kept its original mode when no metadata
   was sent, so setuid/setgid/sticky bits survived an overwrite (a root
   sync could leave a root-owned setuid binary controlled by a client).
2. A shorter payload left stale trailing bytes from the previous version
   because the file was never truncated to the new length.

In the inplace branch of file_to_disk_secure_impl:
- Always trim the file to the new payload length (ftruncate after the
  write) so stale trailing bytes can never survive; sparse targets keep
  their pre-size ftruncate.
- Always normalize the mode after a successful overwrite: apply the
  metadata-derived safe mode when metadata is present (as before), else
  fchmod to a safe default 0644, so setuid/setgid/sticky are cleared in
  both cases.
- The --update newer-destination check still runs before any truncation
  or chmod, preserving the skip semantics.

Adds unit tests in test_file.c: (a) setuid/sticky bits on an existing
destination are cleared after an inplace write with and without metadata,
(b) a shorter inplace payload leaves no trailing stale bytes.
2026-09-05 12:29:46 +02:00
TapTap 98120fc722 fix: #254 bound receiver queue by aggregate payload bytes
The per-connection memory budget (MAX_CONNECTION_MEMORY, 256 MiB) only
charged wire buffers via receive_data_limited.  Decompression buffers and
per-file chunk copies were not accounted for, and the multithreaded
receiver could enqueue up to 100 files (each up to 64 MiB uncompressed)
ahead of a slow disk writer, retaining ~6.4 GiB per connection.  A client
sending highly compressible chunks with little bandwidth could OOM the
host while the reserve never tripped.

Bound the receive pipeline by aggregate payload bytes instead of item
count alone:
- Export MAX_CONNECTION_MEMORY from protocol.h.
- PipelineContextReceiver tracks queued_bytes (payload bytes received but
  not yet released by the disk writer, i.e. queued or in the writer's
  hand) under the existing mutex.
- receiver enqueue now blocks while the queue is full by count OR when
  adding the file would push queued_bytes over the configured byte limit,
  applying backpressure to the sender instead of failing the transfer.
- The disk writer releases the byte budget after each file is freed and
  signals the not-full condition.
- The server sets the byte ceiling to
  MAX_CONNECTION_MEMORY - 2*MAX_CHUNK_SIZE so that the queued payloads
  plus the transient wire/decompression buffers of the one in-flight
  chunk stay within the per-connection budget.

The single-threaded receive path is already bounded: it writes files to
disk before reading the next chunk, so its transient is at most one
chunk's wire + decompressed + copied payload (~3 * MAX_CHUNK_SIZE, below
the budget).  Wire buffers remain charged exactly once by
receive_data_limited; this change does not double charge them.

Adds a deterministic unit test in test_multiprocessing.c proving that an
enqueue which would exceed the byte budget blocks until the writer
releases bytes.
2026-09-05 12:29:41 +02:00
TapTap 5fed0888aa perf: #259 delta_compute hash index over signature blocks 2026-09-05 12:21:26 +02:00
TapTap 9f8b58893b Apply clang-format 18 to merge-conflict resolution code
CI / lint (push) Successful in 15s
CI / sanitizers (address) (push) Successful in 37s
CI / sanitizers (undefined) (push) Successful in 37s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 33s
CI / build-and-test (push) Successful in 1m18s
CI / valgrind (push) Successful in 34s
The Phase 1 merge conflict resolutions introduced formatting that failed
the CI lint job (clang-format 18.1.3). Reformatted with the exact CI
version; no functional changes.
2026-09-05 10:57:41 +02:00
TapTap 48dfd5dfa0 Fix incremental skip regression from modify-window nsec comparison
CI / lint (push) Failing after 3s
CI / build-and-test (push) Skipped
CI / sanitizers (address) (push) Skipped
CI / sanitizers (undefined) (push) Skipped
CI / fuzz-build (push) Skipped
CI / coverage (push) Skipped
CI / valgrind (push) Skipped
metadata_mtime_matches() required exact nanosecond equality at the default
modify_window=0, but destination write-time nsecs never match source
creation-time nsecs unless -M preserves metadata. Compare whole seconds
(rsync's default quick-check) at window 0; keep the subsecond-refined
tolerance for explicit window values.
2026-09-05 10:43:58 +02:00
TapTap 8384a1997b Merge remote-tracking branch 'origin/feat/rsync-delete-during-alias' into dev 2026-09-05 02:53:59 +02:00
TapTap a1981a78ad Merge remote-tracking branch 'origin/feat/rsync-old-dirs-aliases' into dev 2026-09-05 02:48:48 +02:00
TapTap b5646a0b9e Merge remote-tracking branch 'origin/feat/rsync-checksum-choice-alias' into dev 2026-09-05 02:44:04 +02:00
TapTap c6a341bd1e Merge remote-tracking branch 'origin/feat/rsync-partial-progress' into dev 2026-09-05 02:39:03 +02:00
TapTap c1e9509f16 Merge remote-tracking branch 'origin/feat/rsync-old-args' into dev 2026-09-05 02:34:29 +02:00
TapTap 8689778233 Merge remote-tracking branch 'origin/feat/rsync-no-options' into dev 2026-09-05 02:30:50 +02:00
TapTap cf0d10fccb Merge remote-tracking branch 'origin/feat/rsync-info' into dev 2026-09-05 02:25:36 +02:00
TapTap 90112a7585 Merge remote-tracking branch 'origin/feat/max-alloc' into dev 2026-09-05 02:13:35 +02:00
TapTap 3304541337 Merge remote-tracking branch 'origin/feat/compress-threads' into dev 2026-09-05 02:01:43 +02:00
TapTap 6a95efbe41 Merge remote-tracking branch 'origin/feat/skip-compress' into dev 2026-09-04 18:35:19 +02:00
TapTap 3b013bf9d2 Merge remote-tracking branch 'origin/feat/rsync-compression-aliases' into dev 2026-09-04 18:24:21 +02:00
TapTap 4f21498ee6 Merge remote-tracking branch 'origin/feat/chmod' into dev 2026-09-04 18:20:08 +02:00
TapTap 315e572d18 Merge remote-tracking branch 'origin/feat/executability' into dev 2026-09-04 18:13:34 +02:00
TapTap 55172ff6de Merge remote-tracking branch 'origin/feat/ignore-existing' into dev 2026-09-04 17:59:50 +02:00
TapTap c650f9a3d0 Merge remote-tracking branch 'origin/feat/existing' into dev 2026-09-04 17:50:55 +02:00
TapTap 192aff8c46 Merge remote-tracking branch 'origin/feat/update' into dev
# Conflicts:
#	src/client/client_cli.c
#	src/client/usage.c
#	src/shared/file.c
#	src/shared/file.h
#	src/shared/file_receive.c
#	tests/test_client_cli.c
2026-09-04 17:44:07 +02:00
TapTap 6d10116d50 Merge remote-tracking branch 'origin/feat/modify-window' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	src/client/client_cli.c
#	src/client/usage.c
#	src/server/receiver.c
#	src/server/server.c
#	src/shared/config.c
#	src/shared/config.h
#	src/shared/file_receive.c
#	tests/integration/test_features.py
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-04 17:40:16 +02:00
TapTap e42df1bde3 Merge remote-tracking branch 'origin/feat/size-only' into dev
# Conflicts:
#	src/client/client_cli.c
#	src/client/usage.c
#	src/server/receiver.c
#	src/server/server.c
#	src/shared/config.c
#	src/shared/config.h
#	tests/integration/test_features.py
#	tests/test_config.c
2026-09-04 17:30:14 +02:00
TapTap 9fc1e72972 Merge remote-tracking branch 'origin/feat/ignore-times' into dev
# Conflicts:
#	src/shared/config.c
#	tests/integration/test_features.py
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-04 17:27:04 +02:00
TapTap 71f1529222 Merge remote-tracking branch 'origin/feat/whole-file' into dev
# Conflicts:
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-04 17:24:22 +02:00
TapTap 2adfa5a03b Merge remote-tracking branch 'origin/feat/rsync-secluded-args' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	tests/test_client_cli.c
2026-09-04 17:22:53 +02:00
TapTap 47d1cbdae8 Merge remote-tracking branch 'origin/feat/rsync-debug' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	src/client/client_cli.c
#	src/client/usage.c
#	src/shared/log.h
#	src/shared/protocol.c
2026-09-04 17:21:48 +02:00
TapTap b59139589d Merge remote-tracking branch 'origin/feat/rsync-stderr-mode' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	src/shared/log.c
#	src/shared/log.h
#	tests/test_client_cli.c
2026-09-04 17:20:36 +02:00
TapTap 49e4af275f Merge remote-tracking branch 'origin/feat/8-bit-output' into dev
# Conflicts:
#	src/client/client_cli.c
#	src/client/client_send.c
#	src/shared/config.c
#	tests/test_client_cli.c
#	tests/test_config.c
#	tests/test_shared_utils.c
2026-09-04 17:18:01 +02:00
TapTap 8e1bc9bb9c Merge remote-tracking branch 'origin/feat/human-readable' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	src/client/client_send.c
#	tests/test_client_cli.c
2026-09-04 17:16:24 +02:00
TapTap 421a94773f Merge remote-tracking branch 'origin/feat/quiet' into dev
# Conflicts:
#	src/client/client_send.c
2026-09-04 17:14:45 +02:00
TapTap ead4651d12 Merge remote-tracking branch 'origin/feat/remove-source-files' into dev 2026-09-04 17:14:23 +02:00
TapTap 638d953b1d fix: bump protocol version for max alloc wire format
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-04 02:29:06 +02:00
TapTap 04cea295b5 fix: enforce max alloc in delta deserialization
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-04 02:25:26 +02:00
TapTap 820afd334a fix: reject malformed max-alloc sizes
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-04 02:22:00 +02:00
TapTap 2f553a2a43 fix: honor explicit protocol allocation sessions
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-04 02:18:44 +02:00
TapTap f990e86309 fix: make protocol allocation accounting atomic
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-04 02:12:22 +02:00
TapTap 0d306940e1 fix: bind allocation sessions in worker threads
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-04 01:56:34 +02:00
TapTap 552c19d3fe fix: guard skip-compress cleanup
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m17s
CI / valgrind (pull_request) Successful in 34s
2026-09-04 01:49:38 +02:00
TapTap 14c5da98d8 fix: free old snapshot on full transfer fallback
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:51:42 +02:00
TapTap aab45873ac test: cover compression option equals forms
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:47:41 +02:00
TapTap 605f79a450 fix: make rsync info none override verbose
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:44:44 +02:00
TapTap 066c7ed1af fix: address 8-bit output re-review findings
CI / lint (pull_request) Successful in 10s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:40:30 +02:00
TapTap 97a628c3fd docs: document delete-during aliases
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 22:34:52 +02:00
TapTap 9f23b23893 fix: clarify unsupported directory option diagnostics
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:32:09 +02:00
TapTap 2ace6416a7 test: cover configured fsync save path
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:29:40 +02:00
TapTap e491e811e3 fix: enforce max alloc across protocol workers
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:26:29 +02:00
TapTap ec02ee6dde fix: bound compression worker threads
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 22:22:04 +02:00
TapTap 98cbf3495d fix: complete skip-compress protocol handling
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m17s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:19:00 +02:00
TapTap 245a34fa11 fix: guard source removal against file replacement
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:11:18 +02:00
TapTap 68aaf89da7 fix: address update option review findings
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 21:59:55 +02:00
TapTap e37d5b9438 fix: address modify-window review findings
CI / lint (pull_request) Successful in 13s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:54:46 +02:00
TapTap 5b8cdf457b fix: bump protocol version for size-only field
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:51:29 +02:00
TapTap 1ebe261e05 fix: complete ignore-times incremental handling
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:49:10 +02:00
TapTap 05313f00ea fix: prioritize whole-file validation
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 21:46:50 +02:00
TapTap 4152bf379e fix: honor compression choice aliases
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:44:21 +02:00
TapTap f8291d895d fix: classify partial progress as incomplete
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m14s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:40:07 +02:00
TapTap ba236d5b2e fix: clarify secluded args compatibility semantics
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:37:45 +02:00
TapTap ec6160a8b9 fix: propagate SSH child setup failures
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:34:32 +02:00
TapTap 302c3cb664 fix: preserve explicit rsync option negations
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 35s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:30:55 +02:00
TapTap caabd2e314 fix: reject unsupported rsync debug categories
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:28:21 +02:00
TapTap 21e6b1c1bc fix: address rsync info review findings
CI / lint (pull_request) Successful in 12s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:25:56 +02:00
TapTap 5350290f52 fix: reject unsupported rsync client stderr mode
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:21:22 +02:00
TapTap 5b997d5d25 fix: complete 8-bit output negotiation
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 35s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:18:35 +02:00
TapTap fb96c21d93 test: complete human-readable progress coverage
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:14:49 +02:00
TapTap 17b5b31845 fix: complete quiet option handling
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 21:12:43 +02:00
TapTap 48a36b0038 feat: add rsync delete-during alias
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:34:06 +02:00
TapTap ce64fcd4b1 feat: recognize rsync old-dirs aliases
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:31:09 +02:00
TapTap 4982e15969 feat: add rsync-compatible fsync
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 40s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:28:02 +02:00
TapTap d69f5db652 feat: add rsync-compatible max-alloc limit
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:23:45 +02:00
TapTap f443b2986d feat: add compression worker threads
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:15:14 +02:00
TapTap 2a08a7ba5c feat: add skip-compress option
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 17:56:27 +02:00
TapTap 36a373ff5d Add executability preservation option
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 17:39:50 +02:00
TapTap 3b85658024 Add remove-source-files option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 17:33:52 +02:00
TapTap 5e67a721a4 feat: add rsync-compatible update option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / sanitizers (undefined) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 17:07:02 +02:00
TapTap 0b5b1a8643 feat: add rsync modify-window option
CI / lint (pull_request) Successful in 14s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Failing after 1m15s
CI / valgrind (pull_request) Successful in 32s
2026-09-03 17:01:45 +02:00
TapTap 2afb1d9649 feat: add rsync-compatible size-only option
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:54:07 +02:00
TapTap 230401c629 feat: add ignore-times incremental option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:50:07 +02:00
TapTap c91db3266a feat: add whole-file transfer mode
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:47:02 +02:00
TapTap 44ce154e37 feat: add checksum choice alias
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:42:29 +02:00
TapTap cd27886dde feat: add rsync compression aliases
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 16:38:12 +02:00
TapTap a0aff000ad feat: add rsync partial progress alias
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 32s
2026-09-03 16:34:18 +02:00
TapTap 50f9ebb55b feat: add secluded args compatibility option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:31:37 +02:00
TapTap 393a440c78 feat: add rsync old-args compatibility mode
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 16:27:45 +02:00
TapTap 2ff2688ef1 feat: support rsync no-option handling
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:22:18 +02:00
TapTap 5c7d82b79c feat: add rsync debug flags
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:18:46 +02:00
TapTap 4872d425a3 feat: add rsync info flags
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:13:56 +02:00
TapTap 60eb7f11fe feat: add rsync stderr output modes
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:07:51 +02:00
TapTap eb4e9fba1f feat: add 8-bit output option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:03:19 +02:00
TapTap 8ab5026224 feat: add human-readable output flag
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 15:58:48 +02:00
TapTap dbacc4f0e5 feat: add quiet client option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 15:52:36 +02:00
TapTap 190fc5d300 Merge pull request 'fix: harden network security boundaries' (#213) from security-fixes into dev
CI / lint (push) Successful in 11s
CI / sanitizers (undefined) (push) Successful in 37s
CI / sanitizers (address) (push) Successful in 37s
CI / fuzz-build (push) Successful in 14s
CI / coverage (push) Successful in 32s
CI / build-and-test (push) Successful in 1m15s
CI / valgrind (push) Successful in 33s
2026-09-01 20:52:55 +02:00
TapTap 8ae5f82013 style: clang-format file_receive.c
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 35s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 32s
2026-09-01 20:49:39 +02:00
TapTap 0c4855e344 merge: resolve dev (quality refactor) into security-fixes
- file.c split layout retained; security-hardened secure-fs helpers
  (open_secure_parent/to_disk_secure/rename_secure/stat_secure) now live in
  file.c with file_ prefix and are shared with file_receive.c
- file_receive.c takes the security branch's bounded allocations
  (receive_data_limited, data_decompress_limited, size checks) and
  STATUS_ERROR signaling
- file_send.c gains the data consistency check on file->data
- client_validation.c: stricter --tls requiring --ca, log_message style
- utils.c: hardened openat/mkdirat mkdir_r from security branch
2026-09-01 20:46:07 +02:00
TapTap 265f0c0c09 Merge pull request 'refactor: quality cleanup — table-driven CLI, file.c split, scanner helpers, unified error reporting' (#216) from refactor/quality-cleanup into dev
CI / lint (push) Successful in 12s
CI / sanitizers (address) (push) Successful in 35s
CI / sanitizers (undefined) (push) Successful in 35s
CI / fuzz-build (push) Successful in 14s
CI / coverage (push) Successful in 32s
CI / build-and-test (push) Successful in 1m15s
CI / valgrind (push) Successful in 32s
2026-09-01 20:39:31 +02:00
TapTap 047a9a1906 style: apply clang-format 18
CI / lint (pull_request) Successful in 23s
CI / sanitizers (address) (pull_request) Successful in 54s
CI / sanitizers (undefined) (pull_request) Successful in 54s
CI / fuzz-build (pull_request) Successful in 14s
CI / build-and-test (pull_request) Successful in 1m17s
CI / coverage (pull_request) Successful in 31s
CI / valgrind (pull_request) Successful in 33s
2026-08-30 14:20:33 +02:00
TapTap 6d82967c68 refactor: standardize error reporting on the log module
- Add log_perror() helper (context + strerror(errno)) to the log module
- Replace all bare perror() calls with log_perror() so errors are routed
  through the unified logger (stderr sink + optional --log-file sink)
- Convert fprintf(stderr, "Error:/Warning: ...") in client code to
  log_message(); raw fprintf kept only for progress/stats output
2026-08-30 14:06:48 +02:00
TapTap ae95211a05 refactor: unify send_files cleanup and share progress printing
- Extract print_transfer_progress() shared by single-threaded loop and
  the multithreaded progress thread
- Route all send_files() exits through a single send_fail cleanup path
- Fix pre-existing manifest leak on success without --delete
2026-08-30 14:02:12 +02:00
TapTap d639dfdc07 refactor: split file.c into file_send.c and file_receive.c
- file.c: File/FileMetadata lifecycle and local disk helpers (~110 lines)
- file_send.c: client-side send path (file_send_single_calls, file_send_sendfile)
- file_receive.c: server-side receive/save path (file_receive, receive_incremental_check,
  receive_manifest, file_save_to_disk)
- file_types.h holds shared struct definitions; file.h remains an umbrella header
  so existing includes are unaffected
Completes the transfer/protocol separation started in PR #212
2026-08-30 13:59:43 +02:00
TapTap 3fa3e150ce refactor: split parallel_scanner_create_with_options into focused helpers
- parallel_scanner_init(): result queue + sync primitive setup with unwinding
- batch_files(): root-file chunk batching, reusable by other scan paths
- scan_root_directory()/scan_root_entry(): root-dir scanning
- spawn_parallel_workers(): worker thread creation with per-thread arg setup
Main function reduced from ~230 to ~40 lines
2026-08-30 13:56:27 +02:00
TapTap e3e766ba3d refactor: table-driven CLI option parsing in client_cli
- Add OPTION_TABLE for options that map directly to Config fields
  (flag/string/pos-int/nonneg-int/ull kinds)
- Extract parse_ull_arg() replacing 5 duplicated strtoull blocks
- Extract config_add_pattern() replacing duplicated --exclude/--include
  append logic, also reused by read_patterns_from_file()
- parse_args() reduced from ~275 to ~160 lines
2026-08-30 13:53:59 +02:00
TapTap f2917eb163 refactor: remove dead API, rename to_disk/config_is_remote_dest, fix perror newlines
- Delete unused public array_list_extend (made static)
- Delete legacy 16-parameter parallel_scanner_create wrapper; migrate test to parallel_scanner_create_with_options
- Rename to_disk -> file_write_to_disk and is_remote_dest -> config_is_remote_dest for module_action naming convention
- Remove stray newlines in perror calls (perror already appends one)
2026-08-30 13:50:42 +02:00
TapTap 6c4d0246bc merge: resolve origin dev refactor conflicts
CI / lint (pull_request) Successful in 28s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-08-16 09:56:15 +02:00
TapTap 1391564ce7 test: harden allocation checks
CI / lint (pull_request) Successful in 31s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m14s
CI / valgrind (pull_request) Successful in 33s
2026-08-16 09:37:28 +02:00
TapTap d102622e28 fix: close remaining PR review gaps
CI / lint (pull_request) Failing after 31s
CI / build-and-test (pull_request) Has been skipped
CI / sanitizers (address) (pull_request) Has been skipped
CI / sanitizers (undefined) (pull_request) Has been skipped
CI / fuzz-build (pull_request) Has been skipped
CI / coverage (pull_request) Has been skipped
CI / valgrind (pull_request) Has been skipped
2026-08-16 09:35:26 +02:00
TapTap 1abc17142f Merge pull request 'refactor: split transfer and protocol responsibilities' (#212) from refactor/codebase-structure into dev
CI / lint (push) Successful in 30s
CI / sanitizers (undefined) (push) Successful in 38s
CI / sanitizers (address) (push) Successful in 39s
CI / fuzz-build (push) Successful in 14s
CI / coverage (push) Successful in 31s
CI / build-and-test (push) Successful in 1m15s
CI / valgrind (push) Successful in 33s
Reviewed-on: #212
2026-08-16 09:11:58 +02:00
TapTap 059dc2ac75 fix: close remaining PR review gaps
CI / lint (pull_request) Successful in 32s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 13s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 20:38:04 +02:00
TapTap 7cffff0b8b fix: fail closed on authorization setup failure
CI / lint (pull_request) Successful in 30s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 35s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 20:02:09 +02:00
TapTap d3b4c62f51 Merge branch 'dev' into refactor/codebase-structure
CI / lint (pull_request) Successful in 30s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 32s
2026-08-15 15:09:00 +02:00
TapTap 2f804ecfdd fix: address remaining PR 212 review issues
CI / lint (pull_request) Successful in 31s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:52:38 +02:00
TapTap 0c24136052 Merge pull request 'docs: clarify rsync compatibility roadmap' (#215) from docs/rsync-compatible-readme-clean into dev
CI / lint (push) Successful in 34s
CI / sanitizers (address) (push) Successful in 35s
CI / sanitizers (undefined) (push) Successful in 35s
CI / fuzz-build (push) Successful in 15s
CI / build-and-test (push) Successful in 1m15s
CI / coverage (push) Successful in 31s
CI / valgrind (push) Successful in 33s
Reviewed-on: #215
2026-08-15 13:48:01 +02:00
TapTap d19fc68803 Merge branch 'dev' into docs/rsync-compatible-readme-clean
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:47:55 +02:00
TapTap 78876b110e fix: harden remaining review findings
CI / lint (pull_request) Successful in 34s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:44:31 +02:00
TapTap 6b7213db5c docs: align compatibility guidance
CI / lint (pull_request) Failing after 2s
CI / build-and-test (pull_request) Has been skipped
CI / sanitizers (address) (pull_request) Has been skipped
CI / sanitizers (undefined) (pull_request) Has been skipped
CI / fuzz-build (pull_request) Has been skipped
CI / coverage (pull_request) Has been skipped
CI / valgrind (pull_request) Has been skipped
2026-08-15 13:41:23 +02:00
TapTap b3a724afc8 fix: preserve scanner file ownership on failure
CI / lint (pull_request) Successful in 32s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-08-15 13:40:31 +02:00
TapTap daf662cc2d fix: address remaining PR review findings
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:24:13 +02:00
TapTap 98f833980d fix: handle pipeline cancellation failures
CI / lint (pull_request) Successful in 30s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 32s
2026-08-15 13:20:50 +02:00
TapTap de537810e5 docs: fix server destination root examples
CI / lint (pull_request) Failing after 2s
CI / build-and-test (pull_request) Has been skipped
CI / sanitizers (address) (pull_request) Has been skipped
CI / sanitizers (undefined) (pull_request) Has been skipped
CI / fuzz-build (pull_request) Has been skipped
CI / coverage (pull_request) Has been skipped
CI / valgrind (pull_request) Has been skipped
2026-08-15 13:14:28 +02:00
TapTap 298bfe0d0f fix: address delegated review findings
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 35s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 14s
CI / build-and-test (pull_request) Successful in 1m14s
CI / coverage (pull_request) Successful in 30s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:13:57 +02:00
TapTap 604a14f0be fix: close remaining PR review gaps
CI / lint (pull_request) Successful in 30s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 13:13:50 +02:00
TapTap def58554d9 docs: clarify rsync compatibility roadmap
CI / lint (pull_request) Failing after 2s
CI / build-and-test (pull_request) Has been skipped
CI / sanitizers (address) (pull_request) Has been skipped
CI / sanitizers (undefined) (pull_request) Has been skipped
CI / fuzz-build (pull_request) Has been skipped
CI / coverage (pull_request) Has been skipped
CI / valgrind (pull_request) Has been skipped
2026-08-15 12:54:38 +02:00
TapTap 6f8847899c fix: validate remaining wire booleans
CI / lint (pull_request) Successful in 34s
CI / sanitizers (address) (pull_request) Successful in 35s
CI / fuzz-build (pull_request) Successful in 14s
CI / sanitizers (undefined) (pull_request) Successful in 35s
CI / build-and-test (pull_request) Successful in 1m14s
CI / coverage (pull_request) Successful in 31s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 12:49:30 +02:00
TapTap 09454413d4 fix: address PR 212 review issues
CI / lint (pull_request) Successful in 30s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-08-15 12:48:31 +02:00
TapTap ff189aeef2 fix: harden network security boundaries
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 13s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 32s
2026-08-15 12:34:41 +02:00
TapTap 41b624db5a Merge pull request 'fix: satisfy clang-format in CLI test' (#211) from fix/dev-ci-clang-format into dev
CI / lint (push) Successful in 34s
CI / sanitizers (address) (push) Successful in 37s
CI / sanitizers (undefined) (push) Successful in 37s
CI / fuzz-build (push) Successful in 14s
CI / coverage (push) Successful in 31s
CI / build-and-test (push) Successful in 1m16s
CI / valgrind (push) Successful in 34s
Reviewed-on: #211
2026-08-15 11:55:41 +02:00
77 changed files with 8508 additions and 1942 deletions

No files matched your search

+1 -1
View File
@@ -58,7 +58,7 @@ endif()
find_package(OpenSSL REQUIRED) find_package(OpenSSL REQUIRED)
file(GLOB SHARED_SRCS "src/shared/*.c") file(GLOB SHARED_SRCS "src/shared/*.c")
set(FILE_STORE_SRCS src/shared/file_store.c) set(FILE_STORE_SRCS "${CMAKE_CURRENT_SOURCE_DIR}/src/shared/file_store.c")
list(REMOVE_ITEM SHARED_SRCS ${FILE_STORE_SRCS}) list(REMOVE_ITEM SHARED_SRCS ${FILE_STORE_SRCS})
file(GLOB SERVER_SRCS "src/server/*.c") file(GLOB SERVER_SRCS "src/server/*.c")
set(SERVER_RECEIVER_SRCS src/server/receiver.c) set(SERVER_RECEIVER_SRCS src/server/receiver.c)
+229 -41
View File
@@ -1,4 +1,4 @@
# FastSync #FastSync
FastSync is a high-performance file synchronization tool designed to become a FastSync is a high-performance file synchronization tool designed to become a
drop-in replacement for common `rsync` workflows. It keeps the familiar drop-in replacement for common `rsync` workflows. It keeps the familiar
@@ -45,7 +45,7 @@ replacement for every rsync feature or protocol mode.
- Recursive directory scanning. - Recursive directory scanning.
- Rsync-style source and destination arguments. - Rsync-style source and destination arguments.
- SSH transport using `user@host:/path` destinations. - SSH transport using `user@host:destination` paths below the remote authorized root.
- TCP client/server transfers. - TCP client/server transfers.
- Dry runs, excludes, includes, size filters, backups, statistics, and - Dry runs, excludes, includes, size filters, backups, statistics, and
bandwidth limiting. bandwidth limiting.
@@ -66,8 +66,12 @@ replacement for every rsync feature or protocol mode.
- Owner/group, ACL, xattr, hard-link, device, and special-file handling is - Owner/group, ACL, xattr, hard-link, device, and special-file handling is
incomplete or unavailable. incomplete or unavailable.
- Sparse-file handling does not yet preserve all holes correctly. - Sparse-file handling does not yet preserve all holes correctly.
- `--partial`, `--partial-dir`, `--append`, and `--append-verify` are not yet - `--partial`, `--partial-dir`, `-P`, `--append`, and `--append-verify` are not
full rsync-style resumable transfers. yet full rsync-style resumable transfers. Interrupted files are not retained
for resumption.
- `--dirs` is not implemented. Its compatibility aliases `--old-dirs` and
`--old-d` are recognized but rejected explicitly rather than silently using
FastSync's recursive directory behavior.
- Several rsync short options currently have FastSync-specific meanings. Do - Several rsync short options currently have FastSync-specific meanings. Do
not assume every short option is interchangeable yet. not assume every short option is interchangeable yet.
@@ -79,8 +83,172 @@ partial, alternate, and planned behavior.
### Build ### Build
Requirements: C11 compiler, CMake 3.22 or newer, zstd, OpenSSL, pthreads, ### Client
and an SSH client for SSH transport.
| Argument | Description |
|----------|-------------|
| Positional | `<source> <dest>` — automatic SSH detection if dest contains `:` |
| `-c [level]` | Compression with optional level (1–22, default 5) |
| `-z [level]` | Alias for `-c` |
| `-a, --archive` | Archive mode: enables `-c -m -M` (no `-s`) |
| `-m` | Multithreading mode |
| `-s` | Chunk serialization (batch all files per chunk) |
| `--secluded-args` | Accepted as an rsync compatibility option with no effect; `-s` remains chunk serialization. |
| `-f, --sendfile` | Sendfile zero-copy. Incompatible with `-c` / `-s`. TCP only. |
| `-M, --preserve` | Preserve supported file metadata (mode and mtime; ownership and atime are unsupported) |
| `-n, --dry-run` | Scan and print what would be transferred |
| `-p <port>` | SSH port (default: 22) |
| `-v, --verbose` | Enable debug logging |
| `-q, --quiet` | Suppress non-error output |
| `--progress` | Show real-time transfer speed |
| `-P` | Enables partial-transfer mode and progress output (partial retention is incomplete) |
| `--delete` | Delete files on receiver not present in source |
| `--exclude <pattern>` | Exclude files matching glob pattern (repeatable) |
| `--exclude-from <file>` | Read exclude patterns from a file (one per line) |
| `--include <pattern>` | Only transfer files matching glob pattern (repeatable, whitelist) |
| `--max-size <n>` | Skip files larger than n bytes |
| `--min-size <n>` | Skip files smaller than n bytes |
| `--max-alloc <SIZE>` | Maximum single allocation (binary units: B, K, M, G, T, P, E; default 1G) |
| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `-s`. |
| `--existing` | Skip files not already present at the destination; update existing files normally. |
| `--bwlimit <KB/s>` | Bandwidth limit in kilobytes per second |
| `--chunk-size <n>` | Chunk size in bytes (default: 10485760) |
| `--timeout <sec>` | I/O timeout in seconds (default: 30) |
| `--contimeout <sec>` | Connection timeout in seconds (default: 10) |
| `--backup` | Backup existing destination files before overwriting |
| `--backup-dir <dir>` | Target directory for backups (requires `--backup`) |
| `--stats` | Print transfer statistics at end (bytes, files, timing) |
| `-h, --human-readable` | Format transfer byte sizes with binary units |
| `--max-depth <n>` | Maximum directory depth to recurse (0 = unlimited, default: 0) |
| `--log-file <path>` | Write log messages to file instead of stderr |
| `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
| `--dest-dir <path>` | Server destination directory (overrides `FASTSYNC_DEST_DIR`) |
| `--save-to-disk` | Write received files to disk |
| `--server-host <ip>` | Server IP address (default: `127.0.0.1`) |
| `--server-port <n>` | Server port (default: `8080`) |
| `--tls` | Enable TLS encryption |
| `--cert <path>` | TLS certificate file (PEM) |
| `--key <path>` | TLS private key file (PEM) |
| `--ca <path>` | TLS CA certificate file for verification (PEM) |
| `--client-cn <name>` | Required TLS client certificate common name |
### Server
| Argument | Description |
|----------|-------------|
| `--stdio` | Run in stdio mode (for SSH transport; single connection then exits) |
| `-p <port>` | TCP listen port (default: 8080, range: 1–65535) |
| `--tls` | Enable TLS encryption |
| `--cert <path>` | TLS certificate file (PEM) |
| `--key <path>` | TLS private key file (PEM) |
| `--ca <path>` | TLS CA certificate file for verification (PEM) |
| `--destination-root <path>` | Authorized destination root (default: `.`) |
| `--allow-delete` | Permit manifest deletion |
| `--allow-unauthenticated` | Permit plaintext TCP clients |
| `-v, --verbose` | Enable debug logging |
| `--help` | Show help |
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `FASTSYNC_SOURCE_DIR` | — | Source directory fallback |
| `FASTSYNC_DEST_DIR` | — | Destination directory fallback |
| `FASTSYNC_SAVE_TO_DISK` | `false` | Disk persistence fallback |
| `FASTSYNC_SSH_PORT` | `22` | Default SSH port |
| `FASTSYNC_SERVER_HOST` | `127.0.0.1` | Default server host |
| `FASTSYNC_SERVER_PORT` | `8080` | Default server port |
| `FASTSYNC_TLS_CERT` | — | Default TLS certificate path |
| `FASTSYNC_TLS_KEY` | — | Default TLS private key path |
| `FASTSYNC_TLS_CA` | — | Default TLS CA certificate path |
## Implementation Details
### Data Structures
1. **Chunk** — collection of files (~10 MB total by default)
2. **File** — path, content (`Data`), optional `FileMetadata` pointer
3. **FileMetadata** — `mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`;
uid / gid are advisory wire fields and are never applied by the receiver;
atime is unsupported
4. **Config** — runtime parameters (transported over wire, TLS settings excluded). Includes `timeout`, `contimeout`, `quiet`, `backup`, `backup_dir`, `stats`, `max_depth`, `log_file`, `queue_size`.
5. **Queue** — thread-safe bounded queue with condition variables
6. **DirectoryScanner** — recursive BFS traversal with exclude and include pattern support, max-depth enforcement
### Key Algorithms
1. **File scanning** — BFS directory traversal;
entries matched against exclude and include patterns,
max - depth enforced 2. * *Chunking ** — files accumulated until `chunk_size` threshold,
then flushed 3. *
*Compression ** — streaming zstd
via `ZSTD_compressStream2` / `ZSTD_decompressStream` 4. *
*Network protocol ** — status -
code - driven exchange with metadata packing,
keep - alive,
and abort support 5. * *Incremental check ** — client sends `STATUS_CHECK` + path + size +
mtime and,
with `--checksum`, XXH64 content checksum; server compares against destination. Can be batched via `STATUS_CHECK_BATCH` for reduced round-trips.
6. **Bandwidth limiting** — token-bucket algorithm with `nanosleep` throttling on 64 KB write chunks
7. **Metadata restoration** — `chmod()`, `chown()`, `utimensat()` on the receiving side
8. **`--delete`** — sender tracks all sent paths;
receiver walks destination tree and removes unlisted files / directories 9. *
*SSH transport *
* — `socketpair()` + `fork()` + `execvp("ssh",
...)` with `ControlMaster` and port support
10. *
*TLS transport ** — OpenSSL `SSL_CTX` with TLS
1.2 minimum,
mutual CA verification,
transparent `SSL_read`/`SSL_write` via `io_set_ssl()` 11. *
*Path traversal protection ** — `has_path_traversal()` rejects any file path
containing `..` components,
preventing directory escape attacks 12. *
*Connection limiting ** — server tracks active connections and rejects
new ones beyond `max_connections` (default 100)13. *
*Keep
- alive ** — idle connections receive periodic `STATUS_KEEPALIVE` to detect half
- open TCP connections 14. * *Abort handling ** — `SIGINT` sets an abort flag; the next protocol operation sends `STATUS_ABORT` for clean server cleanup
15. **Atomic writes** — files are written to a `.tmp` suffix then atomically renamed via `rename()`, preventing partial files
16. **Backup** — before overwriting, existing files are moved to `--backup-dir` (or same directory with `~` suffix) preserving the original
## Security Features
### Path Traversal Protection
All received file paths are validated by `has_path_traversal()` before any disk operation. Any path containing `..` components is rejected with `STATUS_ERROR`, preventing directory escape attacks.
### TLS Certificate Verification
TLS requires `--ca` and performs mutual TLS verification (`SSL_VERIFY_PEER` with depth 4). Connections without certificate verification are rejected.
### Connection Limits
The server enforces a maximum of 100 concurrent connections (configurable via `max_connections` in `Server`). When the limit is reached, new connections are immediately rejected and closed.
### Abort Handling
If the client receives `SIGINT` (Ctrl+C) during a transfer, it sends `STATUS_ABORT` to the server. The server then cleans up temporary files and exits the child process, preventing incomplete files from remaining on disk.
### Atomic Writes
Received files are written to a temporary path (suffixed with `.tmp`) and then atomically renamed to the final filename via `rename()`. This prevents partial or corrupted files from appearing at the destination if the transfer is interrupted.
## Build Requirements
- C11 compiler
- CMake >= 3.22
- zstd library
- OpenSSL (development headers and libraries)
- pthreads
- SSH client (for SSH transport mode only)
### Installing Dependencies
**Ubuntu/Debian:**
```bash
sudo apt install cmake build-essential libzstd-dev libssl-dev openssh-client
```
**Nix:**
```bash
nix-shell # provides zstd, openssl, cmake, gcc
```
## Building
```bash ```bash
cmake -B build -S . cmake -B build -S .
@@ -98,10 +266,13 @@ cmake --build build -j$(nproc)
### SSH transfer ### SSH transfer
The remote host must have `fastsync-server` available in `PATH`, or use The remote host must have `fastsync-server` available in `PATH`, or use
`--fastsync-server-path`. `--fastsync-server-path`. SSH starts `fastsync-server --stdio` in its remote
working directory, so use a destination below that directory unless the
remote server is otherwise configured with a matching authorized root.
```bash ```bash
./build/client /path/to/source user@host:/path/to/destination ssh user@host 'mkdir -p destination'
./build/client /path/to/source user@host:destination
``` ```
### TCP transfer ### TCP transfer
@@ -109,7 +280,7 @@ The remote host must have `fastsync-server` available in `PATH`, or use
Start the FastSync server: Start the FastSync server:
```bash ```bash
./build/server -p 8080 ./build/server --destination-root /path/to -p 8080
``` ```
Then run the client: Then run the client:
@@ -120,10 +291,12 @@ Then run the client:
--save-to-disk --save-to-disk
``` ```
### TLS transfer Plain TCP requires the explicit `--allow-unauthenticated` server option. Use TLS for
authenticated network connections.
### TLS transfer
```bash ```bash
./build/server --tls --cert server.pem --key server-key.pem -p 8443 ./build/server --destination-root /path/to --tls --cert server.pem --key server-key.pem -p 8443
./build/client --tls --cert client.pem --key client-key.pem --ca ca.pem \ ./build/client --tls --cert client.pem --key client-key.pem --ca ca.pem \
--server-host example.com --server-port 8443 \ --server-host example.com --server-port 8443 \
--source-dir /path/to/source --dest-dir /path/to/destination \ --source-dir /path/to/source --dest-dir /path/to/destination \
@@ -136,34 +309,34 @@ These examples show the intended rsync-style workflow. Options marked as
FastSync-native are optional performance or transport extensions. FastSync-native are optional performance or transport extensions.
```bash ```bash
# Basic synchronization #Basic synchronization
./build/client /source/ /destination/ ./build/client /source/ /destination/
# Archive-style synchronization (current FastSync archive behavior) #Archive - style synchronization(current FastSync archive behavior)
./build/client -a /source/ user@host:/destination/ ./build/client -a /source/ user@host:destination/
# Preview a transfer without changing the destination #Preview a transfer without changing the destination
./build/client -n /source/ /destination/ ./build/client -n /source/ /destination/
# Exclude temporary and object files #Exclude temporary and object files
./build/client --exclude '*.tmp' --exclude '*.o' \ ./build/client --exclude '*.tmp' --exclude '*.o' \
/source/ user@host:/destination/ /source/ user@host:destination/
# Remove destination entries not present in the source #Remove destination entries not present in the source
./build/client --delete /source/ user@host:/destination/ ./build/client --delete /source/ user@host:destination/
# Skip unchanged files using size and modification time #Skip unchanged files using size and modification time
./build/client --incremental /source/ user@host:/destination/ ./build/client --incremental /source/ user@host:destination/
# Verify content when size and time are not sufficient #Verify content when size and time are not sufficient
./build/client --incremental --checksum /source/ user@host:/destination/ ./build/client --incremental --checksum /source/ user@host:destination/
# Preserve supported mode and timestamp metadata #Preserve supported mode and timestamp metadata
./build/client -M /source/ user@host:/destination/ ./build/client -M /source/ user@host:destination/
# Keep backups of overwritten destination files #Keep backups of overwritten destination files
./build/client --backup --backup-dir /backups \ ./build/client --backup --backup-dir backups \
/source/ user@host:/destination/ /source/ user@host:destination/
``` ```
## FastSync Extensions ## FastSync Extensions
@@ -176,6 +349,10 @@ features without changing the meaning of ordinary compatibility options.
| `-m` | Enable the multithreaded scanner/loader/sender pipeline. | | `-m` | Enable the multithreaded scanner/loader/sender pipeline. |
| `-c [level]`, `-z [level]` | Enable streaming zstd compression, levels 1-22. | | `-c [level]`, `-z [level]` | Enable streaming zstd compression, levels 1-22. |
| `--compress-level <n>` | Set the zstd compression level. | | `--compress-level <n>` | Set the zstd compression level. |
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` and `none`. |
| `--zl <n>` | Alias for `--compress-level`. |
| `--skip-compress <list>` | Skip compression for comma-separated suffixes; incompatible with `-s`. |
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
| `--chunk-size <bytes>` | Set the transfer chunk size. | | `--chunk-size <bytes>` | Set the transfer chunk size. |
| `-s` | Enable FastSync chunk serialization. | | `-s` | Enable FastSync chunk serialization. |
| `-f`, `--sendfile` | Use TCP `sendfile()` zero-copy transfer. Incompatible with compression and chunk serialization. | | `-f`, `--sendfile` | Use TCP `sendfile()` zero-copy transfer. Incompatible with compression and chunk serialization. |
@@ -196,6 +373,11 @@ particular, FastSync currently uses `-p` for SSH port, `-s` for chunk
serialization, and `-S` for sparse handling. These meanings must be reconciled serialization, and `-S` for sparse handling. These meanings must be reconciled
before FastSync can claim full rsync CLI compatibility. before FastSync can claim full rsync CLI compatibility.
`--secluded-args` is accepted as a long-form compatibility no-op. It does not
change FastSync's transport or protocol behavior. The rsync short form `-s` is
intentionally not aliased because it remains FastSync's chunk-serialization
option.
## Client Options ## Client Options
### Selection and transfer ### Selection and transfer
@@ -211,14 +393,16 @@ before FastSync can claim full rsync CLI compatibility.
| `--include-from <file>` | Read include patterns from a file. | | `--include-from <file>` | Read include patterns from a file. |
| `--max-size <bytes>` | Skip files larger than the limit. | | `--max-size <bytes>` | Skip files larger than the limit. |
| `--min-size <bytes>` | Skip files smaller than the limit. | | `--min-size <bytes>` | Skip files smaller than the limit. |
| `--max-depth <n>` | Limit recursive scanning depth; zero means unlimited. | | `--max-depth <n>` | Limit recursive scanning depth;
| `--incremental` | Skip files matching destination size and mtime. | zero means unlimited.| | `--incremental` | Skip files matching destination size and mtime.|
| `--checksum` | Include xxHash64 content checks in incremental comparisons. | | `--checksum` | Include xxHash64 content checks in incremental comparisons.| | `--backup` |
| `--backup` | Back up overwritten files. | Back up overwritten files.| | `--backup - dir<dir>` | Store backups under a separate directory.|
| `--backup-dir <dir>` | Store backups under a separate directory. | | `--suffix<suffix>` | Set the backup filename suffix.| | `--partial` |
| `--suffix <suffix>` | Set the backup filename suffix. | Select partial - transfer handling.With `--partial - dir`,
| `--partial` | Keep received data under the configured partial location. Full resume semantics are planned. | completed files are written there;
| `--partial-dir <dir>` | Set the partial transfer directory. | resumable transfers are not implemented.| | `--partial - dir<dir>` |
Set a relative partial - transfer directory below the server destination root;
use with `--partial`. |
| `--inplace` | Write directly to the destination instead of using a temporary file. | | `--inplace` | Write directly to the destination instead of using a temporary file. |
### Metadata and links ### Metadata and links
@@ -226,7 +410,8 @@ before FastSync can claim full rsync CLI compatibility.
| Option | Description | | Option | Description |
|---|---| |---|---|
| `-M`, `--preserve` | Preserve supported file metadata, currently mode and modification time. | | `-M`, `--preserve` | Preserve supported file metadata, currently mode and modification time. |
| `-l`, `--links` | Request symlink preservation; link-target transfer remains incomplete. | | `-l`, `--links` | Request symlink preservation;
link-target transfer remains incomplete. |
| `--copy-links` | Copy symlink referents. | | `--copy-links` | Copy symlink referents. |
| `--safe-links` | Skip symlinks that point outside the transfer tree. | | `--safe-links` | Skip symlinks that point outside the transfer tree. |
| `--copy-unsafe-links` | Copy unsafe symlink referents. | | `--copy-unsafe-links` | Copy unsafe symlink referents. |
@@ -269,6 +454,8 @@ before FastSync can claim full rsync CLI compatibility.
| `--cert <path>` | TLS certificate file. | | `--cert <path>` | TLS certificate file. |
| `--key <path>` | TLS private key file. | | `--key <path>` | TLS private key file. |
| `--ca <path>` | CA file for peer verification. | | `--ca <path>` | CA file for peer verification. |
| `--destination-root <path>` | Confine received files to this server-side root;
defaults to the current directory. |
| `--allow-delete` | Permit client delete manifests. Deletion is refused by default. | | `--allow-delete` | Permit client delete manifests. Deletion is refused by default. |
| `-v`, `--verbose` | Enable debug logging. | | `-v`, `--verbose` | Enable debug logging. |
| `--help` | Print server usage. | | `--help` | Print server usage. |
@@ -297,11 +484,12 @@ before FastSync can claim full rsync CLI compatibility.
## Protocol and Security ## Protocol and Security
FastSync protocol version `2.2.0` is shared by the client and server. The FastSync protocol version `2.5.0` is shared by the client and server. The
current protocol is sender-driven and includes configuration negotiation, current protocol is sender-driven and includes configuration negotiation,
incremental checks, checksums, manifests, keep-alives, abort handling, and including the maximum allocation limit, incremental checks, checksums,
FastSync-native delta messages. Client and server versions must currently manifests, keep-alives, abort handling, per-file remove-source results, and
match exactly. FastSync-native delta messages.
Client and server versions must currently match exactly.
TLS provides encrypted TCP transport. Supplying `--ca` enables certificate TLS provides encrypted TCP transport. Supplying `--ca` enables certificate
verification; without it, traffic is encrypted but peer identity is not verification; without it, traffic is encrypted but peer identity is not
+38 -36
View File
@@ -6,11 +6,12 @@ This document maps rsync's full feature set to FastSync's current implementation
| Status | Count | Description | | Status | Count | Description |
|--------|-------|-------------| |--------|-------|-------------|
| ✅ Implemented | 34 | Feature works end-to-end | | ✅ Implemented | 35 | Feature works end-to-end |
| 🔀 Alt Arg | 3 | Functionality exists but under different flag/semantics | | 🔀 Alt Arg | 4 | Functionality exists but under different flag/semantics |
| ⚠️ Partial | 1 | Flag parsed/stored but behavior incomplete | | ⚠️ Partial | 5 | Flag parsed/stored but behavior incomplete |
| ❌ Not Implemented | 98 | Flag not recognized or no behavior | | 🔄 Compatibility No-op | 1 | Flag is accepted for CLI compatibility but has no effect |
| **Total** | **136** | | | ❌ Not Implemented | 95 | Flag not recognized or no behavior |
| **Total** | **141** | |
--- ---
@@ -20,12 +21,12 @@ This document maps rsync's full feature set to FastSync's current implementation
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `-a`, `--archive` | Archive mode is -rlptgoD | 🔀 Alt Arg | Maps to -c -m -M (compression + multithread + metadata) | | `-a`, `--archive` | Archive mode is -rlptgoD | 🔀 Alt Arg | Maps to -c -m -M (compression + multithread + metadata) |
| `-v`, `--verbose` | Increase verbosity | ✅ Implemented | Sets `log_level=DEBUG` | | `-v`, `--verbose` | Increase verbosity | ✅ Implemented | Sets `log_level=DEBUG` |
| `-q`, `--quiet` | Suppress non-error messages | ❌ Not Implemented | Removed because it had no effect | | `-q`, `--quiet` | Suppress non-error messages | ✅ Implemented | Suppresses client output while preserving errors |
| `--help` | Show help | ✅ Implemented | Prints usage and exits; `-h` is not accepted | | `--help` | Show help | ✅ Implemented | Prints usage and exits; `-h` is not accepted |
| `-V`, `--version` | Print version | ✅ Implemented | | | `-V`, `--version` | Print version | ✅ Implemented | |
| `--info=FLAGS` | Fine-grained info verbosity | ❌ Not Implemented | Removed because it had no effect | | `--info=FLAGS` | Fine-grained info verbosity | ✅ Implemented | Supports `copy`, `misc`, `skip`, `stats`, `all`, and `none`; explicit flags override `--verbose`, and `none` suppresses info output; unsupported names are rejected |
| `--debug=FLAGS` | Fine-grained debug verbosity | ❌ Not Implemented | Removed because it had no effect | | `--debug=FLAGS` | Fine-grained debug verbosity | ✅ Implemented | `io`, `proto`, `pack`, and `util` are supported; `--debug=help` lists flags; other rsync categories are rejected |
| `--stderr=MODE` | Change stderr output mode | ❌ Not Implemented | | | `--stderr=MODE` | Change stderr output mode | ⚠️ Partial | `errors` (default) and `all` are supported; `client` is rejected because FastSync has no rsync message channel |
| `--no-motd` | Suppress daemon MOTD | ❌ Not Implemented | | | `--no-motd` | Suppress daemon MOTD | ❌ Not Implemented | |
| `--exclude=PATTERN` | Exclude files matching pattern | ✅ Implemented | Glob matching in scanner | | `--exclude=PATTERN` | Exclude files matching pattern | ✅ Implemented | Glob matching in scanner |
| `--include=PATTERN` | Include files matching pattern | ✅ Implemented | Glob matching in scanner | | `--include=PATTERN` | Include files matching pattern | ✅ Implemented | Glob matching in scanner |
@@ -36,14 +37,14 @@ This document maps rsync's full feature set to FastSync's current implementation
| Flag | Rsync Description | FastSync Status | Notes | | Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `--stats` | Give transfer stats | ✅ Implemented | Prints file/byte counts | | `--stats` | Give transfer stats | ✅ Implemented | Prints file/byte counts |
| `-h`, `--human-readable` | Human-readable numbers | ❌ Not Implemented | Removed because it had no effect | | `-h`, `--human-readable` | Human-readable numbers | ✅ Implemented | Formats transfer byte sizes using binary units |
| `-i`, `--itemize-changes` | Per-file change summary | ❌ Not Implemented | Removed because it had no effect | | `-i`, `--itemize-changes` | Per-file change summary | ❌ Not Implemented | Removed because it had no effect |
| `--progress` | Show progress | ✅ Implemented | Progress callback in sender | | `--progress` | Show progress | ✅ Implemented | Progress callback in sender |
| `-P` | Same as --partial --progress | ❌ Not Implemented | | | `-P` | Same as --partial --progress | ⚠️ Partial | Parses and enables progress, but interrupted files are not retained for resumable transfers |
| `--out-format=FORMAT` | Custom output format | ❌ Not Implemented | Removed because it had no effect | | `--out-format=FORMAT` | Custom output format | ❌ Not Implemented | Removed because it had no effect |
| `--log-file=FILE` | Log to file | ✅ Implemented | `log_file` config field | | `--log-file=FILE` | Log to file | ✅ Implemented | `log_file` config field |
| `--log-file-format=FMT` | Log format | ❌ Not Implemented | | | `--log-file-format=FMT` | Log format | ❌ Not Implemented | |
| `--8-bit-output` | Leave high-bit chars unescaped | ❌ Not Implemented | | | `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Implemented | Applies to displayed paths and protocol debug output |
| `--list-only` | List files instead of copying | ❌ Not Implemented | Removed because it had no effect | | `--list-only` | List files instead of copying | ❌ Not Implemented | Removed because it had no effect |
## 3. File Selection ## 3. File Selection
@@ -58,11 +59,11 @@ This document maps rsync's full feature set to FastSync's current implementation
| `--max-size=SIZE` | Skip files larger than SIZE | ✅ Implemented | `max_size` in scanner | | `--max-size=SIZE` | Skip files larger than SIZE | ✅ Implemented | `max_size` in scanner |
| `--min-size=SIZE` | Skip files smaller than SIZE | ✅ Implemented | `min_size` in scanner | | `--min-size=SIZE` | Skip files smaller than SIZE | ✅ Implemented | `min_size` in scanner |
| `-I`, `--ignore-times` | Don't skip files matching size+time | ❌ Not Implemented | | | `-I`, `--ignore-times` | Don't skip files matching size+time | ❌ Not Implemented | |
| `--size-only` | Skip based on size only | ❌ Not Implemented | | | `--size-only` | Skip based on size only | ✅ Implemented | With `--incremental`, ignores mtime |
| `-@`, `--modify-window=NUM` | Mod-time comparison accuracy | ❌ Not Implemented | | | `-@`, `--modify-window=NUM` | Mod-time comparison accuracy | ✅ Implemented | Whole-second tolerance with nanosecond-aware comparisons |
| `--existing` | Skip creating new files on receiver | ❌ Not Implemented | | | `--existing` | Skip creating new files on receiver | ✅ Implemented | Existing destination files continue through normal update handling |
| `--ignore-existing` | Skip updating existing files | ❌ Not Implemented | | | `--ignore-existing` | Skip updating existing files | ❌ Not Implemented | |
| `--remove-source-files` | Sender removes synced files | ❌ Not Implemented | | | `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Implemented | |
## 4. Directory Options ## 4. Directory Options
@@ -71,7 +72,7 @@ This document maps rsync's full feature set to FastSync's current implementation
| `-r`, `--recursive` | Recurse into directories | ✅ Implemented | Default behavior | | `-r`, `--recursive` | Recurse into directories | ✅ Implemented | Default behavior |
| `-R`, `--relative` | Use relative path names | ❌ Not Implemented | Removed because it had no effect | | `-R`, `--relative` | Use relative path names | ❌ Not Implemented | Removed because it had no effect |
| `--no-implied-dirs` | Don't send implied dirs with -R | ❌ Not Implemented | | | `--no-implied-dirs` | Don't send implied dirs with -R | ❌ Not Implemented | |
| `-d`, `--dirs` | Transfer dirs without recursing | ❌ Not Implemented | | | `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ❌ Not Implemented | The aliases are recognized and rejected explicitly; they depend on the unimplemented `--dirs` behavior |
| `--mkpath` | Create missing path components | ❌ Not Implemented | | | `--mkpath` | Create missing path components | ❌ Not Implemented | |
## 5. Transfer Modifications ## 5. Transfer Modifications
@@ -101,7 +102,7 @@ This document maps rsync's full feature set to FastSync's current implementation
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `--delete` | Delete extraneous files from dest | ✅ Implemented | `use_delete` config field | | `--delete` | Delete extraneous files from dest | ✅ Implemented | `use_delete` config field |
| `--delete-before` | Delete before transfer | ❌ Not Implemented | Removed because it had no effect | | `--delete-before` | Delete before transfer | ❌ Not Implemented | Removed because it had no effect |
| `--delete-during` | Delete during transfer | ❌ Not Implemented | | | `--del`, `--delete-during` | Delete during transfer | ❌ Not Implemented | Both flags are recognized but rejected; delete timing is not implemented |
| `--delete-delay` | Find deletions during, delete after | ❌ Not Implemented | | | `--delete-delay` | Find deletions during, delete after | ❌ Not Implemented | |
| `--delete-after` | Delete after transfer | ❌ Not Implemented | Removed because it had no effect | | `--delete-after` | Delete after transfer | ❌ Not Implemented | Removed because it had no effect |
| `--delete-excluded` | Also delete excluded files | ❌ Not Implemented | Removed because it had no effect | | `--delete-excluded` | Also delete excluded files | ❌ Not Implemented | Removed because it had no effect |
@@ -119,8 +120,8 @@ This document maps rsync's full feature set to FastSync's current implementation
| `-o`, `--owner` | Preserve owner | ✅ Implemented | Part of -M | | `-o`, `--owner` | Preserve owner | ✅ Implemented | Part of -M |
| `-g`, `--group` | Preserve group | ✅ Implemented | Part of -M | | `-g`, `--group` | Preserve group | ✅ Implemented | Part of -M |
| `-t`, `--times` | Preserve modification times | ✅ Implemented | Part of -M | | `-t`, `--times` | Preserve modification times | ✅ Implemented | Part of -M |
| `-E`, `--executability` | Preserve executability | ❌ Not Implemented | | | `-E`, `--executability` | Preserve executability | ✅ Implemented | Preserves executable permission bits (implies metadata preservation) |
| `--chmod=CHMOD` | Affect file permissions | ❌ Not Implemented | | | `--chmod=CHMOD` | Affect file permissions | ✅ Implemented | Supports numeric and symbolic `ugo` `rwx` changes; retains receiver safety masking |
| `-A`, `--acls` | Preserve ACLs | ❌ Not Implemented | Removed because it had no effect | | `-A`, `--acls` | Preserve ACLs | ❌ Not Implemented | Removed because it had no effect |
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Not Implemented | Removed because it had no effect | | `-X`, `--xattrs` | Preserve extended attributes | ❌ Not Implemented | Removed because it had no effect |
| `-H`, `--hard-links` | Preserve hard links | ❌ Not Implemented | Removed because it had no effect | | `-H`, `--hard-links` | Preserve hard links | ❌ Not Implemented | Removed because it had no effect |
@@ -152,14 +153,14 @@ This document maps rsync's full feature set to FastSync's current implementation
| Flag | Rsync Description | FastSync Status | Notes | | Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `-S`, `--sparse` | Sparse block handling | ✅ Implemented | `preserve_sparse` config field | | `-S`, `--sparse` | Sparse block handling | ⚠️ Partial | Flag is accepted, but full hole preservation is not implemented |
| `--preallocate` | Allocate dest files before writing | ❌ Not Implemented | | | `--preallocate` | Allocate dest files before writing | ❌ Not Implemented | |
## 11. Checksum & Comparison ## 11. Checksum & Comparison
| Flag | Rsync Description | FastSync Status | Notes | | Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `--checksum` | Skip based on checksum | ❌ Not Implemented | Removed because it had no effect; `-c` means compression | | `--checksum` | Skip based on checksum | ✅ Implemented | With `--incremental`, compares xxHash64 content checksums; `-c` remains compression |
| `--checksum-choice=STR` | Choose checksum algorithm | ❌ Not Implemented | xxHash used internally | | `--checksum-choice=STR` | Choose checksum algorithm | ❌ Not Implemented | xxHash used internally |
| `--compare-dest=DIR` | Compare dest files relative to DIR | ❌ Not Implemented | Removed because it had no effect | | `--compare-dest=DIR` | Compare dest files relative to DIR | ❌ Not Implemented | Removed because it had no effect |
| `--copy-dest=DIR` | Include copies of unchanged files | ❌ Not Implemented | Removed because it had no effect | | `--copy-dest=DIR` | Include copies of unchanged files | ❌ Not Implemented | Removed because it had no effect |
@@ -171,10 +172,10 @@ This document maps rsync's full feature set to FastSync's current implementation
| Flag | Rsync Description | FastSync Status | Notes | | Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `-z`, `--compress` | Compress file data | 🔀 Alt Arg | Always uses zstd (rsync supports multiple algorithms) | | `-z`, `--compress` | Compress file data | 🔀 Alt Arg | Always uses zstd (rsync supports multiple algorithms) |
| `--compress-choice=STR` | Choose compression algorithm | ❌ Not Implemented | Removed because it had no effect; FastSync always uses zstd | | `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ✅ Implemented | FastSync supports `zstd` and `none` |
| `--compress-level=NUM` | Set compression level | ✅ Implemented | 1-22, default 5 | | `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Implemented | 1-22, default 5 |
| `--compress-threads=NUM` | Set compression threads | ❌ Not Implemented | | | `--compress-threads=NUM` | Set compression threads | ❌ Not Implemented | |
| `--skip-compress=LIST` | Skip compress for suffixes | ❌ Not Implemented | Internal skip for hardcoded types; not user-configurable | | `--skip-compress=LIST` | Skip compress for suffixes | ✅ Implemented | Comma-separated, case-insensitive suffix list; empty list skips none; incompatible with FastSync chunk serialization (`-s`) |
## 13. Connectivity ## 13. Connectivity
@@ -211,7 +212,7 @@ This document maps rsync's full feature set to FastSync's current implementation
| Max data/string/chunk sizes | Prevent OOM attacks | ✅ Implemented | Per-message limits | | Max data/string/chunk sizes | Prevent OOM attacks | ✅ Implemented | Per-message limits |
| Per-connection memory limit | 1GB per connection | ✅ Implemented | `MAX_CONNECTION_MEMORY` | | Per-connection memory limit | 1GB per connection | ✅ Implemented | `MAX_CONNECTION_MEMORY` |
| `--trust-sender` | Trust remote sender's file list | ❌ Not Implemented | | | `--trust-sender` | Trust remote sender's file list | ❌ Not Implemented | |
| `--old-args` | Disable modern arg protection | ❌ Not Implemented | | | `--old-args` | Disable modern arg protection | ✅ Implemented | SSH-only legacy mode; restores raw remote command construction and permits shell interpretation of the configured server path |
| `--ignore-missing-args` | Ignore missing source args | ❌ Not Implemented | | | `--ignore-missing-args` | Ignore missing source args | ❌ Not Implemented | |
| `--delete-missing-args` | Delete missing source args | ❌ Not Implemented | | | `--delete-missing-args` | Delete missing source args | ❌ Not Implemented | |
@@ -229,12 +230,12 @@ This document maps rsync's full feature set to FastSync's current implementation
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `--stop-after=MINS` | Stop after N minutes | ❌ Not Implemented | | | `--stop-after=MINS` | Stop after N minutes | ❌ Not Implemented | |
| `--stop-at=TIME` | Stop at specified time | ❌ Not Implemented | | | `--stop-at=TIME` | Stop at specified time | ❌ Not Implemented | |
| `--fsync` | Fsync every written file | ❌ Not Implemented | | | `--fsync` | Fsync every written file before publication | ✅ Implemented | |
| `--protocol=NUM` | Force older protocol version | ❌ Not Implemented | | | `--protocol=NUM` | Force older protocol version | ❌ Not Implemented | |
| `--iconv=CONVERT_SPEC` | Charset conversion | ❌ Not Implemented | | | `--iconv=CONVERT_SPEC` | Charset conversion | ❌ Not Implemented | |
| `--checksum-seed=NUM` | Set checksum seed | ❌ Not Implemented | | | `--checksum-seed=NUM` | Set checksum seed | ❌ Not Implemented | |
| `-s`, `--secluded-args` | Use protocol to send args | ❌ Not Implemented | | | `--secluded-args` | Use protocol to send args | 🔄 Compatibility No-op | Accepted for CLI compatibility; it does not change FastSync transport or protocol behavior. `-s` remains chunk serialization. |
| `--no-OPTION` | Turn off implied option | ❌ Not Implemented | | | `--no-OPTION` | Turn off implied option | ✅ Supported | Supported boolean FastSync options and archive-implied options; unsafe or value-taking options are rejected. |
--- ---
@@ -247,13 +248,14 @@ Ranked by user demand, implementation complexity, and interoperability impact:
| 1 | `--whole-file` / `-W` | Low | High — users expect opt-out of delta | | 1 | `--whole-file` / `-W` | Low | High — users expect opt-out of delta |
| 2 | `--ignore-times` / `-I` | Low | Medium — useful for forcing re-transfer | | 2 | `--ignore-times` / `-I` | Low | Medium — useful for forcing re-transfer |
| 3 | `--size-only` | Low | Medium — common migration scenario | | 3 | `--size-only` | Low | Medium — common migration scenario |
| 4 | `--existing` / `--ignore-existing` | Low | Medium — common sync patterns | | 4 | `--ignore-existing` | Low | Medium — common sync patterns |
| 5 | `--remove-source-files` | Low | High — common for moves/backup | | 5 | `--existing` | Low | Medium — common sync patterns |
| 6 | `--delete-during` | Medium | High — performance improvement | | 6 | `--remove-source-files` | Low | High — common for moves/backup |
| 7 | `--delay-updates` | Medium | High — atomic updates | | 7 | `--delete-during` | Medium | High — performance improvement |
| 8 | `--chmod` | Low | Medium — permission flexibility | | 8 | `--delay-updates` | Medium | High — atomic updates |
| 9 | `--executability` / `-E` | Low | Low — simple flag | | 9 | `--chmod` | Low | Medium — permission flexibility |
| 10 | `--skip-compress` | Low | Medium — performance tuning | | 10 | `--executability` / `-E` | Low | Low — simple flag |
| 11 | `--skip-compress` | Low | Medium — performance tuning |
--- ---
+775 -217
View File
File diff suppressed because it is too large. Load diff
+378 -111
View File
@@ -16,11 +16,14 @@
#include "transport_ssh.h" #include "transport_ssh.h"
#include "transport_tls.h" #include "transport_tls.h"
#include "utils.h" #include "utils.h"
#include <fcntl.h>
#include <limits.h>
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <threads.h> #include <threads.h>
#include <time.h> #include <time.h>
#include <sys/stat.h>
#include <unistd.h> #include <unistd.h>
#define STREAM_THRESHOLD (64ULL * 1024 * 1024) #define STREAM_THRESHOLD (64ULL * 1024 * 1024)
@@ -28,6 +31,14 @@
/* Forward declaration for progress-reporting thread used in multithreaded send. */ /* Forward declaration for progress-reporting thread used in multithreaded send. */
static int progress_thread_fn(void* arg); static int progress_thread_fn(void* arg);
static const char* display_bytes(unsigned long long bytes, bool human_readable, char* buffer,
size_t buffer_size) {
if (human_readable && format_human_bytes(bytes, buffer, buffer_size))
return buffer;
snprintf(buffer, buffer_size, "%.1f MB", bytes / 1048576.0);
return buffer;
}
static ScannerOptions scanner_options_from_config(const Config* config, int num_threads) { static ScannerOptions scanner_options_from_config(const Config* config, int num_threads) {
ScannerOptions options = { ScannerOptions options = {
config->use_metadata, config->chunk_size, config->exclude_patterns, config->use_metadata, config->chunk_size, config->exclude_patterns,
@@ -42,11 +53,11 @@ static ScannerOptions scanner_options_from_config(const Config* config, int num_
static Client* connect_transfer_client(const Config* config) { static Client* connect_transfer_client(const Config* config) {
if (config->transport == TRANSPORT_SSH) { if (config->transport == TRANSPORT_SSH) {
if (config->use_sendfile) { if (config->use_sendfile) {
fprintf(stderr, "Error: -f/--sendfile is not supported with SSH transport\n"); log_message(LOG_LEVEL_ERROR, "-f/--sendfile is not supported with SSH transport");
return NULL; return NULL;
} }
return client_connect_ssh(config->ssh_destination, config->ssh_port, return client_connect_ssh(config->ssh_destination, config->ssh_port,
config->fastsync_server_path); config->fastsync_server_path, config->old_args);
} }
Client* client = client_create(); Client* client = client_create();
@@ -60,6 +71,7 @@ static Client* connect_transfer_client(const Config* config) {
connected = client_connect(client, config->server_host, config->server_port); connected = client_connect(client, config->server_host, config->server_port);
} }
if (!connected) { if (!connected) {
client_disconnect(client);
client_delete(client); client_delete(client);
return NULL; return NULL;
} }
@@ -97,10 +109,92 @@ static bool add_chunk_to_manifest(ArrayList* manifest, const Chunk* chunk) {
return true; return true;
} }
static bool finalize_transfer(Client* client) { /* (finalize_transfer is defined after the SourceFile helpers below.) */
Status status;
return send_status(client->file_descriptor, STATUS_FINISHED) && typedef struct SourceFile {
receive_status(client->file_descriptor, &status) && status == STATUS_OK; char* path;
dev_t device;
ino_t inode;
bool skipped; /* receiver reported the file was not written */
} SourceFile;
static void source_file_destroy(void* item) {
SourceFile* source = item;
if (source) {
free(source->path);
free(source);
}
}
/* Remove only the same regular source file that was sent. */
static void remove_transferred_sources(const Config* config, ArrayList* paths) {
if (!config->remove_source_files || !paths)
return;
for (int i = 0; i < paths->size; i++) {
SourceFile* source = paths->items[i];
if (source->skipped)
continue;
const char* slash = strrchr(source->path, '/');
const char* leaf = slash ? slash + 1 : source->path;
char parent[PATH_MAX];
if (slash) {
size_t parent_length = (size_t)(slash - source->path);
if (parent_length == 0)
parent_length = 1;
if (parent_length >= sizeof(parent))
continue;
memcpy(parent, source->path, parent_length);
parent[parent_length] = '\0';
} else {
(void)snprintf(parent, sizeof(parent), ".");
}
int dirfd = open(parent, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
if (dirfd < 0)
continue;
struct stat st;
if (fstatat(dirfd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0 || !S_ISREG(st.st_mode) ||
st.st_dev != source->device || st.st_ino != source->inode) {
close(dirfd);
continue;
}
if (unlinkat(dirfd, leaf, 0) != 0)
log_message(LOG_LEVEL_WARNING, "Could not remove source file %s", source->path);
close(dirfd);
}
}
static SourceFile* source_file_create(const File* file) {
if (!file || !file->path)
return NULL;
struct stat st;
if (lstat(file->path, &st) != 0 || !S_ISREG(st.st_mode))
return NULL;
SourceFile* source = malloc(sizeof(*source));
if (!source)
return NULL;
source->path = str_dup(file->path);
source->device = st.st_dev;
source->inode = st.st_ino;
source->skipped = false;
if (!source->path) {
source_file_destroy(source);
return NULL;
}
return source;
}
static bool remember_source_file(ArrayList* paths, const File* file) {
if (!paths || !file || !file->path)
return true;
SourceFile* source = source_file_create(file);
if (!source)
return true;
if (!array_list_add(paths, source)) {
source_file_destroy(source);
return false;
}
return true;
} }
static void mark_sender_done(PipelineContextSender* context) { static void mark_sender_done(PipelineContextSender* context) {
@@ -109,6 +203,33 @@ static void mark_sender_done(PipelineContextSender* context) {
mtx_unlock(&context->mutex_progress); mtx_unlock(&context->mutex_progress);
} }
/* Send the final STATUS_FINISHED frame and await the receiver's verdict.
When --remove-source-files is active the receiver acknowledges each data
file it processed, in send order: STATUS_NEXT means the file was written,
STATUS_OK means the file was skipped/unchanged. Skipped sources are marked
so the later removal pass keeps them. */
static bool finalize_transfer(Client* client, const Config* config, ArrayList* remove_sources) {
if (!send_status(client->file_descriptor, STATUS_FINISHED))
return false;
if (config->remove_source_files && remove_sources) {
for (int i = 0; i < remove_sources->size; i++) {
Status per_file;
if (!receive_status(client->file_descriptor, &per_file))
return false;
if (per_file == STATUS_ERROR)
return false;
if (per_file == STATUS_OK) {
((SourceFile*)remove_sources->items[i])->skipped = true;
} else if (per_file != STATUS_NEXT) {
log_message(LOG_LEVEL_ERROR, "Unexpected per-file status from receiver");
return false;
}
}
}
Status status;
return receive_status(client->file_descriptor, &status) && status == STATUS_OK;
}
static void pipeline_cancel(PipelineContextSender* context) { static void pipeline_cancel(PipelineContextSender* context) {
mtx_lock(&context->mutex_scanner); mtx_lock(&context->mutex_scanner);
mtx_lock(&context->mutex_loader); mtx_lock(&context->mutex_loader);
@@ -133,22 +254,46 @@ static int send_dry_run_manifest(const Config* config) {
Chunk* chunk; Chunk* chunk;
int file_count = 0; int file_count = 0;
unsigned long long total_bytes = 0; unsigned long long total_bytes = 0;
printf("Dry run: files to be transferred\n"); char size_buffer[32];
if (!config->quiet)
printf("Dry run: files to be transferred\n");
while ((chunk = directory_scanner_next(scanner)) != NULL) { while ((chunk = directory_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count; i++) { for (int i = 0; i < chunk->element_count; i++) {
printf(" %s (%zu bytes)\n", chunk->items[i]->path, chunk->items[i]->data->size); if (!config->quiet) {
char* escaped_path = output_escape(chunk->items[i]->path, config->eight_bit_output);
if (!escaped_path) {
chunk_destroy(chunk);
directory_scanner_destroy(scanner);
return -1;
}
if (config->human_readable)
printf(
" %s (%s)\n", escaped_path,
display_bytes(chunk->items[i]->data->size, true, size_buffer, sizeof(size_buffer)));
else
printf(" %s (%zu bytes)\n", escaped_path, chunk->items[i]->data->size);
free(escaped_path);
}
total_bytes += chunk->items[i]->data->size; total_bytes += chunk->items[i]->data->size;
file_count++; file_count++;
} }
chunk_destroy(chunk); chunk_destroy(chunk);
} }
directory_scanner_destroy(scanner); directory_scanner_destroy(scanner);
printf("Total: %d files, %.1f MB\n", file_count, total_bytes / 1048576.0); if (!config->quiet) {
if (config->human_readable)
printf("Total: %d files, %s\n", file_count,
display_bytes(total_bytes, true, size_buffer, sizeof(size_buffer)));
else
printf("Total: %d files, %.1f MB\n", file_count, total_bytes / 1048576.0);
}
return 0; return 0;
} }
/* Send the delete manifest (list of files) to the server. Returns 0 on success, -1 on failure. */ /* Send the delete manifest (list of files) to the server. Returns 0 on success, -1 on failure. */
static int send_delete_manifest(int fd, ArrayList* manifest) { static int send_delete_manifest(int fd, ArrayList* manifest) {
if (!manifest)
return -1;
if (!send_status(fd, STATUS_MANIFEST)) if (!send_status(fd, STATUS_MANIFEST))
return -1; return -1;
if (!send_int(fd, manifest->size)) if (!send_int(fd, manifest->size))
@@ -169,10 +314,13 @@ static int incremental_check(Client* client, File* file, const Config* config,
return -1; return -1;
unsigned long long fsize = file->data->size; unsigned long long fsize = file->data->size;
long long mtime = file->metadata ? file->metadata->mtime_sec : 0; long long mtime = file->metadata ? file->metadata->mtime_sec : 0;
long long mtime_nsec = file->metadata ? file->metadata->mtime_nsec : 0;
if (!send_n_data(client->file_descriptor, &fsize, sizeof(fsize))) if (!send_n_data(client->file_descriptor, &fsize, sizeof(fsize)))
return -1; return -1;
if (!send_n_data(client->file_descriptor, &mtime, sizeof(mtime))) if (!send_n_data(client->file_descriptor, &mtime, sizeof(mtime)))
return -1; return -1;
if (!send_n_data(client->file_descriptor, &mtime_nsec, sizeof(mtime_nsec)))
return -1;
if (config->checksum) { if (config->checksum) {
uint64_t checksum; uint64_t checksum;
if (!file_checksum(file, &checksum) || if (!file_checksum(file, &checksum) ||
@@ -190,17 +338,22 @@ static int incremental_check(Client* client, File* file, const Config* config,
return 1; return 1;
if (s == STATUS_DELTA_SIGNATURE) { if (s == STATUS_DELTA_SIGNATURE) {
Data* sig_data = receive_data(client->file_descriptor); Data* sig_data = receive_data(client->file_descriptor);
if (!sig_data) if (!sig_data) {
send_status(client->file_descriptor, STATUS_ERROR);
return -1; return -1;
}
DeltaSignature* sig = delta_signature_deserialize(sig_data); DeltaSignature* sig = delta_signature_deserialize(sig_data);
data_destroy(sig_data); data_destroy(sig_data);
if (!sig) if (!sig) {
send_status(client->file_descriptor, STATUS_ERROR);
return -1; return -1;
}
*out_sig = sig; *out_sig = sig;
return 2; return 2;
} }
if (s != STATUS_NEXT) { if (s != STATUS_NEXT) {
log_message(LOG_LEVEL_ERROR, "Unexpected server status"); log_message(LOG_LEVEL_ERROR, "Unexpected server status");
send_status(client->file_descriptor, STATUS_ERROR);
return -1; return -1;
} }
return 0; return 0;
@@ -208,8 +361,10 @@ static int incremental_check(Client* client, File* file, const Config* config,
static int send_delta(Client* client, File* file, DeltaSignature* sig, Config* config) { static int send_delta(Client* client, File* file, DeltaSignature* sig, Config* config) {
Delta* delta = delta_compute(file->data->data, file->data->size, sig, config->delta_block_size); Delta* delta = delta_compute(file->data->data, file->data->size, sig, config->delta_block_size);
/* The receiver is blocked after sending the signature. Every local
fallback therefore needs the explicit NEXT response before full data. */
if (!delta) if (!delta)
return 1; return send_status(client->file_descriptor, STATUS_NEXT) ? 1 : -1;
if (!delta_is_worthwhile(delta, file->data->size)) { if (!delta_is_worthwhile(delta, file->data->size)) {
delta_destroy(delta); delta_destroy(delta);
@@ -221,14 +376,17 @@ static int send_delta(Client* client, File* file, DeltaSignature* sig, Config* c
Data* delta_data = delta_serialize(delta); Data* delta_data = delta_serialize(delta);
delta_destroy(delta); delta_destroy(delta);
if (!delta_data) if (!delta_data)
return -1; return send_status(client->file_descriptor, STATUS_NEXT) ? 1 : -1;
Data* to_send = delta_data; Data* to_send = delta_data;
if (config->use_compression) { int skip_count = config->skip_compress_set ? config->skip_compress_count : -1;
to_send = data_compress(delta_data, config->compression_level); if (config->use_compression && !compression_should_skip_with_suffixes(
file->path, config->skip_compress_suffixes, skip_count)) {
to_send = data_compress_with_threads(delta_data, config->compression_level,
config->compression_threads);
data_destroy(delta_data); data_destroy(delta_data);
if (!to_send) if (!to_send)
return -1; return send_status(client->file_descriptor, STATUS_NEXT) ? 1 : -1;
} }
bool ok = send_status(client->file_descriptor, STATUS_DELTA_DATA) && bool ok = send_status(client->file_descriptor, STATUS_DELTA_DATA) &&
@@ -241,20 +399,25 @@ static int send_delta(Client* client, File* file, DeltaSignature* sig, Config* c
return ok ? 0 : -1; return ok ? 0 : -1;
} }
typedef bool (*file_send_fn)(File*, int, bool, int, bool);
// Send a single file directly (non-incremental path). // Send a single file directly (non-incremental path).
static bool send_file_direct(File* file, int fd, bool use_metadata, int compression_level) { static bool send_file_direct(File* file, int fd, bool use_metadata, int compression_level,
const Config* config) {
if (!send_status(fd, STATUS_NEXT)) if (!send_status(fd, STATUS_NEXT))
return false; return false;
return file_send_single_calls(file, fd, use_metadata, compression_level, true); int skip_count = config->skip_compress_set ? config->skip_compress_count : -1;
return file_send_single_calls_with_skip(file, fd, use_metadata, compression_level, true,
config->skip_compress_suffixes, skip_count,
config->compression_threads);
} }
// Send a single file directly via sendfile (non-incremental path). // Send a single file directly via sendfile (non-incremental path).
static bool send_file_direct_sendfile(File* file, int fd, bool use_metadata) { static bool send_file_direct_sendfile(File* file, int fd, bool use_metadata, const Config* config) {
if (!send_status(fd, STATUS_NEXT)) if (!send_status(fd, STATUS_NEXT))
return false; return false;
return file_send_sendfile(file, fd, use_metadata, 0, true); int skip_count = config->skip_compress_set ? config->skip_compress_count : -1;
return file_send_sendfile_with_skip(file, fd, use_metadata, 0, true,
config->skip_compress_suffixes, skip_count,
config->compression_threads);
} }
// Process one file in a chunk: either via incremental check or direct send. // Process one file in a chunk: either via incremental check or direct send.
@@ -262,13 +425,16 @@ static bool send_file_direct_sendfile(File* file, int fd, bool use_metadata) {
static int send_single_file(Client* client, File* file, Config* config, bool use_incremental, static int send_single_file(Client* client, File* file, Config* config, bool use_incremental,
bool use_sendfile) { bool use_sendfile) {
int compression_level = config->use_compression ? config->compression_level : 0; int compression_level = config->use_compression ? config->compression_level : 0;
log_info_message(LOG_INFO_COPY, "Transferring %s", file->path);
if (!use_incremental) { if (!use_incremental) {
if (use_sendfile) { if (use_sendfile) {
return send_file_direct_sendfile(file, client->file_descriptor, config->use_metadata) ? 0 return send_file_direct_sendfile(file, client->file_descriptor, config->use_metadata, config)
: -1; ? 0
: -1;
} }
return send_file_direct(file, client->file_descriptor, config->use_metadata, compression_level) return send_file_direct(file, client->file_descriptor, config->use_metadata, compression_level,
config)
? 0 ? 0
: -1; : -1;
} }
@@ -278,6 +444,7 @@ static int send_single_file(Client* client, File* file, Config* config, bool use
DeltaSignature* sig = NULL; DeltaSignature* sig = NULL;
int rc = incremental_check(client, file, config, &sig); int rc = incremental_check(client, file, config, &sig);
if (rc == 1) { if (rc == 1) {
log_info_message(LOG_INFO_SKIP, "Skipping unchanged %s", file->path);
delta_signature_destroy(sig); delta_signature_destroy(sig);
return 1; return 1;
} }
@@ -294,13 +461,15 @@ static int send_single_file(Client* client, File* file, Config* config, bool use
return -1; return -1;
} }
// Fall through: send full file via sendfile (pass 0 for compression_level) // Fall through: send full file via sendfile (pass 0 for compression_level)
if (!file_send_sendfile(file, client->file_descriptor, config->use_metadata, 0, false)) int skip_count = config->skip_compress_set ? config->skip_compress_count : -1;
if (!file_send_sendfile_with_skip(file, client->file_descriptor, config->use_metadata, 0, false,
config->skip_compress_suffixes, skip_count,
config->compression_threads))
return -1; return -1;
return 0; return 0;
} }
// Incremental path with single_calls (supports compression and delta) // Incremental path with single_calls (supports compression and delta)
file_send_fn send_fn = (file_send_fn)file_send_single_calls;
DeltaSignature* sig = NULL; DeltaSignature* sig = NULL;
int rc = incremental_check(client, file, config, &sig); int rc = incremental_check(client, file, config, &sig);
if (rc < 0) { if (rc < 0) {
@@ -308,10 +477,11 @@ static int send_single_file(Client* client, File* file, Config* config, bool use
return -1; return -1;
} }
if (rc == 1) { if (rc == 1) {
log_info_message(LOG_INFO_SKIP, "Skipping unchanged %s", file->path);
delta_signature_destroy(sig); delta_signature_destroy(sig);
return 1; return 1;
} }
if (rc == 2 && config->use_delta) { if (rc == 2 && config->use_delta && !config->whole_file) {
int drc = send_delta(client, file, sig, config); int drc = send_delta(client, file, sig, config);
delta_signature_destroy(sig); delta_signature_destroy(sig);
if (drc == 0) if (drc == 0)
@@ -328,18 +498,29 @@ static int send_single_file(Client* client, File* file, Config* config, bool use
return -1; return -1;
} }
} }
if (!send_fn(file, client->file_descriptor, config->use_metadata, compression_level, false)) int skip_count = config->skip_compress_set ? config->skip_compress_count : -1;
if (!file_send_single_calls_with_skip(file, client->file_descriptor, config->use_metadata,
compression_level, false, config->skip_compress_suffixes,
skip_count, config->compression_threads))
return -1; return -1;
return 0; return 0;
} }
int send_chunk(Client* client, Chunk* chunk, Config* config) { static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
ArrayList* remove_sources) {
if (config->use_chunk_serialization) { if (config->use_chunk_serialization) {
if (remove_sources) {
for (int i = 0; i < chunk->element_count; i++) {
if (!remember_source_file(remove_sources, chunk->items[i]))
return -1;
}
}
if (!send_status(client->file_descriptor, STATUS_CHUNK)) if (!send_status(client->file_descriptor, STATUS_CHUNK))
return -1; return -1;
Data* data; Data* data;
if (config->use_compression) { if (config->use_compression) {
data = chunk_compress(chunk, config->compression_level, config->use_metadata); data = chunk_compress_with_threads(chunk, config->compression_level, config->use_metadata,
config->compression_threads);
} else { } else {
data = chunk_serialize(chunk, config->use_metadata); data = chunk_serialize(chunk, config->use_metadata);
} }
@@ -358,23 +539,38 @@ int send_chunk(Client* client, Chunk* chunk, Config* config) {
if (f == NULL) if (f == NULL)
continue; continue;
bool stream = f->data->data == NULL && f->data->size > 0; bool stream = f->data->data == NULL && f->data->size > 0;
bool use_sendfile = (config->use_sendfile && !config->use_compression) || stream; bool use_sendfile =
(config->use_sendfile && !config->use_compression) || (stream && !config->use_compression);
SourceFile* source = remove_sources ? source_file_create(f) : NULL;
int rc = send_single_file(client, f, config, config->use_incremental, use_sendfile); int rc = send_single_file(client, f, config, config->use_incremental, use_sendfile);
if (rc == 1) if (rc == 1) {
source_file_destroy(source);
continue; continue;
if (rc < 0) }
if (rc < 0) {
source_file_destroy(source);
return -1; return -1;
}
if (source && !array_list_add(remove_sources, source)) {
source_file_destroy(source);
return -1;
}
} }
return 0; return 0;
} }
int send_chunk(Client* client, Chunk* chunk, Config* config) {
return send_chunk_with_removal(client, chunk, config, NULL);
}
static int send_chunks_multithreaded(void* pipeline_context) { static int send_chunks_multithreaded(void* pipeline_context) {
PipelineContextSender* context = (PipelineContextSender*)pipeline_context; PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
Client* client = connect_transfer_client(context->config); Client* client = connect_transfer_client(context->config);
if (!client) { if (!client) {
if (context->config->transport == TRANSPORT_TCP) if (context->config->transport == TRANSPORT_TCP)
fprintf(stderr, "Error: could not connect to server%s\n", log_message(LOG_LEVEL_ERROR, "could not connect to server%s",
context->config->use_tls ? " via TLS" : ""); context->config->use_tls ? " via TLS" : "");
pipeline_cancel(context);
mark_sender_done(context); mark_sender_done(context);
return thrd_error; return thrd_error;
} }
@@ -383,6 +579,7 @@ static int send_chunks_multithreaded(void* pipeline_context) {
protocol_session_set_ssl(&session, (SSL*)client->ssl); protocol_session_set_ssl(&session, (SSL*)client->ssl);
protocol_session_bind(&session); protocol_session_bind(&session);
if (!config_send(client->file_descriptor, context->config)) { if (!config_send(client->file_descriptor, context->config)) {
pipeline_cancel(context);
disconnect_transfer_client(client); disconnect_transfer_client(client);
mark_sender_done(context); mark_sender_done(context);
protocol_session_unbind(); protocol_session_unbind();
@@ -394,11 +591,28 @@ static int send_chunks_multithreaded(void* pipeline_context) {
context->queue_loader, &context->mutex_loader, &context->condition_not_empty_loader, context->queue_loader, &context->mutex_loader, &context->condition_not_empty_loader,
&context->condition_not_full_loader, &context->loader_done); &context->condition_not_full_loader, &context->loader_done);
if (current_chunk == NULL) { if (current_chunk == NULL) {
if (atomic_load(&context->cancelled)) {
pipeline_cancel(context);
disconnect_transfer_client(client);
mark_sender_done(context);
protocol_session_unbind();
return thrd_error;
}
if (context->config->use_delete) { if (context->config->use_delete) {
if (send_delete_manifest(client->file_descriptor, context->manifest) != 0) if (send_delete_manifest(client->file_descriptor, context->manifest) != 0)
goto send_fail; goto send_fail;
} }
bool ok = finalize_transfer(client); bool ok = finalize_transfer(client, context->config, context->remove_source_files);
if (ok)
remove_transferred_sources(context->config, context->remove_source_files);
mtx_lock(&context->mutex_progress);
int total_files = context->total_files;
unsigned long long total_bytes = context->total_bytes;
mtx_unlock(&context->mutex_progress);
if (context->config->stats)
fprintf(stderr, "Stats: %d files, %.1f MB\n", total_files, total_bytes / 1048576.0);
log_info_message(LOG_INFO_STATS, "Transfer summary: %d files, %.1f MB", total_files,
total_bytes / 1048576.0);
disconnect_transfer_client(client); disconnect_transfer_client(client);
mark_sender_done(context); mark_sender_done(context);
protocol_session_unbind(); protocol_session_unbind();
@@ -411,8 +625,9 @@ static int send_chunks_multithreaded(void* pipeline_context) {
protocol_session_unbind(); protocol_session_unbind();
return thrd_error; return thrd_error;
} }
if (send_chunk(client, current_chunk, context->config) != 0) { if (send_chunk_with_removal(client, current_chunk, context->config,
fprintf(stderr, "Error: unexpected error while sending chunk\n"); context->remove_source_files) != 0) {
log_message(LOG_LEVEL_ERROR, "unexpected error while sending chunk");
chunk_destroy(current_chunk); chunk_destroy(current_chunk);
pipeline_cancel(context); pipeline_cancel(context);
disconnect_transfer_client(client); disconnect_transfer_client(client);
@@ -420,30 +635,35 @@ static int send_chunks_multithreaded(void* pipeline_context) {
protocol_session_unbind(); protocol_session_unbind();
return thrd_error; return thrd_error;
} }
if (context->config->show_progress) { unsigned long long chunk_bytes = 0;
unsigned long long chunk_bytes = 0; int chunk_files = 0;
for (int i = 0; i < current_chunk->element_count; i++) { for (int i = 0; i < current_chunk->element_count; i++) {
if (current_chunk->items[i] && current_chunk->items[i]->data) if (current_chunk->items[i] && current_chunk->items[i]->data) {
chunk_bytes += current_chunk->items[i]->data->size; chunk_files++;
chunk_bytes += current_chunk->items[i]->data->size;
} }
mtx_lock(&context->mutex_progress);
context->progress_bytes += chunk_bytes;
mtx_unlock(&context->mutex_progress);
} }
mtx_lock(&context->mutex_progress);
context->total_files += chunk_files;
context->total_bytes += chunk_bytes;
context->progress_bytes = context->total_bytes;
mtx_unlock(&context->mutex_progress);
chunk_destroy(current_chunk); chunk_destroy(current_chunk);
} }
} }
static int scan_directory_multithreaded(void* pipeline_context) { static int scan_directory_multithreaded(void* pipeline_context) {
PipelineContextSender* context = (PipelineContextSender*)pipeline_context; PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
protocol_session_bind(&context->allocation_session);
ScannerOptions options = scanner_options_from_config(context->config, 4); ScannerOptions options = scanner_options_from_config(context->config, 4);
ParallelScanner* scanner = ParallelScanner* scanner = parallel_scanner_create_with_options(
parallel_scanner_create_with_options(context->config->send_directory, &options); context->config->send_directory, &options, &context->allocation_session);
Chunk* current_chunk; Chunk* current_chunk;
if (scanner == NULL) { if (scanner == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to create parallel scanner"); log_message(LOG_LEVEL_ERROR, "Failed to create parallel scanner");
pipeline_cancel(context); pipeline_cancel(context);
protocol_session_unbind();
return thrd_error; return thrd_error;
} }
while ((current_chunk = parallel_scanner_next(scanner)) != NULL) { while ((current_chunk = parallel_scanner_next(scanner)) != NULL) {
@@ -455,6 +675,7 @@ static int scan_directory_multithreaded(void* pipeline_context) {
pipeline_cancel(context); pipeline_cancel(context);
chunk_destroy(current_chunk); chunk_destroy(current_chunk);
parallel_scanner_destroy(scanner); parallel_scanner_destroy(scanner);
protocol_session_unbind();
return thrd_error; return thrd_error;
} }
} }
@@ -465,6 +686,7 @@ static int scan_directory_multithreaded(void* pipeline_context) {
chunk_destroy(current_chunk); chunk_destroy(current_chunk);
pipeline_cancel(context); pipeline_cancel(context);
parallel_scanner_destroy(scanner); parallel_scanner_destroy(scanner);
protocol_session_unbind();
return thrd_error; return thrd_error;
} }
} }
@@ -476,6 +698,7 @@ static int scan_directory_multithreaded(void* pipeline_context) {
cnd_broadcast(&context->condition_not_full_scanner); cnd_broadcast(&context->condition_not_full_scanner);
mtx_unlock(&context->mutex_scanner); mtx_unlock(&context->mutex_scanner);
pipeline_cancel(context); pipeline_cancel(context);
protocol_session_unbind();
return thrd_error; return thrd_error;
} }
mtx_lock(&context->mutex_scanner); mtx_lock(&context->mutex_scanner);
@@ -484,11 +707,13 @@ static int scan_directory_multithreaded(void* pipeline_context) {
mtx_unlock(&context->mutex_scanner); mtx_unlock(&context->mutex_scanner);
parallel_scanner_destroy(scanner); parallel_scanner_destroy(scanner);
protocol_session_unbind();
return thrd_success; return thrd_success;
} }
static int load_files_multithreaded(void* pipeline_context) { static int load_files_multithreaded(void* pipeline_context) {
PipelineContextSender* context = (PipelineContextSender*)pipeline_context; PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
protocol_session_bind(&context->allocation_session);
while (true) { while (true) {
Chunk* chunk = queue_dequeue_multithreaded( Chunk* chunk = queue_dequeue_multithreaded(
context->queue_scanner, &context->mutex_scanner, &context->condition_not_empty_scanner, context->queue_scanner, &context->mutex_scanner, &context->condition_not_empty_scanner,
@@ -498,17 +723,20 @@ static int load_files_multithreaded(void* pipeline_context) {
context->loader_done = true; context->loader_done = true;
cnd_signal(&context->condition_not_empty_loader); cnd_signal(&context->condition_not_empty_loader);
mtx_unlock(&context->mutex_loader); mtx_unlock(&context->mutex_loader);
protocol_session_unbind();
return thrd_success; return thrd_success;
} }
if (!context->config->use_sendfile) { if (!context->config->use_sendfile) {
for (int i = 0; i < chunk->element_count; i++) { for (int i = 0; i < chunk->element_count; i++) {
File* f = chunk->items[i]; File* f = chunk->items[i];
if (f->data->size > STREAM_THRESHOLD) if (f->data->size > STREAM_THRESHOLD && !context->config->use_compression)
continue; continue;
if (!file_load_data(f)) { if (!file_load_data(f)) {
log_message(LOG_LEVEL_ERROR, "Failed to load file data, skipping"); log_message(LOG_LEVEL_ERROR, "Failed to load file data");
file_destroy(f); chunk_destroy(chunk);
chunk->items[i] = NULL; pipeline_cancel(context);
protocol_session_unbind();
return thrd_error;
} }
} }
} }
@@ -517,14 +745,34 @@ static int load_files_multithreaded(void* pipeline_context) {
&context->condition_not_full_loader, &context->condition_not_full_loader,
&context->cancelled)) { &context->cancelled)) {
chunk_destroy(chunk); chunk_destroy(chunk);
atomic_store(&context->cancelled, true); pipeline_cancel(context);
cnd_broadcast(&context->condition_not_full_loader); protocol_session_unbind();
cnd_broadcast(&context->condition_not_empty_loader);
return thrd_error; return thrd_error;
} }
} }
} }
/* Print a one-line transfer progress report to stderr. `suffix` ends the
line (e.g. "Done.\n") or is "" for in-place refresh. Shared by the
single-threaded loop and the multithreaded progress thread. */
static void print_transfer_progress(unsigned long long total_bytes, time_t start,
const char* suffix, bool human_readable) {
double elapsed = difftime(time(NULL), start);
double rate = elapsed > 0.0 ? total_bytes / (1048576.0 * elapsed) : 0.0;
if (human_readable) {
char total_buffer[32];
char rate_buffer[32];
fprintf(stderr, "\rSent %s (%s/s) %s",
display_bytes(total_bytes, true, total_buffer, sizeof(total_buffer)),
display_bytes((unsigned long long)(rate * 1048576.0), true, rate_buffer,
sizeof(rate_buffer)),
suffix);
} else {
fprintf(stderr, "\rSent %.1f MB (%.1f MB/s) %s", total_bytes / 1048576.0, rate, suffix);
}
fflush(stderr);
}
/* Progress-reporting thread for multithreaded send. Runs in parallel with /* Progress-reporting thread for multithreaded send. Runs in parallel with
the scanner/loader/sender threads and prints periodic progress to stderr. */ the scanner/loader/sender threads and prints periodic progress to stderr. */
static int progress_thread_fn(void* arg) { static int progress_thread_fn(void* arg) {
@@ -539,20 +787,14 @@ static int progress_thread_fn(void* arg) {
mtx_unlock(&context->mutex_progress); mtx_unlock(&context->mutex_progress);
if (done) { if (done) {
time_t now = time(NULL); print_transfer_progress(total, start, "Done.\n", context->config->human_readable);
double elapsed = difftime(now, start);
double rate = elapsed > 0.0 ? total / (1048576.0 * elapsed) : 0.0;
fprintf(stderr, "\rSent %.1f MB (%.1f MB/s) Done.\n", total / 1048576.0, rate);
break; break;
} }
time_t now = time(NULL); time_t now = time(NULL);
if (now - last_progress >= 1) { if (now - last_progress >= 1) {
last_progress = now; last_progress = now;
double elapsed = difftime(now, start); print_transfer_progress(total, start, "", context->config->human_readable);
double rate = elapsed > 0.0 ? total / (1048576.0 * elapsed) : 0.0;
fprintf(stderr, "\rSent %.1f MB (%.1f MB/s) ", total / 1048576.0, rate);
fflush(stderr);
} }
struct timespec ts = {0, 100 * 1000000L}; /* 100 ms */ struct timespec ts = {0, 100 * 1000000L}; /* 100 ms */
@@ -568,36 +810,33 @@ int send_files(Config* config) {
Client* client = connect_transfer_client(config); Client* client = connect_transfer_client(config);
if (!client) { if (!client) {
if (config->transport == TRANSPORT_TCP) if (config->transport == TRANSPORT_TCP)
fprintf(stderr, "Error: could not connect to server%s\n", config->use_tls ? " via TLS" : ""); log_message(LOG_LEVEL_ERROR, "could not connect to server%s",
config->use_tls ? " via TLS" : "");
return 1; return 1;
} }
ProtocolSession session; ProtocolSession session;
protocol_session_init(&session, client->file_descriptor, client->file_descriptor); protocol_session_init(&session, client->file_descriptor, client->file_descriptor);
protocol_session_set_ssl(&session, (SSL*)client->ssl); protocol_session_set_ssl(&session, (SSL*)client->ssl);
protocol_session_bind(&session); protocol_session_bind(&session);
if (!config_send(client->file_descriptor, config)) { int ret = 1;
disconnect_transfer_client(client); DirectoryScanner* scanner = NULL;
protocol_session_unbind(); ArrayList* manifest = NULL;
return 1; ArrayList* remove_sources = NULL;
} if (!config_send(client->file_descriptor, config))
goto send_fail;
ScannerOptions scanner_options = scanner_options_from_config(config, 0); ScannerOptions scanner_options = scanner_options_from_config(config, 0);
DirectoryScanner* scanner = scanner = directory_scanner_create_with_options(config->send_directory, &scanner_options);
directory_scanner_create_with_options(config->send_directory, &scanner_options); manifest = create_transfer_manifest(config);
if (config->remove_source_files)
remove_sources = array_list_create(source_file_destroy);
if (!scanner || (config->use_delete && !manifest) ||
(config->remove_source_files && !remove_sources))
goto send_fail;
Chunk* current_chunk; Chunk* current_chunk;
unsigned long long total_bytes = 0; unsigned long long total_bytes = 0;
int total_files = 0; int total_files = 0;
time_t last_progress = 0; time_t last_progress = 0;
time_t start = time(NULL); time_t start = time(NULL);
ArrayList* manifest = create_transfer_manifest(config);
if (!scanner || (config->use_delete && !manifest)) {
if (scanner)
directory_scanner_destroy(scanner);
if (manifest)
array_list_delete(manifest);
disconnect_transfer_client(client);
protocol_session_unbind();
return 1;
}
while ((current_chunk = directory_scanner_next(scanner)) != NULL) { while ((current_chunk = directory_scanner_next(scanner)) != NULL) {
unsigned long long chunk_bytes = 0; unsigned long long chunk_bytes = 0;
for (int i = 0; i < current_chunk->element_count; i++) { for (int i = 0; i < current_chunk->element_count; i++) {
@@ -609,17 +848,23 @@ int send_files(Config* config) {
goto send_fail; goto send_fail;
} }
if (!config->use_sendfile) { if (!config->use_sendfile) {
bool load_ok = true;
for (int i = 0; i < current_chunk->element_count; i++) { for (int i = 0; i < current_chunk->element_count; i++) {
File* f = current_chunk->items[i]; File* f = current_chunk->items[i];
if (f->data->size > STREAM_THRESHOLD) if (f->data->size > STREAM_THRESHOLD && !config->use_compression)
continue; continue;
if (!file_load_data(f)) { if (!file_load_data(f)) {
log_message(LOG_LEVEL_ERROR, "Failed to load file data"); log_message(LOG_LEVEL_ERROR, "Failed to load file data");
continue; load_ok = false;
break;
} }
} }
if (!load_ok) {
chunk_destroy(current_chunk);
goto send_fail;
}
} }
if (send_chunk(client, current_chunk, config) != 0) { if (send_chunk_with_removal(client, current_chunk, config, remove_sources) != 0) {
log_message(LOG_LEVEL_ERROR, "Failed to send chunk"); log_message(LOG_LEVEL_ERROR, "Failed to send chunk");
chunk_destroy(current_chunk); chunk_destroy(current_chunk);
if (manifest) if (manifest)
@@ -627,15 +872,12 @@ int send_files(Config* config) {
manifest = NULL; manifest = NULL;
break; break;
} }
if (config->show_progress) { total_bytes += chunk_bytes;
total_bytes += chunk_bytes; if (config->show_progress && !config->quiet) {
time_t now = time(NULL); time_t now = time(NULL);
if (now - last_progress >= 1) { if (now - last_progress >= 1) {
last_progress = now; last_progress = now;
double elapsed = difftime(now, start); print_transfer_progress(total_bytes, start, "", config->human_readable);
double rate = elapsed > 0 ? total_bytes / (1048576.0 * elapsed) : 0;
fprintf(stderr, "\rSent %.1f MB (%.1f MB/s) ", total_bytes / 1048576.0, rate);
fflush(stderr);
} }
} }
chunk_destroy(current_chunk); chunk_destroy(current_chunk);
@@ -645,37 +887,54 @@ int send_files(Config* config) {
if (config->use_delete) { if (config->use_delete) {
if (send_delete_manifest(client->file_descriptor, manifest) != 0) { if (send_delete_manifest(client->file_descriptor, manifest) != 0) {
array_list_delete(manifest); array_list_delete(manifest);
manifest = NULL;
goto send_fail; goto send_fail;
} }
array_list_delete(manifest); array_list_delete(manifest);
manifest = NULL; manifest = NULL;
} }
bool ok = finalize_transfer(client); bool ok = finalize_transfer(client, config, remove_sources);
double elapsed_total = difftime(time(NULL), start); if (ok)
if (config->show_progress) { remove_transferred_sources(config, remove_sources);
if (config->show_progress && !config->quiet)
print_transfer_progress(total_bytes, start, "Done.\n", config->human_readable);
if (config->stats && !config->quiet) {
double elapsed_total = difftime(time(NULL), start);
double rate = elapsed_total > 0 ? total_bytes / (1048576.0 * elapsed_total) : 0; double rate = elapsed_total > 0 ? total_bytes / (1048576.0 * elapsed_total) : 0;
fprintf(stderr, "\rSent %.1f MB (%.1f MB/s) Done.\n", total_bytes / 1048576.0, rate); if (config->human_readable) {
char total_buffer[32];
char rate_buffer[32];
fprintf(stderr, "Stats: %d files, %s, %s/s\n", total_files,
display_bytes(total_bytes, true, total_buffer, sizeof(total_buffer)),
display_bytes((unsigned long long)(rate * 1048576.0), true, rate_buffer,
sizeof(rate_buffer)));
} else {
fprintf(stderr, "Stats: %d files, %.1f MB, %.1f MB/s\n", total_files, total_bytes / 1048576.0,
rate);
}
} }
if (config->stats) { log_info_message(LOG_INFO_STATS, "Transfer summary: %d files, %.1f MB", total_files,
double rate = elapsed_total > 0 ? total_bytes / (1048576.0 * elapsed_total) : 0; total_bytes / 1048576.0);
fprintf(stderr, "Stats: %d files, %.1f MB, %.1f MB/s\n", total_files, total_bytes / 1048576.0, ret = ok ? 0 : 1;
rate);
}
directory_scanner_destroy(scanner);
disconnect_transfer_client(client);
protocol_session_unbind();
return ok ? 0 : 1;
send_fail: send_fail:
/* Single cleanup path for all exits. The manifest is intentionally deleted
here even on success without --delete, fixing a pre-existing leak. */
if (manifest) if (manifest)
array_list_delete(manifest); array_list_delete(manifest);
directory_scanner_destroy(scanner); if (remove_sources)
array_list_delete(remove_sources);
if (scanner)
directory_scanner_destroy(scanner);
disconnect_transfer_client(client); disconnect_transfer_client(client);
protocol_session_unbind(); protocol_session_unbind();
return 1; return ret;
} }
int send_files_multithreaded(Config* config) { int send_files_multithreaded(Config** config_ptr) {
if (!config_ptr || !*config_ptr)
return 1;
Config* config = *config_ptr;
if (config->dry_run) if (config->dry_run)
return send_dry_run_manifest(config); return send_dry_run_manifest(config);
@@ -706,8 +965,16 @@ int send_files_multithreaded(Config* config) {
queue_destroy(q2); queue_destroy(q2);
return 1; return 1;
} }
*config_ptr = NULL; /* context now owns config through all remaining paths */
if (config->use_delete) if (config->use_delete)
context->manifest = create_transfer_manifest(config); context->manifest = array_list_create(free);
if (config->remove_source_files)
context->remove_source_files = array_list_create(source_file_destroy);
if ((config->use_delete && !context->manifest) ||
(config->remove_source_files && !context->remove_source_files)) {
pipeline_context_sender_destroy(context);
return 1;
}
thrd_t scanner, loader, sender; thrd_t scanner, loader, sender;
bool scanner_created = false; bool scanner_created = false;
@@ -721,7 +988,7 @@ int send_files_multithreaded(Config* config) {
sender_created = (thrd_create(&sender, send_chunks_multithreaded, context) == thrd_success); sender_created = (thrd_create(&sender, send_chunks_multithreaded, context) == thrd_success);
if (!scanner_created || !loader_created || !sender_created) { if (!scanner_created || !loader_created || !sender_created) {
perror("Error creating threads.\n"); log_perror("Error creating threads");
pipeline_cancel(context); pipeline_cancel(context);
mtx_lock(&context->mutex_progress); mtx_lock(&context->mutex_progress);
context->sender_done = true; context->sender_done = true;
@@ -738,10 +1005,10 @@ int send_files_multithreaded(Config* config) {
thrd_t progress; thrd_t progress;
bool progress_created = false; bool progress_created = false;
if (config->show_progress) { if (config->show_progress && !config->quiet) {
progress_created = (thrd_create(&progress, progress_thread_fn, context) == thrd_success); progress_created = (thrd_create(&progress, progress_thread_fn, context) == thrd_success);
if (!progress_created) { if (!progress_created) {
perror("Error creating progress thread.\n"); log_perror("Error creating progress thread");
/* Non-fatal; continue without progress reporting */ /* Non-fatal; continue without progress reporting */
} }
} }
+2 -1
View File
@@ -7,6 +7,7 @@
int send_chunk(Client* client, Chunk* chunk, Config* config); int send_chunk(Client* client, Chunk* chunk, Config* config);
int send_files(Config* config); int send_files(Config* config);
int send_files_multithreaded(Config* config); /* Takes ownership only when *config is set to NULL on return. */
int send_files_multithreaded(Config** config);
#endif #endif
+23 -13
View File
@@ -1,37 +1,47 @@
#include "client_validation.h" #include "client_validation.h"
#include "log.h"
#include "usage.h" #include "usage.h"
#include <stdio.h> #include <stdio.h>
/* Validate config after parsing. Returns true if valid. */ /* Validate config after parsing. Returns true if valid. */
bool validate_config(const Config* config) { bool validate_config(const Config* config) {
if (!config->send_directory || !config->receive_root_directory) { if (!config->send_directory || !config->receive_root_directory) {
fprintf(stderr, "Error: source and destination directories are required\n"); log_message(LOG_LEVEL_ERROR, "source and destination directories are required");
print_usage(); print_usage();
return false; return false;
} }
if (config->use_sendfile && (config->use_chunk_serialization || config->use_compression)) { if (config->use_sendfile && (config->use_chunk_serialization || config->use_compression)) {
fprintf(stderr, "Error: -f/--sendfile cannot be combined with -c (compression) or -s (chunk " log_message(LOG_LEVEL_ERROR, "-f/--sendfile cannot be combined with -c (compression) or -s "
"serialization)\n"); "(chunk serialization)");
return false;
}
if (config->compression_threads > 0 && !config->use_compression) {
log_message(LOG_LEVEL_ERROR, "--compress-threads requires compression (-c or -z)");
return false; return false;
} }
if (config->transport == TRANSPORT_SSH && config->use_sendfile) { if (config->transport == TRANSPORT_SSH && config->use_sendfile) {
fprintf(stderr, "Error: -f/--sendfile is not supported with SSH transport\n"); log_message(LOG_LEVEL_ERROR, "-f/--sendfile is not supported with SSH transport");
return false; return false;
} }
if (config->use_incremental && config->use_chunk_serialization) { if (config->use_incremental && config->use_chunk_serialization) {
fprintf(stderr, "Error: --incremental is not supported with -s (chunk serialization)\n"); log_message(LOG_LEVEL_ERROR, "--incremental is not supported with -s (chunk serialization)");
return false; return false;
} }
if (config->use_delta && !config->use_incremental) { if (config->skip_compress_set && config->use_chunk_serialization) {
fprintf(stderr, "Error: --delta requires --incremental\n"); log_message(LOG_LEVEL_ERROR,
"--skip-compress cannot be combined with -s (chunk serialization)");
return false; return false;
} }
if (config->use_delta && config->use_chunk_serialization) { if (config->use_delta && !config->whole_file && !config->use_incremental) {
fprintf(stderr, "Error: --delta cannot be combined with -s (chunk serialization)\n"); log_message(LOG_LEVEL_ERROR, "--delta requires --incremental");
return false; return false;
} }
if (config->use_delta && config->use_sendfile) { if (config->use_delta && !config->whole_file && config->use_chunk_serialization) {
fprintf(stderr, "Error: --delta cannot be combined with -f (sendfile)\n"); log_message(LOG_LEVEL_ERROR, "--delta cannot be combined with -s (chunk serialization)");
return false;
}
if (config->use_delta && !config->whole_file && config->use_sendfile) {
log_message(LOG_LEVEL_ERROR, "--delta cannot be combined with -f (sendfile)");
return false; return false;
} }
if (config->append || config->append_verify) { if (config->append || config->append_verify) {
@@ -41,8 +51,8 @@ bool validate_config(const Config* config) {
return false; return false;
} }
if (config->use_tls) { if (config->use_tls) {
if (!config->tls_cert || !config->tls_key) { if (!config->tls_cert || !config->tls_key || !config->tls_ca) {
fprintf(stderr, "Error: --tls requires --cert and --key\n"); log_message(LOG_LEVEL_ERROR, "--tls requires --cert, --key, and --ca");
return false; return false;
} }
} }
+255 -208
View File
@@ -1,3 +1,4 @@
#include "log.h"
#include "scanner.h" #include "scanner.h"
#include "array_list.h" #include "array_list.h"
#include "chunk.h" #include "chunk.h"
@@ -226,7 +227,7 @@ static int open_next_directory(DirectoryScanner* scanner) {
free(de); free(de);
scanner->current_dir = opendir(scanner->current_path); scanner->current_dir = opendir(scanner->current_path);
if (scanner->current_dir == NULL) { if (scanner->current_dir == NULL) {
perror("Could not open directory"); log_perror("Could not open directory");
free(scanner->current_path); free(scanner->current_path);
scanner->current_path = NULL; scanner->current_path = NULL;
scanner->failed = true; scanner->failed = true;
@@ -350,10 +351,14 @@ typedef struct {
char** dirs; char** dirs;
int dir_count; int dir_count;
ScannerOptions options; ScannerOptions options;
ProtocolSession* allocation_session;
} ParallelWorkerArg; } ParallelWorkerArg;
static int parallel_worker_thread(void* arg) { static int parallel_worker_thread(void* arg) {
ParallelWorkerArg* wa = (ParallelWorkerArg*)arg; ParallelWorkerArg* wa = (ParallelWorkerArg*)arg;
ProtocolSession* allocation_session = wa->allocation_session;
if (allocation_session)
protocol_session_bind(allocation_session);
for (int i = 0; i < wa->dir_count; i++) { for (int i = 0; i < wa->dir_count; i++) {
DirectoryScanner* ds = directory_scanner_create_with_options(wa->dirs[i], &wa->options); DirectoryScanner* ds = directory_scanner_create_with_options(wa->dirs[i], &wa->options);
if (!ds) { if (!ds) {
@@ -363,6 +368,8 @@ static int parallel_worker_thread(void* arg) {
cnd_broadcast(&wa->ps->result_not_empty); cnd_broadcast(&wa->ps->result_not_empty);
cnd_broadcast(&wa->ps->result_not_full); cnd_broadcast(&wa->ps->result_not_full);
mtx_unlock(&wa->ps->result_mutex); mtx_unlock(&wa->ps->result_mutex);
for (int j = i; j < wa->dir_count; j++)
free(wa->dirs[j]);
break; break;
} }
Chunk* chunk; Chunk* chunk;
@@ -395,21 +402,28 @@ static int parallel_worker_thread(void* arg) {
cnd_signal(&ps->result_not_empty); cnd_signal(&ps->result_not_empty);
} }
mtx_unlock(&ps->result_mutex); mtx_unlock(&ps->result_mutex);
if (allocation_session)
protocol_session_unbind();
return thrd_success; return thrd_success;
} }
ParallelScanner* parallel_scanner_create_with_options(const char* root_directory, static void parallel_scanner_creation_failed(ParallelScanner* ps) {
const ScannerOptions* options) { mtx_lock(&ps->result_mutex);
if (!root_directory || !options) ps->failed = true;
return NULL; atomic_store(&ps->cancelled, true);
ParallelScanner* ps = calloc(1, sizeof(ParallelScanner)); ps->expected_threads = ps->created_threads;
if (!ps) if (ps->completed >= ps->expected_threads)
return NULL; ps->done = true;
cnd_broadcast(&ps->result_not_empty);
cnd_broadcast(&ps->result_not_full);
mtx_unlock(&ps->result_mutex);
}
/* Initialize result queue and synchronization primitives. Returns true on success. */
static bool parallel_scanner_init(ParallelScanner* ps) {
ps->result_queue = queue_create(100, chunk_destroy); ps->result_queue = queue_create(100, chunk_destroy);
if (!ps->result_queue) { if (!ps->result_queue)
free(ps); return false;
return NULL;
}
atomic_init(&ps->cancelled, false); atomic_init(&ps->cancelled, false);
int init = 0; int init = 0;
bool ok = true; bool ok = true;
@@ -434,223 +448,251 @@ ParallelScanner* parallel_scanner_create_with_options(const char* root_directory
if (init >= 1) if (init >= 1)
mtx_destroy(&ps->result_mutex); mtx_destroy(&ps->result_mutex);
queue_destroy(ps->result_queue); queue_destroy(ps->result_queue);
ps->result_queue = NULL;
return false;
}
return true;
}
/* Split files into chunks of roughly chunk_size bytes. Returns the first chunk (also stored
* chunks beyond the first are enqueued on `queue`). Nulls out consumed entries in `files`.
* Sets *failed on allocation/enqueue errors. */
static Chunk* batch_files(ArrayList* files, unsigned long long chunk_size, Queue* queue,
bool* failed) {
Chunk* first = NULL;
if (files->size <= 0)
return NULL;
ArrayList* batch = array_list_create(NULL);
if (!batch) {
*failed = true;
return NULL;
}
unsigned long long batch_size = 0;
for (int i = 0; i < files->size; i++) {
File* f = (File*)files->items[i];
if (!array_list_add(batch, f)) {
*failed = true;
break;
}
batch_size += f->data->size;
if (batch_size >= chunk_size || i == files->size - 1) {
void** items = array_list_to_array(batch);
if (!items) {
*failed = true;
array_list_delete(batch);
batch = NULL;
break;
}
Chunk* c = chunk_create((File**)items, batch->size);
free(items);
if (!c) {
*failed = true;
array_list_delete(batch);
batch = NULL;
break;
}
int batch_start = i - batch->size + 1;
for (int j = batch_start; j <= i; j++)
files->items[j] = NULL;
batch->item_destroyer = NULL;
array_list_delete(batch);
batch = NULL;
if (!first) {
first = c;
} else {
if (!queue_enqueue(queue, c)) {
chunk_destroy(c);
*failed = true;
}
}
if (i < files->size - 1) {
batch = array_list_create(NULL);
if (!batch) {
*failed = true;
break;
}
batch_size = 0;
}
}
}
if (batch) {
batch->item_destroyer = NULL;
array_list_delete(batch);
}
return first;
}
/* Scan one root-directory entry into either the subdirs or files list. */
static void scan_root_entry(const ScannerOptions* options, const char* root_directory,
const struct dirent* entry, ArrayList* root_files, ArrayList* subdirs,
ParallelScanner* ps) {
ScannerEntry inspected;
int inspection =
scanner_inspect_entry(options, root_directory, root_directory, entry->d_name, &inspected);
if (inspection < 0) {
ps->failed = true;
return;
}
if (inspection == 0)
return;
char* cur_path = inspected.path;
struct stat st = inspected.stats;
if (inspected.is_directory) {
if (!array_list_add(subdirs, cur_path)) {
free(cur_path);
ps->failed = true;
}
return;
}
File* file = file_create(cur_path);
free(cur_path);
if (!file) {
ps->failed = true;
return;
}
file->data->size = st.st_size;
if (options->use_metadata)
file->metadata = file_metadata_create(&st);
if (options->use_metadata && !file->metadata) {
file_destroy(file);
ps->failed = true;
return;
}
if (!array_list_add(root_files, file)) {
file_destroy(file);
ps->failed = true;
}
}
/* Scan the root directory itself, collecting root files and subdirectories.
* Returns false if the root directory could not be opened. */
static bool scan_root_directory(ParallelScanner* ps, const char* root_directory,
const ScannerOptions* options, ArrayList* root_files,
ArrayList* subdirs) {
DIR* dir = opendir(root_directory);
if (!dir) {
log_perror("Could not open root directory for parallel scan");
return false;
}
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
scan_root_entry(options, root_directory, entry, root_files, subdirs, ps);
}
closedir(dir);
return true;
}
/* Spawn worker threads, one per group of subdirectories. */
static void spawn_parallel_workers(ParallelScanner* ps, ArrayList* subdirs,
const ScannerOptions* options, unsigned long long cs) {
if (subdirs->size <= 0)
return;
int n = options->num_threads > 0 ? options->num_threads : 4;
if (n > subdirs->size)
n = subdirs->size;
ps->num_threads = n;
ps->expected_threads = n;
ps->threads = calloc(n, sizeof(thrd_t));
if (!ps->threads) {
ps->num_threads = 0;
ps->expected_threads = 0;
ps->failed = true;
return;
}
int dirs_per_thread = subdirs->size / n;
int remainder = subdirs->size % n;
int start = 0;
ps->num_threads = 0;
for (int t = 0; t < n; t++) {
int count = dirs_per_thread + (t < remainder ? 1 : 0);
if (count == 0)
break;
ParallelWorkerArg* wa = calloc(1, sizeof(ParallelWorkerArg));
if (!wa) {
parallel_scanner_creation_failed(ps);
break;
}
wa->ps = ps;
wa->dirs = calloc(count, sizeof(char*));
if (!wa->dirs) {
free(wa);
parallel_scanner_creation_failed(ps);
break;
}
bool dup_ok = true;
for (int j = 0; j < count; j++) {
wa->dirs[j] = str_dup((char*)subdirs->items[start + j]);
if (!wa->dirs[j])
dup_ok = false;
}
if (!dup_ok) {
for (int j = 0; j < count; j++)
free(wa->dirs[j]);
free(wa->dirs);
free(wa);
parallel_scanner_creation_failed(ps);
break;
}
wa->dir_count = count;
wa->options = *options;
wa->options.chunk_size = cs;
wa->allocation_session = ps->allocation_session;
start += count;
if (thrd_create(&ps->threads[t], parallel_worker_thread, wa) != thrd_success) {
for (int j = 0; j < count; j++)
free(wa->dirs[j]);
free(wa->dirs);
free(wa);
parallel_scanner_creation_failed(ps);
break;
}
ps->num_threads++;
ps->created_threads++;
}
}
ParallelScanner* parallel_scanner_create_with_options(const char* root_directory,
const ScannerOptions* options,
ProtocolSession* allocation_session) {
if (!root_directory || !options)
return NULL;
ParallelScanner* ps = calloc(1, sizeof(ParallelScanner));
if (!ps)
return NULL;
if (!parallel_scanner_init(ps)) {
free(ps); free(ps);
return NULL; return NULL;
} }
ps->allocation_session = allocation_session;
DIR* dir = opendir(root_directory);
if (!dir) {
perror("Could not open root directory for parallel scan");
parallel_scanner_destroy(ps);
return NULL;
}
ArrayList* root_files = array_list_create(file_destroy); ArrayList* root_files = array_list_create(file_destroy);
ArrayList* subdirs = array_list_create(free); ArrayList* subdirs = array_list_create(free);
if (!root_files || !subdirs) { if (!root_files || !subdirs) {
array_list_delete(root_files); array_list_delete(root_files);
array_list_delete(subdirs); array_list_delete(subdirs);
closedir(dir);
parallel_scanner_destroy(ps); parallel_scanner_destroy(ps);
return NULL; return NULL;
} }
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) { if (!scan_root_directory(ps, root_directory, options, root_files, subdirs)) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) array_list_delete(root_files);
continue; array_list_delete(subdirs);
ScannerEntry inspected; parallel_scanner_destroy(ps);
int inspection = return NULL;
scanner_inspect_entry(options, root_directory, root_directory, entry->d_name, &inspected);
if (inspection < 0) {
ps->failed = true;
continue;
}
if (inspection == 0)
continue;
char* cur_path = inspected.path;
struct stat st = inspected.stats;
if (inspected.is_directory) {
if (!array_list_add(subdirs, cur_path)) {
free(cur_path);
ps->failed = true;
}
} else {
File* file = file_create(cur_path);
free(cur_path);
if (!file) {
ps->failed = true;
continue;
}
file->data->size = st.st_size;
if (options->use_metadata)
file->metadata = file_metadata_create(&st);
if (options->use_metadata && !file->metadata) {
file_destroy(file);
ps->failed = true;
continue;
}
if (!array_list_add(root_files, file)) {
file_destroy(file);
ps->failed = true;
}
}
} }
closedir(dir);
unsigned long long cs = options->chunk_size > 0 ? options->chunk_size : DESIRED_CHUNK_SIZE; unsigned long long cs = options->chunk_size > 0 ? options->chunk_size : DESIRED_CHUNK_SIZE;
if (root_files->size > 0) { ps->initial_chunk = batch_files(root_files, cs, ps->result_queue, &ps->failed);
ArrayList* batch = array_list_create(NULL);
if (!batch) {
ps->failed = true;
array_list_delete(root_files);
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
return NULL;
}
unsigned long long batch_size = 0;
Chunk* first = NULL;
for (int i = 0; i < root_files->size; i++) {
File* f = (File*)root_files->items[i];
if (!array_list_add(batch, f)) {
ps->failed = true;
break;
}
batch_size += f->data->size;
if (batch_size >= cs || i == root_files->size - 1) {
void** items = array_list_to_array(batch);
if (!items) {
ps->failed = true;
batch->item_destroyer = file_destroy;
array_list_delete(batch);
batch = NULL;
break;
}
Chunk* c = chunk_create((File**)items, batch->size);
free(items);
if (!c) {
ps->failed = true;
batch->item_destroyer = file_destroy;
array_list_delete(batch);
batch = NULL;
break;
}
batch->item_destroyer = NULL;
array_list_delete(batch);
batch = NULL;
if (!first) {
first = c;
} else {
if (!queue_enqueue(ps->result_queue, c)) {
chunk_destroy(c);
ps->failed = true;
}
}
if (i < root_files->size - 1) {
batch = array_list_create(NULL);
if (!batch) {
ps->failed = true;
break;
}
batch_size = 0;
}
}
}
if (batch) {
batch->item_destroyer = NULL;
array_list_delete(batch);
}
ps->initial_chunk = first;
root_files->item_destroyer = NULL;
}
array_list_delete(root_files); array_list_delete(root_files);
int n = options->num_threads > 0 ? options->num_threads : 4; spawn_parallel_workers(ps, subdirs, options, cs);
if (n > subdirs->size)
n = subdirs->size > 0 ? subdirs->size : 1;
if (subdirs->size > 0) {
ps->num_threads = n;
ps->expected_threads = n;
ps->threads = calloc(n, sizeof(thrd_t));
if (!ps->threads) {
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
return NULL;
}
int dirs_per_thread = subdirs->size / n;
int remainder = subdirs->size % n;
int start = 0;
ps->num_threads = 0;
for (int t = 0; t < n; t++) {
int count = dirs_per_thread + (t < remainder ? 1 : 0);
if (count == 0)
break;
ParallelWorkerArg* wa = calloc(1, sizeof(ParallelWorkerArg));
if (!wa) {
ps->failed = true;
break;
}
wa->ps = ps;
wa->dirs = calloc(count, sizeof(char*));
if (!wa->dirs) {
free(wa);
ps->failed = true;
break;
}
bool dup_ok = true;
for (int j = 0; j < count; j++) {
wa->dirs[j] = str_dup((char*)subdirs->items[start + j]);
if (!wa->dirs[j])
dup_ok = false;
}
if (!dup_ok) {
for (int j = 0; j < count; j++)
free(wa->dirs[j]);
free(wa->dirs);
free(wa);
ps->failed = true;
break;
}
wa->dir_count = count;
wa->options = *options;
wa->options.chunk_size = cs;
start += count;
if (thrd_create(&ps->threads[t], parallel_worker_thread, wa) != thrd_success) {
for (int j = 0; j < count; j++)
free(wa->dirs[j]);
free(wa->dirs);
free(wa);
ps->failed = true;
atomic_store(&ps->cancelled, true);
ps->expected_threads = ps->created_threads;
mtx_lock(&ps->result_mutex);
cnd_broadcast(&ps->result_not_empty);
cnd_broadcast(&ps->result_not_full);
mtx_unlock(&ps->result_mutex);
break;
}
ps->num_threads++;
ps->created_threads++;
}
}
array_list_delete(subdirs); array_list_delete(subdirs);
return ps; return ps;
} }
ParallelScanner* parallel_scanner_create(const char* root_directory, bool use_metadata,
unsigned long long chunk_size, char** exclude_patterns,
int exclude_count, char** include_patterns,
int include_count, unsigned long long max_size,
unsigned long long min_size, int max_depth,
int num_threads, bool follow_symlinks, bool copy_links,
bool safe_links, bool copy_unsafe_links, bool checksum) {
ScannerOptions options = {use_metadata, chunk_size, exclude_patterns, exclude_count,
include_patterns, include_count, max_size, min_size,
max_depth, num_threads, follow_symlinks, copy_links,
safe_links, copy_unsafe_links, checksum};
return parallel_scanner_create_with_options(root_directory, &options);
}
Chunk* parallel_scanner_next(ParallelScanner* ps) { Chunk* parallel_scanner_next(ParallelScanner* ps) {
if (ps->initial_chunk) { if (ps->initial_chunk) {
Chunk* c = ps->initial_chunk; Chunk* c = ps->initial_chunk;
@@ -659,6 +701,11 @@ Chunk* parallel_scanner_next(ParallelScanner* ps) {
} }
if (ps->num_threads == 0) { if (ps->num_threads == 0) {
mtx_lock(&ps->result_mutex); mtx_lock(&ps->result_mutex);
if (!queue_is_empty(ps->result_queue)) {
Chunk* chunk = queue_dequeue(ps->result_queue);
mtx_unlock(&ps->result_mutex);
return chunk;
}
ps->done = true; ps->done = true;
mtx_unlock(&ps->result_mutex); mtx_unlock(&ps->result_mutex);
return NULL; return NULL;
+4 -8
View File
@@ -2,6 +2,7 @@
#define SCANNER_H #define SCANNER_H
#include "chunk.h" #include "chunk.h"
#include "protocol.h"
#include "queue.h" #include "queue.h"
#include <dirent.h> #include <dirent.h>
#include <stdbool.h> #include <stdbool.h>
@@ -62,6 +63,7 @@ typedef struct {
atomic_bool cancelled; atomic_bool cancelled;
int completed; int completed;
Chunk* initial_chunk; Chunk* initial_chunk;
ProtocolSession* allocation_session;
} ParallelScanner; } ParallelScanner;
DirectoryScanner* directory_scanner_create(const char* root_directory, bool use_metadata, DirectoryScanner* directory_scanner_create(const char* root_directory, bool use_metadata,
@@ -77,15 +79,9 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner);
bool directory_scanner_failed(const DirectoryScanner* scanner); bool directory_scanner_failed(const DirectoryScanner* scanner);
void directory_scanner_destroy(DirectoryScanner* scanner); void directory_scanner_destroy(DirectoryScanner* scanner);
ParallelScanner* parallel_scanner_create(const char* root_directory, bool use_metadata,
unsigned long long chunk_size, char** exclude_patterns,
int exclude_count, char** include_patterns,
int include_count, unsigned long long max_size,
unsigned long long min_size, int max_depth,
int num_threads, bool follow_symlinks, bool copy_links,
bool safe_links, bool copy_unsafe_links, bool checksum);
ParallelScanner* parallel_scanner_create_with_options(const char* root_directory, ParallelScanner* parallel_scanner_create_with_options(const char* root_directory,
const ScannerOptions* options); const ScannerOptions* options,
ProtocolSession* allocation_session);
Chunk* parallel_scanner_next(ParallelScanner* scanner); Chunk* parallel_scanner_next(ParallelScanner* scanner);
bool parallel_scanner_failed(const ParallelScanner* scanner); bool parallel_scanner_failed(const ParallelScanner* scanner);
void parallel_scanner_destroy(ParallelScanner* scanner); void parallel_scanner_destroy(ParallelScanner* scanner);
+40
View File
@@ -18,26 +18,51 @@ void print_usage(void) {
printf(" -z [level] Alias for -c\n"); printf(" -z [level] Alias for -c\n");
printf(" -a, --archive Archive mode (-c -m -M)\n"); printf(" -a, --archive Archive mode (-c -m -M)\n");
printf(" -n, --dry-run Show what would be transferred\n"); printf(" -n, --dry-run Show what would be transferred\n");
printf(" --remove-source-files Remove regular source files after successful transfer\n");
printf(" -p <port> SSH port (default: 22)\n"); printf(" -p <port> SSH port (default: 22)\n");
printf(" --progress Show transfer progress\n"); printf(" --progress Show transfer progress\n");
printf(" -P Partial mode with progress (retention incomplete)\n");
printf(" -8, --8-bit-output Leave high-bit characters unescaped in output\n");
printf(" --delete Delete files on receiver not in source\n"); printf(" --delete Delete files on receiver not in source\n");
printf(" --ignore-existing Skip files that already exist on receiver\n");
printf(
" --dirs, --old-dirs, --old-d Transfer directories without recursing (not implemented)\n");
printf(" --del Alias for --delete-during (not implemented)\n");
printf(" --exclude <pattern> Exclude files matching pattern\n"); printf(" --exclude <pattern> Exclude files matching pattern\n");
printf(" --include <pattern> Only include files matching pattern\n"); printf(" --include <pattern> Only include files matching pattern\n");
printf(" --exclude-from <file> Read exclude patterns from file\n"); printf(" --exclude-from <file> Read exclude patterns from file\n");
printf(" --include-from <file> Read include patterns from file\n"); printf(" --include-from <file> Read include patterns from file\n");
printf(" --max-size <n> Skip files larger than n bytes\n"); printf(" --max-size <n> Skip files larger than n bytes\n");
printf(" --min-size <n> Skip files smaller than n bytes\n"); printf(" --min-size <n> Skip files smaller than n bytes\n");
printf(" --max-alloc <SIZE> Maximum single allocation (default: 1G)\n");
printf(" --incremental Skip files unchanged since last transfer\n"); printf(" --incremental Skip files unchanged since last transfer\n");
printf(" --size-only Skip incremental files matching in size, ignoring mtime\n");
printf(" -I, --ignore-times Transfer files even when size and mtime match\n");
printf(" -@, --modify-window <sec> Modification time tolerance\n");
printf(" -u, --update Skip files newer than the source on receiver\n");
printf(" --existing Skip files not already present at destination\n");
printf(" --checksum-choice, --cc <alg> Checksum algorithm (not supported yet; xxHash64 is "
"used)\n");
printf(" --delta Delta transfer for changed files (requires --incremental)\n"); printf(" --delta Delta transfer for changed files (requires --incremental)\n");
printf(" -W, --whole-file Transfer changed files without delta processing\n");
printf(" --delta-block <n> Delta block size in bytes (default: %d)\n", printf(" --delta-block <n> Delta block size in bytes (default: %d)\n",
DELTA_BLOCK_SIZE_DEFAULT); DELTA_BLOCK_SIZE_DEFAULT);
printf(" --delta-max <n> Max file size for delta transfer (default: %llu)\n", printf(" --delta-max <n> Max file size for delta transfer (default: %llu)\n",
DELTA_MAX_FILE_SIZE); DELTA_MAX_FILE_SIZE);
printf(" -m Enable multithreading\n"); printf(" -m Enable multithreading\n");
printf(" -s Enable chunk serialization\n"); printf(" -s Enable chunk serialization\n");
printf(" --secluded-args Accept rsync compatibility option (no effect)\n");
printf(" -f Enable sendfile (TCP only, not with -c or -s)\n"); printf(" -f Enable sendfile (TCP only, not with -c or -s)\n");
printf(" --compress-choice <alg> Compression algorithm (default: zstd)\n");
printf(" --zc <alg> Alias for --compress-choice\n");
printf(" -v, --verbose Enable debug logging\n"); printf(" -v, --verbose Enable debug logging\n");
printf(" -q, --quiet Suppress non-error output\n");
printf(" --debug=FLAGS Fine-grained debug logging (use --debug=help for flags)\n");
printf(" --info=FLAGS Fine-grained info: copy,misc,skip,stats,all,none\n");
printf(" none suppresses info even with --verbose\n");
printf(" -M, --preserve Preserve file metadata\n"); printf(" -M, --preserve Preserve file metadata\n");
printf(" -E, --executability Preserve executable permission bits\n");
printf(" --chmod <changes> Modify transferred permissions (rsync syntax)\n");
printf(" --chunk-size <n> Chunk size in bytes (default: %d)\n", DEFAULT_CHUNK_SIZE); printf(" --chunk-size <n> Chunk size in bytes (default: %d)\n", DEFAULT_CHUNK_SIZE);
printf(" --source-dir <path> Source directory\n"); printf(" --source-dir <path> Source directory\n");
printf(" --dest-dir <path> Destination directory\n"); printf(" --dest-dir <path> Destination directory\n");
@@ -56,19 +81,34 @@ void print_usage(void) {
printf(" --backup-dir <dir> Directory for backups (requires --backup)\n"); printf(" --backup-dir <dir> Directory for backups (requires --backup)\n");
printf(" --suffix <str> Backup suffix (default: ~)\n"); printf(" --suffix <str> Backup suffix (default: ~)\n");
printf(" --stats Print transfer statistics at end\n"); printf(" --stats Print transfer statistics at end\n");
printf(" -h, --human-readable Print byte sizes in human-readable form\n");
printf(" --max-depth <n> Maximum directory depth (0=unlimited)\n"); printf(" --max-depth <n> Maximum directory depth (0=unlimited)\n");
printf(" --log-file <path> Write log messages to file\n"); printf(" --log-file <path> Write log messages to file\n");
printf(" --stderr=MODE Route logging to stderr: errors or all\n");
printf(" --partial Keep partial files on interrupted transfer\n"); printf(" --partial Keep partial files on interrupted transfer\n");
printf(" --partial-dir <dir> Directory for partial files\n"); printf(" --partial-dir <dir> Directory for partial files\n");
printf(" --fastsync-server-path <path>\n"); printf(" --fastsync-server-path <path>\n");
printf(" Path to fastsync-server on remote (default: fastsync-server)\n"); printf(" Path to fastsync-server on remote (default: fastsync-server)\n");
printf(
" --old-args Disable safe SSH command argument quoting (legacy compatibility)\n");
printf(" -l, --links Copy symlinks as symlinks\n"); printf(" -l, --links Copy symlinks as symlinks\n");
printf(" --copy-links Transform symlinks into referent files\n"); printf(" --copy-links Transform symlinks into referent files\n");
printf(" --safe-links Skip symlinks that point outside transfer tree\n"); printf(" --safe-links Skip symlinks that point outside transfer tree\n");
printf(" --copy-unsafe-links Only transform unsafe symlinks into referent files\n"); printf(" --copy-unsafe-links Only transform unsafe symlinks into referent files\n");
printf(" -S, --sparse Handle sparse files efficiently\n"); printf(" -S, --sparse Handle sparse files efficiently\n");
printf(" --inplace Update files in-place (no temp+rename)\n"); printf(" --inplace Update files in-place (no temp+rename)\n");
printf(" --fsync Fsync every written file before publication\n");
printf(" --compress-level <n> Compression level (default: 5)\n"); printf(" --compress-level <n> Compression level (default: 5)\n");
printf(" --zl <n> Alias for --compress-level\n");
printf(" --skip-compress=LIST Skip compression for comma-separated suffixes\n");
printf(" --compress-threads <n> Compression worker threads (requires zstd threaded support)\n");
printf(" --no-OPTION Disable a supported boolean option\n");
printf(" --help Show this help\n"); printf(" --help Show this help\n");
printf(" -V, --version Show version\n"); printf(" -V, --version Show version\n");
} }
void print_debug_usage(void) {
printf("Supported debug flags: IO,PROTO,PACK,UTIL,ALL,NONE\n");
printf("Flags may be comma-separated, for example: --debug=io,proto\n");
printf("Other rsync debug flags are unsupported and rejected.\n");
}
+1
View File
@@ -2,5 +2,6 @@
#define USAGE_H #define USAGE_H
void print_usage(void); void print_usage(void);
void print_debug_usage(void);
#endif #endif
+116 -13
View File
@@ -1,15 +1,65 @@
#include "receiver.h" #include "receiver.h"
#include "chunk.h" #include "chunk.h"
#include "file_receive.h"
#include "log.h" #include "log.h"
#include "metadata.h"
#include "protocol.h" #include "protocol.h"
#include "utils.h" #include "utils.h"
#include <stdlib.h> #include <stdlib.h>
#include <sys/stat.h> #include <sys/stat.h>
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code) {
if (!outcomes)
return false;
if (outcomes->count == outcomes->capacity) {
size_t new_capacity = outcomes->capacity == 0 ? 64 : outcomes->capacity * 2;
if (new_capacity < outcomes->capacity)
return false;
unsigned char* grown = realloc(outcomes->entries, new_capacity);
if (!grown)
return false;
outcomes->entries = grown;
outcomes->capacity = new_capacity;
}
outcomes->entries[outcomes->count++] = code;
return true;
}
void receiver_outcomes_destroy(ReceiverOutcomes* outcomes) {
if (!outcomes)
return;
free(outcomes->entries);
outcomes->entries = NULL;
outcomes->count = 0;
outcomes->capacity = 0;
}
/* End-of-transfer success frame. When --remove-source-files was negotiated
each processed data file is acknowledged first (STATUS_NEXT = written,
STATUS_OK = skipped) so the sender never removes a source the receiver did
not actually store. The frame always ends with a plain STATUS_OK. */
bool receiver_send_final_success(int fd, const Config* config, const ReceiverOutcomes* outcomes) {
if (!config->remove_source_files)
return send_status(fd, STATUS_OK);
size_t count = outcomes ? outcomes->count : 0;
for (size_t i = 0; i < count; i++) {
Status per_file = outcomes->entries[i] == FILE_SAVE_WRITTEN ? STATUS_NEXT : STATUS_OK;
if (!send_status(fd, per_file))
return false;
}
return send_status(fd, STATUS_OK);
}
static bool receiver_process_chunk(Chunk* chunk, const ReceiverSink* sink) { static bool receiver_process_chunk(Chunk* chunk, const ReceiverSink* sink) {
if (!chunk || !sink || !sink->store_file)
return false;
for (int i = 0; i < chunk->element_count; i++) { for (int i = 0; i < chunk->element_count; i++) {
File* file = chunk->items[i]; File* file = chunk->items[i];
if (!file) {
chunk_destroy(chunk);
return false;
}
chunk->items[i] = NULL; chunk->items[i] = NULL;
if (!sink->store_file(file, sink->context)) { if (!sink->store_file(file, sink->context)) {
chunk_destroy(chunk); chunk_destroy(chunk);
@@ -31,9 +81,13 @@ static bool receiver_process_batch(Config* config, int file_descriptor) {
return false; return false;
unsigned long long check_size; unsigned long long check_size;
long long check_mtime; long long check_mtime;
long long check_mtime_nsec;
if (!receive_n_data(file_descriptor, &check_size, sizeof(check_size)) || if (!receive_n_data(file_descriptor, &check_size, sizeof(check_size)) ||
!receive_n_data(file_descriptor, &check_mtime, sizeof(check_mtime))) { !receive_n_data(file_descriptor, &check_mtime, sizeof(check_mtime)) ||
!receive_n_data(file_descriptor, &check_mtime_nsec, sizeof(check_mtime_nsec)) ||
check_mtime_nsec < 0 || check_mtime_nsec >= 1000000000LL) {
free(check_path); free(check_path);
send_status(file_descriptor, STATUS_ERROR);
return false; return false;
} }
if (!utils_valid_batch_path(check_path)) { if (!utils_valid_batch_path(check_path)) {
@@ -41,11 +95,28 @@ static bool receiver_process_batch(Config* config, int file_descriptor) {
send_status(file_descriptor, STATUS_ERROR); send_status(file_descriptor, STATUS_ERROR);
return false; return false;
} }
if (check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
free(check_path);
send_status(file_descriptor, STATUS_ERROR);
return false;
}
char* full_path = path_cat(config->receive_root_directory, check_path); char* full_path = path_cat(config->receive_root_directory, check_path);
if (!full_path) {
free(check_path);
send_status(file_descriptor, STATUS_ERROR);
return false;
}
struct stat st; struct stat st;
bool has_old = full_path && lstat(full_path, &st) == 0; bool has_old = file_stat_secure(full_path, &st);
bool match = has_old && (unsigned long long)st.st_size == check_size && long long old_mtime_nsec = 0;
(long long)st.st_mtime == check_mtime; if (has_old) {
#ifdef __linux__
old_mtime_nsec = st.st_mtim.tv_nsec;
#endif
}
bool match = !config->ignore_times && has_old && (unsigned long long)st.st_size == check_size &&
metadata_mtime_matches(st.st_mtime, old_mtime_nsec, (time_t)check_mtime,
(long)check_mtime_nsec, config->modify_window);
bool sent = send_status(file_descriptor, match ? STATUS_OK : STATUS_NEXT); bool sent = send_status(file_descriptor, match ? STATUS_OK : STATUS_NEXT);
free(full_path); free(full_path);
free(check_path); free(check_path);
@@ -102,8 +173,14 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status"); log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
goto receive_error; goto receive_error;
} }
if (sink->send_success && !send_status(file_descriptor, STATUS_OK)) if (sink->send_success) {
return -1; if (sink->send_success_frame) {
if (!sink->send_success_frame(file_descriptor, sink->context))
return -1;
} else if (!send_status(file_descriptor, STATUS_OK)) {
return -1;
}
}
return 0; return 0;
receive_error: receive_error:
@@ -112,15 +189,41 @@ receive_error:
return -1; return -1;
} }
static bool receiver_save_file(File* file, void* context) { /* ---- Single-threaded sink (used by receiver_receive_files) ---- */
Config* config = context;
bool success = typedef struct {
!config->save_to_disk || file_save_to_disk(config->receive_root_directory, file, config); Config* config;
ReceiverOutcomes outcomes;
} ReceiverSaveContext;
static bool receiver_save_file(File* file, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
FileSaveResult result = FILE_SAVE_ERROR;
if (!context->config->save_to_disk) {
/* Nothing is stored; report the file as not-written so a
--remove-source-files sender keeps its source. */
result = FILE_SAVE_SKIPPED;
} else {
result = file_save_to_disk_full(context->config->receive_root_directory, file, context->config);
}
if (result != FILE_SAVE_ERROR && context->config->remove_source_files &&
!receiver_outcomes_append(&context->outcomes, (unsigned char)result)) {
file_destroy(file);
return false;
}
file_destroy(file); file_destroy(file);
return success; return result != FILE_SAVE_ERROR;
}
static bool receiver_send_success_frame(int fd, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
return receiver_send_final_success(fd, context->config, &context->outcomes);
} }
int receiver_receive_files(Config* config, int file_descriptor) { int receiver_receive_files(Config* config, int file_descriptor) {
ReceiverSink sink = {receiver_save_file, config, true, true}; ReceiverSaveContext context = {.config = config, .outcomes = {0}};
return receiver_process(config, file_descriptor, &sink); ReceiverSink sink = {receiver_save_file, &context, true, true, receiver_send_success_frame};
int ret = receiver_process(config, file_descriptor, &sink);
receiver_outcomes_destroy(&context.outcomes);
return ret;
} }
+21
View File
@@ -3,16 +3,37 @@
#include "config.h" #include "config.h"
#include "file.h" #include "file.h"
#include "file_receive.h"
typedef bool (*ReceiverFileSink)(File* file, void* context); typedef bool (*ReceiverFileSink)(File* file, void* context);
/* Ordered per-file save outcomes for one connection. One entry is appended
for every data-bearing file the receiver processes (in the order the files
were sent) so the sender of a --remove-source-files transfer can be told
which sources were actually written versus skipped on the receiver. */
typedef struct {
unsigned char* entries; /* FILE_SAVE_WRITTEN or FILE_SAVE_SKIPPED */
size_t count;
size_t capacity;
} ReceiverOutcomes;
typedef bool (*ReceiverSuccessFrame)(int fd, void* context);
typedef struct { typedef struct {
ReceiverFileSink store_file; ReceiverFileSink store_file;
void* context; void* context;
bool send_error; bool send_error;
bool send_success; bool send_success;
/* Emits the end-of-transfer success frame. When the sender requested
--remove-source-files this includes one per-file status per processed
data file followed by the final STATUS_OK; otherwise just STATUS_OK. */
ReceiverSuccessFrame send_success_frame;
} ReceiverSink; } ReceiverSink;
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code);
void receiver_outcomes_destroy(ReceiverOutcomes* outcomes);
bool receiver_send_final_success(int fd, const Config* config, const ReceiverOutcomes* outcomes);
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink); int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink);
int receiver_receive_files(Config* config, int file_descriptor); int receiver_receive_files(Config* config, int file_descriptor);
+139 -35
View File
@@ -2,11 +2,11 @@
#include "file.h" #include "file.h"
#include "log.h" #include "log.h"
#include "multiprocessing.h" #include "multiprocessing.h"
#include "protocol.h"
#include "queue.h" #include "queue.h"
#include "receiver.h" #include "receiver.h"
#include "transport_tcp.h" #include "transport_tcp.h"
#include "transport_tls.h" #include "transport_tls.h"
#include "unistd.h"
#include "utils.h" #include "utils.h"
#include <fcntl.h> #include <fcntl.h>
#include <limits.h> #include <limits.h>
@@ -14,11 +14,48 @@
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <sys/stat.h> #include <unistd.h>
#include <openssl/x509.h>
static char* authorized_root; static char* authorized_root;
static int authorized_root_fd = -1; static int authorized_root_fd = -1;
static bool allow_delete; static bool allow_delete;
static bool allow_unauthenticated;
static const char* required_client_cn;
/* Aggregate payload bytes the multithreaded receiver may buffer ahead of the
slow disk writer. Receiving one more chunk adds up to ~2 * MAX_CHUNK_SIZE
of transient wire/decompression buffers on top of the queued payloads, so
this ceiling keeps total per-connection receive memory (decompressed and
per-file copied chunk buffers included) within MAX_CONNECTION_MEMORY. */
#define RECEIVER_QUEUE_MAX_BYTES (MAX_CONNECTION_MEMORY - 2 * MAX_CHUNK_SIZE)
static bool tls_client_identity_allowed(SSL* ssl) {
if (!ssl || !required_client_cn)
return false;
X509* certificate = SSL_get1_peer_certificate(ssl);
if (!certificate)
return false;
char common_name[256];
int length = X509_NAME_get_text_by_NID(X509_get_subject_name(certificate), NID_commonName,
common_name, sizeof(common_name));
size_t required_length = strlen(required_client_cn);
bool allowed = length >= 0 && (size_t)length == required_length &&
required_length < sizeof(common_name) &&
memcmp(common_name, required_client_cn, required_length) == 0;
X509_free(certificate);
return allowed;
}
static void release_authorization(void) {
file_set_authorized_root(-1, NULL);
utils_set_authorized_root_fd(-1);
if (authorized_root_fd >= 0)
close(authorized_root_fd);
authorized_root_fd = -1;
free(authorized_root);
authorized_root = NULL;
}
static bool path_is_within(const char* root, const char* path) { static bool path_is_within(const char* root, const char* path) {
size_t n = strlen(root); size_t n = strlen(root);
@@ -27,19 +64,44 @@ static bool path_is_within(const char* root, const char* path) {
static bool __attribute__((unused)) configure_authorization(const char* root) { static bool __attribute__((unused)) configure_authorization(const char* root) {
char resolved[PATH_MAX]; char resolved[PATH_MAX];
if (!root || !realpath(root, resolved)) if (!root) {
file_set_authorized_root(-1, NULL);
utils_set_authorized_root(-1, NULL);
return false; return false;
}
int root_fd = open(root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (root_fd < 0) {
file_set_authorized_root(-1, NULL);
utils_set_authorized_root(-1, NULL);
return false;
}
char fd_path[64];
int fd_path_length = snprintf(fd_path, sizeof(fd_path), "/proc/self/fd/%d", root_fd);
if (fd_path_length < 0 || (size_t)fd_path_length >= sizeof(fd_path) ||
!realpath(fd_path, resolved)) {
close(root_fd);
file_set_authorized_root(-1, NULL);
utils_set_authorized_root(-1, NULL);
return false;
}
authorized_root = str_dup(resolved); authorized_root = str_dup(resolved);
if (!authorized_root) if (!authorized_root) {
close(root_fd);
file_set_authorized_root(-1, NULL);
utils_set_authorized_root(-1, NULL);
return false; return false;
authorized_root_fd = open(resolved, O_RDONLY | O_DIRECTORY | O_CLOEXEC); }
if (authorized_root_fd < 0) { authorized_root_fd = root_fd;
if (!file_set_authorized_root(authorized_root_fd, authorized_root) ||
!utils_set_authorized_root(authorized_root_fd, authorized_root)) {
file_set_authorized_root(-1, NULL);
utils_set_authorized_root(-1, NULL);
close(authorized_root_fd);
authorized_root_fd = -1;
free(authorized_root); free(authorized_root);
authorized_root = NULL; authorized_root = NULL;
return false; return false;
} }
file_set_authorized_root(authorized_root_fd, authorized_root);
utils_set_authorized_root_fd(authorized_root_fd);
return true; return true;
} }
@@ -56,6 +118,7 @@ void handler(int file_descriptor) {
protocol_session_unbind(); protocol_session_unbind();
return; return;
} }
protocol_set_8_bit_output(config->eight_bit_output);
if (!authorized_root) { if (!authorized_root) {
log_message(LOG_LEVEL_ERROR, "No server-side destination root configured"); log_message(LOG_LEVEL_ERROR, "No server-side destination root configured");
config_delete(config); config_delete(config);
@@ -63,25 +126,37 @@ void handler(int file_descriptor) {
protocol_session_unbind(); protocol_session_unbind();
return; return;
} }
char resolved_destination[PATH_MAX]; if (!allow_unauthenticated && ssl == NULL) {
char* canonical_destination = realpath(config->receive_root_directory, NULL); log_message(LOG_LEVEL_ERROR, "Rejected unauthenticated plaintext connection");
const char* destination =
canonical_destination ? canonical_destination : config->receive_root_directory;
if (has_path_traversal(destination) || !path_is_within(authorized_root, destination)) {
log_message(LOG_LEVEL_ERROR, "Rejected destination outside authorized root");
free(canonical_destination);
config_delete(config); config_delete(config);
close(file_descriptor); close(file_descriptor);
protocol_session_unbind(); protocol_session_unbind();
return; return;
} }
if (canonical_destination) if (ssl && required_client_cn && !tls_client_identity_allowed(ssl)) {
snprintf(resolved_destination, sizeof(resolved_destination), "%s", canonical_destination); log_message(LOG_LEVEL_ERROR, "Rejected TLS client with unauthorized identity");
else config_delete(config);
snprintf(resolved_destination, sizeof(resolved_destination), "%s", destination); close(file_descriptor);
free(canonical_destination); return;
free(config->receive_root_directory); }
config->receive_root_directory = str_dup(resolved_destination); char* destination = config->receive_root_directory;
char* joined_destination = NULL;
if (destination && destination[0] != '/')
joined_destination = path_cat(authorized_root, destination);
if (joined_destination)
destination = joined_destination;
if (!destination || has_path_traversal(destination) ||
!path_is_within(authorized_root, destination)) {
log_message(LOG_LEVEL_ERROR, "Rejected destination outside authorized root");
free(joined_destination);
config_delete(config);
close(file_descriptor);
return;
}
if (joined_destination) {
free(config->receive_root_directory);
config->receive_root_directory = joined_destination;
}
if (!config->receive_root_directory) { if (!config->receive_root_directory) {
config_delete(config); config_delete(config);
close(file_descriptor); close(file_descriptor);
@@ -106,13 +181,17 @@ void handler(int file_descriptor) {
protocol_session_unbind(); protocol_session_unbind();
return; return;
} }
protocol_session_set_max_alloc(&context->session, config->max_alloc);
atomic_store(&context->session.total_allocated_bytes,
atomic_load(&session.total_allocated_bytes));
pipeline_context_receiver_set_queue_byte_limit(context, RECEIVER_QUEUE_MAX_BYTES);
thrd_t receiver, writer; thrd_t receiver, writer;
bool receiver_created = thrd_create(&receiver, receive_thread, context) == thrd_success; bool receiver_created = thrd_create(&receiver, receive_thread, context) == thrd_success;
bool writer_created = false; bool writer_created = false;
if (receiver_created) if (receiver_created)
writer_created = thrd_create(&writer, write_thread, context) == thrd_success; writer_created = thrd_create(&writer, write_thread, context) == thrd_success;
if (!receiver_created || !writer_created) { if (!receiver_created || !writer_created) {
perror("Error creating Threads"); log_perror("Error creating Threads");
if (receiver_created) { if (receiver_created) {
mtx_lock(&context->mutex); mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true); atomic_store(&context->cancelled, true);
@@ -134,9 +213,15 @@ void handler(int file_descriptor) {
int writer_result; int writer_result;
thrd_join(receiver, &receiver_result); thrd_join(receiver, &receiver_result);
thrd_join(writer, &writer_result); thrd_join(writer, &writer_result);
send_status(file_descriptor, receiver_result == thrd_success && writer_result == thrd_success bool transfer_ok = receiver_result == thrd_success && writer_result == thrd_success;
? STATUS_OK if (transfer_ok) {
: STATUS_ERROR); if (!receiver_send_final_success(file_descriptor, config, &context->outcomes))
transfer_ok = false;
} else {
send_status(file_descriptor, STATUS_ERROR);
}
if (!transfer_ok)
log_message(LOG_LEVEL_ERROR, "Transfer failed");
pipeline_context_receiver_destroy(context); pipeline_context_receiver_destroy(context);
} else { } else {
if (receiver_receive_files(config, file_descriptor) != 0) if (receiver_receive_files(config, file_descriptor) != 0)
@@ -167,8 +252,10 @@ static void print_server_usage(void) {
printf(" --cert <path> TLS certificate file (PEM)\n"); printf(" --cert <path> TLS certificate file (PEM)\n");
printf(" --key <path> TLS private key file (PEM)\n"); printf(" --key <path> TLS private key file (PEM)\n");
printf(" --ca <path> TLS CA certificate file (PEM)\n"); printf(" --ca <path> TLS CA certificate file (PEM)\n");
printf(" --client-cn <name> Required TLS client certificate CN\n");
printf(" --destination-root <path> Authorized destination root (default: .)\n"); printf(" --destination-root <path> Authorized destination root (default: .)\n");
printf(" --allow-delete Permit manifest deletion\n"); printf(" --allow-delete Permit manifest deletion\n");
printf(" --allow-unauthenticated Allow plaintext/anonymous network clients\n");
printf(" -v, --verbose Enable debug logging\n"); printf(" -v, --verbose Enable debug logging\n");
printf(" --help Show this help\n"); printf(" --help Show this help\n");
} }
@@ -189,6 +276,7 @@ int main(int argc, char* argv[]) {
stdio_mode = true; stdio_mode = true;
} else if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) { } else if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) {
set_log_level(LOG_LEVEL_DEBUG); set_log_level(LOG_LEVEL_DEBUG);
set_log_debug_flags(LOG_DEBUG_ALL);
} else if (strcmp(argv[i], "--tls") == 0) { } else if (strcmp(argv[i], "--tls") == 0) {
use_tls = true; use_tls = true;
} else if (strcmp(argv[i], "--cert") == 0 && i + 1 < argc) { } else if (strcmp(argv[i], "--cert") == 0 && i + 1 < argc) {
@@ -197,20 +285,29 @@ int main(int argc, char* argv[]) {
tls_key = argv[++i]; tls_key = argv[++i];
} else if (strcmp(argv[i], "--ca") == 0 && i + 1 < argc) { } else if (strcmp(argv[i], "--ca") == 0 && i + 1 < argc) {
tls_ca = argv[++i]; tls_ca = argv[++i];
} else if (strcmp(argv[i], "--client-cn") == 0 && i + 1 < argc) {
required_client_cn = argv[++i];
} else if (strcmp(argv[i], "--destination-root") == 0 && i + 1 < argc) { } else if (strcmp(argv[i], "--destination-root") == 0 && i + 1 < argc) {
destination_root = argv[++i]; destination_root = argv[++i];
} else if (strcmp(argv[i], "--allow-delete") == 0) { } else if (strcmp(argv[i], "--allow-delete") == 0) {
allow_delete = true; allow_delete = true;
} else if (strcmp(argv[i], "--allow-unauthenticated") == 0) {
allow_unauthenticated = true;
} else if (strcmp(argv[i], "-p") == 0 && i + 1 < argc) { } else if (strcmp(argv[i], "-p") == 0 && i + 1 < argc) {
char* end; char* end;
long p = strtol(argv[++i], &end, 10); long p = strtol(argv[++i], &end, 10);
if (*end || p <= 0 || p > 65535) { if (*end || p <= 0 || p > 65535) {
fprintf(stderr, "Error: invalid port '%s' (must be 1-65535)\n", argv[i]); char* escaped = output_escape(argv[i], false);
fprintf(stderr, "Error: invalid port '%s' (must be 1-65535)\n",
escaped ? escaped : "<allocation failed>");
free(escaped);
return 1; return 1;
} }
port = (int)p; port = (int)p;
} else if (argv[i][0] == '-') { } else if (argv[i][0] == '-') {
fprintf(stderr, "Unknown option: %s\n", argv[i]); char* escaped = output_escape(argv[i], false);
fprintf(stderr, "Unknown option: %s\n", escaped ? escaped : "<allocation failed>");
free(escaped);
print_server_usage(); print_server_usage();
return 1; return 1;
} }
@@ -220,39 +317,46 @@ int main(int argc, char* argv[]) {
signal(SIGINT, cleanup); signal(SIGINT, cleanup);
signal(SIGTERM, cleanup); signal(SIGTERM, cleanup);
if (!configure_authorization(destination_root)) { if (!configure_authorization(destination_root)) {
fprintf(stderr, "Error: invalid destination root '%s'\n", destination_root); char* escaped = output_escape(destination_root, false);
fprintf(stderr, "Error: invalid destination root '%s'\n",
escaped ? escaped : "<allocation failed>");
free(escaped);
return 1; return 1;
} }
if (stdio_mode) { if (stdio_mode) {
/* SSH authenticates the stdio transport outside of FastSync. */
allow_unauthenticated = true;
io_set_fds(STDIN_FILENO, STDOUT_FILENO); io_set_fds(STDIN_FILENO, STDOUT_FILENO);
handler(STDIN_FILENO); handler(STDIN_FILENO);
file_set_authorized_root(-1, NULL); release_authorization();
utils_set_authorized_root_fd(-1);
close(authorized_root_fd);
free(authorized_root);
return 0; return 0;
} }
g_server = server_create(port); g_server = server_create(port);
if (!g_server) { if (!g_server) {
log_message(LOG_LEVEL_ERROR, "Failed to create server"); log_message(LOG_LEVEL_ERROR, "Failed to create server");
release_authorization();
return 1; return 1;
} }
if (use_tls) { if (use_tls) {
if (!tls_cert || !tls_key) { if (!tls_cert || !tls_key || !tls_ca || !required_client_cn) {
fprintf(stderr, "Error: --tls requires --cert and --key\n"); fprintf(stderr, "Error: --tls requires --cert, --key, --ca, and --client-cn\n");
server_delete(&g_server); server_delete(&g_server);
release_authorization();
return 1; return 1;
} }
tls_global_init(); tls_global_init();
if (!server_create_tls(g_server, tls_cert, tls_key, tls_ca)) { if (!server_create_tls(g_server, tls_cert, tls_key, tls_ca)) {
log_message(LOG_LEVEL_ERROR, "Failed to set up TLS"); log_message(LOG_LEVEL_ERROR, "Failed to set up TLS");
server_delete(&g_server); server_delete(&g_server);
release_authorization();
return 1; return 1;
} }
server_listen_tls(g_server, handler); server_listen_tls(g_server, handler);
} else { } else {
server_listen(g_server, handler); server_listen(g_server, handler);
} }
server_delete(&g_server);
release_authorization();
return 0; return 0;
} }
#endif #endif
+10 -8
View File
@@ -1,16 +1,18 @@
#include "log.h"
#include "array_list.h" #include "array_list.h"
#include "protocol.h"
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
ArrayList* array_list_create(void (*item_destroyer)(void* item)) { ArrayList* array_list_create(void (*item_destroyer)(void* item)) {
ArrayList* list = (ArrayList*)malloc(sizeof(ArrayList)); ArrayList* list = (ArrayList*)protocol_alloc(sizeof(ArrayList));
if (list == NULL) { if (list == NULL) {
perror("ERROR: Could not allocate memory for array list struct"); log_perror("ERROR: Could not allocate memory for array list struct");
return NULL; return NULL;
} }
list->items = malloc(INITIAL_ARRAY_SIZE * sizeof(void*)); list->items = protocol_alloc(INITIAL_ARRAY_SIZE * sizeof(void*));
if (list->items == NULL) { if (list->items == NULL) {
free(list); free(list);
return NULL; return NULL;
@@ -34,15 +36,15 @@ void array_list_delete(ArrayList* array_list) {
free(array_list); free(array_list);
} }
bool array_list_extend(ArrayList* array_list) { static bool array_list_extend(ArrayList* array_list) {
if (array_list == NULL) if (array_list == NULL)
return false; return false;
int new_capacity = array_list->capacity * 2; int new_capacity = array_list->capacity * 2;
if (new_capacity == 0) if (new_capacity == 0)
new_capacity = INITIAL_ARRAY_SIZE; new_capacity = INITIAL_ARRAY_SIZE;
void* new_items = realloc(array_list->items, new_capacity * sizeof(void*)); void* new_items = protocol_realloc(array_list->items, new_capacity * sizeof(void*));
if (new_items == NULL) { if (new_items == NULL) {
perror("ERROR: Could not reallocate memory for array list items"); log_perror("ERROR: Could not reallocate memory for array list items");
return false; return false;
} }
array_list->items = new_items; array_list->items = new_items;
@@ -66,9 +68,9 @@ void** array_list_to_array(const ArrayList* array_list) {
if (array_list == NULL) { if (array_list == NULL) {
return NULL; return NULL;
} }
void** array = malloc(array_list->size * sizeof(void*)); void** array = protocol_alloc(array_list->size * sizeof(void*));
if (array == NULL) { if (array == NULL) {
perror("Could not malloc space for array from array list!"); log_perror("Could not malloc space for array from array list!");
return NULL; return NULL;
} }
memcpy(array, array_list->items, array_list->size * sizeof(void*)); memcpy(array, array_list->items, array_list->size * sizeof(void*));
-1
View File
@@ -14,7 +14,6 @@ typedef struct ArrayList {
ArrayList* array_list_create(void (*item_destroyer)(void* item)); ArrayList* array_list_create(void (*item_destroyer)(void* item));
void array_list_delete(ArrayList* array_list); void array_list_delete(ArrayList* array_list);
bool array_list_extend(ArrayList* array_list);
bool array_list_add(ArrayList* array_list, void* item); bool array_list_add(ArrayList* array_list, void* item);
void** array_list_to_array(const ArrayList* array_list); void** array_list_to_array(const ArrayList* array_list);
+90
View File
@@ -0,0 +1,90 @@
#include "chmod.h"
#include <stddef.h>
#include <string.h>
static bool parse_clause(mode_t* mode, const char* begin, const char* end) {
const char* p = begin;
unsigned who = 0;
while (p < end && strchr("ugoa", *p)) {
if (*p == 'a')
who = 7;
else
who |= *p == 'u' ? 1U : (*p == 'g' ? 2U : 4U);
p++;
}
if (who == 0)
who = 7;
if (p == end || (*p != '+' && *p != '-' && *p != '='))
return false;
char operation = *p++;
mode_t bits = 0;
while (p < end) {
mode_t bit;
switch (*p++) {
case 'r':
bit = 4;
break;
case 'w':
bit = 2;
break;
case 'x':
bit = 1;
break;
default:
return false;
}
bits |= bit;
}
for (unsigned class_index = 0; class_index < 3; class_index++) {
unsigned class_bit = 1U << class_index;
if (!(who & class_bit))
continue;
mode_t shift = (mode_t)((2U - class_index) * 3U);
mode_t mask = (mode_t)(7U << shift);
mode_t class_bits = (mode_t)(bits << shift);
if (operation == '+')
*mode |= class_bits;
else if (operation == '-')
*mode &= ~class_bits;
else
*mode = (*mode & ~mask) | class_bits;
}
return true;
}
bool chmod_apply(mode_t mode, const char* spec, mode_t* result) {
if (!spec || !*spec || !result)
return false;
bool numeric = true;
size_t length = strlen(spec);
if (length > 4)
numeric = false;
for (size_t i = 0; i < length && numeric; i++)
numeric = spec[i] >= '0' && spec[i] <= '7';
if (numeric) {
if (length == 0 || length > 4)
return false;
mode_t parsed = 0;
for (size_t i = 0; i < length; i++)
parsed = (mode_t)((parsed << 3) | (spec[i] - '0'));
*result = parsed;
return true;
}
mode_t changed = mode;
const char* begin = spec;
while (*begin) {
const char* end = strchr(begin, ',');
if (!end)
end = begin + strlen(begin);
if (!parse_clause(&changed, begin, end))
return false;
if (*end == '\0')
break;
begin = end + 1;
if (!*begin)
return false;
}
*result = changed;
return true;
}
+10
View File
@@ -0,0 +1,10 @@
#ifndef CHMOD_H
#define CHMOD_H
#include <stdbool.h>
#include <sys/stat.h>
/* Apply the supported rsync --chmod syntax to a permission mode. */
bool chmod_apply(mode_t mode, const char* spec, mode_t* result);
#endif
+124 -23
View File
@@ -1,4 +1,6 @@
#include <stddef.h> #include <stddef.h>
#include <stdint.h>
#include <limits.h>
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
@@ -11,21 +13,33 @@
#include "log.h" #include "log.h"
#include "metadata.h" #include "metadata.h"
#include "protocol.h" #include "protocol.h"
#include "utils.h"
/* Maximum individual file data size within a chunk (64 MB) */ /* Maximum individual file data size within a chunk (64 MB) */
#define MAX_FILE_DATA_SIZE (64ULL * 1024 * 1024) #define MAX_FILE_DATA_SIZE (64ULL * 1024 * 1024)
#define MAX_FILES_PER_CHUNK 65536U
Chunk* chunk_create(File** items, int element_count) { Chunk* chunk_create(File** items, int element_count) {
Chunk* chunk = (Chunk*)malloc(sizeof(Chunk)); if (element_count < 0 || (element_count > 0 && items == NULL))
return NULL;
Chunk* chunk = (Chunk*)protocol_alloc(sizeof(Chunk));
if (chunk == NULL) { if (chunk == NULL) {
perror("ERROR: Could not allocate memory for chunk structure"); log_perror("ERROR: Could not allocate memory for chunk structure");
return NULL; return NULL;
} }
chunk->items = (File**)malloc(element_count * sizeof(File*)); if (element_count == 0) {
if (chunk->items == NULL) { chunk->items = NULL;
free(chunk); } else {
return NULL; if ((size_t)element_count > SIZE_MAX / sizeof(File*)) {
free(chunk);
return NULL;
}
chunk->items = (File**)protocol_alloc((size_t)element_count * sizeof(File*));
if (chunk->items == NULL) {
free(chunk);
return NULL;
}
} }
for (int i = 0; i < element_count; i++) { for (int i = 0; i < element_count; i++) {
@@ -50,15 +64,37 @@ void chunk_destroy(void* item) {
} }
static unsigned long long per_file_serialize_size(File* file, bool use_metadata) { static unsigned long long per_file_serialize_size(File* file, bool use_metadata) {
return sizeof(size_t) + strlen(file->path) + unsigned long long size = sizeof(size_t);
(use_metadata ? sizeof(int) + (file->metadata ? FILE_METADATA_WIRE_SIZE : 0) : 0) + size_t path_len = strlen(file->path);
sizeof(size_t) + file->data->size; unsigned long long metadata_size =
use_metadata ? sizeof(int) + (file->metadata ? FILE_METADATA_WIRE_SIZE : 0) : 0;
if ((unsigned long long)path_len > ULLONG_MAX - size)
return 0;
size += path_len;
if (metadata_size > ULLONG_MAX - size)
return 0;
size += metadata_size;
if (sizeof(size_t) > ULLONG_MAX - size)
return 0;
size += sizeof(size_t);
if ((unsigned long long)file->data->size > ULLONG_MAX - size)
return 0;
return size + file->data->size;
} }
Data* chunk_serialize(Chunk* chunk, bool use_metadata) { Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
if (!chunk || chunk->element_count < 0 || (chunk->element_count > 0 && chunk->items == NULL))
return NULL;
unsigned long long data_size = 0; unsigned long long data_size = 0;
for (int i = 0; i < chunk->element_count; i++) { for (int i = 0; i < chunk->element_count; i++) {
data_size += per_file_serialize_size(chunk->items[i], use_metadata); if (!chunk->items[i] || !chunk->items[i]->path || !chunk->items[i]->data ||
(chunk->items[i]->data->size > 0 && !chunk->items[i]->data->data) ||
chunk->items[i]->path[0] == '\0' || has_path_traversal(chunk->items[i]->path))
return NULL;
unsigned long long file_size = per_file_serialize_size(chunk->items[i], use_metadata);
if (file_size == 0 || file_size > ULLONG_MAX - data_size || data_size + file_size > SIZE_MAX)
return NULL;
data_size += file_size;
} }
Data* data = data_create_empty(data_size); Data* data = data_create_empty(data_size);
if (data == NULL) { if (data == NULL) {
@@ -87,11 +123,20 @@ Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
} }
Chunk* chunk_deserialize(Data* data, bool use_metadata) { Chunk* chunk_deserialize(Data* data, bool use_metadata) {
if (!data || (!data->data && data->size != 0))
return NULL;
ArrayList* files = array_list_create(file_destroy); ArrayList* files = array_list_create(file_destroy);
if (files == NULL)
return NULL;
char* data_pointer = data->data; char* data_pointer = data->data;
size_t remaining_size = data->size; size_t remaining_size = data->size;
while (remaining_size > 0) { while (remaining_size > 0) {
if ((unsigned int)files->size >= MAX_FILES_PER_CHUNK) {
log_message(LOG_LEVEL_ERROR, "Chunk contains too many files");
array_list_delete(files);
return NULL;
}
if (remaining_size < sizeof(size_t)) { if (remaining_size < sizeof(size_t)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for path length"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for path length");
array_list_delete(files); array_list_delete(files);
@@ -103,48 +148,77 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
data_pointer += sizeof(size_t); data_pointer += sizeof(size_t);
remaining_size -= sizeof(size_t); remaining_size -= sizeof(size_t);
if (remaining_size < path_len) { if (path_len > SIZE_MAX - 1 || remaining_size < path_len) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for path"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for path");
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
char* path = malloc(path_len + 1); if (path_len == SIZE_MAX) {
array_list_delete(files);
return NULL;
}
char* path = protocol_alloc(path_len + 1);
if (path == NULL) { if (path == NULL) {
perror("Could not allocate memory for file path"); log_perror("Could not allocate memory for file path");
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
memcpy(path, data_pointer, path_len); memcpy(path, data_pointer, path_len);
path[path_len] = '\0'; path[path_len] = '\0';
if (memchr(path, '\0', path_len) != NULL) {
free(path);
array_list_delete(files);
return NULL;
}
data_pointer += path_len; data_pointer += path_len;
remaining_size -= path_len; remaining_size -= path_len;
if (path_len == 0 || has_path_traversal(path)) {
free(path);
array_list_delete(files);
return NULL;
}
File* file = file_create(path); File* file = file_create(path);
free(path); free(path);
if (file == NULL) {
array_list_delete(files);
return NULL;
}
if (use_metadata) { if (use_metadata) {
if (remaining_size < sizeof(int)) { if (remaining_size < sizeof(int)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
// Peek at present flag to determine total size needed before reading // Peek at present flag to determine total size needed before reading
int present_flag; int present_flag;
memcpy(&present_flag, data_pointer, sizeof(int)); memcpy(&present_flag, data_pointer, sizeof(int));
if (present_flag && remaining_size < sizeof(int) + FILE_METADATA_WIRE_SIZE) { if ((present_flag != 0 && present_flag != 1) ||
(present_flag == 1 && remaining_size < sizeof(int) + FILE_METADATA_WIRE_SIZE)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata body"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata body");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
file->metadata = metadata_from_buf(&data_pointer); file->metadata = metadata_from_buf(&data_pointer);
remaining_size -= sizeof(int); remaining_size -= sizeof(int);
if (file->metadata) if (present_flag == 1) {
if (file->metadata == NULL) {
file_destroy(file);
array_list_delete(files);
return NULL;
}
remaining_size -= FILE_METADATA_WIRE_SIZE; remaining_size -= FILE_METADATA_WIRE_SIZE;
}
} }
if (remaining_size < sizeof(size_t)) { if (remaining_size < sizeof(size_t)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for data size"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for data size");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
@@ -156,6 +230,7 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
if (remaining_size < file_data_size) { if (remaining_size < file_data_size) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for file content"); log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for file content");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
@@ -164,29 +239,50 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
if (file_data_size > MAX_FILE_DATA_SIZE) { if (file_data_size > MAX_FILE_DATA_SIZE) {
log_message(LOG_LEVEL_ERROR, "File data size %zu exceeds maximum %llu", file_data_size, log_message(LOG_LEVEL_ERROR, "File data size %zu exceeds maximum %llu", file_data_size,
(unsigned long long)MAX_FILE_DATA_SIZE); (unsigned long long)MAX_FILE_DATA_SIZE);
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
void* file_data = malloc(file_data_size); size_t allocation_size = file_data_size > 0 ? file_data_size : 1;
void* file_data = protocol_alloc(allocation_size);
if (file_data == NULL) { if (file_data == NULL) {
perror("Could not allocate memory for file data"); log_perror("Could not allocate memory for file data");
file_destroy(file);
array_list_delete(files); array_list_delete(files);
return NULL; return NULL;
} }
memcpy(file_data, data_pointer, file_data_size); memcpy(file_data, data_pointer, file_data_size);
Data* replacement = data_create(file_data, file_data_size);
if (replacement == NULL) {
file_destroy(file);
array_list_delete(files);
return NULL;
}
data_destroy(file->data); data_destroy(file->data);
file->data = data_create(file_data, file_data_size); file->data = replacement;
data_pointer += file_data_size; data_pointer += file_data_size;
remaining_size -= file_data_size; remaining_size -= file_data_size;
array_list_add(files, file); if (!array_list_add(files, file)) {
file_destroy(file);
array_list_delete(files);
return NULL;
}
} }
File** file_array = (File**)array_list_to_array(files); File** file_array = (File**)array_list_to_array(files);
if (files->size > 0 && file_array == NULL) {
array_list_delete(files);
return NULL;
}
Chunk* chunk = chunk_create(file_array, files->size); Chunk* chunk = chunk_create(file_array, files->size);
free(file_array); free(file_array);
if (chunk == NULL) {
array_list_delete(files);
return NULL;
}
files->item_destroyer = NULL; files->item_destroyer = NULL;
array_list_delete(files); array_list_delete(files);
@@ -194,27 +290,32 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
} }
Data* chunk_compress(Chunk* chunk, int compression_level, bool use_metadata) { Data* chunk_compress(Chunk* chunk, int compression_level, bool use_metadata) {
return chunk_compress_with_threads(chunk, compression_level, use_metadata, 0);
}
Data* chunk_compress_with_threads(Chunk* chunk, int compression_level, bool use_metadata,
int compression_threads) {
log_message(LOG_LEVEL_DEBUG, "Starting to compress chunk"); log_message(LOG_LEVEL_DEBUG, "Starting to compress chunk");
Data* serialized = chunk_serialize(chunk, use_metadata); Data* serialized = chunk_serialize(chunk, use_metadata);
if (serialized == NULL) if (serialized == NULL)
return NULL; return NULL;
Data* compressed = data_compress(serialized, compression_level); Data* compressed = data_compress_with_threads(serialized, compression_level, compression_threads);
data_destroy(serialized); data_destroy(serialized);
if (compressed == NULL) if (compressed == NULL)
return NULL; return NULL;
log_message(LOG_LEVEL_DEBUG, "Chunk successfully compressed"); log_debug_message(LOG_DEBUG_PACK, "Chunk successfully compressed");
return compressed; return compressed;
} }
Chunk* receive_chunk_data(int fd, const Config* config) { Chunk* receive_chunk_data(int fd, const Config* config) {
Data* chunk_data = receive_data(fd); Data* chunk_data = receive_data_limited(fd, MAX_CHUNK_SIZE);
if (chunk_data == NULL) { if (chunk_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to receive chunk data"); log_message(LOG_LEVEL_ERROR, "Failed to receive chunk data");
return NULL; return NULL;
} }
Data* data_to_process = chunk_data; Data* data_to_process = chunk_data;
if (config->use_compression) { if (config->use_compression) {
data_to_process = data_decompress(chunk_data); data_to_process = data_decompress_limited(chunk_data, MAX_CHUNK_SIZE);
data_destroy(chunk_data); data_destroy(chunk_data);
if (data_to_process == NULL) { if (data_to_process == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk"); log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk");
+2
View File
@@ -19,6 +19,8 @@ void chunk_destroy(void* chunk);
Data* chunk_serialize(Chunk* chunk, bool use_metadata); Data* chunk_serialize(Chunk* chunk, bool use_metadata);
Chunk* chunk_deserialize(Data* data, bool use_metadata); Chunk* chunk_deserialize(Data* data, bool use_metadata);
Data* chunk_compress(Chunk* chunk, int compression_level, bool use_metadata); Data* chunk_compress(Chunk* chunk, int compression_level, bool use_metadata);
Data* chunk_compress_with_threads(Chunk* chunk, int compression_level, bool use_metadata,
int compression_threads);
Chunk* receive_chunk_data(int fd, const Config* config); Chunk* receive_chunk_data(int fd, const Config* config);
#endif #endif
+74 -18
View File
@@ -1,31 +1,53 @@
#include "compression.h" #include "compression.h"
#include "data.h" #include "data.h"
#include "log.h" #include "log.h"
#include "protocol.h"
#include <stdlib.h> #include <stdlib.h>
#include <limits.h>
#include <stdint.h>
#include <string.h> #include <string.h>
#include <strings.h> #include <strings.h>
#include <unistd.h>
#include <zstd.h> #include <zstd.h>
#define INITIAL_DECOMPRESS_BUF_SIZE (1024 * 1024) #define INITIAL_DECOMPRESS_BUF_SIZE (1024 * 1024)
#define MAX_DECOMPRESSED_SIZE (100ULL * 1024 * 1024) /* 100 MB hard ceiling */ #define MAX_DECOMPRESSED_SIZE (100ULL * 1024 * 1024) /* 100 MB hard ceiling */
static const char* SKIP_COMPRESSION_EXTENSIONS[] = {".jpg", ".jpeg", ".png", ".gif", ".mp4", ".mkv", static char* SKIP_COMPRESSION_EXTENSIONS[] = {".jpg", ".jpeg", ".png", ".gif", ".mp4", ".mkv",
".zip", ".gz", ".xz", ".zst", NULL}; ".zip", ".gz", ".xz", ".zst", NULL};
bool compression_should_skip(const char* path) { bool compression_should_skip(const char* path) {
return compression_should_skip_with_suffixes(path, NULL, -1);
}
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count) {
if (!path) if (!path)
return false; return false;
const char* dot = strrchr(path, '.'); const char* dot = strrchr(path, '.');
if (!dot) if (!dot)
return false; return false;
for (int i = 0; SKIP_COMPRESSION_EXTENSIONS[i]; i++) { if (count < 0) {
if (strcasecmp(dot, SKIP_COMPRESSION_EXTENSIONS[i]) == 0) suffixes = SKIP_COMPRESSION_EXTENSIONS;
count = 0;
while (SKIP_COMPRESSION_EXTENSIONS[count])
count++;
}
for (int i = 0; i < count; i++) {
if (strcasecmp(dot, suffixes[i]) == 0)
return true; return true;
} }
return false; return false;
} }
Data* data_compress(Data* data_to_compress, int compression_level) { Data* data_compress(Data* data_to_compress, int compression_level) {
return data_compress_with_threads(data_to_compress, compression_level, 0);
}
Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
int compression_threads) {
if (!data_to_compress || (!data_to_compress->data && data_to_compress->size != 0) ||
compression_threads < 0 || compression_threads > COMPRESSION_MAX_THREADS)
return NULL;
log_message(LOG_LEVEL_DEBUG, "Starting to compress data"); log_message(LOG_LEVEL_DEBUG, "Starting to compress data");
size_t dst_size = ZSTD_compressBound(data_to_compress->size); size_t dst_size = ZSTD_compressBound(data_to_compress->size);
Data* compressed_data = data_create_empty(dst_size); Data* compressed_data = data_create_empty(dst_size);
@@ -47,6 +69,30 @@ Data* data_compress(Data* data_to_compress, int compression_level) {
return NULL; return NULL;
} }
if (compression_threads > 0) {
long online_cpus = sysconf(_SC_NPROCESSORS_ONLN);
int available_threads = online_cpus > 0 && online_cpus < compression_threads
? (int)online_cpus
: compression_threads;
zret = ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, available_threads);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression threads: %s",
ZSTD_getErrorName(zret));
ZSTD_freeCCtx(cctx);
data_destroy(compressed_data);
return NULL;
}
/* Streaming compression needs the source size before threaded mode can end a frame. */
zret = ZSTD_CCtx_setPledgedSrcSize(cctx, data_to_compress->size);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression source size: %s",
ZSTD_getErrorName(zret));
ZSTD_freeCCtx(cctx);
data_destroy(compressed_data);
return NULL;
}
}
ZSTD_inBuffer input = {data_to_compress->data, data_to_compress->size, 0}; ZSTD_inBuffer input = {data_to_compress->data, data_to_compress->size, 0};
ZSTD_outBuffer output = {compressed_data->data, dst_size, 0}; ZSTD_outBuffer output = {compressed_data->data, dst_size, 0};
@@ -64,13 +110,16 @@ Data* data_compress(Data* data_to_compress, int compression_level) {
compressed_data->size = output.pos; compressed_data->size = output.pos;
ZSTD_freeCCtx(cctx); ZSTD_freeCCtx(cctx);
log_message(LOG_LEVEL_DEBUG, "Data succesfully compressed from %zu to %zu", log_debug_message(LOG_DEBUG_UTIL, "Data succesfully compressed from %zu to %zu",
data_to_compress->size, compressed_data->size); data_to_compress->size, compressed_data->size);
return compressed_data; return compressed_data;
} }
Data* data_decompress(Data* compressed_data) { Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
log_message(LOG_LEVEL_DEBUG, "Start to decompress data"); if (!compressed_data || (!compressed_data->data && compressed_data->size != 0) ||
maximum_size == 0)
return NULL;
log_debug_message(LOG_DEBUG_UTIL, "Start to decompress data");
unsigned long long dst_size = unsigned long long dst_size =
ZSTD_getFrameContentSize(compressed_data->data, compressed_data->size); ZSTD_getFrameContentSize(compressed_data->data, compressed_data->size);
if (ZSTD_isError(dst_size)) { if (ZSTD_isError(dst_size)) {
@@ -82,15 +131,16 @@ Data* data_decompress(Data* compressed_data) {
// ZSTD_CONTENTSIZE_UNKNOWN (~2^64) can cause massive allocation; // ZSTD_CONTENTSIZE_UNKNOWN (~2^64) can cause massive allocation;
// fall back to a conservative estimate (3x compressed size) when unknown. // fall back to a conservative estimate (3x compressed size) when unknown.
if (dst_size == ZSTD_CONTENTSIZE_UNKNOWN) { if (dst_size == ZSTD_CONTENTSIZE_UNKNOWN) {
if (compressed_data->size > ULLONG_MAX / 3)
return NULL;
dst_size = compressed_data->size * 3; dst_size = compressed_data->size * 3;
if (dst_size < INITIAL_DECOMPRESS_BUF_SIZE) if (dst_size < INITIAL_DECOMPRESS_BUF_SIZE)
dst_size = INITIAL_DECOMPRESS_BUF_SIZE; dst_size = INITIAL_DECOMPRESS_BUF_SIZE;
if (dst_size > MAX_DECOMPRESSED_SIZE)
dst_size = MAX_DECOMPRESSED_SIZE;
} }
if (dst_size > MAX_DECOMPRESSED_SIZE) { unsigned long long hard_limit =
log_message(LOG_LEVEL_ERROR, "Declared decompressed size exceeds %llu bytes", maximum_size < MAX_DECOMPRESSED_SIZE ? maximum_size : MAX_DECOMPRESSED_SIZE;
(unsigned long long)MAX_DECOMPRESSED_SIZE); if (dst_size > hard_limit) {
log_message(LOG_LEVEL_ERROR, "Declared decompressed size exceeds %llu bytes", hard_limit);
return NULL; return NULL;
} }
@@ -101,6 +151,8 @@ Data* data_decompress(Data* compressed_data) {
} }
size_t buf_size = (dst_size > 0) ? (size_t)dst_size : INITIAL_DECOMPRESS_BUF_SIZE; size_t buf_size = (dst_size > 0) ? (size_t)dst_size : INITIAL_DECOMPRESS_BUF_SIZE;
if (buf_size > maximum_size)
buf_size = maximum_size;
Data* uncompressed_data = data_create_empty(buf_size); Data* uncompressed_data = data_create_empty(buf_size);
if (!uncompressed_data) { if (!uncompressed_data) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate decompression buffer"); log_message(LOG_LEVEL_ERROR, "Failed to allocate decompression buffer");
@@ -121,7 +173,7 @@ Data* data_decompress(Data* compressed_data) {
return NULL; return NULL;
} }
if (ret > 0 && output.pos == output.size) { if (ret > 0 && output.pos == output.size) {
if (buf_size >= MAX_DECOMPRESSED_SIZE) { if (buf_size >= hard_limit || buf_size > SIZE_MAX / 2) {
log_message(LOG_LEVEL_ERROR, "Decompressed data exceeds %llu bytes", log_message(LOG_LEVEL_ERROR, "Decompressed data exceeds %llu bytes",
(unsigned long long)MAX_DECOMPRESSED_SIZE); (unsigned long long)MAX_DECOMPRESSED_SIZE);
ZSTD_freeDCtx(dctx); ZSTD_freeDCtx(dctx);
@@ -129,9 +181,9 @@ Data* data_decompress(Data* compressed_data) {
return NULL; return NULL;
} }
buf_size *= 2; buf_size *= 2;
if (buf_size > MAX_DECOMPRESSED_SIZE) if (buf_size > hard_limit)
buf_size = MAX_DECOMPRESSED_SIZE; buf_size = (size_t)hard_limit;
void* new_data = realloc(uncompressed_data->data, buf_size); void* new_data = protocol_realloc(uncompressed_data->data, buf_size);
if (!new_data) { if (!new_data) {
log_message(LOG_LEVEL_ERROR, "Failed to grow decompression buffer"); log_message(LOG_LEVEL_ERROR, "Failed to grow decompression buffer");
ZSTD_freeDCtx(dctx); ZSTD_freeDCtx(dctx);
@@ -147,6 +199,10 @@ Data* data_decompress(Data* compressed_data) {
uncompressed_data->size = output.pos; uncompressed_data->size = output.pos;
ZSTD_freeDCtx(dctx); ZSTD_freeDCtx(dctx);
log_message(LOG_LEVEL_DEBUG, "Decompressed data successfully"); log_debug_message(LOG_DEBUG_UTIL, "Decompressed data successfully");
return uncompressed_data; return uncompressed_data;
} }
Data* data_decompress(Data* compressed_data) {
return data_decompress_limited(compressed_data, MAX_DECOMPRESSED_SIZE);
}
+6
View File
@@ -4,8 +4,14 @@
#include "data.h" #include "data.h"
#include <stdbool.h> #include <stdbool.h>
#define COMPRESSION_MAX_THREADS 64
Data* data_compress(Data* data_to_compress, int compression_level); Data* data_compress(Data* data_to_compress, int compression_level);
Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
int compression_threads);
Data* data_decompress(Data* compressed_data); Data* data_decompress(Data* compressed_data);
Data* data_decompress_limited(Data* compressed_data, size_t maximum_size);
bool compression_should_skip(const char* path); bool compression_should_skip(const char* path);
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count);
#endif #endif
+212 -67
View File
@@ -1,4 +1,5 @@
#include "config.h" #include "config.h"
#include "chmod.h"
#include "delta.h" #include "delta.h"
#include "log.h" #include "log.h"
#include "protocol.h" #include "protocol.h"
@@ -17,10 +18,14 @@ static void config_set_defaults(Config* config) {
config->use_chunk_serialization = false; config->use_chunk_serialization = false;
config->use_compression = false; config->use_compression = false;
config->use_metadata = false; config->use_metadata = false;
config->use_executability = false;
config->metadata_explicitly_disabled = false;
config->show_progress = false; config->show_progress = false;
config->dry_run = false; config->dry_run = false;
config->remove_source_files = false;
config->use_delete = false; config->use_delete = false;
config->compression_level = 5; config->compression_level = 5;
config->compression_threads = 0;
config->use_sendfile = false; config->use_sendfile = false;
config->chunk_size = DEFAULT_CHUNK_SIZE; config->chunk_size = DEFAULT_CHUNK_SIZE;
config->ssh_port = 22; config->ssh_port = 22;
@@ -33,8 +38,13 @@ static void config_set_defaults(Config* config) {
config->include_count = 0; config->include_count = 0;
config->max_size = 0; config->max_size = 0;
config->min_size = 0; config->min_size = 0;
config->max_alloc = DEFAULT_MAX_ALLOC;
config->use_incremental = false; config->use_incremental = false;
config->ignore_times = false;
config->size_only = false;
config->use_delta = false; config->use_delta = false;
config->whole_file = false;
config->modify_window = 0;
config->delta_block_size = DELTA_BLOCK_SIZE_DEFAULT; config->delta_block_size = DELTA_BLOCK_SIZE_DEFAULT;
config->delta_max_file_size = DELTA_MAX_FILE_SIZE; config->delta_max_file_size = DELTA_MAX_FILE_SIZE;
config->use_tls = false; config->use_tls = false;
@@ -68,8 +78,12 @@ static void config_set_defaults(Config* config) {
config->debug_level = 0; config->debug_level = 0;
config->list_only = false; config->list_only = false;
config->human_readable = false; config->human_readable = false;
config->eight_bit_output = false;
config->existing = false;
config->ignore_existing = false;
config->update = false; config->update = false;
config->inplace = false; config->inplace = false;
config->use_fsync = false;
config->append = false; config->append = false;
config->append_verify = false; config->append_verify = false;
config->delete_excluded = false; config->delete_excluded = false;
@@ -82,6 +96,7 @@ static void config_set_defaults(Config* config) {
config->relative = false; config->relative = false;
config->rsh_command = NULL; config->rsh_command = NULL;
config->rsync_path = NULL; config->rsync_path = NULL;
config->old_args = false;
config->temp_dir = NULL; config->temp_dir = NULL;
config->compare_dest = NULL; config->compare_dest = NULL;
config->copy_dest = NULL; config->copy_dest = NULL;
@@ -98,6 +113,56 @@ static void config_set_defaults(Config* config) {
config->server_mode = false; config->server_mode = false;
config->checksum = false; config->checksum = false;
config->compress_choice = NULL; config->compress_choice = NULL;
config->chmod_spec = NULL;
config->skip_compress_suffixes = NULL;
config->skip_compress_count = 0;
config->skip_compress_set = false;
}
static bool valid_wire_bool(int value) {
return value == 0 || value == 1;
}
static bool receive_wire_bool(int fd, bool* value) {
int wire_value;
if (!receive_int(fd, &wire_value) || !valid_wire_bool(wire_value))
return false;
*value = wire_value != 0;
return true;
}
static bool validate_received_config(const Config* config) {
return valid_wire_bool(config->save_to_disk) && valid_wire_bool(config->use_multithreading) &&
valid_wire_bool(config->use_chunk_serialization) &&
valid_wire_bool(config->use_compression) && valid_wire_bool(config->use_metadata) &&
valid_wire_bool(config->use_executability) && valid_wire_bool(config->use_sendfile) &&
valid_wire_bool(config->use_delete) && valid_wire_bool(config->use_incremental) &&
valid_wire_bool(config->size_only) && valid_wire_bool(config->ignore_times) &&
valid_wire_bool(config->use_delta) && valid_wire_bool(config->backup) &&
valid_wire_bool(config->remove_source_files) && valid_wire_bool(config->follow_symlinks) &&
valid_wire_bool(config->copy_links) && valid_wire_bool(config->safe_links) &&
valid_wire_bool(config->copy_unsafe_links) &&
valid_wire_bool(config->preserve_hard_links) && valid_wire_bool(config->preserve_acls) &&
valid_wire_bool(config->preserve_xattrs) && valid_wire_bool(config->preserve_devices) &&
valid_wire_bool(config->preserve_sparse) && valid_wire_bool(config->ignore_existing) &&
valid_wire_bool(config->existing) && valid_wire_bool(config->update) &&
valid_wire_bool(config->inplace) && valid_wire_bool(config->append) &&
valid_wire_bool(config->use_fsync) && valid_wire_bool(config->append_verify) &&
valid_wire_bool(config->delete_excluded) && valid_wire_bool(config->delete_after) &&
valid_wire_bool(config->relative) && valid_wire_bool(config->prune_empty_dirs) &&
valid_wire_bool(config->partial) && valid_wire_bool(config->delete_before) &&
valid_wire_bool(config->checksum) && valid_wire_bool(config->eight_bit_output) &&
!(config->skip_compress_set && config->use_chunk_serialization) &&
(!config->use_compression ||
(config->compression_level >= 1 && config->compression_level <= 22)) &&
config->chunk_size > 0 && config->chunk_size <= MAX_CHUNK_SIZE &&
config->delta_block_size >= DELTA_BLOCK_SIZE_MIN &&
config->delta_block_size <= DELTA_BLOCK_SIZE_MAX &&
config->delta_max_file_size <= DELTA_MAX_FILE_SIZE && config->modify_window >= 0 &&
config->max_delete >= 0 && config->skip_compress_count >= 0 &&
config->skip_compress_count <= 10000 && config->max_alloc > 0 &&
(!config->chmod_spec || !*config->chmod_spec ||
chmod_apply(0, config->chmod_spec, &(mode_t){0}));
} }
Config* config_create(void) { Config* config_create(void) {
@@ -108,7 +173,7 @@ Config* config_create(void) {
return config; return config;
} }
bool is_remote_dest(const char* s) { bool config_is_remote_dest(const char* s) {
if (s == NULL) if (s == NULL)
return false; return false;
const char* colon = strchr(s, ':'); const char* colon = strchr(s, ':');
@@ -124,7 +189,7 @@ bool is_remote_dest(const char* s) {
} }
void config_parse_ssh_dest(Config* config) { void config_parse_ssh_dest(Config* config) {
if (!is_remote_dest(config->receive_root_directory)) if (!config_is_remote_dest(config->receive_root_directory))
return; return;
config->transport = TRANSPORT_SSH; config->transport = TRANSPORT_SSH;
config->ssh_destination = str_dup(config->receive_root_directory); config->ssh_destination = str_dup(config->receive_root_directory);
@@ -137,6 +202,10 @@ void config_parse_ssh_dest(Config* config) {
void config_delete(Config* config) { void config_delete(Config* config) {
if (config == NULL) if (config == NULL)
return; return;
if (config->log_file) {
fclose(config->log_file);
config->log_file = NULL;
}
free(config->version); free(config->version);
free(config->send_directory); free(config->send_directory);
free(config->receive_root_directory); free(config->receive_root_directory);
@@ -167,6 +236,12 @@ void config_delete(Config* config) {
free(config->bind_address); free(config->bind_address);
free(config->daemon_config); free(config->daemon_config);
free(config->compress_choice); free(config->compress_choice);
free(config->chmod_spec);
if (config->skip_compress_suffixes) {
for (int i = 0; i < config->skip_compress_count; i++)
free(config->skip_compress_suffixes[i]);
free(config->skip_compress_suffixes);
}
if (config->filters) { if (config->filters) {
array_list_delete(config->filters); array_list_delete(config->filters);
} }
@@ -177,149 +252,209 @@ void config_delete(Config* config) {
* helper call order in config_send and config_receive unchanged when adding * helper call order in config_send and config_receive unchanged when adding
* fields. */ * fields. */
static bool send_core_fields(int fd, const Config* c) { static bool send_core_fields(int fd, const Config* c) {
return send_str(fd, c->version) && send_str(fd, c->send_directory) && if (!send_str(fd, c->version) || !send_int(fd, c->eight_bit_output))
send_str(fd, c->receive_root_directory) && send_int(fd, c->save_to_disk) && return false;
send_int(fd, c->use_multithreading) && send_int(fd, c->use_chunk_serialization) && protocol_set_8_bit_output(c->eight_bit_output);
send_int(fd, c->use_compression) && send_int(fd, c->use_metadata) && if (!send_n_data(fd, &c->max_alloc, sizeof(c->max_alloc)))
return false;
return send_str(fd, c->send_directory) && send_str(fd, c->receive_root_directory) &&
send_int(fd, c->save_to_disk) && send_int(fd, c->use_multithreading) &&
send_int(fd, c->use_chunk_serialization) && send_int(fd, c->use_compression) &&
send_int(fd, c->use_metadata) && send_int(fd, c->use_executability) &&
send_int(fd, c->compression_level) && send_int(fd, c->compression_level) &&
send_n_data(fd, &c->chunk_size, sizeof(c->chunk_size)) && send_int(fd, c->use_sendfile); send_n_data(fd, &c->chunk_size, sizeof(c->chunk_size)) && send_int(fd, c->use_sendfile);
} }
static bool send_delta_fields(int fd, const Config* c) { static bool send_delta_fields(int fd, const Config* c) {
return send_int(fd, c->use_delete) && send_int(fd, c->use_incremental) && return send_int(fd, c->use_delete) && send_int(fd, c->use_incremental) &&
send_int(fd, c->use_delta) && send_int(fd, c->size_only) && send_int(fd, c->ignore_times) &&
send_int(fd, c->use_delta && !c->whole_file) &&
send_n_data(fd, &c->delta_block_size, sizeof(c->delta_block_size)) && send_n_data(fd, &c->delta_block_size, sizeof(c->delta_block_size)) &&
send_n_data(fd, &c->delta_max_file_size, sizeof(unsigned long long)); send_n_data(fd, &c->delta_max_file_size, sizeof(unsigned long long));
} }
static bool send_file_options(int fd, const Config* c) { static bool send_file_options(int fd, const Config* c) {
return send_int(fd, c->backup) && send_str(fd, c->backup_dir ? c->backup_dir : "") && return send_int(fd, c->backup) && send_str(fd, c->backup_dir ? c->backup_dir : "") &&
send_int(fd, c->follow_symlinks) && send_int(fd, c->copy_links) && send_int(fd, c->remove_source_files) && send_int(fd, c->follow_symlinks) &&
send_int(fd, c->safe_links) && send_int(fd, c->copy_unsafe_links) && send_int(fd, c->copy_links) && send_int(fd, c->safe_links) &&
send_int(fd, c->preserve_hard_links) && send_int(fd, c->preserve_acls) && send_int(fd, c->copy_unsafe_links) && send_int(fd, c->preserve_hard_links) &&
send_int(fd, c->preserve_xattrs) && send_int(fd, c->preserve_devices) && send_int(fd, c->preserve_acls) && send_int(fd, c->preserve_xattrs) &&
send_int(fd, c->preserve_sparse); send_int(fd, c->preserve_devices) && send_int(fd, c->preserve_sparse);
} }
static bool send_selection_options(int fd, const Config* c) { static bool send_selection_options(int fd, const Config* c) {
return send_int(fd, c->update) && send_int(fd, c->inplace) && send_int(fd, c->append) && return send_int(fd, c->ignore_existing) && send_int(fd, c->existing) && send_int(fd, c->update) &&
send_int(fd, c->inplace) && send_int(fd, c->append) && send_int(fd, c->use_fsync) &&
send_int(fd, c->append_verify) && send_int(fd, c->delete_excluded) && send_int(fd, c->append_verify) && send_int(fd, c->delete_excluded) &&
send_int(fd, c->delete_after) && send_n_data(fd, &c->max_delete, sizeof(c->max_delete)) && send_int(fd, c->delete_after) && send_n_data(fd, &c->max_delete, sizeof(c->max_delete)) &&
send_int(fd, c->relative) && send_int(fd, c->prune_empty_dirs); send_int(fd, c->relative) && send_int(fd, c->prune_empty_dirs);
} }
static bool send_skip_compress_options(int fd, const Config* c) {
if (!send_int(fd, c->skip_compress_set) || !send_int(fd, c->skip_compress_count))
return false;
for (int i = 0; i < c->skip_compress_count; i++) {
if (!send_str(fd, c->skip_compress_suffixes[i]))
return false;
}
return true;
}
static bool send_resume_options(int fd, const Config* c) { static bool send_resume_options(int fd, const Config* c) {
return send_str(fd, c->temp_dir ? c->temp_dir : "") && send_int(fd, c->partial) && return send_str(fd, c->temp_dir ? c->temp_dir : "") && send_int(fd, c->partial) &&
send_str(fd, c->partial_dir ? c->partial_dir : "") && send_str(fd, c->partial_dir ? c->partial_dir : "") &&
send_str(fd, c->suffix ? c->suffix : "") && send_int(fd, c->delete_before) && send_str(fd, c->suffix ? c->suffix : "") && send_int(fd, c->delete_before) &&
send_int(fd, c->checksum) && send_str(fd, c->compress_choice ? c->compress_choice : ""); send_int(fd, c->checksum) && send_int(fd, c->modify_window) &&
send_str(fd, c->compress_choice ? c->compress_choice : "") &&
send_str(fd, c->chmod_spec ? c->chmod_spec : "") && send_skip_compress_options(fd, c);
} }
static bool receive_core_fields(int fd, Config* c) { static bool receive_core_fields(int fd, Config* c) {
int value; int value;
if (!receive_wire_bool(fd, &c->eight_bit_output))
return false;
protocol_set_8_bit_output(c->eight_bit_output);
if (!receive_n_data(fd, &c->max_alloc, sizeof(c->max_alloc)) || c->max_alloc == 0)
return false;
if (c->max_alloc > MAX_SERVER_ALLOC)
c->max_alloc = MAX_SERVER_ALLOC;
protocol_session_set_max_alloc(NULL, c->max_alloc);
c->send_directory = receive_str(fd); c->send_directory = receive_str(fd);
c->receive_root_directory = receive_str(fd); c->receive_root_directory = receive_str(fd);
if (!c->send_directory || !c->receive_root_directory) if (!c->send_directory || !c->receive_root_directory)
return false; return false;
if (!receive_int(fd, &value)) if (!receive_wire_bool(fd, &c->save_to_disk) || !receive_wire_bool(fd, &c->use_multithreading) ||
!receive_wire_bool(fd, &c->use_chunk_serialization) ||
!receive_wire_bool(fd, &c->use_compression) || !receive_wire_bool(fd, &c->use_metadata) ||
!receive_wire_bool(fd, &c->use_executability))
return false; return false;
c->save_to_disk = value;
if (!receive_int(fd, &value))
return false;
c->use_multithreading = value;
if (!receive_int(fd, &value))
return false;
c->use_chunk_serialization = value;
if (!receive_int(fd, &value))
return false;
c->use_compression = value;
if (!receive_int(fd, &value))
return false;
c->use_metadata = value;
if (!receive_int(fd, &value)) if (!receive_int(fd, &value))
return false; return false;
c->compression_level = value; c->compression_level = value;
if (!receive_n_data(fd, &c->chunk_size, sizeof(c->chunk_size))) if (!receive_n_data(fd, &c->chunk_size, sizeof(c->chunk_size)))
return false; return false;
if (!receive_int(fd, &value)) if (!receive_wire_bool(fd, &c->use_sendfile))
return false; return false;
c->use_sendfile = value;
return true; return true;
} }
static bool receive_delta_fields(int fd, Config* c) { static bool receive_delta_fields(int fd, Config* c) {
int value; if (!receive_wire_bool(fd, &c->use_delete))
if (!receive_int(fd, &value))
return false; return false;
c->use_delete = value; if (!receive_wire_bool(fd, &c->use_incremental))
if (!receive_int(fd, &value))
return false; return false;
c->use_incremental = value; if (!receive_wire_bool(fd, &c->size_only))
if (!receive_int(fd, &value)) return false;
if (!receive_wire_bool(fd, &c->ignore_times))
return false;
if (!receive_wire_bool(fd, &c->use_delta))
return false; return false;
c->use_delta = value;
return receive_n_data(fd, &c->delta_block_size, sizeof(c->delta_block_size)) && return receive_n_data(fd, &c->delta_block_size, sizeof(c->delta_block_size)) &&
receive_n_data(fd, &c->delta_max_file_size, sizeof(unsigned long long)); receive_n_data(fd, &c->delta_max_file_size, sizeof(unsigned long long));
} }
static bool receive_file_options(int fd, Config* c) { static bool receive_file_options(int fd, Config* c) {
int value; if (!receive_wire_bool(fd, &c->backup))
if (!receive_int(fd, &value))
return false; return false;
c->backup = value; char* backup_dir = receive_str(fd);
c->backup_dir = receive_str(fd); if (!backup_dir)
if (!c->backup_dir) return false;
if (*backup_dir != '\0') {
c->backup_dir = backup_dir;
} else {
/* The sender serializes an unset (NULL) string as "", so canonicalize the
empty wire value back to NULL to preserve NULL-vs-empty semantics. */
free(backup_dir);
}
if (!receive_wire_bool(fd, &c->remove_source_files))
return false; return false;
bool* flags[] = {&c->follow_symlinks, &c->copy_links, &c->safe_links, bool* flags[] = {&c->follow_symlinks, &c->copy_links, &c->safe_links,
&c->copy_unsafe_links, &c->preserve_hard_links, &c->preserve_acls, &c->copy_unsafe_links, &c->preserve_hard_links, &c->preserve_acls,
&c->preserve_xattrs, &c->preserve_devices, &c->preserve_sparse}; &c->preserve_xattrs, &c->preserve_devices, &c->preserve_sparse};
for (size_t i = 0; i < sizeof(flags) / sizeof(flags[0]); i++) { for (size_t i = 0; i < sizeof(flags) / sizeof(flags[0]); i++) {
if (!receive_int(fd, &value)) if (!receive_wire_bool(fd, flags[i]))
return false; return false;
*flags[i] = value;
} }
return true; return true;
} }
static bool receive_selection_options(int fd, Config* c) { static bool receive_selection_options(int fd, Config* c) {
int value; bool* flags[] = {&c->ignore_existing, &c->existing, &c->update,
bool* flags[] = {&c->update, &c->inplace, &c->append, &c->inplace, &c->append, &c->use_fsync,
&c->append_verify, &c->delete_excluded, &c->delete_after}; &c->append_verify, &c->delete_excluded, &c->delete_after};
for (size_t i = 0; i < sizeof(flags) / sizeof(flags[0]); i++) { for (size_t i = 0; i < sizeof(flags) / sizeof(flags[0]); i++) {
if (!receive_int(fd, &value)) if (!receive_wire_bool(fd, flags[i]))
return false; return false;
*flags[i] = value;
} }
if (!receive_n_data(fd, &c->max_delete, sizeof(c->max_delete))) if (!receive_n_data(fd, &c->max_delete, sizeof(c->max_delete)))
return false; return false;
if (!receive_int(fd, &value)) if (!receive_wire_bool(fd, &c->relative))
return false; return false;
c->relative = value; if (!receive_wire_bool(fd, &c->prune_empty_dirs))
if (!receive_int(fd, &value))
return false; return false;
c->prune_empty_dirs = value;
return true; return true;
} }
static bool receive_resume_options(int fd, Config* c) { static bool receive_resume_options(int fd, Config* c) {
int value; char* temp_dir = receive_str(fd);
c->temp_dir = receive_str(fd); if (!temp_dir)
if (!c->temp_dir || !receive_int(fd, &value))
return false; return false;
c->partial = value; if (*temp_dir != '\0') {
c->partial_dir = receive_str(fd); c->temp_dir = temp_dir;
c->suffix = c->partial_dir ? receive_str(fd) : NULL; } else {
if (!c->partial_dir || !c->suffix || !receive_int(fd, &value)) free(temp_dir);
}
if (!receive_wire_bool(fd, &c->partial))
return false; return false;
c->delete_before = value; /* These options have NULL client defaults, so the sender transmits an empty
if (!receive_int(fd, &value)) string for "unset". Canonicalize the empty wire value back to NULL so
receivers observe exactly what the client configured (plain --backup, for
example, must not look like --backup-dir ""). */
char* partial_dir = receive_str(fd);
if (!partial_dir)
return false;
if (*partial_dir != '\0') {
c->partial_dir = partial_dir;
} else {
free(partial_dir);
}
char* suffix = receive_str(fd);
if (!suffix)
return false;
if (*suffix != '\0') {
c->suffix = suffix;
} else {
free(suffix);
}
if (!receive_wire_bool(fd, &c->delete_before))
return false;
if (!receive_wire_bool(fd, &c->checksum))
return false;
if (!receive_n_data(fd, &c->modify_window, sizeof(c->modify_window)))
return false; return false;
c->checksum = value;
c->compress_choice = receive_str(fd); c->compress_choice = receive_str(fd);
return c->compress_choice != NULL; if (!c->compress_choice)
return false;
c->chmod_spec = receive_str(fd);
if (!c->chmod_spec || !receive_wire_bool(fd, &c->skip_compress_set) ||
!receive_int(fd, &c->skip_compress_count) || c->skip_compress_count < 0 ||
c->skip_compress_count > 10000)
return false;
if (c->skip_compress_count > 0) {
c->skip_compress_suffixes = calloc((size_t)c->skip_compress_count, sizeof(char*));
if (!c->skip_compress_suffixes)
return false;
for (int i = 0; i < c->skip_compress_count; i++) {
c->skip_compress_suffixes[i] = receive_str(fd);
if (!c->skip_compress_suffixes[i])
return false;
}
}
return true;
} }
bool config_send(int file_descriptor, const Config* config) { bool config_send(int file_descriptor, const Config* config) {
protocol_session_set_max_alloc(NULL, config->max_alloc);
if (!send_core_fields(file_descriptor, config) || !send_delta_fields(file_descriptor, config) || if (!send_core_fields(file_descriptor, config) || !send_delta_fields(file_descriptor, config) ||
!send_file_options(file_descriptor, config) || !send_file_options(file_descriptor, config) ||
!send_selection_options(file_descriptor, config) || !send_selection_options(file_descriptor, config) ||
@@ -344,8 +479,10 @@ Config* config_receive(int file_descriptor) {
if (!config->version) if (!config->version)
goto error; goto error;
if (strcmp(config->version, PROTOCOL_VERSION) != 0) { if (strcmp(config->version, PROTOCOL_VERSION) != 0) {
fprintf(stderr, "Protocol version mismatch: client=%s, server=%s\n", config->version, char* escaped_version = output_escape(config->version, false);
PROTOCOL_VERSION); fprintf(stderr, "Protocol version mismatch: client=%s, server=%s\n",
escaped_version ? escaped_version : "<allocation failed>", PROTOCOL_VERSION);
free(escaped_version);
send_status(file_descriptor, STATUS_ERROR); send_status(file_descriptor, STATUS_ERROR);
goto error; goto error;
} }
@@ -357,7 +494,15 @@ Config* config_receive(int file_descriptor) {
goto error; goto error;
if (config->compress_choice[0] != '\0' && strcmp(config->compress_choice, "zstd") != 0 && if (config->compress_choice[0] != '\0' && strcmp(config->compress_choice, "zstd") != 0 &&
strcmp(config->compress_choice, "none") != 0) { strcmp(config->compress_choice, "none") != 0) {
fprintf(stderr, "Unsupported compression choice: %s\n", config->compress_choice); char* escaped_choice = output_escape(config->compress_choice, config->eight_bit_output);
fprintf(stderr, "Unsupported compression choice: %s\n",
escaped_choice ? escaped_choice : "<allocation failed>");
free(escaped_choice);
send_status(file_descriptor, STATUS_ERROR);
goto error;
}
if (!validate_received_config(config)) {
fprintf(stderr, "Invalid configuration received from client\n");
send_status(file_descriptor, STATUS_ERROR); send_status(file_descriptor, STATUS_ERROR);
goto error; goto error;
} }
+21 -2
View File
@@ -18,10 +18,14 @@ typedef struct Config {
bool use_compression; bool use_compression;
bool use_sendfile; bool use_sendfile;
bool use_metadata; bool use_metadata;
bool use_executability;
bool metadata_explicitly_disabled;
bool show_progress; bool show_progress;
bool dry_run; bool dry_run;
bool remove_source_files;
bool use_delete; bool use_delete;
int compression_level; int compression_level;
int compression_threads;
unsigned long long chunk_size; unsigned long long chunk_size;
int ssh_port; int ssh_port;
TransportType transport; TransportType transport;
@@ -33,8 +37,13 @@ typedef struct Config {
int include_count; int include_count;
unsigned long long max_size; unsigned long long max_size;
unsigned long long min_size; unsigned long long min_size;
unsigned long long max_alloc;
bool use_incremental; bool use_incremental;
bool ignore_times;
bool size_only;
bool use_delta; bool use_delta;
bool whole_file;
int modify_window;
uint32_t delta_block_size; uint32_t delta_block_size;
unsigned long long delta_max_file_size; unsigned long long delta_max_file_size;
bool use_tls; bool use_tls;
@@ -74,10 +83,14 @@ typedef struct Config {
int debug_level; int debug_level;
bool list_only; bool list_only;
bool human_readable; bool human_readable;
bool eight_bit_output;
// Issue #127: Transfer modes // Issue #127: Transfer modes
bool existing;
bool ignore_existing;
bool update; bool update;
bool inplace; bool inplace;
bool use_fsync;
bool append; bool append;
bool append_verify; bool append_verify;
@@ -96,6 +109,7 @@ typedef struct Config {
// Issue #130: Remote shell/connection options // Issue #130: Remote shell/connection options
char* rsh_command; char* rsh_command;
char* rsync_path; char* rsync_path;
bool old_args;
char* temp_dir; char* temp_dir;
char* compare_dest; char* compare_dest;
char* copy_dest; char* copy_dest;
@@ -126,16 +140,21 @@ typedef struct Config {
// PR #184: Compression algorithm negotiation // PR #184: Compression algorithm negotiation
char* compress_choice; char* compress_choice;
char* chmod_spec;
char** skip_compress_suffixes;
int skip_compress_count;
bool skip_compress_set;
} Config; } Config;
#define PROTOCOL_VERSION "2.2.0" #define PROTOCOL_VERSION "2.5.0"
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024) #define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
Config* config_create(void); Config* config_create(void);
void config_delete(Config* config); void config_delete(Config* config);
bool config_send(int file_descriptor, const Config* config); bool config_send(int file_descriptor, const Config* config);
Config* config_receive(int file_descriptor); Config* config_receive(int file_descriptor);
bool is_remote_dest(const char* s); bool config_is_remote_dest(const char* s);
void config_parse_ssh_dest(Config* config); void config_parse_ssh_dest(Config* config);
#endif #endif
+8 -3
View File
@@ -1,11 +1,12 @@
#include "data.h" #include "data.h"
#include "log.h" #include "log.h"
#include "protocol.h"
#include <stdlib.h> #include <stdlib.h>
Data* data_create_empty(size_t data_size) { Data* data_create_empty(size_t data_size) {
/* malloc(0) is UB; allocate at least 1 byte but preserve requested size */ /* malloc(0) is UB; allocate at least 1 byte but preserve requested size */
size_t alloc_size = data_size > 0 ? data_size : 1; size_t alloc_size = data_size > 0 ? data_size : 1;
void* data = malloc(alloc_size); void* data = protocol_alloc(alloc_size);
if (data == NULL) { if (data == NULL) {
log_message(LOG_LEVEL_ERROR, "Could not allocate memory for empty data"); log_message(LOG_LEVEL_ERROR, "Could not allocate memory for empty data");
return NULL; return NULL;
@@ -14,18 +15,19 @@ Data* data_create_empty(size_t data_size) {
} }
Data* data_create_reserve(size_t size) { Data* data_create_reserve(size_t size) {
Data* d = malloc(sizeof(Data)); Data* d = protocol_alloc(sizeof(Data));
if (d == NULL) { if (d == NULL) {
log_message(LOG_LEVEL_ERROR, "Could not allocate memory for data"); log_message(LOG_LEVEL_ERROR, "Could not allocate memory for data");
return NULL; return NULL;
} }
d->data = NULL; d->data = NULL;
d->size = size; d->size = size;
d->protocol_charge = 0;
return d; return d;
} }
Data* data_create(void* data, size_t data_size) { Data* data_create(void* data, size_t data_size) {
Data* new_data = malloc(sizeof(Data)); Data* new_data = protocol_alloc(sizeof(Data));
if (new_data == NULL) { if (new_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Could not allocate memory for data"); log_message(LOG_LEVEL_ERROR, "Could not allocate memory for data");
free(data); free(data);
@@ -33,12 +35,15 @@ Data* data_create(void* data, size_t data_size) {
} }
new_data->data = data; new_data->data = data;
new_data->size = data_size; new_data->size = data_size;
new_data->protocol_charge = 0;
return new_data; return new_data;
} }
void data_destroy(Data* data) { void data_destroy(Data* data) {
if (data == NULL) if (data == NULL)
return; return;
if (data->protocol_charge != 0)
protocol_release_memory(data->protocol_charge);
free(data->data); free(data->data);
free(data); free(data);
} }
+3
View File
@@ -6,11 +6,14 @@
typedef struct { typedef struct {
void* data; void* data;
size_t size; size_t size;
/* Non-zero only for a buffer charged to the protocol connection budget. */
size_t protocol_charge;
} Data; } Data;
Data* data_create_empty(size_t data_size); Data* data_create_empty(size_t data_size);
Data* data_create_reserve(size_t size); Data* data_create_reserve(size_t size);
Data* data_create(void* data, size_t data_size); Data* data_create(void* data, size_t data_size);
void data_destroy(Data* data); void data_destroy(Data* data);
void protocol_release_memory(size_t charge);
#endif #endif
+251 -88
View File
@@ -1,6 +1,8 @@
#include "delta.h" #include "delta.h"
#include "log.h" #include "log.h"
#include "protocol.h"
#include <stdint.h> #include <stdint.h>
#include <limits.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
@@ -36,16 +38,24 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
if (old_file_data == NULL || old_file_size == 0 || block_size == 0) if (old_file_data == NULL || old_file_size == 0 || block_size == 0)
return NULL; return NULL;
if (old_file_size > DELTA_MAX_FILE_SIZE || block_size > DELTA_BLOCK_SIZE_MAX ||
old_file_size > UINT32_MAX * (uint64_t)block_size)
return NULL;
uint32_t block_count = (uint32_t)((old_file_size + block_size - 1) / block_size); uint32_t block_count = (uint32_t)((old_file_size + block_size - 1) / block_size);
DeltaSignature* sig = malloc(sizeof(DeltaSignature)); DeltaSignature* sig = protocol_alloc(sizeof(DeltaSignature));
if (!sig) if (!sig)
return NULL; return NULL;
sig->file_size = old_file_size; sig->file_size = old_file_size;
sig->block_size = block_size; sig->block_size = block_size;
sig->block_count = block_count; sig->block_count = block_count;
sig->blocks = malloc(block_count * sizeof(DeltaBlockSig)); if (block_count == 0) {
free(sig);
return NULL;
}
sig->blocks = protocol_alloc((size_t)block_count * sizeof(DeltaBlockSig));
if (!sig->blocks) { if (!sig->blocks) {
free(sig); free(sig);
return NULL; return NULL;
@@ -67,10 +77,13 @@ Data* delta_signature_serialize(const DeltaSignature* sig) {
if (!sig) if (!sig)
return NULL; return NULL;
uint64_t total = sizeof(uint64_t) + sizeof(uint32_t) + sizeof(uint32_t) + uint64_t block_bytes = (uint64_t)sig->block_count * (sizeof(uint32_t) + sizeof(uint32_t));
(uint64_t)sig->block_count * (sizeof(uint32_t) + sizeof(uint32_t)); uint64_t total = sizeof(uint64_t) + sizeof(uint32_t) + sizeof(uint32_t) + block_bytes;
if (block_bytes > UINT64_MAX - (sizeof(uint64_t) + sizeof(uint32_t) + sizeof(uint32_t)) ||
total > SIZE_MAX)
return NULL;
uint8_t* buf = malloc((size_t)total); uint8_t* buf = protocol_alloc((size_t)total);
if (!buf) if (!buf)
return NULL; return NULL;
@@ -99,7 +112,7 @@ DeltaSignature* delta_signature_deserialize(const Data* data) {
const uint8_t* buf = (const uint8_t*)data->data; const uint8_t* buf = (const uint8_t*)data->data;
size_t pos = 0; size_t pos = 0;
DeltaSignature* sig = malloc(sizeof(DeltaSignature)); DeltaSignature* sig = protocol_alloc(sizeof(DeltaSignature));
if (!sig) if (!sig)
return NULL; return NULL;
@@ -118,6 +131,13 @@ DeltaSignature* delta_signature_deserialize(const Data* data) {
return NULL; return NULL;
} }
if (sig->block_size == 0 || sig->block_size > DELTA_BLOCK_SIZE_MAX ||
sig->file_size > DELTA_MAX_FILE_SIZE || sig->file_size == 0 ||
(sig->file_size + sig->block_size - 1) / sig->block_size != sig->block_count) {
free(sig);
return NULL;
}
uint64_t expected = sizeof(uint64_t) + sizeof(uint32_t) + sizeof(uint32_t) + uint64_t expected = sizeof(uint64_t) + sizeof(uint32_t) + sizeof(uint32_t) +
(uint64_t)sig->block_count * (sizeof(uint32_t) + sizeof(uint32_t)); (uint64_t)sig->block_count * (sizeof(uint32_t) + sizeof(uint32_t));
if (data->size < expected) { if (data->size < expected) {
@@ -130,7 +150,7 @@ DeltaSignature* delta_signature_deserialize(const Data* data) {
free(sig); free(sig);
return NULL; return NULL;
} }
sig->blocks = malloc((size_t)blocks_size); sig->blocks = protocol_alloc((size_t)blocks_size);
if (!sig->blocks) { if (!sig->blocks) {
free(sig); free(sig);
return NULL; return NULL;
@@ -156,8 +176,10 @@ void delta_signature_destroy(DeltaSignature* sig) {
static bool ensure_capacity(DeltaInstruction** instrs, uint32_t* capacity, uint32_t count) { static bool ensure_capacity(DeltaInstruction** instrs, uint32_t* capacity, uint32_t count) {
if (count < *capacity) if (count < *capacity)
return true; return true;
if (*capacity > MAX_DELTA_INSTRUCTIONS / 2)
return false;
uint32_t new_cap = *capacity * 2; uint32_t new_cap = *capacity * 2;
DeltaInstruction* tmp = realloc(*instrs, new_cap * sizeof(DeltaInstruction)); DeltaInstruction* tmp = protocol_realloc(*instrs, (size_t)new_cap * sizeof(DeltaInstruction));
if (!tmp) if (!tmp)
return false; return false;
*instrs = tmp; *instrs = tmp;
@@ -169,10 +191,12 @@ static bool flush_literal(DeltaInstruction** instrs, uint32_t* capacity, uint32_
const uint8_t* data, uint64_t start, uint64_t end) { const uint8_t* data, uint64_t start, uint64_t end) {
if (start >= end) if (start >= end)
return true; return true;
if (end - start > UINT32_MAX || *count >= MAX_DELTA_INSTRUCTIONS)
return false;
uint32_t lit_len = (uint32_t)(end - start); uint32_t lit_len = (uint32_t)(end - start);
if (!ensure_capacity(instrs, capacity, *count)) if (!ensure_capacity(instrs, capacity, *count))
return false; return false;
uint8_t* lit_data = malloc(lit_len); uint8_t* lit_data = protocol_alloc(lit_len);
if (!lit_data) if (!lit_data)
return false; return false;
memcpy(lit_data, data + start, lit_len); memcpy(lit_data, data + start, lit_len);
@@ -183,19 +207,161 @@ static bool flush_literal(DeltaInstruction** instrs, uint32_t* capacity, uint32_
return true; return true;
} }
static void free_instructions(DeltaInstruction* instrs, uint32_t count) {
if (!instrs)
return;
for (uint32_t i = 0; i < count; i++)
if (instrs[i].type == DELTA_INSTR_LITERAL)
free(instrs[i].literal.data);
free(instrs);
}
/* Sentinel meaning "no signature block" in the lookup index chains. Block
* counts are bounded well below UINT32_MAX, so it doubles as a null link. */
#define DELTA_NO_BLOCK UINT32_MAX
/* Avalanche mix for the rolling checksum so blocks do not cluster in the
* bucket table when the weak checksum has little entropy (e.g. all-zero or
* patterned files). */
static uint32_t delta_adler_mix(uint32_t h) {
h ^= h >> 16;
h *= 0x7feb352dU;
h ^= h >> 15;
h *= 0x846ca68bU;
h ^= h >> 16;
return h;
}
/* Smallest power of two >= v. v must be non-zero. */
static uint32_t delta_next_pow2(uint32_t v) {
v--;
v |= v >> 1;
v |= v >> 2;
v |= v >> 4;
v |= v >> 8;
v |= v >> 16;
return v + 1;
}
/* Build a hash index over sig->blocks keyed by the (mixed) rolling checksum.
* All blocks sharing an Adler-32 value land in the same bucket; collisions
* are chained through a single contiguous allocation:
*
* [0, bucket_count) heads (first block per bucket)
* [bucket_count, 2*bucket_count) tails (last block per bucket)
* [2*bucket_count, ...) per-block chain links
*
* Blocks are inserted in ascending index order so every bucket chain is
* ordered exactly like the historical linear scan. Returns the base pointer
* (also the heads array) or NULL when no index could be allocated; callers
* then fall back to the linear scan. */
static uint32_t* delta_build_index(const DeltaSignature* sig, uint32_t bucket_count) {
if (sig->block_count == 0 || bucket_count == 0)
return NULL;
size_t entries = (size_t)2 * bucket_count + sig->block_count;
if (entries > SIZE_MAX / sizeof(uint32_t))
return NULL;
uint32_t* index = protocol_alloc(entries * sizeof(uint32_t));
if (!index)
return NULL;
uint32_t* heads = index;
uint32_t* tails = index + bucket_count;
uint32_t* next = index + 2 * bucket_count;
uint32_t mask = bucket_count - 1;
memset(heads, 0xFF, (size_t)bucket_count * sizeof(uint32_t));
memset(tails, 0xFF, (size_t)bucket_count * sizeof(uint32_t));
for (uint32_t j = 0; j < sig->block_count; j++) {
uint32_t b = delta_adler_mix(sig->blocks[j].adler32) & mask;
if (heads[b] == DELTA_NO_BLOCK)
heads[b] = j;
else
next[tails[b]] = j;
tails[b] = j;
next[j] = DELTA_NO_BLOCK;
}
return index;
}
/* Locate the signature block matching the byte window at new_data[i].
*
* Mirrors the original per-window behaviour exactly: only a full block_size
* window can match, candidates are accepted only when the weak (Adler-32) and
* strong (xxHash32) checksums both agree, and the lowest block index wins so
* the emitted op stream is byte-identical to the linear scan. When heads is
* non-NULL the candidate set is reached through the bucket index (expected
* O(1) per window); otherwise an exact linear scan is used. */
static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uint32_t adler,
bool full_window, const DeltaSignature* sig, const uint32_t* heads,
const uint32_t* next, uint32_t mask) {
if (!full_window || sig->block_count == 0)
return DELTA_NO_BLOCK;
if (heads) {
uint32_t b = delta_adler_mix(adler) & mask;
uint32_t window_xxh = 0;
bool have_xxh = false;
for (uint32_t j = heads[b]; j != DELTA_NO_BLOCK; j = next[j]) {
if (sig->blocks[j].adler32 != adler)
continue;
if (!have_xxh) {
window_xxh = delta_xxhash32(window, window_len);
have_xxh = true;
}
if (window_xxh == sig->blocks[j].xxhash)
return j;
}
return DELTA_NO_BLOCK;
}
/* Fallback used when the index could not be allocated. */
for (uint32_t j = 0; j < sig->block_count; j++) {
if (sig->blocks[j].adler32 == adler) {
uint32_t window_xxh = delta_xxhash32(window, window_len);
if (window_xxh == sig->blocks[j].xxhash)
return j;
}
}
return DELTA_NO_BLOCK;
}
Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const DeltaSignature* sig, Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const DeltaSignature* sig,
uint32_t block_size) { uint32_t block_size) {
if (!new_file_data || !sig || new_file_size == 0 || block_size == 0) if (!new_file_data || !sig || !sig->blocks || new_file_size == 0 || block_size == 0 ||
block_size > DELTA_BLOCK_SIZE_MAX || sig->block_size != block_size)
return NULL; return NULL;
const uint8_t* new_data = (const uint8_t*)new_file_data; const uint8_t* new_data = (const uint8_t*)new_file_data;
uint32_t capacity = 64; uint32_t capacity = 64;
uint32_t count = 0; uint32_t count = 0;
DeltaInstruction* instrs = malloc(capacity * sizeof(DeltaInstruction)); DeltaInstruction* instrs = protocol_alloc((size_t)capacity * sizeof(DeltaInstruction));
if (!instrs) if (!instrs)
return NULL; return NULL;
/* Build a one-time bucket index over the signature blocks keyed by the weak
* checksum. This turns the per-byte-window candidate lookup from an
* O(block_count) linear scan into an expected O(1) probe, which dominates
* the cost for large mostly-matching files (the diff steps one byte at a
* time through changed regions). On allocation failure the probe falls back
* to the original linear scan, so behaviour is unchanged under memory
* pressure. */
uint32_t* index = NULL;
const uint32_t* chain_next = NULL;
uint32_t mask = 0;
if (sig->block_count > 0) {
uint32_t bucket_count = delta_next_pow2(sig->block_count);
index = delta_build_index(sig, bucket_count);
if (index) {
chain_next = index + 2 * bucket_count;
mask = bucket_count - 1;
}
}
uint64_t literal_start = 0; uint64_t literal_start = 0;
bool has_literal = false; bool has_literal = false;
@@ -230,34 +396,32 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
} }
bool matched = false; bool matched = false;
for (uint32_t j = 0; j < sig->block_count; j++) { uint32_t match_block = delta_find_match(new_data + i, window_len, adler, full_window, sig,
if (adler == sig->blocks[j].adler32 && full_window) { index, chain_next, mask);
uint32_t xxh = delta_xxhash32(new_data + i, window_len); if (match_block != DELTA_NO_BLOCK) {
if (xxh == sig->blocks[j].xxhash) { if (has_literal) {
if (has_literal) { if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, i)) {
if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, i)) { free_instructions(instrs, count);
free(instrs); free(index);
return NULL; return NULL;
}
has_literal = false;
}
if (!ensure_capacity(&instrs, &capacity, count)) {
free(instrs);
return NULL;
}
instrs[count].type = DELTA_INSTR_BLOCK_MATCH;
instrs[count].match.block_index = j;
instrs[count].match.block_offset = 0;
instrs[count].match.length = window_len;
count++;
i += window_len;
rolling_valid = false;
matched = true;
break;
} }
has_literal = false;
} }
if (!ensure_capacity(&instrs, &capacity, count)) {
free_instructions(instrs, count);
free(index);
return NULL;
}
instrs[count].type = DELTA_INSTR_BLOCK_MATCH;
instrs[count].match.block_index = match_block;
instrs[count].match.block_offset = 0;
instrs[count].match.length = window_len;
count++;
i += window_len;
rolling_valid = false;
matched = true;
} }
if (!matched) { if (!matched) {
@@ -269,20 +433,18 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
} }
} }
free(index);
if (has_literal) { if (has_literal) {
if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, new_file_size)) { if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, new_file_size)) {
free(instrs); free_instructions(instrs, count);
return NULL; return NULL;
} }
} }
Delta* delta = malloc(sizeof(Delta)); Delta* delta = protocol_alloc(sizeof(Delta));
if (!delta) { if (!delta) {
for (uint32_t k = 0; k < count; k++) { free_instructions(instrs, count);
if (instrs[k].type == DELTA_INSTR_LITERAL)
free(instrs[k].literal.data);
}
free(instrs);
return NULL; return NULL;
} }
@@ -292,11 +454,24 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
delta->delta_size = 0; delta->delta_size = 0;
for (uint32_t k = 0; k < count; k++) { for (uint32_t k = 0; k < count; k++) {
if (delta->delta_size == UINT64_MAX) {
delta_destroy(delta);
return NULL;
}
delta->delta_size += 1; delta->delta_size += 1;
if (instrs[k].type == DELTA_INSTR_BLOCK_MATCH) { if (instrs[k].type == DELTA_INSTR_BLOCK_MATCH) {
if (delta->delta_size > UINT64_MAX - sizeof(uint32_t) * 3) {
delta_destroy(delta);
return NULL;
}
delta->delta_size += sizeof(uint32_t) * 3; delta->delta_size += sizeof(uint32_t) * 3;
} else { } else {
delta->delta_size += sizeof(uint32_t) + instrs[k].literal.length; uint64_t extra = sizeof(uint32_t) + instrs[k].literal.length;
if (delta->delta_size > UINT64_MAX - extra) {
delta_destroy(delta);
return NULL;
}
delta->delta_size += extra;
} }
} }
@@ -307,8 +482,13 @@ Data* delta_serialize(const Delta* delta) {
if (!delta) if (!delta)
return NULL; return NULL;
uint64_t total = sizeof(uint64_t) + sizeof(uint32_t) + delta->delta_size; if (delta->instruction_count > 0 && !delta->instructions)
uint8_t* buf = malloc((size_t)total); return NULL;
uint64_t header_size = sizeof(uint64_t) + sizeof(uint32_t);
if (delta->delta_size > UINT64_MAX - header_size || header_size + delta->delta_size > SIZE_MAX)
return NULL;
uint64_t total = header_size + delta->delta_size;
uint8_t* buf = protocol_alloc((size_t)total);
if (!buf) if (!buf)
return NULL; return NULL;
@@ -348,7 +528,7 @@ Delta* delta_deserialize(const Data* data) {
const uint8_t* buf = (const uint8_t*)data->data; const uint8_t* buf = (const uint8_t*)data->data;
size_t pos = 0; size_t pos = 0;
Delta* delta = malloc(sizeof(Delta)); Delta* delta = protocol_alloc(sizeof(Delta));
if (!delta) if (!delta)
return NULL; return NULL;
@@ -365,8 +545,11 @@ Delta* delta_deserialize(const Data* data) {
return NULL; return NULL;
} }
delta->instructions = malloc(delta->instruction_count * sizeof(DeltaInstruction)); delta->instructions =
if (!delta->instructions) { delta->instruction_count == 0
? NULL
: protocol_alloc((size_t)delta->instruction_count * sizeof(DeltaInstruction));
if (delta->instruction_count > 0 && !delta->instructions) {
free(delta); free(delta);
return NULL; return NULL;
} }
@@ -375,11 +558,7 @@ Delta* delta_deserialize(const Data* data) {
for (uint32_t i = 0; i < delta->instruction_count; i++) { for (uint32_t i = 0; i < delta->instruction_count; i++) {
if (pos >= data->size) { if (pos >= data->size) {
for (uint32_t k = 0; k < i; k++) { free_instructions(delta->instructions, i);
if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
free(delta->instructions[k].literal.data);
}
free(delta->instructions);
free(delta); free(delta);
return NULL; return NULL;
} }
@@ -391,12 +570,8 @@ Delta* delta_deserialize(const Data* data) {
delta->delta_size += 1; delta->delta_size += 1;
if (type == DELTA_OP_BLOCK_MATCH) { if (type == DELTA_OP_BLOCK_MATCH) {
if (pos + sizeof(uint32_t) * 3 > data->size) { if (data->size - pos < sizeof(uint32_t) * 3) {
for (uint32_t k = 0; k < i; k++) { free_instructions(delta->instructions, i);
if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
free(delta->instructions[k].literal.data);
}
free(delta->instructions);
free(delta); free(delta);
return NULL; return NULL;
} }
@@ -409,12 +584,8 @@ Delta* delta_deserialize(const Data* data) {
pos += sizeof(uint32_t); pos += sizeof(uint32_t);
delta->delta_size += sizeof(uint32_t) * 3; delta->delta_size += sizeof(uint32_t) * 3;
} else if (type == DELTA_OP_LITERAL) { } else if (type == DELTA_OP_LITERAL) {
if (pos + sizeof(uint32_t) > data->size) { if (data->size - pos < sizeof(uint32_t)) {
for (uint32_t k = 0; k < i; k++) { free_instructions(delta->instructions, i);
if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
free(delta->instructions[k].literal.data);
}
free(delta->instructions);
free(delta); free(delta);
return NULL; return NULL;
} }
@@ -423,23 +594,15 @@ Delta* delta_deserialize(const Data* data) {
pos += sizeof(uint32_t); pos += sizeof(uint32_t);
uint32_t lit_len = delta->instructions[i].literal.length; uint32_t lit_len = delta->instructions[i].literal.length;
if (pos + lit_len > data->size) { if (lit_len > data->size - pos) {
for (uint32_t k = 0; k < i; k++) { free_instructions(delta->instructions, i);
if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
free(delta->instructions[k].literal.data);
}
free(delta->instructions);
free(delta); free(delta);
return NULL; return NULL;
} }
delta->instructions[i].literal.data = malloc(lit_len); delta->instructions[i].literal.data = protocol_alloc(lit_len ? lit_len : 1);
if (!delta->instructions[i].literal.data) { if (!delta->instructions[i].literal.data) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate %u bytes for literal data", lit_len); log_message(LOG_LEVEL_ERROR, "Failed to allocate %u bytes for literal data", lit_len);
for (uint32_t k = 0; k < i; k++) { free_instructions(delta->instructions, i);
if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
free(delta->instructions[k].literal.data);
}
free(delta->instructions);
free(delta); free(delta);
return NULL; return NULL;
} }
@@ -447,11 +610,7 @@ Delta* delta_deserialize(const Data* data) {
pos += lit_len; pos += lit_len;
delta->delta_size += sizeof(uint32_t) + lit_len; delta->delta_size += sizeof(uint32_t) + lit_len;
} else { } else {
for (uint32_t k = 0; k < i; k++) { free_instructions(delta->instructions, i);
if (delta->instructions[k].type == DELTA_INSTR_LITERAL)
free(delta->instructions[k].literal.data);
}
free(delta->instructions);
free(delta); free(delta);
return NULL; return NULL;
} }
@@ -463,10 +622,11 @@ Delta* delta_deserialize(const Data* data) {
void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta, void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta,
uint32_t block_size) { uint32_t block_size) {
if (!old_data || !delta || (delta->new_file_size > 0 && delta->instructions == NULL) || if (!old_data || !delta || (delta->new_file_size > 0 && delta->instructions == NULL) ||
(delta->instruction_count > 0 && block_size == 0)) (delta->instruction_count > 0 && block_size == 0) ||
delta->new_file_size > DELTA_MAX_FILE_SIZE || delta->new_file_size > SIZE_MAX)
return NULL; return NULL;
void* output = malloc((size_t)delta->new_file_size); void* output = protocol_alloc(delta->new_file_size ? (size_t)delta->new_file_size : 1);
if (!output) if (!output)
return NULL; return NULL;
@@ -491,7 +651,7 @@ void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta,
} }
memcpy(out + out_pos, old + src_offset, len); memcpy(out + out_pos, old + src_offset, len);
out_pos += len; out_pos += len;
} else { } else if (delta->instructions[i].type == DELTA_INSTR_LITERAL) {
uint32_t len = delta->instructions[i].literal.length; uint32_t len = delta->instructions[i].literal.length;
if (out_pos > delta->new_file_size || (uint64_t)len > delta->new_file_size - out_pos) { if (out_pos > delta->new_file_size || (uint64_t)len > delta->new_file_size - out_pos) {
free(output); free(output);
@@ -499,6 +659,9 @@ void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta,
} }
memcpy(out + out_pos, delta->instructions[i].literal.data, len); memcpy(out + out_pos, delta->instructions[i].literal.data, len);
out_pos += len; out_pos += len;
} else {
free(output);
return NULL;
} }
} }
@@ -534,7 +697,7 @@ bool delta_should_attempt(uint64_t old_size, uint64_t new_size, uint64_t max_fil
} }
bool delta_is_worthwhile(const Delta* delta, uint64_t new_file_size) { bool delta_is_worthwhile(const Delta* delta, uint64_t new_file_size) {
if (!delta || delta->instruction_count == 0) if (!delta || delta->instruction_count == 0 || new_file_size == 0)
return false; return false;
bool has_match = false; bool has_match = false;
+333 -705
View File
File diff suppressed because it is too large. Load diff
+35 -29
View File
@@ -1,45 +1,51 @@
#ifndef FILE_H #ifndef FILE_H
#define FILE_H #define FILE_H
#include "config.h" #include "file_send.h"
#include "data.h" #include "file_receive.h"
#include "file_types.h"
#include <stdbool.h> #include <stdbool.h>
#include <stdint.h>
#include <sys/stat.h> #include <sys/stat.h>
typedef enum { FILE_TYPE_REGULAR, FILE_TYPE_SYMLINK, FILE_TYPE_DIR } FileType; /* File/FileMetadata lifecycle, local disk helpers, and secure filesystem
primitives shared by the send/receive pipelines. */
typedef struct {
mode_t mode;
uid_t uid;
gid_t gid;
time_t mtime_sec;
long mtime_nsec;
} FileMetadata;
typedef struct {
char* path;
Data* data;
FileMetadata* metadata;
bool skip;
} File;
File* file_create(const char* path); File* file_create(const char* path);
void file_destroy(void* item); void file_destroy(void* item);
bool file_load_data(File* file); bool file_load_data(File* file);
bool file_checksum(File* file, uint64_t* checksum); bool file_checksum(File* file, uint64_t* checksum);
File* file_receive(const Config* config, int file_descriptor);
bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path);
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
bool send_path);
size_t file_content_to_buffer(File* file); size_t file_content_to_buffer(File* file);
FileMetadata* file_metadata_create(const struct stat* stats); FileMetadata* file_metadata_create(const struct stat* stats);
void file_metadata_destroy(void* metadata); void file_metadata_destroy(void* metadata);
bool to_disk(const char* path, const void* data, unsigned long long data_size, bool inplace, bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
bool sparse); bool inplace, bool sparse);
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
void file_set_authorized_root(int fd, const char* canonical_path); /* A configured fd without a canonical identity deliberately rejects paths. */
File* receive_incremental_check(int fd, const Config* config, bool* skipped); bool file_set_authorized_root(int fd, const char* canonical_path);
int receive_manifest(int fd, const Config* config, int* next_status);
/* Secure path/filesystem primitives (symlink-safe, O_NOFOLLOW, root-confined). */
bool file_path_exists_secure(const char* path);
bool file_stat_secure(const char* path, struct stat* st);
bool file_destination_is_newer_secure(const char* path, const FileMetadata* metadata);
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
bool file_ensure_directory_secure(const char* path);
bool file_rename_secure(const char* old_path, const char* new_path);
bool file_to_disk_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability);
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync);
/* With update enabled, an existing newer destination is left untouched. The
check is descriptor-based for inplace writes; atomic replacement still has
an unavoidable final rename race without filesystem locking. */
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability);
bool file_to_disk_secure_no_replace(const char* path, const void* data,
unsigned long long data_size, bool sparse,
const FileMetadata* metadata, bool preserve_executability);
#endif #endif
+700
View File
@@ -0,0 +1,700 @@
#include <errno.h>
#include <fcntl.h>
#include <libgen.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
#include "array_list.h"
#include "chmod.h"
#include "compression.h"
#include "config.h"
#include "data.h"
#include "delta.h"
#include "file.h"
#include "log.h"
#include "metadata.h"
#include "protocol.h"
#include "utils.h"
#define MAX_SERVER_DELETE_COUNT 100000U
#define MAX_FILE_DATA_SIZE MAX_RECEIVE_WHOLE_FILE_SIZE
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config) {
return file_save_to_disk_full(root_directory, file, config) != FILE_SAVE_ERROR;
}
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
const Config* config) {
/* Backups are incompatible with ignore-existing: moving the entry first
would make a concurrent no-replace commit overwrite its old name. */
bool backup_enabled = config && config->backup && !config->ignore_existing;
bool inplace = config && config->inplace;
bool sparse = config && config->preserve_sparse;
bool preserve_executability = config && config->use_executability;
const char* backup_suffix = (config && config->suffix) ? config->suffix : "~";
const char* backup_dir = (config && config->backup_dir) ? config->backup_dir : NULL;
const char* partial_dir = (config && config->partial_dir) ? config->partial_dir : NULL;
bool use_partial_root = partial_dir && config && config->partial;
char *confined_backup = NULL, *confined_partial = NULL, *disk_path = NULL;
char* destination_path = NULL;
char *backup_path = NULL, *parent_copy = NULL;
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data) ||
has_path_traversal(file->path) ||
(backup_enabled &&
(!backup_suffix || backup_suffix[0] == '\0' || strchr(backup_suffix, '/') != NULL ||
strcmp(backup_suffix, ".") == 0 || strcmp(backup_suffix, "..") == 0))) {
log_message(LOG_LEVEL_ERROR, "Invalid file or path received");
return FILE_SAVE_ERROR;
}
/* These options arrive from the client. They are names below the server
root, never independent filesystem roots. */
if ((backup_dir && (backup_dir[0] == '/' || has_path_traversal(backup_dir))) ||
(partial_dir && (partial_dir[0] == '/' || has_path_traversal(partial_dir))))
return FILE_SAVE_ERROR;
if (backup_dir && !(confined_backup = path_cat(root_directory, backup_dir)))
return FILE_SAVE_ERROR;
if (partial_dir && !(confined_partial = path_cat(root_directory, partial_dir))) {
free(confined_backup);
return FILE_SAVE_ERROR;
}
const char* actual_root = use_partial_root ? confined_partial : root_directory;
destination_path = path_cat(root_directory, file->path);
disk_path = path_cat(actual_root, file->path);
if (destination_path == NULL || disk_path == NULL) {
free(confined_backup);
free(confined_partial);
free(destination_path);
free(disk_path);
return FILE_SAVE_ERROR;
}
/* --existing checks the final destination, not a temporary partial path. */
if (config && config->existing && !file_path_exists_secure(destination_path)) {
free(confined_backup);
free(confined_partial);
free(destination_path);
free(disk_path);
return FILE_SAVE_SKIPPED;
}
/* --ignore-existing checks the final destination before partial files or
overwrite policies can modify it. */
if (config && config->ignore_existing) {
bool exists = file_path_exists_secure(destination_path);
if (exists) {
free(confined_backup);
free(confined_partial);
free(destination_path);
free(disk_path);
return FILE_SAVE_SKIPPED;
}
}
/* --update is receiver-side policy: never replace a newer destination.
In partial-dir mode the entry that would be replaced is the real
destination, not the temporary partial file. The secure stat does not
require read permission on the destination. */
const char* update_target = use_partial_root ? destination_path : disk_path;
if (config && config->update && file_destination_is_newer_secure(update_target, file->metadata)) {
free(confined_backup);
free(confined_partial);
free(destination_path);
free(disk_path);
return FILE_SAVE_SKIPPED;
}
if (backup_enabled) {
/* Back up the entry that the incoming write will replace. When writing
through a partial dir the pre-existing destination file is the one to
preserve; any stale partial file is overwritten without a backup. */
const char* replace_target = use_partial_root ? destination_path : disk_path;
struct stat backup_stat;
if (file_stat_secure(replace_target, &backup_stat)) {
if (backup_dir) {
backup_path = path_cat(confined_backup, file->path);
} else {
size_t path_len = strlen(replace_target);
size_t suffix_len = strlen(backup_suffix);
if (path_len > SIZE_MAX - suffix_len - 1)
goto fail;
backup_path = malloc(path_len + suffix_len + 1);
if (backup_path) {
memcpy(backup_path, replace_target, path_len);
memcpy(backup_path + path_len, backup_suffix, suffix_len + 1);
}
}
if (!backup_path)
goto fail;
parent_copy = str_dup(backup_path);
if (!parent_copy || !file_ensure_directory_secure(dirname(parent_copy)))
goto fail;
free(parent_copy);
parent_copy = NULL;
if (!file_rename_secure(replace_target, backup_path))
goto fail;
free(backup_path);
backup_path = NULL;
}
}
FileMetadata adjusted_metadata;
const FileMetadata* metadata = file->metadata;
if (metadata && config && config->chmod_spec && *config->chmod_spec) {
adjusted_metadata = *metadata;
if (!chmod_apply(adjusted_metadata.mode, config->chmod_spec, &adjusted_metadata.mode))
goto fail;
metadata = &adjusted_metadata;
}
bool ok = config && config->ignore_existing
? file_to_disk_secure_no_replace(disk_path, file->data->data, file->data->size,
sparse, metadata, preserve_executability)
: config && config->update
? file_to_disk_secure_update(disk_path, file->data->data, file->data->size, inplace,
sparse, metadata, preserve_executability)
: file_to_disk_secure_with_fsync(disk_path, file->data->data, file->data->size,
inplace, sparse, metadata, preserve_executability,
config && config->use_fsync);
if (!ok)
goto fail;
/* --partial --partial-dir writes the complete file under the partial dir so
interrupted transfers leave a resumable copy there. Once the file is
fully written it must be atomically installed at the real destination;
otherwise completed transfers would linger under the partial dir. */
if (use_partial_root) {
if (!file_rename_secure(disk_path, destination_path))
goto fail;
}
free(parent_copy);
free(backup_path);
free(confined_backup);
free(confined_partial);
free(destination_path);
free(disk_path);
return FILE_SAVE_WRITTEN;
fail:
free(parent_copy);
free(backup_path);
free(confined_backup);
free(confined_partial);
free(destination_path);
free(disk_path);
return FILE_SAVE_ERROR;
}
static File* receive_delta_file(int fd, const Config* config, const char* check_path,
void* old_data, unsigned long long old_size, bool* failed) {
if (!old_data) {
*failed = true;
return NULL;
}
DeltaSignature* sig = delta_signature_create(old_data, old_size, config->delta_block_size);
if (!sig) {
free(old_data);
*failed = true;
return NULL;
}
Data* sig_data = delta_signature_serialize(sig);
if (!sig_data) {
delta_signature_destroy(sig);
free(old_data);
*failed = true;
return NULL;
}
bool sig_sent = send_status(fd, STATUS_DELTA_SIGNATURE) && send_data(fd, sig_data);
data_destroy(sig_data);
if (!sig_sent) {
delta_signature_destroy(sig);
free(old_data);
*failed = true;
return NULL;
}
Status resp;
if (!receive_status(fd, &resp)) {
delta_signature_destroy(sig);
free(old_data);
*failed = true;
return NULL;
}
if (resp == STATUS_DELTA_DATA) {
Data* delta_data = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
if (!delta_data) {
delta_signature_destroy(sig);
free(old_data);
*failed = true;
return NULL;
}
Data* raw_delta = delta_data;
if (config->use_compression &&
!compression_should_skip_with_suffixes(
check_path, config->skip_compress_suffixes,
config->skip_compress_set ? config->skip_compress_count : -1)) {
raw_delta = data_decompress_limited(delta_data, MAX_RECEIVE_WHOLE_FILE_SIZE);
data_destroy(delta_data);
if (!raw_delta) {
free(old_data);
delta_signature_destroy(sig);
*failed = true;
return NULL;
}
}
Delta* delta = delta_deserialize(raw_delta);
data_destroy(raw_delta);
if (!delta) {
free(old_data);
delta_signature_destroy(sig);
*failed = true;
return NULL;
}
uint64_t new_size = delta->new_file_size;
if (new_size > MAX_RECEIVE_WHOLE_FILE_SIZE || new_size > SIZE_MAX) {
delta_destroy(delta);
free(old_data);
delta_signature_destroy(sig);
send_status(fd, STATUS_ERROR);
*failed = true;
return NULL;
}
void* new_data = delta_apply(old_data, old_size, delta, config->delta_block_size);
delta_destroy(delta);
if (!new_data) {
free(old_data);
delta_signature_destroy(sig);
*failed = true;
return NULL;
}
File* file = file_create(check_path);
if (!file) {
free(new_data);
free(old_data);
delta_signature_destroy(sig);
*failed = true;
return NULL;
}
if (config->use_metadata) {
int meta_ok = 1;
file->metadata = metadata_receive(fd, &meta_ok);
if (!meta_ok) {
file_destroy(file);
free(new_data);
free(old_data);
delta_signature_destroy(sig);
*failed = true;
return NULL;
}
}
Data* replacement = data_create(new_data, (size_t)new_size);
if (replacement == NULL) {
file_destroy(file);
free(old_data);
delta_signature_destroy(sig);
send_status(fd, STATUS_ERROR);
*failed = true;
return NULL;
}
data_destroy(file->data);
file->data = replacement;
free(old_data);
delta_signature_destroy(sig);
return file;
}
if (resp == STATUS_NEXT) {
delta_signature_destroy(sig);
free(old_data);
File* file = file_create(check_path);
if (!file) {
*failed = true;
return NULL;
}
if (config->use_metadata) {
int meta_ok = 1;
file->metadata = metadata_receive(fd, &meta_ok);
if (!meta_ok) {
file_destroy(file);
*failed = true;
return NULL;
}
}
Data* file_data = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
if (file_data == NULL) {
file_destroy(file);
*failed = true;
return NULL;
}
if (config->use_compression &&
!compression_should_skip_with_suffixes(
file->path, config->skip_compress_suffixes,
config->skip_compress_set ? config->skip_compress_count : -1)) {
Data* uncompressed = data_decompress_limited(file_data, MAX_RECEIVE_WHOLE_FILE_SIZE);
data_destroy(file_data);
if (uncompressed == NULL) {
file_destroy(file);
*failed = true;
return NULL;
}
if (uncompressed->size > MAX_FILE_DATA_SIZE) {
data_destroy(uncompressed);
file_destroy(file);
send_status(fd, STATUS_ERROR);
*failed = true;
return NULL;
}
file_data = uncompressed;
}
data_destroy(file->data);
file->data = file_data;
return file;
}
delta_signature_destroy(sig);
free(old_data);
send_status(fd, STATUS_ERROR);
*failed = true;
return NULL;
}
File* receive_incremental_check(int fd, const Config* config, bool* skipped) {
if (!config || !skipped) {
send_status(fd, STATUS_ERROR);
return NULL;
}
*skipped = false;
char* check_path = receive_str(fd);
if (check_path == NULL) {
return NULL;
}
unsigned long long check_size;
long long check_mtime;
long long check_mtime_nsec;
uint64_t check_checksum = 0;
if (!receive_n_data(fd, &check_size, sizeof(check_size)) ||
!receive_n_data(fd, &check_mtime, sizeof(check_mtime))) {
free(check_path);
return NULL;
}
if (!receive_n_data(fd, &check_mtime_nsec, sizeof(check_mtime_nsec)) || check_mtime_nsec < 0 ||
check_mtime_nsec >= 1000000000LL) {
free(check_path);
send_status(fd, STATUS_ERROR);
return NULL;
}
if (config->checksum && !receive_n_data(fd, &check_checksum, sizeof(check_checksum))) {
free(check_path);
return NULL;
}
if (check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
free(check_path);
send_status(fd, STATUS_ERROR);
return NULL;
}
if (has_path_traversal(check_path)) {
char* escaped_path = output_escape(check_path, log_get_8_bit_output());
log_message(LOG_LEVEL_ERROR, "Path traversal detected: %s",
escaped_path ? escaped_path : "<allocation failed>");
free(escaped_path);
free(check_path);
return NULL;
}
char* full_path = path_cat(config->receive_root_directory, check_path);
if (!full_path) {
free(check_path);
send_status(fd, STATUS_ERROR);
return NULL;
}
/* Open the existing destination entry (if any) once and keep the descriptor
until the quick-check below decides whether the old contents are needed. */
struct stat st;
bool has_old_file = false;
int old_fd = -1;
char* leaf = NULL;
int parent_fd = file_open_secure_parent(full_path, &leaf, false);
if (parent_fd >= 0) {
old_fd = openat(parent_fd, leaf, O_RDONLY | O_CLOEXEC | O_NOFOLLOW);
free(leaf);
close(parent_fd);
has_old_file = old_fd >= 0 && fstat(old_fd, &st) == 0 && S_ISREG(st.st_mode);
}
if (!has_old_file && old_fd >= 0) {
close(old_fd);
old_fd = -1;
}
unsigned long long old_size = has_old_file ? (unsigned long long)st.st_size : 0;
/* Decide from metadata alone whether the receiver already holds the file
the sender is offering. The old contents are only read into memory when
a checksum comparison or a delta transfer actually requires them. */
bool size_equal = has_old_file && old_size == check_size;
bool match_by_metadata = false;
if (size_equal && !config->ignore_times && !config->size_only) {
long long old_mtime_nsec = 0;
#ifdef __linux__
old_mtime_nsec = st.st_mtim.tv_nsec;
#endif
match_by_metadata = metadata_mtime_matches(st.st_mtime, old_mtime_nsec, (time_t)check_mtime,
(long)check_mtime_nsec, config->modify_window);
}
bool try_delta = config->use_delta && !config->whole_file && has_old_file &&
delta_should_attempt(old_size, check_size, config->delta_max_file_size);
bool checksum_needs_read = size_equal && !config->ignore_times && config->checksum;
bool need_old_data = checksum_needs_read || try_delta;
void* old_data = NULL;
if (need_old_data && has_old_file && old_size > 0 && old_size <= MAX_RECEIVE_WHOLE_FILE_SIZE &&
old_size <= SIZE_MAX) {
old_data = protocol_alloc((size_t)old_size);
if (old_data) {
size_t got = 0;
while (got < (size_t)old_size) {
ssize_t n = read(old_fd, (char*)old_data + got, (size_t)old_size - got);
if (n <= 0) {
free(old_data);
old_data = NULL;
break;
}
got += (size_t)n;
}
}
}
/* Quick-skip decision. If no content comparison is required this is final
and the old file was never read; if the read failed the file is not
skipped and the transfer proceeds with the full new contents. */
bool match = false;
if (checksum_needs_read) {
if (old_size == 0)
match = delta_xxhash64("", 0) == check_checksum;
else
match = old_data != NULL && delta_xxhash64(old_data, (size_t)old_size) == check_checksum;
} else if (size_equal && !config->ignore_times) {
match = config->size_only || match_by_metadata;
}
if (match) {
free(old_data);
if (!send_status(fd, STATUS_OK)) {
close(old_fd);
free(full_path);
free(check_path);
return NULL;
}
close(old_fd);
free(full_path);
free(check_path);
*skipped = true;
return NULL;
}
if (try_delta && old_data != NULL) {
bool delta_failed = false;
File* delta_file =
receive_delta_file(fd, config, check_path, old_data, old_size, &delta_failed);
old_data = NULL; /* receive_delta_file consumes the snapshot on every path */
if (delta_file) {
close(old_fd);
free(full_path);
free(check_path);
return delta_file;
}
if (delta_failed) {
close(old_fd);
free(full_path);
free(check_path);
return NULL;
}
}
free(old_data);
old_data = NULL;
if (!send_status(fd, STATUS_NEXT)) {
close(old_fd);
free(full_path);
free(check_path);
return NULL;
}
close(old_fd);
File* file = file_create(check_path);
free(check_path);
free(full_path);
if (file == NULL) {
return NULL;
}
if (config->use_metadata) {
int meta_ok = 1;
file->metadata = metadata_receive(fd, &meta_ok);
if (!meta_ok) {
file_destroy(file);
return NULL;
}
}
Data* file_data = receive_data_limited(fd, MAX_RECEIVE_WHOLE_FILE_SIZE);
if (file_data == NULL) {
file_destroy(file);
return NULL;
}
if (config->use_compression &&
!compression_should_skip_with_suffixes(file->path, config->skip_compress_suffixes,
config->skip_compress_set ? config->skip_compress_count
: -1)) {
Data* uncompressed = data_decompress_limited(file_data, MAX_RECEIVE_WHOLE_FILE_SIZE);
data_destroy(file_data);
if (uncompressed == NULL) {
file_destroy(file);
return NULL;
}
if (uncompressed->size > MAX_FILE_DATA_SIZE) {
data_destroy(uncompressed);
file_destroy(file);
return NULL;
}
file_data = uncompressed;
}
data_destroy(file->data);
file->data = file_data;
return file;
}
File* file_receive(const Config* config, int file_descriptor) {
char* path = receive_str(file_descriptor);
if (path == NULL)
return NULL;
if (path[0] == '\0' || has_path_traversal(path)) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_ERROR, "Invalid received file path: %s",
escaped_path ? escaped_path : "<allocation failed>");
free(escaped_path);
free(path);
return NULL;
}
File* file = file_create(path);
free(path);
if (file == NULL)
return NULL;
if (config->use_metadata) {
int meta_ok = 1;
file->metadata = metadata_receive(file_descriptor, &meta_ok);
if (!meta_ok) {
file_destroy(file);
return NULL;
}
}
Data* file_data = receive_data_limited(file_descriptor, MAX_RECEIVE_WHOLE_FILE_SIZE);
if (file_data == NULL) {
file_destroy(file);
return NULL;
}
if (config->use_compression &&
!compression_should_skip_with_suffixes(file->path, config->skip_compress_suffixes,
config->skip_compress_set ? config->skip_compress_count
: -1)) {
Data* file_data_uncompressed = data_decompress_limited(file_data, MAX_RECEIVE_WHOLE_FILE_SIZE);
data_destroy(file_data);
if (file_data_uncompressed == NULL) {
file_destroy(file);
return NULL;
}
if (file_data_uncompressed->size > MAX_FILE_DATA_SIZE) {
data_destroy(file_data_uncompressed);
file_destroy(file);
return NULL;
}
file_data = file_data_uncompressed;
}
data_destroy(file->data);
file->data = file_data;
return file;
}
int receive_manifest(int fd, const Config* config, int* next_status) {
if (!config) {
send_status(fd, STATUS_ERROR);
return -1;
}
int received_status = STATUS_ERROR;
int* status_out = next_status ? next_status : &received_status;
int count;
if (!receive_int(fd, &count)) {
send_status(fd, STATUS_ERROR);
return -1;
}
if (count < 0 || count > MAX_MANIFEST_ENTRIES) {
send_status(fd, STATUS_ERROR);
return -1;
}
ArrayList* manifest = array_list_create(free);
if (!manifest) {
send_status(fd, STATUS_ERROR);
return -1;
}
size_t manifest_bytes = 0;
for (int i = 0; i < count; i++) {
char* s = receive_str(fd);
size_t entry_size = s ? strlen(s) : 0;
if (!s || s[0] == '\0' || s[0] == '/' || has_path_traversal(s) ||
entry_size > MAX_MANIFEST_BYTES - manifest_bytes ||
(manifest_bytes += entry_size) > MAX_MANIFEST_BYTES || !array_list_add(manifest, s)) {
free(s);
array_list_delete(manifest);
send_status(fd, STATUS_ERROR);
return -1;
}
}
if (!receive_status(fd, status_out)) {
array_list_delete(manifest);
send_status(fd, STATUS_ERROR);
return -1;
}
/* Deletion is a commit operation: never perform it until the sender has
completed the manifest frame successfully. */
if (*status_out != STATUS_FINISHED || !config->use_delete) {
array_list_delete(manifest);
if (*status_out != STATUS_FINISHED)
send_status(fd, STATUS_ERROR);
return *status_out == STATUS_FINISHED ? 0 : -1;
}
fprintf(stderr, "Deleting files not in manifest...\n");
bool deletion_ok =
delete_extras_limited(config->receive_root_directory, manifest, MAX_SERVER_DELETE_COUNT);
array_list_delete(manifest);
if (!deletion_ok)
send_status(fd, STATUS_ERROR);
return deletion_ok ? 0 : -1;
}
+23
View File
@@ -0,0 +1,23 @@
#ifndef FILE_RECEIVE_H
#define FILE_RECEIVE_H
#include "config.h"
#include "file_types.h"
#include <stdbool.h>
/* Server-side file receive/save path. */
File* file_receive(const Config* config, int file_descriptor);
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
int receive_manifest(int fd, const Config* config, int* next_status);
/* Outcome of a single file_save_to_disk operation. The receiver needs to
distinguish "written" from "skipped" so --remove-source-files can be told
which sources were actually stored. */
typedef enum { FILE_SAVE_ERROR = 0, FILE_SAVE_WRITTEN = 1, FILE_SAVE_SKIPPED = 2 } FileSaveResult;
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
const Config* config);
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
#endif
+154
View File
@@ -0,0 +1,154 @@
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <poll.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/sendfile.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include "compression.h"
#include "data.h"
#include "file.h"
#include "log.h"
#include "metadata.h"
#include "protocol.h"
bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path) {
return file_send_single_calls_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, NULL, -1, 0);
}
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path,
char* const* skip_suffixes, int skip_count,
int compression_threads) {
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data))
return false;
const Data* data_to_send = file->data;
Data* compressed_data = NULL;
if (compression_level > 0 &&
!compression_should_skip_with_suffixes(file->path, skip_suffixes, skip_count)) {
compressed_data =
data_compress_with_threads(file->data, compression_level, compression_threads);
if (compressed_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to compress file data");
return false;
}
data_to_send = compressed_data;
}
if (send_path && !send_str(file_descriptor, file->path)) {
data_destroy(compressed_data);
return false;
}
if (use_metadata && !metadata_send(file_descriptor, file->metadata)) {
data_destroy(compressed_data);
return false;
}
if (!send_data(file_descriptor, data_to_send)) {
data_destroy(compressed_data);
return false;
}
data_destroy(compressed_data);
return true;
}
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
bool send_path) {
return file_send_sendfile_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, NULL, -1, 0);
}
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path, char* const* skip_suffixes,
int skip_count, int compression_threads) {
if (!file || !file->path || !file->data)
return false;
if (compression_level > 0)
return file_send_single_calls_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, skip_suffixes, skip_count,
compression_threads);
if (send_path && !send_str(file_descriptor, file->path))
return false;
if (use_metadata && !metadata_send(file_descriptor, file->metadata))
return false;
int fd = open(file->path, O_RDONLY);
if (fd == -1) {
log_perror("Could not open file for sendfile");
return false;
}
unsigned long long file_size = file->data->size;
struct stat source_stat;
if (fstat(fd, &source_stat) != 0 || !S_ISREG(source_stat.st_mode) ||
(unsigned long long)source_stat.st_size < file_size) {
close(fd);
return false;
}
if (!send_n_data(file_descriptor, &file_size, sizeof(unsigned long long))) {
close(fd);
return false;
}
/* sendfile cannot encrypt TLS records. Keep the framing identical but
route encrypted transfers through the deadline-aware IO layer. */
if (io_get_ssl() != NULL) {
unsigned char buffer[64 * 1024];
unsigned long long remaining = file_size;
bool ok = true;
while (remaining > 0) {
size_t want = remaining > sizeof(buffer) ? sizeof(buffer) : (size_t)remaining;
ssize_t got = read(fd, buffer, want);
if (got <= 0 || !send_n_data(file_descriptor, buffer, (size_t)got)) {
ok = false;
break;
}
remaining -= (unsigned long long)got;
}
close(fd);
return ok;
}
off_t offset = 0;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += 60;
while ((unsigned long long)offset < file_size) {
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
long long remaining = (long long)(deadline.tv_sec - now.tv_sec) * 1000LL +
(deadline.tv_nsec - now.tv_nsec) / 1000000LL;
if (remaining <= 0) {
close(fd);
return false;
}
struct pollfd pfd = {.fd = file_descriptor, .events = POLLOUT};
int timeout = remaining > INT_MAX ? INT_MAX : (int)remaining;
int polled = poll(&pfd, 1, timeout);
if (polled <= 0 || (pfd.revents & (POLLERR | POLLHUP | POLLNVAL))) {
close(fd);
return false;
}
ssize_t sent = sendfile(file_descriptor, fd, &offset, file_size - offset);
if (sent == -1) {
if (errno == EAGAIN || errno == EINTR)
continue;
log_perror("sendfile failed");
close(fd);
return false;
}
if (sent == 0) {
close(fd);
return false;
}
}
close(fd);
return true;
}
+21
View File
@@ -0,0 +1,21 @@
#ifndef FILE_SEND_H
#define FILE_SEND_H
#include "file_types.h"
#include <stdbool.h>
/* Client-side file send path. */
bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path);
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path,
char* const* skip_suffixes, int skip_count,
int compression_threads);
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
bool send_path);
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path, char* const* skip_suffixes,
int skip_count, int compression_threads);
#endif
+52 -13
View File
@@ -20,10 +20,18 @@ static bool path_is_within_root(const char* root, const char* path) {
(path[root_length] == '\0' || path[root_length] == '/'); (path[root_length] == '\0' || path[root_length] == '/');
} }
void file_store_set_authorized_root(int fd, const char* canonical_path) { bool file_store_set_authorized_root(int fd, const char* canonical_path) {
authorized_root_fd = fd; char* new_path = canonical_path ? str_dup(canonical_path) : NULL;
if (canonical_path && !new_path) {
authorized_root_fd = -1;
free(authorized_root_path);
authorized_root_path = NULL;
return false;
}
free(authorized_root_path); free(authorized_root_path);
authorized_root_path = canonical_path ? str_dup(canonical_path) : NULL; authorized_root_path = new_path;
authorized_root_fd = fd;
return true;
} }
int file_store_open_secure_parent(const char* path, char** leaf_out) { int file_store_open_secure_parent(const char* path, char** leaf_out) {
@@ -38,9 +46,19 @@ int file_store_open_secure_parent(const char* path, char** leaf_out) {
return -1; return -1;
} }
int fd; int fd;
if (authorized_root_fd >= 0 && authorized_root_path && path[0] == '/' && if (authorized_root_fd >= 0) {
path_is_within_root(authorized_root_path, path)) { if (!authorized_root_path || path[0] != '/' ||
!path_is_within_root(authorized_root_path, path)) {
free(copy);
free(leaf);
return -1;
}
fd = dup(authorized_root_fd); fd = dup(authorized_root_fd);
if (fd < 0) {
free(copy);
free(leaf);
return -1;
}
size_t root_length = strlen(authorized_root_path); size_t root_length = strlen(authorized_root_path);
char* relative = str_dup(path + root_length); char* relative = str_dup(path + root_length);
if (!relative) { if (!relative) {
@@ -64,10 +82,18 @@ int file_store_open_secure_parent(const char* path, char** leaf_out) {
char* save = NULL; char* save = NULL;
char* component = strtok_r(parent, "/", &save); char* component = strtok_r(parent, "/", &save);
while (component) { while (component) {
if (strcmp(component, ".") != 0 && strcmp(component, "..") != 0) { if (strcmp(component, "..") == 0) {
close(fd);
free(copy);
free(leaf);
return -1;
}
if (strcmp(component, ".") != 0) {
int next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); int next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (next < 0 && errno == ENOENT && mkdirat(fd, component, 0755) == 0) if (next < 0 && errno == ENOENT) {
next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); if (mkdirat(fd, component, 0755) == 0 || errno == EEXIST)
next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
if (next < 0) { if (next < 0) {
close(fd); close(fd);
free(copy); free(copy);
@@ -114,7 +140,8 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
} }
bool file_store_write_secure(const char* path, const void* data, unsigned long long data_size, bool file_store_write_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata) { bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability) {
char* leaf = NULL; char* leaf = NULL;
int dirfd = file_store_open_secure_parent(path, &leaf); int dirfd = file_store_open_secure_parent(path, &leaf);
if (dirfd < 0) if (dirfd < 0)
@@ -127,12 +154,23 @@ bool file_store_write_secure(const char* path, const void* data, unsigned long l
if (!sparse || data_size == 0 || ftruncate(fd, (off_t)data_size) == 0) if (!sparse || data_size == 0 || ftruncate(fd, (off_t)data_size) == 0)
ok = write_all(fd, data, data_size); ok = write_all(fd, data, data_size);
if (ok && metadata) if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata); ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
} }
} else { } else {
char tmp[NAME_MAX]; int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%u", leaf, (long)getpid(), 99U);
if (tmp_size < 0) {
close(dirfd);
free(leaf);
return false;
}
char* tmp = malloc((size_t)tmp_size + 1);
if (!tmp) {
close(dirfd);
free(leaf);
return false;
}
for (unsigned int i = 0; i < 100 && !ok; ++i) { for (unsigned int i = 0; i < 100 && !ok; ++i) {
snprintf(tmp, sizeof(tmp), ".%s.tmp.%ld.%u", leaf, (long)getpid(), i); snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
fd = openat(dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600); fd = openat(dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
if (fd < 0) if (fd < 0)
continue; continue;
@@ -141,7 +179,7 @@ bool file_store_write_secure(const char* path, const void* data, unsigned long l
if (ok || (!sparse || data_size == 0)) if (ok || (!sparse || data_size == 0))
ok = write_all(fd, data, data_size); ok = write_all(fd, data, data_size);
if (ok && metadata) if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata); ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
if (close(fd) != 0) if (close(fd) != 0)
ok = false; ok = false;
fd = -1; fd = -1;
@@ -150,6 +188,7 @@ bool file_store_write_secure(const char* path, const void* data, unsigned long l
if (!ok) if (!ok)
unlinkat(dirfd, tmp, 0); unlinkat(dirfd, tmp, 0);
} }
free(tmp);
} }
if (fd >= 0) if (fd >= 0)
close(fd); close(fd);
+3 -2
View File
@@ -4,10 +4,11 @@
#include "file.h" #include "file.h"
#include <stdbool.h> #include <stdbool.h>
void file_store_set_authorized_root(int fd, const char* canonical_path); bool file_store_set_authorized_root(int fd, const char* canonical_path);
int file_store_open_secure_parent(const char* path, char** leaf_out); int file_store_open_secure_parent(const char* path, char** leaf_out);
bool file_store_rename_secure(const char* old_path, const char* new_path); bool file_store_rename_secure(const char* old_path, const char* new_path);
bool file_store_write_secure(const char* path, const void* data, unsigned long long data_size, bool file_store_write_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata); bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability);
#endif #endif
+25
View File
@@ -0,0 +1,25 @@
#ifndef FILE_TYPES_H
#define FILE_TYPES_H
#include "data.h"
#include <stdbool.h>
#include <sys/stat.h>
typedef enum { FILE_TYPE_REGULAR, FILE_TYPE_SYMLINK, FILE_TYPE_DIR } FileType;
typedef struct {
mode_t mode;
uid_t uid;
gid_t gid;
time_t mtime_sec;
long mtime_nsec;
} FileMetadata;
typedef struct {
char* path;
Data* data;
FileMetadata* metadata;
bool skip;
} File;
#endif
+89 -1
View File
@@ -1,20 +1,61 @@
#include "log.h" #include "log.h"
#include <errno.h>
#include <stdbool.h>
#include <stdarg.h> #include <stdarg.h>
#include <stdio.h> #include <stdio.h>
#include <string.h>
#include <time.h> #include <time.h>
static const char* log_level_strings[] = {"DEBUG", "INFO", "WARN", "ERROR"}; static const char* log_level_strings[] = {"DEBUG", "INFO", "WARN", "ERROR"};
static LogLevel current_log_level = LOG_LEVEL_WARNING; static LogLevel current_log_level = LOG_LEVEL_WARNING;
static uint32_t current_debug_flags = 0;
static uint32_t info_flags = 0;
static bool info_flags_explicit = false;
static FILE* log_fp = NULL; static FILE* log_fp = NULL;
static _Thread_local bool eight_bit_output;
static LogStderrMode stderr_mode = LOG_STDERR_ERRORS;
void set_log_level(LogLevel level) { void set_log_level(LogLevel level) {
current_log_level = level; current_log_level = level;
} }
void set_log_debug_flags(uint32_t flags) {
current_debug_flags = flags;
}
uint32_t get_log_debug_flags(void) {
return current_debug_flags;
}
void set_log_info_flags(uint32_t flags) {
info_flags = flags;
info_flags_explicit = true;
}
uint32_t get_log_info_flags(void) {
return info_flags;
}
void log_set_file(FILE* fp) { void log_set_file(FILE* fp) {
log_fp = fp; log_fp = fp;
} }
void log_set_8_bit_output(bool enabled) {
eight_bit_output = enabled;
}
bool log_get_8_bit_output(void) {
return eight_bit_output;
}
void log_set_stderr_mode(LogStderrMode mode) {
stderr_mode = mode;
}
LogStderrMode log_get_stderr_mode(void) {
return stderr_mode;
}
static inline void write_message(FILE* dest_io, LogLevel log_level, struct tm t, const char* format, static inline void write_message(FILE* dest_io, LogLevel log_level, struct tm t, const char* format,
va_list args) { va_list args) {
fprintf(dest_io, "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t.tm_year + 1900, t.tm_mon + 1, fprintf(dest_io, "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t.tm_year + 1900, t.tm_mon + 1,
@@ -35,7 +76,7 @@ void log_message(LogLevel log_level, const char* format, ...) {
return; return;
FILE* dest_io = stdout; FILE* dest_io = stdout;
if (log_level == LOG_LEVEL_ERROR) { if (stderr_mode == LOG_STDERR_ALL || log_level == LOG_LEVEL_ERROR) {
dest_io = stderr; dest_io = stderr;
} }
@@ -50,3 +91,50 @@ void log_message(LogLevel log_level, const char* format, ...) {
va_end(args); va_end(args);
} }
} }
void log_debug_message(LogDebugFlag flag, const char* format, ...) {
if (current_log_level > LOG_LEVEL_DEBUG || !(current_debug_flags & flag))
return;
time_t now = time(NULL);
struct tm t;
if (!localtime_r(&now, &t))
return;
va_list args;
va_start(args, format);
write_message(stdout, LOG_LEVEL_DEBUG, t, format, args);
va_end(args);
if (log_fp) {
va_start(args, format);
write_message(log_fp, LOG_LEVEL_DEBUG, t, format, args);
va_end(args);
}
}
void log_info_message(LogInfoFlag flag, const char* format, ...) {
if ((info_flags_explicit && (info_flags & flag) == 0) ||
(!info_flags_explicit && current_log_level > LOG_LEVEL_DEBUG))
return;
time_t now = time(NULL);
struct tm t;
if (!localtime_r(&now, &t))
return;
va_list args;
va_start(args, format);
write_message(stdout, LOG_LEVEL_INFO, t, format, args);
va_end(args);
if (log_fp) {
va_start(args, format);
write_message(log_fp, LOG_LEVEL_INFO, t, format, args);
va_end(args);
}
}
void log_perror(const char* context) {
log_message(LOG_LEVEL_ERROR, "%s: %s", context, strerror(errno));
}
+30
View File
@@ -2,11 +2,41 @@
#define LOG_H #define LOG_H
#include <stdio.h> #include <stdio.h>
#include <stdbool.h>
#include <stdint.h>
typedef enum { LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_WARNING, LOG_LEVEL_ERROR } LogLevel; typedef enum { LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_WARNING, LOG_LEVEL_ERROR } LogLevel;
typedef enum { LOG_STDERR_ERRORS, LOG_STDERR_ALL } LogStderrMode;
typedef enum {
LOG_DEBUG_IO = 1u << 0,
LOG_DEBUG_PROTO = 1u << 1,
LOG_DEBUG_PACK = 1u << 2,
LOG_DEBUG_UTIL = 1u << 3,
LOG_DEBUG_ALL = (1u << 4) - 1,
} LogDebugFlag;
typedef enum {
LOG_INFO_COPY = 1u << 0,
LOG_INFO_MISC = 1u << 1,
LOG_INFO_SKIP = 1u << 2,
LOG_INFO_STATS = 1u << 3,
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS,
} LogInfoFlag;
void log_message(LogLevel log_level, const char* message, ...); void log_message(LogLevel log_level, const char* message, ...);
void log_perror(const char* context);
void set_log_level(LogLevel level); void set_log_level(LogLevel level);
void set_log_debug_flags(uint32_t flags);
uint32_t get_log_debug_flags(void);
void log_debug_message(LogDebugFlag flag, const char* message, ...);
void set_log_info_flags(uint32_t flags);
uint32_t get_log_info_flags(void);
void log_info_message(LogInfoFlag flag, const char* message, ...);
void log_set_file(FILE* fp); void log_set_file(FILE* fp);
void log_set_8_bit_output(bool enabled);
bool log_get_8_bit_output(void);
void log_set_stderr_mode(LogStderrMode mode);
LogStderrMode log_get_stderr_mode(void);
#endif #endif
+65 -10
View File
@@ -2,6 +2,7 @@
#include "file.h" #include "file.h"
#include "log.h" #include "log.h"
#include "protocol.h" #include "protocol.h"
#include "utils.h"
#include <errno.h> #include <errno.h>
#include <fcntl.h> #include <fcntl.h>
#include <stdint.h> #include <stdint.h>
@@ -24,6 +25,29 @@ typedef char static_assert_mode_t_fits[(sizeof(mode_t) <= sizeof(int32_t)) ? 1 :
typedef char static_assert_uid_t_fits[(sizeof(uid_t) <= sizeof(int32_t)) ? 1 : -1]; typedef char static_assert_uid_t_fits[(sizeof(uid_t) <= sizeof(int32_t)) ? 1 : -1];
typedef char static_assert_gid_t_fits[(sizeof(gid_t) <= sizeof(int32_t)) ? 1 : -1]; typedef char static_assert_gid_t_fits[(sizeof(gid_t) <= sizeof(int32_t)) ? 1 : -1];
bool metadata_mtime_matches(time_t left_sec, long left_nsec, time_t right_sec, long right_nsec,
int modify_window) {
int64_t left = (int64_t)left_sec;
int64_t right = (int64_t)right_sec;
int64_t seconds;
int64_t nanoseconds;
if (left > right || (left == right && left_nsec >= right_nsec)) {
seconds = left - right;
nanoseconds = (int64_t)left_nsec - (int64_t)right_nsec;
} else {
seconds = right - left;
nanoseconds = (int64_t)right_nsec - (int64_t)left_nsec;
}
if (nanoseconds < 0) {
seconds--;
nanoseconds += 1000000000LL;
}
if (modify_window == 0)
return left == right;
return seconds < modify_window || (seconds == modify_window && nanoseconds == 0);
}
void metadata_to_buf(char** buf, const FileMetadata* m) { void metadata_to_buf(char** buf, const FileMetadata* m) {
int32_t present = (m != NULL) ? 1 : 0; int32_t present = (m != NULL) ? 1 : 0;
memcpy(*buf, &present, sizeof(present)); memcpy(*buf, &present, sizeof(present));
@@ -51,9 +75,11 @@ FileMetadata* metadata_from_buf(char** buf) {
int32_t present; int32_t present;
memcpy(&present, *buf, sizeof(present)); memcpy(&present, *buf, sizeof(present));
*buf += sizeof(present); *buf += sizeof(present);
if (present != 0 && present != 1)
return NULL;
if (!present) if (!present)
return NULL; return NULL;
FileMetadata* m = malloc(sizeof(FileMetadata)); FileMetadata* m = protocol_alloc(sizeof(FileMetadata));
if (m == NULL) if (m == NULL)
return NULL; return NULL;
int32_t mode; int32_t mode;
@@ -76,10 +102,15 @@ FileMetadata* metadata_from_buf(char** buf) {
memcpy(&mtime_nsec, *buf, sizeof(mtime_nsec)); memcpy(&mtime_nsec, *buf, sizeof(mtime_nsec));
*buf += sizeof(mtime_nsec); *buf += sizeof(mtime_nsec);
m->mtime_nsec = (long)mtime_nsec; m->mtime_nsec = (long)mtime_nsec;
if (present != 1 || mtime_nsec < 0 || mtime_nsec >= 1000000000LL || mode < 0 || uid < 0 ||
gid < 0) {
free(m);
return NULL;
}
return m; return m;
} }
bool metadata_send(int file_descriptor, FileMetadata* m) { bool metadata_send(int file_descriptor, const FileMetadata* m) {
if (m == NULL) { if (m == NULL) {
int32_t zero = 0; int32_t zero = 0;
return send_n_data(file_descriptor, &zero, sizeof(zero)); return send_n_data(file_descriptor, &zero, sizeof(zero));
@@ -115,7 +146,7 @@ FileMetadata* metadata_receive(int file_descriptor, int* ok) {
*ok = 0; *ok = 0;
return NULL; return NULL;
} }
FileMetadata* m = malloc(sizeof(FileMetadata)); FileMetadata* m = protocol_alloc(sizeof(FileMetadata));
if (m == NULL) { if (m == NULL) {
if (ok) if (ok)
*ok = 0; *ok = 0;
@@ -172,11 +203,27 @@ FileMetadata* metadata_receive(int file_descriptor, int* ok) {
return m; return m;
} }
void file_restore_metadata(const char* path, const FileMetadata* metadata) { static mode_t metadata_mode(const FileMetadata* metadata, mode_t current_mode,
bool preserve_executability) {
const mode_t execute_bits = S_IXUSR | S_IXGRP | S_IXOTH;
if (preserve_executability)
return (current_mode & 0777 & ~execute_bits) | (metadata->mode & execute_bits);
return metadata->mode & 0777 & ~(S_IWGRP | S_IWOTH);
}
void file_restore_metadata(const char* path, const FileMetadata* metadata,
bool preserve_executability) {
if (metadata == NULL) if (metadata == NULL)
return; return;
if (chmod(path, metadata->mode & 07777 & ~(S_ISUID | S_ISGID)) != 0) struct stat current;
log_message(LOG_LEVEL_WARNING, "Failed to chmod %s: %s", path, strerror(errno)); mode_t current_mode = stat(path, &current) == 0 ? current.st_mode : 0;
mode_t safe_mode = metadata_mode(metadata, current_mode, preserve_executability);
if (chmod(path, safe_mode) != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Failed to chmod %s: %s",
escaped_path ? escaped_path : "<allocation failed>", strerror(errno));
free(escaped_path);
}
/* Never apply client-supplied ownership. The descriptor API below is the /* Never apply client-supplied ownership. The descriptor API below is the
receiver write path; retain this legacy API only for compatibility. */ receiver write path; retain this legacy API only for compatibility. */
struct timespec times[2]; struct timespec times[2];
@@ -184,15 +231,23 @@ void file_restore_metadata(const char* path, const FileMetadata* metadata) {
times[0].tv_nsec = UTIME_OMIT; times[0].tv_nsec = UTIME_OMIT;
times[1].tv_sec = metadata->mtime_sec; times[1].tv_sec = metadata->mtime_sec;
times[1].tv_nsec = metadata->mtime_nsec; times[1].tv_nsec = metadata->mtime_nsec;
if (utimensat(AT_FDCWD, path, times, 0) != 0) if (utimensat(AT_FDCWD, path, times, 0) != 0) {
log_message(LOG_LEVEL_WARNING, "Failed to set timestamps on %s: %s", path, strerror(errno)); char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Failed to set timestamps on %s: %s",
escaped_path ? escaped_path : "<allocation failed>", strerror(errno));
free(escaped_path);
}
} }
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata) { bool file_restore_metadata_fd(int fd, const FileMetadata* metadata, bool preserve_executability) {
if (fd < 0 || metadata == NULL) if (fd < 0 || metadata == NULL)
return metadata == NULL; return metadata == NULL;
bool ok = true; bool ok = true;
if (fchmod(fd, metadata->mode & 07777 & ~(S_ISUID | S_ISGID)) != 0) struct stat current;
if (fstat(fd, &current) != 0)
return false;
mode_t safe_mode = metadata_mode(metadata, current.st_mode, preserve_executability);
if (fchmod(fd, safe_mode) != 0)
ok = false; ok = false;
/* Client uid/gid values are deliberately not authoritative. */ /* Client uid/gid values are deliberately not authoritative. */
struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT}, struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
+9 -3
View File
@@ -5,6 +5,7 @@
#include <stdbool.h> #include <stdbool.h>
#include <stdint.h> #include <stdint.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <time.h>
/* /*
* Wire format (introduced in protocol version 2.0.0): * Wire format (introduced in protocol version 2.0.0):
@@ -27,9 +28,14 @@
void metadata_to_buf(char** buf, const FileMetadata* m); void metadata_to_buf(char** buf, const FileMetadata* m);
FileMetadata* metadata_from_buf(char** buf); FileMetadata* metadata_from_buf(char** buf);
bool metadata_send(int file_descriptor, FileMetadata* m); bool metadata_send(int file_descriptor, const FileMetadata* m);
FileMetadata* metadata_receive(int file_descriptor, int* ok); FileMetadata* metadata_receive(int file_descriptor, int* ok);
void file_restore_metadata(const char* path, const FileMetadata* metadata); void file_restore_metadata(const char* path, const FileMetadata* metadata,
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata); bool preserve_executability);
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata, bool preserve_executability);
/* Compare timestamps using rsync's whole-second modification window. */
bool metadata_mtime_matches(time_t left_sec, long left_nsec, time_t right_sec, long right_nsec,
int modify_window);
#endif #endif
+119 -11
View File
@@ -26,9 +26,14 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
context->scanner_done = false; context->scanner_done = false;
context->loader_done = false; context->loader_done = false;
context->manifest = NULL; context->manifest = NULL;
context->remove_source_files = NULL;
context->total_files = 0;
context->progress_bytes = 0; context->progress_bytes = 0;
context->total_bytes = 0;
context->sender_done = false; context->sender_done = false;
atomic_init(&context->cancelled, false); atomic_init(&context->cancelled, false);
protocol_session_init(&context->allocation_session, -1, -1);
protocol_session_set_max_alloc(&context->allocation_session, config->max_alloc);
int init = 0; int init = 0;
if (mtx_init(&context->mutex_scanner, mtx_plain) != thrd_success) if (mtx_init(&context->mutex_scanner, mtx_plain) != thrd_success)
goto fail; goto fail;
@@ -55,7 +60,7 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
return context; return context;
fail: fail:
perror("Error initializing synchronization objects"); log_perror("Error initializing synchronization objects");
if (init >= 6) if (init >= 6)
cnd_destroy(&context->condition_not_empty_loader); cnd_destroy(&context->condition_not_empty_loader);
if (init >= 5) if (init >= 5)
@@ -76,6 +81,8 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
if (context->manifest) { if (context->manifest) {
array_list_delete(context->manifest); array_list_delete(context->manifest);
} }
if (context->remove_source_files)
array_list_delete(context->remove_source_files);
config_delete(context->config); config_delete(context->config);
queue_destroy(context->queue_scanner); queue_destroy(context->queue_scanner);
queue_destroy(context->queue_loader); queue_destroy(context->queue_loader);
@@ -98,9 +105,14 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
context->queue = queue; context->queue = queue;
context->file_descriptor = file_descriptor; context->file_descriptor = file_descriptor;
context->ssl = ssl; context->ssl = ssl;
context->outcomes.entries = NULL;
context->outcomes.count = 0;
context->outcomes.capacity = 0;
protocol_session_init(&context->session, file_descriptor, file_descriptor); protocol_session_init(&context->session, file_descriptor, file_descriptor);
protocol_session_set_ssl(&context->session, ssl); protocol_session_set_ssl(&context->session, ssl);
context->receiver_done = false; context->receiver_done = false;
context->queued_bytes = 0;
context->max_queue_bytes = 0;
atomic_init(&context->cancelled, false); atomic_init(&context->cancelled, false);
int init = 0; int init = 0;
if (mtx_init(&context->mutex, mtx_plain) != thrd_success) if (mtx_init(&context->mutex, mtx_plain) != thrd_success)
@@ -116,7 +128,7 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
return context; return context;
fail: fail:
perror("Error initializing synchronization objects"); log_perror("Error initializing synchronization objects");
if (init >= 3) if (init >= 3)
cnd_destroy(&context->condition_not_empty); cnd_destroy(&context->condition_not_empty);
if (init >= 2) if (init >= 2)
@@ -130,20 +142,80 @@ fail:
void pipeline_context_receiver_destroy(PipelineContextReceiver* context) { void pipeline_context_receiver_destroy(PipelineContextReceiver* context) {
config_delete(context->config); config_delete(context->config);
queue_destroy(context->queue); queue_destroy(context->queue);
receiver_outcomes_destroy(&context->outcomes);
mtx_destroy(&context->mutex); mtx_destroy(&context->mutex);
cnd_destroy(&context->condition_not_full); cnd_destroy(&context->condition_not_full);
cnd_destroy(&context->condition_not_empty); cnd_destroy(&context->condition_not_empty);
free(context); free(context);
} }
void pipeline_context_receiver_set_queue_byte_limit(PipelineContextReceiver* context,
size_t max_bytes) {
if (context == NULL)
return;
mtx_lock(&context->mutex);
context->max_queue_bytes = max_bytes;
context->queued_bytes = 0;
cnd_broadcast(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
void pipeline_context_receiver_note_bytes_released(PipelineContextReceiver* context,
size_t released_bytes) {
if (context == NULL || context->max_queue_bytes == 0 || released_bytes == 0)
return;
mtx_lock(&context->mutex);
if (released_bytes >= context->queued_bytes)
context->queued_bytes = 0;
else
context->queued_bytes -= released_bytes;
cnd_signal(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, File* file) {
if (context == NULL || file == NULL)
return false;
size_t file_bytes = file->data ? file->data->size : 0;
mtx_lock(&context->mutex);
while (!atomic_load(&context->cancelled)) {
bool blocked_by_count = queue_is_full(context->queue);
bool blocked_by_budget = false;
if (context->max_queue_bytes > 0) {
size_t budget = context->max_queue_bytes;
size_t used = context->queued_bytes;
if (used >= budget) {
blocked_by_budget = true;
} else if (file_bytes > budget - used) {
/* A single payload larger than the whole budget (not possible with
the per-file receive cap) is only admitted to an empty pipeline so
the wait can never deadlock. */
blocked_by_budget = used != 0;
}
}
if (!blocked_by_count && !blocked_by_budget)
break;
cnd_wait(&context->condition_not_full, &context->mutex);
}
if (atomic_load(&context->cancelled)) {
mtx_unlock(&context->mutex);
file_destroy(file);
return false;
}
if (!queue_enqueue(context->queue, file)) {
mtx_unlock(&context->mutex);
file_destroy(file);
return false;
}
context->queued_bytes += file_bytes;
cnd_signal(&context->condition_not_empty);
mtx_unlock(&context->mutex);
return true;
}
static bool receiver_enqueue_file(File* file, void* context_pointer) { static bool receiver_enqueue_file(File* file, void* context_pointer) {
PipelineContextReceiver* context = context_pointer; PipelineContextReceiver* context = (PipelineContextReceiver*)context_pointer;
if (queue_enqueue_multithreaded_cancel(context->queue, file, &context->mutex, return pipeline_context_receiver_enqueue_file(context, file);
&context->condition_not_empty,
&context->condition_not_full, &context->cancelled))
return true;
file_destroy(file);
return false;
} }
static void receiver_thread_fail(PipelineContextReceiver* context) { static void receiver_thread_fail(PipelineContextReceiver* context) {
@@ -163,7 +235,7 @@ int receive_thread(void* pipeline_context) {
const Config* config = context->config; const Config* config = context->config;
mtx_unlock(&context->mutex); mtx_unlock(&context->mutex);
ReceiverSink sink = {receiver_enqueue_file, context, false, false}; ReceiverSink sink = {receiver_enqueue_file, context, false, false, NULL};
if (receiver_process((Config*)config, file_descriptor, &sink) != 0) { if (receiver_process((Config*)config, file_descriptor, &sink) != 0) {
receiver_thread_fail(context); receiver_thread_fail(context);
protocol_session_unbind(); protocol_session_unbind();
@@ -179,10 +251,21 @@ int receive_thread(void* pipeline_context) {
int write_thread(void* pipeline_context) { int write_thread(void* pipeline_context) {
PipelineContextReceiver* context = (PipelineContextReceiver*)pipeline_context; PipelineContextReceiver* context = (PipelineContextReceiver*)pipeline_context;
protocol_session_bind(&context->session);
mtx_lock(&context->mutex); mtx_lock(&context->mutex);
bool save_to_disk = context->config->save_to_disk; bool save_to_disk = context->config->save_to_disk;
char* root_directory = str_dup(context->config->receive_root_directory); char* root_directory = str_dup(context->config->receive_root_directory);
mtx_unlock(&context->mutex); mtx_unlock(&context->mutex);
if (save_to_disk && !root_directory) {
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
protocol_session_unbind();
return thrd_error;
}
while (true) { while (true) {
File* file = File* file =
@@ -190,10 +273,33 @@ int write_thread(void* pipeline_context) {
&context->condition_not_full, &context->receiver_done); &context->condition_not_full, &context->receiver_done);
if (file == NULL) { if (file == NULL) {
free(root_directory); free(root_directory);
protocol_session_unbind();
return thrd_success; return thrd_success;
} }
if (save_to_disk && !file_save_to_disk(root_directory, file, context->config)) { size_t file_bytes = file->data ? file->data->size : 0;
FileSaveResult result = FILE_SAVE_SKIPPED;
if (save_to_disk) {
result = file_save_to_disk_full(root_directory, file, context->config);
if (result == FILE_SAVE_ERROR) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
free(root_directory);
protocol_session_unbind();
return thrd_error;
}
}
/* Record the per-file outcome so a --remove-source-files sender learns
which sources were actually written versus skipped on the receiver. */
if (context->config->remove_source_files &&
!receiver_outcomes_append(&context->outcomes, (unsigned char)result)) {
file_destroy(file); file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex); mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true); atomic_store(&context->cancelled, true);
context->receiver_done = true; context->receiver_done = true;
@@ -201,8 +307,10 @@ int write_thread(void* pipeline_context) {
cnd_broadcast(&context->condition_not_empty); cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex); mtx_unlock(&context->mutex);
free(root_directory); free(root_directory);
protocol_session_unbind();
return thrd_error; return thrd_error;
} }
file_destroy(file); file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
} }
} }
+26
View File
@@ -9,6 +9,7 @@
#include "file.h" #include "file.h"
#include "protocol.h" #include "protocol.h"
#include "queue.h" #include "queue.h"
#include "receiver.h"
#include <openssl/ssl.h> #include <openssl/ssl.h>
typedef struct { typedef struct {
@@ -24,10 +25,14 @@ typedef struct {
cnd_t condition_not_empty_loader; cnd_t condition_not_empty_loader;
bool loader_done; bool loader_done;
ArrayList* manifest; ArrayList* manifest;
ArrayList* remove_source_files;
mtx_t mutex_progress; mtx_t mutex_progress;
int total_files;
unsigned long long progress_bytes; unsigned long long progress_bytes;
unsigned long long total_bytes;
bool sender_done; bool sender_done;
atomic_bool cancelled; atomic_bool cancelled;
ProtocolSession allocation_session;
} PipelineContextSender; } PipelineContextSender;
typedef struct PipelineContextReceiver { typedef struct PipelineContextReceiver {
@@ -36,11 +41,20 @@ typedef struct PipelineContextReceiver {
int file_descriptor; int file_descriptor;
SSL* ssl; SSL* ssl;
ProtocolSession session; ProtocolSession session;
ReceiverOutcomes outcomes;
mtx_t mutex; mtx_t mutex;
cnd_t condition_not_full; cnd_t condition_not_full;
cnd_t condition_not_empty; cnd_t condition_not_empty;
bool receiver_done; bool receiver_done;
atomic_bool cancelled; atomic_bool cancelled;
/* Aggregate payload bytes that have been received but not yet released by
the disk writer (queued or in the writer's hand). Guarded by `mutex`.
When `max_queue_bytes` is non-zero the receiver blocks before enqueuing
once this total would exceed it, so decompressed/copied file payloads
buffered ahead of a slow disk writer respect the per-connection memory
budget instead of growing without bound. */
size_t queued_bytes;
size_t max_queue_bytes;
} PipelineContextReceiver; } PipelineContextReceiver;
PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* queue_scanner, PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* queue_scanner,
@@ -49,6 +63,18 @@ void pipeline_context_sender_destroy(PipelineContextSender* context);
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver, PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
int file_descriptor, SSL* ssl); int file_descriptor, SSL* ssl);
void pipeline_context_receiver_destroy(PipelineContextReceiver* context); void pipeline_context_receiver_destroy(PipelineContextReceiver* context);
/* Bound the bytes buffered ahead of the disk writer (see max_queue_bytes). */
void pipeline_context_receiver_set_queue_byte_limit(PipelineContextReceiver* context,
size_t max_bytes);
/* Blocking enqueue used by the receive pipeline sink. Blocks while the queue
is full by element count or when adding `file` would push queued_bytes over
the configured byte limit; waits until the disk writer releases bytes.
Takes ownership of `file` on success and destroys it on failure/cancel. */
bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, File* file);
/* Account for `released_bytes` of payload memory that has been freed by the
disk writer, unblocking a receiver that is waiting on the byte limit. */
void pipeline_context_receiver_note_bytes_released(PipelineContextReceiver* context,
size_t released_bytes);
int receive_thread(void* pipeline_context); int receive_thread(void* pipeline_context);
int write_thread(void* pipeline_context); int write_thread(void* pipeline_context);
#endif #endif
+192 -85
View File
@@ -1,5 +1,6 @@
#include "protocol.h" #include "protocol.h"
#include "log.h" #include "log.h"
#include "utils.h"
#include <errno.h> #include <errno.h>
#include <limits.h> #include <limits.h>
#include <openssl/ssl.h> #include <openssl/ssl.h>
@@ -13,21 +14,45 @@
#define RECEIVE_TIMEOUT_SEC 60 /* 60 second per-message timeout */ #define RECEIVE_TIMEOUT_SEC 60 /* 60 second per-message timeout */
#define SEND_TIMEOUT_SEC 60 #define SEND_TIMEOUT_SEC 60
#define MAX_CONNECTION_MEMORY (1024ULL * 1024 * 1024) /* 1 GB total per connection */
static __thread int io_read_fd = -1; static __thread int io_read_fd = -1;
static __thread int io_write_fd = -1; static __thread int io_write_fd = -1;
static __thread SSL* io_ssl; static __thread SSL* io_ssl;
static __thread ProtocolSession* bound_session; static __thread ProtocolSession* bound_session;
static __thread ProtocolSession legacy_io_session = {
.read_fd = -1, .write_fd = -1, .max_alloc = DEFAULT_MAX_ALLOC};
static unsigned long long io_bwlimit = 0; static unsigned long long io_bwlimit = 0;
static long long bw_tokens = 0;
static struct timespec bw_last_refill = {0, 0};
static mtx_t bw_mutex; static mtx_t bw_mutex;
static once_flag bw_mutex_once = ONCE_FLAG_INIT; static once_flag bw_mutex_once = ONCE_FLAG_INIT;
static __thread unsigned long long total_allocated_bytes = 0; static unsigned long long global_bwlimit(void);
static bool protocol_reserve_memory(ProtocolSession* session, size_t charge) {
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
while (true) {
if (allocated > MAX_CONNECTION_MEMORY ||
(unsigned long long)charge > MAX_CONNECTION_MEMORY - allocated)
return false;
if (atomic_compare_exchange_weak(&session->total_allocated_bytes, &allocated,
allocated + (unsigned long long)charge))
return true;
}
}
static void protocol_release_memory_for_session(ProtocolSession* session, size_t charge) {
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
while (true) {
unsigned long long remaining = (unsigned long long)charge >= allocated ? 0 : allocated - charge;
if (atomic_compare_exchange_weak(&session->total_allocated_bytes, &allocated, remaining))
break;
}
}
void protocol_release_memory(size_t charge) {
ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
protocol_release_memory_for_session(session, charge);
}
void io_set_fds(int read_fd, int write_fd) { void io_set_fds(int read_fd, int write_fd) {
bound_session = NULL; bound_session = NULL;
io_read_fd = read_fd; io_read_fd = read_fd;
@@ -35,7 +60,13 @@ void io_set_fds(int read_fd, int write_fd) {
/* A descriptor switch starts a new transport; never reuse a TLS object /* A descriptor switch starts a new transport; never reuse a TLS object
belonging to a previous connection or test pipe. */ belonging to a previous connection or test pipe. */
io_ssl = NULL; io_ssl = NULL;
total_allocated_bytes = 0; legacy_io_session.read_fd = read_fd;
legacy_io_session.write_fd = write_fd;
legacy_io_session.ssl = NULL;
legacy_io_session.eight_bit_output = false;
atomic_store(&legacy_io_session.total_allocated_bytes, 0);
legacy_io_session.max_alloc = DEFAULT_MAX_ALLOC;
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
} }
void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd) { void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd) {
@@ -44,12 +75,46 @@ void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd)
memset(session, 0, sizeof(*session)); memset(session, 0, sizeof(*session));
session->read_fd = read_fd; session->read_fd = read_fd;
session->write_fd = write_fd; session->write_fd = write_fd;
if (io_bwlimit) session->max_alloc = DEFAULT_MAX_ALLOC;
protocol_session_set_bwlimit(session, io_bwlimit); atomic_init(&session->total_allocated_bytes, 0);
protocol_session_set_bwlimit(session, global_bwlimit());
}
void protocol_session_set_max_alloc(ProtocolSession* session, unsigned long long max_alloc) {
if (!session)
session = bound_session ? bound_session : &legacy_io_session;
session->max_alloc = max_alloc;
}
static bool allocation_allowed(const ProtocolSession* session, size_t size) {
return (unsigned long long)size <= session->max_alloc;
}
static void* protocol_alloc_for_session(const ProtocolSession* session, size_t size) {
if (!allocation_allowed(session, size))
return NULL;
return malloc(size);
}
static void* protocol_realloc_for_session(const ProtocolSession* session, void* ptr, size_t size) {
if (!allocation_allowed(session, size))
return NULL;
return realloc(ptr, size);
}
void* protocol_alloc(size_t size) {
const ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
return protocol_alloc_for_session(session, size);
}
void* protocol_realloc(void* ptr, size_t size) {
const ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
return protocol_realloc_for_session(session, ptr, size);
} }
void protocol_session_bind(ProtocolSession* session) { void protocol_session_bind(ProtocolSession* session) {
bound_session = session; bound_session = session;
log_set_8_bit_output(session && session->eight_bit_output);
} }
void protocol_session_unbind(void) { void protocol_session_unbind(void) {
@@ -65,26 +130,48 @@ static void bw_mutex_init(void) {
mtx_init(&bw_mutex, mtx_plain); mtx_init(&bw_mutex, mtx_plain);
} }
static unsigned long long global_bwlimit(void) {
unsigned long long limit;
call_once(&bw_mutex_once, bw_mutex_init);
mtx_lock(&bw_mutex);
limit = io_bwlimit;
mtx_unlock(&bw_mutex);
return limit;
}
void io_set_bwlimit(unsigned long long bytes_per_sec) { void io_set_bwlimit(unsigned long long bytes_per_sec) {
call_once(&bw_mutex_once, bw_mutex_init); call_once(&bw_mutex_once, bw_mutex_init);
mtx_lock(&bw_mutex); mtx_lock(&bw_mutex);
io_bwlimit = bytes_per_sec; io_bwlimit =
bw_tokens = (long long)io_bwlimit; bytes_per_sec > (unsigned long long)LLONG_MAX ? (unsigned long long)LLONG_MAX : bytes_per_sec;
clock_gettime(CLOCK_MONOTONIC, &bw_last_refill);
mtx_unlock(&bw_mutex); mtx_unlock(&bw_mutex);
} }
void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long bytes_per_sec) { void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long bytes_per_sec) {
if (!session) if (!session)
return; return;
session->bwlimit = bytes_per_sec; session->bwlimit =
session->bw_tokens = (long long)bytes_per_sec; bytes_per_sec > (unsigned long long)LLONG_MAX ? (unsigned long long)LLONG_MAX : bytes_per_sec;
session->bw_tokens = (long long)session->bwlimit;
struct timespec now; struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now); clock_gettime(CLOCK_MONOTONIC, &now);
session->bw_last_refill_sec = now.tv_sec; session->bw_last_refill_sec = now.tv_sec;
session->bw_last_refill_nsec = now.tv_nsec; session->bw_last_refill_nsec = now.tv_nsec;
} }
void protocol_session_set_8_bit_output(ProtocolSession* session, bool enabled) {
if (!session)
return;
session->eight_bit_output = enabled;
if (session == bound_session)
log_set_8_bit_output(enabled);
}
void protocol_set_8_bit_output(bool enabled) {
ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
protocol_session_set_8_bit_output(session, enabled);
}
static void bw_throttle_session(ProtocolSession* session, size_t bytes_written) { static void bw_throttle_session(ProtocolSession* session, size_t bytes_written) {
if (session->bwlimit == 0) if (session->bwlimit == 0)
return; return;
@@ -126,32 +213,31 @@ SSL* io_get_ssl(void) {
return io_ssl; return io_ssl;
} }
static ProtocolSession* legacy_session(void) { static ProtocolSession* legacy_session(int read_fd, int write_fd) {
static __thread ProtocolSession session;
if (bound_session) if (bound_session)
return bound_session; return bound_session;
session.read_fd = io_read_fd; int target_read_fd = io_read_fd != -1 ? io_read_fd : read_fd;
session.write_fd = io_write_fd; int target_write_fd = io_write_fd != -1 ? io_write_fd : write_fd;
session.ssl = io_ssl; if (legacy_io_session.read_fd != target_read_fd ||
session.bwlimit = io_bwlimit; legacy_io_session.write_fd != target_write_fd) {
session.bw_tokens = (unsigned long long)(bw_tokens < 0 ? 0 : bw_tokens); legacy_io_session.read_fd = target_read_fd;
session.bw_last_refill_sec = bw_last_refill.tv_sec; legacy_io_session.write_fd = target_write_fd;
session.bw_last_refill_nsec = bw_last_refill.tv_nsec; atomic_store(&legacy_io_session.total_allocated_bytes, 0);
return &session; legacy_io_session.max_alloc = DEFAULT_MAX_ALLOC;
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
} else if (legacy_io_session.bwlimit != global_bwlimit()) {
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
}
legacy_io_session.ssl = io_ssl;
return &legacy_io_session;
} }
bool send_n_data(int file_descriptor, const void* data, size_t data_size) { bool send_n_data(int file_descriptor, const void* data, size_t data_size) {
ProtocolSession* session = legacy_session(); return protocol_send_n_data(legacy_session(-1, file_descriptor), data, data_size);
if (session->write_fd == -1)
session->write_fd = file_descriptor;
return protocol_send_n_data(session, data, data_size);
} }
bool receive_n_data(int file_descriptor, void* data, size_t data_size) { bool receive_n_data(int file_descriptor, void* data, size_t data_size) {
ProtocolSession* session = legacy_session(); return protocol_receive_n_data(legacy_session(file_descriptor, -1), data, data_size);
if (session->read_fd == -1)
session->read_fd = file_descriptor;
return protocol_receive_n_data(session, data, data_size);
} }
static int deadline_remaining_ms(const struct timespec* deadline) { static int deadline_remaining_ms(const struct timespec* deadline) {
@@ -166,7 +252,9 @@ static int deadline_remaining_ms(const struct timespec* deadline) {
} }
bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t data_size) { bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t data_size) {
log_message(LOG_LEVEL_DEBUG, " Sending n Data: %zu", data_size); if (!data && data_size != 0)
return false;
log_debug_message(LOG_DEBUG_IO, " Sending n Data: %zu", data_size);
if (!session) if (!session)
return false; return false;
int fd = session->write_fd; int fd = session->write_fd;
@@ -207,13 +295,15 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
} }
bw_throttle_session(session, (size_t)bytes_send); bw_throttle_session(session, (size_t)bytes_send);
total_bytes_send += bytes_send; total_bytes_send += bytes_send;
if (session->ssl)
wait_events = POLLOUT;
} }
log_message(LOG_LEVEL_DEBUG, " Send n Data: %zu", total_bytes_send); log_debug_message(LOG_DEBUG_IO, " Send n Data: %zu", total_bytes_send);
return true; return true;
} }
bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_size) { bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_size) {
log_message(LOG_LEVEL_DEBUG, " Receiving n Data: %zu", data_size); log_debug_message(LOG_DEBUG_IO, " Receiving n Data: %zu", data_size);
if (!session) if (!session)
return false; return false;
int fd = session->read_fd; int fd = session->read_fd;
@@ -225,20 +315,22 @@ bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_s
size_t total_bytes_received = 0; size_t total_bytes_received = 0;
short wait_events = POLLIN; short wait_events = POLLIN;
while (total_bytes_received < data_size) { while (total_bytes_received < data_size) {
struct pollfd pfd = {.fd = fd, .events = wait_events}; if (!session->ssl || SSL_pending(session->ssl) == 0) {
int poll_result = poll(&pfd, 1, deadline_remaining_ms(&deadline)); struct pollfd pfd = {.fd = fd, .events = wait_events};
if (poll_result == 0) { int poll_result = poll(&pfd, 1, deadline_remaining_ms(&deadline));
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", RECEIVE_TIMEOUT_SEC); if (poll_result == 0) {
return false; log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", RECEIVE_TIMEOUT_SEC);
return false;
}
if (poll_result < 0) {
if (errno == EINTR)
continue;
return false;
}
/* POLLHUP may accompany the final readable bytes on pipes/sockets. */
if (pfd.revents & (POLLERR | POLLNVAL))
return false;
} }
if (poll_result < 0) {
if (errno == EINTR)
continue;
return false;
}
/* POLLHUP may accompany the final readable bytes on pipes/sockets. */
if (pfd.revents & (POLLERR | POLLNVAL))
return false;
ssize_t bytes_received; ssize_t bytes_received;
if (session->ssl) if (session->ssl)
@@ -261,9 +353,11 @@ bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_s
log_message(LOG_LEVEL_ERROR, "Could not receive bytes"); log_message(LOG_LEVEL_ERROR, "Could not receive bytes");
return false; return false;
} }
total_bytes_received += bytes_received; total_bytes_received += (size_t)bytes_received;
if (session->ssl)
wait_events = POLLIN;
} }
log_message(LOG_LEVEL_DEBUG, " Received n Data: %zu", total_bytes_received); log_debug_message(LOG_DEBUG_IO, " Received n Data: %zu", total_bytes_received);
return true; return true;
} }
@@ -304,7 +398,7 @@ bool protocol_send_str(ProtocolSession* session, const char* data) {
return false; return false;
if (!protocol_send_n_data(session, data, size)) if (!protocol_send_n_data(session, data, size))
return false; return false;
log_message(LOG_LEVEL_DEBUG, "Send String: %s", data); log_debug_message(LOG_DEBUG_PROTO, "Send String: %s", data);
return true; return true;
} }
@@ -312,125 +406,138 @@ char* protocol_receive_str(ProtocolSession* session) {
size_t size; size_t size;
if (!protocol_receive_n_data(session, &size, sizeof(size_t))) if (!protocol_receive_n_data(session, &size, sizeof(size_t)))
return NULL; return NULL;
if (size > MAX_STRING_SIZE || size > SIZE_MAX - 1 || if (size > MAX_STRING_SIZE || size > SIZE_MAX - 1) {
size + 1 > MAX_CONNECTION_MEMORY - session->total_allocated_bytes) {
log_message(LOG_LEVEL_ERROR, "String size %zu exceeds maximum %llu", size, log_message(LOG_LEVEL_ERROR, "String size %zu exceeds maximum %llu", size,
(unsigned long long)MAX_STRING_SIZE); (unsigned long long)MAX_STRING_SIZE);
return NULL; return NULL;
} }
char* data = (char*)malloc(size + 1); char* data = (char*)protocol_alloc_for_session(session, size + 1);
if (data == NULL) if (data == NULL)
return NULL; return NULL;
if (!protocol_receive_n_data(session, data, size)) { if (!protocol_receive_n_data(session, data, size)) {
free(data); free(data);
return NULL; return NULL;
} }
if (memchr(data, '\0', size) != NULL) {
free(data);
log_message(LOG_LEVEL_ERROR, "Received string contains an embedded NUL");
return NULL;
}
data[size] = '\0'; data[size] = '\0';
session->total_allocated_bytes += size + 1; log_debug_message(LOG_DEBUG_PROTO, "Received String: %s", data);
log_message(LOG_LEVEL_DEBUG, "Received String: %s", data);
return data; return data;
} }
bool protocol_send_data(ProtocolSession* session, const Data* data) { bool protocol_send_data(ProtocolSession* session, const Data* data) {
if (!data || (!data->data && data->size != 0))
return false;
if (!session)
return false;
unsigned long long data_size = data->size; unsigned long long data_size = data->size;
if (!protocol_send_n_data(session, &data_size, sizeof(unsigned long long))) if (!protocol_send_n_data(session, &data_size, sizeof(unsigned long long)))
return false; return false;
if (!protocol_send_n_data(session, data->data, data_size)) if (!protocol_send_n_data(session, data->data, data_size))
return false; return false;
log_message(LOG_LEVEL_DEBUG, "Send %lld data", data_size); log_debug_message(LOG_DEBUG_PROTO, "Send %lld data", data_size);
return true; return true;
} }
Data* protocol_receive_data(ProtocolSession* session) { Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long maximum_size) {
if (!session)
return NULL;
unsigned long long size = 0; unsigned long long size = 0;
if (!protocol_receive_n_data(session, &size, sizeof(unsigned long long))) if (!protocol_receive_n_data(session, &size, sizeof(unsigned long long)))
return NULL; return NULL;
if (size > MAX_DATA_PAYLOAD_SIZE) { if (size > MAX_DATA_PAYLOAD_SIZE || size > maximum_size) {
log_message(LOG_LEVEL_ERROR, "Data size %llu exceeds maximum %llu", size, log_message(LOG_LEVEL_ERROR, "Data size %llu exceeds maximum %llu", size,
(unsigned long long)MAX_DATA_PAYLOAD_SIZE); (unsigned long long)MAX_DATA_PAYLOAD_SIZE);
return NULL; return NULL;
} }
if (size > SIZE_MAX)
return NULL;
size_t allocation_size = size == 0 ? 1 : (size_t)size; size_t allocation_size = size == 0 ? 1 : (size_t)size;
if (allocation_size > MAX_CONNECTION_MEMORY - session->total_allocated_bytes) { if (!protocol_reserve_memory(session, allocation_size)) {
log_message(LOG_LEVEL_ERROR, "Per-connection memory limit exceeded (%llu + %llu > %llu)", log_message(LOG_LEVEL_ERROR, "Per-connection memory limit exceeded (%llu + %llu > %llu)",
(unsigned long long)session->total_allocated_bytes, size, (unsigned long long)atomic_load(&session->total_allocated_bytes), size,
(unsigned long long)MAX_CONNECTION_MEMORY); (unsigned long long)MAX_CONNECTION_MEMORY);
return NULL; return NULL;
} }
void* data = malloc(allocation_size); void* data = protocol_alloc_for_session(session, allocation_size);
if (data == NULL) if (data == NULL) {
protocol_release_memory_for_session(session, allocation_size);
return NULL; return NULL;
}
if (!protocol_receive_n_data(session, data, (size_t)size)) { if (!protocol_receive_n_data(session, data, (size_t)size)) {
free(data); free(data);
protocol_release_memory_for_session(session, allocation_size);
return NULL; return NULL;
} }
session->total_allocated_bytes += allocation_size; log_debug_message(LOG_DEBUG_PROTO, "Received %lld data", size);
log_message(LOG_LEVEL_DEBUG, "Received %lld data", size);
Data* result = data_create(data, (size_t)size); Data* result = data_create(data, (size_t)size);
if (!result) { if (!result) {
free(data); protocol_release_memory_for_session(session, allocation_size);
session->total_allocated_bytes -= allocation_size; return NULL;
} }
result->protocol_charge = allocation_size;
return result; return result;
} }
Data* protocol_receive_data(ProtocolSession* session) {
return protocol_receive_data_limited(session, MAX_DATA_PAYLOAD_SIZE);
}
bool protocol_send_int(ProtocolSession* session, int data) { bool protocol_send_int(ProtocolSession* session, int data) {
if (!protocol_send_n_data(session, &data, sizeof(int))) if (!protocol_send_n_data(session, &data, sizeof(int)))
return false; return false;
log_message(LOG_LEVEL_DEBUG, "Send Int: %d", data); log_debug_message(LOG_DEBUG_PROTO, "Send Int: %d", data);
return true; return true;
} }
bool protocol_receive_int(ProtocolSession* session, int* data) { bool protocol_receive_int(ProtocolSession* session, int* data) {
if (!protocol_receive_n_data(session, data, sizeof(int))) if (!protocol_receive_n_data(session, data, sizeof(int)))
return false; return false;
log_message(LOG_LEVEL_DEBUG, "Received Int: %d", *data); log_debug_message(LOG_DEBUG_PROTO, "Received Int: %d", *data);
return true; return true;
} }
bool protocol_send_status(ProtocolSession* session, Status status) { bool protocol_send_status(ProtocolSession* session, Status status) {
if (!protocol_send_n_data(session, &status, sizeof(Status))) if (!protocol_send_n_data(session, &status, sizeof(Status)))
return false; return false;
log_message(LOG_LEVEL_DEBUG, "Send Status: %s", status_to_string(status)); log_debug_message(LOG_DEBUG_PROTO, "Send Status: %s", status_to_string(status));
return true; return true;
} }
bool protocol_receive_status(ProtocolSession* session, Status* status) { bool protocol_receive_status(ProtocolSession* session, Status* status) {
if (!protocol_receive_n_data(session, status, sizeof(Status))) if (!protocol_receive_n_data(session, status, sizeof(Status)))
return false; return false;
log_message(LOG_LEVEL_DEBUG, "Received Status: %s", status_to_string(*status)); log_debug_message(LOG_DEBUG_PROTO, "Received Status: %s", status_to_string(*status));
return true; return true;
} }
bool send_str(int fd, const char* data) { bool send_str(int fd, const char* data) {
(void)fd; return protocol_send_str(legacy_session(-1, fd), data);
return protocol_send_str(legacy_session(), data);
} }
char* receive_str(int fd) { char* receive_str(int fd) {
(void)fd; return protocol_receive_str(legacy_session(fd, -1));
return protocol_receive_str(legacy_session());
} }
bool send_data(int fd, const Data* data) { bool send_data(int fd, const Data* data) {
(void)fd; return protocol_send_data(legacy_session(-1, fd), data);
return protocol_send_data(legacy_session(), data);
} }
Data* receive_data(int fd) { Data* receive_data(int fd) {
(void)fd; return protocol_receive_data_limited(legacy_session(fd, -1), MAX_DATA_PAYLOAD_SIZE);
return protocol_receive_data(legacy_session()); }
Data* receive_data_limited(int fd, unsigned long long maximum_size) {
return protocol_receive_data_limited(legacy_session(fd, -1), maximum_size);
} }
bool send_int(int fd, int data) { bool send_int(int fd, int data) {
(void)fd; return protocol_send_int(legacy_session(-1, fd), data);
return protocol_send_int(legacy_session(), data);
} }
bool receive_int(int fd, int* data) { bool receive_int(int fd, int* data) {
(void)fd; return protocol_receive_int(legacy_session(fd, -1), data);
return protocol_receive_int(legacy_session(), data);
} }
bool send_status(int fd, Status status) { bool send_status(int fd, Status status) {
(void)fd; return protocol_send_status(legacy_session(-1, fd), status);
return protocol_send_status(legacy_session(), status);
} }
bool receive_status(int fd, Status* status) { bool receive_status(int fd, Status* status) {
(void)fd; return protocol_receive_status(legacy_session(fd, -1), status);
return protocol_receive_status(legacy_session(), status);
} }
+28 -3
View File
@@ -4,18 +4,34 @@
#include "data.h" #include "data.h"
#include <stdbool.h> #include <stdbool.h>
#include <stddef.h> #include <stddef.h>
#include <stdatomic.h>
/* Maximum allowed string size for receive_str (64 KB) */ /* Maximum allowed string size for receive_str (64 KB) */
#define MAX_STRING_SIZE (64 * 1024) #define MAX_STRING_SIZE (64 * 1024)
/* Maximum allowed data payload size for receive_data (100 MB) */ /* Maximum uncompressed file payload accepted by the receiver's whole-file
#define MAX_DATA_PAYLOAD_SIZE (100ULL * 1024 * 1024) * paths. A single whole file is charged against the per-connection memory
* reservation (MAX_CONNECTION_MEMORY) and against the server allocation
* ceiling (MAX_SERVER_ALLOC), so this mirrors those 256 MB bounds rather than
* the older 64 MB chunk-era cap. Chunk-serialized payloads keep their own
* 64 MB cap (MAX_CHUNK_SIZE). */
#define MAX_RECEIVE_WHOLE_FILE_SIZE (256ULL * 1024 * 1024)
/* Maximum allowed data payload size for receive_data (whole-file bound) */
#define MAX_DATA_PAYLOAD_SIZE MAX_RECEIVE_WHOLE_FILE_SIZE
/* Maximum chunk size (64 MB) — prevents unbounded allocation from the wire */ /* Maximum chunk size (64 MB) — prevents unbounded allocation from the wire */
#define MAX_CHUNK_SIZE (64ULL * 1024 * 1024) #define MAX_CHUNK_SIZE (64ULL * 1024 * 1024)
#define MAX_MANIFEST_ENTRIES (1024 * 1024) #define MAX_MANIFEST_ENTRIES (1024 * 1024)
/* Aggregate bytes retained by one received deletion manifest. */ /* Aggregate bytes retained by one received deletion manifest. */
#define MAX_MANIFEST_BYTES (16ULL * 1024 * 1024) #define MAX_MANIFEST_BYTES (16ULL * 1024 * 1024)
#define DEFAULT_MAX_ALLOC (1ULL * 1024 * 1024 * 1024)
/* Server policy ceiling for a client-provided allocation limit. */
#define MAX_SERVER_ALLOC (256ULL * 1024 * 1024)
/* Bounded cumulative per-connection receive budget. In-flight wire buffers,
decompression buffers and queued (not yet written) file payloads for a
connection must stay within this ceiling. */
#define MAX_CONNECTION_MEMORY (256ULL * 1024 * 1024)
typedef struct ssl_st SSL; typedef struct ssl_st SSL;
@@ -33,7 +49,9 @@ typedef struct ProtocolSession {
long long bw_tokens; long long bw_tokens;
long long bw_last_refill_sec; long long bw_last_refill_sec;
long bw_last_refill_nsec; long bw_last_refill_nsec;
unsigned long long total_allocated_bytes; atomic_ullong total_allocated_bytes;
bool eight_bit_output;
unsigned long long max_alloc;
} ProtocolSession; } ProtocolSession;
typedef int Status; typedef int Status;
@@ -63,12 +81,18 @@ void protocol_session_bind(ProtocolSession* session);
void protocol_session_unbind(void); void protocol_session_unbind(void);
void protocol_session_set_ssl(ProtocolSession* session, SSL* ssl); void protocol_session_set_ssl(ProtocolSession* session, SSL* ssl);
void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long bytes_per_sec); void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long bytes_per_sec);
void protocol_session_set_max_alloc(ProtocolSession* session, unsigned long long max_alloc);
void* protocol_alloc(size_t size);
void* protocol_realloc(void* ptr, size_t size);
void protocol_session_set_8_bit_output(ProtocolSession* session, bool enabled);
void protocol_set_8_bit_output(bool enabled);
bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t data_size); bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t data_size);
bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_size); bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_size);
bool protocol_send_str(ProtocolSession* session, const char* data); bool protocol_send_str(ProtocolSession* session, const char* data);
char* protocol_receive_str(ProtocolSession* session); char* protocol_receive_str(ProtocolSession* session);
bool protocol_send_data(ProtocolSession* session, const Data* data); bool protocol_send_data(ProtocolSession* session, const Data* data);
Data* protocol_receive_data(ProtocolSession* session); Data* protocol_receive_data(ProtocolSession* session);
Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long maximum_size);
bool protocol_send_int(ProtocolSession* session, int data); bool protocol_send_int(ProtocolSession* session, int data);
bool protocol_receive_int(ProtocolSession* session, int* data); bool protocol_receive_int(ProtocolSession* session, int* data);
bool protocol_send_status(ProtocolSession* session, Status status); bool protocol_send_status(ProtocolSession* session, Status status);
@@ -80,6 +104,7 @@ bool send_str(int file_descriptor, const char* data);
char* receive_str(int file_descriptor); char* receive_str(int file_descriptor);
bool send_data(int file_descriptor, const Data* data); bool send_data(int file_descriptor, const Data* data);
Data* receive_data(int file_descriptor); Data* receive_data(int file_descriptor);
Data* receive_data_limited(int file_descriptor, unsigned long long maximum_size);
bool send_int(int file_descriptor, int data); bool send_int(int file_descriptor, int data);
bool receive_int(int file_descriptor, int* data); bool receive_int(int file_descriptor, int* data);
bool send_status(int file_descriptor, Status status); bool send_status(int file_descriptor, Status status);
+4 -3
View File
@@ -1,3 +1,4 @@
#include "log.h"
#include <stdbool.h> #include <stdbool.h>
#include <limits.h> #include <limits.h>
#include <stdio.h> #include <stdio.h>
@@ -13,7 +14,7 @@ Queue* queue_create(int capacity, void (*destroyer)(void* item)) {
Queue* queue = (Queue*)malloc(sizeof(Queue)); Queue* queue = (Queue*)malloc(sizeof(Queue));
if (queue == NULL) { if (queue == NULL) {
perror("ERROR: Could not allocate memory for queue structure"); log_perror("ERROR: Could not allocate memory for queue structure");
return NULL; return NULL;
} }
@@ -72,7 +73,7 @@ static bool queue_double_capacity(Queue* queue) {
new_capacity = 100; new_capacity = 100;
void** new_items = malloc(new_capacity * sizeof(void*)); void** new_items = malloc(new_capacity * sizeof(void*));
if (new_items == NULL) { if (new_items == NULL) {
perror("ERROR: Could not allocate memory for doubling capacity of queue."); log_perror("ERROR: Could not allocate memory for doubling capacity of queue.");
return false; return false;
} }
for (int i = 0; i < queue->size; i++) for (int i = 0; i < queue->size; i++)
@@ -127,7 +128,7 @@ bool queue_enqueue_multithreaded_cancel(Queue* queue, void* item, mtx_t* mutex,
void* queue_dequeue(Queue* queue) { void* queue_dequeue(Queue* queue) {
if (queue == NULL || queue_is_empty(queue)) { if (queue == NULL || queue_is_empty(queue)) {
perror("ERROR: Could not dequeue from null or empty queue."); log_perror("ERROR: Could not dequeue from null or empty queue.");
return NULL; return NULL;
} }
+78 -19
View File
@@ -1,7 +1,9 @@
#include "log.h"
#include "transport_ssh.h" #include "transport_ssh.h"
#include "utils.h" #include "utils.h"
#include <fcntl.h> #include <fcntl.h>
#include <stdio.h> #include <stdio.h>
#include <stdint.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <sys/socket.h> #include <sys/socket.h>
@@ -14,6 +16,12 @@ typedef struct {
char* remote_path; char* remote_path;
} RemoteDest; } RemoteDest;
static void ssh_child_setup_failed(int status_fd) {
ssize_t wret = write(status_fd, "x", 1);
(void)wret;
_exit(1);
}
static void remote_dest_destroy(RemoteDest* r) { static void remote_dest_destroy(RemoteDest* r) {
free(r->user); free(r->user);
free(r->host); free(r->host);
@@ -67,16 +75,63 @@ static int parse_remote_dest(const char* dest, RemoteDest* r) {
return 0; return 0;
} }
Client* client_connect_ssh(const char* destination, int port, const char* server_path) { char* ssh_build_remote_command(const char* server_path, bool old_args) {
const char* path = server_path ? server_path : "fastsync-server";
const char* suffix = " --stdio";
size_t path_len = strlen(path);
size_t suffix_len = strlen(suffix);
if (old_args) {
if (path_len > SIZE_MAX - suffix_len - 1)
return NULL;
char* command = malloc(path_len + suffix_len + 1);
if (!command)
return NULL;
memcpy(command, path, path_len);
memcpy(command + path_len, suffix, suffix_len + 1);
return command;
}
/* Quote the executable as one remote-shell word. This is the default safety boundary. */
size_t quote_count = 0;
for (const char* p = path; *p; p++)
if (*p == '\'')
quote_count++;
if (path_len > SIZE_MAX - suffix_len - 4 ||
quote_count > (SIZE_MAX - path_len - suffix_len - 4) / 4)
return NULL;
size_t command_len = path_len + quote_count * 4 + suffix_len + 4;
char* command = malloc(command_len + 1);
if (!command)
return NULL;
char* out = command;
*out++ = '\'';
for (const char* p = path; *p; p++) {
if (*p == '\'') {
memcpy(out, "'\\''", 4);
out += 4;
} else {
*out++ = *p;
}
}
*out++ = '\'';
memcpy(out, suffix, suffix_len + 1);
return command;
}
Client* client_connect_ssh(const char* destination, int port, const char* server_path,
bool old_args) {
RemoteDest r; RemoteDest r;
if (parse_remote_dest(destination, &r) != 0) { if (parse_remote_dest(destination, &r) != 0) {
fprintf(stderr, "Invalid remote destination: %s\n", destination); char* escaped = output_escape(destination, false);
fprintf(stderr, "Invalid remote destination: %s\n", escaped ? escaped : "<allocation failed>");
free(escaped);
return NULL; return NULL;
} }
int sv[2]; int sv[2];
if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0) { if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0) {
perror("socketpair failed"); log_perror("socketpair failed");
remote_dest_destroy(&r); remote_dest_destroy(&r);
return NULL; return NULL;
} }
@@ -89,7 +144,7 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
int exec_pipe[2]; int exec_pipe[2];
if (pipe(exec_pipe) < 0) { if (pipe(exec_pipe) < 0) {
perror("pipe failed"); log_perror("pipe failed");
close(sv[0]); close(sv[0]);
close(sv[1]); close(sv[1]);
remote_dest_destroy(&r); remote_dest_destroy(&r);
@@ -98,7 +153,7 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
pid_t pid = fork(); pid_t pid = fork();
if (pid < 0) { if (pid < 0) {
perror("fork failed"); log_perror("fork failed");
close(sv[0]); close(sv[0]);
close(sv[1]); close(sv[1]);
close(exec_pipe[0]); close(exec_pipe[0]);
@@ -110,11 +165,12 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
if (pid == 0) { if (pid == 0) {
close(sv[0]); close(sv[0]);
close(exec_pipe[0]); close(exec_pipe[0]);
fcntl(exec_pipe[1], F_SETFD, FD_CLOEXEC); if (fcntl(exec_pipe[1], F_SETFD, FD_CLOEXEC) < 0)
if (sv[1] != STDIN_FILENO) ssh_child_setup_failed(exec_pipe[1]);
dup2(sv[1], STDIN_FILENO); if (sv[1] != STDIN_FILENO && dup2(sv[1], STDIN_FILENO) < 0)
if (sv[1] != STDOUT_FILENO) ssh_child_setup_failed(exec_pipe[1]);
dup2(sv[1], STDOUT_FILENO); if (sv[1] != STDOUT_FILENO && dup2(sv[1], STDOUT_FILENO) < 0)
ssh_child_setup_failed(exec_pipe[1]);
if (sv[1] > 1) if (sv[1] > 1)
close(sv[1]); close(sv[1]);
@@ -125,7 +181,7 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
ssh_user_len = strlen(r.host) + 1; ssh_user_len = strlen(r.host) + 1;
char* ssh_user = malloc(ssh_user_len); char* ssh_user = malloc(ssh_user_len);
if (!ssh_user) if (!ssh_user)
_exit(1); ssh_child_setup_failed(exec_pipe[1]);
if (r.user && r.user[0] != '\0') if (r.user && r.user[0] != '\0')
snprintf(ssh_user, ssh_user_len, "%s@%s", r.user, r.host); snprintf(ssh_user, ssh_user_len, "%s@%s", r.user, r.host);
else else
@@ -134,6 +190,9 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
char* ssh_argv[16]; char* ssh_argv[16];
int ac = 0; int ac = 0;
char port_str[16]; char port_str[16];
char* remote_command = ssh_build_remote_command(server_path, old_args);
if (!remote_command)
ssh_child_setup_failed(exec_pipe[1]);
ssh_argv[ac++] = "ssh"; ssh_argv[ac++] = "ssh";
ssh_argv[ac++] = "-o"; ssh_argv[ac++] = "-o";
ssh_argv[ac++] = "Compression=no"; ssh_argv[ac++] = "Compression=no";
@@ -147,14 +206,11 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
ssh_argv[ac++] = port_str; ssh_argv[ac++] = port_str;
} }
ssh_argv[ac++] = ssh_user; ssh_argv[ac++] = ssh_user;
ssh_argv[ac++] = (char*)(server_path ? server_path : "fastsync-server"); ssh_argv[ac++] = remote_command;
ssh_argv[ac++] = "--stdio";
ssh_argv[ac] = NULL; ssh_argv[ac] = NULL;
execvp("ssh", ssh_argv); execvp("ssh", ssh_argv);
perror("exec of ssh failed"); log_perror("exec of ssh failed");
ssize_t wret = write(exec_pipe[1], "x", 1); ssh_child_setup_failed(exec_pipe[1]);
(void)wret;
_exit(1);
} }
close(sv[1]); close(sv[1]);
@@ -164,12 +220,15 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
ssize_t n = read(exec_pipe[0], &exec_status, 1); ssize_t n = read(exec_pipe[0], &exec_status, 1);
close(exec_pipe[0]); close(exec_pipe[0]);
if (n > 0) { if (n != 0) {
close(sv[0]); close(sv[0]);
waitpid(pid, NULL, 0); waitpid(pid, NULL, 0);
remote_dest_destroy(&r); remote_dest_destroy(&r);
const char* path = server_path ? server_path : "fastsync-server";
char* escaped = output_escape(path, false);
fprintf(stderr, "Error: could not launch '%s --stdio' on remote\n", fprintf(stderr, "Error: could not launch '%s --stdio' on remote\n",
server_path ? server_path : "fastsync-server"); escaped ? escaped : "<allocation failed>");
free(escaped);
return NULL; return NULL;
} }
+3 -1
View File
@@ -3,6 +3,8 @@
#include "transport_tcp.h" #include "transport_tcp.h"
Client* client_connect_ssh(const char* destination, int port, const char* server_path); Client* client_connect_ssh(const char* destination, int port, const char* server_path,
bool old_args);
char* ssh_build_remote_command(const char* server_path, bool old_args);
#endif #endif
+16 -8
View File
@@ -1,6 +1,7 @@
#include "transport_tcp.h" #include "transport_tcp.h"
#include "log.h" #include "log.h"
#include "protocol.h" #include "protocol.h"
#include "utils.h"
#include <arpa/inet.h> #include <arpa/inet.h>
#include <errno.h> #include <errno.h>
#include <netdb.h> #include <netdb.h>
@@ -30,20 +31,20 @@ static void sigchld_handler(int sig) {
Server* server_create(int port) { Server* server_create(int port) {
Server* server = (Server*)malloc(sizeof(Server)); Server* server = (Server*)malloc(sizeof(Server));
if (server == NULL) { if (server == NULL) {
perror("Could not allocate space for Server"); log_perror("Could not allocate space for Server");
return NULL; return NULL;
} }
int file_descriptor = socket(AF_INET, SOCK_STREAM, 0); int file_descriptor = socket(AF_INET, SOCK_STREAM, 0);
if (file_descriptor < 0) { if (file_descriptor < 0) {
perror("Could not create Socket!"); log_perror("Could not create Socket!");
free(server); free(server);
return NULL; return NULL;
} }
server->file_descriptor = file_descriptor; server->file_descriptor = file_descriptor;
int opt = 1; int opt = 1;
if (setsockopt(server->file_descriptor, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt))) { if (setsockopt(server->file_descriptor, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt))) {
perror("Error setting a socket option!"); log_perror("Error setting a socket option!");
close(server->file_descriptor); close(server->file_descriptor);
free(server); free(server);
return NULL; return NULL;
@@ -59,7 +60,7 @@ Server* server_create(int port) {
if (bind(server->file_descriptor, (struct sockaddr*)&server->address, server->address_length) < if (bind(server->file_descriptor, (struct sockaddr*)&server->address, server->address_length) <
0) { 0) {
perror("Could not bind server"); log_perror("Could not bind server");
close(server->file_descriptor); close(server->file_descriptor);
free(server); free(server);
return NULL; return NULL;
@@ -83,7 +84,7 @@ void server_delete(Server** server) {
static void accept_loop(Server* server, void (*child_fn)(int, void*), void* child_ctx, static void accept_loop(Server* server, void (*child_fn)(int, void*), void* child_ctx,
const char* log_fmt) { const char* log_fmt) {
if (listen(server->file_descriptor, SOMAXCONN) < 0) { if (listen(server->file_descriptor, SOMAXCONN) < 0) {
perror("Could not listen on port!"); log_perror("Could not listen on port!");
return; return;
} }
signal(SIGCHLD, sigchld_handler); signal(SIGCHLD, sigchld_handler);
@@ -92,7 +93,7 @@ static void accept_loop(Server* server, void (*child_fn)(int, void*), void* chil
socklen_t client_len = sizeof(client_addr); socklen_t client_len = sizeof(client_addr);
int fd = accept(server->file_descriptor, (struct sockaddr*)&client_addr, &client_len); int fd = accept(server->file_descriptor, (struct sockaddr*)&client_addr, &client_len);
if (fd < 0) { if (fd < 0) {
perror("Could not accept the connection"); log_perror("Could not accept the connection");
continue; continue;
} }
tcp_apply_socket_timeout(fd); tcp_apply_socket_timeout(fd);
@@ -151,6 +152,10 @@ int tcp_get_contimeout_sec(void) {
return g_contimeout_sec; return g_contimeout_sec;
} }
int tcp_get_timeout_sec(void) {
return g_timeout_sec;
}
static void tcp_apply_socket_timeout(int fd) { static void tcp_apply_socket_timeout(int fd) {
struct timeval tv; struct timeval tv;
tv.tv_sec = g_timeout_sec; tv.tv_sec = g_timeout_sec;
@@ -186,7 +191,10 @@ bool tcp_connect_socket(Client* client, char* host, int port) {
int err = getaddrinfo(host, port_str, &hints, &result); int err = getaddrinfo(host, port_str, &hints, &result);
if (err != 0 || result == NULL) { if (err != 0 || result == NULL) {
fprintf(stderr, "Could not resolve host: %s (%s)\n", host, gai_strerror(err)); char* escaped_host = output_escape(host, false);
fprintf(stderr, "Could not resolve host: %s (%s)\n",
escaped_host ? escaped_host : "<allocation failed>", gai_strerror(err));
free(escaped_host);
return false; return false;
} }
@@ -218,7 +226,7 @@ bool tcp_connect_socket(Client* client, char* host, int port) {
freeaddrinfo(result); freeaddrinfo(result);
if (!connected) { if (!connected) {
perror("Could not connect to Server!"); log_perror("Could not connect to Server!");
return false; return false;
} }
+1
View File
@@ -35,5 +35,6 @@ void client_disconnect(Client* client);
void client_delete(Client* client); void client_delete(Client* client);
void tcp_set_timeouts(int timeout_sec, int contimeout_sec); void tcp_set_timeouts(int timeout_sec, int contimeout_sec);
int tcp_get_contimeout_sec(void); int tcp_get_contimeout_sec(void);
int tcp_get_timeout_sec(void);
#endif #endif
+61 -10
View File
@@ -2,6 +2,7 @@
#include "log.h" #include "log.h"
#include "protocol.h" #include "protocol.h"
#include "transport_tcp.h" #include "transport_tcp.h"
#include "utils.h"
#include <arpa/inet.h> #include <arpa/inet.h>
#include <openssl/err.h> #include <openssl/err.h>
#include <openssl/ssl.h> #include <openssl/ssl.h>
@@ -9,6 +10,10 @@
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h> #include <unistd.h>
bool tls_global_init(void) { bool tls_global_init(void) {
@@ -31,6 +36,10 @@ static void log_ssl_errors(void) {
static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key, static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key,
const char* ca_path) { const char* ca_path) {
if (!is_server && !ca_path) {
log_message(LOG_LEVEL_ERROR, "TLS clients require a CA certificate path");
return NULL;
}
const SSL_METHOD* method = is_server ? TLS_server_method() : TLS_client_method(); const SSL_METHOD* method = is_server ? TLS_server_method() : TLS_client_method();
SSL_CTX* ctx = SSL_CTX_new(method); SSL_CTX* ctx = SSL_CTX_new(method);
if (!ctx) { if (!ctx) {
@@ -39,17 +48,37 @@ static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key
return NULL; return NULL;
} }
SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION); if (SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION) != 1) {
SSL_CTX_free(ctx);
return NULL;
}
if (SSL_CTX_set_cipher_list(ctx, "HIGH:!aNULL:!eNULL:!MD5:!RC4:!3DES") != 1) {
SSL_CTX_free(ctx);
return NULL;
}
if (cert && key) { if (cert && key) {
struct stat key_stat;
if (stat(key, &key_stat) != 0 || !S_ISREG(key_stat.st_mode) || key_stat.st_uid != geteuid() ||
(key_stat.st_mode & (S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH))) {
log_message(LOG_LEVEL_ERROR, "TLS private key must be owned by the current user and private");
SSL_CTX_free(ctx);
return NULL;
}
if (SSL_CTX_use_certificate_file(ctx, cert, SSL_FILETYPE_PEM) <= 0) { if (SSL_CTX_use_certificate_file(ctx, cert, SSL_FILETYPE_PEM) <= 0) {
log_message(LOG_LEVEL_ERROR, "Failed to load certificate: %s", cert); char* escaped = output_escape(cert, false);
log_message(LOG_LEVEL_ERROR, "Failed to load certificate: %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
log_ssl_errors(); log_ssl_errors();
SSL_CTX_free(ctx); SSL_CTX_free(ctx);
return NULL; return NULL;
} }
if (SSL_CTX_use_PrivateKey_file(ctx, key, SSL_FILETYPE_PEM) <= 0) { if (SSL_CTX_use_PrivateKey_file(ctx, key, SSL_FILETYPE_PEM) <= 0) {
log_message(LOG_LEVEL_ERROR, "Failed to load private key: %s", key); char* escaped = output_escape(key, false);
log_message(LOG_LEVEL_ERROR, "Failed to load private key: %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
log_ssl_errors(); log_ssl_errors();
SSL_CTX_free(ctx); SSL_CTX_free(ctx);
return NULL; return NULL;
@@ -63,7 +92,10 @@ static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key
if (ca_path) { if (ca_path) {
if (!SSL_CTX_load_verify_locations(ctx, ca_path, NULL)) { if (!SSL_CTX_load_verify_locations(ctx, ca_path, NULL)) {
log_message(LOG_LEVEL_ERROR, "Failed to load CA: %s", ca_path); char* escaped = output_escape(ca_path, false);
log_message(LOG_LEVEL_ERROR, "Failed to load CA: %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
log_ssl_errors(); log_ssl_errors();
SSL_CTX_free(ctx); SSL_CTX_free(ctx);
return NULL; return NULL;
@@ -83,15 +115,22 @@ static SSL* wrap_fd_with_ssl(int fd, SSL_CTX* ctx, bool is_server, const char* h
log_message(LOG_LEVEL_ERROR, "Failed to create SSL object"); log_message(LOG_LEVEL_ERROR, "Failed to create SSL object");
return NULL; return NULL;
} }
SSL_set_fd(ssl, fd); if (SSL_set_fd(ssl, fd) != 1) {
SSL_free(ssl);
return NULL;
}
// Enable hostname verification for client connections when a hostname is provided. // Enable hostname verification for client connections when a hostname is provided.
// Must be done before SSL_connect to take effect during the handshake. // Must be done before SSL_connect to take effect during the handshake.
if (!is_server && hostname) { if (!is_server && hostname) {
SSL_set1_host(ssl, hostname); if (SSL_set1_host(ssl, hostname) != 1) {
SSL_free(ssl);
return NULL;
}
} }
// Retry SSL_accept/SSL_connect on WANT_READ/WANT_WRITE (non-blocking handshake) // Retry SSL_accept/SSL_connect on WANT_READ/WANT_WRITE (non-blocking handshake)
time_t deadline = time(NULL) + (is_server ? tcp_get_timeout_sec() : tcp_get_contimeout_sec());
int ret; int ret;
do { do {
if (is_server) if (is_server)
@@ -101,7 +140,8 @@ static SSL* wrap_fd_with_ssl(int fd, SSL_CTX* ctx, bool is_server, const char* h
if (ret <= 0) { if (ret <= 0) {
int ssl_err = SSL_get_error(ssl, ret); int ssl_err = SSL_get_error(ssl, ret);
if (ssl_err == SSL_ERROR_WANT_READ || ssl_err == SSL_ERROR_WANT_WRITE) if ((ssl_err == SSL_ERROR_WANT_READ || ssl_err == SSL_ERROR_WANT_WRITE) &&
time(NULL) < deadline)
continue; continue;
log_message(LOG_LEVEL_ERROR, "SSL %s failed", is_server ? "accept" : "connect"); log_message(LOG_LEVEL_ERROR, "SSL %s failed", is_server ? "accept" : "connect");
log_ssl_errors(); log_ssl_errors();
@@ -129,8 +169,10 @@ struct tls_child_ctx {
static void tls_child_fn(int fd, void* arg) { static void tls_child_fn(int fd, void* arg) {
struct tls_child_ctx* ctx = (struct tls_child_ctx*)arg; struct tls_child_ctx* ctx = (struct tls_child_ctx*)arg;
SSL* ssl = wrap_fd_with_ssl(fd, ctx->ssl_ctx, true, NULL); SSL* ssl = wrap_fd_with_ssl(fd, ctx->ssl_ctx, true, NULL);
if (!ssl) if (!ssl) {
io_set_ssl(NULL);
return; return;
}
io_set_ssl(ssl); io_set_ssl(ssl);
ctx->handler(fd); ctx->handler(fd);
SSL_shutdown(ssl); SSL_shutdown(ssl);
@@ -146,12 +188,19 @@ bool server_listen_tls(Server* server, void (*handler)(int file_descriptor)) {
bool client_connect_tls(Client* client, char* host, int port, const char* cert_path, bool client_connect_tls(Client* client, char* host, int port, const char* cert_path,
const char* key_path, const char* ca_path) { const char* key_path, const char* ca_path) {
if (!tcp_connect_socket(client, host, port)) if (!tcp_connect_socket(client, host, port)) {
if (client->file_descriptor >= 0)
close(client->file_descriptor);
client->file_descriptor = -1;
return false; return false;
}
SSL_CTX* ctx = create_ssl_ctx(false, cert_path, key_path, ca_path); SSL_CTX* ctx = create_ssl_ctx(false, cert_path, key_path, ca_path);
if (!ctx) if (!ctx) {
close(client->file_descriptor);
client->file_descriptor = -1;
return false; return false;
}
client->ssl_ctx = ctx; client->ssl_ctx = ctx;
// Pass the server hostname for TLS hostname verification (SSL_set1_host // Pass the server hostname for TLS hostname verification (SSL_set1_host
@@ -161,6 +210,8 @@ bool client_connect_tls(Client* client, char* host, int port, const char* cert_p
if (!ssl) { if (!ssl) {
SSL_CTX_free(ctx); SSL_CTX_free(ctx);
client->ssl_ctx = NULL; client->ssl_ctx = NULL;
close(client->file_descriptor);
client->file_descriptor = -1;
return false; return false;
} }
+168 -66
View File
@@ -1,69 +1,85 @@
#include "utils.h" #include "utils.h"
#include "array_list.h" #include "array_list.h"
#include "libgen.h" #include "log.h"
#include <dirent.h> #include <dirent.h>
#include <errno.h> #include <errno.h>
#include <fcntl.h> #include <fcntl.h>
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <stdint.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <unistd.h> #include <unistd.h>
static int authorized_root_fd = -1; static int authorized_root_fd = -1;
static char* authorized_root_path;
void utils_set_authorized_root_fd(int fd) { bool utils_set_authorized_root(int fd, const char* canonical_path) {
char* path_copy = canonical_path ? str_dup(canonical_path) : NULL;
if (canonical_path && !path_copy) {
authorized_root_fd = -1;
free(authorized_root_path);
authorized_root_path = NULL;
return false;
}
authorized_root_fd = fd; authorized_root_fd = fd;
free(authorized_root_path);
authorized_root_path = path_copy;
return true;
} }
bool mkdir_r(const char* path) { void utils_set_authorized_root_fd(int fd) {
size_t path_len = strlen(path); (void)utils_set_authorized_root(fd, NULL);
char* path_duplicate = malloc(path_len + 1); }
if (!path_duplicate)
return false; static bool path_is_within_root(const char* root, const char* path) {
memcpy(path_duplicate, path, path_len + 1); size_t root_len = strlen(root);
size_t capacity = path_len + 2; return strncmp(root, path, root_len) == 0 && (path[root_len] == '\0' || path[root_len] == '/');
char* path_current = (char*)malloc(capacity * sizeof(char)); }
if (!path_current) {
free(path_duplicate); static int open_authorized_destination(const char* dest_root) {
return false; if (authorized_root_fd < 0 || !authorized_root_path || !dest_root ||
!path_is_within_root(authorized_root_path, dest_root))
return -1;
int dirfd = dup(authorized_root_fd);
if (dirfd < 0)
return -1;
const char* relative_path = dest_root + strlen(authorized_root_path);
while (*relative_path == '/')
relative_path++;
char* relative = str_dup(*relative_path ? relative_path : ".");
if (!relative) {
close(dirfd);
return -1;
} }
char* path_current_position = path_current;
if (path[0] == '/') { char* saveptr = NULL;
path_current[0] = '/'; char* component = strtok_r(relative, "/", &saveptr);
path_current[1] = '\0'; while (component) {
path_current_position += 1; if (strcmp(component, "..") == 0) {
} else { free(relative);
path_current[0] = '\0'; close(dirfd);
} return -1;
const char* delimiter = "/";
char* saveptr;
const char* part = strtok_r(path_duplicate, delimiter, &saveptr);
bool ok = true;
while (part != NULL) {
size_t part_len = strlen(part);
if ((size_t)(path_current_position - path_current) + part_len + 2 > capacity) {
ok = false;
break;
} }
memcpy(path_current_position, part, part_len); if (strcmp(component, ".") == 0) {
path_current_position += part_len; component = strtok_r(NULL, "/", &saveptr);
path_current_position[0] = '/'; continue;
path_current_position[1] = '\0';
path_current_position++;
struct stat st;
if (stat(path_current, &st) != 0) {
if (mkdir(path_current, 0755) != 0) {
perror("Could not create directory");
ok = false;
break;
}
} }
part = strtok_r(NULL, delimiter, &saveptr); int next = openat(dirfd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (next < 0) {
free(relative);
close(dirfd);
return -1;
}
close(dirfd);
dirfd = next;
component = strtok_r(NULL, "/", &saveptr);
} }
free(path_duplicate);
free(path_current); free(relative);
return ok; return dirfd;
} }
char* str_dup(const char* string) { char* str_dup(const char* string) {
@@ -77,6 +93,32 @@ char* str_dup(const char* string) {
return new_string; return new_string;
} }
char* output_escape(const char* string, bool eight_bit_output) {
if (!string)
return NULL;
size_t length = strlen(string);
if (length > (SIZE_MAX - 1) / 5)
return NULL;
char* escaped = malloc(length * 5 + 1);
if (!escaped)
return NULL;
size_t out = 0;
for (size_t i = 0; i < length; i++) {
unsigned char byte = (unsigned char)string[i];
if ((byte >= 32 && byte <= 126) || (eight_bit_output && byte >= 128)) {
escaped[out++] = (char)byte;
} else {
escaped[out++] = '\\';
escaped[out++] = '#';
escaped[out++] = (char)('0' + ((byte >> 6) & 7));
escaped[out++] = (char)('0' + ((byte >> 3) & 7));
escaped[out++] = (char)('0' + (byte & 7));
}
}
escaped[out] = '\0';
return escaped;
}
/* Match a glob pattern against a string. Supported wildcards: /* Match a glob pattern against a string. Supported wildcards:
* ? matches any single character except '/'. * ? matches any single character except '/'.
* * matches any sequence of characters within one path component (no '/'). * * matches any sequence of characters within one path component (no '/').
@@ -132,6 +174,25 @@ bool glob_match(const char* pattern, const char* str) {
return *str == '\0'; return *str == '\0';
} }
bool format_human_bytes(unsigned long long bytes, char* buffer, size_t buffer_size) {
static const char* const units[] = {"B", "KB", "MB", "GB", "TB", "PB", "EB"};
double value = (double)bytes;
size_t unit = 0;
int written;
if (!buffer || buffer_size == 0)
return false;
while (value >= 1024.0 && unit < sizeof(units) / sizeof(units[0]) - 1) {
value /= 1024.0;
unit++;
}
if (unit == 0)
written = snprintf(buffer, buffer_size, "%llu %s", bytes, units[unit]);
else
written = snprintf(buffer, buffer_size, "%.1f %s", value, units[unit]);
return written >= 0 && (size_t)written < buffer_size;
}
static bool is_dir_in_manifest(const char* rel_path, ArrayList* manifest) { static bool is_dir_in_manifest(const char* rel_path, ArrayList* manifest) {
size_t len = strlen(rel_path); size_t len = strlen(rel_path);
for (int i = 0; i < manifest->size; i++) { for (int i = 0; i < manifest->size; i++) {
@@ -143,7 +204,8 @@ static bool is_dir_in_manifest(const char* rel_path, ArrayList* manifest) {
return false; return false;
} }
static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifest) { static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifest,
size_t max_delete, size_t* deleted_count) {
int scanfd = dup(dirfd); int scanfd = dup(dirfd);
if (scanfd < 0) if (scanfd < 0)
return false; return false;
@@ -152,15 +214,20 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
close(scanfd); close(scanfd);
return false; return false;
} }
bool all_removed = true;
bool operation_ok = true; bool operation_ok = true;
const struct dirent* entry; const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) { while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue; continue;
char* child_rel = path_cat((char*)rel_path, entry->d_name); char* child_rel = path_cat((char*)rel_path, entry->d_name);
if (!child_rel) {
operation_ok = false;
continue;
}
struct stat st; struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) { if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT)
operation_ok = false;
free(child_rel); free(child_rel);
continue; continue;
} }
@@ -173,14 +240,24 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_removed = false; bool child_removed = false;
if (childfd >= 0) { if (childfd >= 0) {
child_removed = delete_extras_fd(childfd, child_rel, manifest); child_removed = delete_extras_fd(childfd, child_rel, manifest, max_delete, deleted_count);
if (!child_removed)
operation_ok = false;
close(childfd); close(childfd);
} } else if (errno != ENOENT) {
if (child_removed && !is_dir_in_manifest(child_rel, manifest) &&
unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0 && errno != ENOENT) {
operation_ok = false; operation_ok = false;
} else if (!child_removed) { }
all_removed = false; if (child_removed && !is_dir_in_manifest(child_rel, manifest)) {
if (*deleted_count >= max_delete) {
operation_ok = false;
} else {
if (unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0) {
if (errno != ENOENT)
operation_ok = false;
} else {
(*deleted_count)++;
}
}
} }
} else { } else {
// Check if relative path is in manifest // Check if relative path is in manifest
@@ -192,38 +269,61 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
} }
} }
if (!found) { if (!found) {
if (unlinkat(dirfd, entry->d_name, 0) != 0 && errno != ENOENT) if (*deleted_count >= max_delete) {
operation_ok = false; operation_ok = false;
fprintf(stderr, " Deleted: %s\n", child_rel); free(child_rel);
} else { continue;
all_removed = false; }
if (unlinkat(dirfd, entry->d_name, 0) != 0) {
if (errno != ENOENT)
operation_ok = false;
} else {
(*deleted_count)++;
}
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
free(escaped_path);
} }
} }
free(child_rel); free(child_rel);
} }
closedir(dir); closedir(dir);
(void)all_removed;
return operation_ok; return operation_ok;
} }
bool delete_extras(const char* dest_root, ArrayList* manifest) { bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete) {
int rootfd = authorized_root_fd >= 0 if (!manifest)
? dup(authorized_root_fd) return false;
: open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); int rootfd;
if (authorized_root_fd >= 0) {
if (authorized_root_path)
rootfd = open_authorized_destination(dest_root);
else if (dest_root == NULL)
rootfd = dup(authorized_root_fd);
else
rootfd = -1;
} else {
rootfd = open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
if (rootfd < 0) if (rootfd < 0)
return false; return false;
bool ok = delete_extras_fd(rootfd, "", manifest); size_t deleted_count = 0;
bool ok = delete_extras_fd(rootfd, "", manifest, max_delete, &deleted_count);
if (close(rootfd) != 0) if (close(rootfd) != 0)
ok = false; ok = false;
return ok; return ok;
} }
bool delete_extras(const char* dest_root, ArrayList* manifest) {
return delete_extras_limited(dest_root, manifest, SIZE_MAX);
}
bool has_path_traversal(const char* path) { bool has_path_traversal(const char* path) {
if (!path) if (!path)
return false; return true;
char* dup = str_dup(path); char* dup = str_dup(path);
if (!dup) if (!dup)
return false; return true;
char* saveptr; char* saveptr;
const char* part = strtok_r(dup, "/", &saveptr); const char* part = strtok_r(dup, "/", &saveptr);
while (part) { while (part) {
@@ -255,6 +355,8 @@ char* path_cat(const char* path1, const char* path2) {
offset = 1; offset = 1;
path2_len -= 1; path2_len -= 1;
} }
if (path1_len > SIZE_MAX - path2_len - 2)
return NULL;
char* new_path = malloc(path1_len + path2_len + 2); char* new_path = malloc(path1_len + path2_len + 2);
if (new_path == NULL) if (new_path == NULL)
return NULL; return NULL;
+7 -1
View File
@@ -2,15 +2,21 @@
#define UTILS_H #define UTILS_H
#include "array_list.h" #include "array_list.h"
#include <stddef.h>
#include <stdbool.h> #include <stdbool.h>
bool mkdir_r(const char* path);
char* str_dup(const char* string); char* str_dup(const char* string);
char* output_escape(const char* string, bool eight_bit_output);
char* path_cat(const char* path1, const char* path2); char* path_cat(const char* path1, const char* path2);
bool glob_match(const char* pattern, const char* str); bool glob_match(const char* pattern, const char* str);
bool delete_extras(const char* dest_root, ArrayList* manifest); bool delete_extras(const char* dest_root, ArrayList* manifest);
bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete);
bool utils_set_authorized_root(int fd, const char* canonical_path);
/* The fd-only compatibility form is fail-closed for path-based operations;
* callers should use utils_set_authorized_root with the canonical identity. */
void utils_set_authorized_root_fd(int fd); void utils_set_authorized_root_fd(int fd);
bool has_path_traversal(const char* path); bool has_path_traversal(const char* path);
bool utils_valid_batch_path(const char* path); bool utils_valid_batch_path(const char* path);
bool format_human_bytes(unsigned long long bytes, char* buffer, size_t buffer_size);
#endif #endif
+3 -1
View File
@@ -25,7 +25,9 @@ class ServerManager:
def start(self, extra_args=None): def start(self, extra_args=None):
self.stop() self.stop()
self._port = _find_free_port() self._port = _find_free_port()
cmd = SERVER_CMD + ["-p", str(self._port)] # Plain TCP is intentionally explicit in the server; integration tests
# exercise that opt-in mode rather than relying on the secure default.
cmd = SERVER_CMD + ["-p", str(self._port), "--allow-unauthenticated"]
if extra_args: if extra_args:
cmd += extra_args cmd += extra_args
self._proc = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) self._proc = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+744
View File
@@ -1,4 +1,5 @@
"""Feature tests: incremental sync, bandwidth limiting, dry run, metadata, filters.""" """Feature tests: incremental sync, bandwidth limiting, dry run, metadata, filters."""
import filecmp
import os import os
import shutil import shutil
import sys import sys
@@ -26,6 +27,12 @@ def setup_test_data():
class TestDryRun: class TestDryRun:
def test_human_readable_dry_run(self):
result, dur = run_client(SOURCE_DIR, DEST_DIR, flags=["-h", "--dry-run"])
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}"
assert "Total:" in result.stdout
assert "KB" in result.stdout
def test_dry_run(self): def test_dry_run(self):
clean_dir(DEST_DIR) clean_dir(DEST_DIR)
result, dur = run_client( result, dur = run_client(
@@ -35,6 +42,118 @@ class TestDryRun:
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}" assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}"
assert "Dry run:" in result.stdout, f"No dry run output: {result.stdout[:200]}" assert "Dry run:" in result.stdout, f"No dry run output: {result.stdout[:200]}"
def test_quiet_suppresses_dry_run_output(self):
result, dur = run_client(
SOURCE_DIR, DEST_DIR,
flags=["-q", "-n", "--progress", "--stats"],
)
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}"
assert result.stdout == ""
assert result.stderr == ""
def test_quiet_preserves_errors(self):
result, dur = run_client(
SOURCE_DIR, DEST_DIR,
flags=["--quiet", "--server-port", "1"],
)
assert result.returncode != 0
assert result.stderr != ""
@pytest.mark.parametrize("flags", [["-q", "-v"], ["-v", "-q"]])
def test_quiet_successful_transfer_and_verbose_order(self, shared_server, flags):
clean_dir(DEST_DIR)
result, dur = run_client(
SOURCE_DIR, DEST_DIR,
flags=flags,
port=shared_server.port,
)
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}"
assert result.stdout == ""
assert result.stderr == ""
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
class TestRemoveSourceFiles:
def test_removes_only_transferred_regular_files(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remove_source")
dest = os.path.join(TEST_DATA_DIR, "remove_dest")
clean_dir(source)
clean_dir(dest)
with open(os.path.join(source, "one.txt"), "wb") as f:
f.write(b"one")
with open(os.path.join(source, "two.txt"), "wb") as f:
f.write(b"two")
os.makedirs(os.path.join(source, "directory"))
os.symlink("one.txt", os.path.join(source, "link.txt"))
result, _ = run_client(source, dest, flags=["--remove-source-files", "-m"],
port=shared_server.port)
assert result.returncode == 0, f"Remove-source sync failed: {result.stderr[:200]}"
assert not os.path.exists(os.path.join(source, "one.txt"))
assert not os.path.exists(os.path.join(source, "two.txt"))
assert os.path.isdir(os.path.join(source, "directory"))
assert os.path.islink(os.path.join(source, "link.txt"))
def test_single_threaded_removes_transferred_file(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remove_single_source")
dest = os.path.join(TEST_DATA_DIR, "remove_single_dest")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "file.txt")
with open(source_file, "wb") as f:
f.write(b"single threaded")
result, _ = run_client(source, dest, flags=["--remove-source-files"],
port=shared_server.port)
assert result.returncode == 0
assert not os.path.exists(source_file)
def test_dry_run_preserves_source_files(self):
source = os.path.join(TEST_DATA_DIR, "remove_dry_source")
dest = os.path.join(TEST_DATA_DIR, "remove_dry_dest")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "file.txt")
with open(source_file, "wb") as f:
f.write(b"keep")
result, _ = run_client(source, dest, flags=["--remove-source-files", "--dry-run"])
assert result.returncode == 0
assert os.path.isfile(source_file)
def test_failed_connection_preserves_source_files(self):
source = os.path.join(TEST_DATA_DIR, "remove_failed_source")
dest = os.path.join(TEST_DATA_DIR, "remove_failed_dest")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "file.txt")
with open(source_file, "wb") as f:
f.write(b"keep after failure")
result, _ = run_client(source, dest, flags=["--remove-source-files"], port=1)
assert result.returncode != 0
assert os.path.isfile(source_file)
def test_incremental_skip_preserves_source_file(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remove_skipped_source")
dest = os.path.join(TEST_DATA_DIR, "remove_skipped_dest")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "file.txt")
with open(source_file, "wb") as f:
f.write(b"keep after skip")
result, _ = run_client(source, dest, port=shared_server.port)
assert result.returncode == 0
result, _ = run_client(source, dest,
flags=["--remove-source-files", "--incremental"],
port=shared_server.port)
assert result.returncode == 0
assert os.path.isfile(source_file)
class TestArchiveMode: class TestArchiveMode:
def test_archive_mode(self, shared_server): def test_archive_mode(self, shared_server):
@@ -51,6 +170,136 @@ class TestArchiveMode:
assert not missing, f"Missing: {missing}" assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}" assert not mismatches, f"Mismatch: {mismatches}"
def test_archive_implied_options_can_be_negated(self, shared_server):
clean_dir(DEST_DIR)
result, dur = run_client(
SOURCE_DIR, DEST_DIR,
flags=["--archive", "--no-compress", "--no-m", "--no-preserve"],
port=shared_server.port,
)
if result.returncode != 0:
pytest.fail(f"Exit {result.returncode}: {(result.stderr or result.stdout)[:200]}")
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
class TestExecutability:
def test_preserves_only_executable_bits(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "executability_source")
dest = os.path.join(TEST_DATA_DIR, "executability_dest")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "tool.sh")
with open(source_file, "w") as f:
f.write("#!/bin/sh\necho test\n")
os.chmod(source_file, 0o751)
result, _ = run_client(source, dest, flags=["-E"], port=shared_server.port)
assert result.returncode == 0, f"Executability sync failed: {result.stderr[:200]}"
received_file = os.path.join(get_dest_received_dir(dest, source), "tool.sh")
received_mode = os.stat(received_file).st_mode
assert received_mode & 0o111 == 0o111
assert received_mode & 0o600 == 0o600
assert received_mode & 0o077 == 0o011
class TestChmod:
def test_chmod_applies_to_transferred_files(self, shared_server):
clean_dir(DEST_DIR)
source_file = os.path.join(SOURCE_DIR, "small.txt")
os.chmod(source_file, 0o777)
result, dur = run_client(
SOURCE_DIR, DEST_DIR,
flags=["--chmod=u=rw,go=r"],
port=shared_server.port,
)
if result.returncode != 0:
pytest.fail(f"Exit {result.returncode}: {(result.stderr or result.stdout)[:200]}")
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
assert (os.stat(os.path.join(received, "small.txt")).st_mode & 0o777) == 0o644
class TestCompressionChoice:
def test_zstd_choice_compresses(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["--zc", "zstd"],
port=shared_server.port)
assert result.returncode == 0, f"zstd sync failed: {(result.stderr or result.stdout)[:200]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
def test_none_choice_disables_compression(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-z", "--compress-choice", "none"],
port=shared_server.port)
assert result.returncode == 0, f"none sync failed: {(result.stderr or result.stdout)[:200]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
class TestSkipCompress:
def test_skip_compress_case_insensitive(self, shared_server):
clean_dir(DEST_DIR)
with open(os.path.join(SOURCE_DIR, "skip-case.TXT"), "wb") as f:
f.write((b"skip compression case test\n" * 100))
result, _ = run_client(
SOURCE_DIR, DEST_DIR,
flags=["-c", "--skip-compress=.txt"],
port=shared_server.port,
)
assert result.returncode == 0, f"Skip-compress sync failed: {(result.stderr or result.stdout)[:200]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
with open(os.path.join(received, "skip-case.TXT"), "rb") as f:
assert f.read() == b"skip compression case test\n" * 100
def test_skip_compress_empty_list(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(
SOURCE_DIR, DEST_DIR,
flags=["-c", "--skip-compress="],
port=shared_server.port,
)
assert result.returncode == 0, f"Empty skip-compress sync failed: {(result.stderr or result.stdout)[:200]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
def test_skip_compress_incremental_full_fallback(self, shared_server):
clean_dir(DEST_DIR)
path = os.path.join(SOURCE_DIR, "incremental-skip.TXT")
with open(path, "wb") as f:
f.write(b"original skipped content\n")
flags = ["-c", "-M", "--skip-compress=.txt"]
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
assert result.returncode == 0, f"Initial sync failed: {(result.stderr or result.stdout)[:200]}"
with open(path, "wb") as f:
f.write(b"updated skipped content\n")
result, _ = run_client(
SOURCE_DIR, DEST_DIR,
flags=flags + ["--incremental"],
port=shared_server.port,
)
assert result.returncode == 0, f"Incremental sync failed: {(result.stderr or result.stdout)[:200]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
with open(os.path.join(received, "incremental-skip.TXT"), "rb") as f:
assert f.read() == b"updated skipped content\n"
def test_skip_compress_rejects_chunk_serialization(self, shared_server):
result, _ = run_client(
SOURCE_DIR, DEST_DIR,
flags=["-c", "-s", "--skip-compress=.txt"],
port=shared_server.port,
)
assert result.returncode != 0
assert "cannot be combined" in (result.stderr or result.stdout)
class TestExclude: class TestExclude:
def test_exclude_single(self, shared_server): def test_exclude_single(self, shared_server):
@@ -213,6 +462,225 @@ class TestIncremental:
with open(received_file, "rb") as f: with open(received_file, "rb") as f:
assert f.read() == b"hello world\n" assert f.read() == b"hello world\n"
def test_size_only_skips_same_size_with_different_mtime(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-M"], port=shared_server.port)
assert result.returncode == 0
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
received_file = os.path.join(received, "small.txt")
with open(received_file, "wb") as f:
f.write(b"different!!\n")
os.utime(received_file, (time.time() - 3600, time.time() - 3600))
result, _ = run_client(
SOURCE_DIR, DEST_DIR,
flags=["-M", "--incremental", "--size-only"],
port=shared_server.port,
)
assert result.returncode == 0, f"Size-only sync failed: {result.stderr[:200]}"
with open(received_file, "rb") as f:
assert f.read() == b"different!!\n"
def test_ignore_times_transfers_same_size_and_mtime(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-M"], port=shared_server.port)
assert result.returncode == 0
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
source_file = os.path.join(SOURCE_DIR, "small.txt")
received_file = os.path.join(received, "small.txt")
source_stat = os.stat(source_file)
with open(received_file, "wb") as f:
f.write(b"stale data!\n")
os.utime(received_file, (source_stat.st_atime, source_stat.st_mtime))
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=["-M", "--incremental", "--ignore-times"],
port=shared_server.port)
assert result.returncode == 0, f"Ignore-times sync failed: {result.stderr[:200]}"
with open(received_file, "rb") as f:
assert f.read() == b"hello world\n"
def test_modify_window_allows_subsecond_mtime_difference(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-M"], port=shared_server.port)
assert result.returncode == 0
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
source_file = os.path.join(SOURCE_DIR, "small.txt")
received_file = os.path.join(received, "small.txt")
source_stat = os.stat(source_file)
with open(received_file, "wb") as f:
f.write(b"modified!!!\n")
os.utime(received_file, ns=(source_stat.st_atime_ns,
source_stat.st_mtime_ns - 1500000000))
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=["-M", "--incremental", "--modify-window=2"],
port=shared_server.port)
assert result.returncode == 0, f"Modify-window sync failed: {result.stderr[:200]}"
with open(received_file, "rb") as f:
assert f.read() == b"modified!!!\n"
def test_whole_file_disables_delta_and_keeps_compression(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-M"], port=shared_server.port)
assert result.returncode == 0
source_file = os.path.join(SOURCE_DIR, "medium.txt")
with open(source_file, "wb") as f:
f.write(b"whole-file replacement\n" * 5000)
result, _ = run_client(
SOURCE_DIR,
DEST_DIR,
flags=["-M", "--incremental", "--delta", "-W", "-c"],
port=shared_server.port,
)
assert result.returncode == 0, f"Whole-file sync failed: {(result.stderr or result.stdout)[:200]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
class TestUpdate:
def test_update_skips_older_destination_and_allows_equal_or_newer_source(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-u"], port=shared_server.port)
assert result.returncode == 0
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
source_file = os.path.join(SOURCE_DIR, "small.txt")
received_file = os.path.join(received, "small.txt")
source_stat = os.stat(source_file)
with open(received_file, "wb") as f:
f.write(b"newer destination\n")
os.utime(received_file, ns=(source_stat.st_atime_ns, source_stat.st_mtime_ns + 10_000_000_000))
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-u"], port=shared_server.port)
assert result.returncode == 0
with open(received_file, "rb") as f:
assert f.read() == b"newer destination\n"
os.utime(received_file, ns=(source_stat.st_atime_ns, source_stat.st_mtime_ns))
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-u"], port=shared_server.port)
assert result.returncode == 0
with open(received_file, "rb") as f:
assert f.read() == b"hello world\n"
def test_update_skips_unreadable_newer_destination(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-u"], port=shared_server.port)
assert result.returncode == 0
received_file = os.path.join(get_dest_received_dir(DEST_DIR, SOURCE_DIR), "small.txt")
source_stat = os.stat(os.path.join(SOURCE_DIR, "small.txt"))
with open(received_file, "wb") as f:
f.write(b"protected destination\n")
os.utime(received_file, ns=(source_stat.st_atime_ns, source_stat.st_mtime_ns + 10_000_000_000))
original_mode = os.stat(received_file).st_mode
try:
os.chmod(received_file, 0)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-u"], port=shared_server.port)
assert result.returncode == 0
os.chmod(received_file, original_mode)
with open(received_file, "rb") as f:
assert f.read() == b"protected destination\n"
finally:
os.chmod(received_file, original_mode)
with open(received_file, "wb") as f:
f.write(b"older destination\n")
os.utime(received_file, ns=(source_stat.st_atime_ns, source_stat.st_mtime_ns - 10_000_000_000))
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-u"], port=shared_server.port)
assert result.returncode == 0
with open(received_file, "rb") as f:
assert f.read() == b"hello world\n"
class TestExisting:
def test_existing_updates_existing_and_skips_new(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=["-M"], port=shared_server.port)
assert result.returncode == 0, f"Initial sync failed: {(result.stderr or result.stdout)[:200]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
source_file = os.path.join(SOURCE_DIR, "small.txt")
new_source_file = os.path.join(SOURCE_DIR, "new-existing-test.txt")
with open(source_file, "wb") as f:
f.write(b"updated existing content\n")
with open(new_source_file, "wb") as f:
f.write(b"this file must not be created\n")
try:
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=["-M", "--existing"], port=shared_server.port)
assert result.returncode == 0, f"--existing sync failed: {(result.stderr or result.stdout)[:200]}"
with open(os.path.join(received, "small.txt"), "rb") as f:
assert f.read() == b"updated existing content\n"
assert not os.path.exists(os.path.join(received, "new-existing-test.txt"))
finally:
os.unlink(new_source_file)
with open(source_file, "wb") as f:
f.write(b"hello world\n")
class TestIgnoreExisting:
def test_ignore_existing_preserves_existing_and_transfers_new(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, port=shared_server.port)
assert result.returncode == 0
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
existing_file = os.path.join(received, "small.txt")
with open(existing_file, "wb") as f:
f.write(b"destination content\n")
new_source = os.path.join(SOURCE_DIR, "new.txt")
try:
with open(new_source, "wb") as f:
f.write(b"new file\n")
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=["--ignore-existing"], port=shared_server.port)
assert result.returncode == 0, f"Sync failed: {(result.stderr or result.stdout)[:200]}"
with open(existing_file, "rb") as f:
assert f.read() == b"destination content\n"
with open(os.path.join(received, "new.txt"), "rb") as f:
assert f.read() == b"new file\n"
finally:
if os.path.lexists(new_source):
os.unlink(new_source)
class TestIgnoreExisting:
def test_ignore_existing_preserves_existing_and_transfers_new(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, port=shared_server.port)
assert result.returncode == 0
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
existing_file = os.path.join(received, "small.txt")
with open(existing_file, "wb") as f:
f.write(b"destination content\n")
new_source = os.path.join(SOURCE_DIR, "new.txt")
try:
with open(new_source, "wb") as f:
f.write(b"new file\n")
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=["--ignore-existing"], port=shared_server.port)
assert result.returncode == 0, f"Sync failed: {(result.stderr or result.stdout)[:200]}"
with open(existing_file, "rb") as f:
assert f.read() == b"destination content\n"
with open(os.path.join(received, "new.txt"), "rb") as f:
assert f.read() == b"new file\n"
finally:
if os.path.lexists(new_source):
os.unlink(new_source)
class TestDelete: class TestDelete:
def test_delete_removes_extra_files(self, shared_server): def test_delete_removes_extra_files(self, shared_server):
@@ -263,6 +731,75 @@ class TestProgress:
assert "Sent " in output and "MB" in output, "--progress produced no stable byte marker" assert "Sent " in output and "MB" in output, "--progress produced no stable byte marker"
assert "Done." in output, "--progress did not report completion" assert "Done." in output, "--progress did not report completion"
def test_human_readable_stats(self, shared_server):
clean_dir(DEST_DIR)
result, dur = run_client(
SOURCE_DIR, DEST_DIR,
flags=["-h", "--stats"],
port=shared_server.port,
)
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}"
assert "Stats:" in result.stderr
assert "KB" in result.stderr
def test_human_readable_progress_multithreaded(self, shared_server):
clean_dir(DEST_DIR)
result, dur = run_client(
SOURCE_DIR, DEST_DIR,
flags=["-m", "-h", "--progress"],
port=shared_server.port,
)
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}"
output = result.stdout + result.stderr
assert "Sent " in output
assert "KB" in output
assert "Done." in output
class TestInfo:
def test_info_copy_reports_transfers(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(
SOURCE_DIR, DEST_DIR,
flags=["--info=copy"],
port=shared_server.port,
)
assert result.returncode == 0, f"Info sync failed: {(result.stderr or result.stdout)[:200]}"
output = result.stdout + result.stderr
assert "[INFO]" in output and "Transferring" in output
def test_info_stats_reports_multithreaded_transfer(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(
SOURCE_DIR, DEST_DIR,
flags=["-m", "--info=stats"],
port=shared_server.port,
)
assert result.returncode == 0, f"Info stats sync failed: {(result.stderr or result.stdout)[:200]}"
output = result.stdout + result.stderr
assert "[INFO]" in output and "Transfer summary:" in output
def test_info_rejects_unknown_flag(self):
result, _ = run_client(
SOURCE_DIR, DEST_DIR,
flags=["--info=unknown"],
)
assert result.returncode != 0
assert "unsupported --info flag" in result.stderr
@pytest.mark.parametrize("flags", [
["--info=none", "--verbose"],
["--verbose", "--info=none"],
])
def test_info_none_suppresses_verbose_info(self, shared_server, flags):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
assert result.returncode == 0, f"Info sync failed: {(result.stderr or result.stdout)[:200]}"
output = result.stdout + result.stderr
assert "[INFO]" not in output
assert "Transferring" not in output
assert "Transfer summary:" not in output
class TestBandwidthLimit: class TestBandwidthLimit:
def test_bwlimit_runs(self, shared_server): def test_bwlimit_runs(self, shared_server):
@@ -277,3 +814,210 @@ class TestBandwidthLimit:
mismatches, missing = verify_transfer(SOURCE_DIR, received) mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}" assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}" assert not mismatches, f"Mismatch: {mismatches}"
def _read_file(path):
with open(path, "rb") as fh:
return fh.read()
class TestRemoveSourceFilesSkips:
"""--remove-source-files must not delete sources the receiver skipped
(rsync reference behavior)."""
def test_existing_first_sync_keeps_new_source(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remove_rsf_existing_src")
dest = os.path.join(TEST_DATA_DIR, "remove_rsf_existing_dst")
clean_dir(source)
clean_dir(dest)
with open(os.path.join(source, "only.txt"), "wb") as f:
f.write(b"keep me")
result, _ = run_client(source, dest, flags=["--remove-source-files", "--existing"],
port=shared_server.port)
assert result.returncode == 0, f"Sync failed: {result.stderr[:200]}"
# The file exists only on the source side, so --existing makes the
# receiver skip it; the source must therefore not be removed.
assert os.path.isfile(os.path.join(source, "only.txt"))
received = get_dest_received_dir(dest, source)
assert not os.path.exists(os.path.join(received, "only.txt"))
def test_ignore_existing_keeps_skipped_source(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remove_rsf_ignore_src")
dest = os.path.join(TEST_DATA_DIR, "remove_rsf_ignore_dst")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "file.txt")
with open(source_file, "wb") as f:
f.write(b"payload")
result, _ = run_client(source, dest, port=shared_server.port)
assert result.returncode == 0
result, _ = run_client(source, dest, flags=["--remove-source-files", "--ignore-existing"],
port=shared_server.port)
assert result.returncode == 0, f"Sync failed: {result.stderr[:200]}"
# Destination already has the file, so the second run is a receiver
# skip; the source file must survive.
assert os.path.isfile(source_file)
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, "file.txt")) == b"payload"
def test_update_newer_destination_keeps_source(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remove_rsf_update_src")
dest = os.path.join(TEST_DATA_DIR, "remove_rsf_update_dst")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "file.txt")
with open(source_file, "wb") as f:
f.write(b"source payload")
result, _ = run_client(source, dest, port=shared_server.port)
assert result.returncode == 0
received = get_dest_received_dir(dest, source)
received_file = os.path.join(received, "file.txt")
with open(received_file, "wb") as f:
f.write(b"newer destination payload")
os.utime(received_file, ns=(time.time_ns() + 10**9, time.time_ns() + 10**9))
result, _ = run_client(source, dest, flags=["--remove-source-files", "--update"],
port=shared_server.port)
assert result.returncode == 0, f"Sync failed: {result.stderr[:200]}"
# --update skips a destination that is newer than the source, so the
# source must not be removed.
assert os.path.isfile(source_file)
assert _read_file(received_file) == b"newer destination payload"
def test_multithreaded_ignore_existing_keeps_skipped_source(self, shared_server):
"""The multithreaded writer path must also report per-file outcomes so a
--remove-source-files sender does not delete skipped sources."""
source = os.path.join(TEST_DATA_DIR, "remove_rsf_mt_ignore_src")
dest = os.path.join(TEST_DATA_DIR, "remove_rsf_mt_ignore_dst")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "file.txt")
with open(source_file, "wb") as f:
f.write(b"payload")
result, _ = run_client(source, dest, port=shared_server.port)
assert result.returncode == 0
result, _ = run_client(source, dest,
flags=["--remove-source-files", "--ignore-existing", "-m"],
port=shared_server.port)
assert result.returncode == 0, f"Sync failed: {result.stderr[:200]}"
# Destination already has the file, so the receiver (writer thread)
# skips it; the source must survive.
assert os.path.isfile(source_file)
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, "file.txt")) == b"payload"
class TestBackup:
def _sync(self, source, dest, flags, port):
return run_client(source, dest, flags=flags, port=port)
def test_plain_backup_keeps_previous_version(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "backup_src")
dest = os.path.join(TEST_DATA_DIR, "backup_dst")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "f.txt")
with open(source_file, "wb") as f:
f.write(b"AAAA")
result, _ = self._sync(source, dest, ["--backup"], shared_server.port)
assert result.returncode == 0, f"Backup sync failed: {result.stderr[:200]}"
with open(source_file, "wb") as f:
f.write(b"BBBB")
result, _ = self._sync(source, dest, ["--backup"], shared_server.port)
assert result.returncode == 0, f"Backup sync failed: {result.stderr[:200]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, "f.txt")) == b"BBBB"
# rsync default suffix "~" keeps the overwritten version.
assert _read_file(os.path.join(received, "f.txt~")) == b"AAAA"
def test_backup_custom_suffix(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "backup_suffix_src")
dest = os.path.join(TEST_DATA_DIR, "backup_suffix_dst")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "f.txt")
with open(source_file, "wb") as f:
f.write(b"AAAA")
flags = ["--backup", "--suffix", ".bak"]
result, _ = self._sync(source, dest, flags, shared_server.port)
assert result.returncode == 0, f"Backup sync failed: {result.stderr[:200]}"
with open(source_file, "wb") as f:
f.write(b"BBBB")
result, _ = self._sync(source, dest, flags, shared_server.port)
assert result.returncode == 0, f"Backup sync failed: {result.stderr[:200]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, "f.txt")) == b"BBBB"
assert _read_file(os.path.join(received, "f.txt.bak")) == b"AAAA"
def test_backup_dir_stores_backups_separately(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "backup_dir_src")
dest = os.path.join(TEST_DATA_DIR, "backup_dir_dst")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "f.txt")
with open(source_file, "wb") as f:
f.write(b"AAAA")
flags = ["--backup", "--backup-dir", "backups"]
result, _ = self._sync(source, dest, flags, shared_server.port)
assert result.returncode == 0, f"Backup sync failed: {result.stderr[:200]}"
with open(source_file, "wb") as f:
f.write(b"BBBB")
result, _ = self._sync(source, dest, flags, shared_server.port)
assert result.returncode == 0, f"Backup sync failed: {result.stderr[:200]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, "f.txt")) == b"BBBB"
backup = os.path.join(dest, "backups", os.path.relpath(source_file, os.path.sep))
assert _read_file(backup) == b"AAAA"
class TestPartialDir:
def test_completed_transfer_installed_in_destination(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "partial_src")
dest = os.path.join(TEST_DATA_DIR, "partial_dst")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "f.txt")
with open(source_file, "wb") as f:
f.write(b"partial payload")
result, _ = run_client(source, dest, flags=["--partial", "--partial-dir", ".partial"],
port=shared_server.port)
assert result.returncode == 0, f"Partial sync failed: {result.stderr[:200]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, "f.txt")) == b"partial payload"
# A completed transfer must not remain under the partial directory.
partial = os.path.join(dest, ".partial", os.path.relpath(source_file, os.path.sep))
assert not os.path.exists(partial)
class TestLargeFile:
def test_transfer_100mb_file(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "large_src")
dest = os.path.join(TEST_DATA_DIR, "large_dst")
clean_dir(source)
clean_dir(dest)
source_file = os.path.join(source, "big.bin")
chunk = os.urandom(1024 * 1024)
with open(source_file, "wb") as f:
for _ in range(100):
f.write(chunk)
result, _ = run_client(source, dest, port=shared_server.port)
assert result.returncode == 0, f"Large-file sync failed: {result.stderr[:200]}"
received = get_dest_received_dir(dest, source)
assert filecmp.cmp(source_file, os.path.join(received, "big.bin"), shallow=False)
+5
View File
@@ -71,6 +71,11 @@ class TestTCPFlags:
r = _run_tcp_test("Compression (-c)", shared_server.port, ["-c"]) r = _run_tcp_test("Compression (-c)", shared_server.port, ["-c"])
assert r["status"] == "Success", r["error"] assert r["status"] == "Success", r["error"]
def test_compression_threads(self, shared_server):
r = _run_tcp_test("Compression threads (-c --compress-threads=2)", shared_server.port,
["-c", "--compress-threads=2"])
assert r["status"] == "Success", r["error"]
def test_chunk_serialization(self, shared_server): def test_chunk_serialization(self, shared_server):
r = _run_tcp_test("Chunk Serialization (-s)", shared_server.port, ["-s"]) r = _run_tcp_test("Chunk Serialization (-s)", shared_server.port, ["-s"])
assert r["status"] == "Success", r["error"] assert r["status"] == "Success", r["error"]
+3
View File
@@ -102,6 +102,7 @@ class TestTLSBasic:
with ServerManager() as server: with ServerManager() as server:
server.start(extra_args=[ server.start(extra_args=[
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"], "--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
"--ca", certs["ca"], "--client-cn", "fastsync-client",
]) ])
result, dur = run_client( result, dur = run_client(
SOURCE_DIR, DEST_DIR, SOURCE_DIR, DEST_DIR,
@@ -125,6 +126,7 @@ class TestTLSBasic:
with ServerManager() as server: with ServerManager() as server:
server.start(extra_args=[ server.start(extra_args=[
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"], "--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
"--ca", certs["ca"], "--client-cn", "fastsync-client",
]) ])
result, dur = run_client( result, dur = run_client(
SOURCE_DIR, DEST_DIR, SOURCE_DIR, DEST_DIR,
@@ -149,6 +151,7 @@ class TestTLSBasic:
with ServerManager() as server: with ServerManager() as server:
server.start(extra_args=[ server.start(extra_args=[
"--tls", "--cert", certs["server_cert"], "--key", certs["server_key"], "--tls", "--cert", certs["server_cert"], "--key", certs["server_key"],
"--ca", certs["ca"], "--client-cn", "fastsync-client",
]) ])
result, dur = run_client( result, dur = run_client(
SOURCE_DIR, DEST_DIR, SOURCE_DIR, DEST_DIR,
+4
View File
@@ -17,6 +17,8 @@ void test_array_list() {
// Test adding // Test adding
int* val1 = malloc(sizeof(int)); int* val1 = malloc(sizeof(int));
if (!val1)
return;
*val1 = 42; *val1 = 42;
array_list_add(list, val1); array_list_add(list, val1);
EXPECT_EQ_INT(list->size, 1); EXPECT_EQ_INT(list->size, 1);
@@ -26,6 +28,8 @@ void test_array_list() {
// Initial capacity is 100. Let's add 105 elements. // Initial capacity is 100. Let's add 105 elements.
for (int i = 0; i < 105; i++) { for (int i = 0; i < 105; i++) {
int* val = malloc(sizeof(int)); int* val = malloc(sizeof(int));
if (!val)
return;
*val = i; *val = i;
array_list_add(list, val); array_list_add(list, val);
} }
+3 -3
View File
@@ -11,7 +11,7 @@ static void test_file_operations() {
char* test_content = "Hello, Chunk System!"; char* test_content = "Hello, Chunk System!";
unsigned long long test_len = strlen(test_content); unsigned long long test_len = strlen(test_content);
to_disk(test_path, test_content, test_len, false, false); file_write_to_disk(test_path, test_content, test_len, false, false);
File* f = file_create(test_path); File* f = file_create(test_path);
EXPECT_NOT_NULL(f); EXPECT_NOT_NULL(f);
@@ -43,8 +43,8 @@ static void test_chunk_operations() {
char* content2 = "chunk item number 2"; char* content2 = "chunk item number 2";
unsigned long long len2 = strlen(content2); unsigned long long len2 = strlen(content2);
to_disk(path1, content1, len1, false, false); file_write_to_disk(path1, content1, len1, false, false);
to_disk(path2, content2, len2, false, false); file_write_to_disk(path2, content2, len2, false, false);
struct stat st1, st2; struct stat st1, st2;
stat(path1, &st1); stat(path1, &st1);
+877 -10
View File
@@ -1,8 +1,11 @@
#include "test_client_cli.h" #include "test_client_cli.h"
#include "client_validation.h" #include "client_validation.h"
#include "chmod.h"
#include "config.h" #include "config.h"
#include "log.h"
#include "test_utils.h" #include "test_utils.h"
#include "utils.h" #include "utils.h"
#include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <unistd.h> #include <unistd.h>
@@ -36,6 +39,15 @@ static void test_validate_config_incompatible_options() {
cfg->use_incremental = true; cfg->use_incremental = true;
cfg->use_chunk_serialization = true; cfg->use_chunk_serialization = true;
EXPECT_FALSE(validate_config(cfg)); EXPECT_FALSE(validate_config(cfg));
cfg->use_incremental = false;
cfg->skip_compress_set = true;
EXPECT_FALSE(validate_config(cfg));
cfg->skip_compress_set = false;
cfg->compression_threads = 2;
EXPECT_FALSE(validate_config(cfg));
cfg->use_compression = true;
cfg->use_sendfile = false;
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg); config_delete(cfg);
} }
@@ -46,6 +58,8 @@ static void test_validate_config_tls_requirements() {
cfg->tls_cert = str_dup("cert.pem"); cfg->tls_cert = str_dup("cert.pem");
EXPECT_FALSE(validate_config(cfg)); EXPECT_FALSE(validate_config(cfg));
cfg->tls_key = str_dup("key.pem"); cfg->tls_key = str_dup("key.pem");
EXPECT_FALSE(validate_config(cfg));
cfg->tls_ca = str_dup("ca.pem");
EXPECT_TRUE(validate_config(cfg)); EXPECT_TRUE(validate_config(cfg));
config_delete(cfg); config_delete(cfg);
} }
@@ -57,6 +71,13 @@ static void test_validate_config_delta_sendfile_constraints() {
cfg->use_incremental = true; cfg->use_incremental = true;
cfg->use_sendfile = true; cfg->use_sendfile = true;
EXPECT_FALSE(validate_config(cfg)); EXPECT_FALSE(validate_config(cfg));
/* Whole-file makes delta selection inactive, so these combinations are valid. */
cfg->whole_file = true;
EXPECT_TRUE(validate_config(cfg));
cfg->use_incremental = false;
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg); config_delete(cfg);
} }
@@ -105,6 +126,27 @@ static void test_cli_dry_run() {
config_delete(cfg); config_delete(cfg);
} }
static void test_cli_remove_source_files() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
EXPECT_FALSE(cfg->remove_source_files);
cfg->remove_source_files = true;
EXPECT_TRUE(cfg->remove_source_files);
config_delete(cfg);
}
static void test_parse_args_remove_source_files() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--remove-source-files", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
int ret = parse_args(cfg, 4, argv, positional_args, &positional_count);
EXPECT_EQ_INT(ret, 0);
EXPECT_TRUE(cfg->remove_source_files);
config_delete(cfg);
}
/* Test that --delete sets use_delete */ /* Test that --delete sets use_delete */
static void test_cli_delete_flag() { static void test_cli_delete_flag() {
Config* cfg = config_create(); Config* cfg = config_create();
@@ -179,6 +221,80 @@ static void test_parse_args_valid_port() {
config_delete(cfg); config_delete(cfg);
} }
/* Test parse_args with --size-only. */
static void test_parse_args_size_only() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--size-only", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->size_only);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
static void test_parse_args_ignore_existing() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--ignore-existing", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
int ret = parse_args(cfg, 4, argv, positional_args, &positional_count);
EXPECT_EQ_INT(ret, 0);
EXPECT_TRUE(cfg->ignore_existing);
config_delete(cfg);
}
static void test_parse_args_executability() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-E", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_executability);
EXPECT_TRUE(cfg->use_metadata);
config_delete(cfg);
}
static void test_parse_args_chmod() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--chmod=u=rw,go=r", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->chmod_spec, "u=rw,go=r");
EXPECT_TRUE(cfg->use_metadata);
mode_t result;
EXPECT_TRUE(chmod_apply(0777, cfg->chmod_spec, &result));
EXPECT_EQ_INT(result, 0644);
config_delete(cfg);
}
static void test_parse_args_numeric_chmod() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--chmod", "7777", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->chmod_spec, "7777");
EXPECT_TRUE(cfg->use_metadata);
config_delete(cfg);
}
static void test_parse_args_rejects_invalid_chmod() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--chmod=a+X", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
/* Test parse_args rejects port > 65535 */ /* Test parse_args rejects port > 65535 */
static void test_parse_args_invalid_port() { static void test_parse_args_invalid_port() {
Config* cfg = config_create(); Config* cfg = config_create();
@@ -245,6 +361,178 @@ static void test_parse_args_valid_compression_level() {
config_delete(cfg); config_delete(cfg);
} }
static void test_parse_args_debug_flags() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--debug=io,proto,pack,util", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->debug_level, LOG_DEBUG_ALL);
EXPECT_EQ_INT(get_log_debug_flags(), LOG_DEBUG_ALL);
config_delete(cfg);
}
static void test_parse_args_debug_help() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--debug=help"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 2, argv, positional_args, &positional_count), 1);
config_delete(cfg);
}
static void test_parse_args_debug_flags_validation() {
static const char* const values[] = {"", "io,", ",io", "io,,proto", "acl", "tls", "unknown"};
for (size_t i = 0; i < sizeof(values) / sizeof(values[0]); i++) {
Config* cfg = config_create();
char option[64];
snprintf(option, sizeof(option), "--debug=%s", values[i]);
char* argv[] = {"fastsync", option, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
}
static void test_parse_args_modify_window() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--modify-window=3", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->modify_window, 3);
config_delete(cfg);
cfg = config_create();
char* short_argv[] = {"fastsync", "-@", "7", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, short_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->modify_window, 7);
config_delete(cfg);
cfg = config_create();
char* attached_argv[] = {"fastsync", "-@11", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, attached_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->modify_window, 11);
config_delete(cfg);
}
static void test_parse_args_rejects_invalid_modify_window() {
const char* values[] = {"-1", "not-a-number", ""};
for (size_t i = 0; i < sizeof(values) / sizeof(values[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--modify-window", (char*)values[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
}
static void test_parse_args_max_alloc_sizes() {
const char* values[] = {"1", "4K", "2m", "3G", "1T", "1P", "1E", "512B"};
const unsigned long long expected[] = {1,
4ULL * 1024,
2ULL * 1024 * 1024,
3ULL * 1024 * 1024 * 1024,
1ULL * 1024 * 1024 * 1024 * 1024,
1ULL * 1024 * 1024 * 1024 * 1024 * 1024,
1ULL * 1024 * 1024 * 1024 * 1024 * 1024 * 1024,
512};
for (size_t i = 0; i < sizeof(values) / sizeof(values[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--max-alloc", (char*)values[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->max_alloc == expected[i]);
config_delete(cfg);
}
Config* cfg = config_create();
char* argv[] = {"fastsync", "--max-alloc=8M", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->max_alloc == 8ULL * 1024 * 1024);
config_delete(cfg);
}
static void test_parse_args_rejects_invalid_max_alloc() {
const char* values[] = {"0", "-1", "+1", " 1", "1 ",
"1Z", "1K2", "1 K", "1\tK", "18446744073709551615K"};
for (size_t i = 0; i < sizeof(values) / sizeof(values[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--max-alloc", (char*)values[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
}
static void test_parse_args_skip_compress() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--skip-compress=.ZIP, .GZ", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->skip_compress_set);
EXPECT_EQ_INT(cfg->skip_compress_count, 2);
EXPECT_EQ_STR(cfg->skip_compress_suffixes[0], ".ZIP");
EXPECT_EQ_STR(cfg->skip_compress_suffixes[1], ".GZ");
config_delete(cfg);
}
static void test_parse_args_empty_skip_compress() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--skip-compress=", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->skip_compress_set);
EXPECT_EQ_INT(cfg->skip_compress_count, 0);
config_delete(cfg);
}
static void test_parse_args_compression_threads() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--compress-threads", "4", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_threads, 4);
config_delete(cfg);
cfg = config_create();
char* equals_argv[] = {"fastsync", "--compress-threads=3", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, equals_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_threads, 3);
config_delete(cfg);
cfg = config_create();
char* invalid_argv[] = {"fastsync", "--compress-threads=0", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, invalid_argv, positional_args, &positional_count), -1);
config_delete(cfg);
cfg = config_create();
char* excessive_argv[] = {"fastsync", "--compress-threads=65", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, excessive_argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
/* Test parse_args unknown option returns error */ /* Test parse_args unknown option returns error */
static void test_parse_args_unknown_option() { static void test_parse_args_unknown_option() {
Config* cfg = config_create(); Config* cfg = config_create();
@@ -258,11 +546,53 @@ static void test_parse_args_unknown_option() {
config_delete(cfg); config_delete(cfg);
} }
/* Directory aliases must report the unsupported directory-only behavior clearly. */
static void test_parse_args_rejects_dirs_aliases() {
static const char* const options[] = {"--dirs", "--old-dirs", "--old-d"};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
FILE* log_file = tmpfile();
char log_buffer[256] = {0};
EXPECT_NOT_NULL(log_file);
log_set_file(log_file);
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
fflush(log_file);
rewind(log_file);
EXPECT_TRUE(fread(log_buffer, 1, sizeof(log_buffer) - 1, log_file) > 0);
EXPECT_TRUE(strstr(log_buffer, options[i]) != NULL);
EXPECT_TRUE(strstr(log_buffer, "directory-only transfer is not implemented") != NULL);
EXPECT_TRUE(strstr(log_buffer, "requires --dirs") == NULL);
log_set_file(NULL);
fclose(log_file);
config_delete(cfg);
}
}
/* --del is recognized as the rsync alias, but its timing mode is not implemented. */
static void test_parse_args_delete_during_alias_unimplemented() {
static const char* const options[] = {"--del", "--delete-during"};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
EXPECT_FALSE(cfg->use_delete);
config_delete(cfg);
}
}
/* Parsed-but-unimplemented options must fail instead of being silently accepted. */ /* Parsed-but-unimplemented options must fail instead of being silently accepted. */
static void test_parse_args_rejects_unimplemented_options() { static void test_parse_args_rejects_unimplemented_options() {
static const char* const options[] = {"-q", static const char* const options[] = {"--silent",
"--quiet",
"--silent",
"--queue-size", "--queue-size",
"-H", "-H",
"--hard-links", "--hard-links",
@@ -275,13 +605,7 @@ static void test_parse_args_rejects_unimplemented_options() {
"-i", "-i",
"--itemize-changes", "--itemize-changes",
"--out-format", "--out-format",
"--info",
"--debug",
"--list-only", "--list-only",
"-h",
"--human-readable",
"-u",
"--update",
"--append", "--append",
"--append-verify", "--append-verify",
"--delete-excluded", "--delete-excluded",
@@ -308,7 +632,7 @@ static void test_parse_args_rejects_unimplemented_options() {
"--daemon", "--daemon",
"--config", "--config",
"--server", "--server",
"--compress-choice"}; "--checksum-choice"};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) { for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create(); Config* cfg = config_create();
@@ -321,6 +645,92 @@ static void test_parse_args_rejects_unimplemented_options() {
} }
} }
/* Test both rsync-compatible quiet spellings and option ordering. */
static void test_parse_args_quiet() {
static const char* const options[][2] = {
{"-q", "-v"}, {"-v", "-q"}, {"--quiet", "-v"}, {"-v", "--quiet"}};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i][0], (char*)options[i][1], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->quiet);
config_delete(cfg);
}
}
static void test_parse_args_human_readable() {
static const char* const options[] = {"-h", "--human-readable"};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->human_readable);
config_delete(cfg);
}
}
static void test_parse_args_update() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-u", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->update);
EXPECT_TRUE(cfg->use_metadata);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
static void test_parse_args_info_flags() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--info=copy,skip", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_COPY | LOG_INFO_SKIP);
EXPECT_EQ_INT(get_log_info_flags(), LOG_INFO_COPY | LOG_INFO_SKIP);
config_delete(cfg);
}
static void test_parse_args_info_verbose_order() {
char* argv_info_first[] = {"fastsync", "--info=none", "--verbose", "/src", "/dst"};
char* argv_verbose_first[] = {"fastsync", "--verbose", "--info=none", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
Config* cfg = config_create();
EXPECT_EQ_INT(parse_args(cfg, 5, argv_info_first, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, 0);
EXPECT_EQ_INT(get_log_info_flags(), 0);
config_delete(cfg);
cfg = config_create();
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv_verbose_first, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, 0);
EXPECT_EQ_INT(get_log_info_flags(), 0);
config_delete(cfg);
}
static void test_parse_args_rejects_invalid_info_flag() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--info=copy,unknown", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
/* Test parse_args with --archive flag */ /* Test parse_args with --archive flag */
static void test_parse_args_archive() { static void test_parse_args_archive() {
Config* cfg = config_create(); Config* cfg = config_create();
@@ -337,6 +747,411 @@ static void test_parse_args_archive() {
config_delete(cfg); config_delete(cfg);
} }
/* Negations must override archive's implied options in argument order. */
static void test_parse_args_negations() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--archive", "--no-compress", "--no-m",
"--no-preserve", "--no-dry-run", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 8, argv, positional_args, &positional_count), 0);
EXPECT_FALSE(cfg->use_compression);
EXPECT_FALSE(cfg->use_multithreading);
EXPECT_FALSE(cfg->use_metadata);
EXPECT_FALSE(cfg->dry_run);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
static void test_parse_args_negation_order() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--no-z", "-c", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_compression);
config_delete(cfg);
}
static void test_parse_args_no_preserve_blocks_implicit_metadata() {
static const char* const options[][3] = {
{"--incremental", "--no-preserve", "/src"},
{"--no-preserve", "--incremental", "/src"},
{"--delta", "--no-preserve", "/src"},
{"--no-preserve", "--delta", "/src"},
};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i][0], (char*)options[i][1], (char*)options[i][2],
"/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_FALSE(cfg->use_metadata);
EXPECT_TRUE(cfg->metadata_explicitly_disabled);
config_delete(cfg);
}
}
/* Checksum-choice spellings are recognized and rejected until algorithms are implemented. */
static void test_parse_args_checksum_choice_aliases() {
static const char* const options[] = {"--checksum-choice", "--cc"};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i], "xxh64", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
}
static void test_parse_args_rejects_unsafe_negation() {
static const char* const options[] = {"--no-archive", "--no-timeout", "--no-unknown"};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
}
/* Both checksum-choice spellings require a value. */
static void test_parse_args_checksum_choice_requires_value() {
static const char* const options[] = {"--checksum-choice", "--cc"};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i]};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 2, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
}
static void test_parse_args_old_args() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--old-args", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->old_args);
config_delete(cfg);
}
static void test_parse_args_fsync() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--fsync", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_fsync);
config_delete(cfg);
}
static void test_parse_args_existing() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--existing", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->existing);
config_delete(cfg);
}
static void test_parse_args_ignore_times() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-I", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->ignore_times);
config_delete(cfg);
cfg = config_create();
char* long_argv[] = {"fastsync", "--ignore-times", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, long_argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->ignore_times);
config_delete(cfg);
}
/* --secluded-args is accepted for compatibility but has no effect. */
static void test_parse_args_secluded_args() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--secluded-args", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_FALSE(cfg->use_chunk_serialization);
config_delete(cfg);
}
static void test_parse_args_short_s_remains_chunk_serialization() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-s", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_chunk_serialization);
config_delete(cfg);
}
static void test_parse_args_8_bit_output() {
Config* cfg = config_create();
char* long_argv[] = {"fastsync", "--8-bit-output", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, long_argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->eight_bit_output);
config_delete(cfg);
cfg = config_create();
char* short_argv[] = {"fastsync", "-8", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, short_argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->eight_bit_output);
config_delete(cfg);
}
static void test_parse_args_stderr_modes() {
static const char* const modes[] = {"errors", "all", "e", "a"};
static const LogStderrMode expected[] = {LOG_STDERR_ERRORS, LOG_STDERR_ALL, LOG_STDERR_ERRORS,
LOG_STDERR_ALL};
for (size_t i = 0; i < sizeof(modes) / sizeof(modes[0]); i++) {
Config* cfg = config_create();
char option[32];
snprintf(option, sizeof(option), "--stderr=%s", modes[i]);
char* argv[] = {"fastsync", option, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(log_get_stderr_mode(), expected[i]);
config_delete(cfg);
}
log_set_stderr_mode(LOG_STDERR_ERRORS);
}
static void test_parse_args_rejects_unsupported_stderr_modes() {
static const char* const modes[] = {"client", "c", "invalid"};
for (size_t i = 0; i < sizeof(modes) / sizeof(modes[0]); i++) {
Config* cfg = config_create();
char option[32];
snprintf(option, sizeof(option), "--stderr=%s", modes[i]);
char* argv[] = {"fastsync", option, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
log_set_stderr_mode(LOG_STDERR_ERRORS);
}
/* Test both whole-file spellings and its precedence over delta selection. */
static void test_parse_args_whole_file() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--delta", "--incremental", "-W", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->whole_file);
EXPECT_TRUE(cfg->use_delta);
config_delete(cfg);
cfg = config_create();
char* long_argv[] = {"fastsync", "--whole-file", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, long_argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->whole_file);
config_delete(cfg);
}
/* Test rsync-compatible compression-choice and compression-level aliases. */
static void test_parse_args_compression_aliases() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--zc", "zstd", "--zl", "10", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
int ret = parse_args(cfg, 7, argv, positional_args, &positional_count);
EXPECT_EQ_INT(ret, 0);
EXPECT_EQ_STR(cfg->compress_choice, "zstd");
EXPECT_EQ_INT(cfg->compression_level, 10);
EXPECT_TRUE(cfg->use_compression);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
/* Test rsync-compatible -P parsing; resumable partial-file retention is not implied. */
static void test_parse_args_partial_progress() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-P", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
int ret = parse_args(cfg, 4, argv, positional_args, &positional_count);
EXPECT_EQ_INT(ret, 0);
EXPECT_TRUE(cfg->partial);
EXPECT_TRUE(cfg->show_progress);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
static void test_parse_args_compression_equals_and_none() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-z", "--zc=none", "--zl=7", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
int ret = parse_args(cfg, 6, argv, positional_args, &positional_count);
EXPECT_EQ_INT(ret, 0);
EXPECT_EQ_STR(cfg->compress_choice, "none");
EXPECT_EQ_INT(cfg->compression_level, 7);
EXPECT_FALSE(cfg->use_compression);
config_delete(cfg);
}
static void test_parse_args_compression_canonical_equals() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--compress-choice=zstd", "--compress-level=7", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->compress_choice, "zstd");
EXPECT_EQ_INT(cfg->compression_level, 7);
EXPECT_TRUE(cfg->use_compression);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
static void test_parse_args_compression_alias_equals() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--zc=zstd", "--zl=7", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->compress_choice, "zstd");
EXPECT_EQ_INT(cfg->compression_level, 7);
EXPECT_TRUE(cfg->use_compression);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
static void test_parse_args_rejects_invalid_compression_level_equals() {
static const char* const values[] = {"0", "23", "invalid"};
for (size_t i = 0; i < sizeof(values) / sizeof(values[0]); i++) {
Config* cfg = config_create();
char option[32];
snprintf(option, sizeof(option), "--compress-level=%s", values[i]);
char* argv[] = {"fastsync", option, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
}
static void test_parse_args_rejects_invalid_compression_choice() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--compress-choice=bogus", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
/* Every value-taking table option accepts an inline "--opt=value" form. */
static void test_parse_args_table_equals_size_options() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--max-size=2G", "--min-size=1K", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->max_size == 2ULL * 1024 * 1024 * 1024);
EXPECT_TRUE(cfg->min_size == 1024ULL);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
static void test_parse_args_table_equals_string_and_int_options() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--suffix=.bak", "--timeout=30", "--max-depth=5", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, argv, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->suffix, ".bak");
EXPECT_EQ_INT(cfg->timeout, 30);
EXPECT_EQ_INT(cfg->max_depth, 5);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
cfg = config_create();
char* backup_argv[] = {"fastsync", "--backup-dir=/tmp/bak", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, backup_argv, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->backup_dir, "/tmp/bak");
config_delete(cfg);
}
/* Options that take a separate value must report "missing argument", not the
* generic "Unknown option", when they are the final argv entry. */
static void test_parse_args_missing_argument_diagnostic() {
static const char* const options[] = {"--exclude", "--server-port", "--skip-compress", "-T"};
for (size_t i = 0; i < sizeof(options) / sizeof(options[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)options[i]};
int positional_args[2];
int positional_count = 0;
FILE* log_file = tmpfile();
char log_buffer[512] = {0};
EXPECT_NOT_NULL(log_file);
log_set_file(log_file);
EXPECT_EQ_INT(parse_args(cfg, 2, argv, positional_args, &positional_count), -1);
fflush(log_file);
rewind(log_file);
EXPECT_TRUE(fread(log_buffer, 1, sizeof(log_buffer) - 1, log_file) > 0);
EXPECT_TRUE(strstr(log_buffer, "missing argument") != NULL);
EXPECT_TRUE(strstr(log_buffer, "Unknown option") == NULL);
log_set_file(NULL);
fclose(log_file);
config_delete(cfg);
}
}
void test_client_cli() { void test_client_cli() {
test_validate_config_required_paths(); test_validate_config_required_paths();
test_validate_config_incompatible_options(); test_validate_config_incompatible_options();
@@ -345,17 +1160,69 @@ void test_client_cli() {
test_cli_help(); test_cli_help();
test_cli_archive_flags(); test_cli_archive_flags();
test_cli_dry_run(); test_cli_dry_run();
test_cli_remove_source_files();
test_parse_args_remove_source_files();
test_cli_delete_flag(); test_cli_delete_flag();
test_cli_exclude_patterns(); test_cli_exclude_patterns();
test_parse_args_help(); test_parse_args_help();
test_parse_args_version(); test_parse_args_version();
test_parse_args_valid_port(); test_parse_args_valid_port();
test_parse_args_size_only();
test_parse_args_ignore_existing();
test_parse_args_executability();
test_parse_args_chmod();
test_parse_args_numeric_chmod();
test_parse_args_rejects_invalid_chmod();
test_parse_args_invalid_port(); test_parse_args_invalid_port();
test_parse_args_non_numeric_port(); test_parse_args_non_numeric_port();
test_parse_args_invalid_server_port(); test_parse_args_invalid_server_port();
test_parse_args_invalid_compression_level(); test_parse_args_invalid_compression_level();
test_parse_args_valid_compression_level(); test_parse_args_valid_compression_level();
test_parse_args_debug_flags();
test_parse_args_debug_help();
test_parse_args_debug_flags_validation();
test_parse_args_modify_window();
test_parse_args_rejects_invalid_modify_window();
test_parse_args_skip_compress();
test_parse_args_empty_skip_compress();
test_parse_args_compression_threads();
test_parse_args_max_alloc_sizes();
test_parse_args_rejects_invalid_max_alloc();
test_parse_args_unknown_option(); test_parse_args_unknown_option();
test_parse_args_rejects_dirs_aliases();
test_parse_args_delete_during_alias_unimplemented();
test_parse_args_rejects_unimplemented_options(); test_parse_args_rejects_unimplemented_options();
test_parse_args_quiet();
test_parse_args_human_readable();
test_parse_args_update();
test_parse_args_info_flags();
test_parse_args_info_verbose_order();
test_parse_args_rejects_invalid_info_flag();
test_parse_args_archive(); test_parse_args_archive();
test_parse_args_negations();
test_parse_args_negation_order();
test_parse_args_no_preserve_blocks_implicit_metadata();
test_parse_args_rejects_unsafe_negation();
test_parse_args_old_args();
test_parse_args_fsync();
test_parse_args_existing();
test_parse_args_ignore_times();
test_parse_args_8_bit_output();
test_parse_args_stderr_modes();
test_parse_args_rejects_unsupported_stderr_modes();
test_parse_args_secluded_args();
test_parse_args_short_s_remains_chunk_serialization();
test_parse_args_whole_file();
test_parse_args_compression_aliases();
test_parse_args_compression_equals_and_none();
test_parse_args_compression_canonical_equals();
test_parse_args_compression_alias_equals();
test_parse_args_rejects_invalid_compression_level_equals();
test_parse_args_rejects_invalid_compression_choice();
test_parse_args_table_equals_size_options();
test_parse_args_table_equals_string_and_int_options();
test_parse_args_missing_argument_diagnostic();
test_parse_args_partial_progress();
test_parse_args_checksum_choice_aliases();
test_parse_args_checksum_choice_requires_value();
} }
+31 -2
View File
@@ -13,6 +13,8 @@ static void test_data_compress_decompress_roundtrip() {
size_t len = strlen(original); size_t len = strlen(original);
char* buf = malloc(len); char* buf = malloc(len);
if (!buf)
return;
memcpy(buf, original, len); memcpy(buf, original, len);
Data* original_data = data_create(buf, len); Data* original_data = data_create(buf, len);
EXPECT_NOT_NULL(original_data); EXPECT_NOT_NULL(original_data);
@@ -53,6 +55,31 @@ static void test_data_compress_decompress_large() {
data_destroy(decompressed); data_destroy(decompressed);
} }
static void test_skip_compress_suffix_matching() {
char* suffixes[] = {".ZIP", ".GZ"};
EXPECT_TRUE(compression_should_skip_with_suffixes("archive.zip", suffixes, 2));
EXPECT_TRUE(compression_should_skip_with_suffixes("backup.TAR.GZ", suffixes, 2));
EXPECT_FALSE(compression_should_skip_with_suffixes("notes.txt", suffixes, 2));
EXPECT_FALSE(compression_should_skip_with_suffixes("archive.zip", suffixes, 0));
}
static void test_data_compress_with_threads_roundtrip() {
const size_t size = 8 * 1024 * 1024;
Data* input = data_create_empty(size);
EXPECT_NOT_NULL(input);
for (size_t i = 0; i < size; i++)
((char*)input->data)[i] = (char)((i / 4096) % 7);
Data* compressed = data_compress_with_threads(input, 3, 2);
EXPECT_NOT_NULL(compressed);
Data* decompressed = data_decompress(compressed);
EXPECT_NOT_NULL(decompressed);
EXPECT_EQ_INT((int)decompressed->size, (int)size);
EXPECT_EQ_INT(memcmp(decompressed->data, input->data, size), 0);
data_destroy(input);
data_destroy(compressed);
data_destroy(decompressed);
}
static void test_chunk_compress_decompress_roundtrip() { static void test_chunk_compress_decompress_roundtrip() {
char* path1 = "temp_comp_test_1.txt"; char* path1 = "temp_comp_test_1.txt";
char* content1 = "chunk compression test file 1"; char* content1 = "chunk compression test file 1";
@@ -62,8 +89,8 @@ static void test_chunk_compress_decompress_roundtrip() {
char* content2 = "chunk compression test file 2 with more data"; char* content2 = "chunk compression test file 2 with more data";
unsigned long long len2 = strlen(content2); unsigned long long len2 = strlen(content2);
to_disk(path1, content1, len1, false, false); file_write_to_disk(path1, content1, len1, false, false);
to_disk(path2, content2, len2, false, false); file_write_to_disk(path2, content2, len2, false, false);
struct stat st1, st2; struct stat st1, st2;
EXPECT_EQ_INT(stat(path1, &st1), 0); EXPECT_EQ_INT(stat(path1, &st1), 0);
@@ -113,5 +140,7 @@ static void test_chunk_compress_decompress_roundtrip() {
void test_compression() { void test_compression() {
test_data_compress_decompress_roundtrip(); test_data_compress_decompress_roundtrip();
test_data_compress_decompress_large(); test_data_compress_decompress_large();
test_skip_compress_suffix_matching();
test_data_compress_with_threads_roundtrip();
test_chunk_compress_decompress_roundtrip(); test_chunk_compress_decompress_roundtrip();
} }
+157 -19
View File
@@ -95,6 +95,7 @@ static void test_pipeline_sender_lifecycle() {
EXPECT_EQ_INT(pcs->queue_loader->capacity, 15); EXPECT_EQ_INT(pcs->queue_loader->capacity, 15);
EXPECT_FALSE(pcs->scanner_done); EXPECT_FALSE(pcs->scanner_done);
EXPECT_FALSE(pcs->loader_done); EXPECT_FALSE(pcs->loader_done);
EXPECT_EQ_INT((int)pcs->allocation_session.max_alloc, (int)cfg->max_alloc);
pipeline_context_sender_destroy(pcs); pipeline_context_sender_destroy(pcs);
} }
@@ -121,11 +122,25 @@ static void test_config_send_receive() {
send_cfg->receive_root_directory = str_dup("/send/dst"); send_cfg->receive_root_directory = str_dup("/send/dst");
send_cfg->save_to_disk = true; send_cfg->save_to_disk = true;
send_cfg->use_multithreading = true; send_cfg->use_multithreading = true;
send_cfg->use_chunk_serialization = true; send_cfg->use_chunk_serialization = false;
send_cfg->use_compression = true; send_cfg->use_compression = true;
send_cfg->use_metadata = true; send_cfg->use_metadata = true;
send_cfg->use_executability = true;
send_cfg->use_delta = true;
send_cfg->whole_file = true;
send_cfg->ignore_times = true;
send_cfg->size_only = true;
send_cfg->compression_level = 5; send_cfg->compression_level = 5;
send_cfg->chunk_size = 1024; send_cfg->chunk_size = 1024;
send_cfg->eight_bit_output = true;
send_cfg->modify_window = 4;
send_cfg->existing = true;
send_cfg->ignore_existing = true;
send_cfg->skip_compress_set = true;
send_cfg->skip_compress_count = 1;
send_cfg->skip_compress_suffixes = calloc(1, sizeof(char*));
send_cfg->skip_compress_suffixes[0] = str_dup(".zip");
send_cfg->max_alloc = MAX_SERVER_ALLOC + 1;
/* Use socketpair for bidirectional communication */ /* Use socketpair for bidirectional communication */
int p[2]; int p[2];
@@ -154,12 +169,33 @@ static void test_config_send_receive() {
ok = false; ok = false;
if (!recv_cfg->use_multithreading) if (!recv_cfg->use_multithreading)
ok = false; ok = false;
if (!recv_cfg->use_chunk_serialization) if (recv_cfg->use_chunk_serialization)
ok = false; ok = false;
if (recv_cfg->compression_level != 5) if (recv_cfg->compression_level != 5)
ok = false; ok = false;
if (recv_cfg->chunk_size != 1024) if (recv_cfg->chunk_size != 1024)
ok = false; ok = false;
if (!recv_cfg->use_executability)
ok = false;
if (!recv_cfg->size_only)
ok = false;
if (!recv_cfg->ignore_times)
ok = false;
if (!recv_cfg->eight_bit_output)
ok = false;
if (recv_cfg->use_delta)
ok = false;
if (recv_cfg->modify_window != 4)
ok = false;
if (!recv_cfg->existing)
ok = false;
if (!recv_cfg->ignore_existing)
ok = false;
if (!recv_cfg->skip_compress_set || recv_cfg->skip_compress_count != 1 ||
strcmp(recv_cfg->skip_compress_suffixes[0], ".zip") != 0)
ok = false;
if (recv_cfg->max_alloc != MAX_SERVER_ALLOC)
ok = false;
} }
config_delete(recv_cfg); config_delete(recv_cfg);
close(p[0]); close(p[0]);
@@ -185,11 +221,11 @@ static void test_config_send_receive() {
} }
static void test_config_send_receive_version_mismatch() { static void test_config_send_receive_version_mismatch() {
/* Create a config with a different protocol version */ /* A peer using the previous wire format must be rejected. */
Config* cfg = config_create(); Config* cfg = config_create();
EXPECT_NOT_NULL(cfg); EXPECT_NOT_NULL(cfg);
free(cfg->version); free(cfg->version);
cfg->version = str_dup("0.0"); cfg->version = str_dup("2.3.0");
cfg->send_directory = str_dup("/src"); cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst"); cfg->receive_root_directory = str_dup("/dst");
@@ -233,6 +269,8 @@ static void test_config_receive_truncated() {
/* A valid prefix exercises cleanup after allocated wire strings and a /* A valid prefix exercises cleanup after allocated wire strings and a
* partially received scalar field. */ * partially received scalar field. */
EXPECT_TRUE(send_str(p[1], PROTOCOL_VERSION)); EXPECT_TRUE(send_str(p[1], PROTOCOL_VERSION));
unsigned long long max_alloc = DEFAULT_MAX_ALLOC;
EXPECT_TRUE(send_n_data(p[1], &max_alloc, sizeof(max_alloc)));
EXPECT_TRUE(send_str(p[1], "/src")); EXPECT_TRUE(send_str(p[1], "/src"));
EXPECT_TRUE(send_str(p[1], "/dst")); EXPECT_TRUE(send_str(p[1], "/dst"));
EXPECT_TRUE(send_int(p[1], 1)); EXPECT_TRUE(send_int(p[1], 1));
@@ -244,26 +282,125 @@ static void test_config_receive_truncated() {
close(p[1]); close(p[1]);
} }
static void test_is_remote_dest() { static bool config_string_roundtrip_matches(const Config* send_cfg, Config* recv) {
/* The sender serializes NULL strings as "" on the wire. Receivers must
canonicalize those empty values back to NULL for the options whose client
default is NULL (backup_dir, temp_dir, partial_dir, suffix), while a real
non-empty value round-trips unchanged. */
const char* fields[4];
char* const* recv_fields[4];
fields[0] = send_cfg->backup_dir;
recv_fields[0] = &recv->backup_dir;
fields[1] = send_cfg->temp_dir;
recv_fields[1] = &recv->temp_dir;
fields[2] = send_cfg->partial_dir;
recv_fields[2] = &recv->partial_dir;
fields[3] = send_cfg->suffix;
recv_fields[3] = &recv->suffix;
for (int i = 0; i < 4; i++) {
const char* sent = fields[i];
const char* got = *recv_fields[i];
if (sent == NULL || sent[0] == '\0') {
if (got != NULL)
return false;
} else if (got == NULL || strcmp(sent, got) != 0) {
return false;
}
}
return true;
}
static bool roundtrip_config_ok(const Config* send_cfg) {
int p[2];
if (socketpair(AF_UNIX, SOCK_STREAM, 0, p) != 0)
return false;
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
Config* recv = config_receive(p[0]);
bool ok = recv != NULL;
if (ok) {
ok = recv->version != NULL && strcmp(recv->version, PROTOCOL_VERSION) == 0;
ok = ok && recv->send_directory && recv->receive_root_directory;
ok = ok && config_string_roundtrip_matches(send_cfg, recv);
}
config_delete(recv);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
bool sent = config_send(p[1], send_cfg);
int status;
waitpid(pid, &status, 0);
close(p[1]);
return sent && WIFEXITED(status) && WEXITSTATUS(status) == 0;
}
}
/* Issue #252: NULL-vs-empty must survive the wire for backup_dir, temp_dir,
partial_dir, and suffix. NULL and explicitly-empty client values are both
serialized as "" and must be reconstructed as NULL so plain --backup (with
no --suffix/--backup-dir) works exactly like the client configured it. */
static void test_config_string_null_vs_empty_roundtrip() {
if (is_running_under_valgrind())
return;
/* NULL values on the wire must come back as NULL. */
Config* a = config_create();
EXPECT_NOT_NULL(a);
a->send_directory = str_dup("/src");
a->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(roundtrip_config_ok(a));
config_delete(a);
/* Explicitly empty strings (indistinguishable on the wire from NULL) must
be canonicalized to NULL by the receiver. */
Config* b = config_create();
EXPECT_NOT_NULL(b);
b->send_directory = str_dup("/src");
b->receive_root_directory = str_dup("/dst");
b->backup_dir = str_dup("");
b->temp_dir = str_dup("");
b->partial_dir = str_dup("");
b->suffix = str_dup("");
EXPECT_TRUE(roundtrip_config_ok(b));
config_delete(b);
/* Non-empty values must round-trip unchanged. */
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->backup_dir = str_dup("backups");
c->temp_dir = str_dup("/tmp/fast");
c->partial_dir = str_dup(".partial");
c->suffix = str_dup(".bak");
EXPECT_TRUE(roundtrip_config_ok(c));
config_delete(c);
}
static void test_config_is_remote_dest() {
/* Valid SSH-style destinations */ /* Valid SSH-style destinations */
EXPECT_TRUE(is_remote_dest("user@host:/path")); EXPECT_TRUE(config_is_remote_dest("user@host:/path"));
EXPECT_TRUE(is_remote_dest("host:/path")); EXPECT_TRUE(config_is_remote_dest("host:/path"));
EXPECT_TRUE(is_remote_dest("user@192.168.1.1:/remote/path")); EXPECT_TRUE(config_is_remote_dest("user@192.168.1.1:/remote/path"));
/* Invalid destinations */ /* Invalid destinations */
EXPECT_FALSE(is_remote_dest(NULL)); EXPECT_FALSE(config_is_remote_dest(NULL));
EXPECT_FALSE(is_remote_dest("")); EXPECT_FALSE(config_is_remote_dest(""));
EXPECT_FALSE(is_remote_dest(":")); EXPECT_FALSE(config_is_remote_dest(":"));
EXPECT_FALSE(is_remote_dest("/local/path")); EXPECT_FALSE(config_is_remote_dest("/local/path"));
EXPECT_FALSE(is_remote_dest("relative/path")); EXPECT_FALSE(config_is_remote_dest("relative/path"));
/* C:/windows/path is treated as remote (colon with no preceding slash) */ /* C:/windows/path is treated as remote (colon with no preceding slash) */
EXPECT_TRUE(is_remote_dest("C:/windows/path")); EXPECT_TRUE(config_is_remote_dest("C:/windows/path"));
/* Edge cases */ /* Edge cases */
EXPECT_FALSE(is_remote_dest("noslash")); EXPECT_FALSE(config_is_remote_dest("noslash"));
EXPECT_FALSE(is_remote_dest("/")); EXPECT_FALSE(config_is_remote_dest("/"));
EXPECT_TRUE(is_remote_dest("host:")); EXPECT_TRUE(config_is_remote_dest("host:"));
EXPECT_TRUE(is_remote_dest("user@host:")); EXPECT_TRUE(config_is_remote_dest("user@host:"));
} }
void test_config() { void test_config() {
@@ -277,6 +414,7 @@ void test_config() {
test_config_send_receive(); test_config_send_receive();
test_config_send_receive_version_mismatch(); test_config_send_receive_version_mismatch();
test_config_receive_truncated(); test_config_receive_truncated();
test_config_string_null_vs_empty_roundtrip();
} }
test_is_remote_dest(); test_config_is_remote_dest();
} }
+293
View File
@@ -343,6 +343,297 @@ static void test_delta_apply_rejects_output_overflow() {
EXPECT_TRUE(delta_apply(old_data, sizeof(old_data), &delta, 1) == NULL); EXPECT_TRUE(delta_apply(old_data, sizeof(old_data), &delta, 1) == NULL);
} }
/* ---------------------------------------------------------------------------
* Hash-index lookup differential tests.
*
* delta_compute buckets signature blocks by their weak checksum. These tests
* prove the bucket-indexed candidate lookup is behaviour-identical to the
* original per-window linear scan: the emitted instruction stream (types,
* lengths, literal bytes and chosen block indices) must match a naive linear
* reference exactly, and the delta must reconstruct the new buffer.
* ------------------------------------------------------------------------- */
#define REF_NO_MATCH UINT32_MAX
typedef struct {
DeltaInstruction* items;
uint32_t count;
uint32_t cap;
} RefDelta;
static void ref_delta_free(RefDelta* ref) {
if (!ref->items)
return;
for (uint32_t i = 0; i < ref->count; i++)
if (ref->items[i].type == DELTA_INSTR_LITERAL)
free(ref->items[i].literal.data);
free(ref->items);
ref->items = NULL;
ref->count = 0;
ref->cap = 0;
}
static bool ref_delta_push(RefDelta* ref, DeltaInstruction instr) {
if (ref->count == ref->cap) {
uint32_t new_cap = ref->cap ? ref->cap * 2 : 16;
DeltaInstruction* tmp = realloc(ref->items, (size_t)new_cap * sizeof(DeltaInstruction));
if (!tmp)
return false;
ref->items = tmp;
ref->cap = new_cap;
}
ref->items[ref->count++] = instr;
return true;
}
static bool ref_delta_flush_literal(RefDelta* ref, const uint8_t* data, uint64_t start,
uint64_t end) {
if (start >= end)
return true;
uint8_t* lit = malloc((size_t)(end - start));
if (!lit)
return false;
memcpy(lit, data + start, (size_t)(end - start));
DeltaInstruction instr = {
.type = DELTA_INSTR_LITERAL,
.literal = {.data = lit, .length = (uint32_t)(end - start)},
};
return ref_delta_push(ref, instr);
}
/* Naive O(windows x blocks) re-implementation of the historical delta_compute
* candidate scan: only full windows may match, a candidate needs both the weak
* (Adler-32) and strong (xxHash32) checksums to agree, and the lowest matching
* block index is selected. */
static bool ref_delta_build(RefDelta* ref, const uint8_t* new_data, uint64_t new_size,
const DeltaSignature* sig) {
uint32_t block_size = sig->block_size;
uint64_t i = 0;
uint64_t literal_start = 0;
bool has_literal = false;
while (i < new_size) {
uint32_t window_len = (uint32_t)((new_size - i < block_size) ? (new_size - i) : block_size);
bool full_window = (window_len == block_size);
uint32_t matched = REF_NO_MATCH;
if (full_window) {
uint32_t adler = delta_adler32(new_data + i, window_len);
for (uint32_t j = 0; j < sig->block_count; j++) {
if (sig->blocks[j].adler32 == adler &&
delta_xxhash32(new_data + i, window_len) == sig->blocks[j].xxhash) {
matched = j;
break;
}
}
}
if (matched != REF_NO_MATCH) {
if (has_literal) {
if (!ref_delta_flush_literal(ref, new_data, literal_start, i))
return false;
has_literal = false;
}
DeltaInstruction instr = {
.type = DELTA_INSTR_BLOCK_MATCH,
.match = {.block_index = matched, .block_offset = 0, .length = window_len},
};
if (!ref_delta_push(ref, instr))
return false;
i += window_len;
} else {
if (!has_literal) {
literal_start = i;
has_literal = true;
}
i++;
}
}
if (has_literal && !ref_delta_flush_literal(ref, new_data, literal_start, new_size))
return false;
return true;
}
static bool ref_delta_matches(const RefDelta* ref, const Delta* delta) {
if (ref->count != delta->instruction_count)
return false;
for (uint32_t i = 0; i < ref->count; i++) {
const DeltaInstruction* a = &ref->items[i];
const DeltaInstruction* b = &delta->instructions[i];
if (a->type != b->type)
return false;
if (a->type == DELTA_INSTR_BLOCK_MATCH) {
if (a->match.block_index != b->match.block_index ||
a->match.block_offset != b->match.block_offset || a->match.length != b->match.length)
return false;
} else {
if (a->literal.length != b->literal.length ||
memcmp(a->literal.data, b->literal.data, a->literal.length) != 0)
return false;
}
}
return true;
}
static void expect_linear_reference_match(const uint8_t* old_data, uint64_t old_size,
const uint8_t* new_data, uint64_t new_size,
uint32_t block_size, const char* label) {
DeltaSignature* sig = delta_signature_create(old_data, old_size, block_size);
if (!sig) {
printf(" [FAIL] %s: signature creation failed\n", label);
EXPECT_NOT_NULL(sig);
return;
}
Delta* delta = delta_compute(new_data, new_size, sig, block_size);
if (!delta) {
printf(" [FAIL] %s: delta_compute returned NULL\n", label);
delta_signature_destroy(sig);
EXPECT_NOT_NULL(delta);
return;
}
RefDelta ref = {0};
bool ok = ref_delta_build(&ref, new_data, new_size, sig);
if (ok)
ok = ref_delta_matches(&ref, delta);
if (!ok) {
printf(" [FAIL] %s: instruction stream differs from linear reference "
"(linear=%u indexed=%u)\n",
label, ref.count, delta->instruction_count);
}
ref_delta_free(&ref);
delta_destroy(delta);
delta_signature_destroy(sig);
EXPECT_TRUE(ok);
}
static void fill_delta_pattern(uint8_t* buf, uint64_t size, uint32_t seed) {
uint32_t x = seed ? seed : 1;
for (uint64_t i = 0; i < size; i++) {
x ^= x << 13;
x ^= x >> 17;
x ^= x << 5;
buf[i] = (uint8_t)(x >> 24);
}
}
static void test_delta_hash_index_matches_linear_reference() {
/* Identical file (full block alignment). */
uint8_t old_a[32768];
uint8_t new_a[32768];
fill_delta_pattern(old_a, sizeof(old_a), 42);
memcpy(new_a, old_a, sizeof(old_a));
expect_linear_reference_match(old_a, sizeof(old_a), new_a, sizeof(new_a), 2048,
"identical 32KiB @ 2KiB");
/* Scattered single-byte edits in the middle of each block. */
uint8_t new_b[32768];
memcpy(new_b, old_a, sizeof(old_a));
for (size_t p = 100; p < sizeof(new_b); p += 4096)
new_b[p] ^= 0x5A;
expect_linear_reference_match(old_a, sizeof(old_a), new_b, sizeof(new_b), 2048,
"32KiB scattered single-byte edits @ 2KiB");
/* Non-aligned old file (partial final block) with a single edit. */
uint8_t old_c[30000];
uint8_t new_c[30000];
fill_delta_pattern(old_c, sizeof(old_c), 7);
memcpy(new_c, old_c, sizeof(old_c));
new_c[15000] ^= 0x3C;
expect_linear_reference_match(old_c, sizeof(old_c), new_c, sizeof(new_c), 2048,
"30KiB partial-tail single edit @ 2KiB");
/* Growth: appended data after an identical prefix. */
uint8_t old_d[24576];
uint8_t new_d[34576];
fill_delta_pattern(old_d, sizeof(old_d), 11);
memcpy(new_d, old_d, sizeof(old_d));
fill_delta_pattern(new_d + sizeof(old_d), sizeof(new_d) - sizeof(old_d), 23);
expect_linear_reference_match(old_d, sizeof(old_d), new_d, sizeof(new_d), 2048,
"24KiB -> 34KiB appended @ 2KiB");
/* Insertion shifting everything after the edit point (rsync re-sync). */
uint8_t old_e[65536];
uint8_t new_e[65536 + 3000];
fill_delta_pattern(old_e, sizeof(old_e), 99);
memcpy(new_e, old_e, 20000);
fill_delta_pattern(new_e + 20000, 3000, 101);
memcpy(new_e + 23000, old_e + 20000, sizeof(old_e) - 20000);
expect_linear_reference_match(old_e, sizeof(old_e), new_e, sizeof(new_e), 2048,
"64KiB + 3KiB insertion @ 2KiB");
/* Deletion shrinking the file. */
uint8_t new_f[sizeof(old_e) - 5000];
memcpy(new_f, old_e, 30000);
memcpy(new_f + 30000, old_e + 35000, sizeof(old_e) - 35000);
expect_linear_reference_match(old_e, sizeof(old_e), new_f, sizeof(new_f), 2048,
"64KiB - 5KiB deletion @ 2KiB");
/* Repeated identical blocks must resolve to the lowest block index. */
uint8_t old_g[4 * 4096];
uint8_t new_g[4 * 4096];
for (uint32_t b = 0; b < 4; b++)
fill_delta_pattern(old_g + b * 4096, 4096, b % 2 == 0 ? 500 : 501); /* block0==block2 */
memcpy(new_g, old_g, sizeof(old_g));
new_g[4096 + 5] ^= 0x11; /* edit inside the second (duplicated) chunk */
expect_linear_reference_match(old_g, sizeof(old_g), new_g, sizeof(new_g), 4096,
"duplicated chunks @ 4KiB");
/* Block larger than the file: nothing can match, all literal. */
uint8_t old_h[1000];
uint8_t new_h[1000];
fill_delta_pattern(old_h, sizeof(old_h), 3);
memcpy(new_h, old_h, sizeof(old_h));
expect_linear_reference_match(old_h, sizeof(old_h), new_h, sizeof(new_h), 4096,
"1KiB file @ 4KiB block");
}
static void test_delta_hash_index_large_mostly_matching() {
const uint64_t size = 4ULL * 1024 * 1024;
const uint32_t block_size = 8192;
uint8_t* old_data = malloc((size_t)size);
uint8_t* new_data = malloc((size_t)size);
EXPECT_TRUE(old_data != NULL && new_data != NULL);
fill_delta_pattern(old_data, size, 1234);
memcpy(new_data, old_data, (size_t)size);
/* Scattered single-byte changes across the whole buffer. Each change forces
* the diff to re-synchronise by walking one byte at a time through the
* affected block, which is exactly the case that used to cost O(bytes x
* blocks) with the linear scan. */
const uint64_t nchanges = 64;
for (uint64_t c = 0; c < nchanges; c++) {
uint64_t pos = (c * (size / nchanges)) + (c % 17);
new_data[pos] ^= (uint8_t)(0xA0 + (c % 16));
}
DeltaSignature* sig = delta_signature_create(old_data, size, block_size);
EXPECT_NOT_NULL(sig);
EXPECT_EQ_INT((int)sig->block_count, (int)(size / block_size));
Delta* delta = delta_compute(new_data, size, sig, block_size);
EXPECT_NOT_NULL(delta);
EXPECT_EQ_INT((int)delta->new_file_size, (int)size);
uint32_t match_count = 0;
for (uint32_t i = 0; i < delta->instruction_count; i++)
if (delta->instructions[i].type == DELTA_INSTR_BLOCK_MATCH)
match_count++;
EXPECT_TRUE(match_count > 0);
void* result = delta_apply(old_data, size, delta, block_size);
EXPECT_NOT_NULL(result);
EXPECT_EQ_INT(memcmp(result, new_data, (size_t)size), 0);
free(result);
delta_destroy(delta);
delta_signature_destroy(sig);
free(old_data);
free(new_data);
}
void test_delta() { void test_delta() {
test_adler32_basic(); test_adler32_basic();
test_adler32_different_data(); test_adler32_different_data();
@@ -360,4 +651,6 @@ void test_delta() {
test_is_worthwhile(); test_is_worthwhile();
test_large_file_delta(); test_large_file_delta();
test_delta_apply_rejects_output_overflow(); test_delta_apply_rejects_output_overflow();
test_delta_hash_index_matches_linear_reference();
test_delta_hash_index_large_mostly_matching();
} }
+468 -20
View File
@@ -1,13 +1,16 @@
#include "test_file.h" #include "test_file.h"
#include "file.h" #include "file.h"
#include "data.h" #include "data.h"
#include "config.h"
#include "utils.h" #include "utils.h"
#include "protocol.h" #include "protocol.h"
#include "test_utils.h" #include "test_utils.h"
#include <fcntl.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <sys/wait.h> #include <sys/wait.h>
#include <time.h>
#include <unistd.h> #include <unistd.h>
static void test_file_create() { static void test_file_create() {
@@ -34,7 +37,8 @@ static void test_file_destroy_normal() {
static void test_file_load_data() { static void test_file_load_data() {
const char* content = "Hello Load Test"; const char* content = "Hello Load Test";
EXPECT_TRUE(to_disk("test_file_load_data.txt", content, strlen(content), false, false)); EXPECT_TRUE(
file_write_to_disk("test_file_load_data.txt", content, strlen(content), false, false));
struct stat st; struct stat st;
EXPECT_EQ_INT(stat("test_file_load_data.txt", &st), 0); EXPECT_EQ_INT(stat("test_file_load_data.txt", &st), 0);
@@ -87,15 +91,282 @@ static void test_file_save_to_disk() {
rmdir("test_save_tmp"); rmdir("test_save_tmp");
} }
static void test_to_disk_basic() { static void test_file_save_to_disk_with_fsync_config() {
const char* content = "Basic to_disk test"; File* f = file_create("saved_file_fsync.txt");
EXPECT_TRUE(to_disk("test_to_disk_basic.txt", content, strlen(content), false, false)); EXPECT_NOT_NULL(f);
const char* content = "Save to disk with fsync";
f->data->data = malloc(strlen(content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, content, strlen(content));
f->data->size = strlen(content);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->use_fsync = true;
EXPECT_TRUE(file_save_to_disk("test_save_fsync_tmp", f, config));
struct stat st; struct stat st;
EXPECT_EQ_INT(stat("test_to_disk_basic.txt", &st), 0); EXPECT_EQ_INT(stat("test_save_fsync_tmp/saved_file_fsync.txt", &st), 0);
EXPECT_EQ_INT((int)st.st_size, (int)strlen(content)); EXPECT_EQ_INT((int)st.st_size, (int)strlen(content));
FILE* fp = fopen("test_to_disk_basic.txt", "rb"); file_destroy(f);
config_delete(config);
unlink("test_save_fsync_tmp/saved_file_fsync.txt");
rmdir("test_save_fsync_tmp");
}
static void test_file_save_to_disk_existing() {
const char* root = "test_existing_tmp";
const char* existing_path = "test_existing_tmp/existing.txt";
const char* missing_path = "test_existing_tmp/missing.txt";
EXPECT_TRUE(file_write_to_disk(existing_path, "old", 3, false, false));
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->existing = true;
File* existing = file_create("existing.txt");
EXPECT_NOT_NULL(existing);
existing->data->data = malloc(3);
EXPECT_NOT_NULL(existing->data->data);
memcpy(existing->data->data, "new", 3);
existing->data->size = 3;
EXPECT_TRUE(file_save_to_disk(root, existing, cfg));
file_destroy(existing);
File* missing = file_create("missing.txt");
EXPECT_NOT_NULL(missing);
missing->data->data = malloc(7);
EXPECT_NOT_NULL(missing->data->data);
memcpy(missing->data->data, "skipped", 7);
missing->data->size = 7;
EXPECT_TRUE(file_save_to_disk(root, missing, cfg));
file_destroy(missing);
FILE* fp = fopen(existing_path, "rb");
char content[4] = {0};
EXPECT_NOT_NULL(fp);
// cppcheck-suppress knownConditionTrueFalse
if (fp) {
EXPECT_EQ_INT((int)fread(content, 1, 3, fp), 3);
fclose(fp);
}
EXPECT_EQ_STR(content, "new");
EXPECT_EQ_INT(access(missing_path, F_OK), -1);
config_delete(cfg);
unlink(existing_path);
rmdir("test_existing_tmp");
}
static void test_file_save_to_disk_ignore_existing() {
const char* path = "test_ignore_existing_tmp/existing.txt";
EXPECT_TRUE(file_write_to_disk(path, "old", 3, false, false));
File* file = file_create("existing.txt");
EXPECT_NOT_NULL(file);
file->data->data = malloc(3);
EXPECT_NOT_NULL(file->data->data);
memcpy(file->data->data, "new", 3);
file->data->size = 3;
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->ignore_existing = true;
EXPECT_TRUE(file_save_to_disk("test_ignore_existing_tmp", file, config));
FILE* stream = fopen(path, "rb");
char content[4] = {0};
EXPECT_NOT_NULL(stream);
// cppcheck-suppress knownConditionTrueFalse
if (stream) {
EXPECT_EQ_INT((int)fread(content, 1, 3, stream), 3);
fclose(stream);
}
EXPECT_EQ_STR(content, "old");
file_destroy(file);
config_delete(config);
unlink(path);
rmdir("test_ignore_existing_tmp");
}
static void test_file_save_to_disk_ignore_existing_entry_types() {
const char* root = "test_ignore_existing_entries_tmp";
const char* directory = "test_ignore_existing_entries_tmp/directory";
const char* link = "test_ignore_existing_entries_tmp/link";
const char* target = "test_ignore_existing_entries_tmp/target";
const char* backup = "test_ignore_existing_entries_tmp/backup.txt~";
const char* backup_file = "test_ignore_existing_entries_tmp/backup.txt";
Config* config = config_create();
File* file = file_create("unused");
unlink(link);
unlink(target);
unlink(backup);
unlink(backup_file);
rmdir(directory);
rmdir(root);
EXPECT_NOT_NULL(config);
EXPECT_NOT_NULL(file);
// cppcheck-suppress knownConditionTrueFalse
if (!config || !file)
return;
config->ignore_existing = true;
config->backup = true;
file->data->data = malloc(3);
EXPECT_NOT_NULL(file->data->data);
// cppcheck-suppress knownConditionTrueFalse
if (!file->data->data) {
file_destroy(file);
config_delete(config);
return;
}
memcpy(file->data->data, "new", 3);
file->data->size = 3;
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(directory, 0755), 0);
EXPECT_TRUE(file_write_to_disk(target, "old", 3, false, false));
EXPECT_EQ_INT(symlink("target", link), 0);
free(file->path);
file->path = str_dup("directory");
EXPECT_TRUE(file_save_to_disk(root, file, config));
free(file->path);
file->path = str_dup("link");
EXPECT_TRUE(file_save_to_disk(root, file, config));
free(file->path);
file->path = str_dup("backup.txt");
EXPECT_TRUE(file_write_to_disk(backup_file, "old", 3, false, false));
EXPECT_TRUE(file_save_to_disk(root, file, config));
EXPECT_TRUE(file_path_exists_secure(backup_file));
EXPECT_FALSE(file_path_exists_secure(backup));
file_destroy(file);
config_delete(config);
unlink(link);
unlink(target);
unlink(backup_file);
rmdir(directory);
rmdir(root);
}
/* Issue #253: with --partial --partial-dir a completed write must be installed
at the real destination rather than left under the partial directory. */
static void test_file_save_to_disk_partial_install() {
const char* root = "test_partial_install_tmp";
const char* dest_file = "test_partial_install_tmp/file.txt";
const char* partial_file = "test_partial_install_tmp/.partial/file.txt";
unlink(dest_file);
unlink(partial_file);
rmdir("test_partial_install_tmp/.partial");
rmdir(root);
File* f = file_create("file.txt");
EXPECT_NOT_NULL(f);
const char* content = "partial-dir content";
f->data->data = malloc(strlen(content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, content, strlen(content));
f->data->size = strlen(content);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->partial = true;
config->partial_dir = str_dup(".partial");
EXPECT_EQ_INT(file_save_to_disk_full(root, f, config), FILE_SAVE_WRITTEN);
FILE* fp = fopen(dest_file, "rb");
EXPECT_NOT_NULL(fp);
// cppcheck-suppress knownConditionTrueFalse
if (fp) {
char buf[64] = {0};
size_t nread = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
EXPECT_EQ_INT((int)nread, (int)strlen(content));
EXPECT_EQ_INT(memcmp(buf, content, strlen(content)), 0);
}
/* A completed transfer must not linger under the partial dir. */
EXPECT_EQ_INT(access(partial_file, F_OK), -1);
file_destroy(f);
config_delete(config);
unlink(dest_file);
rmdir(root);
}
/* Issue #251: file_save_to_disk_full must distinguish receiver-side skips
(--existing/--ignore-existing/--update) from real writes so the sender can
decide whether --remove-source-files may unlink its source. */
static void test_file_save_to_disk_reports_skips() {
const char* root = "test_save_skip_tmp";
const char* existing_path = "test_save_skip_tmp/existing.txt";
unlink(existing_path);
rmdir(root);
EXPECT_TRUE(file_write_to_disk(existing_path, "old", 3, false, false));
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
File* new_file = file_create("missing.txt");
EXPECT_NOT_NULL(new_file);
new_file->data->data = malloc(7);
EXPECT_NOT_NULL(new_file->data->data);
memcpy(new_file->data->data, "skipped", 7);
new_file->data->size = 7;
/* --existing: destination is missing -> skipped, not an error. */
cfg->existing = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, new_file, cfg), FILE_SAVE_SKIPPED);
cfg->existing = false;
/* --ignore-existing: destination present -> skipped. */
File* present = file_create("existing.txt");
EXPECT_NOT_NULL(present);
present->data->data = malloc(3);
EXPECT_NOT_NULL(present->data->data);
memcpy(present->data->data, "new", 3);
present->data->size = 3;
cfg->ignore_existing = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, present, cfg), FILE_SAVE_SKIPPED);
cfg->ignore_existing = false;
/* A normal overwrite of an existing file is a real write. */
EXPECT_EQ_INT(file_save_to_disk_full(root, present, cfg), FILE_SAVE_WRITTEN);
/* --update: a newer destination is skipped. */
struct stat st;
EXPECT_EQ_INT(stat(existing_path, &st), 0);
time_t now = time(NULL);
FileMetadata metadata = {.mode = st.st_mode,
.uid = st.st_uid,
.gid = st.st_gid,
.mtime_sec = now - 100,
.mtime_nsec = 0};
present->metadata = &metadata;
cfg->update = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, present, cfg), FILE_SAVE_SKIPPED);
present->metadata = NULL;
file_destroy(new_file);
file_destroy(present);
config_delete(cfg);
unlink(existing_path);
rmdir(root);
}
static void test_file_write_to_disk_basic() {
const char* content = "Basic file_write_to_disk test";
EXPECT_TRUE(file_write_to_disk("test_file_write_to_disk_basic.txt", content, strlen(content),
false, false));
struct stat st;
EXPECT_EQ_INT(stat("test_file_write_to_disk_basic.txt", &st), 0);
EXPECT_EQ_INT((int)st.st_size, (int)strlen(content));
FILE* fp = fopen("test_file_write_to_disk_basic.txt", "rb");
EXPECT_NOT_NULL(fp); EXPECT_NOT_NULL(fp);
char buf[100]; char buf[100];
size_t nread = fread(buf, 1, sizeof(buf), fp); size_t nread = fread(buf, 1, sizeof(buf), fp);
@@ -103,12 +374,24 @@ static void test_to_disk_basic() {
EXPECT_EQ_INT((int)nread, (int)strlen(content)); EXPECT_EQ_INT((int)nread, (int)strlen(content));
EXPECT_EQ_INT(memcmp(buf, content, strlen(content)), 0); EXPECT_EQ_INT(memcmp(buf, content, strlen(content)), 0);
unlink("test_to_disk_basic.txt"); unlink("test_file_write_to_disk_basic.txt");
} }
static void test_to_disk_creates_dirs() { static void test_file_write_to_disk_with_fsync() {
const char* path = "test_file_write_to_disk_fsync.txt";
const char* content = "fsync file content";
EXPECT_TRUE(file_to_disk_secure_with_fsync(path, content, strlen(content), false, false, NULL,
false, true));
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT((int)st.st_size, (int)strlen(content));
unlink(path);
}
static void test_file_write_to_disk_creates_dirs() {
const char* content = "Nested dir test"; const char* content = "Nested dir test";
EXPECT_TRUE(to_disk("test_nested_tmp/nested/file.txt", content, strlen(content), false, false)); EXPECT_TRUE(file_write_to_disk("test_nested_tmp/nested/file.txt", content, strlen(content), false,
false));
struct stat st; struct stat st;
EXPECT_EQ_INT(stat("test_nested_tmp/nested/file.txt", &st), 0); EXPECT_EQ_INT(stat("test_nested_tmp/nested/file.txt", &st), 0);
@@ -126,15 +409,15 @@ static void test_to_disk_creates_dirs() {
rmdir("test_nested_tmp"); rmdir("test_nested_tmp");
} }
static void test_to_disk_does_not_follow_symlink() { static void test_file_write_to_disk_does_not_follow_symlink() {
const char* outside = "test_to_disk_outside.txt"; const char* outside = "test_file_write_to_disk_outside.txt";
const char* link = "test_to_disk_link.txt"; const char* link = "test_file_write_to_disk_link.txt";
const char* content = "confined"; const char* content = "confined";
unlink(outside); unlink(outside);
unlink(link); unlink(link);
EXPECT_TRUE(to_disk(outside, "outside", 7, false, false)); EXPECT_TRUE(file_write_to_disk(outside, "outside", 7, false, false));
EXPECT_EQ_INT(symlink(outside, link), 0); EXPECT_EQ_INT(symlink(outside, link), 0);
EXPECT_TRUE(to_disk(link, content, strlen(content), false, false)); EXPECT_TRUE(file_write_to_disk(link, content, strlen(content), false, false));
FILE* fp = fopen(outside, "rb"); FILE* fp = fopen(outside, "rb");
char buf[16] = {0}; char buf[16] = {0};
EXPECT_NOT_NULL(fp); EXPECT_NOT_NULL(fp);
@@ -151,7 +434,7 @@ static void test_to_disk_does_not_follow_symlink() {
static void test_file_content_to_buffer() { static void test_file_content_to_buffer() {
const char* content = "Buffer content test"; const char* content = "Buffer content test";
EXPECT_TRUE(to_disk("test_buffer_file.txt", content, strlen(content), false, false)); EXPECT_TRUE(file_write_to_disk("test_buffer_file.txt", content, strlen(content), false, false));
File* f = file_create("test_buffer_file.txt"); File* f = file_create("test_buffer_file.txt");
EXPECT_NOT_NULL(f); EXPECT_NOT_NULL(f);
@@ -273,7 +556,7 @@ static void test_file_send_no_path() {
} }
static void test_file_metadata_create() { static void test_file_metadata_create() {
EXPECT_TRUE(to_disk("test_meta_file.txt", "metadata test", 13, false, false)); EXPECT_TRUE(file_write_to_disk("test_meta_file.txt", "metadata test", 13, false, false));
struct stat st; struct stat st;
EXPECT_EQ_INT(stat("test_meta_file.txt", &st), 0); EXPECT_EQ_INT(stat("test_meta_file.txt", &st), 0);
@@ -382,7 +665,7 @@ static void test_file_send_single_calls_metadata_and_path() {
/* Create a real file on disk so we can have metadata */ /* Create a real file on disk so we can have metadata */
const char* content = "File with metadata"; const char* content = "File with metadata";
size_t len = strlen(content); size_t len = strlen(content);
EXPECT_TRUE(to_disk("test_meta_send.txt", content, len, false, false)); EXPECT_TRUE(file_write_to_disk("test_meta_send.txt", content, len, false, false));
struct stat st; struct stat st;
EXPECT_EQ_INT(stat("test_meta_send.txt", &st), 0); EXPECT_EQ_INT(stat("test_meta_send.txt", &st), 0);
@@ -448,6 +731,161 @@ static void test_file_send_single_calls_metadata_and_path() {
} }
} }
static void test_inplace_overwrite_clears_special_mode_bits() {
const char* root = "test_inplace_tmp";
const char* path = "test_inplace_tmp/priv.txt";
const char* content = "olddata";
unlink(path);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
/* Create a destination carrying setuid + sticky bits. */
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
EXPECT_TRUE(fd >= 0);
// cppcheck-suppress knownConditionTrueFalse
if (fd < 0) {
rmdir(root);
return;
}
EXPECT_EQ_INT((int)write(fd, content, strlen(content)), (int)strlen(content));
EXPECT_EQ_INT(fchmod(fd, S_ISUID | S_ISVTX | 0755), 0);
EXPECT_EQ_INT(close(fd), 0);
/* Overwrite in place without metadata: the mode must be normalized to a
safe default (0644) and the setuid/sticky bits must be gone. */
File* f = file_create("priv.txt");
EXPECT_NOT_NULL(f);
const char* new_content = "newdata";
f->data->data = malloc(strlen(new_content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, new_content, strlen(new_content));
f->data->size = strlen(new_content);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->inplace = true;
EXPECT_TRUE(file_save_to_disk(root, f, cfg));
file_destroy(f);
config_delete(cfg);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT((int)(st.st_mode & (S_ISUID | S_ISGID | S_ISVTX)), 0);
EXPECT_EQ_INT((int)(st.st_mode & 0777), 0644);
FILE* stream = fopen(path, "rb");
char buf[16] = {0};
EXPECT_NOT_NULL(stream);
// cppcheck-suppress knownConditionTrueFalse
if (stream) {
size_t nread = fread(buf, 1, sizeof(buf) - 1, stream);
fclose(stream);
EXPECT_EQ_INT((int)nread, (int)strlen(new_content));
}
EXPECT_EQ_STR(buf, new_content);
unlink(path);
rmdir(root);
}
static void test_inplace_overwrite_metadata_strips_special_bits() {
const char* root = "test_inplace_meta_tmp";
const char* path = "test_inplace_meta_tmp/meta.txt";
const char* source = "test_inplace_meta_source.txt";
unlink(path);
unlink(source);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
/* Existing destination with setuid+sticky set. */
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
EXPECT_TRUE(fd >= 0);
// cppcheck-suppress knownConditionTrueFalse
if (fd < 0) {
rmdir(root);
return;
}
EXPECT_EQ_INT((int)write(fd, "olddata", 7), 7);
EXPECT_EQ_INT(fchmod(fd, S_ISUID | S_ISVTX | 0755), 0);
EXPECT_EQ_INT(close(fd), 0);
/* Build source metadata carrying a plain executable mode (no specials). */
EXPECT_TRUE(file_write_to_disk(source, "source", 6, false, false));
EXPECT_EQ_INT(chmod(source, 0755), 0);
struct stat source_st;
EXPECT_EQ_INT(stat(source, &source_st), 0);
File* f = file_create("meta.txt");
EXPECT_NOT_NULL(f);
const char* new_content = "meta";
f->data->data = malloc(strlen(new_content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, new_content, strlen(new_content));
f->data->size = strlen(new_content);
f->metadata = file_metadata_create(&source_st);
EXPECT_NOT_NULL(f->metadata);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->inplace = true;
EXPECT_TRUE(file_save_to_disk(root, f, cfg));
file_destroy(f);
config_delete(cfg);
unlink(source);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
/* Metadata-derived mode is applied and never includes setuid/setgid/sticky. */
EXPECT_EQ_INT((int)(st.st_mode & (S_ISUID | S_ISGID | S_ISVTX)), 0);
EXPECT_EQ_INT((int)(st.st_mode & 0777), 0755);
unlink(path);
rmdir(root);
}
static void test_inplace_overwrite_truncates_shorter_payload() {
const char* root = "test_inplace_trunc_tmp";
const char* path = "test_inplace_trunc_tmp/big.txt";
unlink(path);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
const char* old_content = "0123456789abcdef"; /* 16 bytes */
EXPECT_TRUE(file_write_to_disk(path, old_content, strlen(old_content), false, false));
File* f = file_create("big.txt");
EXPECT_NOT_NULL(f);
const char* new_content = "hi";
f->data->data = malloc(strlen(new_content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, new_content, strlen(new_content));
f->data->size = strlen(new_content);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->inplace = true;
EXPECT_TRUE(file_save_to_disk(root, f, cfg));
file_destroy(f);
config_delete(cfg);
/* A shorter payload must truncate the file: no stale trailing bytes. */
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT((int)st.st_size, (int)strlen(new_content));
FILE* stream = fopen(path, "rb");
char buf[32] = {0};
EXPECT_NOT_NULL(stream);
// cppcheck-suppress knownConditionTrueFalse
if (stream) {
size_t nread = fread(buf, 1, sizeof(buf) - 1, stream);
fclose(stream);
EXPECT_EQ_INT((int)nread, (int)strlen(new_content));
}
EXPECT_EQ_STR(buf, new_content);
unlink(path);
rmdir(root);
}
void test_file() { void test_file() {
test_file_create(); test_file_create();
test_file_destroy_null(); test_file_destroy_null();
@@ -455,9 +893,16 @@ void test_file() {
test_file_load_data(); test_file_load_data();
test_file_load_data_missing_file(); test_file_load_data_missing_file();
test_file_save_to_disk(); test_file_save_to_disk();
test_to_disk_basic(); test_file_save_to_disk_with_fsync_config();
test_to_disk_creates_dirs(); test_file_save_to_disk_existing();
test_to_disk_does_not_follow_symlink(); test_file_save_to_disk_ignore_existing();
test_file_save_to_disk_ignore_existing_entry_types();
test_file_save_to_disk_partial_install();
test_file_save_to_disk_reports_skips();
test_file_write_to_disk_basic();
test_file_write_to_disk_with_fsync();
test_file_write_to_disk_creates_dirs();
test_file_write_to_disk_does_not_follow_symlink();
test_file_content_to_buffer(); test_file_content_to_buffer();
test_file_save_to_disk_path_traversal(); test_file_save_to_disk_path_traversal();
test_file_save_to_disk_deep_traversal(); test_file_save_to_disk_deep_traversal();
@@ -473,4 +918,7 @@ void test_file() {
test_file_send_single_calls_metadata_and_path(); test_file_send_single_calls_metadata_and_path();
} }
test_file_metadata_create(); test_file_metadata_create();
test_inplace_overwrite_clears_special_mode_bits();
test_inplace_overwrite_metadata_strips_special_bits();
test_inplace_overwrite_truncates_shorter_payload();
} }
+4 -4
View File
@@ -15,7 +15,7 @@
static void test_sendfile_basic() { static void test_sendfile_basic() {
const char* content = "Hello from sendfile test!"; const char* content = "Hello from sendfile test!";
size_t len = strlen(content); size_t len = strlen(content);
EXPECT_TRUE(to_disk("test_sendfile_basic.txt", content, len, false, false)); EXPECT_TRUE(file_write_to_disk("test_sendfile_basic.txt", content, len, false, false));
File* file = file_create("test_sendfile_basic.txt"); File* file = file_create("test_sendfile_basic.txt");
EXPECT_NOT_NULL(file); EXPECT_NOT_NULL(file);
@@ -77,7 +77,7 @@ static void test_sendfile_basic() {
static void test_sendfile_empty_file() { static void test_sendfile_empty_file() {
const char* content = ""; const char* content = "";
size_t len = 0; size_t len = 0;
EXPECT_TRUE(to_disk("test_sendfile_empty.txt", content, len, false, false)); EXPECT_TRUE(file_write_to_disk("test_sendfile_empty.txt", content, len, false, false));
File* file = file_create("test_sendfile_empty.txt"); File* file = file_create("test_sendfile_empty.txt");
EXPECT_NOT_NULL(file); EXPECT_NOT_NULL(file);
@@ -156,7 +156,7 @@ static void test_sendfile_missing_file() {
static void test_sendfile_compression_fallback() { static void test_sendfile_compression_fallback() {
const char* content = "Compression fallback content"; const char* content = "Compression fallback content";
size_t len = strlen(content); size_t len = strlen(content);
EXPECT_TRUE(to_disk("test_sendfile_comp.txt", content, len, false, false)); EXPECT_TRUE(file_write_to_disk("test_sendfile_comp.txt", content, len, false, false));
struct stat st; struct stat st;
EXPECT_EQ_INT(stat("test_sendfile_comp.txt", &st), 0); EXPECT_EQ_INT(stat("test_sendfile_comp.txt", &st), 0);
@@ -222,7 +222,7 @@ static void test_sendfile_compression_fallback() {
static void test_sendfile_no_path() { static void test_sendfile_no_path() {
const char* content = "No path sendfile test"; const char* content = "No path sendfile test";
size_t len = strlen(content); size_t len = strlen(content);
EXPECT_TRUE(to_disk("test_sendfile_nopath.txt", content, len, false, false)); EXPECT_TRUE(file_write_to_disk("test_sendfile_nopath.txt", content, len, false, false));
File* file = file_create("test_sendfile_nopath.txt"); File* file = file_create("test_sendfile_nopath.txt");
EXPECT_NOT_NULL(file); EXPECT_NOT_NULL(file);
+1 -1
View File
@@ -101,7 +101,7 @@ static void test_fuzz_delta_deserialize() {
/* Smoke test for metadata_from_buf fuzz target */ /* Smoke test for metadata_from_buf fuzz target */
static void test_fuzz_metadata_from_buf() { static void test_fuzz_metadata_from_buf() {
/* Create a real file to get metadata from */ /* Create a real file to get metadata from */
EXPECT_TRUE(to_disk("fuzz_meta_test.txt", "metadata test", 13, false, false)); EXPECT_TRUE(file_write_to_disk("fuzz_meta_test.txt", "metadata test", 13, false, false));
struct stat st; struct stat st;
EXPECT_EQ_INT(stat("fuzz_meta_test.txt", &st), 0); EXPECT_EQ_INT(stat("fuzz_meta_test.txt", &st), 0);
+26
View File
@@ -1,6 +1,8 @@
#include "test_log.h" #include "test_log.h"
#include "log.h" #include "log.h"
#include "test_utils.h" #include "test_utils.h"
#include <string.h>
#include <unistd.h>
/* Test default log level: WARNING and ERROR should print, DEBUG and INFO should not. /* Test default log level: WARNING and ERROR should print, DEBUG and INFO should not.
* We can't easily capture stderr in unit tests, so we verify the functions don't crash * We can't easily capture stderr in unit tests, so we verify the functions don't crash
@@ -90,6 +92,29 @@ static void test_log_filtering() {
EXPECT_TRUE(true); EXPECT_TRUE(true);
} }
static void test_log_stderr_mode_all() {
int pipe_fds[2];
EXPECT_EQ_INT(pipe(pipe_fds), 0);
int saved_stderr = dup(STDERR_FILENO);
EXPECT_TRUE(saved_stderr >= 0);
EXPECT_TRUE(dup2(pipe_fds[1], STDERR_FILENO) >= 0);
close(pipe_fds[1]);
set_log_level(LOG_LEVEL_WARNING);
log_set_stderr_mode(LOG_STDERR_ALL);
log_message(LOG_LEVEL_WARNING, "warning routed to stderr");
fflush(stderr);
EXPECT_TRUE(dup2(saved_stderr, STDERR_FILENO) >= 0);
close(saved_stderr);
char output[128] = {0};
ssize_t length = read(pipe_fds[0], output, sizeof(output) - 1);
close(pipe_fds[0]);
EXPECT_TRUE(length > 0);
EXPECT_TRUE(strstr(output, "warning routed to stderr") != NULL);
log_set_stderr_mode(LOG_STDERR_ERRORS);
}
/* Test that log_message handles various format strings */ /* Test that log_message handles various format strings */
static void test_log_message_formats() { static void test_log_message_formats() {
set_log_level(LOG_LEVEL_DEBUG); set_log_level(LOG_LEVEL_DEBUG);
@@ -112,5 +137,6 @@ void test_log() {
test_log_set_level_info(); test_log_set_level_info();
test_log_set_level_error(); test_log_set_level_error();
test_log_filtering(); test_log_filtering();
test_log_stderr_mode_all();
test_log_message_formats(); test_log_message_formats();
} }
+69 -2
View File
@@ -1,4 +1,5 @@
#include "test_metadata.h" #include "test_metadata.h"
#include "chmod.h"
#include "metadata.h" #include "metadata.h"
#include "protocol.h" #include "protocol.h"
#include "test_utils.h" #include "test_utils.h"
@@ -122,10 +123,22 @@ static void test_metadata_rejects_invalid_values() {
close(p[1]); close(p[1]);
} }
static void test_metadata_mtime_window() {
EXPECT_TRUE(metadata_mtime_matches(100, 100000000, 101, 600000000, 2));
EXPECT_FALSE(metadata_mtime_matches(100, 100000000, 102, 600000000, 2));
EXPECT_TRUE(metadata_mtime_matches(100, 100000000, 102, 100000000, 2));
EXPECT_TRUE(metadata_mtime_matches(100, 900000000, 102, 100000000, 2));
EXPECT_FALSE(metadata_mtime_matches(100, 100000000, 102, 900000000, 2));
EXPECT_TRUE(metadata_mtime_matches(100, 900000000, 102, 900000000, 2));
EXPECT_FALSE(metadata_mtime_matches(100, 900000000, 101, 100000001, 0));
EXPECT_TRUE(metadata_mtime_matches(100, 100000000, 100, 100000001, 0));
EXPECT_TRUE(metadata_mtime_matches(100, 100000000, 100, 100000000, 0));
}
static void test_file_restore_metadata() { static void test_file_restore_metadata() {
const char* path = "temp_meta_restore_test.txt"; const char* path = "temp_meta_restore_test.txt";
const char* content = "test content"; const char* content = "test content";
EXPECT_TRUE(to_disk(path, content, strlen(content), false, false)); EXPECT_TRUE(file_write_to_disk(path, content, strlen(content), false, false));
FileMetadata m; FileMetadata m;
m.mode = 0644; m.mode = 0644;
@@ -134,7 +147,7 @@ static void test_file_restore_metadata() {
m.mtime_sec = 1234567890; m.mtime_sec = 1234567890;
m.mtime_nsec = 0; m.mtime_nsec = 0;
file_restore_metadata(path, &m); file_restore_metadata(path, &m, false);
struct stat st; struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0); EXPECT_EQ_INT(stat(path, &st), 0);
@@ -144,6 +157,56 @@ static void test_file_restore_metadata() {
unlink(path); unlink(path);
} }
static void test_file_restore_executability_only() {
const char* path = "temp_exec_restore_test.txt";
EXPECT_TRUE(file_write_to_disk(path, "x", 1, false, false));
EXPECT_EQ_INT(chmod(path, 0644), 0);
FileMetadata m = {
.mode = 0751, .uid = getuid(), .gid = getgid(), .mtime_sec = 0, .mtime_nsec = 0};
file_restore_metadata(path, &m, true);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT(st.st_mode & 0777, 0755);
unlink(path);
}
static void test_directory_restore_executability_only() {
const char* path = "temp_exec_restore_test_dir";
EXPECT_EQ_INT(mkdir(path, 0700), 0);
FileMetadata m = {
.mode = 0755, .uid = getuid(), .gid = getgid(), .mtime_sec = 0, .mtime_nsec = 0};
file_restore_metadata(path, &m, true);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT(st.st_mode & 0777, 0711);
rmdir(path);
}
static void test_chmod_changes() {
mode_t result;
EXPECT_TRUE(chmod_apply(0777, "u=rw,go=r", &result));
EXPECT_EQ_INT(result, 0644);
EXPECT_TRUE(chmod_apply(0644, "a+x", &result));
EXPECT_EQ_INT(result, 0755);
result = 0777;
EXPECT_TRUE(chmod_apply(0777, "0000", &result));
EXPECT_EQ_INT(result, 0000);
result = 0777;
EXPECT_TRUE(chmod_apply(0777, "7777", &result));
EXPECT_EQ_INT(result, 07777);
result = 0777;
EXPECT_TRUE(chmod_apply(0777, "755", &result));
EXPECT_EQ_INT(result, 0755);
EXPECT_FALSE(chmod_apply(0777, "888", &result));
EXPECT_FALSE(chmod_apply(0777, "10000", &result));
EXPECT_FALSE(chmod_apply(0777, "a+X", &result));
EXPECT_FALSE(chmod_apply(0777, "a+r,", &result));
}
void test_metadata() { void test_metadata() {
test_metadata_to_from_buf_roundtrip(); test_metadata_to_from_buf_roundtrip();
test_metadata_to_buf_null(); test_metadata_to_buf_null();
@@ -151,5 +214,9 @@ void test_metadata() {
test_metadata_send_receive_roundtrip(); test_metadata_send_receive_roundtrip();
test_metadata_send_null(); test_metadata_send_null();
test_metadata_rejects_invalid_values(); test_metadata_rejects_invalid_values();
test_metadata_mtime_window();
test_file_restore_metadata(); test_file_restore_metadata();
test_file_restore_executability_only();
test_directory_restore_executability_only();
test_chmod_changes();
} }
+82
View File
@@ -250,6 +250,87 @@ static void test_write_thread_done() {
config_delete(cfg); config_delete(cfg);
} }
typedef struct {
PipelineContextReceiver* context;
File* file;
atomic_bool* done;
atomic_bool* result;
} ByteBudgetEnqueueArg;
static int byte_budget_enqueue_worker(void* arg) {
ByteBudgetEnqueueArg* worker = arg;
bool ok = pipeline_context_receiver_enqueue_file(worker->context, worker->file);
atomic_store(worker->result, ok);
atomic_store(worker->done, true);
return thrd_success;
}
/* A receiver must not buffer more decompressed/copied payload bytes ahead of
the (slow) disk writer than the configured byte budget: an enqueue that
would exceed the budget blocks until the writer releases bytes. */
static void test_receiver_enqueue_byte_budget() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
free(cfg->version);
cfg->version = str_dup(PROTOCOL_VERSION);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
cfg->save_to_disk = false;
Queue* q = queue_create(16, file_destroy);
EXPECT_NOT_NULL(q);
PipelineContextReceiver* ctx = pipeline_context_receiver_create(cfg, q, -1, NULL);
EXPECT_NOT_NULL(ctx);
pipeline_context_receiver_set_queue_byte_limit(ctx, 3000);
ctx->receiver_done = false;
File* first = file_create("budget_file_1");
EXPECT_NOT_NULL(first);
first->data->size = 2000;
EXPECT_TRUE(pipeline_context_receiver_enqueue_file(ctx, first));
EXPECT_EQ_INT((int)ctx->queued_bytes, 2000);
/* Second 2000-byte payload would push the pipeline to 4000 > 3000 budget,
so the enqueue must block until the first payload is released. */
File* second = file_create("budget_file_2");
EXPECT_NOT_NULL(second);
second->data->size = 2000;
atomic_bool done;
atomic_bool result;
atomic_init(&done, false);
atomic_init(&result, false);
ByteBudgetEnqueueArg arg = {ctx, second, &done, &result};
thrd_t enqueuer;
EXPECT_EQ_INT(thrd_create(&enqueuer, byte_budget_enqueue_worker, &arg), thrd_success);
/* Give a broken (unbounded) implementation every chance to enqueue. */
struct timespec wait = {0, 200 * 1000000L};
thrd_sleep(&wait, NULL);
EXPECT_FALSE(atomic_load(&done));
EXPECT_EQ_INT((int)ctx->queued_bytes, 2000); /* budget still honored */
/* Simulate the disk writer: dequeue + destroy + release the first file. */
File* drained = queue_dequeue_multithreaded(q, &ctx->mutex, &ctx->condition_not_empty,
&ctx->condition_not_full, &ctx->receiver_done);
EXPECT_NOT_NULL(drained);
file_destroy(drained);
pipeline_context_receiver_note_bytes_released(ctx, 2000);
EXPECT_EQ_INT((int)ctx->queued_bytes, 0);
EXPECT_EQ_INT(thrd_join(enqueuer, NULL), thrd_success);
EXPECT_TRUE(atomic_load(&done));
EXPECT_TRUE(atomic_load(&result));
EXPECT_EQ_INT((int)ctx->queued_bytes, 2000); /* second payload now in flight */
/* Tear down: the second file is still queued and is freed by queue_destroy. */
mtx_destroy(&ctx->mutex);
cnd_destroy(&ctx->condition_not_full);
cnd_destroy(&ctx->condition_not_empty);
free(ctx);
queue_destroy(q);
config_delete(cfg);
}
void test_multiprocessing() { void test_multiprocessing() {
test_sender_create_destroy(); test_sender_create_destroy();
test_receiver_create_destroy(); test_receiver_create_destroy();
@@ -261,4 +342,5 @@ void test_multiprocessing() {
test_receive_thread_failure_wakes_writer(); test_receive_thread_failure_wakes_writer();
} }
test_write_thread_done(); test_write_thread_done();
test_receiver_enqueue_byte_budget();
} }
+5 -1
View File
@@ -14,6 +14,8 @@
static Data* random_data(int min_size, int max_size) { static Data* random_data(int min_size, int max_size) {
int size = min_size + rand() % (max_size - min_size + 1); int size = min_size + rand() % (max_size - min_size + 1);
char* buf = malloc(size); char* buf = malloc(size);
if (!buf)
return NULL;
for (int i = 0; i < size; i++) for (int i = 0; i < size; i++)
buf[i] = (char)(rand() % 256); buf[i] = (char)(rand() % 256);
return data_create(buf, size); return data_create(buf, size);
@@ -81,10 +83,12 @@ static void test_property_chunk_roundtrip() {
int content_len = 1 + rand() % 4096; int content_len = 1 + rand() % 4096;
char* content = malloc(content_len); char* content = malloc(content_len);
if (!content)
return;
for (int i = 0; i < content_len; i++) for (int i = 0; i < content_len; i++)
content[i] = (char)(rand() % 256); content[i] = (char)(rand() % 256);
to_disk(path, content, content_len, false, false); file_write_to_disk(path, content, content_len, false, false);
struct stat st; struct stat st;
stat(path, &st); stat(path, &st);
+233
View File
@@ -3,6 +3,60 @@
#include <limits.h> #include <limits.h>
#include <string.h> #include <string.h>
#include <unistd.h> #include <unistd.h>
#include <threads.h>
typedef struct {
ProtocolSession* session;
bool allocation_allowed;
} AllocationWorkerArg;
static int allocation_worker(void* arg) {
AllocationWorkerArg* worker = arg;
protocol_session_bind(worker->session);
void* allocation = protocol_alloc(8);
worker->allocation_allowed = allocation != NULL;
free(allocation);
protocol_session_unbind();
return thrd_success;
}
typedef struct {
ProtocolSession* session;
int read_fd;
bool released;
} AccountingWorkerArg;
typedef struct {
ProtocolSession* session;
atomic_int* ready;
atomic_bool* release;
bool received;
} ConcurrentAccountingWorkerArg;
static int accounting_worker(void* arg) {
AccountingWorkerArg* worker = arg;
protocol_session_bind(worker->session);
Data* data = protocol_receive_data_limited(worker->session, 8);
if (data) {
data_destroy(data);
worker->released = atomic_load(&worker->session->total_allocated_bytes) == 0;
}
protocol_session_unbind();
return data ? thrd_success : thrd_error;
}
static int concurrent_accounting_worker(void* arg) {
ConcurrentAccountingWorkerArg* worker = arg;
protocol_session_bind(worker->session);
Data* data = protocol_receive_data_limited(worker->session, 8);
worker->received = data != NULL;
atomic_fetch_add(worker->ready, 1);
while (!atomic_load(worker->release))
thrd_yield();
data_destroy(data);
protocol_session_unbind();
return thrd_success;
}
static void test_send_receive_n_data() { static void test_send_receive_n_data() {
int p[2]; int p[2];
@@ -187,6 +241,177 @@ static void test_receive_str_truncated() {
close(p[0]); close(p[0]);
} }
static void test_max_alloc_rejects_single_buffer() {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
ProtocolSession session;
protocol_session_init(&session, p[0], p[1]);
protocol_session_set_max_alloc(&session, 4);
protocol_session_bind(&session);
char payload[8] = {0};
EXPECT_TRUE(write(p[1], &(size_t){sizeof(payload)}, sizeof(size_t)) == sizeof(size_t));
EXPECT_NULL(protocol_receive_str(&session));
protocol_session_unbind();
close(p[0]);
close(p[1]);
}
static void test_explicit_session_max_alloc_cannot_be_bypassed() {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
ProtocolSession explicit_session;
ProtocolSession unrelated_session;
protocol_session_init(&explicit_session, p[0], p[1]);
protocol_session_init(&unrelated_session, p[0], p[1]);
protocol_session_set_max_alloc(&explicit_session, 4);
protocol_session_set_max_alloc(&unrelated_session, 64);
protocol_session_bind(&unrelated_session);
unsigned long long size = 8;
EXPECT_EQ_INT((int)write(p[1], &size, sizeof(size)), (int)sizeof(size));
EXPECT_EQ_INT((int)write(p[1], "12345678", 8), 8);
EXPECT_NULL(protocol_receive_data_limited(&explicit_session, 8));
EXPECT_EQ_INT((int)atomic_load(&explicit_session.total_allocated_bytes), 0);
protocol_session_unbind();
close(p[0]);
close(p[1]);
}
static void test_max_alloc_allows_configured_buffer() {
ProtocolSession session;
protocol_session_init(&session, -1, -1);
protocol_session_set_max_alloc(&session, 4);
protocol_session_bind(&session);
void* allowed = protocol_alloc(4);
const void* rejected = protocol_alloc(5);
EXPECT_NOT_NULL(allowed);
EXPECT_NULL(rejected);
free(allowed);
protocol_session_unbind();
}
static void test_max_alloc_is_bound_in_worker_threads() {
enum { WORKER_COUNT = 4 };
ProtocolSession sessions[WORKER_COUNT];
AllocationWorkerArg args[WORKER_COUNT] = {0};
thrd_t threads[WORKER_COUNT];
for (int i = 0; i < WORKER_COUNT; i++) {
protocol_session_init(&sessions[i], -1, -1);
protocol_session_set_max_alloc(&sessions[i], 4);
args[i].session = &sessions[i];
EXPECT_EQ_INT(thrd_create(&threads[i], allocation_worker, &args[i]), thrd_success);
}
for (int i = 0; i < WORKER_COUNT; i++) {
int result;
EXPECT_EQ_INT(thrd_join(threads[i], &result), thrd_success);
EXPECT_EQ_INT(result, thrd_success);
EXPECT_FALSE(args[i].allocation_allowed);
}
}
static void test_protocol_accounting_is_released_in_worker_threads() {
enum { WORKER_COUNT = 4 };
ProtocolSession sessions[WORKER_COUNT];
AccountingWorkerArg args[WORKER_COUNT] = {0};
thrd_t threads[WORKER_COUNT];
for (int i = 0; i < WORKER_COUNT; i++) {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
protocol_session_init(&sessions[i], p[0], p[1]);
protocol_session_set_max_alloc(&sessions[i], 64);
unsigned long long size = 8;
EXPECT_EQ_INT((int)write(p[1], &size, sizeof(size)), (int)sizeof(size));
EXPECT_EQ_INT((int)write(p[1], "12345678", 8), 8);
close(p[1]);
args[i].session = &sessions[i];
args[i].read_fd = p[0];
EXPECT_EQ_INT(thrd_create(&threads[i], accounting_worker, &args[i]), thrd_success);
}
for (int i = 0; i < WORKER_COUNT; i++) {
int result;
EXPECT_EQ_INT(thrd_join(threads[i], &result), thrd_success);
EXPECT_EQ_INT(result, thrd_success);
EXPECT_TRUE(args[i].released);
EXPECT_EQ_INT((int)atomic_load(&sessions[i].total_allocated_bytes), 0);
close(args[i].read_fd);
}
}
static void test_protocol_accounting_reservation_is_atomic() {
enum { WORKER_COUNT = 8 };
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
ProtocolSession session;
protocol_session_init(&session, p[0], p[1]);
protocol_session_set_max_alloc(&session, 64);
const unsigned long long budget_before = MAX_SERVER_ALLOC - 8;
atomic_store(&session.total_allocated_bytes, budget_before);
for (int i = 0; i < WORKER_COUNT; i++) {
unsigned long long size = 8;
EXPECT_EQ_INT((int)write(p[1], &size, sizeof(size)), (int)sizeof(size));
EXPECT_EQ_INT((int)write(p[1], "12345678", 8), 8);
}
close(p[1]);
atomic_int ready;
atomic_bool release;
atomic_init(&ready, 0);
atomic_init(&release, false);
ConcurrentAccountingWorkerArg args[WORKER_COUNT] = {0};
thrd_t threads[WORKER_COUNT];
for (int i = 0; i < WORKER_COUNT; i++) {
args[i].session = &session;
args[i].ready = &ready;
args[i].release = &release;
EXPECT_EQ_INT(thrd_create(&threads[i], concurrent_accounting_worker, &args[i]), thrd_success);
}
while (atomic_load(&ready) != WORKER_COUNT)
thrd_yield();
bool budget_ok = atomic_load(&session.total_allocated_bytes) == budget_before + 8;
atomic_store(&release, true);
int received = 0;
for (int i = 0; i < WORKER_COUNT; i++) {
int result;
EXPECT_EQ_INT(thrd_join(threads[i], &result), thrd_success);
EXPECT_EQ_INT(result, thrd_success);
received += args[i].received ? 1 : 0;
}
EXPECT_EQ_INT(received, 1);
EXPECT_TRUE(budget_ok);
EXPECT_EQ_INT((int)atomic_load(&session.total_allocated_bytes), (int)budget_before);
close(p[0]);
}
static void test_protocol_string_accounting_is_transient() {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
ProtocolSession session;
protocol_session_init(&session, p[0], p[1]);
protocol_session_set_max_alloc(&session, 64);
EXPECT_TRUE(protocol_send_str(&session, "temporary"));
char* received = protocol_receive_str(&session);
EXPECT_NOT_NULL(received);
EXPECT_EQ_STR(received, "temporary");
EXPECT_EQ_INT((int)atomic_load(&session.total_allocated_bytes), 0);
free(received);
close(p[0]);
close(p[1]);
}
static void test_protocol_accounting_release_does_not_underflow() {
ProtocolSession session;
protocol_session_init(&session, -1, -1);
atomic_store(&session.total_allocated_bytes, 4);
protocol_session_bind(&session);
protocol_release_memory(8);
EXPECT_EQ_INT((int)atomic_load(&session.total_allocated_bytes), 0);
protocol_release_memory(1);
EXPECT_EQ_INT((int)atomic_load(&session.total_allocated_bytes), 0);
protocol_session_unbind();
}
void test_protocol() { void test_protocol() {
test_send_receive_n_data(); test_send_receive_n_data();
test_send_receive_n_data_zero(); test_send_receive_n_data_zero();
@@ -198,4 +423,12 @@ void test_protocol() {
test_send_receive_status(); test_send_receive_status();
test_receive_n_data_truncated(); test_receive_n_data_truncated();
test_receive_str_truncated(); test_receive_str_truncated();
test_max_alloc_rejects_single_buffer();
test_explicit_session_max_alloc_cannot_be_bypassed();
test_max_alloc_allows_configured_buffer();
test_max_alloc_is_bound_in_worker_threads();
test_protocol_accounting_is_released_in_worker_threads();
test_protocol_accounting_reservation_is_atomic();
test_protocol_string_accounting_is_transient();
test_protocol_accounting_release_does_not_underflow();
} }
+4
View File
@@ -122,6 +122,8 @@ static void test_queue_destroyer() {
for (int i = 0; i < 3; i++) { for (int i = 0; i < 3; i++) {
int* val = malloc(sizeof(int)); int* val = malloc(sizeof(int));
if (!val)
break;
*val = i; *val = i;
queue_enqueue(q, val); queue_enqueue(q, val);
} }
@@ -181,6 +183,8 @@ static void test_queue_multithreaded() {
for (int i = 1; i <= 100; i++) { for (int i = 1; i <= 100; i++) {
int* val = malloc(sizeof(int)); int* val = malloc(sizeof(int));
if (!val)
break;
*val = i; *val = i;
queue_enqueue_multithreaded(q, val, &mutex, &cnd_empty, &cnd_full); queue_enqueue_multithreaded(q, val, &mutex, &cnd_empty, &cnd_full);
} }
+16 -1
View File
@@ -13,7 +13,7 @@
static void test_chunk_deserialize_truncated() { static void test_chunk_deserialize_truncated() {
char* path = "test_rob_trunc.txt"; char* path = "test_rob_trunc.txt";
char* content = "hello"; char* content = "hello";
to_disk(path, content, strlen(content), false, false); file_write_to_disk(path, content, strlen(content), false, false);
struct stat st; struct stat st;
stat(path, &st); stat(path, &st);
@@ -109,6 +109,20 @@ static void test_delta_deserialize_garbage() {
data_destroy(d); data_destroy(d);
} }
static void test_delta_deserialize_respects_max_alloc() {
unsigned char serialized[sizeof(uint64_t) + sizeof(uint32_t)] = {0};
Data data = {.data = serialized, .size = sizeof(serialized)};
ProtocolSession session;
protocol_session_init(&session, -1, -1);
protocol_session_set_max_alloc(&session, sizeof(Delta) - 1);
protocol_session_bind(&session);
const Delta* result = delta_deserialize(&data);
EXPECT_NULL(result);
protocol_session_unbind();
}
static void test_delta_signature_deserialize_truncated() { static void test_delta_signature_deserialize_truncated() {
char old_data[4096]; char old_data[4096];
for (int i = 0; i < 4096; i++) for (int i = 0; i < 4096; i++)
@@ -206,6 +220,7 @@ void test_robustness() {
test_delta_deserialize_truncated(); test_delta_deserialize_truncated();
test_delta_deserialize_empty(); test_delta_deserialize_empty();
test_delta_deserialize_garbage(); test_delta_deserialize_garbage();
test_delta_deserialize_respects_max_alloc();
test_delta_deserialize_truncated_instructions(); test_delta_deserialize_truncated_instructions();
test_delta_signature_deserialize_truncated(); test_delta_signature_deserialize_truncated();
test_delta_apply_null(); test_delta_apply_null();
+31 -1
View File
@@ -7,7 +7,7 @@
#include <unistd.h> #include <unistd.h>
static void create_test_file(const char* path, const char* content) { static void create_test_file(const char* path, const char* content) {
(void)to_disk(path, content, strlen(content), false, false); (void)file_write_to_disk(path, content, strlen(content), false, false);
} }
static void test_scanner_single_file() { static void test_scanner_single_file() {
@@ -385,6 +385,35 @@ static void test_scanner_no_patterns() {
rmdir(dir); rmdir(dir);
} }
static void test_parallel_scanner_root_chunks_without_workers() {
const char* dir = "test_parallel_scan_root";
const char* file1 = "test_parallel_scan_root/a.txt";
const char* file2 = "test_parallel_scan_root/b.txt";
EXPECT_EQ_INT(mkdir(dir, 0755), 0);
create_test_file(file1, "a");
create_test_file(file2, "b");
ScannerOptions options = {false, 1, NULL, 0, NULL, 0, 0, 0,
0, 0, false, false, false, false, false};
ParallelScanner* scanner = parallel_scanner_create_with_options(dir, &options, NULL);
EXPECT_NOT_NULL(scanner);
int total_files = 0;
Chunk* chunk;
while ((chunk = parallel_scanner_next(scanner)) != NULL) {
total_files += chunk->element_count;
chunk_destroy(chunk);
}
EXPECT_EQ_INT(total_files, 2);
EXPECT_FALSE(parallel_scanner_failed(scanner));
parallel_scanner_destroy(scanner);
unlink(file1);
unlink(file2);
rmdir(dir);
}
void test_scanner() { void test_scanner() {
test_scanner_single_file(); test_scanner_single_file();
test_scanner_multiple_files(); test_scanner_multiple_files();
@@ -399,4 +428,5 @@ void test_scanner() {
test_scanner_size_range(); test_scanner_size_range();
test_scanner_mixed_patterns(); test_scanner_mixed_patterns();
test_scanner_no_patterns(); test_scanner_no_patterns();
test_parallel_scanner_root_chunks_without_workers();
} }
+279
View File
@@ -1,14 +1,18 @@
#include "test_server.h" #include "test_server.h"
#include "config.h" #include "config.h"
#include "delta.h"
#include "file.h" #include "file.h"
#include "protocol.h" #include "protocol.h"
#include "test_utils.h" #include "test_utils.h"
#include "utils.h" #include "utils.h"
#include <fcntl.h>
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <sys/socket.h> #include <sys/socket.h>
#include <sys/stat.h>
#include <sys/wait.h> #include <sys/wait.h>
#include <time.h>
#include <unistd.h> #include <unistd.h>
#include "receiver.h" #include "receiver.h"
@@ -182,11 +186,286 @@ static void test_receive_manifest_rejects_traversal() {
config_delete(cfg); config_delete(cfg);
} }
static void test_receive_incremental_check_rejects_invalid_nanoseconds() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->receive_root_directory = str_dup("/tmp/dst");
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
EXPECT_TRUE(send_str(p[1], "file.txt"));
unsigned long long size = 0;
long long mtime = 100;
long long mtime_nsec = 1000000000LL;
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
bool skipped = false;
EXPECT_NULL(receive_incremental_check(p[0], cfg, &skipped));
Status status;
EXPECT_TRUE(receive_status(p[1], &status));
EXPECT_EQ_INT(status, STATUS_ERROR);
EXPECT_FALSE(skipped);
close(p[0]);
close(p[1]);
config_delete(cfg);
}
static char* make_check_root(const char* tag) {
char tmpl[128];
snprintf(tmpl, sizeof(tmpl), "/tmp/fastsync_%s_XXXXXX", tag);
char* path = str_dup(tmpl);
if (!path)
return NULL;
if (!mkdtemp(path)) {
free(path);
return NULL;
}
return path;
}
static void write_check_file(const char* dir, const char* name, const char* content) {
char path[1024];
snprintf(path, sizeof(path), "%s/%s", dir, name);
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd >= 0) {
size_t len = strlen(content);
if (write(fd, content, len) != (ssize_t)len) {
/* intentionally ignored in tests */
}
close(fd);
}
}
/* Issue #255: a same-size/mtime match is decided from metadata alone, so the
receiver answers STATUS_OK (skip) and never asks for a data body. */
static void test_incremental_check_quick_skip_by_mtime() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
char* root = make_check_root("qskip");
EXPECT_NOT_NULL(root);
cfg->receive_root_directory = str_dup(root);
write_check_file(root, "file.txt", "0123456789abcdef");
char path[1024];
snprintf(path, sizeof(path), "%s/file.txt", root);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
pid_t pid = fork();
if (pid == 0) {
alarm(30);
close(p[1]);
io_set_fds(p[0], p[0]);
bool skipped = false;
File* file = receive_incremental_check(p[0], cfg, &skipped);
bool ok = file == NULL && skipped;
file_destroy(file);
config_delete(cfg);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
EXPECT_TRUE(send_str(p[1], "file.txt"));
unsigned long long size = (unsigned long long)st.st_size;
long long mtime = (long long)st.st_mtime;
long long mtime_nsec = 0;
#ifdef __linux__
mtime_nsec = (long long)st.st_mtim.tv_nsec;
#endif
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_OK);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(cfg);
unlink(path);
rmdir(root);
free(root);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
/* Issue #255: a size mismatch cannot be a skip, so the receiver answers
STATUS_NEXT and consumes the full data body that follows. */
static void test_incremental_check_size_mismatch_full_transfer() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
char* root = make_check_root("qnext");
EXPECT_NOT_NULL(root);
cfg->receive_root_directory = str_dup(root);
write_check_file(root, "file.txt", "0123456789abcdef");
char path[1024];
snprintf(path, sizeof(path), "%s/file.txt", root);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
pid_t pid = fork();
if (pid == 0) {
alarm(30);
close(p[1]);
io_set_fds(p[0], p[0]);
bool skipped = false;
File* file = receive_incremental_check(p[0], cfg, &skipped);
bool ok = file != NULL && !skipped && file->path != NULL && strcmp(file->path, "file.txt") == 0;
file_destroy(file);
config_delete(cfg);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
EXPECT_TRUE(send_str(p[1], "file.txt"));
unsigned long long size = (unsigned long long)st.st_size + 1;
long long mtime = (long long)st.st_mtime;
long long mtime_nsec = 0;
#ifdef __linux__
mtime_nsec = (long long)st.st_mtim.tv_nsec;
#endif
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_NEXT);
Data* body = data_create_reserve(8);
EXPECT_NOT_NULL(body);
body->data = malloc(8);
EXPECT_NOT_NULL(body->data);
memcpy(body->data, "replaced", 8);
body->size = 8;
EXPECT_TRUE(send_data(p[1], body));
data_destroy(body);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(cfg);
unlink(path);
rmdir(root);
free(root);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
/* Issue #256: when a received delta claims a result above the whole-file cap,
receive_delta_file must mark the operation failed so the caller aborts with
STATUS_ERROR instead of emitting STATUS_NEXT and waiting for a body that
never arrives. */
static void test_incremental_check_delta_oversize_reports_failure() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
char* root = make_check_root("qdelta");
EXPECT_NOT_NULL(root);
cfg->receive_root_directory = str_dup(root);
cfg->use_delta = true;
char content[20000];
memset(content, 'a', sizeof(content));
content[sizeof(content) - 1] = '\0';
write_check_file(root, "file.txt", content);
char path[1024];
snprintf(path, sizeof(path), "%s/file.txt", root);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT((int)st.st_size, 19999);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
pid_t pid = fork();
if (pid == 0) {
alarm(30);
close(p[1]);
io_set_fds(p[0], p[0]);
bool skipped = false;
File* file = receive_incremental_check(p[0], cfg, &skipped);
bool ok = file == NULL && !skipped;
if (ok)
send_status(p[0], STATUS_ERROR); /* mirror the server error path */
file_destroy(file);
config_delete(cfg);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
EXPECT_TRUE(send_str(p[1], "file.txt"));
unsigned long long size = (unsigned long long)st.st_size;
long long mtime = 1; /* different from the file mtime: force a transfer */
long long mtime_nsec = 0;
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_DELTA_SIGNATURE);
Data* sig_data = receive_data(p[1]);
EXPECT_NOT_NULL(sig_data);
DeltaSignature* sig = delta_signature_deserialize(sig_data);
EXPECT_NOT_NULL(sig);
delta_signature_destroy(sig);
data_destroy(sig_data);
/* Send a delta whose claimed output size exceeds the whole-file cap. */
Delta delta;
memset(&delta, 0, sizeof(delta));
delta.new_file_size = MAX_RECEIVE_WHOLE_FILE_SIZE + 1;
Data* bogus = delta_serialize(&delta);
EXPECT_NOT_NULL(bogus);
EXPECT_TRUE(send_status(p[1], STATUS_DELTA_DATA));
EXPECT_TRUE(bogus != NULL && send_data(p[1], bogus));
data_destroy(bogus);
/* The receiver must answer with an error, never with STATUS_NEXT. */
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_ERROR);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(cfg);
unlink(path);
rmdir(root);
free(root);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
void test_server() { void test_server() {
if (!is_running_under_valgrind()) { if (!is_running_under_valgrind()) {
test_receive_files_finished(); test_receive_files_finished();
test_receive_files_single_file(); test_receive_files_single_file();
test_receive_files_abort(); test_receive_files_abort();
test_receive_manifest_rejects_traversal(); test_receive_manifest_rejects_traversal();
test_receive_incremental_check_rejects_invalid_nanoseconds();
test_incremental_check_quick_skip_by_mtime();
test_incremental_check_size_mismatch_full_transfer();
test_incremental_check_delta_oversize_reports_failure();
} }
} }
+56
View File
@@ -1,10 +1,66 @@
#include "test_shared_utils.h" #include "test_shared_utils.h"
#include "utils.h" #include "utils.h"
#include "protocol.h"
#include "test_utils.h" #include "test_utils.h"
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <threads.h>
typedef struct {
bool eight_bit_output;
const char* expected;
int failed;
} EscapeThreadArgs;
static int escape_thread(void* arg) {
EscapeThreadArgs* args = arg;
for (int i = 0; i < 1000; i++) {
char* escaped = output_escape("x\xc3\xa9\n", args->eight_bit_output);
if (!escaped || strcmp(escaped, args->expected) != 0)
args->failed = 1;
free(escaped);
}
return 0;
}
void test_shared_utils() { void test_shared_utils() {
char formatted[32];
EXPECT_TRUE(format_human_bytes(0, formatted, sizeof(formatted)));
EXPECT_EQ_STR(formatted, "0 B");
EXPECT_TRUE(format_human_bytes(1024, formatted, sizeof(formatted)));
EXPECT_EQ_STR(formatted, "1.0 KB");
EXPECT_TRUE(format_human_bytes(1536 * 1024, formatted, sizeof(formatted)));
EXPECT_EQ_STR(formatted, "1.5 MB");
EXPECT_FALSE(format_human_bytes(1024, formatted, 4));
char high_bit[] = {'a', (char)0xc3, (char)0xa9, '\n', '\0'};
char* escaped = output_escape(high_bit, false);
EXPECT_EQ_STR(escaped, "a\\#303\\#251\\#012");
free(escaped);
escaped = output_escape(high_bit, true);
EXPECT_EQ_STR(escaped, "a\xc3\xa9\\#012");
free(escaped);
ProtocolSession safe_session;
ProtocolSession eight_bit_session;
protocol_session_init(&safe_session, -1, -1);
protocol_session_init(&eight_bit_session, -1, -1);
protocol_session_set_8_bit_output(&safe_session, false);
protocol_session_set_8_bit_output(&eight_bit_session, true);
EXPECT_FALSE(safe_session.eight_bit_output);
EXPECT_TRUE(eight_bit_session.eight_bit_output);
EscapeThreadArgs safe_args = {false, "x\\#303\\#251\\#012", 0};
EscapeThreadArgs eight_bit_args = {true, "x\xc3\xa9\\#012", 0};
thrd_t safe_thread;
thrd_t eight_bit_thread;
EXPECT_EQ_INT(thrd_create(&safe_thread, escape_thread, &safe_args), thrd_success);
EXPECT_EQ_INT(thrd_create(&eight_bit_thread, escape_thread, &eight_bit_args), thrd_success);
EXPECT_EQ_INT(thrd_join(safe_thread, NULL), thrd_success);
EXPECT_EQ_INT(thrd_join(eight_bit_thread, NULL), thrd_success);
EXPECT_FALSE(safe_args.failed);
EXPECT_FALSE(eight_bit_args.failed);
// Test str_dup // Test str_dup
const char* dup_null = str_dup(NULL); const char* dup_null = str_dup(NULL);
EXPECT_NULL(dup_null); EXPECT_NULL(dup_null);
+6
View File
@@ -32,6 +32,8 @@ static int mpmc_producer_func(void* arg) {
ProducerCtx* ctx = (ProducerCtx*)arg; ProducerCtx* ctx = (ProducerCtx*)arg;
for (int i = 1; i <= ITEMS_PER_PRODUCER; i++) { for (int i = 1; i <= ITEMS_PER_PRODUCER; i++) {
int* val = malloc(sizeof(int)); int* val = malloc(sizeof(int));
if (!val)
return thrd_error;
*val = ctx->producer_id * ITEMS_PER_PRODUCER + i; *val = ctx->producer_id * ITEMS_PER_PRODUCER + i;
queue_enqueue_multithreaded(ctx->q, val, ctx->mutex, ctx->cnd_empty, ctx->cnd_full); queue_enqueue_multithreaded(ctx->q, val, ctx->mutex, ctx->cnd_empty, ctx->cnd_full);
} }
@@ -121,6 +123,8 @@ static int bp_producer_func(void* arg) {
BackpressureCtx* ctx = (BackpressureCtx*)arg; BackpressureCtx* ctx = (BackpressureCtx*)arg;
for (int i = 0; i < 5; i++) { for (int i = 0; i < 5; i++) {
int* val = malloc(sizeof(int)); int* val = malloc(sizeof(int));
if (!val)
return thrd_error;
*val = i + 1; *val = i + 1;
queue_enqueue_multithreaded(ctx->q, val, ctx->mutex, ctx->cnd_empty, ctx->cnd_full); queue_enqueue_multithreaded(ctx->q, val, ctx->mutex, ctx->cnd_empty, ctx->cnd_full);
ctx->items_sent++; ctx->items_sent++;
@@ -194,6 +198,8 @@ static void test_queue_rapid_create_destroy() {
for (int j = 0; j < 3; j++) { for (int j = 0; j < 3; j++) {
int* val = malloc(sizeof(int)); int* val = malloc(sizeof(int));
if (!val)
break;
*val = j; *val = j;
queue_enqueue(q, val); queue_enqueue(q, val);
} }
+33 -13
View File
@@ -4,34 +4,39 @@
static void test_ssh_connect_invalid_dest_no_colon() { static void test_ssh_connect_invalid_dest_no_colon() {
/* cppcheck-suppress constVariablePointer */ /* cppcheck-suppress constVariablePointer */
Client* client = client_connect_ssh("invalid-destination-no-colon", 22, NULL); Client* client = client_connect_ssh("invalid-destination-no-colon", 22, NULL, false);
EXPECT_NULL(client); EXPECT_NULL(client);
} }
static void test_ssh_connect_invalid_dest_empty() { static void test_ssh_connect_invalid_dest_empty() {
/* cppcheck-suppress constVariablePointer */ /* cppcheck-suppress constVariablePointer */
Client* client = client_connect_ssh("", 22, NULL); Client* client = client_connect_ssh("", 22, NULL, false);
EXPECT_NULL(client); EXPECT_NULL(client);
} }
/* Test client_connect_ssh with malformed destination (just a colon). /* A child that cannot exec ssh must not be returned as a successful client. */
* parse_remote_dest succeeds, ssh is exec'd and fails, but the function
* creates a Client that must be cleaned up. */
static void test_ssh_connect_malformed() { static void test_ssh_connect_malformed() {
Client* client = client_connect_ssh(":", 22, NULL); const char* old_path = getenv("PATH");
/* ssh binary exists, so exec succeeds; the function returns a Client. char* saved_path = old_path ? strdup(old_path) : NULL;
* We just verify it doesn't crash and clean up properly. */ setenv("PATH", "", 1);
if (client != NULL) {
client_disconnect(client); /* cppcheck-suppress constVariablePointer */
client_delete(client); Client* client = client_connect_ssh(":", 22, NULL, false);
if (saved_path) {
setenv("PATH", saved_path, 1);
free(saved_path);
} else {
unsetenv("PATH");
} }
EXPECT_TRUE(true);
EXPECT_NULL(client);
} }
/* Test client_connect_ssh with valid format but unreachable host. /* Test client_connect_ssh with valid format but unreachable host.
* The function launches ssh which will fail to connect, returns a Client. */ * The function launches ssh which will fail to connect, returns a Client. */
static void test_ssh_connect_unreachable() { static void test_ssh_connect_unreachable() {
Client* client = client_connect_ssh("nonexistent.invalid:/remote/path", 22, NULL); Client* client = client_connect_ssh("nonexistent.invalid:/remote/path", 22, NULL, false);
if (client != NULL) { if (client != NULL) {
client_disconnect(client); client_disconnect(client);
client_delete(client); client_delete(client);
@@ -39,9 +44,24 @@ static void test_ssh_connect_unreachable() {
EXPECT_TRUE(true); EXPECT_TRUE(true);
} }
static void test_ssh_remote_command_argument_modes() {
char* command = ssh_build_remote_command("fast sync; touch /tmp/pwned", false);
EXPECT_EQ_STR(command, "'fast sync; touch /tmp/pwned' --stdio");
free(command);
command = ssh_build_remote_command("fast'sync", false);
EXPECT_EQ_STR(command, "'fast'\\''sync' --stdio");
free(command);
command = ssh_build_remote_command("fast sync; touch /tmp/pwned", true);
EXPECT_EQ_STR(command, "fast sync; touch /tmp/pwned --stdio");
free(command);
}
void test_transport_ssh() { void test_transport_ssh() {
test_ssh_connect_invalid_dest_no_colon(); test_ssh_connect_invalid_dest_no_colon();
test_ssh_connect_invalid_dest_empty(); test_ssh_connect_invalid_dest_empty();
test_ssh_connect_malformed(); test_ssh_connect_malformed();
test_ssh_connect_unreachable(); test_ssh_connect_unreachable();
test_ssh_remote_command_argument_modes();
} }