Incremental check causes per-file network round-trip -- batch it #106

New Issue

2026-07-21T16:20:16+02:00

TapTap commented

2026-07-21 16:20:16 +02:00

Description

incremental_check() sends one request per file and waits for a reply. For millions of small unchanged files, this adds a full network round-trip per file. On high-latency links this destroys throughput.

Location

src/client/client_send.c:25-62 (incremental_check)
src/shared/file.c:442-560 (receive_incremental_check)

Current behavior

Client sends STATUS_CHECK + (path, size, mtime) per file. Server replies STATUS_OK (unchanged) or STATUS_NEXT (send file). For 1M unchanged files on a 100ms RTT link, this takes ~28 hours of round-trips.

Suggested Fix

Batch incremental checks: design a STATUS_CHECK_BATCH message type that sends an array of (path, size, mtime) tuples and receives a bitmap of which files changed. This reduces the O(n) round trips to O(1).

Expected impact

On high-latency links, batching could reduce incremental check time from hours to seconds.

Severity

high

Sub-Agent

perf-analyst (PA-1)

This issue was automatically generated by the issue-creator agent.

## Description `incremental_check()` sends one request per file and waits for a reply. For millions of small unchanged files, this adds a full network round-trip per file. On high-latency links this destroys throughput. ## Location src/client/client_send.c:25-62 (incremental_check) src/shared/file.c:442-560 (receive_incremental_check) ## Current behavior Client sends STATUS_CHECK + (path, size, mtime) per file. Server replies STATUS_OK (unchanged) or STATUS_NEXT (send file). For 1M unchanged files on a 100ms RTT link, this takes ~28 hours of round-trips. ## Suggested Fix Batch incremental checks: design a `STATUS_CHECK_BATCH` message type that sends an array of (path, size, mtime) tuples and receives a bitmap of which files changed. This reduces the O(n) round trips to O(1). ## Expected impact On high-latency links, batching could reduce incremental check time from hours to seconds. ## Severity high ## Category performance ## Sub-Agent perf-analyst (PA-1) --- _This issue was automatically generated by the issue-creator agent._

TapTap referenced this issue from a commit

2026-07-21 16:47:11 +02:00

feat: protocol improvements (#100-#102, #106)

TapTap referenced this issue

2026-07-21 16:48:31 +02:00

Protocol improvements: keepalive, abort, atomic writes, batch incremental check (#100-#102, #106) #140

TapTap referenced this issue

2026-07-21 18:01:08 +02:00

Merge all 5 batch PRs: security, CLI features, protocol, performance, tests/docs #143

TapTap referenced this issue

2026-07-29 18:11:26 +02:00

Merge all 5 batch PRs: security, CLI features, protocol, performance, tests/docs #146