Fixed queue capacity 100 may cause pipeline stalls for varied file sizes #108

New Issue

2026-07-21T16:20:24+02:00

TapTap commented

2026-07-21 16:20:24 +02:00

Description

With capacity 100 in both queues and ~10MB chunks, the pipeline can hold ~1GB of data in flight. But for many small files, the scanner fills the queue quickly and blocks, then the loader drains it. The fixed size does not adapt to file sizes, causing unnecessary stalls.

Location

src/client/client_send.c:521-522

Suggested Fix

Make queue capacity dynamic based on total bytes in queue rather than item count. Or use a two-threshold system: minimum capacity 100 but grow as needed up to a max memory limit.

Severity

medium

Sub-Agent

perf-analyst (PA-3)

This issue was automatically generated by the issue-creator agent.

## Description With capacity 100 in both queues and ~10MB chunks, the pipeline can hold ~1GB of data in flight. But for many small files, the scanner fills the queue quickly and blocks, then the loader drains it. The fixed size does not adapt to file sizes, causing unnecessary stalls. ## Location src/client/client_send.c:521-522 ## Suggested Fix Make queue capacity dynamic based on total bytes in queue rather than item count. Or use a two-threshold system: minimum capacity 100 but grow as needed up to a max memory limit. ## Severity medium ## Category performance ## Sub-Agent perf-analyst (PA-3) --- _This issue was automatically generated by the issue-creator agent._

TapTap referenced this issue

2026-07-21 17:00:03 +02:00

Performance: multi-threaded scanner, adaptive queue, compression skip, streaming, bw throttle (#107-#111) #141

TapTap closed this issue

2026-07-29 18:27:17 +02:00

TapTap commented

2026-07-29 18:27:33 +02:00