Single-threaded BFS scanner is a bottleneck for large directory trees #109

New Issue

2026-07-21T16:20:25+02:00

TapTap commented

2026-07-21 16:20:25 +02:00

Description

The scanner runs in a single thread. For filesystems with millions of files in deep trees, opendir/readdir/lstat per entry is I/O bound and cannot be parallelized.

Location

src/client/scanner.c

Suggested Fix

Use multiple scanner threads each handling different subdirectories (root→queue→workers pattern). Or use fts(3) / nftw(3) which can be faster than manual BFS with opendir/readdir/lstat.

Severity

medium

Sub-Agent

perf-analyst (PA-4)

This issue was automatically generated by the issue-creator agent.

## Description The scanner runs in a single thread. For filesystems with millions of files in deep trees, `opendir`/`readdir`/`lstat` per entry is I/O bound and cannot be parallelized. ## Location src/client/scanner.c ## Suggested Fix Use multiple scanner threads each handling different subdirectories (root→queue→workers pattern). Or use `fts(3)` / `nftw(3)` which can be faster than manual BFS with opendir/readdir/lstat. ## Severity medium ## Category performance ## Sub-Agent perf-analyst (PA-4) --- _This issue was automatically generated by the issue-creator agent._

TapTap referenced this issue

2026-07-21 17:00:03 +02:00

Performance: multi-threaded scanner, adaptive queue, compression skip, streaming, bw throttle (#107-#111) #141

TapTap closed this issue

2026-07-29 18:27:17 +02:00

TapTap commented

2026-07-29 18:27:33 +02:00