Loader thread loads all file data into memory causing high memory usage #107

New Issue

2026-07-21T16:20:18+02:00

TapTap commented

2026-07-21 16:20:18 +02:00

Description

load_files_multithreaded() calls file_load_data() which reads the entire file into a mallocd buffer (up to STREAM_THRESHOLD = 64MB). With queue depth 100 chunks of ~10MB each, memory usage can reach ~1GB.

Location

src/client/client_send.c:331-357 (load_files_multithreaded)
src/shared/file.c:92-119 (file_load_data)

Suggested Fix

Use mmap() instead of malloc() + read() for files below STREAM_THRESHOLD — this allows the OS to page data in/out
Reduce default queue depth for memory-constrained environments
Use a memory budget rather than fixed queue slot count

Severity

high

Sub-Agent

perf-analyst (PA-2)

This issue was automatically generated by the issue-creator agent.

## Description `load_files_multithreaded()` calls `file_load_data()` which reads the entire file into a mallocd buffer (up to STREAM_THRESHOLD = 64MB). With queue depth 100 chunks of ~10MB each, memory usage can reach ~1GB. ## Location src/client/client_send.c:331-357 (load_files_multithreaded) src/shared/file.c:92-119 (file_load_data) ## Suggested Fix 1. Use `mmap()` instead of `malloc()` + `read()` for files below STREAM_THRESHOLD — this allows the OS to page data in/out 2. Reduce default queue depth for memory-constrained environments 3. Use a memory budget rather than fixed queue slot count ## Severity high ## Category performance ## Sub-Agent perf-analyst (PA-2) --- _This issue was automatically generated by the issue-creator agent._

TapTap referenced this issue

2026-07-21 17:00:03 +02:00

Performance: multi-threaded scanner, adaptive queue, compression skip, streaming, bw throttle (#107-#111) #141

TapTap closed this issue

2026-07-29 18:27:17 +02:00

TapTap commented

2026-07-29 18:27:33 +02:00