performance: entire file content loaded into RAM -- no streaming for large files #40

New Issue

2026-07-20T14:58:39+02:00

TapTap commented

2026-07-20 14:58:39 +02:00

FastSync loads entire file contents into memory before sending. This is visible in several locations:

src/client/client_send.c:294 -- file_load_data reads the full file into a Data buffer
src/shared/file.c:505-518 -- file_content_to_buffer uses fread to read the entire file into a single mallocd buffer
src/shared/file.c:436-468 -- file_send_sendfile is the exception using sendfile with zero-copy, but it is incompatible with compression and chunk serialization

This means:

A 10 GB file requires 10 GB of RAM
Multiple large files in a chunk consume even more memory
Partial reading/streaming is not possible
Systems with limited RAM cannot transfer files larger than available memory
The sendfile path (which avoids this problem) cannot be combined with compression

For comparison, rsync can stream files with minimal memory overhead, especially when combined with compression.

Suggested Fix: Add a streaming send path for individual files that reads and sends in fixed-size blocks rather than loading the entire file. This would also enable streaming compression.

Severity: medium

FastSync loads entire file contents into memory before sending. This is visible in several locations: 1. src/client/client_send.c:294 -- file_load_data reads the full file into a Data buffer 2. src/shared/file.c:505-518 -- file_content_to_buffer uses fread to read the entire file into a single mallocd buffer 3. src/shared/file.c:436-468 -- file_send_sendfile is the exception using sendfile with zero-copy, but it is incompatible with compression and chunk serialization This means: - A 10 GB file requires 10 GB of RAM - Multiple large files in a chunk consume even more memory - Partial reading/streaming is not possible - Systems with limited RAM cannot transfer files larger than available memory - The sendfile path (which avoids this problem) cannot be combined with compression For comparison, rsync can stream files with minimal memory overhead, especially when combined with compression. Suggested Fix: Add a streaming send path for individual files that reads and sends in fixed-size blocks rather than loading the entire file. This would also enable streaming compression. Severity: medium

TapTap changed title from ~~performance: entire file content loaded into RAM — no streaming for large files~~ to performance: entire file content loaded into RAM -- no streaming for large files

2026-07-20 14:58:44 +02:00

TapTap referenced this issue from a commit

2026-07-20 18:25:17 +02:00

fix: features and enhancements — issues #70, #36, #34, #33, #32, #57, #40, #37

TapTap referenced this issue

2026-07-20 18:25:20 +02:00

Features and enhancements (#70, #36, #34, #33, #32, #57, #40, #37) #77

TapTap referenced this issue from a commit

2026-07-20 18:30:12 +02:00

fix: features and enhancements — issues #70, #36, #34, #33, #32, #57, #40, #37

TapTap referenced this issue from a commit

2026-07-20 18:37:01 +02:00