FastSync loads entire file contents into memory before sending. This is visible in several locations:
src/client/client_send.c:294 -- file_load_data reads the full file into a Data buffer
src/shared/file.c:505-518 -- file_content_to_buffer uses fread to read the entire file into a single mallocd buffer
src/shared/file.c:436-468 -- file_send_sendfile is the exception using sendfile with zero-copy, but it is incompatible with compression and chunk serialization
This means:
A 10 GB file requires 10 GB of RAM
Multiple large files in a chunk consume even more memory
Partial reading/streaming is not possible
Systems with limited RAM cannot transfer files larger than available memory
The sendfile path (which avoids this problem) cannot be combined with compression
For comparison, rsync can stream files with minimal memory overhead, especially when combined with compression.
Suggested Fix: Add a streaming send path for individual files that reads and sends in fixed-size blocks rather than loading the entire file. This would also enable streaming compression.
Severity: medium
FastSync loads entire file contents into memory before sending. This is visible in several locations:
1. src/client/client_send.c:294 -- file_load_data reads the full file into a Data buffer
2. src/shared/file.c:505-518 -- file_content_to_buffer uses fread to read the entire file into a single mallocd buffer
3. src/shared/file.c:436-468 -- file_send_sendfile is the exception using sendfile with zero-copy, but it is incompatible with compression and chunk serialization
This means:
- A 10 GB file requires 10 GB of RAM
- Multiple large files in a chunk consume even more memory
- Partial reading/streaming is not possible
- Systems with limited RAM cannot transfer files larger than available memory
- The sendfile path (which avoids this problem) cannot be combined with compression
For comparison, rsync can stream files with minimal memory overhead, especially when combined with compression.
Suggested Fix: Add a streaming send path for individual files that reads and sends in fixed-size blocks rather than loading the entire file. This would also enable streaming compression.
Severity: medium
TapTap
changed title from performance: entire file content loaded into RAM — no streaming for large files to performance: entire file content loaded into RAM -- no streaming for large files2026-07-20 14:58:44 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
FastSync loads entire file contents into memory before sending. This is visible in several locations:
This means:
For comparison, rsync can stream files with minimal memory overhead, especially when combined with compression.
Suggested Fix: Add a streaming send path for individual files that reads and sends in fixed-size blocks rather than loading the entire file. This would also enable streaming compression.
Severity: medium
performance: entire file content loaded into RAM — no streaming for large filesto performance: entire file content loaded into RAM -- no streaming for large files