parity: stream a single file >256 MiB (unblocks basis-dir and fuzzy rows) #318

Closed
opened 2026-09-23 02:10:49 +02:00 by TapTap · 1 comment
Owner

MAX_RECEIVE_WHOLE_FILE_SIZE (256 MiB) caps a single whole-file payload, so a basis MISS over the limit is refused (--compare-dest/--copy-dest/--link-dest ⚠️) and -y/--fuzzy's whole-basis buffer is capped.

Work: remove the whole-file buffering ceiling by streaming the payload/basis through a bounded buffer end-to-end. Closes/narrows four ⚠️ rows (three basis dirs + fuzzy tie-break/buffer).

Files: src/shared/incremental_check.c, src/shared/file_receive.c, src/shared/file_send.c, src/client/client_send.c.

`MAX_RECEIVE_WHOLE_FILE_SIZE` (256 MiB) caps a single whole-file payload, so a basis MISS over the limit is refused (`--compare-dest`/`--copy-dest`/`--link-dest` ⚠️) and `-y/--fuzzy`'s whole-basis buffer is capped. **Work:** remove the whole-file buffering ceiling by streaming the payload/basis through a bounded buffer end-to-end. Closes/narrows four ⚠️ rows (three basis dirs + fuzzy tie-break/buffer). Files: `src/shared/incremental_check.c`, `src/shared/file_receive.c`, `src/shared/file_send.c`, `src/client/client_send.c`.
TapTap added the enhancementperformance labels 2026-09-23 02:10:49 +02:00
Author
Owner

Fixed in PR #328. The receiver reads the length-prefixed whole-file payload in bounded 64 KiB chunks and spools above the streaming bound (incrementally decompressed for zstd/zlib) through the existing bounded basis-copy store path; sender-side streaming compression added. --compare-dest/--copy-dest/--link-dest no longer refuse an over-limit basis MISS, and -y/--fuzzy streams the sibling signature/reconstruction. Verified with a real 300 MiB transfer (sha256 match; peak RSS 7 MB client / 3 MB receiver with -z). The four affected ⚠️ rows had their over-256 MiB residual dropped. Remaining residuals: --append/delta over-limit reconstruction on the destination path still materializes; lz4 cannot stream. Closing as completed.

Fixed in PR #328. The receiver reads the length-prefixed whole-file payload in bounded 64 KiB chunks and spools above the streaming bound (incrementally decompressed for zstd/zlib) through the existing bounded basis-copy store path; sender-side streaming compression added. `--compare-dest`/`--copy-dest`/`--link-dest` no longer refuse an over-limit basis MISS, and `-y/--fuzzy` streams the sibling signature/reconstruction. Verified with a real 300 MiB transfer (sha256 match; peak RSS 7 MB client / 3 MB receiver with `-z`). The four affected ⚠️ rows had their over-256 MiB residual dropped. Remaining residuals: `--append`/delta over-limit reconstruction on the destination path still materializes; lz4 cannot stream. Closing as completed.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: TapTap/FastSync#318