Fair benchmark: random data, exclude dry run from metrics #10

2026-07-16T13:43:42+02:00

TapTap commented

2026-07-16 13:43:42 +02:00

Two fairness fixes for the benchmark suite:

Random data instead of all-zeros: Bulk files now use random.randbytes() instead of b"0" * n. Zero-filled data compresses to near-nothing, making zstd compression look unrealistically effective. Random data is incompressible, so speedups reflect real protocol efficiency.
Dry run excluded from best-client metrics: print_metrics now filters out dry-run results, which transfer zero bytes but always appear as the "best" configuration. Real transfer configs are correctly compared against rsync.

Before (zeros): compression won by compressing payload to near-zero
After (random): Standard (no metadata) is fastest at 0.81s, 6.79x vs rsync archive, 1.75x vs rsync compress — honest numbers from protocol architecture, not data artifacts.

Two fairness fixes for the benchmark suite: 1. **Random data instead of all-zeros**: Bulk files now use `random.randbytes()` instead of `b"0" * n`. Zero-filled data compresses to near-nothing, making zstd compression look unrealistically effective. Random data is incompressible, so speedups reflect real protocol efficiency. 2. **Dry run excluded from best-client metrics**: `print_metrics` now filters out dry-run results, which transfer zero bytes but always appear as the "best" configuration. Real transfer configs are correctly compared against rsync. Before (zeros): compression won by compressing payload to near-zero After (random): Standard (no metadata) is fastest at 0.81s, 6.79x vs rsync archive, 1.75x vs rsync compress — honest numbers from protocol architecture, not data artifacts.

TapTap added 1 commit 2026-07-16 13:43:42 +02:00

Fair benchmark: random data instead of zeros, exclude dry run from metrics ea551c4a1d

TapTap merged commit 07207cd71d into main

2026-07-16 13:46:27 +02:00

TapTap deleted branch fair-benchmark

2026-07-16 13:46:29 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: TapTap/FastSync#10