Performance of push varies by protocol filter and compression mode. Typically the remote push locations IO bandwidth is typically the bottleneck.
Here we are testing against a RAM disk on the local system. This removes the network and remote IO performance from the benchmark. Focusing entirely on the FMADIO fetch filter and compress performance.
Parallel GZIP (pigz) running on an FMADIO 100Gv2 Analytics System (96 CPUs total). Making good use of all the CPUs
.png?sv=2022-11-02&spr=https&st=2025-05-06T18%3A51%3A10Z&se=2025-05-06T19%3A02%3A10Z&sr=c&sp=r&sig=2mezlGVhyRDmCCsCf8K11HSWKLBzL1DjMJ5jsmj2vWY%3D)
Parallel GZIP 64 CPUs
XZ compression using 64 CPUs, more utilization than Parallel GZ
.png?sv=2022-11-02&spr=https&st=2025-05-06T18%3A51%3A10Z&se=2025-05-06T19%3A02%3A10Z&sr=c&sp=r&sig=2mezlGVhyRDmCCsCf8K11HSWKLBzL1DjMJ5jsmj2vWY%3D)
xz compression 64 CPUs
Traffic Profile ISP
The dataset we are testing with is a WAN connection that has an ISP like L2-L7 packet distribution. As its ISP like traffic the data compression rate is not high.
Description | Gbps | Size | Ratio |
|---|---|---|---|
raw (not compressed) | 6.9 Gbps | 45.5G | x1.0 |
lz4 | 2.5Gbps | 41.2GB | x1.104 |
gzip -1 (fast) | 0.16Gbps | 40.5GB | x1.123 |
gzip default | 0.15Gbps | 40.2GB | x1.129 |
pigz 64 CPU (Parallel GZIP with 64 CPUs) | 4.8Gbps | 40.3GB | x1.129 |
pigz 32 CPU (Parallel GZIP with 32 CPUs) | 4.4Gbps | 40.3GB | x1.129 |
pigz 8 CPU (Parallel GZIP with 8 CPUs) | 1.04Gbps | 40.3GB | x1.129 |
zstd default | 0.77Gbps | 39.5GB | x1.15 |
zstd —fast | 2.4Gbps | 40.3GB | x1.129 |
xz default | 0.019Gbps | 39.2GB | x1.160 |
xz 64 CPU (-T 64) | 0.909Gbps | 38.9GB | x1.17 |
xz 32 CPU (-T 32) | 0.502Gbps | 39.3GB | x1.157 |
Traffic Profile Finance
The data set tested is a full days worth of OPRA A+B Feed dataset, raw uncompressed data size is just under 1TB. Financial data typically gets x2 to x3 compression ratio with xz maxing out at x5.
Description | Gbps | Time | Size | Ratio |
|---|---|---|---|---|
raw (not compressed) | 4.29Gbps | 0.6H | 979GB | x1.0 |
lz4 | 1.65Gbps | 1.3H | 372GB | x2.629 |
gzip -1 (fast) | 0.341Gbps | 6.3H | 266GB | x3.67 |
gzip (default) | 0.129Gbps | 16.6H | 311GB | x3.14 |
pigz 64 CPU (parallel gzip) | 3.92Gbps | 0.55H | 264GB | x3.70 |
pigz 32 CPU (parallel gzip) | 3.92Gbps | 0.55H | 264GB | x3.70 |
pigz 16 CPU (parallel gzip) | 1.98Gbps | 1.09H | 264GB | x3.70 |
pigz 8 CPU (parallel gzip) | 0.997Gbps | 2.1H | 264GB | x3.70 |
zstd default | 1.118Gbps | 1.9H | 255GB | x3.83 |
zstd —fast | 1.90Gbps | 1.13H | 294GB | x3.32 |
xz default | 00.012Gbps | 181H | 184GB | x5.32 |
xz 64 CPU | 0.604Gbps | 3.6H | 194GB | x5.32 |
xz 32 CPU | 0.372Gbps | 5.8H | 194GB | x5.32 |
xz 16 CPU | 0.192Gbps | 11.3H | 184GB | x5.32 |
xz 8 | 0.096Gbps | 22.6H | 184GB | x5.32 |