Replay PCAP on management port

FMADIO Gen4 Replay

Firmware: 12232+

FMADIO includes a high speed replay functionality egressing data out the management interfaces. This is helpful for troubleshooting downstream devices from captured data to reproduce problems found in the PROD environment in the lab, and for further processing of packet data in downstream process.

Replay uses the Intel E810/E830/E835 based management card network chipset sourcing data from the SSD Capture Buffer egress the data in pseudo-realtime or a post processing step. The diagram below shows the high level architecture.

Example

Using the following example to egress on a 25G SFP28 management port (phy13)

 sudo stream_cat --chunked <capture name> | sudo ./pcap_replay  -i phy13 --fast

Example output is shown below

sudo stream_cat --chunked interop17_20260809_0954 | sudo ./pcap_replay  -i phy13 --fast
xsk: phy13 already has an XDP program, using it
xsk: phy13 queue 0 bound in zero copy mode
replay:   2.225950 Mpps  20.326571 Gbps  total      2225950 pkts      2.541 GB  drift max         0 ns
replay:   2.201668 Mpps  20.414176 Gbps  total      4427618 pkts      5.093 GB  drift max         0 ns
replay:   2.108579 Mpps  19.358980 Gbps  total      6536197 pkts      7.512 GB  drift max         0 ns
replay:   1.492322 Mpps  13.664159 Gbps  total      8028519 pkts      9.220 GB  drift max         0 ns
replay:   1.872059 Mpps  17.087027 Gbps  total      9900578 pkts     11.356 GB  drift max         0 ns
replay:   2.236854 Mpps  20.380204 Gbps  total     12137432 pkts     13.904 GB  drift max         0 ns
replay:   2.200246 Mpps  20.174061 Gbps  total     14337678 pkts     16.426 GB  drift max         0 ns
replay:   2.135870 Mpps  19.415229 Gbps  total     16473548 pkts     18.853 GB  drift max         0 ns
replay:   2.224422 Mpps  19.699710 Gbps  total     18697970 pkts     21.315 GB  drift max         0 ns
replay:   2.228391 Mpps  20.016930 Gbps  total     20926361 pkts     23.817 GB  drift max         0 ns
replay:   2.210766 Mpps  20.179698 Gbps  total     23137127 pkts     26.340 GB  drift max         0 ns
replay:   2.201882 Mpps  20.096758 Gbps  total     25339009 pkts     28.852 GB  drift max         0 ns
replay:   2.216802 Mpps  20.091786 Gbps  total     27555811 pkts     31.363 GB  drift max         0 ns
replay:   2.162102 Mpps  19.512131 Gbps  total     29717913 pkts     33.802 GB  drift max         0 ns
replay:   2.210582 Mpps  20.150033 Gbps  total     31928495 pkts     36.321 GB  drift max         0 ns
.
.

Performance (FMADIO-5514-180T)

Using the above cycle on the below hardware configuration

• 10 Disk Capture (X200) PCIe Gen5 SSD

• 2 Disk Scratch

• 2 × 100G Management NIC

Performance (FMADIO-5514-45T)

Using the above cycle on the below hardware configuration

• 10 Disk Capture (X100) PCIe Gen4 SSD

• 2 Disk Scratch

• 4 × 25G Management NIC

Performance (FMADIO-5524-90T)

TBD: 2U 24 Disk array, significantly more IO bandwidth avaliable

Legacy (Gen2)

PCAP replay on the management port can be helpful for troubleshooting issues in the lab

The tcpreplay version on the system

fmadio@fmadio100v2-228U:~$ tcpreplay -V
Warning: May need to run as root to get access to all network interfaces.
tcpreplay version: 4.3.1 (build git:v4.3.1)
Copyright 2013-2018 by Fred Klassen <tcpreplay at appneta dot com> - AppNeta
Copyright 2000-2012 by Aaron Turner <aturner at synfin dot net>
The entire Tcpreplay Suite is licensed under the GPLv3
Cache file supported: 04
Not compiled with libdnet.
Compiled against libpcap: 1.10.4
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Packet editing: disabled
Fragroute engine: disabled
Injection method: PF_PACKET send()
Not compiled with netmap
fmadio@fmadio100v2-228U:~$

To replay a static PCAP on the man10 interface

cat /mnt/store0/tmp2/test.pcap | sudo tcpreplay -i man10 --stats=1 -

The example output looks similar to below, example file was replayed at around 550Mbps

fmadio@fmadio100v2-228U:/mnt/store0/tmp2$ cat /mnt/store0/tmp2/vxlan2.pcap | sudo tcpreplay -i man10 -
Warning: - was captured using a snaplen of 16384 bytes.  This may mean you have truncated packets.
Warning: Unable to send packet: Error with PF_PACKET send() [8951]: Message too long (errno = 90)

Actual: 8950 packets (9981543 bytes) sent in 0.145007 seconds
Rated: 68834904.5 Bps, 550.67 Mbps, 61721.15 pps

Statistics for network device: man10
        Successful packets:        8950
        Failed packets:            1
        Truncated packets:         0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0
fmadio@fmadio100v2-228U:/mnt/store0/tmp2$

Replay a FMADIO Capture file on the man10 interface

sudo stream_cat opra_port0_20240728_0405 | sudo tcpreplay -i man10 --stats=1 -

The output looks similar to the following

fmadio@fmadio100v2-228U:/mnt/store0/tmp2$ sudo stream_cat opra_port0_20240728_0405 | sudo tcpreplay -i man10 --stats=1 -
Warning: - was captured using a snaplen of 16384 bytes.  This may mean you have truncated packets.
Test start: 2024-09-30 01:15:33.433149 ...

Actual: 583688 packets (93339006 bytes) sent in 1.00 seconds
Rated: 93333126.0 Bps, 746.66 Mbps, 583651.22 pps
Actual: 1178587 packets (178732888 bytes) sent in 2.00 seconds

Rated: 89363584.3 Bps, 714.90 Mbps, 589274.64 pps
Actual: 1773394 packets (261483666 bytes) sent in 3.00 seconds

Rated: 87159362.6 Bps, 697.27 Mbps, 591118.72 pps
Actual: 2367922 packets (343940040 bytes) sent in 4.00 seconds

Rated: 85983612.7 Bps, 687.86 Mbps, 591970.88 pps
Actual: 2951907 packets (420438034 bytes) sent in 5.00 seconds
.
.

Replay an FMADIO Capture file on the man10 at specific Mbps

Use the following command. In the below example —mbps=10 is set to replay a 10Mbps

sudo stream_cat --cpu 40 <capture name> | sudo tcpreplay -i man10 --stats=1 --mbps=10 -

Resulting output looks like

fmadio@fmadio100v2-228U:/mnt/store0/tmp2$ sudo stream_cat --cpu 40 opra_port0_20240728_0405 | sudo tcpreplay -i man10 --stats=1 --mbps=10 -
Warning: - was captured using a snaplen of 16384 bytes.  This may mean you have truncated packets.
Test start: 2024-09-30 01:17:43.188774 ...
Actual: 9457 packets (1250110 bytes) sent in 1.00 seconds
Rated: 1249997.5 Bps, 9.99 Mbps, 9456.14 pps

Actual: 19036 packets (2500168 bytes) sent in 2.00 seconds
Rated: 1249999.0 Bps, 9.99 Mbps, 9517.35 pps

Actual: 27924 packets (3750216 bytes) sent in 3.00 seconds
Rated: 1249999.5 Bps, 9.99 Mbps, 9307.46 pps

Actual: 36183 packets (5000300 bytes) sent in 4.00 seconds
Rated: 1249999.6 Bps, 9.99 Mbps, 9045.20 pps

Actual: 45343 packets (6250376 bytes) sent in 5.00 seconds
Rated: 1249999.7 Bps, 9.99 Mbps, 9068.05 pps

There are a number of other useful functions tcpreplay provides, for more details see the man page

https://tcpreplay.appneta.com/