Layer 1 Capture (Packet in Packet)

  • Updated on Apr 23, 2025
  • Published on Mar 29, 2025
  • 9 minute(s) read
  • aA
 Prev Next

Layer 1 Capture Packet in Packet is a full capture solution that records the full XGMII Layer 1 stream in a packetized format without any MAC.

High level architecture diagram

Example Packet in Wireshark

Installing

Configuring the Layer1 PinP Capture requires firmware version 10760+ or later

Step 1) Configure Layer 1 PinP mode

Using fmadiocli command configure using

config system fpga mode capture-4x10G-L1

Example output shown below

[Sat Mar 29 12:09:55 2025] > config system fpga mode capture-4x10G-L1
[Sat Mar 29 12:09:59 2025]     fmad fmadlua Sep  7 2024 (/usr/local/bin/fmadiolua /opt/fmadio/bin/firmware_install.lua --portchange )
[Sat Mar 29 12:10:00 2025]     Change PortMode
[Sat Mar 29 12:10:00 2025]     Copy [cp /mnt/system/boot/bitstream.rom.4x10Gl1 /mnt/system/boot/bitstream.rom]
[Sat Mar 29 12:10:00 2025]     os[sudo /opt/fmadio/bin/bitstream_update.lua --noreboot --write /mnt/system/boot/bitstream.rom]
[Sat Mar 29 12:10:01 2025]     cp: '/mnt/system/boot/bitstream.rom' and '/mnt/system//boot/bitstream.rom' are the same file
[Sat Mar 29 12:10:01 2025]     done 0.005109Sec 0.000085Min
[Sat Mar 29 12:10:01 2025]     Firmware Install Complete
[Sat Mar 29 12:10:01 2025]     done 1.104735Sec 0.018412Min
[Sat Mar 29 12:10:01 2025] set port mode from(8x10G) -> (4x10Gl1)
[Sat Mar 29 12:10:01 2025]
[Sat Mar 29 12:10:01 2025] ****************************************************
[Sat Mar 29 12:10:01 2025] *********** REBOOT IS REQUIRED. ********************
[Sat Mar 29 12:10:01 2025] ****************************************************
[Sat Mar 29 12:10:01 2025]
[Sat Mar 29 12:10:01 2025] Reboot -> Safe Mode -> FPGA Flash -> Reboot -> System ready (5-10 minutes)
[Sat Mar 29 12:10:01 2025]
[Sat Mar 29 12:10:01 2025] >

Step 2) Reboot system

The system needs to be rebooted using command. Use Ctrl-C to exit the fmadiocli prompt

sudo reboot

Example output shown below

fmadio@fmadio200v4-636:~$ sudo reboot
fmadio@fmadio200v4-636:~$ Connection to 192.168.2.215 closed by remote host.
Connection to 192.168.2.215 closed.
$

The system will reboot twice. It puts the system into a “safe mode” then proceeds to flash the FPGA firmware. This takes about 5 minutes. Then reboots the system again back into “normal operating mode”

Step 3) Confirm 4×10G L1 capture mode

Confirm the system is in 4×10G L1 capture mode, using the command

cat /opt/fmadio/status/port_config

Example output shown

fmadio@fmadio200v4-636:~$ cat /opt/fmadio/status/port_config
4x10Gl1
fmadio@fmadio200v4-636:~$

Or using fmadiocli

fmadio@fmadio200v4-636:~$ fmadiocli  "show system fpga mode"
fmad fmadlua Mar 29 2025 (/opt/fmadio/bin/fmadiolua --nocal /opt/fmadio/bin/fmadiocli show system fpga mode )
[Sat Mar 29 12:32:40 2025] CmdLine [show system fpga mode]
[Sat Mar 29 12:32:40 2025] Cmd [show system fpga mode ]
[Sat Mar 29 12:32:40 2025] Current capture port configuration: capture-4x10Gl1
fmadio@fmadio200v4-636:~$

Step 4) Start a capture

Start a capture using the quick capture mode on the GUI, fmadiocli or enable the 247 capture mode

Confirm the capture is running using

fmadiocli "show capture status"

Example correct output, note “Capture Running” status is true

fmadio@fmadio200v4-636:~$ fmadiocli "show capture status"
fmad fmadlua Mar 29 2025 (/opt/fmadio/bin/fmadiolua --nocal /opt/fmadio/bin/fmadiocli show capture status )
 [Sat Mar 29 12:35:07 2025] CmdLine [show capture status]
[Sat Mar 29 12:35:07 2025] Cmd [show capture status ]
[Sat Mar 29 12:35:07 2025]
[Sat Mar 29 12:35:07 2025] Current Capture Status
[Sat Mar 29 12:35:07 2025] ------------------+--------------------
[Sat Mar 29 12:35:07 2025] Capture Running   | true
[Sat Mar 29 12:35:07 2025] Capture Name      | wan0_20250329_1228
[Sat Mar 29 12:35:07 2025] Capture Bytes     |        423131520
[Sat Mar 29 12:35:07 2025] Capture Packets   |           661143
[Sat Mar 29 12:35:07 2025] Capture Drop      |                0
[Sat Mar 29 12:35:07 2025] Capture FCS Error |                0
[Sat Mar 29 12:35:07 2025] Capture Rate      |     0.064147 Gbps
[Sat Mar 29 12:35:07 2025]                   |     0.012529 MPps
[Sat Mar 29 12:35:07 2025] Capture Start     |
[Sat Mar 29 12:35:07 2025] Capture Duration  |
[Sat Mar 29 12:35:07 2025] ------------------+--------------------
fmadio@fmadio200v4-636:~$

Step 5) decode L1 traffic

Using the command

sudo stream_cat | pcap_l1decode

This uses the pcap_l1decode utility located at

https://github.com/fmadio/platform/tree/main/pcap_l1decode

The system firmware includes a pre-build binary of this, the example output shown below

fmadio@fmadio200v4-636:/mnt/store0/git/platform/pcap_l1decode$ sudo stream_cat | ./pcap_l1decode | head -n 100
stream_cat ioqueue: 4
Using Filename [wan0_20250329_1228]
calibrating...
0 : 2749993153           2.7500 cycles/nsec offset:0.007 Mhz
Cycles/Sec 2749993153.0000 Std:       0 cycle std(  0.00000000) Target:2.75 Ghz
StartChunkID: 23548756
StartChunk: 23548756 Offset: 0 Stride: 1
StartChunk: 23548756
PCAP nano
.
.

20250423_232112.504_499_769  29 : cap0     ff 0707070707070707
20250423_232112.504_499_769  30 : cap0     ff 0707070707070707
20250423_232112.504_499_769  31 : cap0     ff 0707070707070707
20250423_232112.504_499_769  32 : cap0     ff 0707070707070707
20250423_232112.504_499_769  33 : cap0     ff 0707070707070707
20250423_232112.504_499_769  34 : cap0     ff 0707070707070707
20250423_232112.504_499_769  35 : cap0     ff 0707070707070707
20250423_232112.504_499_769  36 : cap0     ff 0707070707070707
20250423_232112.504_499_769  37 : cap0     ff 0707070707070707
20250423_232112.504_499_769  38 : cap0 S   f8 07070707fb555555
20250423_232112.504_499_769  39 : cap0     00 555555d5d23548a6
20250423_232112.504_499_769  40 : cap0     00 503ef8f21ebccb61
20250423_232112.504_499_769  41 : cap0     00 8100040708004508
20250423_232112.504_499_769  42 : cap0     00 0040e2b940003f06
20250423_232112.504_499_769  43 : cap0     00 e68dc0a8d20bc0a8
20250423_232112.504_499_769  44 : cap0     00 1f0ce0b600169c47
20250423_232112.504_499_769  45 : cap0     00 7d9718e60857b010
20250423_232112.504_499_769  46 : cap0     00 5ff6e7cf00000101
20250423_232112.504_499_769  47 : cap0     00 080abf8ae77e6813
20250423_232112.504_499_769  48 : cap0     00 08ef0101050a18e6
20250423_232112.504_499_769  49 : cap0     00 0daf18e61307a22e
20250423_232112.504_499_769  50 : cap0   E 3f 3768fd0707070707
20250423_232112.504_499_769  51 : cap0     ff 0707070707070707
20250423_232112.504_499_769  52 : cap0     ff 0707070707070707
20250423_232112.504_499_769  53 : cap0     ff 0707070707070707
20250423_232112.504_499_769  54 : cap0     ff 0707070707070707
20250423_232112.504_499_769  55 : cap0     ff 0707070707070707
20250423_232112.504_499_769  56 : cap0     ff 0707070707070707
20250423_232112.504_499_769  57 : cap0     ff 0707070707070707
20250423_232112.504_499_769  58 : cap0     ff 0707070707070707
20250423_232112.504_499_769  59 : cap0     ff 0707070707070707
20250423_232112.504_499_769  60 : cap0     ff 0707070707070707
20250423_232112.504_499_769  61 : cap0     ff 0707070707070707
20250423_232112.504_499_769  62 : cap0     ff 0707070707070707
20250423_232112.504_499_769  63 : cap0     ff 0707070707070707
20250423_232112.504_500_181   0 : cap0     ff 0707070707070707
20250423_232112.504_500_181   1 : cap0 S   f8 07070707fb555555
20250423_232112.504_500_181   2 : cap0     00 555555d5d23548a6
20250423_232112.504_500_181   3 : cap0     00 503ef8f21ebccb61
20250423_232112.504_500_181   4 : cap0     00 8100040708004508
20250423_232112.504_500_181   5 : cap0     00 0040e2b940003f06
20250423_232112.504_500_181   6 : cap0     00 e68dc0a8d20bc0a8
20250423_232112.504_500_181   7 : cap0     00 1f0ce0b600169c47
20250423_232112.504_500_181   8 : cap0     00 7d9718e60857b010
20250423_232112.504_500_181   9 : cap0     00 5ff6e7cf00000101
20250423_232112.504_500_181  10 : cap0     00 080abf8ae77e6813
20250423_232112.504_500_181  11 : cap0     00 08ef0101050a18e6
20250423_232112.504_500_181  12 : cap0     00 0daf18e61307a22e
20250423_232112.504_500_181  13 : cap0   E 3f 3768fd0707070707
20250423_232112.504_500_181  14 : cap0     ff 0707070707070707
20250423_232112.504_500_181  15 : cap0     ff 0707070707070707
20250423_232112.504_500_181  16 : cap0     ff 0707070707070707
20250423_232112.504_500_181  17 : cap0     ff 0707070707070707
20250423_232112.504_500_181  18 : cap0     ff 0707070707070707
20250423_232112.504_500_181  19 : cap0     ff 0707070707070707

Step 6) Searching for specific event

If you know around what time an event occurs, then a much faster process can be applied by extracting and processing only the relevant captured data.

Using the stream cat time functions

sudo stream_cat --time-start HH:MM:SS --top-stop HH:MM:SS

Can greatly accelerate finding corrupted Layer 1 streams.

For example, event occurs around 12:50:00 local time. In the example below its extracts Layer 1 streams from 12:50:00 only.

fmadio@fmadio200v4-636:/mnt/store0/git/platform/pcap_l1decode$ sudo stream_cat --time-start 12:50:00 | pcap_l1decode | head -n 100
Time Start Filter 12:50:00nstream_cat ioqueue: 4
Using Filename [wan0_20250329_1228]
PCAP nano
20250329_125000.000_958_064   0 : 0     ff 0707070707070707
20250329_125000.000_958_064   1 : 0     ff 0707070707070707
20250329_125000.000_958_064   2 : 0     ff 0707070707070707
20250329_125000.000_958_064   3 : 0     ff 0707070707070707
20250329_125000.000_958_064   4 : 0     ff 0707070707070707
20250329_125000.000_958_064   5 : 0     ff 0707070707070707
20250329_125000.000_958_064   6 : 0     ff 0707070707070707
20250329_125000.000_958_064   7 : 0     ff 0707070707070707
20250329_125000.000_958_064   8 : 0     ff 0707070707070707
20250329_125000.000_958_064   9 : 0     ff 0707070707070707
20250329_125000.000_958_064  10 : 0     ff 0707070707070707
20250329_125000.000_958_064  11 : 0     ff 0707070707070707
20250329_125000.000_958_064  12 : 0     ff 0707070707070707
20250329_125000.000_958_064  13 : 0     ff 0707070707070707
20250329_125000.000_958_064  14 : 0     ff 0707070707070707
20250329_125000.000_958_064  15 : 0     ff 0707070707070707
20250329_125000.000_958_064  16 : 0     ff 0707070707070707
20250329_125000.000_958_064  17 : 0     ff 0707070707070707
20250329_125000.000_958_064  18 : 0     ff 0707070707070707
20250329_125000.000_958_064  19 : 0     ff 0707070707070707
20250329_125000.000_958_064  20 : 0     ff 0707070707070707
20250329_125000.000_958_064  21 : 0     ff 0707070707070707
20250329_125000.000_958_064  22 : 0     ff 0707070707070707
20250329_125000.000_958_064  23 : 0 S   f1 07070707fb555555
20250329_125000.000_958_064  24 : 0     00 555555d5f8f21ebc
20250329_125000.000_958_064  25 : 0     00 cb61303ea7050490
20250329_125000.000_958_064  26 : 0     00 8100069308004500
20250329_125000.000_958_064  27 : 0     00 00ac545540004006
20250329_125000.000_958_064  28 : 0     00 5378c0a8b623c0a8
20250329_125000.000_958_064  29 : 0     00 5b0a03f408012239
20250329_125000.000_958_064  30 : 0     00 fa4b53ba62768018
20250329_125000.000_958_064  31 : 0     00 5ff72c4000000101
20250329_125000.000_958_064  32 : 0     00 080ae496ed0326fd
20250329_125000.000_958_064  33 : 0     00 b2f5800000743076
20250329_125000.000_958_064  34 : 0     00 efbc000000000000
20250329_125000.000_958_064  35 : 0     00 0002000186a30000
20250329_125000.000_958_064  36 : 0     00 0003000000120000
20250329_125000.000_958_064  37 : 0     00 00010000002c0000
20250329_125000.000_958_064  38 : 0     00 000000000013666d
20250329_125000.000_958_064  39 : 0     00 6164696f32307634
20250329_125000.000_958_064  40 : 0     00 2d3638322d6c6974
20250329_125000.000_958_064  41 : 0     00 6500000000000000
20250329_125000.000_958_064  42 : 0     00 0000000000010000
20250329_125000.000_958_064  43 : 0     00 0000000000000000
20250329_125000.000_958_064  44 : 0     00 00000000001c0100
20250329_125000.000_958_064  45 : 0     00 0700389017a20800
20250329_125000.000_958_064  46 : 0     00 0000313e378636f6
20250329_125000.000_958_064  47 : 0     00 48b38dd7cf45518c
20250329_125000.000_958_064  48 : 0   E 03 7355ea129795fd07
20250329_125000.000_958_064  49 : 0     ff 0707070707070707
20250329_125000.000_958_064  50 : 0     ff 0707070707070707
20250329_125000.000_958_064  51 : 0     ff 0707070707070707
20250329_125000.000_958_064  52 : 0     ff 0707070707070707
20250329_125000.000_958_064  53 : 0     ff 0707070707070707
20250329_125000.000_958_064  54 : 0     ff 0707070707070707
20250329_125000.000_958_064  55 : 0     ff 0707070707070707
20250329_125000.000_958_064  56 : 0     ff 0707070707070707
20250329_125000.000_958_064  57 : 0     ff 0707070707070707

This can greatly accelerate finding the the needle in a haystack traffic.

pcap_l1decode application

This application converts the PCAP data into raw XGMII traffic, an explanation of the fields follows. The full source code is publicly available under an MIT license

https://github.com/fmadio/platform/tree/main/pcap_l1decode

Column 1) Time

YYYYMMDD_HHMMSS followed by a nano second resolution sub second value

Example:

20250329_122903.847_345_292   0 : cap0     ff 0707070707070707

This is 2025 March(03) 29th at 12:29:03 local time, milliseconds 847, microseconds 345 nanoseconds 292

Column 2) XGMII Frame number

The next column is the XGMII frame number within the packet. This is always 0 - 63 indicating there are a total of 64 XGMII cycles within a single packet. Example highlighted below

Column 3) Capture Port

Capture port is shown next, indicating which physical capture port the data was recorded on

Column 4) XGMII Control

XGMII 8bit control word is shown next, Example shown below

There is some marked up that helps visually decode the packet

S - Start of packet e.g. 0xFB

E - End of packet e.g 0xFD

Column 5) XGMII Data

Finally the XGMII Data payload is shown next, as highlighted below. As can be seen all XGMII traffic is visible, including the XGMII control words. This can be extremely useful for debugging Layer1 protocol issues.