Getting Packets

  • Updated on Oct 4, 2025
  • Published on Apr 22, 2025
  • 11 minute(s) read
  • aA
 Prev Next

FMADIO LXC system high level architecture is shown below.

This provides a full lossless filtered version of data directly into the LXC container for processing.

Backpressure is provided thru the LXC Ring structure, allowing the application to consume data as fast or slow as possible without dropping anything.

Reference Example

For packets to flow from the capture SSDs as shown above, into an LXC application additional configuration is required. In this example we are using a manual push into the LXC via an FMADIO Ring. Note, for automatic pushing of data into the LXC pls see: https://docs.fmad.io/docs/fmadio-os-lxc-push-lxc

The following steps are provided.

Step 1) Create an LXC ring

Code is as follows

fmadiocli config ring add test1

Example output shown below

fmadio@fmadio100v2-228U:~$ fmadiocli config ring add lxc_test1
[Sat Oct  4 08:10:27 2025] CmdLine [config]
[Sat Oct  4 08:10:27 2025] CmdLine [ring]
[Sat Oct  4 08:10:27 2025] CmdLine [add]
[Sat Oct  4 08:10:27 2025] CmdLine [lxc_test1]
[Sat Oct  4 08:10:27 2025] Cmd [config ring add lxc_test1 ]
[Sat Oct  4 08:10:27 2025] ---------------------------------------------------------------------------------------------------------------------------------
[Sat Oct  4 08:10:27 2025] Created Ring named [lxc_test1]
RING reset
RING file [/opt/fmadio/queue/lxc_test1]
RING[/opt/fmadio/queue/lxc_test1                       ] Size missmatch 0 12595200
RING[/opt/fmadio/queue/lxc_test1                       ] Size   : 12595200 16777216
RING[/opt/fmadio/queue/lxc_test1                       ] Version:        0      100
RING[/opt/fmadio/queue/lxc_test1                       ] version wrong force reset
RING[/opt/fmadio/queue/lxc_test1                       ] Put:0 0 0x7f830e3b7000
RING[/opt/fmadio/queue/lxc_test1                       ] Get:0 0 0x7f830e3b8000
[Sat Oct  4 08:10:28 2025] created ring [/opt/fmadio/queue/lxc_test1]
fmadio@fmadio100v2-228U:~$

This creates the FMADIO Ring in the following directory

/opt/fmadio/queue/lxc_test1

Example shown below

fmadio@fmadio200v4-636:~$ ls -al /opt/fmadio/queue/lxc_test1
-rw-r--r--    1 root     root      12595200 Oct  4 08:26 /opt/fmadio/queue/lxc_test1
fmadio@fmadio200v4-636:~$

Step 2) Configure the LXC

By default the the LXC configuration does not setup FMADIO Rings inside an LXC.

To configure edit the LXC configuration file, in this case we are using the LXC named security-20250912-0819, the configuration file is in

/opt/fmadio/lxc/<container name>/config

Example is shown below

root@fmadio200v4-636:# cat /opt/fmadio/lxc/security-20250912-0819/config
# lxc config generate by security install.lua
# Sun Sep 14 13:14:39 2025
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = x86_64

lxc.rootfs.path = dir:/opt/fmadio/lxc/security-20250912-0819/rootfs
lxc.uts.name = fmadio200v4-636-security

lxc.net.0.type = veth
lxc.net.0.link = fmad0
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 192.168.255.60/24
lxc.net.0.ipv4.gateway = 192.168.255.2

# Mount Data Directory
lxc.mount.entry = /opt/fmadio/lxc/security-20250912-0819/data/  mnt/data/ none bind,create=dir 0 0

# Mount Log Directory
lxc.mount.entry = /opt/fmadio/lxc/security-20250912-0819/log/  mnt/log/ none bind,create=dir 0 0

#set lxc ring mount
lxc.mount.entry = /opt/fmadio/queue/lxc_suricata   opt/fmadio/queue/lxc_suricata none bind,create=file 0 0

# mount test ring into the lxc
lxc.mount.entry = /opt/fmadio/queue/lxc_test1   opt/fmadio/queue/lxc_test1 none bind,create=file 0 0

lxc.prlimit.nofile = 65535
lxc.prlimit.memlock = unlimited

root@fmadio200v4-636:/mnt/store2/lxc/lib/lxc/security-20250912-0819#

The important lines are shown below, this allows the FMADIO Ring file to be passed though into the LXC.

# mount test ring into the lxc
lxc.mount.entry = /opt/fmadio/queue/lxc_test1   opt/fmadio/queue/lxc_test1 none bind,create=file 0 0

Step 3) Restart LXC and validate

The LXC is requires a restart for the new configuration to take effect.

Typically this can be done with a stop/start process as shown below.

sudo lxc-stop -n <container name>
sudo lxc-start -n <container name>

NOTE: if the lxc-start fails, there might be a typeo or incorrect path in the configuration file. This can be debuged by writing the LXC log file to a location on the host system, example shown below.

sudo lxc-start -n <container name> --logfile lxc_log

Then inspecting the lxc_log file for any problems

Once the LXC has started, confirm the FMADIO Ring is present in the LXC. 

ls -al /opt/fmadio/queue/lxc_test1

As shown below

root@fmadio200v4-636:/opt/fmadio/lxc# lxc-attach -n security-20250912-0819
root@fmadio200v4-636-security:/# ls -al /opt/fmadio/queue/lxc_test1
-rw-r--r-- 1 root root 12595200 Oct  4 08:06 /opt/fmadio/queue/lxc_test1
root@fmadio200v4-636-security:/#

Step 4) Show the current Ring status

Using the command below, check the status of the ring. This should online with no packet counts

 fmadiocli show ring status

Example output shows 0 packets (as expected)

fmadio@fmadio100v2-228U:~$ fmadiocli show ring status
[Sat Oct  4 08:12:20 2025] CmdLine [show]
[Sat Oct  4 08:12:20 2025] CmdLine [ring]
[Sat Oct  4 08:12:20 2025] CmdLine [status]
[Sat Oct  4 08:12:20 2025] Cmd [show ring status ]
[Sat Oct  4 08:12:20 2025] Name                                     : Path                                                         :     Status :          Pkt Put :          Pkt Get : Pkt Queued : Desc
[Sat Oct  4 08:12:20 2025] -----------------------------------------+--------------------------------------------------------------+------------+------------------+------------------+------------+------------------------------------
[Sat Oct  4 08:12:20 2025] lxc_test1                                : /opt/fmadio/queue/lxc_test1                                  :     online :                0 :                0 :          0 :
[Sat Oct  4 08:12:20 2025] -----------------------------------------+--------------------------------------------------------------+------------+------------------+------------------+------------+------------------------------------
fmadio@fmadio100v2-228U:~$

Step 5) Start the LXC consuming data

In this example we will just convert the ring data to PCAP data and output to tcpdump, for demonstration purposes. A real application will consume and process this data in some way.

Running inside the ring

fmadio2pcap -i /opt/fmadio/queue/lxc_test1  | tcpdump -r - -nn

Example shown below

root@fmadio200v4-636:/opt/fmadio/lxc# lxc-attach -n security-20250912-0819
root@fmadio200v4-636-security:/# fmadio2pcap -i /opt/fmadio/queue/lxc_test1  | tcpdump -r - -nn
FMAD Ring [/opt/fmadio/queue/lxc_test1]
RING[/opt/fmadio/queue/lxc_test1                       ] Size   : 12595200 12595200 16777216
RING[/opt/fmadio/queue/lxc_test1                       ] Version:      100      100
RING[/opt/fmadio/queue/lxc_test1                       ] Path:/opt/fmadio/queue/lxc_test1RING[/opt/fmadio/queue/lxc_test1                       ] Put:0 0
RING[/opt/fmadio/queue/lxc_test1                       ] Get:0 0

Step 6) Send data to the Ring

Finally manually send data to the ring using the utility stream_cat. Example command is shown below

For more options see: https://docs.fmad.io/docs/fmadio-utils-stream-cat

sudo stream_cat -v <capture name> --ring /opt/fmadio/queue/lxc_test1

Example output shown below

fmadio@fmadio200v4-636:~$ sudo stream_cat -v opra_trump_20250924_0718 --ring /opt/fmadio/queue/lxc_test1
Create FMAD Ring: 0 [/opt/fmadio/queue/lxc_test1]
stream_cat ioqueue: 4
TimeStamp[0] --pcap
TimeStamp[1] --pcap
TimeStamp[2] --pcap
TimeStamp[3] --pcap
TimeStamp[4] --pcap
TimeStamp[5] --pcap
TimeStamp[6] --pcap
TimeStamp[7] --pcap
RING[/opt/fmadio/queue/lxc_test1                       ] 00 : CPU:   0 FilterBPF:[(null)] FilterFrame:[(null)]
StartChunkID: 108797062
StartChunk: 108797062 Offset: 0 Stride: 1
StartChunk: 108797062
RING[/opt/fmadio/queue/lxc_test1                       ] Size   : 12595200 16777216
RING[/opt/fmadio/queue/lxc_test1                       ] Version:      100      100
RING[/opt/fmadio/queue/lxc_test1                       ] Put:0 0 0x7f8cf16ac000
RING[/opt/fmadio/queue/lxc_test1                       ] Get:0 0 0x7f8cf16ad000
RING[/opt/fmadio/queue/lxc_test1                       ] thread:0
RING[/opt/fmadio/queue/lxc_test1                       ] worker thread start
{"tstr":"20251004_082327", "timestamp":1759566207,"PktCnt":               0, "PktByte":               0, "ChunkID":108797062,"PCAPTS":"00:00:00.000.000.000","PendingB":226502377472,"Read_bps":0,"Read_pps":0,"Write_bps":0,"Write_pps":0,"FwdPct":0.000,"CPUIdle":0.000,"CPUFetch":0.604, "CPUSend":0.000}
{"tstr":"20251004_082328", "timestamp":1759566208,"PktCnt":        13003297, "PktByte":      1953324528, "ChunkID":108805307,"PCAPTS":"14:30:01.610.367.686","PendingB":224341000192,"Read_bps":15625407131,"Read_pps":13002308,"Write_bps":15625407131,"Write_pps":13002308,"FwdPct":1.000,"CPUIdle":0.000,"CPUFetch":0.995, "CPUSend":0.000}
{"tstr":"20251004_082329", "timestamp":1759566209,"PktCnt":        24546024, "PktByte":      3945222528, "ChunkID":108813610,"PCAPTS":"14:30:02.377.394.383","PendingB":222164418560,"Read_bps":15934572272,"Read_pps":11542284,"Write_bps":15934572272,"Write_pps":11542284,"FwdPct":1.000,"CPUIdle":0.000,"CPUFetch":0.995, "CPUSend":0.000}
.
.

Step 7) Confirm the LXC is receiving data

In the LXC window per Step 5) the expected output is a stream of tcpdump 

root@fmadio200v4-636-security:/# fmadio2pcap -i /opt/fmadio/queue/lxc_test1  | tcpdump -r - -nn  | head -n 100
FMAD Ring [/opt/fmadio/queue/lxc_test1]
RING[/opt/fmadio/queue/lxc_test1                       ] Size   : 12595200 12595200 16777216
RING[/opt/fmadio/queue/lxc_test1                       ] Version:      100      100
RING[/opt/fmadio/queue/lxc_test1                       ] Path:/opt/fmadio/queue/lxc_test1RING[/opt/fmadio/queue/lxc_test1                       ] Put:23516a6 2a6
RING[/opt/fmadio/queue/lxc_test1                       ] Get:23516a6 2a6
reading from file -, link-type EN10MB (Ethernet), snapshot length 65535
14:15:15.043316 IP 162.69.44.4.45004 > 224.0.204.4.45004: UDP, length 36
14:15:15.043576 IP 162.69.52.4.45004 > 224.0.206.4.45004: UDP, length 36
14:15:15.043883 IP 162.69.44.4.45004 > 224.0.204.4.45004: UDP, length 36
14:15:15.044144 IP 162.69.52.4.45004 > 224.0.206.4.45004: UDP, length 36
14:15:15.047041 IP 162.69.44.4.45004 > 224.0.204.4.45004: UDP, length 36
14:15:15.047301 IP 162.69.52.4.45004 > 224.0.206.4.45004: UDP, length 36
14:15:15.057104 IP 162.69.44.4.45004 > 224.0.204.4.45004: UDP, length 36
14:15:15.057365 IP 162.69.52.4.45004 > 224.0.206.4.45004: UDP, length 36
14:15:15.088217 IP 162.69.44.4.45004 > 224.0.204.4.45004: UDP, length 36
14:15:15.088478 IP 162.69.52.4.45004 > 224.0.206.4.45004: UDP, length 36
14:15:15.093018 IP 162.69.44.4.45004 > 224.0.204.4.45004: UDP, length 36
14:15:15.093279 IP 162.69.52.4.45004 > 224.0.206.4.45004: UDP, length 36
.
.
.

Step 7) Finished

This is a reference walk thru on how to configure, send and received packets inside of the LXC container.

Sending Data to LXC Offline

During development its typically easiest to send data into the LXC container in a one time replay operation. This allows sending the same data repeatedly to the application allow the engineers to debug and test the code.

In this example we will use the application "fmadio2pcap" which converts the LXC ring into a standard PCAP formatted stream of data.

Its recommended to start the data consumer aka LXC application first, then start the replay.

Consumer - fmadio2pcap tcpdump

Run the following on the LXC container to generate TCPDUMP output thru the lxc ring named "general"

/opt/fmadio/platform/fmadio2pcap  -i /opt/fmadio/queue/lxc_ring_general  | tcpdump  -r - -nn

It will look similar to the following on startup, it stops because no packets have been sent to the ring.

root@fmadio40v3-440-centos7:/opt/fmadio/platform/fmadio2pcap# ./fmadio2pcap  -i /opt/fmadio/queue/lxc_ring_general  | tcpdump  -r - -nn
fmadio2pcap
FMAD Ring [/opt/fmadio/queue/lxc_ring_general]
Ring size   : 12595200 12595200 16777216
Ring Version:      100      100
Ring Depth  :      400      400
Ring Mask   :      3ff      3ff
RING: Put:1612f 12f
RING: Get:1612f 12f

Producer - stream_cat sending to LXC ring

On the FMADIO Host system find a capture you want to replay. In this example we are using capture named "inetsample_20230615_1513" filename can be found using "stream_dump"

The command is as follows

sudo stream_cat --ring /opt/fmadio/queue/lxc_ring_general  --ring-filter-bpf /opt/fmadio/queue/lxc_ring_general "icmp" -v inetsample_20230615_1513

Example output of the command shown below

fmadio@fmadio40v3-440:/mnt/store0/tmp2$ sudo stream_cat --ring /opt/fmadio/queue/lxc_ring_general  --ring-filter-bpf /opt/fmadio/queue/lxc_ring_general "icmp" -v inetsample_20230615_1513
Create FMAD Ring:0 [/opt/fmadio/queue/lxc_ring_general]
found ring: [/opt/fmadio/queue/lxc_ring_general] id:0
FMAD Ring:0 [/opt/fmadio/queue/lxc_ring_general] FilterBPF[icmp]
RING[/opt/fmadio/queue/lxc_ring_general                ] 00 : CPU:   0 FilterBPF:[icmp] FilterFrame:[(null)]
StartChunkID: 766030
StartChunk: 766030 Offset: 0 Stride: 1
StartChunk: 766030
RING[/opt/fmadio/queue/lxc_ring_general                ] Size   : 12595200 16777216
RING[/opt/fmadio/queue/lxc_ring_general                ] Version:      100      100
RING[/opt/fmadio/queue/lxc_ring_general                ] Put:54a 14a 0x7f30c8cc1000
RING[/opt/fmadio/queue/lxc_ring_general                ] Get:54a 14a 0x7f30c8cc2000
RING[/opt/fmadio/queue/lxc_ring_general                ] thread:0
RING[/opt/fmadio/queue/lxc_ring_general                ] worker thread start
0M Offset:    0GB ChunkID:766030 TS:00:00:00.000.000.000 | Pending  31590 MB 0.000Gbps 0.000Mpps CPUIdle:0.000 CPUFetch:0.595 CPUSend:0.000
2M Offset:    2GB ChunkID:776263 TS:00:00:00.000.000.000 | Pending  29032 MB 21.164Gbps 2.311Mpps CPUIdle:0.000 CPUFetch:0.955 CPUSend:0.000
4M Offset:    4GB ChunkID:786596 TS:00:00:00.000.000.000 | Pending  26449 MB 21.374Gbps 2.300Mpps CPUIdle:0.000 CPUFetch:0.957 CPUSend:0.000
6M Offset:    7GB ChunkID:796886 TS:00:00:00.000.000.000 | Pending  23876 MB 21.283Gbps 2.318Mpps CPUIdle:0.000 CPUFetch:0.956 CPUSend:0.000
.
.
.

And on the LXC client side the output will show ICMP packets printed in tcpdump format as shown below

root@fmadio40v3-440-centos7:/opt/fmadio/platform/fmadio2pcap# ./fmadio2pcap  -i /opt/fmadio/queue/lxc_ring_general  | tcpdump  -r - -nn | head -n 1000
fmadio2pcap
FMAD Ring [/opt/fmadio/queue/lxc_ring_general]
Ring size   : 12595200 12595200 16777216
Ring Version:      100      100
Ring Depth  :      400      400
Ring Mask   :      3ff      3ff
RING: Put:29c1 1c1
RING: Get:29c1 1c1
reading from file -, link-type EN10MB (Ethernet)
13:39:53.729815 IP 130.128.35.9 > 210.175.175.21: ICMP echo request, id 4494, seq 1, length 64
13:39:53.731604 IP 210.175.175.21 > 130.128.35.9: ICMP echo reply, id 4494, seq 1, length 64
13:39:53.732577 IP 45.0.191.213 > 8.8.8.8: ICMP echo request, id 13542, seq 25606, length 40
13:39:53.733466 IP 202.17.221.191 > 216.58.200.196: ICMP host 202.17.221.191 unreachable, length 123
13:39:53.733501 IP 202.17.221.191 > 216.58.200.196: ICMP host 202.17.221.191 unreachable, length 123
13:39:53.733641 IP 8.8.8.8 > 45.0.191.213: ICMP echo reply, id 13542, seq 25606, length 40
13:39:53.734071 IP 130.128.255.176 > 103.235.46.39: ICMP echo request, id 65301, seq 2, length 64
13:39:53.744961 IP 130.128.255.81 > 210.175.175.21: ICMP echo request, id 4498, seq 1, length 64
13:39:53.746434 IP 210.175.175.21 > 130.128.255.81: ICMP echo reply, id 4498, seq 1, length 64
13:39:53.762392 IP 12.133.183.36 > 45.0.191.123: ICMP echo reply, id 46807, seq 6777, length 28
13:39:53.770100 IP 45.0.252.242 > 8.8.8.8: ICMP echo request, id 6172, seq 0, length 64
13:39:53.771160 IP 8.8.8.8 > 45.0.252.242: ICMP echo reply, id 6172, seq 0, length 64
13:39:53.775954 IP 130.128.5.7 > 172.21.32.22: ICMP echo request, id 63039, seq 32022, length 40
13:39:53.776217 IP 130.128.5.7 > 172.21.32.21: ICMP echo request, id 63039, seq 32278, length 40
13:39:53.776490 IP 130.128.5.7 > 172.21.32.20: ICMP echo request, id 63039, seq 32534, length 40
13:39:53.776777 IP 130.128.5.7 > 172.21.32.19: ICMP echo request, id 63039, seq 32790, length 40
13:39:53.790153 IP 45.0.191.229 > 8.8.8.8: ICMP echo request, id 34054, seq 39636, length 44
13:39:53.800539 IP 45.0.191.122 > 8.8.8.8: ICMP echo request, id 8265, seq 44218, length 64
13:39:53.801850 IP 8.8.8.8 > 45.0.191.122: ICMP echo reply, id 8265, seq 44218, length 64
.
.
.

Custom Application

For a custom application to directly ingest data, the following reference code is provided

Primary header file with all functions and strcutures inline

https://github.com/fmadio/platform/blob/main/include/fmadio_packet.h

Core example code to retreive packets

https://github.com/fmadio/platform/blob/main/fmadio2pcap/main.c#L160-L185

Core loop snippet, this converts from LXC Ring into standard PCAP Packet format

while (!s_Exit)
{
	u64 TS;
	PCAPPacket_t* Pkt	= (PCAPPacket_t*)PktBuffer;

	// fetch packet from ring without blocking
	int ret = FMADPacket_RecvV1(s_RING, false, &TS, &Pkt->LengthWire, &Pkt->LengthCapture, NULL, Pkt + 1);

	// if it has valid data
	if (ret > 0)
	{
		// convert 64b epoch into sec/subsec for pcap
		Pkt->Sec 			= TS / (u64)1e9;
		Pkt->NSec 			= TS % (u64)1e9;

		// write PCAP header and payload 	
		fwrite(PktBuffer, 1, sizeof(PCAPPacket_t) + Pkt->LengthCapture, FPCAP); 

		// general stats
		TotalPkt 	+= 1;
		TotalByte 	+= ret;
	}	

	// end of stream
	if (ret < 0) break;
}