FMADIO Systems have a simple internal ring bus architecture. This Ring bus can be used for sending packet data within the system to containers all without dropping any packets.
In the diagram below its a single Shared Memory lockless Ring.
Producer will write packets into the shared memory PUT slot + updated the PUT pointer
Consumer will read packets from the shared memory GET slot + update the GET pointer
Flow control ensures the PUT and GET pointers do not overflow or underflow
Lock free design. There are not mutexes or locks
.png?sv=2022-11-02&spr=https&st=2025-05-14T23%3A40%3A52Z&se=2025-05-15T00%3A02%3A52Z&sr=c&sp=r&sig=jiXTO2pEdPjfvj0F7bC5SlRbkRpCLZoid%2FgJEbG0GwA%3D "image(83).png")
Its a simple system, by default the FMADIO Ring is only 1024 × 9K jumbo packets deep. This keeps the data hot in the L1, L2, L3 caches while still allow ebbs and flows of the processing.
The location of the rings are in
/opt/fmadio/queue/lxc_*This allows an easy way to pass the the entire FMADIO Ring to an LXC container. For example the FShark2 container has the following in the lxc config
#set lxc ring mount
lxc.mount.entry = /opt/fmadio/queue/lxc_fshark2 opt/fmadio/queue/lxc_fshark2 none bind,create=file 0 0
lxc.prlimit.nofile = 65535
lxc.prlimit.memlock = unlimitedMonitoring Ring Status
fmadiocli
FMADIO CLI can be used to monitor the status of the ring using the command
fmadiocli "show ring status"Example output below:
fmadio@fmadio200v4-636:~$ fmadiocli "show ring status"
fmad fmadlua Oct 18 2024 (/opt/fmadio/bin/fmadiolua --nocal /opt/fmadio/bin/fmadiocli show ring status )
Disable cycle calibration
[Tue Dec 17 19:09:17 2024] CmdLine [show ring status]
[Tue Dec 17 19:09:17 2024] Cmd [show ring status]
[Tue Dec 17 19:09:17 2024] Name : Path : Status : Pkt Put : Pkt Get : Pkt Queued : Desc
[Tue Dec 17 19:09:17 2024] -----------------------------------------+--------------------------------------------------------------+------------+------------------+------------------+------------+------------------------------------
[Tue Dec 17 19:09:17 2024] lxc_fshark2 : /opt/fmadio/queue/lxc_fshark2 : online : 3,137,624 : 3,137,624 : 0 :
[Tue Dec 17 19:09:17 2024] -----------------------------------------+--------------------------------------------------------------+------------+------------------+------------------+------------+------------------------------------
done 0.036744Sec 0.000612Min
fmadio@fmadio200v4-636:~$Producer : stream_cat
The primary packet producer is the stream_cat utility. This will fetch data from the local storage, filter it and write to the specified ring
Offline Capture
Below is the command to write a PCAP without filtering into the ring
sudo stream_cat -v --ring /opt/fmadio/queue/lxc_fshark2 <capture name>Example output looks like below
fmadio@fmadio200v4-636:~$ sudo stream_cat -v --ring /opt/fmadio/queue/lxc_fshark2 wan0_20241217_1319
Create FMAD Ring: 0 [/opt/fmadio/queue/lxc_fshark2]
RING[/opt/fmadio/queue/lxc_fshark2 ] 00 : CPU: 0 FilterBPF:[(null)] FilterFrame:[(null)]
StartChunk: 6338017
RING[/opt/fmadio/queue/lxc_fshark2 ] Size : 12595200 16777216
RING[/opt/fmadio/queue/lxc_fshark2 ] Version: 100 100
RING[/opt/fmadio/queue/lxc_fshark2 ] Put:3ff 3ff 0x7f5af8334000
RING[/opt/fmadio/queue/lxc_fshark2 ] Get:3ff 3ff 0x7f5af8335000
RING[/opt/fmadio/queue/lxc_fshark2 ] thread:0
RING[/opt/fmadio/queue/lxc_fshark2 ] worker thread start
{"tstr":"20241217_190746", "timestamp":1734433666,"PktCnt": 0, "PktByte": 0, "ChunkID":6338017,"PCAPTS":"00:00:00.000.000.000","PendingB":49860575232, "Read_bps":0,"Read_pps":0, "Write_bps":0,"Write_pps":0, "FwdPct":0.000 }
{"tstr":"20241217_190747", "timestamp":1734433667,"PktCnt": 504489, "PktByte": 365485290, "ChunkID":6339449,"PCAPTS":"20:00:03.678.433.179","PendingB":49485185024, "Read_bps":2918460195,"Read_pps":503553, "Write_bps":5829923357,"Write_pps":1007107, "FwdPct":2.000 }
{"tstr":"20241217_190748", "timestamp":1734433668,"PktCnt": 1000274, "PktByte": 797697754, "ChunkID":6341128,"PCAPTS":"20:00:07.418.689.821","PendingB":49045045248, "Read_bps":3456240418,"Read_pps":495576, "Write_bps":6902900114,"Write_pps":991152, "FwdPct":2.000 }
{"tstr":"20241217_190749", "timestamp":1734433669,"PktCnt": 1513100, "PktByte": 1154918776, "ChunkID":6342522,"PCAPTS":"20:00:11.297.875.278","PendingB":48679616512, "Read_bps":2855335713,"Read_pps":512389, "Write_bps":5703834082,"Write_pps":1024779, "FwdPct":2.000 }
{"tstr":"20241217_190750", "timestamp":1734433670,"PktCnt": 2030128, "PktByte": 1479828590, "ChunkID":6343793,"PCAPTS":"20:00:15.832.633.068","PendingB":48346431488, "Read_bps":2593419322,"Read_pps":515863, "Write_bps":5180966837,"Write_pps":1031725, "FwdPct":2.000
.
.
.
Offline Filtered Capture
By adding --ring-filter-bpf <ring> BPF to the command line a BPF filter can be applied to the traffic going into the FMADIO Ring, in this example filtering for vlan 2
sudo stream_cat -v --ring /opt/fmadio/queue/lxc_fshark2 --ring-filter-bpf /opt/fmadio/queue/lxc_fshark2 "vlan 2" <capture name>Realtime Capture
Forwarding the captured data in realtime is achieved using the --follow command line option. Example below
sudo stream_cat -v --follow --ring /opt/fmadio/queue/lxc_fshark2Realtime BPF Filtered Capture
Sending BPF filtered capture data in realtime to the FMADIO Ring, Example below
sudo stream_cat -v --follow --ring /opt/fmadio/queue/lxc_fshark2 --ring-filter-bpf /opt/fmadio/queue/lxc_fshark2 "vlan 2"Realtime BPF Filtered Port Filtered Capture
Finally filtering on the physical capture port and a BPF filter sending data into the FMADIO Ring. This uses the --ring-filter-frame <ring> <frame filter> functionality. Example below
sudo stream_cat -v --follow --ring /opt/fmadio/queue/lxc_fshark2 --ring-filter-bpf /opt/fmadio/queue/lxc_fshark2 "vlan 2" --ring-filter-bpf /opt/fmadio/queue/lxc_fshark2 "capture.port=1"Producer : pcap2fmadio
In addition to stream_cat on the host system there is a utility pcap2fmadio . This does a similar function as stream_cat however it operates using PCAP format on STDIN.
Unfiltered PCAP
For example to send a pcap named test.pcap into the lxc_fshark2 FMADIO Ring, shown below
cat test.pcap | pcap2fmadio -i /opt/fmadio/queue/lxc_fshark2Example output:
fmadio@fmadio200v4-636:~$ cat test.pcap | pcap2fmadio -i /opt/fmadio/queue/lxc_fshark2
Nano PCAP
RING[/opt/fmadio/queue/lxc_fshark2 ] Size : 12595200 16777216
RING[/opt/fmadio/queue/lxc_fshark2 ] Version: 100 100
RING[/opt/fmadio/queue/lxc_fshark2 ] Put:428935 135 0x7ffac3f9a000
RING[/opt/fmadio/queue/lxc_fshark2 ] Get:428536 136 0x7ffac3f9b000Filtered PCAP
A more complicated example can use zcat and tcpdump to filter the data before it goes into the ring buffer. In the example below it is
decompressing in gzip format
filtering for non VLAN tagged packets using tcpdump
writing the filtered packets into lxc_fshark2 ring
zcat test.pcap.gz | tcpdump -r - -w - "not vlan" | pcap2fmadio -i /opt/fmadio/queue/lxc_fshark2Example output:
fmadio@fmadio200v4-636:~$ zcat test.pcap.gz | tcpdump -r - -w - "not vlan" | pcap2fmadio -i /opt/fmadio/queue/lxc_fshark2
reading from file -, link-type EN10MB (Ethernet), snapshot length 16384
Nano PCAP
RING[/opt/fmadio/queue/lxc_fshark2 ] Size : 12595200 16777216
RING[/opt/fmadio/queue/lxc_fshark2 ] Version: 100 100
RING[/opt/fmadio/queue/lxc_fshark2 ] Put:488501 101 0x7efdd0c66000
RING[/opt/fmadio/queue/lxc_fshark2 ] Get:488501 101 0x7efdd0c67000Consumer : fmadio2pcap
On the otherside the utility fmadio2pcap will take the FMADIO Ring and convert that into standard PCAP format. While this is a suboptimal approach almost all network analysis tools have support for PCAPs. This enables a fast way to get up and running with almost any network analysis tool.
tcpdump output
Example piping the output into tcpdump
fmadio2pcap -i /opt/fmadio/queue/lxc_fshark2 | tcpdump -r - -nn Example output below:
.png?sv=2022-11-02&spr=https&st=2025-05-14T23%3A40%3A52Z&se=2025-05-15T00%3A02%3A52Z&sr=c&sp=r&sig=jiXTO2pEdPjfvj0F7bC5SlRbkRpCLZoid%2FgJEbG0GwA%3D "image(84).png")
Library
To enable processing there is a simple API, its part of our public code repo located below:
https://github.com/fmadio/platform/blob/main/include/fmadio_packet.h
Full BSD license opensource library. Its a simple shared memory ring buffer making it simple for direct integration. Basic full source code example is shown below with the fmadio2pcap reference code.
fmadio2pcap example
The above fmadio2pcap example utility is located
https://github.com/fmadio/platform/tree/main/fmadio2pcap
The example fmadio2pcap utility shows how simple it is to get packets off the FMADIO Ring. Core code snippet below
while (!s_Exit)
{
u64 TS;
PCAPPacket_t* Pkt = (PCAPPacket_t*)PktBuffer;
u32 PktFlag = 0;
// fetch packet from ring without blocking
int ret = FMADPacket_RecvV1(s_RING, false, &TS, &Pkt->LengthWire, &Pkt->LengthCapture, NULL, &PktFlag, Pkt + 1);
// if it has valid data
if (ret > 0)
{
// count flaged FCS packets
if (PktFlag & FMADRING_FLAG_FCSERR)
{
TotalPktFCS++;
}
// santize it
assert(Pkt->LengthCapture > 0);
assert(Pkt->LengthCapture < 16*1024);
// convert 64b epoch into sec/subsec for pcap
Pkt->Sec = TS / (u64)1e9;
Pkt->NSec = TS % (u64)1e9;
// write PCAP header and payload
fwrite(PktBuffer, 1, sizeof(PCAPPacket_t) + Pkt->LengthCapture, FPCAP);
// general stats
TotalPkt += 1;
TotalByte += ret;
}
// end of stream
if (ret < 0) break;
}
fflush(stdout);