Getting Started

Push PCAP is a function on the FMADIO device which can filter / split / compress the raw incoming packet data. FMADIO systems will always write to the SSD Storage first, this prevents any packet loss on capture. Once on storage it is then read out and processed by the Push PCAP code.

For example use case

https://www.fmad.io/blog/push-compressed-realtime-filtered-pcap

FMADIO Push PCAP Architecture

Process flow

1) FMADIO Capture

FMADIO Capture device uses the FPGA and SSD Capture buffer to write packets revived on the Capture Ports directly to the SSDs. This path has minimial filtering and processing, with the goal of capturing everything first, ask questions later.

2) FMADIO Push PCAP

Once data is in the Capture Buffer, the builtin FMADIO Push PCAP function reads from the SSD buffer (1TB - 1PB) where it splits and filters using BPF Filters and optionally compresses the data, before writing it to the Local Scratch disk which is a normal linux EXT4 or XFS based storage system.

3) FMADIO Push PCAP Post Processing

After PCAPs have been generated on the local Scratch Disk, various post processing can be performed. Below are examples of features that are built into the FMADIO system

- Remote Copy

In addition to writing to the local disk, the system can copy the PCAP files from the local disk to a remote storage, such as AWS S3 or GCP S3 or Other S3 buckets. This copy process includes checks once completed the file sizes match providing additional layers of monitoring data has arrived at the remote location.

- Market Data Gap Detection

Once PCAPs have been generated on the Local Scratch Disk, additional processing can be performed to confirm all market data has been recevied (e.g. no gaps in the data). This provides an additional verification step in the pipeline enabling further quality checks of the generated PCAP data.

Supported Hardware

Supported Endpoints

The primary endpoint is an S3 bucket, this could be an AWS S3 bucket, GCP S3 bucket or MiniIO S3 Bucket as typical end use cases.

and many more, see the rclone documentation for full list of endpoints supported: https://rclone.org/docs/

Configuration

Configuration is via configuration scripts located at:

/opt/fmadio/etc/push_pcap.lua

If there is no such file above, please copy the basic example from the following location:

/opt/fmadio/etc_ro/push_pcap.lua.basic

An example is shown as follows:

local Config = {}
Config.TimeoutRing      = 5*60e9
Config.Target           = {}
table.insert(Config.Target,
{
        Desc      = "Full",
        Mode      = "File",
        Path      = os.date("/mnt/remote0/pcap/%Y%m%d/all-", PCAPTS),
        Split     = "--split-time "..(60*60*1e9),
        SplitCmd  = "-Z fmadio",
        FileName  = "--filename-tstr-HHMMSS",
        FilterBPF = "",
        PipeCmd   = "zstd -c -T8",
        FileSuffix= ".pcap.zstd",
})
table.insert(Config.Target,
{
        Desc      = "tcp_192_168_1_0",
        Mode      = "File",
        Path      = os.date("/mnt/remote0/pcap/%Y%m%d/tcp_host-", PCAPTS),
        Split     = "--split-time "..(60*60*1e9),
        SplitCmd  = "-Z fmadio",
        FileName  = "--filename-tstr-HHMMSS",

Multiple Push PCAP targets can be specified, there is no real limit however throughput of processing may be impacted.

In the above example there are 2 Push PCAP rules:

A) Push all packet data (no filter)

This Push PCAP target sends all PCAP data the remote NFS share mounted on:

/mnt/remote0

See Mount Remote NFS (Linux) Drive for details on setting up /mnt/remote0 mounting points.

The specified is "FilterBPF=nil" meaning there is no filter, thus all traffic is pushed

B) Push all TCP data from network 192.168.1.0/24

The second example shows pushing all TCP data on the network 192.168.1.0/24 to the specified /mnt/remote0/push/ directory with a PCAP file prefix of "tcp_*"

Note:

FilterBPF=net 192.168.1.0/24 and tcp  

This applies a full BPF (Berkley Packet Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter ) with the filter "tcp" on the packets before writing it to the location. This results in only TCP data written to the /mnt/remote0/push/tcp_*.pcap output files.