Examples

Following is some reference example push_pcap.lua configuration files

Push to NFS Share with BPF Filter and 1 minute PCAPs

local Config = {}

Config.Target = {}

-- push all tcp data to /mnt/remote0/push/tcp_*.pcap
table.insert(Config.Target, 
{
    Desc      = "nfs-tcp", 
    Mode      = "File", 
    Path      = "/mnt/remote0/push/tcp",   
    Split     = "--split-time 60e9", 
    FileName  = "--filename-epoch-sec-startend", 
    FilterBPF = "net 192.168.1.0/24 and tcp" 
})

return Config

Push to NFS Share with BPF Filter and HHMMSS Timezone

Example pushes a single UDP multicast group 1001 at 1 minute snapshots using an Hour Min Sec with Timezone filename.

local Config = {}

Config.Target = {}

-- push all multicast port 10001 data to /mnt/remote0/push/udp-10001_*.pcap
table.insert(Config.Target, 
{
    Desc      = "udp-multicast-1001", 
    Mode      = "File", 
    Path      = "/mnt/remote0/push/udp-10001",   
    Split     = "--split-time 60e9", 
    FileName  = "--filename-tstr-HHMMSS_TZ", 
    FilterBPF = "multicast and port 10001" 
})

Push to NFS Share 1min Splits with BPF Filter and LZ4 compression

Example pushes 1min PCAPs with a BPF filter (port 80) and applying LZ4 compression. LZ4 compression is fast and reasonably good compression rates.

local Config = {}
Config.Target = {}
table.insert(Config.Target,
{
    Desc      = "port80-lz4",
    Mode      = "File",
    Path      = "/mnt/store0/tmp2/push/udp-10001",
    Split     = "--split-time 60e9",
    FileName  = "--filename-tstr-HHMMSS_TZ",
    FilterBPF = "port 80",
    PipeCmd   = "lz4 -c",
    FileSuffix   = ".pcap.lz4",
})

return Config

Push to NFS share 1min Splits with BPF Filter and ZSTD compression

Example pushes 1min PCAPs with a BPF filter (port 80) and applying ZSTD compression. ZSTD is a new compression format with performance close to LZ4 but compression rates close to GZIP.

local Config = {}

Config.Target = {}

table.insert(Config.Target,
{
    Desc      = "port80 zstd",
    Mode      = "File",
    Path      = "/mnt/store0/tmp2/push/udp-10001",
    Split     = "--split-time 60e9",
    FileName  = "--filename-tstr-HHMMSS_TZ",
    FilterBPF = "port 80",
    PipeCmd   = "zstd -c",
    FileSuffix   = ".pcap.zstd",
})

return Config

Push to NFS/CIFS Share 1GB splits

Example pushes the raw data to a remote NFS/CIFS (Windows Share) splitting by 1GB file size writing a gzip compressed PCAP file to the remote location.

local Config = {}

Config.Target = {}

-- push everything to /mnt/remote0/push/capture*.pcap.gz at 1GB splits compressed gz
table.insert(Config.Target, 
{
    Desc      = "capture", 
    Mode      = "File", 
    Path      = "/mnt/remote0/push/capture",   
    Split     = "--split-size 1e9", 
    FileName  = "--filename-tstr-HHMMSS",
    FileSuffix = "pcap.gz", 
    FilterBPF = "" 
})

return Config

Push to AWS S3 Bucket with Compression

Pushing captured PCAP data from the local device to AWS S3 Bucket can be done using the RCLONE support.

Below is an example push_pcap.lua config file for that

-- autogenerated Thu Jun  9 20:07:14 2022 from fmadio_config
local Config = {}
Config.FollowStart = true
Config.Decap       = true
Config.Target      = {}
table.insert(Config.Target,
{
    ["Desc"]        = "S3Cloud",
    ["Mode"]        = "RCLONE",
    ["Path"]        = "fmadio-s3://fmadio-pcap/pcap/fmadio20p3-coffee/full",
    ["Split"]       = "--split-time 60e9",
    ["SplitCmd"]    = "",
    ["PipeCmd"]     = " gzip -c ",
    ["FileSuffix"] = ".gz",
    ["FileName"]    = "--filename-tstr-HHMM",
    ["FilterBPF"]   = "",
    ["FilterFrame"] = "",
})
return Config

This uses gzip to compress the data. Also note we added a PreCapture filter to 64B Slice all traffic to AWS S3 IP address. This prevents the capture size for a run-away explosion.

Below is the resulting output in AWS S3

This does require RCLONE S3 Config to be configured before using.

OPRA 96 Multicast Groups

As part of our testing and system verification, FMADIO has a full FMADIO-5514-180T system deployed into NY2 receiving the full OPRA feed over 2 ×100G Cross connects from NY4. This system provides us with valuable real-world semi-PROD environment we can experiment and test our solution on.

The configuration we have setup is shown below:

In the above example the system is splitting the OPRA A feed into a single 1Min PCAP per each Multicast group. This results in:

• Every 1 minute

• For each Multicast group

A generated PCAP file on the local scratch disk.

The location of this config is:

/opt/fmadio/etc/push_pcap.lua

The actual config file:

local Config = {}

-- start processing from the start of the capture (instead of current write position)
Config.FollowStart = false

-- do not de-encapsulate the packets
Config.Decap = false

Config.Target = {}

-- for each of the 96 OPRA Multicast groups
for MCGroup=1,96 do

    -- generate a PCAP output
    table.insert(Config.Target,
    {
        Mode      = "File",
        -- unique name for each split
        Desc      = string.format("pcap-all-capture0-opra-%02i", MCGroup),

        -- promethues lablel for each metric
        Promethues= string.format('port="capture0",feed="A",group="%02i"', MCGroup),

        -- write location on the local disk
        Path      = os.date(string.format("/mnt/store1/pcap//%Y%m%d/capture0/opra-%02i-", MCGroup), PCAPTS),

        -- copy location (e.g. remote S3 mount point)
        Copy      = os.date(string.format("/mnt/remote0/copy/%Y%m%d/capture0/opra-%02i-", MCGroup), PCAPTS),

        -- split every 1min / 60 seconds
        Split     = "--split-time 60e9 ",

        -- filename format is epoch time
        FileName  = "--filename-epoch-sec-startend",

        -- BPF filter to select only the OPRA A feed destination multicast group
        FilterBPF = "ip and dst host 224.0.204."..MCGroup,


        -- optional additional (disabled)
        --Post        = true,
        --PipeCmd     = "zstd -c -T8",
        --FileSuffix  = ".pcap.zst",
        --Post  = "/opt/fmadio/analytics/push_pcap_copy.lua",
    })
end

return Config

The above config general keeps running in pseudo-realtime during the day. During busier periods there are some delays at market open, which get quickly caught up within the first hour. Capture is not impacted during this period.