Reference

Introduction

Provides a reference documentation on the configuration parameters for Push PCAP.

Configuration file by default is located int

/opt/fmadio/etc/push_pcap.lua

Basic Push PCAP functionality can be configured using FMADIO CLI, for more advanced features direct editing of the push_pcap.lua configuration file is recommended.

Example configuration file is below

-- autogenerated Tue Dec 12 20:37:12 2023 from fmadio_config
local Config = {}
Config.FollowStart = true
Config.Decap       = true
Config.Target      = {}
table.insert(Config.Target,
{
    ["Desc"]        = "pushtest",
    ["Mode"]        = "File",
    ["Path"]        = "/mnt/store0/pushpcap/",
    ["Split"]       = "--split-time 60e9",
    ["SplitCmd"]    = "",
    ["PipeCmd"]     = "zstd -c -T0 ",
    ["FileName"]    = "--filename-tstr-HHMMSS",
    ["FilterBPF"]   = "udp and port 45010",
    ["FilterFrame"] = "",
    ["FileSuffix"] = ".pcap.zstd",
    ["CPU"]         = nil,
})
return Config

Top level Fields

These are fields that apply to all push targets.

Config.FollowStart

This forces the push to start processing at the beginning of the capture file. Usually this is the behavior required for fully lossless processing.

Disabling this can be helpful when the capture rate is very high, FollowStart = false has advantages to start processing data real-time immediately.

Config.Decap

Enables the PCAP de-encapsulation process to be run. This will remove VLANs, MLPS, GRE and custom headers, such as timestamping.

This helps remove all the encapsulation around the inner payload, to make processing the PCAPs inner payload simpler.

Config.TimeoutRing

Custom setting for low bandwidth PCAP pushes

Config.TSModeDisable

This enable the per physical capture port timestamp decoding to be configured. e.g. on a per port basis enabling Arista 7280 / Arista 7130 / Arista 7150 / Cisco 3350 etc decoding of timestamp information.

When configuring the FMADIO Device where its attached to 2 different TapAgg Device each physical port can specifiy what hardware timestamp to use.

Config.StreamCatCmd

This is general work around field, any stream_cat command line argument can be applied here. Generally this is used to configure just released Push PCAP functionality

Per Target Fields

Each individual PCAP file can be customized using these fields

These fields are for settings located inside the Config.Target table. Example below.

table.insert(Config.Target,
{
    ["Desc"]        = "pushtest",
    ["Mode"]        = "File",
    ["Path"]        = "/mnt/store0/pushpcap/",
    ["Split"]       = "--split-time 60e9",
    ["SplitCmd"]    = "",
    ["PipeCmd"]     = "zstd -c -T0 ",
    ["FileName"]    = "--filename-tstr-HHMMSS",
    ["FilterBPF"]   = "udp and port 45010",
    ["FilterFrame"] = "",
    ["FileSuffix"] = ".pcap.zstd",
    ["CPU"]         = nil,
})

Desc

This provides a unique way to identify this push. The name must be unique

Logfiles for this push are located in

/mnt/store0/log/push_pcap_<desc>.cur

Mode

What transport mode to push data to. Usually this is FILE, as in write a normal FILE to local or remote linux file system.

Path

The usage of Path depends on the MODE, this the target destination location of the outputted pcap

Example is as follows

["Path"] = os.date("/mnt/store1/pcap/%Y%m%d/", PCAPTS),

The above example using FILE mode will write PCAPs to the above location. The system will create the directory on the file system.

This example uses the PCAP timestamp in the %Y%m%d e.g. YYYYMMDD Year / Month / Day file format which can be helpful when processing captures offline.

Split

Indicates how to split the captured data stream, typically this is done by time and less frequently by data size.

Both commands can use Scientific Notation and also math operations.

Examples are

Split by 1 Minute

--split-time 60e9

Split by 1 Hour

--split-time 60*60e9

Split by 1 GB

--split-byte 1e9

FilterBPF

This applies a BPF filter to selectively fetch only specific packets for this target. Typical example would be splitting TCP vs UDP traffic, or specific Subnets or Multicast groups

Standard BPF syntax applies

Example: To filter only TCP port 80 traffic

["FilterBPF"] = "tcp and port 80",

FilterFrame

This applies a Frame filter to the push target. Frame filters filter for everything around the payload of the packet. For example filtering on physical capture port.

Example: To filter only on FMADIO Capture port 0

["FilterFrame"] = "capture.port==0",

SplitCmd

SplitCmd is used for additional arguments sent to the pcap_splitter file. Current options are

PipeCmd

Default (none)

Pipe commands are processed on a per PCAP split basis before the end transport is applied. Examples to use this are to GZIP or compress files before sending to the endpoint.

This is a generic stdin/stdout linux application, gzip, lz4 are current examples, Other options are possible, please contact us for more details

PipeCmd="gzip -1 -c"

The above runs gzip with compression level 1 on the split PCAP before sending to the output location. Some examples are shown below

FileSuffix

Default (none)

By default the split PCAP filename suff is .pcap For most operations that is sufficient, however for more complicated operations such as GZIP compressing with PipeCmd a .pcap.gz file suffix is more appropriate. The Following is an example config target that compresses and outputs splits in .pcap.gz file format

table.insert(Config.Target, 
{ 
    Desc       = "pcap-all-gz", 
    Mode       = "rclone", 
    Path       = "s3-fmad://pcap/all", 
    Split      = "--split-time 60e9", 
    FileName   = "--filename-tstr-HHMMSS", 
    FilterBPF  = nil, 
    PipeCmd    = "gzip -c", 
    FileSuffix = ".pcap.gz" 
})

The above example pushes gzip 1minute PCAP splits to an S3 protocol storage device

CPU

Requires FW: 7750+

Sets a specific CPU for stream_cat to run on by overriding the default CPU setting. This is helpful when multiple pushes are running in parallel.

Default: the system assigned CPU number for push (typically CPU 23)

Example as follows

CPU = 30,