Examples

Following is some reference example push_pcap.lua configuration files

Push to NFS Share with BPF Filter and 1 minute PCAPs

local Config = {}

Config.Target = {}

-- push all tcp data to /mnt/remote0/push/tcp_*.pcap
table.insert(Config.Target, 
{
    Desc      = "nfs-tcp", 
    Mode      = "File", 
    Path      = "/mnt/remote0/push/tcp",   
    Split     = "--split-time 60e9", 
    FileName  = "--filename-epoch-sec-startend", 
    FilterBPF = "net 192.168.1.0/24 and tcp" 
})

return Config

Push to NFS Share with BPF Filter and HHMMSS Timezone

Example pushes a single UDP multicast group 1001 at 1 minute snapshots using an Hour Min Sec with Timezone filename.

local Config = {}

Config.Target = {}

-- push all multicast port 10001 data to /mnt/remote0/push/udp-10001_*.pcap
table.insert(Config.Target, 
{
    Desc      = "udp-multicast-1001", 
    Mode      = "File", 
    Path      = "/mnt/remote0/push/udp-10001",   
    Split     = "--split-time 60e9", 
    FileName  = "--filename-tstr-HHMMSS_TZ", 
    FilterBPF = "multicast and port 10001" 
})

Push to NFS Share 1min Splits with BPF Filter and LZ4 compression

Example pushes 1min PCAPs with a BPF filter (port 80) and applying LZ4 compression. LZ4 compression is fast and reasonably good compression rates.

local Config = {}
Config.Target = {}
table.insert(Config.Target,
{
    Desc      = "port80-lz4",
    Mode      = "File",
    Path      = "/mnt/store0/tmp2/push/udp-10001",
    Split     = "--split-time 60e9",
    FileName  = "--filename-tstr-HHMMSS_TZ",
    FilterBPF = "port 80",
    PipeCmd   = "lz4 -c",
    FileSuffix   = ".pcap.lz4",
})

return Config

Push to NFS share 1min Splits with BPF Filter and ZSTD compression

Example pushes 1min PCAPs with a BPF filter (port 80) and applying ZSTD compression. ZSTD is a new compression format with performance close to LZ4 but compression rates close to GZIP.

local Config = {}

Config.Target = {}

table.insert(Config.Target,
{
    Desc      = "port80 zstd",
    Mode      = "File",
    Path      = "/mnt/store0/tmp2/push/udp-10001",
    Split     = "--split-time 60e9",
    FileName  = "--filename-tstr-HHMMSS_TZ",
    FilterBPF = "port 80",
    PipeCmd   = "zstd -c",
    FileSuffix   = ".pcap.zstd",
})

return Config

Push to NFS/CIFS Share 1GB splits

Example pushes the raw data to a remote NFS/CIFS (Windows Share) splitting by 1GB file size writing a gzip compressed PCAP file to the remote location.

local Config = {}

Config.Target = {}

-- push everything to /mnt/remote0/push/capture*.pcap.gz at 1GB splits compressed gz
table.insert(Config.Target, 
{
    Desc      = "capture", 
    Mode      = "File", 
    Path      = "/mnt/remote0/push/capture",   
    Split     = "--split-size 1e9", 
    FileName  = "--filename-tstr-HHMMSS",
    FileSuffix = "pcap.gz", 
    FilterBPF = "" 
})

return Config

Push to MAGPACK over FTP

local Config = {}

Config.Target = {}

table.insert(Config.Target,
{
        Desc            = "magpack-all",
        Mode            = "CURL",
        Path            = "ftp://192.168.1.100/device-prefix",
        Split           = "--split-byte 1e9",
        FileName        = "--filename-tstr-HHMMSS_SUB",
        FilterBPF       = "not (host 192.168.1.105 or host 192.168.1.102)",
        Chunked         = true,
        FollowStart     = true,
        ScriptNew       = "/opt/fmadio/analytics/push_realtime_checkremote.lua 10"
})

return Config

Push to AWS S3 Bucket with Compression

Pushing captured PCAP data from the local device to AWS S3 Bucket can be done using the RCLONE support.

Below is an example push_pcap.lua config file for that

-- autogenerated Thu Jun  9 20:07:14 2022 from fmadio_config
local Config = {}
Config.FollowStart = true
Config.Decap       = true
Config.Target      = {}
table.insert(Config.Target,
{
    ["Desc"]        = "S3Cloud",
    ["Mode"]        = "RCLONE",
    ["Path"]        = "fmadio-s3://fmadio-pcap/pcap/fmadio20p3-coffee/full",
    ["Split"]       = "--split-time 60e9",
    ["SplitCmd"]    = "",
    ["PipeCmd"]     = " gzip -c ",
    ["FileSuffix"] = ".gz",
    ["FileName"]    = "--filename-tstr-HHMM",
    ["FilterBPF"]   = "",
    ["FilterFrame"] = "",
})
return Config

This uses gzip to compress the data. Also note we added a PreCapture filter to 64B Slice all traffic to AWS S3 IP address. This prevents the capture size for a run-away explosion.

Below is the resulting output in AWS S3

This does require RCLONE S3 Config to be configured before using.