The Push PCAP functionality can be extremely useful for splitting filtering and generating the raw PCAP data. For many applications this meets the requirements.
In addition to this, the ability to process these raw PCAPs with arbitrary applications on the capture system can be quite beneficial.
Some examples are:
custom compression settings / algo.
generating high level packet statistics.
automatically running microburst analysis.
checking for market data gaps.
And many more. The key point is that this feature is fully scriptable enabling anything to be run on the filtered split PCAP after it has finished being generated.
Configuration
Step 1) Configure the script in push_pcap.lua
Edit the push_pcap.lua configuration file located in
/opt/fmadio/etc/push_pcap.lua
This file is created in the previous Push PCAP example.
Step 2) Enable the Post script
Add a new field into each target requiring the post processing step, per below:
["Post"] = "/opt/fmadio/etc/push_pcap_post.lua", The filename of this script can be anything, we chose push_pcap_post.lua as an example only.
A simple example push_pcap.lua config with the post processing configured is shown below:
local Config = {}
Config.FollowStart = true
Config.Decap = true
Config.Target = {}
table.insert(Config.Target,
{
["Desc"] = "all",
["Mode"] = "File",
["Path"] = "/mnt/store0/pushpcap/test_postprocess/",
["Split"] = "--split-time 60e9",
["Post"] = "/opt/fmadio/etc/push_pcap_post.lua",
["SplitCmd"] = "",
["FileName"] = "--filename-tstr-HHMMSS",
["FilterBPF"] = "",
["FilterFrame"] = ""
})
return ConfigStep 3) Configure the post processing script
For this example we are compressing the PCAPs with a custom zstd compressor, then generating an MD5 sum of the compressed archive.
The script gets called with the full path to the PCAP e.g.
/opt/fmadio/etc/push_pcap_post.lua /mnt/store0/pushpcap/test_postprocess/20240923_175959.pcap
The example code is below
#!/opt/fmadio/bin/fmadiolua
-- get the full path of the PCAP
local FileName = ARGV[1]
print(FileName)
-- compress it
local Cmd = "/usr/bin/zstdseekable_compression "..FileName.." "..(1024*1024).." "
print(Cmd)
os.execute(Cmd)
-- MD5 it
local Cmd = "/usr/bin/md5sum "..FileName..".zst > "..FileName..".zst.md5"
print(Cmd)
os.execute(Cmd)This could be a bash script, or Python script etc etc, in this location.
NOTE: Reminder to ensure the script has the executable bit set.
Step 4) Test using offline mode
The best way to test and debug the script or push, is to run an offline test. To run an offline test as follows
cd /opt/fmadio/analytics
sudo ./push_pcap.lua --offline <name of capture stream>The name of the capture stream can be found using sudo stream_dump
An example output looks like below
fmadio@fmadio100v2-228U:/opt/fmadio/analytics$ sudo ./push_pcap.lua --offline opra_port0_20240728_0405
fmad fmadlua Oct 17 2024 (/opt/fmadio/bin/fmadiolua ./push_pcap.lua --offline opra_port0_20240728_0405 )
Config name [push_pcap]
Thu Oct 17 00:50:10 2024|{"module":"push_pcap","subsystem":"event","event_type":"config","timestamp":1729140610,"ver":"10093","instance":"push_pcap"}
Offline TSStart {"Id": 0399,"Name": "opra_port0_20240728_0405","StreamByte": 20086521856,"StreamByteStr": "18GB","StreamBytePct": 0.883997,"PktCnt": 145197123,"PktByte": 17756421452,"TSBeginNS": 1722016800024241083,"TSBeginLocal": "2024-07-26 14:00:00.024.241.083","TSBeginUTC": "2024-07-26 18:00:00.024.241.083 UTC","TSEndNS": 1722016839505625856,"TSEndLocal": "2024-07-26 14:00:39.505.625.856","TSEndUTC": "2024-07-26 18:00:39.505.625.856 UTC","TSDurationNS": 39481384773,"PktAvgBps": 3597932859.567382,"PktAvgPps": 3677609.608549,"PctComp": 0.000,"PctInv": 0.000,"PctCap": 0.000,"PctCache": 0.000,"PctDisk": 0.000,"PctDrop": 0.000,"zero": 0}
Push Config File [/opt/fmadio/etc/push_pcap.lua]
OpenCtrl [/opt/fmadio/status/analytics] (fSysAnalytics_t*) Length 1048576B
KillAll [push_pcap] StreamUID[nil] Search[push_pcap]
Cmd[sudo /opt/fmadio/bin/pipe_fwd --null --disable-cyclecal --ring-reset --ring /opt/fmadio/queue/pcap_all]
RING reset
RING file [/opt/fmadio/queue/pcap_all]
RING[/opt/fmadio/queue/pcap_all ] Size : 12595200 16777216
RING[/opt/fmadio/queue/pcap_all ] Version: 100 100
RING[/opt/fmadio/queue/pcap_all ] Put:0 0 0x7f3fcd3e7000
RING[/opt/fmadio/queue/pcap_all ] Get:0 0 0x7f3fcd3e8000
Cmd[sudo /bin/chmod uga+rw /opt/fmadio/queue/pcap_all]
Cmd[mkdir -p /mnt/store0/pushpcap/test_postprocess/]
sudo /opt/fmadio/bin/stream_cat --uid push_pcap_Z_1729140610668500992 opra_port0_20240728_0405 --decap --nop-truncate --ring-eof --ring /opt/fmadio/queue/pcap_all --ring-filter-bpf /opt/fmadio/queue/pcap_all "" --ring-filter-frame /opt/fmadio/queue/pcap_all "" > /mnt/store0/log/push_pcap_stream_20241017_0050 2>&1 &
Cmd[sudo /opt/fmadio/bin/stream_cat --uid push_pcap_Z_1729140610668500992 opra_port0_20240728_0405 --decap --nop-truncate --ring-eof --ring /opt/fmadio/queue/pcap_all --ring-filter-bpf /opt/fmadio/queue/pcap_all "" --ring-filter-frame /opt/fmadio/queue/pcap_all "" > /mnt/store0/log/push_pcap_stream_20241017_0050 2>&1 &]
Cmd[rm /mnt/store0/log/push_pcap_stream.cur]
Cmd[ln -s /mnt/store0/log/push_pcap_stream_20241017_0050 /mnt/store0/log/push_pcap_stream.cur]
/opt/fmadio/bin/pcap_split --uid push_pcap_Z_1729140610668500992_all --ring /opt/fmadio/queue/pcap_all --split-time 1e9 --filename-tstr-HHMMSS --roll-period 3600000000000 --script-close "/opt/fmadio/analytics/push_pcap_close.lua" --script-post "/opt/fmadio/etc/push_pcap_post.lua" -o /mnt/store0/pushpcap/test_postprocess/ > /mnt/store0/log/push_pcap_all_20241017_0050 2>&1 &
Cmd[/opt/fmadio/bin/pcap_split --uid push_pcap_Z_1729140610668500992_all --ring /opt/fmadio/queue/pcap_all --split-time 1e9 --filename-tstr-HHMMSS --roll-period 3600000000000 --script-close "/opt/fmadio/analytics/push_pcap_close.lua" --script-post "/opt/fmadio/etc/push_pcap_post.lua" -o /mnt/store0/pushpcap/test_postprocess/ > /mnt/store0/log/push_pcap_all_20241017_0050 2>&1 &]
Cmd[rm /mnt/store0/log/push_pcap_all.cur]
Cmd[ln -s /mnt/store0/log/push_pcap_all_20241017_0050 /mnt/store0/log/push_pcap_all.cur]
Thu Oct 17 00:50:10 2024|{"module":"push_pcap","subsystem":"event","event_type":"start","timestamp":1729140610,"instance":"push_pcap"}
Thu Oct 17 00:50:10 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1729140610,"ver":"10093","instance":"push_pcap","Process":"stream_cat ","IsUp": true}
Thu Oct 17 00:50:10 2024|{"module":"push_pcap","subsystem":"status","timestamp":1729140610,"ver":"10093","instance":"push_pcap","Process":"all ","IsUp": true,"Splits": 0,"TotalByte": 0,"TotalPkt": 0,"TransferMbps": 0.00,"PCAPTS":0,"FilterBPF":"","FilterFrame":"","Target":"/mnt/store0/pushpcap/test_postprocess/"}
Thu Oct 17 00:50:11 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1729140611,"ver":"10093","instance":"push_pcap","Process":"stream_cat ","IsUp": true}
Thu Oct 17 00:50:11 2024|{"module":"push_pcap","subsystem":"status","timestamp":1729140611,"ver":"10093","instance":"push_pcap","Process":"all ","IsUp": true,"Splits": 0,"TotalByte": 0,"TotalPkt": 0,"TransferMbps": 0.00,"PCAPTS":0,"FilterBPF":"","FilterFrame":"","Target":"/mnt/store0/pushpcap/test_postprocess/"}
Thu Oct 17 00:50:12 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1729140612,"ver":"10093","instance":"push_pcap","Process":"stream_cat ","IsUp": true}
Thu Oct 17 00:50:12 2024|{"module":"push_pcap","subsystem":"status","timestamp":1729140612,"ver":"10093","instance":"push_pcap","Process":"all ","IsUp": true,"Splits": 1,"TotalByte": 40249050,"TotalPkt": 264654,"TransferMbps": 0.00,"PCAPTS":1722016800086973184,"FilterBPF":"","FilterFrame":"","Target":"/mnt/store0/pushpcap/test_postprocess/"}
Thu Oct 17 00:50:13 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1729140613,"ver":"10093","instance":"push_pcap","Process":"stream_cat ","IsUp": true}
Thu Oct 17 00:50:13 2024|{"module":"push_pcap","subsystem":"status","timestamp":1729140613,"ver":"10093","instance":"push_pcap","Process":"all ","IsUp": true,"Splits": 1,"TotalByte": 393606534,"TotalPkt": 2447880,"TransferMbps": 2369.00,"PCAPTS":1722016800396635648,"FilterBPF":"","FilterFrame":"","Target":"/mnt/store0/pushpcap/test_postprocess/"}
Thu Oct 17 00:50:14 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1729140614,"ver":"10093","instance":"push_pcap","Process":"stream_cat ","IsUp": true}
Thu Oct 17 00:50:14 2024|{"module":"push_pcap","subsystem":"status","timestamp":1729140614,"ver":"10093","instance":"push_pcap","Process":"all ","IsUp": true,"Splits": 1,"TotalByte": 724432926,"TotalPkt": 4707716,"TransferMbps": 2218.00,"PCAPTS":1722016800836585984,"FilterBPF":"","FilterFrame":"","Target":"/mnt/store0/pushpcap/test_postprocess/"}
.
.
.
Then looking in the target directory we see:
fmadio@fmadio100v2-228U:/mnt/store0/pushpcap/test_postprocess$ ls -al
total 20559280
drwxr-xr-x 2 fmadio staff 266240 Oct 17 00:51 ./
drwxr-xr-x 3 fmadio staff 4096 Oct 15 06:09 ../
-rw-r--r-- 1 root root 822066616 Oct 17 00:50 20240726_140000.pcap
-rw-r--r-- 1 root root 338052926 Oct 17 00:50 20240726_140000.pcap.zst
-rw-r--r-- 1 root root 97 Oct 17 00:50 20240726_140000.pcap.zst.md5
-rw-r--r-- 1 root root 639887576 Oct 17 00:50 20240726_140001.pcap
-rw-r--r-- 1 root root 258966315 Oct 17 00:50 20240726_140001.pcap.zst
-rw-r--r-- 1 root root 97 Oct 17 00:50 20240726_140001.pcap.zst.md5
-rw-r--r-- 1 root root 574669294 Oct 17 00:50 20240726_140002.pcap
-rw-r--r-- 1 root root 231214587 Oct 17 00:50 20240726_140002.pcap.zst
-rw-r--r-- 1 root root 97 Oct 17 00:50 20240726_140002.pcap.zst.md5
-rw-r--r-- 1 root root 452640756 Oct 17 00:50 20240726_140003.pcap
-rw-r--r-- 1 root root 179413332 Oct 17 00:50 20240726_140003.pcap.zst
-rw-r--r-- 1 root root 97 Oct 17 00:50 20240726_140003.pcap.zst.md5
-rw-r--r-- 1 root root 556311950 Oct 17 00:50 20240726_140004.pcap
-rw-r--r-- 1 root root 223171051 Oct 17 00:50 20240726_140004.pcap.zst
-rw-r--r-- 1 root root 97 Oct 17 00:50 20240726_140004.pcap.zst.md5
-rw-r--r-- 1 root root 541232776 Oct 17 00:50 20240726_140005.pcap
-rw-r--r-- 1 root root 216978904 Oct 17 00:50 20240726_140005.pcap.zst
-rw-r--r-- 1 root root 97 Oct 17 00:50 20240726_140005.pcap.zst.md5
.
.
With many files that are zstd compressed with an MD5 sum file
Step 5) Finished
When the offline output looks correct, enable in online / realtime mode
Debugging
Most of the time it requires a few iterations to get the post processing script working correctly. The logfile are located in
/mnt/store0/log/push_pcap_<desc>.cur
Where <desc> is the Desc of the push target.
In this example the Desc is “all”
.png?sv=2022-11-02&spr=https&st=2025-05-06T10%3A42%3A50Z&se=2025-05-06T11%3A02%3A50Z&sr=c&sp=r&sig=9f3eSb1tWmGKv7RptBB%2BO9AEpviWjZQIGI%2BnwxbblMA%3D "image(13).png")
There for the logfiles full name is
/mnt/store0/log/push_pcap_all.cur
Example output is shown below
fmadio@fmadio100v2-228U:/opt/fmadio/analytics$ head /mnt/store0/log/push_pcap_all.cur -n 100
args
--uid
push_pcap_Z_1729140610668500992_all
--ring
/opt/fmadio/queue/pcap_all
--split-time
1e9
--filename-tstr-HHMMSS
--roll-period
3600000000000
--script-close
/opt/fmadio/analytics/push_pcap_close.lua
--script-post
/opt/fmadio/etc/push_pcap_post.lua
-o
/mnt/store0/pushpcap/test_postprocess/
--uid
UID [push_pcap_Z_1729140610668500992_all]
--ring
Input from lxc_ring:/opt/fmadio/queue/pcap_all
--split-time
Split Every 1.000000 Sec
--filename-tstr-HHMMSS
Filename TimeString HHMMSS
--roll-period
Roll Period 1.000 hours
--script-close
Script Close Hook [/opt/fmadio/analytics/push_pcap_close.lua]
--script-post
Script Post Hook [/opt/fmadio/etc/push_pcap_post.lua]
-o
OutputName [/mnt/store0/pushpcap/test_postprocess/]
RING[/opt/fmadio/queue/pcap_all ] Size : 12595200 12595200 16777216
RING[/opt/fmadio/queue/pcap_all ] Version: 100 100
RING[/opt/fmadio/queue/pcap_all ] Path:/opt/fmadio/queue/pcap_allRING[/opt/fmadio/queue/pcap_all ] Put:0 0
RING[/opt/fmadio/queue/pcap_all ] Get:0 0
Offset to GMT is -14400000000000 (EDT)
RollTime: 1722006000000000000 478335:00:00.000.000.000
[cat > '/mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap.pending']
[0.000 H][2024-07-26 14:00:00] /mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap : Total Bytes 0.000 GB Speed: 0.000 Gbps : New Split
[480316.837 H][2024-07-26 14:00:00] /mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap : Total Bytes 40249050 264654 0.040 GB Speed: 0.000 Gbps 0.000 Mpps : TotalSplit 1 PCAPTS: 1722016800086973219
[0.000 H][2024-07-26 14:00:00] /mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap : Total Bytes 393606534 2447880 0.394 GB Speed: 2.369 Gbps 1.830 Mpps : TotalSplit 1 PCAPTS: 1722016800396635699
[0.000 H][2024-07-26 14:00:00] /mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap : Total Bytes 724432926 4707716 0.724 GB Speed: 2.218 Gbps 1.894 Mpps : TotalSplit 1 PCAPTS: 1722016800836586053
fmad fmadlua Oct 17 2024 (/opt/fmadio/bin/fmadiolua --nocal /opt/fmadio/analytics/push_pcap_close.lua /mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap 822066592 5408171 2888031000 975758976 1722016800000000000 1722016801000000000 1722016800024241083 1722016800999999565 1722016801000000059 )
[0.001 H][2024-07-26 14:00:01] /mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap : Finished : Split Bytes 822066592 (0.822 GB) Split Pkts: 5408171 WallTime: 2888031000 PCAPTime: 975758976
Script [/opt/fmadio/analytics/push_pcap_close.lua /mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap 822066592 5408171 2888031000 975758976 1722016800000000000 1722016801000000000 1722016800024241083 1722016800999999565 1722016801000000059]
[cat > '/mnt/store0/pushpcap/test_postprocess/20240726_140001.pcap.pending']
[0.001 H][2024-07-26 14:00:01] /mnt/store0/pushpcap/test_postprocess/20240726_140001.pcap : Total Bytes 0.822 GB Speed: 1.630 Gbps : New Split
fmad fmadlua Oct 17 2024 (/opt/fmadio/bin/fmadiolua /opt/fmadio/etc/push_pcap_post.lua /mnt/store0/pushpcap/test_postprocess/20240726_140000.pcap )
[0.000 H][2024-07-26 14:00:01] /mnt/store0/pushpcap/test_postprocess/20240726_140001.pcap : Total Bytes 1019636280 6815547 1.020 GB Speed: 1.979 Gbps 1.766 Mpps : TotalSplit 2 PCAPTS: 1722016801336381417
[0.000 H][2024-07-26 14:00:01] /mnt/store0/pushpcap/test_postprocess/20240726_140001.pcap : Total Bytes 1348959272 9082434 1.349 GB Speed: 2.208 Gbps 1.900 Mpps : TotalSplit 2 PCAPTS: 1722016801834124608
fmad fmadlua Oct 17 2024 (/opt/fmadio/bin/fmadiolua --nocal /opt/fmadio/analytics/push_pcap_close.lua /mnt/store0/pushpcap/test_postprocess/20240726_140001.pcap 639887552 4446588 2340251000 999999942 1722016801000000000 1722016802000000000 1722016801000000059 1722016801999999539 1722016802000000001 )
[0.002 H][2024-07-26 14:00:02] /mnt/store0/pushpcap/test_postprocess/20240726_140001.pcap : Finished : Split Bytes 639887552 (0.640 GB) Split Pkts: 4446588 WallTime: 2340251000 PCAPTime: 999999942
Script [/opt/fmadio/analytics/push_pcap_close.lua /mnt/store0/pushpcap/test_postprocess/20240726_140001.pcap 639887552 4446588 2340251000 999999942 1722016801000000000 1722016802000000000 1722016801000000059 1722016801999999539 1722016802000000001]
.
.
Intentionally creating an error (to demonstrate) each time the script gets run the error message can be seen.
We add an assertion error at the end of the post processing script:
#!/opt/fmadio/bin/fmadiolua
local FileName = ARGV[1]
print("POST Processing Sciprt: "..FileName)
-- compress it
local Cmd = "/usr/bin/zstdseekable_compression "..FileName.." "..(1024*1024).." "
print(Cmd)
os.execute(Cmd)
-- MD5 it
local Cmd = "/usr/bin/md5sum "..FileName..".zst > "..FileName..".zst.md5"
print(Cmd)
os.execute(Cmd)
-- Generate an error
assert(false)
The logfile shows this error as follows:
.png?sv=2022-11-02&spr=https&st=2025-05-06T10%3A42%3A50Z&se=2025-05-06T11%3A02%3A50Z&sr=c&sp=r&sig=9f3eSb1tWmGKv7RptBB%2BO9AEpviWjZQIGI%2BnwxbblMA%3D "image(14).png")
If you have questions, problems or feedback related to this topic please contact support@fmad.io.