Manual Offline Push PCAP

  • Updated on Dec 25, 2024
  • Published on Dec 25, 2024
  • 6 minute(s) read
  • aA
 Prev Next

Generally push gets run in realtime, filtering, splitting and pushing data once it has been captured. Pushing historically captured data can be quite useful for many reasons

  • push_pcap.lua configuration file has changed

  • Testing and debugging

The system supports an “offline push” the steps are shown below

Step 1) Disable any realtime push

Temporarily disable any realtime push that is currently running. Usually this is done via the GUI on the Analytics Schedule page by disabling the push for all days.

Example shown below:

Step 2) Confirm push is not running

Confirm no push is actively by running the below on the bash prompt

ps aux | grep push_pcap

The expected output when no push pcap is running is shown below

fmadio@fmadio200v4-636:~$ ps aux | grep push_pcap
49628 fmadio   grep -s push_pcap
fmadio@fmadio200v4-636:~$

Step 3) Find the full capture name to process

Find the full capture name to process using FMADIOCLI Example command is shown below

fmadiocli "show capture list"

Example output shown below

fmadio@fmadio200v4-636:~$ fmadiocli  "show capture list"
fmad fmadlua Oct 18 2024 (/opt/fmadio/bin/fmadiolua --nocal /opt/fmadio/bin/fmadiocli show capture list )
Disable cycle calibration
[Wed Dec 25 13:23:06 2024] CmdLine [show capture list]
[Wed Dec 25 13:23:06 2024] Cmd [show capture list]
[Wed Dec 25 13:23:06 2024] Showing captures
[Wed Dec 25 13:23:06 2024] [asdf_20241217_1914                                                              ]      135,375,880,192 B (Tue . 23:59:59 . 17-12-2024)
[Wed Dec 25 13:23:06 2024] [asdf_20241217_1331                                                              ]      789,110,849,536 B (Tue . 16:58:10 . 17-12-2024)
[Wed Dec 25 13:23:06 2024] [wan0_20241217_1319                                                              ]       49,860,837,376 B (Tue . 04:15:29 . 17-12-2024)
[Wed Dec 25 13:23:06 2024] [asdf_20241216_1505                                                              ]       96,025,968,640 B (Mon . 15:05:53 . 16-12-2024)
[Wed Dec 25 13:23:06 2024] [asdf_20241216_1503                                                              ]       96,025,968,640 B (Mon . 15:03:43 . 16-12-2024)
[Wed Dec 25 13:23:06 2024] [asdf_20241216_1444                                                              ]       48,018,489,344 B (Mon . 14:45:34 . 16-12-2024)
[Wed Dec 25 13:23:06 2024] [asdf_20241122_2211_47                                                           ]      636,793,651,200 B (Thu . 07:31:47 . 01-01-1970)
done 0.090724Sec 0.001512Min
fmadio@fmadio200v4-636:~$

In this example we are using the wan0_20241217_1319 capture file.

Step 4) Run the offline push

Offline push is specified by manually running the push_pcap.lua script directly with the —offline <capture name> command line argument.

  • <capture name> is the full capture name found in step 2 above

sudo /opt/fmadio/analytics/push_pcap.lua --offline <capture name>

Example output shown below

fmadio@fmadio100v2-228U:~$ sudo /opt/fmadio/analytics/push_pcap.lua  --offline teleco_20241218_1922

fmad fmadlua Dec 10 2024 (/opt/fmadio/bin/fmadiolua /opt/fmadio/analytics/push_pcap.lua --offline teleco_20241218_1922 )
Config name [push_pcap]
Wed Dec 25 13:27:05 2024|{"module":"push_pcap","subsystem":"event","event_type":"config","timestamp":1735104425,"ver":"10403","instance":"push_pcap"}
Offline TSStart {"Id":                  1396,"Name":                    "teleco_20241218_1922","StreamByte":    110280835072,"StreamByteStr":   "102GB","StreamBytePct":        0.981114,"PktCnt":              112632941,"PktByte":            108198091703,"TSBeginNS":               1696597672960768794,"TSBeginLocal":     "2023-10-06 21:07:52.960.768.794","TSBeginUTC":         "2023-10-06 13:07:52.960.768.794 UTC","TSEndNS":                1696597683174550272,"TSEndLocal":       "2023-10-06 21:08:03.174.550.272","TSEndUTC":           "2023-10-06 13:08:03.174.550.272 UTC","TSDurationNS":   10213781478,"PktAvgBps":                84746744923.849060,"PktAvgPps":                 11027545.894007,"PctComp":               0.000,"PctInv":                 0.000,"PctCap":                 0.000,"PctCache":               0.000,"PctDisk":                0.000,"PctDrop":                0.000,"zero":              0}
Push Config File [/opt/fmadio/etc/push_pcap.lua]
OpenCtrl [/opt/fmadio/status/analytics] (fSysAnalytics_t*) Length 1048576B
KillAll [push_pcap] StreamUID[nil] Search[push_pcap]
Cmd[sudo /opt/fmadio/bin/pipe_fwd --null --disable-cyclecal --ring-reset --ring /opt/fmadio/queue/pcap_HTTP]
cycle calibration disabled
RING reset
RING file [/opt/fmadio/queue/pcap_HTTP]
RING[/opt/fmadio/queue/pcap_HTTP                       ] Size missmatch 0 12595200
RING[/opt/fmadio/queue/pcap_HTTP                       ] Size   : 12595200 16777216
RING[/opt/fmadio/queue/pcap_HTTP                       ] Version:        0      100
RING[/opt/fmadio/queue/pcap_HTTP                       ] version wrong force reset
RING[/opt/fmadio/queue/pcap_HTTP                       ] Put:0 0 0x7f007b138000
RING[/opt/fmadio/queue/pcap_HTTP                       ] Get:0 0 0x7f007b139000
Cmd[sudo /bin/chmod uga+rw /opt/fmadio/queue/pcap_HTTP]
Cmd[mkdir -p /mnt/store0/pushpcap/]
sudo /opt/fmadio/bin/stream_cat --uid push_pcap_Z_1735104425791180032 teleco_20241218_1922  --decap --nop-truncate --ring-eof    --ring /opt/fmadio/queue/pcap_HTTP  --ring-filter-bpf /opt/fmadio/queue/pcap_HTTP "tcp and port 80"  --ring-filter-frame /opt/fmadio/queue/pcap_HTTP ""  > /mnt/store0/log/push_pcap_stream_20241225_1327 2>&1 &
Cmd[sudo /opt/fmadio/bin/stream_cat --uid push_pcap_Z_1735104425791180032 teleco_20241218_1922  --decap --nop-truncate --ring-eof    --ring /opt/fmadio/queue/pcap_HTTP  --ring-filter-bpf /opt/fmadio/queue/pcap_HTTP "tcp and port 80"  --ring-filter-frame /opt/fmadio/queue/pcap_HTTP ""  > /mnt/store0/log/push_pcap_stream_20241225_1327 2>&1 &]
Cmd[rm /mnt/store0/log/push_pcap_stream.cur]
Cmd[ln -s /mnt/store0/log/push_pcap_stream_20241225_1327 /mnt/store0/log/push_pcap_stream.cur]
/opt/fmadio/bin/pcap_split --uid push_pcap_Z_1735104425791180032_HTTP  --ring /opt/fmadio/queue/pcap_HTTP  --split-time 60000000000 --filename-tstr-HHMMSS --roll-period 3600000000000   --script-close "/opt/fmadio/analytics/push_pcap_close.lua"  -o /mnt/store0/pushpcap/  > /mnt/store0/log/push_pcap_HTTP_20241225_1327 2>&1 &
Cmd[/opt/fmadio/bin/pcap_split --uid push_pcap_Z_1735104425791180032_HTTP  --ring /opt/fmadio/queue/pcap_HTTP  --split-time 60000000000 --filename-tstr-HHMMSS --roll-period 3600000000000   --script-close "/opt/fmadio/analytics/push_pcap_close.lua"  -o /mnt/store0/pushpcap/  > /mnt/store0/log/push_pcap_HTTP_20241225_1327 2>&1 &]
Cmd[rm /mnt/store0/log/push_pcap_HTTP.cur]
rm: can't remove '/mnt/store0/log/push_pcap_HTTP.cur': No such file or directory
Cmd[ln -s /mnt/store0/log/push_pcap_HTTP_20241225_1327 /mnt/store0/log/push_pcap_HTTP.cur]
Wed Dec 25 13:27:05 2024|{"module":"push_pcap","subsystem":"event","event_type":"start","timestamp":1735104425,"instance":"push_pcap"}
Wed Dec 25 13:27:06 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1735104425,"ver":"10403","instance":"push_pcap","Process":"stream_cat                    ","IsUp":  true}
Wed Dec 25 13:27:06 2024|{"module":"push_pcap","subsystem":"status","timestamp":1735104425,"ver":"10403","instance":"push_pcap","Process":"HTTP                          ","IsUp":  true,"Splits":    0,"TotalByte":               0,"TotalPkt":               0,"TransferMbps":      0.00,"PCAPTS":0,"FilterBPF":"tcp and port 80","FilterFrame":"","Target":"/mnt/store0/pushpcap/"}
Wed Dec 25 13:27:07 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1735104427,"ver":"10403","instance":"push_pcap","Process":"stream_cat                    ","IsUp":  true}
Wed Dec 25 13:27:07 2024|{"module":"push_pcap","subsystem":"status","timestamp":1735104427,"ver":"10403","instance":"push_pcap","Process":"HTTP                          ","IsUp":  true,"Splits":    0,"TotalByte":               0,"TotalPkt":               0,"TransferMbps":      0.00,"PCAPTS":0,"FilterBPF":"tcp and port 80","FilterFrame":"","Target":"/mnt/store0/pushpcap/"}
Wed Dec 25 13:27:08 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1735104428,"ver":"10403","instance":"push_pcap","Process":"stream_cat                    ","IsUp":  true}
Wed Dec 25 13:27:08 2024|{"module":"push_pcap","subsystem":"status","timestamp":1735104428,"ver":"10403","instance":"push_pcap","Process":"HTTP                          ","IsUp":  true,"Splits":    0,"TotalByte":               0,"TotalPkt":               0,"TransferMbps":      0.00,"PCAPTS":0,"FilterBPF":"tcp and port 80","FilterFrame":"","Target":"/mnt/store0/pushpcap/"}
Wed Dec 25 13:27:09 2024|{"module":"push_pcap","subsystem":"stream_cat","timestamp":1735104429,"ver":"10403","instance":"push_pcap","Process":"stream_cat                    ","IsUp":  true}
Wed Dec 25 13:27:09 2024|{"module":"push_pcap","subsystem":"status","timestamp":1735104429,"ver":"10403","instance":"push_pcap","Process":"HTTP                          ","IsUp":  true,"Splits":    0,"TotalByte":               0,"TotalPkt":               0,"TransferMbps":      0.00,"PCAPTS":0,"FilterBPF":"tcp and port 80","FilterFrame":"","Target":"/mnt/store0/pushpcap/"}
.
.

Wed Dec 25 13:27:35 2024 stream cat existed
app:stream_cat false
app:pcap_split false
total: false
KillAll [push_pcap] StreamUID[push_pcap_Z_1735104425791180032] Search[push_pcap]
Wed Dec 25 13:27:35 2024|{"module":"push_pcap","subsystem":"event","event_type":"stop","timestamp":1735104455,"ver":"10403","instance":"push_pcap"}
Push shutting down
done 29.546377Sec 0.492440Min
fmadio@fmadio100v2-228U:~$

Step 5) Confirm PCAP file generation

Confirm the PCAP files are generated as expected in the specified PCAP directory