Offline Processing

  • Updated on Oct 11, 2025
  • Published on Oct 11, 2025
  • 7 minute(s) read
  • aA
 Prev Next

PCAP2JSON can process data in an offline mode, e.g. on historical PCAP data.

Typically this is used for debugging problems and system issues, such as configuration files or Clickhouse database settings.

The following commands are used for an offline push

Step 1) Disable realtime pcap2json

Check realtime processing has been disabled, either in the GUI or CLI.

For the CLI run the the following command

fmadiocli show analytics schedule | grep pcap2json

If any line or any Y is shown, pls disable pcap2json from running.

For offline processing, realtime processing must be disabled

Step 2) Find the capture file to process

The system requires the PCAP data to be on the FMADIO Capture Storage array.

This can be done using the following command

 fmadiocli show capture list

Example output shown below

fmadio@fmadio200v4-636:~$ fmadiocli show capture

[Sat Oct 11 09:50:34 2025] Showing captures
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250924_0000                                                          ]       80,455,663,616 B (Wed . 07:18:43 . 24-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250923_0000                                                          ]    1,013,642,690,560 B (Tue . 23:59:59 . 23-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250922_0331                                                          ]      925,163,061,248 B (Mon . 23:59:58 . 22-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250922_0000                                                          ]       92,932,145,152 B (Mon . 03:24:49 . 22-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_1057                                                          ]      898,390,032,384 B (Sun . 23:59:59 . 21-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_1050                                                          ]            4,718,592 B (Sun . 10:50:24 . 21-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_1036                                                          ]           48,234,496 B (Sun . 10:38:08 . 21-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_1000                                                          ]       16,692,019,200 B (Sun . 10:28:34 . 21-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_0000                                                          ]       95,164,039,168 B (Sun . 09:53:53 . 21-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250920_0000                                                          ]      927,700,353,024 B (Sat . 23:59:59 . 20-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250919_0000                                                          ]      962,859,368,448 B (Fri . 23:59:59 . 19-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250918_0000                                                          ]      970,845,585,408 B (Thu . 23:59:59 . 18-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250917_0000                                                          ]    1,037,491,503,104 B (Wed . 23:59:59 . 17-09-2025)
[Sat Oct 11 09:50:34 2025] [sg2_wan0_20250916_1513                                                          ]      787,077,660,672 B (Tue . 23:59:59 . 16-09-2025)
.
.

If there is a specific capture of interest a grep can be used per below. In this case we know the capture name is interop17

fmadio@fmadio200v4-636:~$ fmadiocli show capture list | grep interop
[Sat Oct 11 09:52:26 2025] [interop17_20250905_1024                                                         ]      100,262,215,680 B (Fri . 13:45:03 . 09-06-2017)
fmadio@fmadio200v4-636:~$

Per above we have found the full capture name

interop17_20250905_1024                                                        

Step 3) Confirm pcap2json configuration file

Check the pcap2json configuration file is correct. The file is located in

/opt/fmadio/etc/pcap2json.lua

If the file is not there, a reference file is located in 

/opt/fmadio/etc_ro/pcap2json.lua

NOTE: the default config file is configured for LXC pcap2json Clickhouse/Grafana backend. This may need to be configured

Step 4) Run the offline process

The offline analytics script is located in

/opt/fmadio/analytics/pcap2json_realtime.lua

Use the above full capture name from Step 2) as the argument to the script. Example shown below.

 sudo /opt/fmadio/analytics/pcap2json_realtime.lua  --offline <capture name>

Example output shown below


fmadio@fmadio200v4-636:~$ sudo /opt/fmadio/analytics/pcap2json_realtime.lua  --offline interop17_20250905_1024
fmad fmadlua Sep  4 2025 (/opt/fmadio/bin/fmadiolua /opt/fmadio/analytics/pcap2json_realtime.lua --offline interop17_20250905_1024 )
Args: 1 --offline
Args: 2 interop17_20250905_1024
OpenCtrl [/opt/fmadio/status/analytics] (fSysAnalytics_t*) Length 1048576B
Cmd[sudo killall stream_cat]
Cmd[sudo killall pcap2json]
Cmd[sudo killall pcap2json_backend]
Cmd[sudo mkdir -p /mnt/store0/protocol/pcap2json]
Cmd[sudo chown fmadio.staff /mnt/store0/protocol/pcap2json]
Got BPF Filter []
Got  Filter []
Watchdog Timeout: 300
.
.
StartChunk: 47246298 Offset: 0 Stride: 1
StartChunk: 47246298
SHM Ring Name [/stream_cat_1760179854802560000]
SHMRing Memory: 4.25 MB
Spining up worker thread CPU:36
[Sat Oct 11 10:50:54 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179854,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 0.00,"FE1_Klps":    0.00,"PCAPPendingByte":           0,"BE_LagSec":       0,"BE_dT": 0.0,"BE_dGB": 0.000,"BE_FlowPerSnapshot":       0,"BE_Gbps":  0.00,"BE_Mpps":  0.00,"Out_PushCnt":       0,"Out_Error":       0,"Out_DocCnt":       0,"Out_DocsPerSecK":   0.0,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    1/  300"}
[Sat Oct 11 10:50:55 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179855,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 0.00,"FE1_Klps":    0.00,"PCAPPendingByte":           0,"BE_LagSec":263164261,"BE_dT": 1.0,"BE_dGB": 0.730,"BE_FlowPerSnapshot":       0,"BE_Gbps":  1.46,"BE_Mpps":  0.16,"Out_PushCnt":       1,"Out_Error":       0,"Out_DocCnt":    2567,"Out_DocsPerSecK":   2.6,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"}
[Sat Oct 11 10:50:56 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179856,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 0.00,"FE1_Klps":    0.00,"PCAPPendingByte": 91538000000,"BE_LagSec":263164244,"BE_dT": 1.0,"BE_dGB": 5.760,"BE_FlowPerSnapshot":    6433,"BE_Gbps": 10.34,"BE_Mpps":  1.12,"Out_PushCnt":      19,"Out_Error":       0,"Out_DocCnt":  122399,"Out_DocsPerSecK": 118.1,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"}
[Sat Oct 11 10:50:58 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179858,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 2.56,"FE1_Klps":    6.62,"PCAPPendingByte": 86015000000,"BE_LagSec":263164228,"BE_dT": 1.0,"BE_dGB": 5.470,"BE_FlowPerSnapshot":    6478,"BE_Gbps": 15.89,"BE_Mpps":  1.74,"Out_PushCnt":      37,"Out_Error":       0,"Out_DocCnt":  239710,"Out_DocsPerSecK": 117.2,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"}
[Sat Oct 11 10:50:59 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179859,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 2.54,"FE1_Klps":    8.60,"PCAPPendingByte": 75060000000,"BE_LagSec":263164211,"BE_dT": 1.0,"BE_dGB": 5.970,"BE_FlowPerSnapshot":    6528,"BE_Gbps": 20.42,"BE_Mpps":  2.23,"Out_PushCnt":      55,"Out_Error":       4,"Out_DocCnt":  358970,"Out_DocsPerSecK": 119.1,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"}
[Sat Oct 11 10:51:00 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179860,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 2.49,"FE1_Klps":    6.64,"PCAPPendingByte": 69575000000,"BE_LagSec":263164195,"BE_dT": 1.0,"BE_dGB": 5.450,"BE_FlowPerSnapshot":    7603,"BE_Gbps": 23.31,"BE_Mpps":  2.56,"Out_PushCnt":      73,"Out_Error":      30,"Out_DocCnt":  552178,"Out_DocsPerSecK": 193.0,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"}
[Sat Oct 11 10:51:01 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179861,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 2.51,"FE1_Klps":    6.76,"PCAPPendingByte": 64080000000,"BE_LagSec":263164160,"BE_dT": 2.0,"BE_dGB":11.260,"BE_FlowPerSnapshot":    7300,"BE_Gbps": 27.64,"BE_Mpps":  3.04,"Out_PushCnt":     109,"Out_Error":     102,"Out_DocCnt":  795808,"Out_DocsPerSecK": 122.8,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"}
.
.
.
20251011_105112 : stream_cat FMADRING Waiting worker thread 0
20251011_105112 : FMADRING Worker thread exiting 382470 PayloadCRC:00000000 Exit:0
20251011_105112 : straem_cat FMADRING exit is clean
20251011_105112 18.817s : Pkt:86781871 Byte:98871346464 SUCCESS
[Sat Oct 11 10:51:13 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179872,"IsUp_Backend":false,"IsUp_stream_cat":false,"IsUp_Frontend":false,"FE1_Gbps": 2.58,"FE1_Klps":    7.12,"PCAPPendingByte":  3983000000,"BE_LagSec":263163971,"BE_dT": 1.5,"BE_dGB": 7.610,"BE_FlowPerSnapshot":    6927,"BE_Gbps": 43.98,"BE_Mpps":  4.84,"Out_PushCnt":     310,"Out_Error":     504,"Out_DocCnt": 2183005,"Out_DocsPerSecK": 109.4,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"}
[Sat Oct 11 10:51:13 2025] PCAP2JSON existed
[Sat Oct 11 10:51:13 2025] Shutting down sleep 0 backend:false
Cmd[sudo killall stream_cat]
Cmd[sudo killall pcap2json]
Cmd[sudo killall pcap2json_backend]
[Sat Oct 11 10:51:13 2025] pcap2json {"module":"pcap2json","subsystem":"system","timestamp":1760179873,"event":"stop","description":"RunLoop:0"}
[Sat Oct 11 10:51:13 2025] 0 finished Took: 0.40710804906667 minoffline mode exiting
[Sat Oct 11 10:51:13 2025] finished Took: 0.4071083008 mindone 24.448984Sec 0.407483Min
fmadio@fmadio200v4-636:~$

Step 5) Check the log files

Confirm the offline process ran correctly, the following few sanity checks can be applied.

Check all the processes are up, as shown below

Check the Out_PushCnt is incrementing. This is the number of data batches sent to the Clickhouse Database. Out_DocCnt is the total number of rows sent

The Out_Error should be 0

Step 6) Check the database

Confirm the database is up and running correctly. After the above push there should be non-zero number of rows.

Quick check using the command below (Executed inside the pcap2json LXC)

clickhouse-client -q "select count(*) from pcapflow.meta_1sec"

Example output shown below, which shows 2.1M rows in the database

root@fmadio200v4-636-pcap2json:~/pcap2json# clickhouse-client -q "select count(*) from pcapflow.meta_1sec"
2173481
root@fmadio200v4-636-pcap2json:~/pcap2json#

Step 7) Check the GUI / Grafana

Confirm data is in in Grafana, note the time range. When pushing historical data the time range may need to be expanded to previous years (depending on the source captured data)

default login/pass is admin fmad-secret

Go to the PCAP2JSON Dashboard

Change the time window to 10 Years (as the data we ingested is from 2017)

Zoom in for a bit more detail

Confirmed the GUI looks correct.

Step 8) Finished

Offline data ingestion has compelted

Debugging

Some useful log files for debugging

pcap2json backend stderr, useful for Connectivity checks. e.g. /opt/fmadio/etc/pcap2json.lua  Is pointing to the wrong IP/Port

tail -F /mnt/store0/log/pcap2json_backend.stderr

General pcap2json backend log file

tail -F /mnt/store0/log/pcap2json_backend.cur

Inside the LXC (where Clickhouse is running)

The clickhouse error log file

/mnt/log/clickhouse/clickhouse-server.err.log