--- title: "Offline Processing" slug: "offline-processing" updated: 2025-10-11T11:34:48Z published: 2025-10-11T11:34:48Z canonical: "docs.fmad.io/offline-processing" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.fmad.io/llms.txt > Use this file to discover all available pages before exploring further. # Offline Processing PCAP2JSON can process data in an offline mode, e.g. on historical PCAP data. Typically this is used for debugging problems and system issues, such as configuration files or Clickhouse database settings. The following commands are used for an offline push ## Step 1) Disable realtime pcap2json Check realtime processing has been disabled, either in the GUI or CLI. For the CLI run the the following command ```plaintext fmadiocli show analytics schedule | grep pcap2json ``` If any line or any Y is shown, pls disable pcap2json from running. **For offline processing, realtime processing must be disabled** ## Step 2) Find the capture file to process The system requires the PCAP data to be on the FMADIO Capture Storage array. This can be done using the following command ```plaintext fmadiocli show capture list ``` Example output shown below ```plaintext fmadio@fmadio200v4-636:~$ fmadiocli show capture [Sat Oct 11 09:50:34 2025] Showing captures [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250924_0000                                                          ]       80,455,663,616 B (Wed . 07:18:43 . 24-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250923_0000                                                          ]    1,013,642,690,560 B (Tue . 23:59:59 . 23-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250922_0331                                                          ]      925,163,061,248 B (Mon . 23:59:58 . 22-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250922_0000                                                          ]       92,932,145,152 B (Mon . 03:24:49 . 22-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_1057                                                          ]      898,390,032,384 B (Sun . 23:59:59 . 21-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_1050                                                          ]            4,718,592 B (Sun . 10:50:24 . 21-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_1036                                                          ]           48,234,496 B (Sun . 10:38:08 . 21-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_1000                                                          ]       16,692,019,200 B (Sun . 10:28:34 . 21-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250921_0000                                                          ]       95,164,039,168 B (Sun . 09:53:53 . 21-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250920_0000                                                          ]      927,700,353,024 B (Sat . 23:59:59 . 20-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250919_0000                                                          ]      962,859,368,448 B (Fri . 23:59:59 . 19-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250918_0000                                                          ]      970,845,585,408 B (Thu . 23:59:59 . 18-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250917_0000                                                          ]    1,037,491,503,104 B (Wed . 23:59:59 . 17-09-2025) [Sat Oct 11 09:50:34 2025] [sg2_wan0_20250916_1513                                                          ]      787,077,660,672 B (Tue . 23:59:59 . 16-09-2025) . . ``` If there is a specific capture of interest a grep can be used per below. In this case we know the capture name is `interop17` ```plaintext fmadio@fmadio200v4-636:~$ fmadiocli show capture list | grep interop [Sat Oct 11 09:52:26 2025] [interop17_20250905_1024                                                         ]      100,262,215,680 B (Fri . 13:45:03 . 09-06-2017) fmadio@fmadio200v4-636:~$ ``` Per above we have found the full capture name ```plaintext interop17_20250905_1024                                                         ``` ### Step 3) Confirm pcap2json configuration file Check the pcap2json configuration file is correct. The file is located in ```plaintext /opt/fmadio/etc/pcap2json.lua ``` If the file is not there, a reference file is located in ```plaintext /opt/fmadio/etc_ro/pcap2json.lua ``` NOTE: the default config file is configured for LXC pcap2json Clickhouse/Grafana backend. This may need to be configured ## Step 4) Run the offline process The offline analytics script is located in ```plaintext /opt/fmadio/analytics/pcap2json_realtime.lua ``` Use the above full capture name from Step 2) as the argument to the script. Example shown below. ```plaintext sudo /opt/fmadio/analytics/pcap2json_realtime.lua  --offline ``` Example output shown below ```plaintext fmadio@fmadio200v4-636:~$ sudo /opt/fmadio/analytics/pcap2json_realtime.lua  --offline interop17_20250905_1024 fmad fmadlua Sep  4 2025 (/opt/fmadio/bin/fmadiolua /opt/fmadio/analytics/pcap2json_realtime.lua --offline interop17_20250905_1024 ) Args: 1 --offline Args: 2 interop17_20250905_1024 OpenCtrl [/opt/fmadio/status/analytics] (fSysAnalytics_t*) Length 1048576B Cmd[sudo killall stream_cat] Cmd[sudo killall pcap2json] Cmd[sudo killall pcap2json_backend] Cmd[sudo mkdir -p /mnt/store0/protocol/pcap2json] Cmd[sudo chown fmadio.staff /mnt/store0/protocol/pcap2json] Got BPF Filter [] Got  Filter [] Watchdog Timeout: 300 . . StartChunk: 47246298 Offset: 0 Stride: 1 StartChunk: 47246298 SHM Ring Name [/stream_cat_1760179854802560000] SHMRing Memory: 4.25 MB Spining up worker thread CPU:36 [Sat Oct 11 10:50:54 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179854,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 0.00,"FE1_Klps":    0.00,"PCAPPendingByte":           0,"BE_LagSec":       0,"BE_dT": 0.0,"BE_dGB": 0.000,"BE_FlowPerSnapshot":       0,"BE_Gbps":  0.00,"BE_Mpps":  0.00,"Out_PushCnt":       0,"Out_Error":       0,"Out_DocCnt":       0,"Out_DocsPerSecK":   0.0,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    1/  300"} [Sat Oct 11 10:50:55 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179855,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 0.00,"FE1_Klps":    0.00,"PCAPPendingByte":           0,"BE_LagSec":263164261,"BE_dT": 1.0,"BE_dGB": 0.730,"BE_FlowPerSnapshot":       0,"BE_Gbps":  1.46,"BE_Mpps":  0.16,"Out_PushCnt":       1,"Out_Error":       0,"Out_DocCnt":    2567,"Out_DocsPerSecK":   2.6,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"} [Sat Oct 11 10:50:56 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179856,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 0.00,"FE1_Klps":    0.00,"PCAPPendingByte": 91538000000,"BE_LagSec":263164244,"BE_dT": 1.0,"BE_dGB": 5.760,"BE_FlowPerSnapshot":    6433,"BE_Gbps": 10.34,"BE_Mpps":  1.12,"Out_PushCnt":      19,"Out_Error":       0,"Out_DocCnt":  122399,"Out_DocsPerSecK": 118.1,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"} [Sat Oct 11 10:50:58 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179858,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 2.56,"FE1_Klps":    6.62,"PCAPPendingByte": 86015000000,"BE_LagSec":263164228,"BE_dT": 1.0,"BE_dGB": 5.470,"BE_FlowPerSnapshot":    6478,"BE_Gbps": 15.89,"BE_Mpps":  1.74,"Out_PushCnt":      37,"Out_Error":       0,"Out_DocCnt":  239710,"Out_DocsPerSecK": 117.2,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"} [Sat Oct 11 10:50:59 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179859,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 2.54,"FE1_Klps":    8.60,"PCAPPendingByte": 75060000000,"BE_LagSec":263164211,"BE_dT": 1.0,"BE_dGB": 5.970,"BE_FlowPerSnapshot":    6528,"BE_Gbps": 20.42,"BE_Mpps":  2.23,"Out_PushCnt":      55,"Out_Error":       4,"Out_DocCnt":  358970,"Out_DocsPerSecK": 119.1,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"} [Sat Oct 11 10:51:00 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179860,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 2.49,"FE1_Klps":    6.64,"PCAPPendingByte": 69575000000,"BE_LagSec":263164195,"BE_dT": 1.0,"BE_dGB": 5.450,"BE_FlowPerSnapshot":    7603,"BE_Gbps": 23.31,"BE_Mpps":  2.56,"Out_PushCnt":      73,"Out_Error":      30,"Out_DocCnt":  552178,"Out_DocsPerSecK": 193.0,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"} [Sat Oct 11 10:51:01 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179861,"IsUp_Backend":true,"IsUp_stream_cat":true,"IsUp_Frontend":true,"FE1_Gbps": 2.51,"FE1_Klps":    6.76,"PCAPPendingByte": 64080000000,"BE_LagSec":263164160,"BE_dT": 2.0,"BE_dGB":11.260,"BE_FlowPerSnapshot":    7300,"BE_Gbps": 27.64,"BE_Mpps":  3.04,"Out_PushCnt":     109,"Out_Error":     102,"Out_DocCnt":  795808,"Out_DocsPerSecK": 122.8,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"} . . . 20251011_105112 : stream_cat FMADRING Waiting worker thread 0 20251011_105112 : FMADRING Worker thread exiting 382470 PayloadCRC:00000000 Exit:0 20251011_105112 : straem_cat FMADRING exit is clean 20251011_105112 18.817s : Pkt:86781871 Byte:98871346464 SUCCESS [Sat Oct 11 10:51:13 2025] pcap2json {"module":"pcap2json","subsystem":"monitor","timestamp":1760179872,"IsUp_Backend":false,"IsUp_stream_cat":false,"IsUp_Frontend":false,"FE1_Gbps": 2.58,"FE1_Klps":    7.12,"PCAPPendingByte":  3983000000,"BE_LagSec":263163971,"BE_dT": 1.5,"BE_dGB": 7.610,"BE_FlowPerSnapshot":    6927,"BE_Gbps": 43.98,"BE_Mpps":  4.84,"Out_PushCnt":     310,"Out_Error":     504,"Out_DocCnt": 2183005,"Out_DocsPerSecK": 109.4,"Out_Mbps":0.000,"Out_CompressRatio":0.000,"Watchdog_Error":"    0/  300"} [Sat Oct 11 10:51:13 2025] PCAP2JSON existed [Sat Oct 11 10:51:13 2025] Shutting down sleep 0 backend:false Cmd[sudo killall stream_cat] Cmd[sudo killall pcap2json] Cmd[sudo killall pcap2json_backend] [Sat Oct 11 10:51:13 2025] pcap2json {"module":"pcap2json","subsystem":"system","timestamp":1760179873,"event":"stop","description":"RunLoop:0"} [Sat Oct 11 10:51:13 2025] 0 finished Took: 0.40710804906667 minoffline mode exiting [Sat Oct 11 10:51:13 2025] finished Took: 0.4071083008 mindone 24.448984Sec 0.407483Min fmadio@fmadio200v4-636:~$ ``` ### Step 5) Check the log files Confirm the offline process ran correctly, the following few sanity checks can be applied. Check all the processes are up, as shown below ![](https://cdn.document360.io/b9aab0c4-36d0-4616-9673-30b661455571/Images/Documentation/image(375).png?sv=2026-02-06&spr=https&st=2026-11-13T18%3A04%3A12Z&se=2026-11-13T19%3A04%3A12Z&sr=c&sp=r&sig=dlKGNEsGk%2FYdv1wPlCsvJQ9ioQyBNacdPSBE2BTvZoo%3D) Check the Out_PushCnt is incrementing. This is the number of data batches sent to the Clickhouse Database. Out_DocCnt is the total number of rows sent ![](https://cdn.document360.io/b9aab0c4-36d0-4616-9673-30b661455571/Images/Documentation/image(377).png?sv=2026-02-06&spr=https&st=2026-11-13T18%3A04%3A12Z&se=2026-11-13T19%3A04%3A12Z&sr=c&sp=r&sig=dlKGNEsGk%2FYdv1wPlCsvJQ9ioQyBNacdPSBE2BTvZoo%3D) The Out_Error should be 0 ![](https://cdn.document360.io/b9aab0c4-36d0-4616-9673-30b661455571/Images/Documentation/image(378).png?sv=2026-02-06&spr=https&st=2026-11-13T18%3A04%3A12Z&se=2026-11-13T19%3A04%3A12Z&sr=c&sp=r&sig=dlKGNEsGk%2FYdv1wPlCsvJQ9ioQyBNacdPSBE2BTvZoo%3D) ## Step 6) Check the database Confirm the database is up and running correctly. After the above push there should be non-zero number of rows. Quick check using the command below (Executed inside the pcap2json LXC) ```plaintext clickhouse-client -q "select count(*) from pcapflow.meta_1sec" ``` Example output shown below, which shows 2.1M rows in the database ```plaintext root@fmadio200v4-636-pcap2json:~/pcap2json# clickhouse-client -q "select count(*) from pcapflow.meta_1sec" 2173481 root@fmadio200v4-636-pcap2json:~/pcap2json# ``` ## Step 7) Check the GUI / Grafana Confirm data is in in Grafana, note the time range. When pushing historical data the time range may need to be expanded to previous years (depending on the source captured data) ![](https://cdn.document360.io/b9aab0c4-36d0-4616-9673-30b661455571/Images/Documentation/image(379).png?sv=2026-02-06&spr=https&st=2026-11-13T18%3A04%3A12Z&se=2026-11-13T19%3A04%3A12Z&sr=c&sp=r&sig=dlKGNEsGk%2FYdv1wPlCsvJQ9ioQyBNacdPSBE2BTvZoo%3D) default login/pass is admin fmad-secret Go to the PCAP2JSON Dashboard ![](https://cdn.document360.io/b9aab0c4-36d0-4616-9673-30b661455571/Images/Documentation/image(380).png?sv=2026-02-06&spr=https&st=2026-11-13T18%3A04%3A12Z&se=2026-11-13T19%3A04%3A12Z&sr=c&sp=r&sig=dlKGNEsGk%2FYdv1wPlCsvJQ9ioQyBNacdPSBE2BTvZoo%3D) Change the time window to 10 Years (as the data we ingested is from 2017) ![](https://cdn.document360.io/b9aab0c4-36d0-4616-9673-30b661455571/Images/Documentation/image(381).png?sv=2026-02-06&spr=https&st=2026-11-13T18%3A04%3A12Z&se=2026-11-13T19%3A04%3A12Z&sr=c&sp=r&sig=dlKGNEsGk%2FYdv1wPlCsvJQ9ioQyBNacdPSBE2BTvZoo%3D) Zoom in for a bit more detail ![](https://cdn.document360.io/b9aab0c4-36d0-4616-9673-30b661455571/Images/Documentation/image(382).png?sv=2026-02-06&spr=https&st=2026-11-13T18%3A04%3A12Z&se=2026-11-13T19%3A04%3A12Z&sr=c&sp=r&sig=dlKGNEsGk%2FYdv1wPlCsvJQ9ioQyBNacdPSBE2BTvZoo%3D) Confirmed the GUI looks correct. ## Step 8) Finished Offline data ingestion has compelted --- ## Debugging Some useful log files for debugging pcap2json backend stderr, useful for Connectivity checks. e.g. `/opt/fmadio/etc/pcap2json.lua` Is pointing to the wrong IP/Port ```plaintext tail -F /mnt/store0/log/pcap2json_backend.stderr ``` General pcap2json backend log file ```plaintext tail -F /mnt/store0/log/pcap2json_backend.cur ``` Inside the LXC (where Clickhouse is running) The clickhouse error log file ```plaintext /mnt/log/clickhouse/clickhouse-server.err.log ```