Market Insignt / pcap2json is split into 2 seperate functions
decoding of raw PCAP data into raw flow data
database and visualization of the raw flow data
There are no requirements for these processes to run on the same physical system. However, for smaller deployments they typically are (e.g. single host)
Decode Config
Following steps include how to install the Clickhouse version of the decoders
Step 1) Download
Download the TCZ package from the website, or use the TCZ package pre-installed with the LXC container
Step 2) Install on FMADIO host
Install the TCZ on the FMADIO host system, per command below
sudo plugin_reload.lua fmadio_pcap2json_basic_20250417_1442.tcz Example output shown below
fmadio@fmadio20v4-733-pro:~$ sudo plugin_install.lua fmadio_pcap2json_basic_20250417_1442.tcz [64/9602]
sudo: plugin_install.lua: command not found
fmadio@fmadio20v4-733-pro:~$ sudo plugin_reload.lua fmadio_pcap2json_basic_20250417_1442.tcz
fmad fmadlua Aug 15 2025 (/opt/fmadio/bin/fmadiolua /opt/fmadio/bin/plugin_reload.lua fmadio_pcap2json_basic_20250417_1442.tcz )
Fri Aug 15 17:48:08 2025 Plugin Load
Loading Plugin [fmadio_pcap2json_basic_20250417_1442.tcz]
MD5: bb757d2e1729a78c2bf948bebd2b12ff fmadio_pcap2json_basic_20250417_1442.tcz
reloading pcap2json [basic]
Copying new firmware [fmadio_pcap2json_basic_20250417_1442.tcz] -> /mnt/system/tce14//optional/fmadio_pcap2json_basic.tcz
Cmd [sudo cp -v fmadio_pcap2json_basic_20250417_1442.tcz /mnt/system/tce14//optional/fmadio_pcap2json_basic.tcz]
'fmadio_pcap2json_basic_20250417_1442.tcz' -> '/mnt/system/tce14//optional/fmadio_pcap2json_basic.tcz'
Inserting into boot list
Cmd [echo "fmadio_pcap2json_basic.tcz" >> /mnt/system/tce14//onboot.lst]
Cmd [cp /mnt/system/tce14//onboot.lst /mnt/system/tce14//onboot.lst.bak]
Cmd [cat /mnt/system/tce14//onboot.lst.bak | grep -v analytics > /mnt/system/tce14//onboot.lst]
Killing programs
Cmd [sudo /usr/local/bin/umount /tmp/tcloop/fmadio_pcap2json_basic]
umount: /tmp/tcloop/fmadio_pcap2json_basic: no mount point specified.
Cmd [sudo mkdir -p /tmp/tcloop/fmadio_pcap2json_basic]
Cmd [sudo /usr/local/bin/mount /mnt/system/tce14//optional/fmadio_pcap2json_basic.tcz /tmp/tcloop/fmadio_pcap2json_basic -t squashfs -o loop,ro]
Cmd [yes | sudo cp -ais /tmp/tcloop/fmadio_pcap2json_basic/* / 2>/dev/null ]
-----------------------------------------------
Updated:
-> basic
-> 773
-> Thu Apr 17 14:42:56 2025
-----------------------------------------------
*****************************************************************
*** A System reboot is REQUIRED. Please restart (sudo reboot) ***
*****************************************************************
done 0.105679Sec 0.001761Min
Step 3) Reboot the system (optional)
This is to ensure the plugin gets loaded correctly on the next reboot.
Step 4) Config file
The default config file is shown below. This is a low performance configuration setting, good for 20Gbps of sustained traffic.
local Config =
{
["General"] =
{
IsMultiFE = false,
--StartDelay = 4e9,
WatchdogTimeout = 5*60,
SingleStack = "analytics",
}
,
["TCPEngine"] =
{
Enable = false,
Debug = true,
}
,
["pcap2json"] =
{
"-v",
"--flow-samplerate 1e9 ",
"--flow-index-depth 8",
"--flow-max 4e6",
}
,
-- CH binary
["backend"] =
{
"--index-name pcapflow.raw",
"--pipe-window 100e9",
"--flow-max 4e6",
"--output-debug",
"--output-buffercnt 32",
"--output-buffersize "..(128*1024*1024),
--"--output-null",
"--output-ch-bin",
"--ch-host 192.168.1.2:9000",
"--username default",
'--password "fmad-secret"',
"--index-disable",
}
,
["RTT"] =
{
["Enable"] = false,
["Period"] = 120e9,
["Offset"] = 120e9,
["Duration"] = 60e9,
}
,
["stream_cat"] =
{
--["BPF"] = "net 192.168.1.0/24",
--["FollowNow"] = 1e6,
}
}
return ConfigCreate a file in
/opt/fmadio/etc/pcap2json.luaPlace the above contents into the file
Step 5) Customize Config
The above config will write to the Clickhouse cluster at 192.168.1.2 on port 9000. Please configure the below line to the appropriate address / login / password of the clickhouse cluster.
"--ch-host 192.168.1.2:9000",
"--username default",
'--password "fmad-secret"',Save the file
Step 6) Sanity check the syntax
To verify the syntax of the file is correct run the following
fmadiolua /opt/fmadio/etc/pcap2json.luaExpected output is shown below
fmadio@fmadio20v4-733-pro:~$ fmadiolua /opt/fmadio/etc/pcap2json.lua
fmad fmadlua Aug 15 2025 (fmadiolua /opt/fmadio/etc/pcap2json.lua )
done 0.000051Sec 0.000001Min
fmadio@fmadio20v4-733-pro:~$If any syntax errors are shown, please got back to 5) and fix
Step 7) Run an offline test
Before enabling 24/7 decode operations. Run an offline test. This makes debugging the config significantly easier.
List all the captures on the system with
fmadiocli show capture listExample output
fmadio@fmadio20v4-733-pro:~$ fmadiocli show capture list
[Fri Aug 15 18:24:30 2025] CmdLine [show]
[Fri Aug 15 18:24:30 2025] CmdLine [capture]
[Fri Aug 15 18:24:30 2025] CmdLine [list]
[Fri Aug 15 18:24:30 2025] Cmd [show capture list ]
[Fri Aug 15 18:24:30 2025] Showing captures
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1752 ] 3,690,725,376 B (Fri . 18:24:30 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1746 ] 150,994,944 B (Fri . 17:48:10 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1730 ] 2,044,723,200 B (Fri . 17:34:43 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1724 ] 3,248,226,304 B (Fri . 17:25:59 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1712 ] 51,637,387,264 B (Fri . 17:19:56 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1642 ] 1,130,102,784 B (Fri . 16:56:34 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1605 ] 190,578,688 B (Fri . 16:08:01 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1553 ] 49,683,890,176 B (Fri . 16:01:28 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1543 ] 7,819,231,232 B (Fri . 15:49:03 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_1531 ] 67,895,296 B (Fri . 15:32:01 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250815_0000 ] 708,388,847,616 B (Fri . 15:27:06 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250814_0000 ] 1,315,479,748,608 B (Fri . 00:00:00 . 15-08-2025)
[Fri Aug 15 18:24:31 2025] [sg2-wan0_20250813_0000 ] 1,697,223,278,592 B (Wed . 23:59:59 . 13-08-2025)
fmadio@fmadio20v4-733-pro:~$In this case we will choose a fairly small capture
sg2-wan0_20250815_1553Then run the following (replacing the capture name with your own data)
sudo /opt/fmadio/analytics/pcap2json_realtime.lua --offline sg2-wan0_20250815_1553Example output is shown below
Step 8) Enable 24/7
Finally enable 24/7 processing of the flow data. Go to the CONFIG tab on the GUI and enter
pcap2json_realtimeAs shown below
.png?sv=2022-11-02&spr=https&st=2025-09-11T00%3A23%3A44Z&se=2025-09-11T00%3A41%3A44Z&sr=c&sp=r&sig=4o82AMN94nzWJffl70cRMRDBSpIjNQa7%2BYhIlodJOdE%3D)
All captured data will now be processed by Market Insight and pushed into the database.