Telemetry Syslog
FMADIO Syslog provides rich telemetry stream for monitoring the health of the system.
Telemetry is provided in JSON format for easy ingestion in monitoring and observability infrastructure. Each syslog moniting log event is described below, these are categorized into subsystems as follows
Generic SYSLOG events
FMADIO Temperature (Physical Thermal monitoring)
FMADIO Time (NTP, PTPv2, PPS)
FMADIO Other (uncategorized sensors)
FMADIO Power (status of power supply and consumption)
FMADIO Capture (monitoring state of the Packet Capture )
FMADIO IO (metrics related to Disk IO)
FMADIO Link (capture and management link status)
FMADIO Fan (physical status of FAN and cooling)
FMADIO Disk (status of both capture and OS disks)
FMADIO Cat (status of the stream_cat process used for extracting data off the disks)
FMADIO Push PCAP (status of the PCAP Push process)
FMADIO Push PCAP Split (event indicating each PCAP split completion)
FMADIO Alert (Custom onbox system alerts)
FMADIO Telemetry Service
FMADIO Telemetry service is included without charge on all support contracts. It provides a dashboard for each system, example screen shot shown below.
JSON Header
All FMADIO JSON messages have the following header fields
{
"module": "system",
"subsystem": "alert",
"timestamp": 1686730481,
"ver": "8779",
.
.
}
module
This provides a high level granularity on what the event/status is for, these map the sections below.
subsystem
This provides more granular view on what kind of event/status this is for
timestamp
This is the epoch time in seconds, where the FMADIO Monitoring system generated the event. Not the rsyslog time, although under normal situations they should be the same.
ver
This is the Firmware Version, it allows easy backwards compatibility as the system updates and improves as data is ingested into the monitoring system.
Generic SYSLOG
This includes generic syslog messages in free form plain text events. Example shown below
2023.06.14-15:48:02.323374 (+08:00) | fmadio100v2-228U | authpriv.info | sudo | pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
2023.06.14-15:48:02.500231 (+08:00) | fmadio100v2-228U | authpriv.info | sudo | pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
2023.06.14-15:48:02.541424 (+08:00) | fmadio100v2-228U | authpriv.info | sudo | pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
2023.06.14-15:48:03.359497 (+08:00) | fmadio100v2-228U | authpriv.info | sudo | pam_unix(sudo:session): session closed for user root
2023.06.14-15:48:03.538123 (+08:00) | fmadio100v2-228U | authpriv.info | sudo | pam_unix(sudo:session): session closed for user root
2023.06.14-15:48:03.566477 (+08:00) | fmadio100v2-228U | authpriv.info | sudo | pam_unix(sudo:session): session closed for user root
FMADIO Temperature
This provides thermal monitoring of the system to ensure its running within operating ranges.
2023.06.14-15:51:48.439475 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"temperature","timestamp":1686729108,"ver":"8779","Temperature_CPU0":39.00,"Temperature_CPU1":44.00,"Temperature_PCH":41.00,"Temperature_SYS":37.00,"Temperature_PER":22.00,"Temperature_NIC":48.00,"Temperature_AirIn":22.00,"Temperature_AirOut":0.00,"Temperature_Transceiver0":39.00,"Temperature_Transceiver1":0.00}
Example JSON Pretty
{
"module": "system",
"subsystem": "temperature",
"timestamp": 1686729129,
"ver": "8779",
"Temperature_CPU0": 39,
"Temperature_CPU1": 44,
"Temperature_PCH": 41,
"Temperature_SYS": 37,
"Temperature_PER": 22,
"Temperature_NIC": 48,
"Temperature_AirIn": 22,
"Temperature_AirOut": 0,
"Temperature_Transceiver0": 39,
"Temperature_Transceiver1": 0
}
FMADIO Time
Provides monitoring of time synchronization
2023.06.14-15:53:23.496058 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"ptp" ,"timestamp":1686729203,"ver":"8779","TimeFPGA":1686729203472942027,"TimeSYS":1686729203473304064,"GMOffset": 0.00,"GMSync":false,"GMMaster":"","SysOffset": 0.00,"SysSync":false,"iSysOffset": 0.00,"iSysSync":true,"NTPOffset":-445000.00,"NTPSync":true,"NTPMaster":"*192.168.2.5","GMUpTime":0,"SysUptime":0,"iSysUptime":156545,"PPSUptime":155994,"NTPUptime":156094,"PPSPeriod":3.103035477,"PPSOffset":-1,"PPSdPhase":-4,"PPSdZero":-1,"PPSCnt":157188,"Clk156Period":0.000000,"Clk156Offset":0,"Clk250Period":0.000000,"Clk250Offset":0,"Clk322Period":0.000000,"Clk322Offset":0}
Pretty JSON
{
"module": "system",
"subsystem": "ptp",
"timestamp": 1686729331,
"ver": "8779",
"TimeFPGA": 1686729331198241000,
"TimeSYS": 1686729331198588000,
"GMOffset": 0,
"GMSync": false,
"GMMaster": "",
"SysOffset": 0,
"SysSync": false,
"iSysOffset": 0,
"iSysSync": true,
"NTPOffset": -445000,
"NTPSync": true,
"NTPMaster": "*192.168.2.5",
"GMUpTime": 0,
"SysUptime": 0,
"iSysUptime": 156675,
"PPSUptime": 156124,
"NTPUptime": 156224,
"PPSPeriod": 3.103035483,
"PPSOffset": 0,
"PPSdPhase": -4,
"PPSdZero": 0,
"PPSCnt": 157316,
"Clk156Period": 0,
"Clk156Offset": 0,
"Clk250Period": 0,
"Clk250Offset": 0,
"Clk322Period": 0,
"Clk322Offset": 0
}
FMADIO Other
Miscellaneous other fields
2023.06.14-15:54:27.535663 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"other" ,"timestamp":1686729267,"ver":"8779","UptimeHour":43.52,"MemFree":366353580032,"MemErrorECC":0,"WritebackB":0,"WritebackPct":0.000000,"WritebackDropTotalG":0.00,"WritebackDropG":0.00,"CacheSize":30726047137792,"StoreSize":30725994708992,"CPULoad": 0.39,"SerialNo":"undef-undef-e0d55e5d2150","PortConfig":"2x100G","Version":"fmadio100v2:8779 pcap2json:715"}
Pretty JSON
{
"module": "system",
"subsystem": "other",
"timestamp": 1686729394,
"ver": "8779",
"UptimeHour": 43.55,
"MemFree": 366355021824,
"MemErrorECC": 0,
"WritebackB": 0,
"WritebackPct": 0,
"WritebackDropTotalG": 0,
"WritebackDropG": 0,
"CacheSize": 30726047137792,
"StoreSize": 30725994708992,
"CPULoad": 0.39,
"SerialNo": "undef-undef-e0d55e5d2150",
"PortConfig": "2x100G",
"Version": "fmadio100v2:8779 pcap2json:715"
}
FMADIO Power
Provides power status and utilization information
2023.06.14-15:53:23.488485 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"power" ,"timestamp":1686729203,"ver":"8779","PSU0_Status":false,"PSU1_Status":true,"PSU_PowerWatt":270}
Pretty JSON
{
"module": "system",
"subsystem": "power",
"timestamp": 1686729447,
"ver": "8779",
"PSU0_Status": false,
"PSU1_Status": true,
"PSU_PowerWatt": 270
}
FMADIO Capture
Provides status information around the capture process
2023.06.14-15:56:34.457781 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"capture" ,"timestamp":1686729394,"ver":"8779","CaptureEnb":false,"CapturePkt": 2795520,"CaptureByte": 190095360,"CaptureDrop": 952825443,"CaptureFCSError": 0,"CaptureRateGbps": 0.000000,"CaptureRateMpps": 0.000000,"CaptureName":"","CapturePort0_Byte":0,"CapturePort0_Pkt":0,"CapturePort1_Byte":68000000000,"CapturePort1_Pkt":1000000000,"CapturePort2_Byte":0,"CapturePort2_Pkt":0,"CapturePort3_Byte":0,"CapturePort3_Pkt":0,"CapturePort4_Byte":0,"CapturePort4_Pkt":0,"CapturePort5_Byte":0,"CapturePort5_Pkt":0,"CapturePort6_Byte":0,"CapturePort6_Pkt":0,"CapturePort7_Byte":0,"CapturePort7_Pkt":0}
Pretty JSON
{
"module": "system",
"subsystem": "capture",
"timestamp": 1686673255,
"ver": "8779",
"CaptureEnb": false,
"CapturePkt": 2795520,
"CaptureByte": 190095360,
"CaptureDrop": 952825443,
"CaptureFCSError": 0,
"CaptureRateGbps": 0,
"CaptureRateMpps": 0,
"CaptureName": "",
"CapturePort0_Byte": 0,
"CapturePort0_Pkt": 0,
"CapturePort1_Byte": 68000000000,
"CapturePort1_Pkt": 1000000000,
"CapturePort2_Byte": 0,
"CapturePort2_Pkt": 0,
"CapturePort3_Byte": 0,
"CapturePort3_Pkt": 0,
"CapturePort4_Byte": 0,
"CapturePort4_Pkt": 0,
"CapturePort5_Byte": 0,
"CapturePort5_Pkt": 0,
"CapturePort6_Byte": 0,
"CapturePort6_Pkt": 0,
"CapturePort7_Byte": 0,
"CapturePort7_Pkt": 0
}
FMADIO IO
Provides status and performance of IO disk/network related systems
2023.06.14-15:54:27.528914 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"io" ,"timestamp":1686729267,"ver":"8779","DiskRdGbps": 0.00,"DiskWrGbps": 0.00}
Pretty JSON
{
"module": "system",
"subsystem": "io",
"timestamp": 1686729638,
"ver": "8779",
"DiskRdGbps": 0,
"DiskWrGbps": 0
}
FMADIO Link
Status information around port link status
{"module":"system","subsystem":"link" ,"timestamp":1686729256,"ver":"8779","cap0_link":true,"cap1_link":true,"cap2_link":false,"cap3_link":false,"cap4_link":false,"cap5_link":false,"cap6_link":false,"cap7_link":false,"man0_link":true,"man10_link":true}
Pretty JSON
{
"module": "system",
"subsystem": "link",
"timestamp": 1686729744,
"ver": "8779",
"cap0_link": true,
"cap1_link": true,
"cap2_link": false,
"cap3_link": false,
"cap4_link": false,
"cap5_link": false,
"cap6_link": false,
"cap7_link": false,
"man0_link": true,
"man10_link": true
}
FMADIO Fan
Status information around fans and cooling
2023.06.14-15:53:02.396976 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"fan" ,"timestamp":1686729182,"ver":"8779","Fan_SYS0":21450,"Fan_SYS1":21450,"Fan_SYS2":21300,"Fan_SYS3":21450,"Fan_SYS4":21450,"Fan_SYS5":21450,"Fan_SYS6":21450,"Fan_SYS7":21450}
Pretty JSON
{
"module": "system",
"subsystem": "fan",
"timestamp": 1686729819,
"ver": "8779",
"Fan_SYS0": 21600,
"Fan_SYS1": 21450,
"Fan_SYS2": 21450,
"Fan_SYS3": 21450,
"Fan_SYS4": 21450,
"Fan_SYS5": 21450,
"Fan_SYS6": 21750,
"Fan_SYS7": 21600
}
FMADIO Disk
Status around the capture and os disk
2023.06.14-15:59:56.186455 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"disk" ,"timestamp":1686729596,"ver":"8779","FreeGB_System":6.078,"FreeGB_Store0":5528.059,"FreeGB_Store1":0.000,"FreeGB_Remote0":83294.126,"FreeGB_Remote1":83294.126,"DiskPresent_os0":true,"DiskTemperature_os0":37,"DiskSMART_os0":0,"DiskPresent_ssd0":true,"DiskTemperature_ssd0":31,"DiskSMART_ssd0":0,"DiskPresent_ssd1":true,"DiskTemperature_ssd1":29,"DiskSMART_ssd1":0,"DiskPresent_ssd2":true,"DiskTemperature_ssd2":30,"DiskSMART_ssd2":0,"DiskPresent_ssd3":true,"DiskTemperature_ssd3":29,"DiskSMART_ssd3":0,"DiskPresent_ssd4":true,"DiskTemperature_ssd4":30,"DiskSMART_ssd4":0,"DiskPresent_ssd5":true,"DiskTemperature_ssd5":31,"DiskSMART_ssd5":0,"DiskPresent_ssd6":true,"DiskTemperature_ssd6":31,"DiskSMART_ssd6":0,"DiskPresent_ssd7":true,"DiskTemperature_ssd7":32,"DiskSMART_ssd7":0,"DiskPresent_par0":true,"DiskTemperature_par0":30,"DiskSMART_par0":0}
Pretty JSON
{
"module": "system",
"subsystem": "disk",
"timestamp": 1686729925,
"ver": "8779",
"FreeGB_System": 6.078,
"FreeGB_Store0": 5528.059,
"FreeGB_Store1": 0,
"FreeGB_Remote0": 83294.126,
"FreeGB_Remote1": 83294.126,
"DiskPresent_os0": true,
"DiskTemperature_os0": 37,
"DiskSMART_os0": 0,
"DiskPresent_ssd0": true,
"DiskTemperature_ssd0": 31,
"DiskSMART_ssd0": 0,
"DiskPresent_ssd1": true,
"DiskTemperature_ssd1": 29,
"DiskSMART_ssd1": 0,
"DiskPresent_ssd2": true,
"DiskTemperature_ssd2": 30,
"DiskSMART_ssd2": 0,
"DiskPresent_ssd3": true,
"DiskTemperature_ssd3": 29,
"DiskSMART_ssd3": 0,
"DiskPresent_ssd4": true,
"DiskTemperature_ssd4": 30,
"DiskSMART_ssd4": 0,
"DiskPresent_ssd5": true,
"DiskTemperature_ssd5": 31,
"DiskSMART_ssd5": 0,
"DiskPresent_ssd6": true,
"DiskTemperature_ssd6": 31,
"DiskSMART_ssd6": 0,
"DiskPresent_ssd7": true,
"DiskTemperature_ssd7": 32,
"DiskSMART_ssd7": 0,
"DiskPresent_par0": true,
"DiskTemperature_par0": 30,
"DiskSMART_par0": 0
}
FMADIO Cat
stream_cat is the core FMADIO utility for extracting packets off the storage system. Its used heavily for download, realtime processing and troubleshooting. This provides statistics and state of the currently running stream_cat instances.
Up to 8 simultaniousle stream_cat instances can be run at the same time. cat_*_ is per instance.
2023.06.14-16:06:28.566556 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"cat" ,"timestamp":1686729988,"ver":"8779","cat_0_Enable":false,"cat_0_Mode":"","cat_0_CPUMain":0,"cat_0_TSPCAP":0,"cat_0_ReadPkt":0,"cat_0_ReadByte":0,"cat_0_ReadTotalPkt":0,"cat_0_ReadTotalByte":0,"cat_0_ReadGbps":0.000000,"cat_0_ReadMpps":0.000000,"cat_0_WritePkt":0,"cat_0_WriteByte":0,"cat_0_WriteTotalPkt":0,"cat_0_WriteTotalByte":0,"cat_0_WriteGbps":0.000000,"cat_0_WriteMpps":0.000000,"cat_0_PendingByte":0,"cat_0_PktDiscard":0,"cat_0_PktDiscardTotal":0,"cat_0_PktSlice":0,"cat_0_IOPriority":0,"cat_0_ChunkID":0,"cat_0_CmdLine":"","cat_0_StreamName":"","cat_0_FilterBPF":"","cat_0_CPUIdle":0.0000,"cat_0_CPUFetch":0.0000,"cat_0_CPUProcess":0.0000,"cat_0_CPUSend":0.0000,"cat_1_Enable":false,"cat_1_Mode":"","cat_1_CPUMain":0,"cat_1_TSPCAP":0,"cat_1_ReadPkt":0,"cat_1_ReadByte":0,"cat_1_ReadTotalPkt":0,"cat_1_ReadTotalByte":0,"cat_1_ReadGbps":0.000000,"cat_1_ReadMpps":0.000000,"cat_1_WritePkt":0,"cat_1_WriteByte":0,"cat_1_WriteTotalPkt":0,"cat_1_WriteTotalByte":0,"cat_1_WriteGbps":0.000000,"cat_1_WriteMpps":0.000000,"cat_1_PendingByte":0,"cat_1_PktDiscard":0,"cat_1_PktDiscardTotal":0,"cat_1_PktSlice":0,"cat_1_IOPriority":0,"cat_1_ChunkID":0,"cat_1_CmdLine":"","cat_1_StreamName":"","cat_1_FilterBPF":"","cat_1_CPUIdle":0.0000,"cat_1_CPUFetch":0.0000,"cat_1_CPUProcess":0.0000,"cat_1_CPUSend":0.0000,"cat_2_Enable":false,"cat_2_Mode":"","cat_2_CPUMain":0,"cat_2_TSPCAP":0,"cat_2_ReadPkt":0,"cat_2_ReadByte":0,"cat_2_ReadTotalPkt":0,"cat_2_ReadTotalByte":0,"cat_2_ReadGbps":0.000000,"cat_2_ReadMpps":0.000000,"cat_2_WritePkt":0,"cat_2_WriteByte":0,"cat_2_WriteTotalPkt":0,"cat_2_WriteTotalByte":0,"cat_2_WriteGbps":0.000000,"cat_2_WriteMpps":0.000000,"cat_2_PendingByte":0,"cat_2_PktDiscard":0,"cat_2_PktDiscardTotal":0,"cat_2_PktSlice":0,"cat_2_IOPriority":0,"cat_2_ChunkID":0,"cat_2_CmdLine":"","cat_2_StreamName":"","cat_2_FilterBPF":"","cat_2_CPUIdle":0.0000,"cat_2_CPUFetch":0.0000,"cat_2_CPUProcess":0.0000,"cat_2_CPUSend":0.0000,"cat_3_Enable":false,"cat_3_Mode":"","cat_3_CPUMain":0,"cat_3_TSPCAP":0,"cat_3_ReadPkt":0,"cat_3_ReadByte":0,"cat_3_ReadTotalPkt":0,"cat_3_ReadTotalByte":0,"cat_3_ReadGbps":0.000000,"cat_3_ReadMpps":0.000000,"cat_3_WritePkt":0,"cat_3_WriteByte":0,"cat_3_WriteTotalPkt":0,"cat_3_WriteTotalByte":0,"cat_3_WriteGbps":0.000000,"cat_3_WriteMpps":0.000000,"cat_3_PendingByte":0,"cat_3_PktDiscard":0,"cat_3_PktDiscardTotal":0,"cat_3_PktSlice":0,"cat_3_IOPriority":0,"cat_3_ChunkID":0,"cat_3_CmdLine":"","cat_3_StreamName":"","cat_3_FilterBPF":"","cat_3_CPUIdle":0.0000,"cat_3_CPUFetch":0.0000,"cat_3_CPUProcess":0.0000,"cat_3_CPUSend":0.0000,"cat_4_Enable":false,"cat_4_Mode":"","cat_4_CPUMain":0,"cat_4_TSPCAP":0,"cat_4_ReadPkt":0,"cat_4_ReadByte":0,"cat_4_ReadTotalPkt":0,"cat_4_ReadTotalByte":0,"cat_4_ReadGbps":0.000000,"cat_4_ReadMpps":0.000000,"cat_4_WritePkt":0,"cat_4_WriteByte":0,"cat_4_WriteTotalPkt":0,"cat_4_WriteTotalByte":0,"cat_4_WriteGbps":0.000000,"cat_4_WriteMpps":0.000000,"cat_4_PendingByte":0,"cat_4_PktDiscard":0,"cat_4_PktDiscardTotal":0,"cat_4_PktSlice":0,"cat_4_IOPriority":0,"cat_4_ChunkID":0,"cat_4_CmdLine":"","cat_4_StreamName":"","cat_4_FilterBPF":"","cat_4_CPUIdle":0.0000,"cat_4_CPUFetch":0.0000,"cat_4_CPUProcess":0.0000,"cat_4_CPUSend":0.0000,"cat_5_Enable":false,"cat_5_Mode":"","cat_5_CPUMain":0,"cat_5_TSPCAP":0,"cat_5_ReadPkt":0,"cat_5_ReadByte":0,"cat_5_ReadTotalPkt":0,"cat_5_ReadTotalByte":0,"cat_5_ReadGbps":0.000000,"cat_5_ReadMpps":0.000000,"cat_5_WritePkt":0,"cat_5_WriteByte":0,"cat_5_WriteTotalPkt":0,"cat_5_WriteTotalByte":0,"cat_5_WriteGbps":0.000000,"cat_5_WriteMpps":0.000000,"cat_5_PendingByte":0,"cat_5_PktDiscard":0,"cat_5_PktDiscardTotal":0,"cat_5_PktSlice":0,"cat_5_IOPriority":0,"cat_5_ChunkID":0,"cat_5_CmdLine":"","cat_5_StreamName":"","cat_5_FilterBPF":"","cat_5_CPUIdle":0.0000,"cat_5_CPUFetch":0.0000,"cat_5_CPUProcess":0.0000,"cat_5_CPUSend":0.0000,"cat_6_Enable":false,"cat_6_Mode":"","cat_6_CPUMain":0,"cat_6_TSPCAP":0,"cat_6_ReadPkt":0,"cat_6_ReadByte":0,"cat_6_ReadTotalPkt":0,"cat_6_ReadTotalByte":0,"cat_6_ReadGbps":0.000000,"cat_6_ReadMpps":0.000000,"cat_6_WritePkt":0,"cat_6_WriteByte":0,"cat_6_WriteTotalPkt":0,"cat_6_WriteTotalByte":0,"cat_6_WriteGbps":0.000000,"cat_6_WriteMpps":0.000000,"cat_6_PendingByte":0,"cat_6_PktDiscard":0,"cat_6_PktDiscardTotal":0,"cat_6_PktSlice":0,"cat_6_IOPriority":0,"cat_6_ChunkID":0,"cat_6_CmdLine":"","cat_6_StreamName":"","cat_6_FilterBPF":"","cat_6_CPUIdle":0.0000,"cat_6_CPUFetch":0.0000,"cat_6_CPUProcess":0.0000,"cat_6_CPUSend":0.0000,"cat_7_Enable":false,"cat_7_Mode":"","cat_7_CPUMain":0,"cat_7_TSPCAP":0,"cat_7_ReadPkt":0,"cat_7_ReadByte":0,"cat_7_ReadTotalPkt":0,"cat_7_ReadTotalByte":0,"cat_7_ReadGbps":0.000000,"cat_7_ReadMpps":0.000000,"cat_7_WritePkt":0,"cat_7_WriteByte":0,"cat_7_WriteTotalPkt":0,"cat_7_WriteTotalByte":0,"cat_7_WriteGbps":0.000000,"cat_7_WriteMpps":0.000000,"cat_7_PendingByte":0,"cat_7_PktDiscard":0,"cat_7_PktDiscardTotal":0,"cat_7_PktSlice":0,"cat_7_IOPriority":0,"cat_7_ChunkID":0,"cat_7_CmdLine":"","cat_7_StreamName":"","cat_7_FilterBPF":"","cat_7_CPUIdle":0.0000,"cat_7_CPUFetch":0.0000,"cat_7_CPUProcess":0.0000,"cat_7_CPUSend":0.0000,"cat_EnableCnt":0,"cat_ReadPkt":0,"cat_ReadByte":0,"cat_ReadTotalPkt":0,"cat_ReadTotalByte":0,"cat_ReadGbps":0.000000,"cat_ReadMpps":0.000000,"cat_WritePkt":0,"cat_WriteByte":0,"cat_WriteTotalPkt":0,"cat_WriteTotalByte":0,"cat_WriteGbps":0.000000,"cat_WriteMpps":0.000000}
Pretty JSON
{
"module": "system",
"subsystem": "cat",
"timestamp": 1686730073,
"ver": "8779",
"cat_0_Enable": false,
"cat_0_Mode": "",
"cat_0_CPUMain": 0,
"cat_0_TSPCAP": 0,
"cat_0_ReadPkt": 0,
"cat_0_ReadByte": 0,
"cat_0_ReadTotalPkt": 0,
"cat_0_ReadTotalByte": 0,
"cat_0_ReadGbps": 0,
"cat_0_ReadMpps": 0,
"cat_0_WritePkt": 0,
"cat_0_WriteByte": 0,
"cat_0_WriteTotalPkt": 0,
"cat_0_WriteTotalByte": 0,
"cat_0_WriteGbps": 0,
"cat_0_WriteMpps": 0,
"cat_0_PendingByte": 0,
"cat_0_PktDiscard": 0,
"cat_0_PktDiscardTotal": 0,
"cat_0_PktSlice": 0,
"cat_0_IOPriority": 0,
"cat_0_ChunkID": 0,
"cat_0_CmdLine": "",
"cat_0_StreamName": "",
"cat_0_FilterBPF": "",
"cat_0_CPUIdle": 0,
"cat_0_CPUFetch": 0,
"cat_0_CPUProcess": 0,
"cat_0_CPUSend": 0,
"cat_1_Enable": false,
"cat_1_Mode": "",
"cat_1_CPUMain": 0,
"cat_1_TSPCAP": 0,
"cat_1_ReadPkt": 0,
"cat_1_ReadByte": 0,
"cat_1_ReadTotalPkt": 0,
"cat_1_ReadTotalByte": 0,
"cat_1_ReadGbps": 0,
"cat_1_ReadMpps": 0,
"cat_1_WritePkt": 0,
"cat_1_WriteByte": 0,
"cat_1_WriteTotalPkt": 0,
"cat_1_WriteTotalByte": 0,
"cat_1_WriteGbps": 0,
"cat_1_WriteMpps": 0,
"cat_1_PendingByte": 0,
"cat_1_PktDiscard": 0,
"cat_1_PktDiscardTotal": 0,
"cat_1_PktSlice": 0,
"cat_1_IOPriority": 0,
"cat_1_ChunkID": 0,
"cat_1_CmdLine": "",
"cat_1_StreamName": "",
"cat_1_FilterBPF": "",
"cat_1_CPUIdle": 0,
"cat_1_CPUFetch": 0,
"cat_1_CPUProcess": 0,
"cat_1_CPUSend": 0,
"cat_2_Enable": false,
"cat_2_Mode": "",
"cat_2_CPUMain": 0,
"cat_2_TSPCAP": 0,
"cat_2_ReadPkt": 0,
"cat_2_ReadByte": 0,
"cat_2_ReadTotalPkt": 0,
"cat_2_ReadTotalByte": 0,
"cat_2_ReadGbps": 0,
"cat_2_ReadMpps": 0,
"cat_2_WritePkt": 0,
"cat_2_WriteByte": 0,
"cat_2_WriteTotalPkt": 0,
"cat_2_WriteTotalByte": 0,
"cat_2_WriteGbps": 0,
"cat_2_WriteMpps": 0,
"cat_2_PendingByte": 0,
"cat_2_PktDiscard": 0,
"cat_2_PktDiscardTotal": 0,
"cat_2_PktSlice": 0,
"cat_2_IOPriority": 0,
"cat_2_ChunkID": 0,
"cat_2_CmdLine": "",
"cat_2_StreamName": "",
"cat_2_FilterBPF": "",
"cat_2_CPUIdle": 0,
"cat_2_CPUFetch": 0,
"cat_2_CPUProcess": 0,
"cat_2_CPUSend": 0,
"cat_3_Enable": false,
"cat_3_Mode": "",
"cat_3_CPUMain": 0,
"cat_3_TSPCAP": 0,
"cat_3_ReadPkt": 0,
"cat_3_ReadByte": 0,
"cat_3_ReadTotalPkt": 0,
"cat_3_ReadTotalByte": 0,
"cat_3_ReadGbps": 0,
"cat_3_ReadMpps": 0,
"cat_3_WritePkt": 0,
"cat_3_WriteByte": 0,
"cat_3_WriteTotalPkt": 0,
"cat_3_WriteTotalByte": 0,
"cat_3_WriteGbps": 0,
"cat_3_WriteMpps": 0,
"cat_3_PendingByte": 0,
"cat_3_PktDiscard": 0,
"cat_3_PktDiscardTotal": 0,
"cat_3_PktSlice": 0,
"cat_3_IOPriority": 0,
"cat_3_ChunkID": 0,
"cat_3_CmdLine": "",
"cat_3_StreamName": "",
"cat_3_FilterBPF": "",
"cat_3_CPUIdle": 0,
"cat_3_CPUFetch": 0,
"cat_3_CPUProcess": 0,
"cat_3_CPUSend": 0,
"cat_4_Enable": false,
"cat_4_Mode": "",
"cat_4_CPUMain": 0,
"cat_4_TSPCAP": 0,
"cat_4_ReadPkt": 0,
"cat_4_ReadByte": 0,
"cat_4_ReadTotalPkt": 0,
"cat_4_ReadTotalByte": 0,
"cat_4_ReadGbps": 0,
"cat_4_ReadMpps": 0,
"cat_4_WritePkt": 0,
"cat_4_WriteByte": 0,
"cat_4_WriteTotalPkt": 0,
"cat_4_WriteTotalByte": 0,
"cat_4_WriteGbps": 0,
"cat_4_WriteMpps": 0,
"cat_4_PendingByte": 0,
"cat_4_PktDiscard": 0,
"cat_4_PktDiscardTotal": 0,
"cat_4_PktSlice": 0,
"cat_4_IOPriority": 0,
"cat_4_ChunkID": 0,
"cat_4_CmdLine": "",
"cat_4_StreamName": "",
"cat_4_FilterBPF": "",
"cat_4_CPUIdle": 0,
"cat_4_CPUFetch": 0,
"cat_4_CPUProcess": 0,
"cat_4_CPUSend": 0,
"cat_5_Enable": false,
"cat_5_Mode": "",
"cat_5_CPUMain": 0,
"cat_5_TSPCAP": 0,
"cat_5_ReadPkt": 0,
"cat_5_ReadByte": 0,
"cat_5_ReadTotalPkt": 0,
"cat_5_ReadTotalByte": 0,
"cat_5_ReadGbps": 0,
"cat_5_ReadMpps": 0,
"cat_5_WritePkt": 0,
"cat_5_WriteByte": 0,
"cat_5_WriteTotalPkt": 0,
"cat_5_WriteTotalByte": 0,
"cat_5_WriteGbps": 0,
"cat_5_WriteMpps": 0,
"cat_5_PendingByte": 0,
"cat_5_PktDiscard": 0,
"cat_5_PktDiscardTotal": 0,
"cat_5_PktSlice": 0,
"cat_5_IOPriority": 0,
"cat_5_ChunkID": 0,
"cat_5_CmdLine": "",
"cat_5_StreamName": "",
"cat_5_FilterBPF": "",
"cat_5_CPUIdle": 0,
"cat_5_CPUFetch": 0,
"cat_5_CPUProcess": 0,
"cat_5_CPUSend": 0,
"cat_6_Enable": false,
"cat_6_Mode": "",
"cat_6_CPUMain": 0,
"cat_6_TSPCAP": 0,
"cat_6_ReadPkt": 0,
"cat_6_ReadByte": 0,
"cat_6_ReadTotalPkt": 0,
"cat_6_ReadTotalByte": 0,
"cat_6_ReadGbps": 0,
"cat_6_ReadMpps": 0,
"cat_6_WritePkt": 0,
"cat_6_WriteByte": 0,
"cat_6_WriteTotalPkt": 0,
"cat_6_WriteTotalByte": 0,
"cat_6_WriteGbps": 0,
"cat_6_WriteMpps": 0,
"cat_6_PendingByte": 0,
"cat_6_PktDiscard": 0,
"cat_6_PktDiscardTotal": 0,
"cat_6_PktSlice": 0,
"cat_6_IOPriority": 0,
"cat_6_ChunkID": 0,
"cat_6_CmdLine": "",
"cat_6_StreamName": "",
"cat_6_FilterBPF": "",
"cat_6_CPUIdle": 0,
"cat_6_CPUFetch": 0,
"cat_6_CPUProcess": 0,
"cat_6_CPUSend": 0,
"cat_7_Enable": false,
"cat_7_Mode": "",
"cat_7_CPUMain": 0,
"cat_7_TSPCAP": 0,
"cat_7_ReadPkt": 0,
"cat_7_ReadByte": 0,
"cat_7_ReadTotalPkt": 0,
"cat_7_ReadTotalByte": 0,
"cat_7_ReadGbps": 0,
"cat_7_ReadMpps": 0,
"cat_7_WritePkt": 0,
"cat_7_WriteByte": 0,
"cat_7_WriteTotalPkt": 0,
"cat_7_WriteTotalByte": 0,
"cat_7_WriteGbps": 0,
"cat_7_WriteMpps": 0,
"cat_7_PendingByte": 0,
"cat_7_PktDiscard": 0,
"cat_7_PktDiscardTotal": 0,
"cat_7_PktSlice": 0,
"cat_7_IOPriority": 0,
"cat_7_ChunkID": 0,
"cat_7_CmdLine": "",
"cat_7_StreamName": "",
"cat_7_FilterBPF": "",
"cat_7_CPUIdle": 0,
"cat_7_CPUFetch": 0,
"cat_7_CPUProcess": 0,
"cat_7_CPUSend": 0,
"cat_EnableCnt": 0,
"cat_ReadPkt": 0,
"cat_ReadByte": 0,
"cat_ReadTotalPkt": 0,
"cat_ReadTotalByte": 0,
"cat_ReadGbps": 0,
"cat_ReadMpps": 0,
"cat_WritePkt": 0,
"cat_WriteByte": 0,
"cat_WriteTotalPkt": 0,
"cat_WriteTotalByte": 0,
"cat_WriteGbps": 0,
"cat_WriteMpps": 0
}
FMADIO Push PCAP Status
Provides status information of the currently active Push PCAP proceses
2023.06.14-16:11:44.342725 (+08:00) | fmadio20v2-149 | local7.info | fmadio | {"module":"push_pcap","subsystem":"status","timestamp":1686730304,"ver":"8769","Process":"voip-hk ","IsUp": true,"Splits":57285,"TotalByte": 63860619926,"TotalPkt": 228209873,"TransferMbps": 18.00,"PCAPTS":1686730299890543104,"FilterBPF":"vlan 1804","FilterFrame":"nil","Target":"/mnt/remote0/pcap/20230614/voip-hk-"}
Pretty JSON
{
"module": "push_pcap",
"subsystem": "status",
"timestamp": 1686730363,
"ver": "8769",
"Process": "voip-hk ",
"IsUp": true,
"Splits": 57343,
"TotalByte": 63955460513,
"TotalPkt": 228548682,
"TransferMbps": 12,
"PCAPTS": 1686730357291535600,
"FilterBPF": "vlan 1804",
"FilterFrame": "nil",
"Target": "/mnt/remote0/pcap/20230614/voip-hk-"
}
FMADIO Push PCAP File
Called after completing a PCAP file split event. Helpful to monitoring each and every PCAP split leaving the system
2023.06.14-16:14:43.131050 (+08:00) | fmadio20v2-149 | user.notice | fmadio | {"module":"push_pcap","subsystem":"split","timestamp":1686730483,"ver":"8769","filename":"/mnt/remote0/pcap/20230614/icmp-20230614_161436.pcap.zstd","Byte":896,"Pkt":8,"TimeWall":0.670407,"TimePCAP":1.005172,"PCATSStart":1686730477000000000,"PCAPTSEnd":1686730476874467328}
Pretty JSON
{
"module": "push_pcap",
"subsystem": "split",
"timestamp": 1686730521,
"ver": "8769",
"filename": "/mnt/remote0/pcap/20230614/icmp-20230614_161514.pcap.zstd",
"Byte": 896,
"Pkt": 8,
"TimeWall": 0.930522,
"TimePCAP": 0.988352,
"PCATSStart": 1686730515000000000,
"PCAPTSEnd": 1686730514909767700
}
FMADIO Alert
The system has capibility to generate system alerts based on configuration files. These alerts can be sent to SYSLOG for ingestion by a monitoring system
2023.06.14-16:17:59.032528 (+08:00) | fmadio100v2-228U | local7.info | fmadio | {"module":"system","subsystem":"alert","timestamp":1686730679,"ver":"8779","Subject":"CPU Temperature","Message":"CPU Temperature Over CPU0:39 CPU1:43 > 20"}
Pretty JSON
{
"module": "system",
"subsystem": "alert",
"timestamp": 1686730679,
"ver": "8779",
"Subject": "CPU Temperature",
"Message": "CPU Temperature Over CPU0:39 CPU1:43 > 20"
}
Telemetry Service Setup
Setting up automatic telemetry as follows
Step 1) Generate unique SSH key
FMADIO devices by default have a pre-installed ssh key. To correctly secure and uniquely identify the system generate your own SSH key as follows.
ssh-keygen -t rsa -b 3072
Using a password less key ensures the automatic setup requires no manual intervention.
Example output per below
fmadio@fmadio100p3-539:~$ ssh-keygen -t rsa -b 3072
Generating public/private rsa key pair.
Enter file in which to save the key (/home/fmadio/.ssh/id_rsa):
/home/fmadio/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/fmadio/.ssh/id_rsa
Your public key has been saved in /home/fmadio/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:G/***************************** fmadio@fmadio100p3-539
The key's randomart image is:
+---[RSA 3072]----+
| Eo.oo=Bo . |
| o .o.. o.. + |
| . ..+ +.o o o |
|. . =..+ . ..|
| . +S o o|
| . ..+ o . |
| ..+.oo. o . |
| o. o= = o . |
| oo..o . .oo |
+----[SHA256]-----+
fmadio@fmadio100p3-539:~$
Send the above public key (below) to support@fmad.io
/opt/fmadio/etc/fmadio_id_rsa.pub
Step 2) Copy to persistent Storage
The SSH public/private keys are on the volatile file system. Copy the keys to the persistent storage.
cp .ssh/id_rsa /opt/fmadio/etc/fmadio_id_rsa
cp .ssh/id_rsa.pub /opt/fmadio/etc/fmadio_id_rsa.pub
NOTE: the key is renamed with an fmadio_* prefix. The system copies the keys from this location and renames them in the .ssh/id_rsa .ssh/id_rsa.pub directory during the boot process.
Step 3) Copy the reference boot script
There is a reference boot script located in
/opt/fmadio/etc_ro/boot.lua.telemetry_mon2
Copy this to to the /opt/fmadio/etc/boot.lua file to automatically establish ssh tunnel to the telemetry service.
cp /opt/fmadio/etc_ro/boot.lua.telemetry_mon2 /opt/fmadio/etc/boot.lua
After copying replace the "username" to the username provided by fmadio support and save the file.
Step 4) Copy the reference rsyslog config
In addition to ssh tunnel setup, rsyslog configuration to forward syslog messages to the SSH tunnel.
Copy the reference config to /opt/fmadio/etc/ directory as follows
cp /opt/fmadio/etc_ro/syslogd.conf.tunnel /opt/fmadio/etc/syslogd.conf
No modifications are required.
Step 5) Reboot the system
Reboot the system to check all the above steps are executed correctly
Step 6) Validate
After rebooting log into the Grafana monitoring site with the assigned username and confirm data is being recevied.
Any problems please contact support@fmad.io
Telemetry via API
In addition to syslog, the systems telemetry information snapshot can be fetched using the JSON API.
Example command:
curl -s http://127.0.0.1/api/v1/system/status | jq
With the pretty formatted output as follows. This output will always match the above syslog information.
{
"timestamp": 1695785514,
"ver": "9120",
"temperature": {
"module": "system",
"subsystem": "temperature",
"timestamp": 1695785514,
"ver": "9120",
"Temperature_CPU0": 55,
"Temperature_CPU1": 70,
"Temperature_PCH": 46,
"Temperature_SYS": 42,
"Temperature_PER": 23,
"Temperature_NIC": 49,
"Temperature_AirIn": 23,
"Temperature_AirOut": 0,
"Temperature_Transceiver0": 42,
"Temperature_Transceiver1": 42
},
"fan": {
"module": "system",
"subsystem": "fan",
"timestamp": 1695785514,
"ver": "9120",
"Fan_SYS0": 21450,
"Fan_SYS1": 21450,
"Fan_SYS2": 21300,
"Fan_SYS3": 21450,
"Fan_SYS4": 21450,
"Fan_SYS5": 21450,
"Fan_SYS6": 21450,
"Fan_SYS7": 21600
},
"disk": {
"module": "system",
"subsystem": "disk",
"timestamp": 1695785514,
"ver": "9120",
"FreeGB_System": 8.977,
"FreeGB_Store0": 4720.779,
"FreeGB_Store1": 0,
"FreeGB_Remote0": 46349.53,
"FreeGB_Remote1": 46349.53,
"DiskPresent_os0": true,
"DiskTemperature_os0": 40,
"DiskSMART_os0": 0,
"DiskPresent_ssd0": true,
"DiskTemperature_ssd0": 36,
"DiskSMART_ssd0": 0,
"DiskPresent_ssd1": true,
"DiskTemperature_ssd1": 34,
"DiskSMART_ssd1": 0,
"DiskPresent_ssd2": true,
"DiskTemperature_ssd2": 35,
"DiskSMART_ssd2": 0,
"DiskPresent_ssd3": true,
"DiskTemperature_ssd3": 34,
"DiskSMART_ssd3": 0,
"DiskPresent_ssd4": true,
"DiskTemperature_ssd4": 35,
"DiskSMART_ssd4": 0,
"DiskPresent_ssd5": true,
"DiskTemperature_ssd5": 40,
"DiskSMART_ssd5": 0,
"DiskPresent_ssd6": true,
"DiskTemperature_ssd6": 36,
"DiskSMART_ssd6": 0,
"DiskPresent_ssd7": true,
"DiskTemperature_ssd7": 37,
"DiskSMART_ssd7": 0,
"DiskPresent_par0": true,
"DiskTemperature_par0": 34,
"DiskSMART_par0": 0
},
"link": {
"module": "system",
"subsystem": "link",
"timestamp": 1695785514,
"ver": "9120",
"cap0_link": true,
"cap1_link": true,
"cap2_link": true,
"cap3_link": true,
"cap4_link": true,
"cap5_link": true,
"cap6_link": true,
"cap7_link": true,
"man0_link": true,
"man10_link": true
},
"io": {
"module": "system",
"subsystem": "io",
"timestamp": 1695785514,
"ver": "9120",
"DiskRdGbps": 0.46,
"DiskWrGbps": 0.19
},
"capture": {
"module": "system",
"subsystem": "capture",
"timestamp": 1695785514,
"ver": "9120",
"CaptureEnb": true,
"CapturePkt": 11751922,
"CaptureByte": 16465381240,
"CaptureDrop": 0,
"CaptureFCSError": 0,
"CaptureRateGbps": 0.214141,
"CaptureRateMpps": 0.0188,
"CaptureName": "wan_colo0_20230927_0320",
"CapturePort0_Byte": 0,
"CapturePort0_Pkt": 0,
"CapturePort1_Byte": 16482394142,
"CapturePort1_Pkt": 11763917,
"CapturePort2_Byte": 0,
"CapturePort2_Pkt": 0,
"CapturePort3_Byte": 0,
"CapturePort3_Pkt": 0,
"CapturePort4_Byte": 0,
"CapturePort4_Pkt": 0,
"CapturePort5_Byte": 0,
"CapturePort5_Pkt": 0,
"CapturePort6_Byte": 0,
"CapturePort6_Pkt": 0,
"CapturePort7_Byte": 0,
"CapturePort7_Pkt": 0
},
"power": {
"module": "system",
"subsystem": "power",
"timestamp": 1695785514,
"ver": "9120",
"PSU0_Status": false,
"PSU1_Status": true,
"PSU_PowerWatt": 370
},
"other": {
"module": "system",
"subsystem": "other",
"timestamp": 1695785514,
"ver": "9120",
"UptimeHour": 0.34,
"MemFree": 350465687552,
"MemErrorECC": 0,
"MemCached": 19210211328,
"MemMapped": 5480173568,
"MemBuffer": 57253888,
"MemDirty": 0,
"PageInByte": 0,
"FDCnt": 1188,
"WritebackB": 0,
"WritebackPct": 0,
"WritebackDropTotalG": 0,
"WritebackDropG": 0,
"CacheSize": 30726047137792,
"StoreSize": 30725994708992,
"CPULoad": 5.1,
"SerialNo": "undef-undef-e0d55e5d2150",
"PortConfig": "8x10G",
"Version": "fmadio100v2:9120pcap2json:715"
},
"cat": {
"module": "system",
"subsystem": "cat",
"timestamp": 1695785514,
"ver": "9120",
"cat_0_Enable": true,
"cat_0_Mode": "FMADRing",
"cat_0_CPUMain": 0,
"cat_0_TSPCAP": 1695785512,
"cat_0_ReadPkt": 20194,
"cat_0_ReadByte": 28774880,
"cat_0_ReadTotalPkt": 11727311,
"cat_0_ReadTotalByte": 16508052240,
"cat_0_ReadGbps": 0.221358,
"cat_0_ReadMpps": 0.019418,
"cat_0_WritePkt": 40388,
"cat_0_WriteByte": 57412864,
"cat_0_WriteTotalPkt": 11727311,
"cat_0_WriteTotalByte": 16508052240,
"cat_0_WriteGbps": 0.441662,
"cat_0_WriteMpps": 0.038837,
"cat_0_PendingByte": 30932992,
"cat_0_PktDiscard": 0,
"cat_0_PktDiscardTotal": 0,
"cat_0_PktSlice": 0,
"cat_0_IOPriority": 20,
"cat_0_ChunkID": 11003385,
"cat_0_CmdLine": "/opt/fmadio/bin/stream_cat--uidpush_pcap_1695784865262465024--follow-start--nop-truncate--ring-eof--ring/opt/fmadio/queue/pcap_ring_all1sec--ring-cpu/opt/fmadio/queue/pcap_ring_all1sec23--ring-filter-bpf/opt/fmadio/queue/pcap_ring_all1sec--ring-filter-frame/opt/fmadio/queue/pcap_ring_all1sec--ring/opt/fmadio/queue/pcap_ring_icmp--ring-cpu/opt/fmadio/queue/pcap_ring_icmp23--ring-filter-bpf/opt/fmadio/queue/pcap_ring_icmpicmp--ring-filter-frame/opt/fmadio/queue/pcap_ring_icmp",
"cat_0_StreamName": "wan_colo0_20230927_0320",
"cat_0_FilterBPF": "",
"cat_0_CPUIdle": 0.9769,
"cat_0_CPUFetch": 0.0148,
"cat_0_CPUProcess": 0.0148,
"cat_0_CPUSend": 0,
"cat_1_Enable": true,
"cat_1_Mode": "FMADRing",
"cat_1_CPUMain": 0,
"cat_1_TSPCAP": 1695785504,
"cat_1_ReadPkt": 189594,
"cat_1_ReadByte": 271169120,
"cat_1_ReadTotalPkt": 10854873,
"cat_1_ReadTotalByte": 15340528240,
"cat_1_ReadGbps": 0.215211,
"cat_1_ReadMpps": 0.018809,
"cat_1_WritePkt": 189594,
"cat_1_WriteByte": 269871369,
"cat_1_WriteTotalPkt": 10854873,
"cat_1_WriteTotalByte": 15340528240,
"cat_1_WriteGbps": 0.214182,
"cat_1_WriteMpps": 0.018809,
"cat_1_PendingByte": 31719424,
"cat_1_PktDiscard": 0,
"cat_1_PktDiscardTotal": 0,
"cat_1_PktSlice": 0,
"cat_1_IOPriority": 20,
"cat_1_ChunkID": 11002639,
"cat_1_CmdLine": "/opt/fmadio/bin/stream_cat--uidpush_lxc_1695784924583593984--follow--ring/opt/fmadio/queue/lxc_market2json_euronext_sbe--ring-filter-bpf/opt/fmadio/queue/lxc_market2json_euronext_sbevlan177andnet224.0.208.0/24--ring-filter-frame/opt/fmadio/queue/lxc_market2json_euronext_sbe--cpu23-v--print-period10e9",
"cat_1_StreamName": "wan_colo0_20230927_0320",
"cat_1_FilterBPF": "",
"cat_1_CPUIdle": 0.9928,
"cat_1_CPUFetch": 0.0064,
"cat_1_CPUProcess": 0.0064,
"cat_1_CPUSend": 0,
"cat_2_Enable": false,
"cat_2_Mode": "",
"cat_2_CPUMain": 0,
"cat_2_TSPCAP": 0,
"cat_2_ReadPkt": 0,
"cat_2_ReadByte": 0,
"cat_2_ReadTotalPkt": 0,
"cat_2_ReadTotalByte": 0,
"cat_2_ReadGbps": 0,
"cat_2_ReadMpps": 0,
"cat_2_WritePkt": 0,
"cat_2_WriteByte": 0,
"cat_2_WriteTotalPkt": 0,
"cat_2_WriteTotalByte": 0,
"cat_2_WriteGbps": 0,
"cat_2_WriteMpps": 0,
"cat_2_PendingByte": 0,
"cat_2_PktDiscard": 0,
"cat_2_PktDiscardTotal": 0,
"cat_2_PktSlice": 0,
"cat_2_IOPriority": 0,
"cat_2_ChunkID": 0,
"cat_2_CmdLine": "",
"cat_2_StreamName": "",
"cat_2_FilterBPF": "",
"cat_2_CPUIdle": 0,
"cat_2_CPUFetch": 0,
"cat_2_CPUProcess": 0,
"cat_2_CPUSend": 0,
"cat_3_Enable": false,
"cat_3_Mode":