Alerts can be generated by system automatically either
EMail alerts
Syslog alerts
SNMP Traps
Alert configuration file is located in
By default all Alert triggers are disabled.
An example alert.lua file is shown below. If the file does not exist please create.
System has can trigger an a small but well defined list of critical Events. The following is a description and example for each item. Triggers are enabled or disabled in the following part of the configuration file. Each line enables/disabled or puts a threshold on the trigger
Each trigger is described below.
Monitoring the capture link status is critical to ensure no data is lost. Enabling this option will alert when a capture link goes up or down.
Capture State shows the capture is active or in-active. When using in alert mode it will trigger anytime the capture state changes
Bytes Cached indicates how much capture data has been written to SSD, but not written back into long term storage yet. e.g. Its the delta between the capture SSD rate, and the HDD magnetic storage writeback. Trigger on for example 3TB here provides a good indication the HDD writeback process is running too slow for the sustained incoming capture rate.
(example trigger once Cache goes overt 1TB)
Any time Bytes Over increases an alert is generated. This typically a symptom of capture rates being too high, or HDD writeback too slow (or failing)
Counts FCS errors received on the interface. Any time packet error counts changes an alert is generated. Typically occurs when there are Layer1 link stability issues
Alerts generated when packets are dropped on the capture device.
When space on /mnt/store0 partition is less than this amount (scientific notation) in bytes. Alerts are generated.
In the below example, an alert is generated when less than 4e9 (4GB) of space is free on /mnt/store0 partition
When space on /mnt/store1 (scratch analytics workspace) is less than this amount (scientific notation) in bytes an Alert is generated
When space on the /mnt/remote0 (typically NFS mount partition) is less than this threshold an Alert is generated
Alerts when there is a disk error or RAID error on the device. For example a disk has been lost or HDD RAID redundancy has been reduced.
Alerts on the total number of disk SMART errors. The value is aggregated across all disks, please check the system log files for more details about which specific disk is having an issue.
Minimum number of seconds between alert generation. This is to prevent spamming of alerts due to unexpected system conditions.
Alert events are always output to SYSLOG regardless of the other transport modes (email/snmp etc)
SYSLOG logfile is found in
An example syslog alerts as follows.
Email alerts can be setup as the following, please add the ["Email"] section in the alet configuration file
An example that sends alerts to the address "alerts@fmad.io" is shown below.
In addition fmadio packet capture system uses msrtp as the email client, it requires smtp configuration file
Example configuration as follows. Please edit to match the email smtp provider
FW: 7611+
FMADIO devices can operate in SNMP Broadcast mode. In this mode the system will periodically broadcast all SNMP counter values at a fixed time interval to an SNMP target.
Latest MIB file is found (last updated 2021/12/25)
The general configuration file is used for config
Please edit the section titles ["SNMP"] as follows
The above config enables SNMP Broadcast mode only, while SNMP Trap(Alert) mode is disabled. Broadcast frequency is 60e9 nanoseconds, e.g. every 1 minute.
Broadcast and Trap mode can be use simultaneously if required.
Please update ["Target"] = setting to the correct SNMP collector address. Multiple SNMP targets can be specified separated by spaces. For example
Example output in broadcast mode is as follows, from the /mnt/store0/log/monitor_alert.cur logfile
This translates to
Logfiles are found /mnt/store0/log/monitor_alert.cur
Verbose mode above can be set to "true" to allow additional logging.
FW: 7611+
FMADIO Devices can send SNMP Traps based on the alert triggers described above. This may be preferable to email alerts for infrastructure management.
Latest MIB file is found (last updated 2021/12/25)
The general configuration file is used for config
Please edit the section titles ["SNMP"] as follows
The above config enables SNMP TRAP mode only, SNMP Broadcast mode is disabled. This configuration will only send SNMP TRAP events when a Trigger is alerted.
Please update ["Target"] = setting to the correct SNMP collector address.
An easy way to trouble shoot traps is to se the DiskFreeStore0 threshold to a very large number. In this setup the SNMP TRAP event will be constantly generated (every 1 minute).
Logfiles are found in /mnt/store0/log/monitor_alert.cur
