Prometheus

Introduction

Prometheus offers a simple and easy get started monitoring infrastructure that works out of the box. It uses a pull method where the Prometheus server will poll each device for its status.

The documentation and download for setting up Prometheus is located below

https://prometheus.io/docs/introduction/first_steps/

Configuration

Configuration is simple, by pointing the Prometheus to the following URL on the FMADIO host system

https://<fmadio host IP>/metrics

Step 1) Set a target to FMADIO device

Typically this is done by editing the Prometheus configuration file located

/etc/prometheus/prometheus.yml

An example configuration is shown below, here the FMADIO system is located at 192.168.133.10 over TLS on port 443

fmadio@prometheus0:/etc/prometheus$ cat prometheus.yml
# my global config
global:
  scrape_interval: 5s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  query_log_file: /var/log/prometheus/query.log
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "fmadio20v3-333"
    scrape_interval: 5s
    scrape_timeout: 5s
    scheme: https
    basic_auth:
      username: fmadio
      password: 100g

    static_configs:
      - targets: ["192.168.133.10:443"]
    tls_config:
      insecure_skip_verify: true
fmadio@prometheus0:/etc/prometheus$

The key part is the static_configs targets section, specifically

  - targets: ["192.168.133.10:443"]  

As FMADIO device uses the default /metrics URL only the IP location is required.

Step 2) restart Prometheus

Usually this is done with systemctl service such as below

fmadio@prometheus0:/etc/prometheus$ sudo systemctl restart prometheus.service
fmadio@prometheus0:/etc/prometheus$

Step 3) Confirm Target on the GUI

Then confirm the target is visible on the prometheus servers web page.

Example is plotting out

fmadio_capture_byte

As shown below the target we added “192.168.133.10” is visible in the list, as shown below

Dashboard

FMADIO Reference Dashboard

A reference dashboard using the FMADIO telemetry is located on the FMADIO public Github repo

https://github.com/fmadio/public/blob/master/prometheus/FMADIO.Prometheus-GrafanaDashboard.json

Which gives a good high and detailed overview of the system and provides a starting point for your own dashboarding.

Debugging

/metrics

In the case of problems, a good base test is to curl the /metrics URL directly to confirm the output is valid.

For example.

 curl -k -u fmadio:100g https://192.168.133.10/metrics

Example full output of the all metrics is shown below.

$ curl -k -u fmadio:***** https://192.168.133.10/metrics
# fmadio_ver number
fmadio_ver 9927
# fmadio_cat_cpu_fetch{ioqueue="0"} number
fmadio_cat_cpu_fetch{ioqueue="0"} 0
# fmadio_cat_cpu_idle{ioqueue="0"} number
fmadio_cat_cpu_idle{ioqueue="0"} 0
# fmadio_cat_cpu_main{ioqueue="0"} number
fmadio_cat_cpu_main{ioqueue="0"} 0
# fmadio_cat_cpu_process{ioqueue="0"} number
fmadio_cat_cpu_process{ioqueue="0"} 0
# fmadio_cat_cpu_send{ioqueue="0"} number
fmadio_cat_cpu_send{ioqueue="0"} 0
# fmadio_cat_chunkid{ioqueue="0"} number
fmadio_cat_chunkid{ioqueue="0"} 0
# fmadio_cat_cat_0_CmdLine string
# fmadio_cat_enable{ioqueue="0"} boolean
fmadio_cat_enable{ioqueue="0"} 0
# fmadio_cat_cat_0_FilterBPF string
# fmadio_cat_iopriority{ioqueue="0"} number
fmadio_cat_iopriority{ioqueue="0"} 0
# fmadio_cat_cat_0_Mode string
# fmadio_cat_pending_byte{ioqueue="0"} number
fmadio_cat_pending_byte{ioqueue="0"} 0
# fmadio_cat_pkt_discard{ioqueue="0"} number
fmadio_cat_pkt_discard{ioqueue="0"} 0
# fmadio_cat_pkt_discard_total{ioqueue="0"} number
fmadio_cat_pkt_discard_total{ioqueue="0"} 0
# fmadio_cat_pkt_slice{ioqueue="0"} number
fmadio_cat_pkt_slice{ioqueue="0"} 0
# fmadio_cat_read_byte{ioqueue="0"} number
fmadio_cat_read_byte{ioqueue="0"} 0
# fmadio_cat_read_gbps{ioqueue="0"} number
fmadio_cat_read_gbps{ioqueue="0"} 0
# fmadio_cat_read_mpps{ioqueue="0"} number
fmadio_cat_read_mpps{ioqueue="0"} 0
# fmadio_cat_read_pkt{ioqueue="0"} number
fmadio_cat_read_pkt{ioqueue="0"} 0
# fmadio_cat_read_byte_total{ioqueue="0"} number
fmadio_cat_read_byte_total{ioqueue="0"} 0
# fmadio_cat_read_pkt_total{ioqueue="0"} number
fmadio_cat_read_pkt_total{ioqueue="0"} 0
# fmadio_cat_cat_0_StreamName string
# fmadio_cat_ts_pcap{ioqueue="0"} number
fmadio_cat_ts_pcap{ioqueue="0"} 0
# fmadio_cat_write_byte{ioqueue="0"} number
fmadio_cat_write_byte{ioqueue="0"} 0
# fmadio_cat_write_gbps{ioqueue="0"} number
fmadio_cat_write_gbps{ioqueue="0"} 0
# fmadio_cat_write_mpps{ioqueue="0"} number
fmadio_cat_write_mpps{ioqueue="0"} 0
# fmadio_cat_write_pkt{ioqueue="0"} number
fmadio_cat_write_pkt{ioqueue="0"} 0
# fmadio_cat_write_byte_total{ioqueue="0"} number
fmadio_cat_write_byte_total{ioqueue="0"} 0
# fmadio_cat_write_pkt_total{ioqueue="0"} number
fmadio_cat_write_pkt_total{ioqueue="0"} 0
# fmadio_cat_cpu_fetch{ioqueue="1"} number
fmadio_cat_cpu_fetch{ioqueue="1"} 0
# fmadio_cat_cpu_idle{ioqueue="1"} number
fmadio_cat_cpu_idle{ioqueue="1"} 0
# fmadio_cat_cpu_main{ioqueue="1"} number
fmadio_cat_cpu_main{ioqueue="1"} 0
# fmadio_cat_cpu_process{ioqueue="1"} number
fmadio_cat_cpu_process{ioqueue="1"} 0
# fmadio_cat_cpu_send{ioqueue="1"} number
fmadio_cat_cpu_send{ioqueue="1"} 0
# fmadio_cat_chunkid{ioqueue="1"} number
fmadio_cat_chunkid{ioqueue="1"} 0
# fmadio_cat_cat_1_CmdLine string
# fmadio_cat_enable{ioqueue="1"} boolean
fmadio_cat_enable{ioqueue="1"} 0
# fmadio_cat_cat_1_FilterBPF string
# fmadio_cat_iopriority{ioqueue="1"} number
fmadio_cat_iopriority{ioqueue="1"} 0
# fmadio_cat_cat_1_Mode string
# fmadio_cat_pending_byte{ioqueue="1"} number
fmadio_cat_pending_byte{ioqueue="1"} 0
# fmadio_cat_pkt_discard{ioqueue="1"} number
fmadio_cat_pkt_discard{ioqueue="1"} 0
# fmadio_cat_pkt_discard_total{ioqueue="1"} number
fmadio_cat_pkt_discard_total{ioqueue="1"} 0
# fmadio_cat_pkt_slice{ioqueue="1"} number
fmadio_cat_pkt_slice{ioqueue="1"} 0
# fmadio_cat_read_byte{ioqueue="1"} number
fmadio_cat_read_byte{ioqueue="1"} 0
# fmadio_cat_read_gbps{ioqueue="1"} number
fmadio_cat_read_gbps{ioqueue="1"} 0
# fmadio_cat_read_mpps{ioqueue="1"} number
fmadio_cat_read_mpps{ioqueue="1"} 0
# fmadio_cat_read_pkt{ioqueue="1"} number
fmadio_cat_read_pkt{ioqueue="1"} 0
# fmadio_cat_read_byte_total{ioqueue="1"} number
fmadio_cat_read_byte_total{ioqueue="1"} 0
# fmadio_cat_read_pkt_total{ioqueue="1"} number
fmadio_cat_read_pkt_total{ioqueue="1"} 0
# fmadio_cat_cat_1_StreamName string
# fmadio_cat_ts_pcap{ioqueue="1"} number
fmadio_cat_ts_pcap{ioqueue="1"} 0
# fmadio_cat_write_byte{ioqueue="1"} number
fmadio_cat_write_byte{ioqueue="1"} 0
# fmadio_cat_write_gbps{ioqueue="1"} number
fmadio_cat_write_gbps{ioqueue="1"} 0
# fmadio_cat_write_mpps{ioqueue="1"} number
fmadio_cat_write_mpps{ioqueue="1"} 0
# fmadio_cat_write_pkt{ioqueue="1"} number
fmadio_cat_write_pkt{ioqueue="1"} 0
# fmadio_cat_write_byte_total{ioqueue="1"} number
fmadio_cat_write_byte_total{ioqueue="1"} 0
# fmadio_cat_write_pkt_total{ioqueue="1"} number
fmadio_cat_write_pkt_total{ioqueue="1"} 0
# fmadio_cat_cpu_fetch{ioqueue="2"} number
fmadio_cat_cpu_fetch{ioqueue="2"} 0
# fmadio_cat_cpu_idle{ioqueue="2"} number
fmadio_cat_cpu_idle{ioqueue="2"} 0
# fmadio_cat_cpu_main{ioqueue="2"} number
fmadio_cat_cpu_main{ioqueue="2"} 0
# fmadio_cat_cpu_process{ioqueue="2"} number
fmadio_cat_cpu_process{ioqueue="2"} 0
# fmadio_cat_cpu_send{ioqueue="2"} number
fmadio_cat_cpu_send{ioqueue="2"} 0
# fmadio_cat_chunkid{ioqueue="2"} number
fmadio_cat_chunkid{ioqueue="2"} 0
# fmadio_cat_cat_2_CmdLine string
# fmadio_cat_enable{ioqueue="2"} boolean
fmadio_cat_enable{ioqueue="2"} 0
# fmadio_cat_cat_2_FilterBPF string
# fmadio_cat_iopriority{ioqueue="2"} number
fmadio_cat_iopriority{ioqueue="2"} 0
# fmadio_cat_cat_2_Mode string
# fmadio_cat_pending_byte{ioqueue="2"} number
fmadio_cat_pending_byte{ioqueue="2"} 0
# fmadio_cat_pkt_discard{ioqueue="2"} number
fmadio_cat_pkt_discard{ioqueue="2"} 0
# fmadio_cat_pkt_discard_total{ioqueue="2"} number
fmadio_cat_pkt_discard_total{ioqueue="2"} 0
# fmadio_cat_pkt_slice{ioqueue="2"} number
fmadio_cat_pkt_slice{ioqueue="2"} 0
# fmadio_cat_read_byte{ioqueue="2"} number
fmadio_cat_read_byte{ioqueue="2"} 0
# fmadio_cat_read_gbps{ioqueue="2"} number
fmadio_cat_read_gbps{ioqueue="2"} 0
# fmadio_cat_read_mpps{ioqueue="2"} number
fmadio_cat_read_mpps{ioqueue="2"} 0
# fmadio_cat_read_pkt{ioqueue="2"} number
fmadio_cat_read_pkt{ioqueue="2"} 0
# fmadio_cat_read_byte_total{ioqueue="2"} number
fmadio_cat_read_byte_total{ioqueue="2"} 0
# fmadio_cat_read_pkt_total{ioqueue="2"} number
fmadio_cat_read_pkt_total{ioqueue="2"} 0
# fmadio_cat_cat_2_StreamName string
# fmadio_cat_ts_pcap{ioqueue="2"} number
fmadio_cat_ts_pcap{ioqueue="2"} 0
# fmadio_cat_write_byte{ioqueue="2"} number
fmadio_cat_write_byte{ioqueue="2"} 0
# fmadio_cat_write_gbps{ioqueue="2"} number
fmadio_cat_write_gbps{ioqueue="2"} 0
# fmadio_cat_write_mpps{ioqueue="2"} number
fmadio_cat_write_mpps{ioqueue="2"} 0
# fmadio_cat_write_pkt{ioqueue="2"} number
fmadio_cat_write_pkt{ioqueue="2"} 0
# fmadio_cat_write_byte_total{ioqueue="2"} number
fmadio_cat_write_byte_total{ioqueue="2"} 0
# fmadio_cat_write_pkt_total{ioqueue="2"} number
fmadio_cat_write_pkt_total{ioqueue="2"} 0
# fmadio_cat_cpu_fetch{ioqueue="3"} number
fmadio_cat_cpu_fetch{ioqueue="3"} 0
# fmadio_cat_cpu_idle{ioqueue="3"} number
fmadio_cat_cpu_idle{ioqueue="3"} 0
# fmadio_cat_cpu_main{ioqueue="3"} number
fmadio_cat_cpu_main{ioqueue="3"} 0
# fmadio_cat_cpu_process{ioqueue="3"} number
fmadio_cat_cpu_process{ioqueue="3"} 0
# fmadio_cat_cpu_send{ioqueue="3"} number
fmadio_cat_cpu_send{ioqueue="3"} 0
# fmadio_cat_chunkid{ioqueue="3"} number
fmadio_cat_chunkid{ioqueue="3"} 0
# fmadio_cat_cat_3_CmdLine string
# fmadio_cat_enable{ioqueue="3"} boolean
fmadio_cat_enable{ioqueue="3"} 0
# fmadio_cat_cat_3_FilterBPF string
# fmadio_cat_iopriority{ioqueue="3"} number
fmadio_cat_iopriority{ioqueue="3"} 0
# fmadio_cat_cat_3_Mode string
# fmadio_cat_pending_byte{ioqueue="3"} number
fmadio_cat_pending_byte{ioqueue="3"} 0
# fmadio_cat_pkt_discard{ioqueue="3"} number
fmadio_cat_pkt_discard{ioqueue="3"} 0
# fmadio_cat_pkt_discard_total{ioqueue="3"} number
fmadio_cat_pkt_discard_total{ioqueue="3"} 0
# fmadio_cat_pkt_slice{ioqueue="3"} number
fmadio_cat_pkt_slice{ioqueue="3"} 0
# fmadio_cat_read_byte{ioqueue="3"} number
fmadio_cat_read_byte{ioqueue="3"} 0
# fmadio_cat_read_gbps{ioqueue="3"} number
fmadio_cat_read_gbps{ioqueue="3"} 0
# fmadio_cat_read_mpps{ioqueue="3"} number
fmadio_cat_read_mpps{ioqueue="3"} 0
# fmadio_cat_read_pkt{ioqueue="3"} number
fmadio_cat_read_pkt{ioqueue="3"} 0
# fmadio_cat_read_byte_total{ioqueue="3"} number
fmadio_cat_read_byte_total{ioqueue="3"} 0
# fmadio_cat_read_pkt_total{ioqueue="3"} number
fmadio_cat_read_pkt_total{ioqueue="3"} 0
# fmadio_cat_cat_3_StreamName string
# fmadio_cat_ts_pcap{ioqueue="3"} number
fmadio_cat_ts_pcap{ioqueue="3"} 0
# fmadio_cat_write_byte{ioqueue="3"} number
fmadio_cat_write_byte{ioqueue="3"} 0
# fmadio_cat_write_gbps{ioqueue="3"} number
fmadio_cat_write_gbps{ioqueue="3"} 0
# fmadio_cat_write_mpps{ioqueue="3"} number
fmadio_cat_write_mpps{ioqueue="3"} 0
# fmadio_cat_write_pkt{ioqueue="3"} number
fmadio_cat_write_pkt{ioqueue="3"} 0
# fmadio_cat_write_byte_total{ioqueue="3"} number
fmadio_cat_write_byte_total{ioqueue="3"} 0
# fmadio_cat_write_pkt_total{ioqueue="3"} number
fmadio_cat_write_pkt_total{ioqueue="3"} 0
# fmadio_cat_cpu_fetch{ioqueue="4"} number
fmadio_cat_cpu_fetch{ioqueue="4"} 0
# fmadio_cat_cpu_idle{ioqueue="4"} number
fmadio_cat_cpu_idle{ioqueue="4"} 0
# fmadio_cat_cpu_main{ioqueue="4"} number
fmadio_cat_cpu_main{ioqueue="4"} 0
# fmadio_cat_cpu_process{ioqueue="4"} number
fmadio_cat_cpu_process{ioqueue="4"} 0
# fmadio_cat_cpu_send{ioqueue="4"} number
fmadio_cat_cpu_send{ioqueue="4"} 0
# fmadio_cat_chunkid{ioqueue="4"} number
fmadio_cat_chunkid{ioqueue="4"} 0
# fmadio_cat_cat_4_CmdLine string
# fmadio_cat_enable{ioqueue="4"} boolean
fmadio_cat_enable{ioqueue="4"} 0
# fmadio_cat_cat_4_FilterBPF string
# fmadio_cat_iopriority{ioqueue="4"} number
fmadio_cat_iopriority{ioqueue="4"} 0
# fmadio_cat_cat_4_Mode string
# fmadio_cat_pending_byte{ioqueue="4"} number
fmadio_cat_pending_byte{ioqueue="4"} 0
# fmadio_cat_pkt_discard{ioqueue="4"} number
fmadio_cat_pkt_discard{ioqueue="4"} 0
# fmadio_cat_pkt_discard_total{ioqueue="4"} number
fmadio_cat_pkt_discard_total{ioqueue="4"} 0
# fmadio_cat_pkt_slice{ioqueue="4"} number
fmadio_cat_pkt_slice{ioqueue="4"} 0
# fmadio_cat_read_byte{ioqueue="4"} number
fmadio_cat_read_byte{ioqueue="4"} 0
# fmadio_cat_read_gbps{ioqueue="4"} number
fmadio_cat_read_gbps{ioqueue="4"} 0
# fmadio_cat_read_mpps{ioqueue="4"} number
fmadio_cat_read_mpps{ioqueue="4"} 0
# fmadio_cat_read_pkt{ioqueue="4"} number
fmadio_cat_read_pkt{ioqueue="4"} 0
# fmadio_cat_read_byte_total{ioqueue="4"} number
fmadio_cat_read_byte_total{ioqueue="4"} 0
# fmadio_cat_read_pkt_total{ioqueue="4"} number
fmadio_cat_read_pkt_total{ioqueue="4"} 0
# fmadio_cat_cat_4_StreamName string
# fmadio_cat_ts_pcap{ioqueue="4"} number
fmadio_cat_ts_pcap{ioqueue="4"} 0
# fmadio_cat_write_byte{ioqueue="4"} number
fmadio_cat_write_byte{ioqueue="4"} 0
# fmadio_cat_write_gbps{ioqueue="4"} number
fmadio_cat_write_gbps{ioqueue="4"} 0
# fmadio_cat_write_mpps{ioqueue="4"} number
fmadio_cat_write_mpps{ioqueue="4"} 0
# fmadio_cat_write_pkt{ioqueue="4"} number
fmadio_cat_write_pkt{ioqueue="4"} 0
# fmadio_cat_write_byte_total{ioqueue="4"} number
fmadio_cat_write_byte_total{ioqueue="4"} 0
# fmadio_cat_write_pkt_total{ioqueue="4"} number
fmadio_cat_write_pkt_total{ioqueue="4"} 0
# fmadio_cat_cpu_fetch{ioqueue="5"} number
fmadio_cat_cpu_fetch{ioqueue="5"} 0
# fmadio_cat_cpu_idle{ioqueue="5"} number
fmadio_cat_cpu_idle{ioqueue="5"} 0
# fmadio_cat_cpu_main{ioqueue="5"} number
fmadio_cat_cpu_main{ioqueue="5"} 0
# fmadio_cat_cpu_process{ioqueue="5"} number
fmadio_cat_cpu_process{ioqueue="5"} 0
# fmadio_cat_cpu_send{ioqueue="5"} number
fmadio_cat_cpu_send{ioqueue="5"} 0
# fmadio_cat_chunkid{ioqueue="5"} number
fmadio_cat_chunkid{ioqueue="5"} 0
# fmadio_cat_cat_5_CmdLine string
# fmadio_cat_enable{ioqueue="5"} boolean
fmadio_cat_enable{ioqueue="5"} 0
# fmadio_cat_cat_5_FilterBPF string
# fmadio_cat_iopriority{ioqueue="5"} number
fmadio_cat_iopriority{ioqueue="5"} 0
# fmadio_cat_cat_5_Mode string
# fmadio_cat_pending_byte{ioqueue="5"} number
fmadio_cat_pending_byte{ioqueue="5"} 0
# fmadio_cat_pkt_discard{ioqueue="5"} number
fmadio_cat_pkt_discard{ioqueue="5"} 0
# fmadio_cat_pkt_discard_total{ioqueue="5"} number
fmadio_cat_pkt_discard_total{ioqueue="5"} 0
# fmadio_cat_pkt_slice{ioqueue="5"} number
fmadio_cat_pkt_slice{ioqueue="5"} 0
# fmadio_cat_read_byte{ioqueue="5"} number
fmadio_cat_read_byte{ioqueue="5"} 0
# fmadio_cat_read_gbps{ioqueue="5"} number
fmadio_cat_read_gbps{ioqueue="5"} 0
# fmadio_cat_read_mpps{ioqueue="5"} number
fmadio_cat_read_mpps{ioqueue="5"} 0
# fmadio_cat_read_pkt{ioqueue="5"} number
fmadio_cat_read_pkt{ioqueue="5"} 0
# fmadio_cat_read_byte_total{ioqueue="5"} number
fmadio_cat_read_byte_total{ioqueue="5"} 0
# fmadio_cat_read_pkt_total{ioqueue="5"} number
fmadio_cat_read_pkt_total{ioqueue="5"} 0
# fmadio_cat_cat_5_StreamName string
# fmadio_cat_ts_pcap{ioqueue="5"} number
fmadio_cat_ts_pcap{ioqueue="5"} 0
# fmadio_cat_write_byte{ioqueue="5"} number
fmadio_cat_write_byte{ioqueue="5"} 0
# fmadio_cat_write_gbps{ioqueue="5"} number
fmadio_cat_write_gbps{ioqueue="5"} 0
# fmadio_cat_write_mpps{ioqueue="5"} number
fmadio_cat_write_mpps{ioqueue="5"} 0
# fmadio_cat_write_pkt{ioqueue="5"} number
fmadio_cat_write_pkt{ioqueue="5"} 0
# fmadio_cat_write_byte_total{ioqueue="5"} number
fmadio_cat_write_byte_total{ioqueue="5"} 0
# fmadio_cat_write_pkt_total{ioqueue="5"} number
fmadio_cat_write_pkt_total{ioqueue="5"} 0
# fmadio_cat_cpu_fetch{ioqueue="6"} number
fmadio_cat_cpu_fetch{ioqueue="6"} 0
# fmadio_cat_cpu_idle{ioqueue="6"} number
fmadio_cat_cpu_idle{ioqueue="6"} 0
# fmadio_cat_cpu_main{ioqueue="6"} number
fmadio_cat_cpu_main{ioqueue="6"} 0
# fmadio_cat_cpu_process{ioqueue="6"} number
fmadio_cat_cpu_process{ioqueue="6"} 0
# fmadio_cat_cpu_send{ioqueue="6"} number
fmadio_cat_cpu_send{ioqueue="6"} 0
# fmadio_cat_chunkid{ioqueue="6"} number
fmadio_cat_chunkid{ioqueue="6"} 0
# fmadio_cat_cat_6_CmdLine string
# fmadio_cat_enable{ioqueue="6"} boolean
fmadio_cat_enable{ioqueue="6"} 0
# fmadio_cat_cat_6_FilterBPF string
# fmadio_cat_iopriority{ioqueue="6"} number
fmadio_cat_iopriority{ioqueue="6"} 0
# fmadio_cat_cat_6_Mode string
# fmadio_cat_pending_byte{ioqueue="6"} number
fmadio_cat_pending_byte{ioqueue="6"} 0
# fmadio_cat_pkt_discard{ioqueue="6"} number
fmadio_cat_pkt_discard{ioqueue="6"} 0
# fmadio_cat_pkt_discard_total{ioqueue="6"} number
fmadio_cat_pkt_discard_total{ioqueue="6"} 0
# fmadio_cat_pkt_slice{ioqueue="6"} number
fmadio_cat_pkt_slice{ioqueue="6"} 0
# fmadio_cat_read_byte{ioqueue="6"} number
fmadio_cat_read_byte{ioqueue="6"} 0
# fmadio_cat_read_gbps{ioqueue="6"} number
fmadio_cat_read_gbps{ioqueue="6"} 0
# fmadio_cat_read_mpps{ioqueue="6"} number
fmadio_cat_read_mpps{ioqueue="6"} 0
# fmadio_cat_read_pkt{ioqueue="6"} number
fmadio_cat_read_pkt{ioqueue="6"} 0
# fmadio_cat_read_byte_total{ioqueue="6"} number
fmadio_cat_read_byte_total{ioqueue="6"} 0
# fmadio_cat_read_pkt_total{ioqueue="6"} number
fmadio_cat_read_pkt_total{ioqueue="6"} 0
# fmadio_cat_cat_6_StreamName string
# fmadio_cat_ts_pcap{ioqueue="6"} number
fmadio_cat_ts_pcap{ioqueue="6"} 0
# fmadio_cat_write_byte{ioqueue="6"} number
fmadio_cat_write_byte{ioqueue="6"} 0
# fmadio_cat_write_gbps{ioqueue="6"} number
fmadio_cat_write_gbps{ioqueue="6"} 0
# fmadio_cat_write_mpps{ioqueue="6"} number
fmadio_cat_write_mpps{ioqueue="6"} 0
# fmadio_cat_write_pkt{ioqueue="6"} number
fmadio_cat_write_pkt{ioqueue="6"} 0
# fmadio_cat_write_byte_total{ioqueue="6"} number
fmadio_cat_write_byte_total{ioqueue="6"} 0
# fmadio_cat_write_pkt_total{ioqueue="6"} number
fmadio_cat_write_pkt_total{ioqueue="6"} 0
# fmadio_cat_cpu_fetch{ioqueue="7"} number
fmadio_cat_cpu_fetch{ioqueue="7"} 0
# fmadio_cat_cpu_idle{ioqueue="7"} number
fmadio_cat_cpu_idle{ioqueue="7"} 0
# fmadio_cat_cpu_main{ioqueue="7"} number
fmadio_cat_cpu_main{ioqueue="7"} 0
# fmadio_cat_cpu_process{ioqueue="7"} number
fmadio_cat_cpu_process{ioqueue="7"} 0
# fmadio_cat_cpu_send{ioqueue="7"} number
fmadio_cat_cpu_send{ioqueue="7"} 0
# fmadio_cat_chunkid{ioqueue="7"} number
fmadio_cat_chunkid{ioqueue="7"} 0
# fmadio_cat_cat_7_CmdLine string
# fmadio_cat_enable{ioqueue="7"} boolean
fmadio_cat_enable{ioqueue="7"} 0
# fmadio_cat_cat_7_FilterBPF string
# fmadio_cat_iopriority{ioqueue="7"} number
fmadio_cat_iopriority{ioqueue="7"} 0
# fmadio_cat_cat_7_Mode string
# fmadio_cat_pending_byte{ioqueue="7"} number
fmadio_cat_pending_byte{ioqueue="7"} 0
# fmadio_cat_pkt_discard{ioqueue="7"} number
fmadio_cat_pkt_discard{ioqueue="7"} 0
# fmadio_cat_pkt_discard_total{ioqueue="7"} number
fmadio_cat_pkt_discard_total{ioqueue="7"} 0
# fmadio_cat_pkt_slice{ioqueue="7"} number
fmadio_cat_pkt_slice{ioqueue="7"} 0
# fmadio_cat_read_byte{ioqueue="7"} number
fmadio_cat_read_byte{ioqueue="7"} 0
# fmadio_cat_read_gbps{ioqueue="7"} number
fmadio_cat_read_gbps{ioqueue="7"} 0
# fmadio_cat_read_mpps{ioqueue="7"} number
fmadio_cat_read_mpps{ioqueue="7"} 0
# fmadio_cat_read_pkt{ioqueue="7"} number
fmadio_cat_read_pkt{ioqueue="7"} 0
# fmadio_cat_read_byte_total{ioqueue="7"} number
fmadio_cat_read_byte_total{ioqueue="7"} 0
# fmadio_cat_read_pkt_total{ioqueue="7"} number
fmadio_cat_read_pkt_total{ioqueue="7"} 0
# fmadio_cat_cat_7_StreamName string
# fmadio_cat_ts_pcap{ioqueue="7"} number
fmadio_cat_ts_pcap{ioqueue="7"} 0
# fmadio_cat_write_byte{ioqueue="7"} number
fmadio_cat_write_byte{ioqueue="7"} 0
# fmadio_cat_write_gbps{ioqueue="7"} number
fmadio_cat_write_gbps{ioqueue="7"} 0
# fmadio_cat_write_mpps{ioqueue="7"} number
fmadio_cat_write_mpps{ioqueue="7"} 0
# fmadio_cat_write_pkt{ioqueue="7"} number
fmadio_cat_write_pkt{ioqueue="7"} 0
# fmadio_cat_write_byte_total{ioqueue="7"} number
fmadio_cat_write_byte_total{ioqueue="7"} 0
# fmadio_cat_write_pkt_total{ioqueue="7"} number
fmadio_cat_write_pkt_total{ioqueue="7"} 0
# fmadio_cat_cat_EnableCnt number
fmadio_cat_cat_EnableCnt 0
# fmadio_cat_cat_LockCnt number
fmadio_cat_cat_LockCnt 0
# fmadio_cat_cat_ReadByte number
fmadio_cat_cat_ReadByte 0
# fmadio_cat_cat_ReadGbps number
fmadio_cat_cat_ReadGbps 0
# fmadio_cat_cat_ReadMpps number
fmadio_cat_cat_ReadMpps 0
# fmadio_cat_cat_ReadPkt number
fmadio_cat_cat_ReadPkt 0
# fmadio_cat_cat_ReadTotalByte number
fmadio_cat_cat_ReadTotalByte 0
# fmadio_cat_cat_ReadTotalPkt number
fmadio_cat_cat_ReadTotalPkt 0
# fmadio_cat_cat_WriteByte number
fmadio_cat_cat_WriteByte 0
# fmadio_cat_cat_WriteGbps number
fmadio_cat_cat_WriteGbps 0
# fmadio_cat_cat_WriteMpps number
fmadio_cat_cat_WriteMpps 0
# fmadio_cat_cat_WritePkt number
fmadio_cat_cat_WritePkt 0
# fmadio_cat_cat_WriteTotalByte number
fmadio_cat_cat_WriteTotalByte 0
# fmadio_cat_cat_WriteTotalPkt number
fmadio_cat_cat_WriteTotalPkt 0
# fmadio_cat_module string
# fmadio_cat_subsystem string
# fmadio_cat_timestamp number
# fmadio_cat_ver string
# fmadio_disk_valid{unit="hdd0"} boolean
fmadio_disk_valid{unit="hdd0"} 1
# fmadio_disk_valid{unit="hdd1"} boolean
fmadio_disk_valid{unit="hdd1"} 1
# fmadio_disk_valid{unit="hdd2"} boolean
fmadio_disk_valid{unit="hdd2"} 1
# fmadio_disk_valid{unit="os0"} boolean
fmadio_disk_valid{unit="os0"} 1
# fmadio_disk_valid{unit="par0"} boolean
fmadio_disk_valid{unit="par0"} 1
# fmadio_disk_valid{unit="scr0"} boolean
fmadio_disk_valid{unit="scr0"} 1
# fmadio_disk_valid{unit="scr1"} boolean
fmadio_disk_valid{unit="scr1"} 1
# fmadio_disk_valid{unit="ssd0"} boolean
fmadio_disk_valid{unit="ssd0"} 1
# fmadio_disk_valid{unit="ssd1"} boolean
fmadio_disk_valid{unit="ssd1"} 1
# fmadio_disk_error{unit="hdd0"} number
fmadio_disk_error{unit="hdd0"} 0
# fmadio_disk_error{unit="hdd1"} number
fmadio_disk_error{unit="hdd1"} 0
# fmadio_disk_error{unit="hdd2"} number
fmadio_disk_error{unit="hdd2"} 0
# fmadio_disk_error{unit="os0"} number
fmadio_disk_error{unit="os0"} 0
# fmadio_disk_error{unit="par0"} number
fmadio_disk_error{unit="par0"} 0
# fmadio_disk_error{unit="scr0"} number
fmadio_disk_error{unit="scr0"} 0
# fmadio_disk_error{unit="scr1"} number
fmadio_disk_error{unit="scr1"} 0
# fmadio_disk_error{unit="ssd0"} number
fmadio_disk_error{unit="ssd0"} 0
# fmadio_disk_error{unit="ssd1"} number
fmadio_disk_error{unit="ssd1"} 0
# fmadio_temperature{unit="hdd0"} number
fmadio_temperature{unit="hdd0"} 28
# fmadio_temperature{unit="hdd1"} number
fmadio_temperature{unit="hdd1"} 27
# fmadio_temperature{unit="hdd2"} number
fmadio_temperature{unit="hdd2"} 25
# fmadio_temperature{unit="os0"} number
fmadio_temperature{unit="os0"} 25
# fmadio_temperature{unit="par0"} number
fmadio_temperature{unit="par0"} 26
# fmadio_temperature{unit="scr0"} number
fmadio_temperature{unit="scr0"} 35
# fmadio_temperature{unit="scr1"} number
fmadio_temperature{unit="scr1"} 32
# fmadio_temperature{unit="ssd0"} number
fmadio_temperature{unit="ssd0"} 33
# fmadio_temperature{unit="ssd1"} number
fmadio_temperature{unit="ssd1"} 35
# fmadio_disk_free{unit="remote0"} number
fmadio_disk_free{unit="remote0"} 7896130000000
# fmadio_disk_free{unit="remote1"} number
fmadio_disk_free{unit="remote1"} 7896130000000
# fmadio_disk_free{unit="store0"} number
fmadio_disk_free{unit="store0"} 96389000000
# fmadio_disk_free{unit="store1"} number
fmadio_disk_free{unit="store1"} 1243846000000
# fmadio_disk_free{unit="system"} number
fmadio_disk_free{unit="system"} 1685000000
# fmadio_disk_module string
# fmadio_disk_subsystem string
# fmadio_disk_timestamp number
# fmadio_disk_ver string
# fmadio_counter_module string
# fmadio_counter_subsystem string
# fmadio_counter_timestamp number
# fmadio_counter_updateTS number
fmadio_counter_updateTS 1729674305131685120
# fmadio_counter_ver string
# fmadio_system_cpu number
fmadio_system_cpu 0
# fmadio_other_CacheSize number
fmadio_other_CacheSize 2048418906112
# fmadio_other_FDCnt number
fmadio_other_FDCnt 1681
# fmadio_other_MemBuffer number
fmadio_other_MemBuffer 524369920
# fmadio_other_MemCached number
fmadio_other_MemCached 25623552000
# fmadio_other_MemDirty number
fmadio_other_MemDirty 69632
# fmadio_other_MemErrorECC number
fmadio_other_MemErrorECC 0
# fmadio_system_memfree number
fmadio_system_memfree 47938289664
# fmadio_other_MemMapped number
fmadio_other_MemMapped 3728523264
# fmadio_other_PageInByte number
fmadio_other_PageInByte 0
# fmadio_other_PortConfig string
# fmadio_other_SerialNo string
# fmadio_other_StoreSize number
fmadio_other_StoreSize 12002352168960
# fmadio_other_UptimeHour number
fmadio_other_UptimeHour 166
# fmadio_other_Version string
# fmadio_other_WritebackB number
fmadio_other_WritebackB 0
# fmadio_other_WritebackDropG number
fmadio_other_WritebackDropG 0
# fmadio_other_WritebackDropTotalG number
fmadio_other_WritebackDropTotalG 0
# fmadio_other_WritebackPct number
fmadio_other_WritebackPct 0
# fmadio_other_module string
# fmadio_other_subsystem string
# fmadio_other_timestamp number
# fmadio_other_ver string
# fmadio_cpu_TotalCPU_Cnt number
fmadio_cpu_TotalCPU_Cnt 96
# fmadio_cpu_TotalCPU_IsolCnt number
fmadio_cpu_TotalCPU_IsolCnt 30
# fmadio_cpu_TotalCPU_LXCCnt number
fmadio_cpu_TotalCPU_LXCCnt 7
# fmadio_cpu_TotalCPU_LXCPct number
fmadio_cpu_TotalCPU_LXCPct 0
# fmadio_cpu_TotalCPU_Pct number
fmadio_cpu_TotalCPU_Pct 0
# fmadio_cpu_dSampleTS number
fmadio_cpu_dSampleTS 60212186112
# fmadio_cpu_module string
# fmadio_cpu_subsystem string
# fmadio_cpu_timestamp number
# fmadio_cpu_ver string
# fmadio_capture_byte{interface="all"} number
fmadio_capture_byte{interface="all"} 0
# fmadio_capture_drop{interface="all"} number
fmadio_capture_drop{interface="all"} 0
# fmadio_capture_enable boolean
fmadio_capture_enable 0
# fmadio_capture_fcs_error{interface="all"} number
fmadio_capture_fcs_error{interface="all"} 0
# fmadio_capture_CaptureName string
# fmadio_capture_pkt{interface="all"} number
fmadio_capture_pkt{interface="all"} 0
# fmadio_capture_byte{interface="cap0"} number
fmadio_capture_byte{interface="cap0"} 0
# fmadio_capture_fcs_error{interface="cap0"} number
fmadio_capture_fcs_error{interface="cap0"} 0
# fmadio_capture_pkt{interface="cap0"} number
fmadio_capture_pkt{interface="cap0"} 0
# fmadio_capture_byte{interface="cap1"} number
fmadio_capture_byte{interface="cap1"} 0
# fmadio_capture_fcs_error{interface="cap1"} number
fmadio_capture_fcs_error{interface="cap1"} 0
# fmadio_capture_pkt{interface="cap1"} number
fmadio_capture_pkt{interface="cap1"} 0
# fmadio_capture_byte{interface="cap2"} number
fmadio_capture_byte{interface="cap2"} 0
# fmadio_capture_fcs_error{interface="cap2"} number
fmadio_capture_fcs_error{interface="cap2"} 0
# fmadio_capture_pkt{interface="cap2"} number
fmadio_capture_pkt{interface="cap2"} 0
# fmadio_capture_byte{interface="cap3"} number
fmadio_capture_byte{interface="cap3"} 0
# fmadio_capture_fcs_error{interface="cap3"} number
fmadio_capture_fcs_error{interface="cap3"} 0
# fmadio_capture_pkt{interface="cap3"} number
fmadio_capture_pkt{interface="cap3"} 0
# fmadio_capture_byte{interface="cap4"} number
fmadio_capture_byte{interface="cap4"} 0
# fmadio_capture_fcs_error{interface="cap4"} number
fmadio_capture_fcs_error{interface="cap4"} 0
# fmadio_capture_pkt{interface="cap4"} number
fmadio_capture_pkt{interface="cap4"} 0
# fmadio_capture_byte{interface="cap5"} number
fmadio_capture_byte{interface="cap5"} 0
# fmadio_capture_fcs_error{interface="cap5"} number
fmadio_capture_fcs_error{interface="cap5"} 0
# fmadio_capture_pkt{interface="cap5"} number
fmadio_capture_pkt{interface="cap5"} 0
# fmadio_capture_byte{interface="cap6"} number
fmadio_capture_byte{interface="cap6"} 0
# fmadio_capture_fcs_error{interface="cap6"} number
fmadio_capture_fcs_error{interface="cap6"} 0
# fmadio_capture_pkt{interface="cap6"} number
fmadio_capture_pkt{interface="cap6"} 0
# fmadio_capture_byte{interface="cap7"} number
fmadio_capture_byte{interface="cap7"} 0
# fmadio_capture_fcs_error{interface="cap7"} number
fmadio_capture_fcs_error{interface="cap7"} 0
# fmadio_capture_pkt{interface="cap7"} number
fmadio_capture_pkt{interface="cap7"} 0
# fmadio_capture_CaptureRateGbps number
# fmadio_capture_CaptureRateMpps number
# fmadio_capture_rate_bps number
fmadio_capture_rate_bps 0
# fmadio_capture_rate_pps number
fmadio_capture_rate_pps 0
# fmadio_capture_module string
# fmadio_capture_subsystem string
# fmadio_capture_timestamp number
# fmadio_capture_ver string
# fmadio_temperature{unit="airin"} number
fmadio_temperature{unit="airin"} 22
# fmadio_temperature{unit="airout"} number
fmadio_temperature{unit="airout"} 0
# fmadio_temperature{unit="cpu0"} number
fmadio_temperature{unit="cpu0"} 36
# fmadio_temperature{unit="cpu1"} number
fmadio_temperature{unit="cpu1"} 47
# fmadio_temperature{unit="nic"} number
fmadio_temperature{unit="nic"} 54
# fmadio_temperature{unit="chipset"} number
fmadio_temperature{unit="chipset"} 39
# fmadio_temperature{unit="perhipherals"} number
fmadio_temperature{unit="perhipherals"} 22
# fmadio_temperature{unit="system"} number
fmadio_temperature{unit="system"} 35
# fmadio_temperature{unit="transceiver0"} number
fmadio_temperature{unit="transceiver0"} 38
# fmadio_temperature{unit="transceiver1"} number
fmadio_temperature{unit="transceiver1"} 40
# fmadio_temperature_module string
# fmadio_temperature_subsystem string
# fmadio_temperature_timestamp number
# fmadio_temperature_ver string
# fmadio_ptp_Clk156Offset number
fmadio_ptp_Clk156Offset 0
# fmadio_ptp_Clk156Period number
fmadio_ptp_Clk156Period 6
# fmadio_ptp_Clk250Offset number
fmadio_ptp_Clk250Offset 2
# fmadio_ptp_Clk250Period number
fmadio_ptp_Clk250Period 6
# fmadio_ptp_Clk322Offset number
fmadio_ptp_Clk322Offset 0
# fmadio_ptp_Clk322Period number
fmadio_ptp_Clk322Period 0
# fmadio_ptp_GMMaster string
# fmadio_ptp_GMOffset number
fmadio_ptp_GMOffset 0
# fmadio_ptp_GMSync boolean
fmadio_ptp_GMSync 0
# fmadio_ptp_GMUpTime number
fmadio_ptp_GMUpTime 0
# fmadio_ptp_NTPMaster string
# fmadio_ptp_NTPOffset number
fmadio_ptp_NTPOffset 0
# fmadio_ptp_NTPSync boolean
fmadio_ptp_NTPSync 0
# fmadio_ptp_NTPUptime number
fmadio_ptp_NTPUptime 0
# fmadio_ptp_PPSCnt number
fmadio_ptp_PPSCnt 0
# fmadio_ptp_PPSOffset number
fmadio_ptp_PPSOffset 0
# fmadio_ptp_PPSPeriod number
fmadio_ptp_PPSPeriod 0
# fmadio_ptp_PPSUptime number
fmadio_ptp_PPSUptime 0
# fmadio_ptp_PPSdPhase number
fmadio_ptp_PPSdPhase 0
# fmadio_ptp_PPSdZero number
fmadio_ptp_PPSdZero 0
# fmadio_ptp_SysOffset number
fmadio_ptp_SysOffset 0
# fmadio_ptp_SysSync boolean
fmadio_ptp_SysSync 0
# fmadio_ptp_SysUptime number
fmadio_ptp_SysUptime 0
# fmadio_ptp_TimeFPGA number
fmadio_ptp_TimeFPGA 1729674305130944000
# fmadio_ptp_TimeSYS number
fmadio_ptp_TimeSYS 1729674305130949120
# fmadio_ptp_iSysOffset number
fmadio_ptp_iSysOffset 0
# fmadio_ptp_iSysSync boolean
fmadio_ptp_iSysSync 1
# fmadio_ptp_iSysUptime number
fmadio_ptp_iSysUptime 597953
# fmadio_ptp_module string
# fmadio_ptp_subsystem string
# fmadio_ptp_timestamp number
# fmadio_ptp_ver string
# fmadio_disk_io_bps{unit="read"} number
fmadio_disk_io_bps{unit="read"} 0
# fmadio_disk_io_bps{unit="write"} number
fmadio_disk_io_bps{unit="write"} 0
# fmadio_io_module string
# fmadio_io_subsystem string
# fmadio_io_timestamp number
# fmadio_io_ver string
# fmadio_link_up{interface="cap0"} boolean
fmadio_link_up{interface="cap0"} 1
# fmadio_link_power_rx{interface="cap0"} number
fmadio_link_power_rx{interface="cap0"} 0
# fmadio_link_up{interface="cap1"} boolean
fmadio_link_up{interface="cap1"} 1
# fmadio_link_power_rx{interface="cap1"} number
fmadio_link_power_rx{interface="cap1"} 0
# fmadio_link_up{interface="cap2"} boolean
fmadio_link_up{interface="cap2"} 0
# fmadio_link_power_rx{interface="cap2"} number
fmadio_link_power_rx{interface="cap2"} 0
# fmadio_link_up{interface="cap3"} boolean
fmadio_link_up{interface="cap3"} 0
# fmadio_link_power_rx{interface="cap3"} number
fmadio_link_power_rx{interface="cap3"} 0
# fmadio_link_up{interface="cap4"} boolean
fmadio_link_up{interface="cap4"} 0
# fmadio_link_power_rx{interface="cap4"} number
fmadio_link_power_rx{interface="cap4"} 0
# fmadio_link_up{interface="cap5"} boolean
fmadio_link_up{interface="cap5"} 0
# fmadio_link_power_rx{interface="cap5"} number
fmadio_link_power_rx{interface="cap5"} 0
# fmadio_link_up{interface="cap6"} boolean
fmadio_link_up{interface="cap6"} 0
# fmadio_link_power_rx{interface="cap6"} number
fmadio_link_power_rx{interface="cap6"} 0
# fmadio_link_up{interface="cap7"} boolean
fmadio_link_up{interface="cap7"} 0
# fmadio_link_power_rx{interface="cap7"} number
fmadio_link_power_rx{interface="cap7"} 0
# fmadio_link_man0_dbyte_rx number
fmadio_link_man0_dbyte_rx 0
# fmadio_link_man0_dbyte_tx number
fmadio_link_man0_dbyte_tx 0
# fmadio_link_up{interface="man0"} boolean
fmadio_link_up{interface="man0"} 1
# fmadio_link_man10_dbyte_rx number
fmadio_link_man10_dbyte_rx 0
# fmadio_link_man10_dbyte_tx number
fmadio_link_man10_dbyte_tx 0
# fmadio_link_up{interface="man10"} boolean
fmadio_link_up{interface="man10"} 1
# fmadio_link_man11_dbyte_rx number
fmadio_link_man11_dbyte_rx 0
# fmadio_link_man11_dbyte_tx number
fmadio_link_man11_dbyte_tx 0
# fmadio_link_man1_dbyte_rx number
fmadio_link_man1_dbyte_rx 0
# fmadio_link_man1_dbyte_tx number
fmadio_link_man1_dbyte_tx 0
# fmadio_link_module string
# fmadio_link_phy0_dbyte_rx number
fmadio_link_phy0_dbyte_rx 0
# fmadio_link_phy0_dbyte_tx number
fmadio_link_phy0_dbyte_tx 0
# fmadio_link_phy10_dbyte_rx number
fmadio_link_phy10_dbyte_rx 0
# fmadio_link_phy10_dbyte_tx number
fmadio_link_phy10_dbyte_tx 0
# fmadio_link_phy11_dbyte_rx number
fmadio_link_phy11_dbyte_rx 0
# fmadio_link_phy11_dbyte_tx number
fmadio_link_phy11_dbyte_tx 0
# fmadio_link_phy1_dbyte_rx number
fmadio_link_phy1_dbyte_rx 0
# fmadio_link_phy1_dbyte_tx number
fmadio_link_phy1_dbyte_tx 0
# fmadio_link_subsystem string
# fmadio_link_timestamp number
# fmadio_link_ver string
# fmadio_power_status{psu="0"} boolean
fmadio_power_status{psu="0"} 0
# fmadio_power_status{psu="1"} boolean
fmadio_power_status{psu="1"} 1
# fmadio_power_watt number
fmadio_power_watt 200
# fmadio_power_module string
# fmadio_power_subsystem string
# fmadio_power_timestamp number
# fmadio_power_ver string
# fmadio_fan_system{interface="SYS0"} number
fmadio_fan_system{interface="SYS0"} 13500
# fmadio_fan_system{interface="SYS1"} number
fmadio_fan_system{interface="SYS1"} 33600
# fmadio_fan_system{interface="SYS2"} number
fmadio_fan_system{interface="SYS2"} 33450
# fmadio_fan_system{interface="SYS3"} number
fmadio_fan_system{interface="SYS3"} 13800
# fmadio_fan_system{interface="SYS4"} number
fmadio_fan_system{interface="SYS4"} 13650
# fmadio_fan_system{interface="SYS5"} number
fmadio_fan_system{interface="SYS5"} 13800
# fmadio_fan_system{interface="SYS6"} number
fmadio_fan_system{interface="SYS6"} 13800
# fmadio_fan_system{interface="SYS7"} number
fmadio_fan_system{interface="SYS7"} 13800
# fmadio_fan_module string
# fmadio_fan_subsystem string
# fmadio_fan_timestamp number
# fmadio_fan_ver string

Custom Metrics

The system has the option to add user defined custom metrics, which ends up in the same /metrics file that Prometheus periodically scraps.

The following steps will add a single counter, there are no theoertical limits to the number of counters that can be added.

Step 1) Confirm current metrics file

Logging into the system using the loopback interface 127.0.0.1 confirm the current metrics file is active. For example check the metric

fmadio_counter_updateTS

This is the timestamp the counters were last  checked. Command is shown below.

curl -s http://127.0.0.1/metrics | grep fmadio_counter_updateTS

Example output. NOTE: the number is constantly changing

fmadio@fmadio200v4-636:~$ curl -s http://127.0.0.1/metrics | grep fmadio_counter_updateTS
# fmadio_counter_updateTS number
fmadio_counter_updateTS 1755141989247150080

fmadio@fmadio200v4-636:~$ curl -s http://127.0.0.1/metrics | grep fmadio_counter_updateTS
# fmadio_counter_updateTS number
fmadio_counter_updateTS 1755141990304825088

fmadio@fmadio200v4-636:~$ curl -s http://127.0.0.1/metrics | grep fmadio_counter_updateTS
# fmadio_counter_updateTS number
fmadio_counter_updateTS 1755141991359013120
fmadio@fmadio200v4-636:~$

Step 2) Create a new counter

Create a new counter in the path

/opt/fmadio/status/counter/

For example creating the following file

/opt/fmadio/status/counter/test1

With the files contents shown below (single numeric number)

fmadio@fmadio200v4-636:~$ cat /opt/fmadio/status/counter/test1
1234567
fmadio@fmadio200v4-636:~$

The value “1234567” is the numeric value that Prometheus will use.

Step 3) Check new metric

Check the new metrics returns the correct counter value

curl -s http://127.0.0.1/metrics | grep fmadio_counter_test1

Example output below

fmadio@fmadio200v4-636:~$ curl -s http://127.0.0.1/metrics | grep fmadio_counter_test1
# fmadio_counter_test1 number
fmadio_counter_test1 1234567
fmadio@fmadio200v4-636:~$

At this point the counter

fmadio_counter_test1

Is visible within the Prometheus infrastructure

Step 4) Additional Information (optional)

A counter name by itself may be less useful, having additional parameters to identify what it is measuring can be quite helpful.

This can be done by adding an additional line after the numerical value

prometheus: { market = "nyse", feed = "A" }

The fmadio_counter_test1 example file looks like below

fmadio@fmadio200v4-636:~$ cat /opt/fmadio/status/counter/test1
1234567
prometheus: { market = "nyse", feed = "A" }
fmadio@fmadio200v4-636:~$

Confirming the value in the metrics file

curl -s http://127.0.0.1/metrics | grep fmadio_counter_test1

With expected output shown below.

fmadio@fmadio200v4-636:~$ curl -s http://127.0.0.1/metrics | grep fmadio_counter_test1
# fmadio_counter_test1 { market = "nyse", feed = "A" } number
fmadio_counter_test1 { market = "nyse", feed = "A" } 1234567
fmadio@fmadio200v4-636:~$

Step 5) Finished

Additional notes:

• all values in /opt/fmadio/status/coutner  are NOT persistent and will be lost on system reboot.

• Only numeric values are supported with Prometheus, strings are not supported.