Monitoring

Monitoring the status of the Push PCAP is critical, the following documents the different metrics that are available.

Status

These are high level verification checks, useful for quick at a glance status of the push in operation

Aggregated Throughput Summary

This is a heatmap showing each push target (in this case 16 Multicast groups on OPRA). It gives a high level overview of the throughput across all the Push PCAP Targets.

Aggregated Verification Status

FMADIO Push PCAP has built in multi level status checks for each PCAP generated, this summary shows all the push targets across time. The dashboard should be entirely green, if there is any RED displayed it indicates which Push PCAP Target and what time, enabling easy identification of a problem.

This can also be used for automatic alerting

Target Status

Individual push targets can be monitored when

Per Target Status

When a problem with a specific Target has been identified, the Per Target Status narrows down the cause of the Status failure

• Status - aggregated status

  Metric	Description
  Status	Aggregated status one entry per PCAP per Push Target
  SizeMatch	Confirms the Local PCAP and the Remote PCAP file sizes match
  RemoteStatus	Confirms the Remote PCAP file has been created

Bytes Per PCAP

Monitors the total number of Bytes in per PCAP per Target over time. Useful metric to monitor the size of each Push PCAP Target.

Latency

While Push PCAP is not ultra low latency solution, monitoring the latency from the capture port to the remote Stroage bucket is important. The metrics below provide this information

Latency (Capture → Buffer)

Latency (Capture → Local)

Latency from the Capture Port to the PCAPs on the local disk.

Latency ( Local → Remote)

Latency from The Local Disk to the Remote storage bucket

Latency (Capture → Remote)

Total latency from Capture port to Remote device

Data rate

Capture Data rate

This is the data rate at the capture interface, e.g. the wire data rate for the selected Push target

NOTE: this is per push target

Local Data Rate

The data rate when writing PCAP files to the local disk. Multiple factors effect this, CPU load, IO load, Compression level

NOTE: this is per push target

Remote Data Rate

The data rate when copying the PCAP files to the remote storage

NOTE: this is per push target

Market Data Gap Detection

One of the primary usecases of FMADIO is capturing market data. The Builtin MDGap functionality provides additional confidence once PCAPs are generated they are complete

Reference Promethues

Push PCAP takes the capture systems stored data and splits into PCAP files and pushes them to some other location. The following stats are avalible in both JSON Syslog and Promethues

push_pcap_status

• for every PCAP file generated

This is the top level full status of an individual PCAP file.

push_pcap_local_bps

• for every PCAP file generated

This is the data rate for a specific PCAP write this to the local storage device.

push_pcap_local_bps = total bytes * 8 / total time

push_pcap_local_byte

• for every PCAP file generated

This is the total bytes for a PCAP write to the local storage device.

push_pcap_local_status

• for every PCAP file generated

Indicates the status of the specific PCAP file. Table below shows the meaning

push_pcap_local_time

• for every PCAP file generated

The number of seconds it took to create the local file.

NOTE: this is an integer value

push_pcap_localremote_size_match

• for every PCAP file generated

Indicates the status of local vs remote PCAP file comparision

push_pcap_remote_bps

• for every PCAP file generated

This is the data rate for a specific PCAP write this to the remote storage device.

push_pcap_remote_bps = total bytes * 8 / total time

push_pcap_remote_byte

• for every PCAP file generated

This is the total bytes for a PCAP written to the remote storage device.

push_pcap_remote_retry

• for every PCAP file generated

This is the number of times a remote copy has been retried. e.g. number of copy fails to the remote storage

push_pcap_remote_status

• for every PCAP file generated

Indicates the status of the specific PCAP file. Table below shows the meaning

push_pcap_remote_time

• for every PCAP file generated

The number of seconds it took to create and copy the PCAP from the local storage location to the remote storage location.

NOTE: this is an integer value