Monitoring

Monitoring the status of the Push PCAP is critical, the following documents the different metrics that are available.

Status

These are high level verification checks, useful for quick at a glance status of the push in operation

Aggregated Throughput Summary

This is a heatmap showing each push target (in this case 16 Multicast groups on OPRA). It gives a high level overview of the throughput across all the Push PCAP Targets.

Aggregated Verification Status

FMADIO Push PCAP has built in multi level status checks for each PCAP generated, this summary shows all the push targets across time. The dashboard should be entirely green, if there is any RED displayed it indicates which Push PCAP Target and what time, enabling easy identification of a problem.

This can also be used for automatic alerting

TODO

Target Status

Individual push targets can be monitored when

Per Target Status

When a problem with a specific Target has been identified, the Per Target Status narrows down the cause of the Status failure

• Status - aggregated status

  Metric	Description
  Status	Aggregated status one entry per PCAP per Push Target
  SizeMatch	Confirms the Local PCAP and the Remote PCAP file sizes match
  RemoteStatus	Confirms the Remote PCAP file has been created

Bytes Per PCAP

Monitors the total number of Bytes in per PCAP per Target over time. Useful metric to monitor the size of each Push PCAP Target.

Latency

While Push PCAP is not ultra low latency solution, monitoring the latency from the capture port to the remote Stroage bucket is important. The metrics below provide this information

Latency (Capture → Buffer)

Latency (Capture → Local)

TODO

Latency ( Local → Remote)

TODO

Latency (Capture → Remote)

TODO

Data rate

Capture Data rate

This is the data rate at the capture interface, e.g. the wire data rate for the selected Push target

NOTE: this is per push target

Local Data Rate

The data rate when writing PCAP files to the local disk. Multiple factors effect this, CPU load, IO load, Compression level

NOTE: this is per push target

Remote Data Rate

The data rate when copying the PCAP files to the remote storage

NOTE: this is per push target

TODO

Market Data Gap Detection

One of the primary usecases of FMADIO is capturing market data. The Builtin MDGap functionality provides additional confidence once PCAPs are generated they are complete