Market Data Gap Detection
Last updated
Last updated
FMADIO Capture system allows al full market data gap detection to run on the capture system itself. This enables data loss checks on all market data links to ensure every single byte and bit of data has been correctly captured and stored.
The project is fully open source under an MIT license, we heavily utilize the Open Markets Initative codebase for formatting and parsing protocols
By default the gap detector runs in an LXC container, this container is built and downloaded from the following location
https://old.fmad.io/container.html
Once installed the system uses the LXC Push functionality of the system that writes (optionally) filtered packet data into an internal ring buffer. That packet ring buffer is passed to the LXC MDGap container, which in-turn pulls data off the ring and processes it.
A high level overview is shown below
Alerts are output to syslog allowing a range of options for monitoring and reporting infrastructure.
Example of the gap detector Alert output is as follows
in pretty json format style
