FMADIO Capture system allows al full market data gap detection to run on the capture system itself. This enables data loss checks on all market data links to ensure every single byte and bit of data has been correctly captured and stored.
The project is fully open source under an MIT license, we heavily utilize the Open Markets Initative codebase for formatting and parsing protocols
By default the gap detector runs in an LXC container, this container is built and downloaded from the following location
https://old.fmad.io/container.html
Once installed the system uses the LXC Push functionality of the system that writes (optionally) filtered packet data into an internal ring buffer. That packet ring buffer is passed to the LXC MDGap container, which in-turn pulls data off the ring and processes it.
A high level overview is shown below
Alerts are output to syslog allowing a range of options for monitoring and reporting infrastructure.
Example of the gap detector Alert output is as follows
in pretty json format style
Following is a step by step guide on how to setup and run the FMADIO LXC Market Data Gap container.
Start by downloading the latest LXC container from
For example downloading using curl
2) untar the lxc into /opt/fmadio/lxc/
Unpack the container in the /opt/fmadio/lxc dir, this requires root
This does basic network configuration of the Marke2JSON LXC. By default its configured for a bridged address running on the same physical interface as man0 of the FMADIO Host system.
e.g. it needs a new IP address on the same subnet as the FMADIO Host.
NOTE: this must be run as root
NOTE: This setups all the internal LXC rings, there is no need to manually configure the rings
using the utility
Check the status of all the lxc,
Example output shown below
The container should be listed above. To automatically start the container at boot run
Example output shown below.
The Market2JSON container relies on the FMADIO Host to feed it data. It receives this data using the FMADIO LXC Ring structure.
The data sent from the host to the LXC Ring can be filtered using BPF filter and pushed automatically using the Push LXC configuration
The install script will automatically create all the rings it supports. To check they have been created , run the following command
Example output shown below. Note system may require a reboot for the ring to become active.
This can be very helpful when monitoring how much traffic is going to each ring.
Start the Market2JSON container either directly with the lxc-start command or use the fmadiocli
Example of sucessful operation is shown below
If an error is shown, its likely the ring names or ip address are misconfigured. Please contact support@fmad.io for further assitance
Attach to the container using the command line
Example shown below
Once inside the container check the status of the initialization
Example output shown below, indiciates its still active. Please wait a few minutes for that setup to complete.
After a few minutes the firstboot status will look like below, indiciating initialization is complete.
Once system has started a Grafana login page is presented at port 3000 of the assigned IP address.
Username: admin Password -> contact support@fmad.io
Go to the directory
And list all the FMADIO services provided
Depending on which Market, enable the appropriate service
In this example we are using Euronext, thus enabling the service is as below
Then start the service
Check the status
If all is configured correctly, the output looks like the following
Show the status of all decoders
For testing, run stream cat manually pushing data to the LXC. In this case we have uploaded a historical PCAP into the capture system prior, see our pcap upload documentation on how to upload data to the system.
Example output
Further filtering may be required, e.g. for a specific multicast port. For example Euronext filtering on port 10135 for a specific symbol
The results can be observed in the container running
Example output, the TotalPkt count increasing
Or by viewing on the Grafana GUI as shown below
Assuming the above is working correctly, next is to setup automatic pushing of the data
To configure automatic push of the data to the container we use FMADIO Push LXC commands. Reference documentation is located here
Start by checking the status of the current LXC Push items
Example below shows a single lxc push for IDS Suricata, there is no mdgap pushes currently configured
Add an automatic push using the command
Example output looks like the following
By default the target is DISABLED can be viewed using the show push lxc status command
Next add an appropriate BPF filter, in this case we want to filter on a specific Multicast group and VLAN
This is shown in the "FilterBPF" part of the status display
Then Enable the push
Example status, the mdgap_euronext is now shown as enabled.
NOTE: In addition to this the Analytics schedule needs to have push_lxc enabled, shown below
The status of the gap detector queue can be monitoring using the fmadiocli
Example shown below. In this example we see the mdgap_euronext has 458K packets on both Put (producer) and Get (consumer) meaning both the automatic push and the MDGap application are generating and consuming the data
If the counters are showing 0, it usually means there is a issue on the Producer side
If the its showing "full" / Pkt Get is 0 it means a problem on the Consumer side (MDGap)
logfiles can be found in
logfiles are best using the system logfile
Some reference performance numbers. All using a 1 CPU for processing, for high throughputs multiple CPUs can be used, typically filtering by multicast group number
About 2.5Gbps per 1 CPU throughput
About 0.9Gbps per 1 CPU throughput
About 0.7Gbps per 1 CPU Throughput
SIAC CTS Feed 1.0 Gbps per 1 CPU throughput
SIAC OPRA Full A/B feed all running on a single 1 CPU. 0.4Gbps. Performance can be improved substantially by distributing the multicast groups to multiple CPUs
