Some reference performance numbers. All using a 1 CPU for processing, for high throughputs multiple CPUs can be used, typically filtering by multicast group number
CME
About 2.5Gbps per 1 CPU throughput
EUREX
About 0.9Gbps per 1 CPU throughput
NASDAQ
About 0.7Gbps per 1 CPU Throughput
SIAC CTS
SIAC CTS Feed 1.0 Gbps per 1 CPU throughput
SIAC OPRA
SIAC OPRA Full A/B feed all running on a single 1 CPU. 0.4Gbps. Performance can be improved substantially by distributing the multicast groups to multiple CPUs
Market Data Gap Detection
FMADIO Capture system allows al full market data gap detection to run on the capture system itself. This enables data loss checks on all market data links to ensure every single byte and bit of data has been correctly captured and stored.
The project is fully open source under an MIT license, we heavily utilize the Open Markets Initative codebase for formatting and parsing protocols
By default the gap detector runs in an LXC container, this container is built and downloaded from the following location
Once installed the system uses the LXC Push functionality of the system that writes (optionally) filtered packet data into an internal ring buffer. That packet ring buffer is passed to the LXC MDGap container, which in-turn pulls data off the ring and processes it.
A high level overview is shown below
Alerts are output to syslog allowing a range of options for monitoring and reporting infrastructure.
Example of the gap detector Alert output is as follows
Following is a step by step guide on how to setup and run the FMADIO LXC Market Data Gap container.
1) Download Market2JSON LXC container
Start by downloading the latest LXC container from
For example downloading using curl
fmadio@fmadio100v2-228U:/tmp$ curl -O -k https://firmware.fmad.io/download/container/https://firmware.fmad.io/download/container/market2json-202309221505.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 267M 100 267M 0 0 490k
2) untar the lxc into /opt/fmadio/lxc/
Unpack the container in the /opt/fmadio/lxc dir, this requires root
fmadio@fmadio100v2-228U:/mnt/store0/lxc/lib/lxc$ sudo mv market2json-202309221505.tar.gz .
fmadio@fmadio100v2-228U:/mnt/store0/lxc/lib/lxc$ sudo tar xf market2json-202309221505.tar.gz
2) Configures the containers IP Address
This does basic network configuration of the Marke2JSON LXC. By default its configured for a bridged address running on the same physical interface as man0 of the FMADIO Host system.
e.g. it needs a new IP address on the same subnet as the FMADIO Host.
cd /opt/fmadio/lxc/market2json-xxxx/
sudo su
./install.lua
systemctl status fmadio-market2json-gap-euronext_v4.13.service
If all is configured correctly, the output looks like the following
Show the status of all decoders
root@mdgap-20220927:/etc/systemd/system# systemctl status fmadio-*
* fmadio-mdgap-cme_mdp3_v1.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-cme_mdp3_v1.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-eurex_eobi_v9.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-eurex_eobi_v9.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-nasdaq_itch_v5.service - FMADIO Market Data Gap Detection (Nasdaq.Equities.TotalView.Itch.v5.0.h)
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-nasdaq_itch_v5.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-siac_cqs_v1.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-siac_cqs_v1.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-siac_cts_v1.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-siac_cts_v1.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-siac_opra_v4.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-siac_opra_v4.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap.service; disabled; vendor preset: disabled)
Active: inactive (dead)
root@mdgap-20220927:/etc/systemd/system#
8) Test using a manual stream_cat
For testing, run stream cat manually pushing data to the LXC. In this case we have uploaded a historical PCAP into the capture system prior, see our pcap upload documentation on how to upload data to the system.
Further filtering may be required, e.g. for a specific multicast port. For example Euronext filtering on port 10135 for a specific symbol
sudo stream_cat -v --ring /opt/fmadio/queue/lxc_market2json_euronext_sbe --ring-filter-bpf /opt/fmadio/queue/lxc_market2json_euronext_sbe "vlan 123 and udp and port 10135" euronext_20230701_0028
The results can be observed in the container running
Example status, the mdgap_euronext is now shown as enabled.
[Sat Jul 1 01:10:08 2023] > show push lxc status
[Sat Jul 1 01:10:09 2023] Ring name : Enable : From : Description : Filter Frame : Filter BPF
[Sat Jul 1 01:10:09 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:10:09 2023] /opt/fmadio/queue/lxc_ring_suricata : false : now : : :
[Sat Jul 1 01:10:09 2023] /opt/fmadio/queue/lxc_market2json_euronext_sbe : true : now : : : vlan 177 and port 10135
[Sat Jul 1 01:10:09 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:10:09 2023] >
NOTE: In addition to this the Analytics schedule needs to have push_lxc enabled, shown below
10) Debug and Monitoring
The status of the gap detector queue can be monitoring using the fmadiocli
show ring status
Example shown below. In this example we see the mdgap_euronext has 458K packets on both Put (producer) and Get (consumer) meaning both the automatic push and the MDGap application are generating and consuming the data
[Sat Jul 1 01:13:42 2023] > show ring status
[Sat Jul 1 01:13:42 2023] Name : Path : Status : Pkt Put : Pkt Get : Pkt Queued : Desc
[Sat Jul 1 01:13:42 2023] -------------------------------+----------------------------------------------------+------------+------------------+------------------+------------+------------------------------------L
[Sat Jul 1 01:13:42 2023] lxc_market2json_euronext_sbe : /opt/fmadio/queue/lxc_market2json_euronext_sbe : online : 485,253 : 485,253 : 0 :
[Sat Jul 1 01:13:42 2023] -------------------------------+----------------------------------------------------+------------+------------------+------------------+------------+------------------------------------
[Sat Jul 1 01:13:42 2023] >
If the counters are showing 0, it usually means there is a issue on the Producer side
If the its showing "full" / Pkt Get is 0 it means a problem on the Consumer side (MDGap)
