Following is a step by step guide on how to setup and run the FMADIO LXC Market Data Gap container.
1) Download Market2JSON LXC container
Start by downloading the latest LXC container from
For example downloading using curl
Copy fmadio@fmadio100v2-228U:/tmp$ curl -O -k https://firmware.fmad.io/download/container/https://firmware.fmad.io/download/container/market2json-202309221505.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 267M 100 267M 0 0 490k 2) untar the lxc into /opt/fmadio/lxc/
Unpack the container in the /opt/fmadio/lxc dir, this requires root
Copy fmadio@fmadio100v2-228U:/mnt/store0/lxc/lib/lxc$ sudo mv market2json-202309221505.tar.gz .
fmadio@fmadio100v2-228U:/mnt/store0/lxc/lib/lxc$ sudo tar xf market2json-202309221505.tar.gz
2) Configures the containers IP Address
This does basic network configuration of the Marke2JSON LXC. By default its configured for a bridged address running on the same physical interface as man0 of the FMADIO Host system.
e.g. it needs a new IP address on the same subnet as the FMADIO Host.
Copy cd /opt/fmadio/lxc/market2json-xxxx/
sudo su
./install.lua NOTE: this must be run as root
Copy root@fmadio100v2-228U:/mnt/store0/lxc/lib/lxc/market2json-202309221505# ./install.lua
fmad fmadlua Oct 18 2022 (/opt/fmadio/bin/fmadiolua ./install.lua )
mkdir /mnt/store0/log/lxc/market2json -p
touch /opt/fmadio/queue/lxc_ring_market_cme
touch /opt/fmadio/queue/lxc_ring_market_eurex
touch /opt/fmadio/queue/lxc_ring_market_nasdaq
touch /opt/fmadio/queue/lxc_ring_market_siac_cqs
touch /opt/fmadio/queue/lxc_ring_market_siac_cts
touch /opt/fmadio/queue/lxc_ring_market_siac_opra
Container IP Address? (e.g. 192.168.1.100 )
192.168.2.177
Container Netmask? (e.g. 24 for 255.255.255.0 )
24
Container Gateway? (e.g. 192.168.1.1 )
192.168.2.1
Container DNS? (e.g. 192.168.1.1 )
192.168.2.1
----------------------
IP : 192.168.2.177
CIDR : 24
NETMASK: 255.255.255.0
GW : 192.168.2.1
DNS : 192.168.2.1
.
.
.
echo fmadio100v2-228U-market2jso n > rootfs/etc/hostname
done 14.834738Sec 0.247246Min
root@fmadio100v2-228U:/mnt/store0/lxc/lib/lxc/market2json-20221014# NOTE: This setups all the internal LXC rings, there is no need to manually configure the rings
3) Check the status of the FMADIO lxc
using the utility
Check the status of all the lxc,
Example output shown below
Copy Sat Sep 23 03:03:03 2023] > show lxc status
[Sat Sep 23 03:03:03 2023] Enable : true
[Sat Sep 23 03:03:03 2023] RingCnt : 1
[Sat Sep 23 03:03:03 2023]
[Sat Sep 23 03:03:04 2023] Name OnBoot Install State Desc
[Sat Sep 23 03:03:04 2023] ---------------------------------------------------------------------------------------------------------------------------------
[Sat Sep 23 03:03:04 2023] market2json-202309221636 false yes STOPPED
[Sat Sep 23 03:03:04 2023] ---------------------------------------------------------------------------------------------------------------------------------
[Sat Sep 23 03:03:04 2023] >
The container should be listed above. To automatically start the container at boot run
Copy config lxc boot <container name> Example output shown below.
Copy [Sat Sep 23 03:04:22 2023] > config lxc boot market2json-202309221636
[Sat Sep 23 03:04:23 2023] Set container [market2json-202309221636] to boot on system start
[Sat Sep 23 03:04:24 2023] > show lxc status
[Sat Sep 23 03:04:25 2023] Enable : true
[Sat Sep 23 03:04:25 2023] RingCnt : 1
[Sat Sep 23 03:04:25 2023]
[Sat Sep 23 03:04:25 2023] Name OnBoot Install State Desc
[Sat Sep 23 03:04:25 2023] ---------------------------------------------------------------------------------------------------------------------------------
[Sat Sep 23 03:04:25 2023] market2json-202309221636 true yes STOPPED
[Sat Sep 23 03:04:25 2023] ---------------------------------------------------------------------------------------------------------------------------------
[Sat Sep 23 03:04:25 2023] > 4) Check status of FMADIO ring buffer
The Market2JSON container relies on the FMADIO Host to feed it data. It receives this data using the FMADIO LXC Ring structure.
The data sent from the host to the LXC Ring can be filtered using BPF filter and pushed automatically using the Push LXC configuration
The install script will automatically create all the rings it supports. To check they have been created , run the following command
Example output shown below. Note system may require a reboot for the ring to become active.
Copy [Sat Sep 23 03:06:59 2023] > show ring status
[Sat Sep 23 03:06:59 2023] Name : Path : Status : Pkt Put : Pkt Get : Pkt Queued : Desc
[Sat Sep 23 03:06:59 2023] -----------------------------------------+--------------------------------------------------------------+------------+------------------+------------------+------------+------------------------------------
[Sat Sep 23 03:06:59 2023] lxc_market2json_cboejp_prod_boe : /opt/fmadio/queue/lxc_market2json_cboejp_prod_boe : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_cboejp_prod_pitch : /opt/fmadio/queue/lxc_market2json_cboejp_prod_pitch : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_cboejp_uat_boe : /opt/fmadio/queue/lxc_market2json_cboejp_uat_boe : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_cboejp_uat_pitch : /opt/fmadio/queue/lxc_market2json_cboejp_uat_pitch : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_cme : /opt/fmadio/queue/lxc_market2json_cme : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_eurex : /opt/fmadio/queue/lxc_market2json_eurex : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_euronext_sbe : /opt/fmadio/queue/lxc_market2json_euronext_sbe : online : 189,943,420 : 189,943,420 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_lse_mitch : /opt/fmadio/queue/lxc_market2json_lse_mitch : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_nasdaq_itch : /opt/fmadio/queue/lxc_market2json_nasdaq_itch : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_siac_cqs : /opt/fmadio/queue/lxc_market2json_siac_cqs : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_siac_cts : /opt/fmadio/queue/lxc_market2json_siac_cts : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] lxc_market2json_siac_opra : /opt/fmadio/queue/lxc_market2json_siac_opra : online : 0 : 0 : 0 :
[Sat Sep 23 03:06:59 2023] -----------------------------------------+--------------------------------------------------------------+------------+------------------+------------------+------------+------------------------------------
[Sat Sep 23 03:06:59 2023] > This can be very helpful when monitoring how much traffic is going to each ring.
5) Start the Market2JSON container
Start the Market2JSON container either directly with the lxc-start command or use the fmadiocli
Copy config lxc start market2json-xxxx Example of sucessful operation is shown below
Copy [Sat Sep 23 03:08:28 2023] > config lxc start market2json-202309221636
[Sat Sep 23 03:08:29 2023] sudo lxc-start -n market2json-202309221636 --logfile /tmp/lxc_market2json-202309221636_1695438509485442048
[Sat Sep 23 03:08:30 2023]
[Sat Sep 23 03:08:30 2023] use the following on a shell to attach to the conatiners console
[Sat Sep 23 03:08:30 2023] sudo lxc-attach -n market2json-202309221636
[Sat Sep 23 03:08:30 2023]
[Sat Sep 23 03:08:30 2023] > If an error is shown, its likely the ring names or ip address are misconfigured. Please contact support@fmad.io for further assitance
6) Attach to the Market2JSON container
Attach to the container using the command line
Copy sudo lxc-attach -n market2json-xxxxx Example shown below
Copy fmadio@fmadio100v2-228U:/opt/fmadio/lxc$ sudo lxc-attach -n market2json-202309221636
root@fmadio100v2-228U-market2json:/# Once inside the container check the status of the initialization
Copy systemctl status fmadio-firstboot Example output shown below, indiciates its still active. Please wait a few minutes for that setup to complete.
After a few minutes the firstboot status will look like below, indiciating initialization is complete.
Once system has started a Grafana login page is presented at port 3000 of the assigned IP address.
Username: admin Password -> contact support@fmad.io
7) Use systemctl to enable/start specific protocols to use
Go to the directory
And list all the FMADIO services provided
Copy root@fmadio100v2-228U-market2json:/etc/systemd/system# ls -al fmadio*
-rw-r--r-- 1 root root 335 Sep 22 16:38 fmadio-firstboot.service
-rw-r--r-- 1 root root 539 Sep 23 03:10 fmadio-market2json-decode-cboejp_prod_boe_v1.service
-rw-r--r-- 1 root root 554 Sep 23 03:10 fmadio-market2json-decode-cboejp_prod_pitch_v1.service
-rw-r--r-- 1 root root 534 Sep 23 03:10 fmadio-market2json-decode-cboejp_uat_boe_v1.service
-rw-r--r-- 1 root root 546 Sep 23 03:10 fmadio-market2json-decode-cboejp_uat_pitch_v1.service
-rw-r--r-- 1 root root 538 Sep 23 03:10 fmadio-market2json-decode-euronext_v4.13.service
-rw-r--r-- 1 root root 496 Sep 23 03:10 fmadio-market2json-gap-cme_mdp3_v1.service
-rw-r--r-- 1 root root 505 Sep 23 03:10 fmadio-market2json-gap-eurex_eobi_v9.service
-rw-r--r-- 1 root root 535 Sep 23 03:10 fmadio-market2json-gap-euronext_v4.13.service
-rw-r--r-- 1 root root 514 Sep 23 03:10 fmadio-market2json-gap-lse_mitch_v11.service
-rw-r--r-- 1 root root 523 Sep 23 03:10 fmadio-market2json-gap-nasdaq_itch_v5.service
-rw-r--r-- 1 root root 505 Sep 23 03:10 fmadio-market2json-gap-siac_cqs_v1.service
-rw-r--r-- 1 root root 505 Sep 23 03:10 fmadio-market2json-gap-siac_cts_v1.service
-rw-r--r-- 1 root root 510 Sep 23 03:10 fmadio-market2json-gap-siac_opra_v4.service
root@fmadio100v2-228U-market2json:/etc/systemd/system#
Depending on which Market, enable the appropriate service
In this example we are using Euronext, thus enabling the service is as below
Copy systemctl enable fmadio-market2json-gap-euronext_v4.13.service Then start the service
Copy systemctl start fmadio-market2json-gap-euronext_v4.13.service Check the status
Copy systemctl status fmadio-market2json-gap-euronext_v4.13.service If all is configured correctly, the output looks like the following
Show the status of all decoders
Copy root@mdgap-20220927:/etc/systemd/system# systemctl status fmadio-*
* fmadio-mdgap-cme_mdp3_v1.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-cme_mdp3_v1.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-eurex_eobi_v9.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-eurex_eobi_v9.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-nasdaq_itch_v5.service - FMADIO Market Data Gap Detection (Nasdaq.Equities.TotalView.Itch.v5.0.h)
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-nasdaq_itch_v5.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-siac_cqs_v1.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-siac_cqs_v1.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-siac_cts_v1.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-siac_cts_v1.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap-siac_opra_v4.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap-siac_opra_v4.service; disabled; vendor preset: disabled)
Active: inactive (dead)
* fmadio-mdgap.service - FMADIO Market Data Gap Detection
Loaded: loaded (/etc/systemd/system/fmadio-mdgap.service; disabled; vendor preset: disabled)
Active: inactive (dead)
root@mdgap-20220927:/etc/systemd/system#
8) Test using a manual stream_cat
For testing, run stream cat manually pushing data to the LXC. In this case we have uploaded a historical PCAP into the capture system prior, see our pcap upload documentation on how to upload data to the system.
https://docs.fmad.io/fmadio-documentation/data-download/pcap-upload
Copy sudo stream_cat -v --ring /opt/fmadio/queue/lxc_market2json_euronext_sbe --ring-filter-bpf /opt/fmadio/queue/lxc_market2json_euronext_sbe "vlan 123 and net 224.0.1.0/24" euronext_20230701_0028 Example output
Copy fmadio@fmadio100v2-228U:/mnt/remote0/pcap/market/euronext$ sudo stream_cat -v --ring /opt/fmadio/queue/lxc_ring_mdgap_euronext --ring-filter-bpf /opt/fmadio/queue/lxc_ring_mdgap_euronext "vlan 123" euronext_20230701_0028
Create FMAD Ring:0 [/opt/fmadio/queue/lxc_ring_mdgap_euronext]
RING[/opt/fmadio/queue/lxc_ring_mdgap_euronext ] Size : 12595200 16777216
RING[/opt/fmadio/queue/lxc_ring_mdgap_euronext ] Version: 100 100
RING[/opt/fmadio/queue/lxc_ring_mdgap_euronext ] Put:0 0 0x7fe8b8ac1000
RING[/opt/fmadio/queue/lxc_ring_mdgap_euronext ] Get:0 0 0x7fe8b8ac2000
RING[/opt/fmadio/queue/lxc_ring_mdgap_euronext ] thread:0
RING[/opt/fmadio/queue/lxc_ring_mdgap_euronext ] worker thread start
0M Offset: 0GB ChunkID:114598304 TS:00:00:00.000.000.000 | Pending 944 MB 0.000Gbps 0.000Mpps CPUIdle:0.000 CPUFetch:0.698 CPUSend:0.000
0M Offset: 0GB ChunkID:114598321 TS:00:00:00.000.000.000 | Pending 940 MB 0.018Gbps 0.018Mpps CPUIdle:0.000 CPUFetch:0.003 CPUSend:0.000
0M Offset: 0GB ChunkID:114598337 TS:00:00:00.000.000.000 | Pending 936 MB 0.029Gbps 0.028Mpps CPUIdle:0.000 CPUFetch:0.001 CPUSend:0.000
0M Offset: 0GB ChunkID:114598345 TS:00:00:00.000.000.000 | Pending 934 MB 0.015Gbps 0.015Mpps CPUIdle:0.000 CPUFetch:0.001 CPUSend:0.000
0M Offset: 0GB ChunkID:114598361 TS:00:00:00.000.000.000 | Pending 930 MB 0.029Gbps 0.029Mpps CPUIdle:0.000 CPUFetch:0.001 CPUSend:0.000
0M Offset: 0GB ChunkID:114598371 TS:00:00:00.000.000.000 | Pending 927 MB 0.018Gbps 0.018Mpps CPUIdle:0.000 CPUFetch:0.001 CPUSend:0.000
0M Offset: 0GB ChunkID:114598379 TS:00:00:00.000.000.000 | Pending 925 MB 0.014Gbps 0.015Mpps CPUIdle:0.000 CPUFetch:0.000 CPUSend:0.000
0M Offset: 0GB ChunkID:114598387 TS:00:00:00.000.000.000 | Pending 923 MB 0.015Gbps 0.015Mpps CPUIdle:0.000 CPUFetch:0.000 CPUSend:0.000
0M Offset: 0GB ChunkID:114598396 TS:00:00:00.000.000.000 | Pending 921 MB 0.017Gbps 0.017Mpps CPUIdle:0.000 CPUFetch:0.001 CPUSend:0.000
.
.
. Further filtering may be required, e.g. for a specific multicast port. For example Euronext filtering on port 10135 for a specific symbol
Copy sudo stream_cat -v --ring /opt/fmadio/queue/lxc_market2json_euronext_sbe --ring-filter-bpf /opt/fmadio/queue/lxc_market2json_euronext_sbe "vlan 123 and udp and port 10135" euronext_20230701_0028 The results can be observed in the container running
Example output, the TotalPkt count increasing
Copy Jul 01 04:40:11 fmadio100v2-228U-mdgap fmadio-mdgap.lua[1090]: status 0.299GB 2.108M 0.000Mbps 0.000Mpps 0.000Mmps Gaps: 12737 Drops:7538641014471
Jul 01 04:40:12 fmadio100v2-228U-mdgap fmadio-mdgap.lua[1087]: {"module":"market-data-gap","subsystem":"monitor" ,"timestamp":1688186411,"uptime_min": 13.48,"IsUpPCAP":true,"IsUpMDGAP":true,"Protocol":"./omi/euronext/Euronext.Optiq.MarketDataGateway.Sbe.v4.13.h"}
Jul 01 04:40:13 fmadio100v2-228U-mdgap fmadio-mdgap.lua[1087]: {"module":"market-data-gap","subsystem":"monitor" ,"timestamp":1688186412,"uptime_min": 13.50,"IsUpPCAP":true,"IsUpMDGAP":true,"Protocol":"./omi/euronext/Euronext.Optiq.MarketDataGateway.Sbe.v4.13.h"}
Jul 01 04:40:13 fmadio100v2-228U-mdgap fmadio[35342]: {"module":"market-data-gap","subsystem":"status" ,"timestamp":1688186413779,"PCAPtimestamp":1683626529293444864,"PCAPTime":"2023.05.09_10:02:09.293.444.880","Location":"","Protocol":"./omi/euronext/Euronext.Optiq.MarketDataGateway.Sbe.v4.13.h","TotalByte":298782803,"TotalPkt":2107647,"TotalGap":12737,"TotalDrop":7538641014471,"TotalReset":1158618,"MarketGap_bps":0,"MarketGap_pps":0,"MarketGap_mps":0,"MarketGap_Lag":4559884485984256,"LatencyEMA":9919832,"LatencyMin":-9223372036854775808,"LatencyMax":0}
Jul 01 04:40:13 fmadio100v2-228U-mdgap fmadio-mdgap.lua[1090]: status 0.299GB 2.108M 0.000Mbps 0.000Mpps 0.000Mmps Gaps: 12737 Drops:7538641014471
Jul 01 04:40:14 fmadio100v2-228U-mdgap fmadio-mdgap.lua[1087]: {"module":"market-data-gap","subsystem":"monitor" ,"timestamp":1688186413,"uptime_min": 13.52,"IsUpPCAP":true,"IsUpMDGAP":true,"Protocol":"./omi/euronext/Euronext.Optiq.MarketDataGateway.Sbe.v4.13.h"}
Jul 01 04:40:15 fmadio100v2-228U-mdgap fmadio-mdgap.lua[1087]: {"module":"market-data-gap","subsystem":"monitor" ,"timestamp":1688186414,"uptime_min": 13.53,"IsUpPCAP":true,"IsUpMDGAP":true,"Protocol":"./omi/euronext/Euronext.Optiq.MarketDataGateway.Sbe.v4.13.h"} Jul 01 04:40:15 fmadio100v2-228U-mdgap fmadio[35364]: {"module":"market-data-gap","subsystem":"status" ,"timestamp":1688186415787,"PCAPtimestamp":1683626529293444864,"PCAPTime":"2023.05.09_10:02:09.293.444.880","Location":"","Protocol":"./omi/euronext/Euronext.Optiq.MarketDataGateway.Sbe.v4.13.h","TotalByte":298782803,"TotalPkt":2107647,"TotalGap":12737,"TotalDrop":7538641014471,"TotalReset":1158618,"MarketGap_bps":0,"MarketGap_pps":0,"MarketGap_mps":0,"MarketGap_Lag":4559886494150144,"LatencyEMA":9919832,"LatencyMin":-9223372036854775
808,"LatencyMax":0}
Jul 01 04:40:15 fmadio100v2-228U-mdgap fmadio-mdgap.lua[1090]: status 0.299GB 2.108M 0.000Mbps 0.000Mpps 0.000Mmps Gaps: 12737 Drops:7538641014471
Or by viewing on the Grafana GUI as shown below
Assuming the above is working correctly, next is to setup automatic pushing of the data
9) Automatic Push to the Container
To configure automatic push of the data to the container we use FMADIO Push LXC commands. Reference documentation is located here
https://docs.fmad.io/fmadio-documentation/cli-reference/fmadiocli#automatic-push-to-lxc-container
Start by checking the status of the current LXC Push items
Example below shows a single lxc push for IDS Suricata, there is no mdgap pushes currently configured
Copy [Sat Jul 1 01:01:25 2023] > show push lxc status
[Sat Jul 1 01:01:25 2023]
[Sat Jul 1 01:01:25 2023] Ring name : Enable : From : Description : Filter Frame : Filter BPF
[Sat Jul 1 01:01:25 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:01:25 2023] /opt/fmadio/queue/lxc_ring_suricata : false : now : : :
[Sat Jul 1 01:01:25 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:01:25 2023] > Add an automatic push using the command
Copy config push lxc add lxc_market2json_euronext_sbe Example output looks like the following
Copy [Sat Jul 1 01:03:53 2023] > config push lxc add lxc_market2json_euronext_sbe
[Sat Jul 1 01:03:53 2023] New Push LXC target [/opt/fmadio/queue/lxc_market2json_euronext_sbe]
[Sat Jul 1 01:03:53 2023] > By default the target is DISABLED can be viewed using the show push lxc status command
Copy [Sat Jul 1 01:06:47 2023] > show push lxc status
[Sat Jul 1 01:06:49 2023]
[Sat Jul 1 01:06:49 2023] Ring name : Enable : From : Description : Filter Frame : Filter BPF
[Sat Jul 1 01:06:49 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:06:49 2023] /opt/fmadio/queue/lxc_ring_suricata : false : now : : :
[Sat Jul 1 01:06:49 2023] /opt/fmadio/queue/lxc_market2json_euronext_sbe : false : now : : :
[Sat Jul 1 01:06:49 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:06:49 2023] >
Next add an appropriate BPF filter, in this case we want to filter on a specific Multicast group and VLAN
Copy config push lxc filter-bpf lxc_market2json_euronext_sbe "vlan 123 and port 10135" This is shown in the "FilterBPF" part of the status display
Copy [Sat Jul 1 01:08:26 2023] > show push lxc status
[Sat Jul 1 01:08:27 2023] Ring name : Enable : From : Description : Filter Frame : Filter BPF
[Sat Jul 1 01:08:27 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:08:27 2023] /opt/fmadio/queue/lxc_ring_suricata : false : now : : :
[Sat Jul 1 01:08:27 2023] /opt/fmadio/queue/lxc_market2json_euronext_sbe : false : now : : : vlan 123 and port 10135
[Sat Jul 1 01:08:27 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:08:27 2023] >
Then Enable the push
Copy config push lxc enable lxc_market2json_euronext_sbe Example status, the mdgap_euronext is now shown as enabled.
Copy [Sat Jul 1 01:10:08 2023] > show push lxc status
[Sat Jul 1 01:10:09 2023] Ring name : Enable : From : Description : Filter Frame : Filter BPF
[Sat Jul 1 01:10:09 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:10:09 2023] /opt/fmadio/queue/lxc_ring_suricata : false : now : : :
[Sat Jul 1 01:10:09 2023] /opt/fmadio/queue/lxc_market2json_euronext_sbe : true : now : : : vlan 177 and port 10135
[Sat Jul 1 01:10:09 2023] ------------------------------------------------+--------+------------+--------------------------------+----------------------+-----------------------------------------------------------------
[Sat Jul 1 01:10:09 2023] > NOTE: In addition to this the Analytics schedule needs to have push_lxc enabled, shown below
10) Debug and Monitoring
The status of the gap detector queue can be monitoring using the fmadiocli
Example shown below. In this example we see the mdgap_euronext has 458K packets on both Put (producer) and Get (consumer) meaning both the automatic push and the MDGap application are generating and consuming the data
Copy [Sat Jul 1 01:13:42 2023] > show ring status
[Sat Jul 1 01:13:42 2023] Name : Path : Status : Pkt Put : Pkt Get : Pkt Queued : Desc
[Sat Jul 1 01:13:42 2023] -------------------------------+----------------------------------------------------+------------+------------------+------------------+------------+------------------------------------L
[Sat Jul 1 01:13:42 2023] lxc_market2json_euronext_sbe : /opt/fmadio/queue/lxc_market2json_euronext_sbe : online : 485,253 : 485,253 : 0 :
[Sat Jul 1 01:13:42 2023] -------------------------------+----------------------------------------------------+------------+------------------+------------------+------------+------------------------------------
[Sat Jul 1 01:13:42 2023] > If the counters are showing 0, it usually means there is a issue on the Producer side
If the its showing "full" / Pkt Get is 0 it means a problem on the Consumer side (MDGap)
Debug Logfiles
Host side
logfiles can be found in
Copy /mnt/store0/log/push_lxc* Market2JSON Container
logfiles are best using the system logfile
