Parquet Schema
FMADIO MDGap analysis writes data into the Apache Parquet file format. This enables quick and easy searching for specific data using the duckdb command line.
MDGap Summary
This contains summary information of each PCAP file processed, its considered a high level enabling to locate specific PCAP files that are showing problems.
Field | Type | Unit | Description |
|---|---|---|---|
ts_pcap_min | Int64 | Epoch NS | The minimum PCAP timestamp found in the PCAP |
ts_pcap_max | Int64 | Epoch NS | The maximum PCAP timestamp found in the PCAP |
source | String | Filename | Source PCAP file this was generated from |
vlan | Int32 | VLAN ID (0 - for not present) | |
mac_src | String | MAC Address | Source MAC Address |
proto | String | Exchange Protocol used to decode the payload | |
endpoint | String | Exchange Line the data was decoded on | |
mcgroup | String | IPv4 | Multicast address including destination port number |
port | String | Port | Destination UDP Port number |
gap_cnt | Int64 | Total number of sequence number discontinuities | |
drop_cnt | Int64 | Total number of missing Sequence Numbers | |
dup_cnt | Int64 | Total number of Duplicate Sequence Numbers | |
msg_cnt | Int64 | Total number of messages | |
pkt_cnt | Int64 | Total number of Packets | |
pkt_byte | Int64 | Byte | Total number of bytes in this stream |
latency_exch_min | Int64 | Nanosec | Minimum Exchange Latency |
latency_exch_max | Int64 | Nanosec | Maximum Exchange Latency |
latency_exch_mean | Int64 | Nanosec | Mean Exchange Latency 0 - N/A |
latency_pb_min | Int64 | Nanosec | Minimum Packet Broker Latency 0 - N/A |
latency_pb_max | Int64 | Nanosec | Maximum Packet Broker Latency |
latency_pb_mean | Int64 | Nanosec | Mean Packet Broker Latency |
Exchange Latency
The latency_exch_* fields calculate the time between whats inside the business message, e.g. the exchange inserted a timestamp vs the FMADIO PCAP Capture Timestamp.
Effective its
FMADIO Capture Timestamp - Exchange Timestamp = Latency ExchangeOnly available for Market Data protocols that have exchange timestamps
Packet Broker Latency
The latency_pb_* Packet broker latency uses the Arista 7130 Footer timestamp format to compare again the FMADIO Captured timestamp.
This is a great way to monitor timekeeping drift on the Arista 7130 device, e.g. if the 7130 is being clock disciplined correctly.
Effectively
FMADIO Capture Timestamp - Arista 7130 Ingress Timestamp = Packet Broker LatencyOnly available for data with appropriate Arista 7130 timestamp footer data.
CLI Reference
The following is a reference for command line options for using the tool directly.
fmadio@fmadio200v4-636:/mnt/store1/pcap/20251027$ pcap_mdgap --help
FMADIO Market Data Gap detection (https://fmad.io)
pcap_mdgap 1.26.6 (c0f5f92) built Jul 3 2026 02:25:51
NOTE: Input PCAP`s are always read from STDIN
--version / -V : print pcap_mdgap version
--log-gap : log each and every gap (default false)
--log-reset : log each and every reset (default false)
--streams-file <file> : load and save streams state to file
--state-seq <N> : state sequence number for streams-file (required with --streams-file)
--json : print the stream statistics as JSON rows to stdout
--parquet-summary <file> : write high level summary info into parquet file
--parquet-gap <file> : write all gaps into parquet file (independently of --log-gap)
--parquet-seqno <file> : write all sequence numbers into parquet file
--table : print the stream statistics as a table to stdout
--table-time-unit <ns|us> : sets the time resolution in the table (--table) views: us, ns (default us)
--csv-seqnos <file> : write all sequence numbers into file
--sort <column> : sort table view by column: macsrc, dstport, mcgroup, gapcnt, dupcnt, msgcnt, pktcnt, exchmean (default mcgroup)
--filename : sets the PCAPFile field in the JSON output to the given filename
--disable-microburst : disable microburst detection (default enabled)
--list-endpoints : list all IP:PORT endpoints used for auto detection
--list-protocols : list all the supported protocols
--timezone-utc : use UTC timezone, default is local timezone
--show-undetected : show streams where the protocol was not detected
--arista7130 : MetaMako/Arista 7130 footer enabled
Protocols:
(Defaults to automatic detection per stream if no flag is provided)
--proto-siac-opra-v6 : decode data as SIAC OPRA v6
--proto-cme-mdp3-sbe-v1 : decode data as CME MDP 3 SBE v1
--proto-nasdaq-totalview-itch-v5 : decode data as NASDAQ TOTALVIEW ITCH v5
--proto-set-itch : decode data as SET ITCH
--proto-nyse-integrated-pillar : decode data as NYSE INTEGRATED PILLAR
--proto-sbe-optiq-euronext : decode data as EURONEXT OPTIQ SBE
--proto-siac-cts-cta : decode data as SIAC CTS CTA
--proto-siac-cqs-cta : decode data as SIAC CQS CTA
--proto-eurex-eobi-t7 : decode data as EUREX EOBI T7 V14
--proto-tmx-quantumfeed-xmt : decode data as TMX QUANTUMFEED XMT
--proto-iex-deep : decode data as IEX DEEP
--proto-cboe-bzx-pitch : dcoder data as CBOE BZX PITCH
--proto-bats-edgx-pitch : decode data as BATS EDGX PITCH
--proto-cboe-cfe-top : decode data as CBOE CFE TOP
--proto-memx-memoir-depth : decode data as MEMX MEMOIR DEPTH
--proto-ice-impact : decode data as ICE iMpact MDF
--proto-hkex-omd : decode data as HKEX OMD (OMD-C / OMD-D)
--proto-ose-itch : decode data as OSE ITCH
--proto-tse-flex : decode data as TSE FLEX Standard/Full
--proto-tse-flex-mbo : decode data as TSE FLEX MBO
--proto-jnx-itch : decode data as JNX ITCH
--proto-miax-pearl-dom-mach : decode data as MIAX PEARL DOM MACH
--proto-miax-pearl-tom-mach : decode data as MIAX PEARL TOM MACH
--proto-iqx : decode data as IQX
--proto-bmf-sbe : decode data as BMF SBE
--proto-rtp : decode data as RTP
--proto-custom <proto-def.txt> : decode data as it is defined in proto-def.txt
--proto-perf-test : no decoder (internal usage)—V / —version
Outputs the current version
Example Output
fmadio@fmadio200v4-636:~$ pcap_mdgap -V
pcap_mdgap 1.26.6 (c0f5f92) built Jul 3 2026 02:25:51
fmadio@fmadio200v4-636:~$—help
Prints the current help listing all the command line options