Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Hardware
FMADIO 100G Gen2 system provides lossless full 100Gbps 149Mpps line rate packet capture system in a compact 1U form factor.
100Gbps sustained 24/7 packet capture
200Gbps Burst Packet Capture
x2 QSFP28 Ports
SR4 / LR4 / CX4 and FEC support
200Gbps line rate packet generator
16-256TB RAID5 storage
Hardware Timestamping
Pre Capture Filtering
Inline Packet Capture
Packet Slicing
Simple easy to use HTML GUI
Full scriptable JSON REST API
Network Sustained Capture Speed:
100Gbps Line Data rate
148.88Mpps Packet Rate
Network Capture Ports
x2 QSFP28 100G
2x100G
2x40G
8x10G
4x25G
Packet Size
64B- 9200B
Management Interface
2 x 10G SFP+
2 x 40G QSFP+
4 x 10G QSFP+ (with breakout cable)
2 x 10/100/1000M RJ45
Hardware Time Stamp Resolution
3.2 ns
Clock Synchronization
PTPv2
NTP
PPS
Disks
10 x U.2 2.5" Disks
Storage Size
16TB - 156tB
Storage Redundancy
Full 100Gbps RAID5
Chassis Size
1U Rack muontable
482 mm x 787mm x 44mm
19" x 31" x 1.75"
Rack mount
31" Tool less rails
Cooling
x12 Hot swap 40mm fans
Power
Dual 850W Power Supply
Hot Swap Power Supply
(Operates with single PSU)
Weight
25.8KG
54lbs
There are 4 ways to connect to the appliance for initial configuration:
KVM – The appliance has a VGA port for a monitor and USB for keyboard.
Serial – The serial port can be used with a configuration of 115200, 8, None, 1.
SSH to the OS – There are 2 management ports preconfigured with IP addresses:
IPMITOOL to the IPMI/BMC/iLO/iDARC – The IPMI port is preconfigured with IP address:
To show the current network status use the fmadiocli utility, by typing it on the shell after logging in.
Example
To show the current network IP and Link status use the command
NOTE: system has no way to check the BMC link up/down status.
Once logged in, the first task is to set new IP addresses. Type fmadiocli to enter the configuration CLI, then use the following commands
To set the IPv4 Address
To set the IPv4 netmask
To set the IPv4 Gateway
To set the IPv4 DNS (optional)
Full documentation is at
Using IPMITOOL serial port to login
Configure the IPv4 man0 interface
Configure the IPv4 man0 netmask
Configure IPv4 man0 gateway
Configure the IPv4 man0 DNS (optional)
Errata related to FMADIO100v2 Capture System
Due to design decision of the FPGA clocking QSFP0 Capture port must be connected first or put in shutdown mode using the fmadiocli.
Recommended configuration is always connect QSFP0 port first
The reason is if the frontend of the capture FPGA uses the clock 322mhz clock that runs off the QSFP0 port FPGA GTY transceiver. If the port is left un-connected and not shutdown the system will try automatically link up in different modes. The constant attempted link up, transitioning between modes (eg FEC and non-FEC) requires resetting of the clock resulting in unstable behavior on Capture Port 1
Network port configuration can be achieved using a) the web interface, b) SSH command line interface(CLI). Using the Web interface is the easiest route, however in highly constrained network environments a pure CLI based configuration can be easier
WEB INTERFACE: NETWORK CONFIG
From the dashboard page, Start by selecting the configuration menu option from as shown below (highlighted in green).
Then edit the network configuration`s IP/Netmask/Gateway/DNS setting as shown in the image below. After each field has been edited the system automatically saves and updates the system setting (save button is not required). After completing the update, refresh the web page to confirm the new settings.
Select the tools menu from the top toolbar, as shown in the image below.
And finally select the Power Cycle / Reboot button to restart the system
Layout of the network ports is as follows. for FMADIO100G v2 1U Capture System (10G SFP+ management)
Layout of the network ports for FMADIO100G v2 1U Analytics System (40G QSFP+ Management)
Systems shipped Prior to April 2021, default 1G management port is "L2"
System shipped after April 2021 default 1G management port is "L1"
This was made as L1 port can bridge the IPMI/BMC port (single RJ45 connection for both Server and BMC), the L2 port can not bridged. Please contact support@fmad.io on how to swap.
The IPMI Port is used for out of band communication with the system. It allows power on/off and KVM capabilities. Highly recommend connecting this. Default IP address is 192.168.0.93/24
The Management ports can be 10G SFP+ or 40G QSFP+ depending on the system configuration. These can be run in standard, or link bonded / redundant setup.
FMADIO100Gv2 Capture systems,
Port Count
Port Speed
Interface
2
10G
SFP+
2
1G
SFP
FMADIO100G Gen2 Analytics Systems
Port Count
Port Speed
Interface
2
40Gbps
QSFP+
4
10Gbps
QSFP+ Breakout Cables
Capture ports can be configured in the following way
Port Count
Port Speed
Interface
2
100Gbps
QSFP28 (FEC or no FEC)
2
40Gbps
QSFP+
8
25Gbps
QSFP28 (Not released yet)
8
10Gbps
QSFP+ Breakout Cables
The PPS connector is a 1 Pulse Per Second time synchronization cable. It runs on a 3.3V trigger signal the interface is SMA Coaxial cable
Standard RJ45 Serial port connector
By default FMADIO devices capture ports operate without any MAC or IP information. It receives and records any and all ethernet traffic on the wire. Its essentially a black hole high speed data recorder.
However there are some situations where the Capture interfaces need an IP MAC address, this is for ERSPAN IP targets, and also having the capture ports directly join Mulitcast groups. The follow demonstrates how to setup IP MAC Address,
Using FMAIO DPI Engine we can filter out low bandwidth traffic such as ARP/ICMP requests without any effect on the 100Gbps / 149Mpps packet capture performance. As seen below using a few of the PreCapture filter rules and forwarding a few packets to our ARP/ICMP/IGMP software network stack running on the x86 Server. This allows full ARP and ICMP protocol support on the capture interfaces.
NOTE: Enabling this feature reduces the total number of Pre Capture filter rules, It requires
Depending on the capture port port configuration (2 ports or 8 ports), a maximum of 9 pre-capture filter rules may be used.
To Configure IP/MAC information for the capture ports. edit the files below, by adding 2 new sections ["cap0"] and ["cap1"] i the configuration file
Edit the file such that it look like the following, adding in your own MAC and IP addresses. There is no Netmask or Gateway associated with the capture ports.
Save the file and reboot the system.
The capture interfaces MAC/IP/IGMP is only enabled when there is an active capture. Please start a capture then try arping the interface first
If that is successful then check ICMP ping is functioning correctly also
Any problems please check the log file
It may provide more information on why ARP or ICMP is not responding correctly
Rule Number
Description
15
MAC Broadcast ARP Requests
14
Capture Port 0 MAC filter
13
Capture Port 1 MAC Filter
The performance of a capture system can be characterized in a number of different ways. We provide the following performance dimensions
This is the short time burst capture rate of the system. For the 100G Gen2 system this is burst capture rate that fills up the DDR buffer on the FMAD FPGA Capture card.
For the 100G Gen2 FMADIO Packet Capture system, all storage is on high speed NVMe SSDs, so the Burst Capture Speed is the same as the Sustained Capture Speed.
FMADIO 20G and 40G systems use a mixture of SSD and magnetic disk storage, so for those systems the Burst Capture Speed is higher than the Sustained Capture Speed.
We indicate this as the sustained capture rate, i.e. the capture rate that a system can sustain 24/7 without any packet loss. For systems with Magnetic storage, mixing capture with downloads effects the writeback to magnetic storage speed, this performance metric is Capture Only with no simultaneous/concurrent downloads.
Performance metric is assuming no bottlenecks on the egress (download client) what is the capture performance while simultaneously downloading.
The other metric is Download only speed. This metric is used to calculate the maximum rate data can be moved off the device over 10G or 40G ethernet.
Modifying the network configuration setting in a restricted Colocation environment can be far easier to achieve via the command line. The first step is SSH into the system, change to the specified directory and view the current network settings, as shown below.
Specifically:
interface name
Hardware Port
Description
man0
L1 1G RJ45
1G management port
bmc
IPMI 1G RJ45
1G Out of bands management port
man10
SFP/QSFP 10G/40G
High speed management port
Interface Name
IP Address
Netmask
Gateway
DNS
man0
192.168.0.95
255.255.255.0
192.168.0.1
192.168.0.1
bmc
192.168.0.93
255.255.255.0
192.168.0.1
man10
192.168.15.95
255.255.255.0
Example configuration is shown below
After editing the file, save it and confirm no syntax errors, running the following command
A successful output looks like the following
Reboot the system and confirm the IP settings are correct
There may be a requirement for management VLAN interfaces on the FMADIO packet capture system. These are simple to add using the standard linux infrastructure
Example is to create a VLAN 123 on the 10G man10 interface. Add the following configuration to
Create a new entry names "man10.123" as follows
The only difference is the addition of the "vlan" setting and an entry named "man10.123" all other fields are configured as a normal static interface.
After configuring be sure to run the following to confirm the configuration file is formatted correctly. The following is the correct output
In addition to the above, if the BMC information has been changed, please run the following command to configure BMC IP address on the system.
Example output of BMC IP configuration is shown below
Once Complete confirming the BMC IP network settings using the following command
Example output looks like the following
After confirming the IP address, BMC IP address is pingable also reachable with IPMITOOL, HTTP and HTTPS.
In large server deployments using remote syslogd where syslog entries are written over UDP is quite helpful. This allows a central server to monitor a fleet of servers by receiving all log entries over the network. This is a standard linux feature set. FMADIO Packet Capture devices support this feature, as follows:
Copy the default syslogd.conf to /opt/fmadio/etc/
Then edit the file as follows, replacing the destination IP with configuration specific to your environment
In the above example all syslog log entries are also written to a server at 192.168.1.100 over TCP on port 514.
For UDP on port 514 use the following setting
Its the standard syslogd from inted package additional customization can be done if required. Example syslog output as follows
FMADIO 100G Hardware looks as follows
List of known and tested transcivers that work with the system. Other transciver types should be ok, please check with support@fmad.io for clarification.
FMADIO Packet Capture systems are Transceiver Vendor neutral, 3rd party optics are fully supported.
Module
Speed
Connector
Cable Type
Brand
100G SR4
100Gbps
MPO12
OM4 MMF
Generic
100G LR4
100Gbps
LC
OS2 SMF
Generic
Finisar FTLC1154RDPL
100G CWDM4
100Gbps
LC
OS2 SMF
Generic
Finisar CTLC1157RGPL
100G SWDM4
100Gbps
LC
OM4 MMF
Generic
Finisar FTLC9152RGPL
100G CR4
100Gbps
DAC
Passive Twinax
Generic
40G SR4
40Gbps
MPO12
OM4 MMF
Generic
Finisar FTL410QE4C
40G LR4
40Gbps
LC
OS2 SMF
Generic
40G CR4
40Gbps
DAC
Passive Twinax
Generic
Installation of the rails is tool less please see the following instructions
System is installed with dual 1200W power supply. By default the system is configured as follows
Total Power Consumption
Behavior
LED
< 400W
HOT-STANDBY
Only Primary PSU is active, Secondary PSU in standby mode
Primary, Solid Green
Secondary, Green Blink
> 400W
HOT-HOT
Both Primary and Secondary PSU are in active mode
Both Solid Green
The above behavior can be overridden to always be in HOT-HOT load balancing mode, with a configuration setting. Please contact support@fmad.io for more information
438 x 43.5 x 730mm
17.2" x 1.7" x 28.7"
(Designed for EIA 19” rack)
26KG / 57 lbs
Silver metallic
2 x Redundant Hot swappable 1200W
110-240AC, 12-7A, 50-60Hz
8 x 40x40x56mm (23,000rpm) Chassis
2 x 1 x 40x40x56mm (Power Supply)
Front to back
Operating temperature: 10°C to 28°C
Operating humidity: 8-80% (non-condensing)
10 x U.2 NVMe Front panel drives
1 x IPMI BMC out of band management (RJ45)
2 x RJ45 Management ports
2 x SFP+(or QSFP+) High speed management ports
2 x QSFP28 Capture ports
1 x Serial port (RJ45 connector)
Dual socket Intel Xeon Platform
DDR4 RDIMM RAM
FMADIO Capture systems capture at multiple different link speeds based on the Device Model number selected, we offer all port speeds at no additional charge. The following port configurations are supported FMADIO100v2: - 2x100G - 2x40G - 4x25G (in progress) - 8x10G
Configuring the different port speeds requires updating the FPGA NIC, which requires setting the Capture Port mode and then re-updating the devices firmware. The steps are shown below: Step 1) Select the port configuration "Config Page - > Port Config" as shown below. In this example 2x10G mode is selected.
Step 2) After the port configuration has been chosen, click the "CHANGE" button to change the port speed. This will reboot the system twice as its reconfiguring the FPGA device. It will take 3-5 minutes to complete the operation
Step 3) Once the update has completed, please verify the capture port configuration on the GUI dashboard, as shown below in blue.
There are many cases where the FMADIO Packet Capture system should subscribe to Multicast traffic via the capture Port.
In the above example each capture port is connected to 2 separate switches (e.g. Market Data Feed A and B) such that each capture port receives the a direct Market data feed, instead of relying on SPAN/Mirror sessions in a passive setup.
To configure start with enabling the capture ports IP/MAC address as per
Next is configuring the IGMP groups and port configuration. Example configuration for the OPRA Options market data feed is as follows. Edit or create the file
Example igmp.lua file is as follows.
Note each Capture port (Port0/Port1) is subscribing to a different MC Group. Also the interval for broadcasting the joins by default is 60sec. In the above example this is reduced to 30sec (Interval = 30e9 is in nanoseconds)
After editing the file confirm the syntax is correct by running the command as follows
There should not be any syntax errors or error statements.
Changes to this file requires the capture to be restarted. Please stop the current capture, then restart it. After the restart the system will issue the IGMP joins based on the above.
Troubleshooting logfile can be seen in /mnt/store0/log/fnic_ping.cur this logfile should list all the IGMP joins that get issued, example as follows.
ANALYTICS SKU only
40G management ports can run in 2 x 40G or 4 x 10G mode or 2 x 2 x 10G The utility is located in
To query what port configuration modes are avaliable use
As shown below
Changing the port is simply
As follows
Then restart the system
Requires FW:7256+
Mounting a remote CIFS/Windows file system on the FMADIO capture device can be extremely useful in many cases. It allows end users to use familiar Windows file browser to access and interface with PCAPs and FMADIO systems.
The main usage for remote CIFS mounts is to provide a simple and robust way to push PCAP data automatically from the capture device to a Windows Share drive. Once on a Windows Share drive it is easily accessible from multiple devices.rr
Please edit the configuration file in
Create an entry named "CIFSDisk" as shown below, be careful all punctuation is correct. The mount point should be "remote0" or "remote1" or "remote2" which maps to /mnt/remote0 /mnt/remote1 on the capture device.
Mount options for username and password are specified after a double colon :
After the configuration file has been updated, run the following command to confirm the config file has no syntax errors.
Above is the correct operation. Once complete please reboot the system to confirm the new mount points.
To confirm the mount point please run ls -al /mnt/remote* and check the file/directory contents
FMADIO Packet Capture systems like all capture systems have multiple internal buffers. These internal buffers can sometimes cause problems for low bandwidth connections which requires Packets to be available on disk immediately for downstream processing.
One such example is Financial Order and Entry data, which can sometimes be extremely low bandwidth however downstream systems require packets to be available ASAP for further processing.
FMADIO Gen2 systems buffer between 2MB-4MB of data internally. To support multiple use cases the flushing mechanics can be tuned based on the customers requirements. By default the flushing occurs when there is no new packets in the last 1 second.
Please edit the configuration file in
The relevant sections are (there may be more or less entries in the ["Capture"] config block)
If these options are not visible in the config file, please go to the GUI Config page, change the PCAP Time Resolution to Micro Second, then back to Nano Second. This will write the default values into the config file. Alternatively you can paste the missing lines from the above example.
NOTE: After changing the settings capture must be stopped, and restarted for the new settings to take effect
Description of what each field setting does.
Flushing works by injecting specially marked NOP packets into the system right at the capture port. Its as if the packets arrived on the ingress port, but are never visible or downloadable. This parameter sets the number of packets for each flush per port to be injected. The packets are 256B in length.
Default value is: 2000 pkts * 256B = 512,000 bytes per port.
For usage models where quick Flush to disk is critical, its recommended to use 5,000 or 10,000 packets for a complete flush. Note this will directly effect how much storage is consumed by the flushing behavior
Flushing based on a pre-defined time interval. For example flush the entire pipeline every 1 minute regardless of how much data has been seen. For a 1 minute flush, the value here should be 60e9, scientific notation is accepted and the unit of time is nano seconds.
Default value is: 0 - this disables the periodic flushing
Lowest recommended setting is 1 minute, otherwise excessive flushing will consume disk space.
Flushing based on an in-activity idle timeout. This will flush the pipeline if no new packets are received within X amount of time. For example the default setting is 1 second, if no new packets are received after 1 second a SINGLE pipeline flush is issued. The next pipeline flush will only occur if new packets are received.
Default value is: 1e9 - flush after 1 second of inactivity, value in nanoseconds To disable set to 0
This mode is the default configuration
For Financial and other customers who require a constant flush to disk, the following setting is recommended
This will flush both ports every 1 minute continuously.
1 Hour / 1 Min = 60 flushes
1 Flush 2 x 5000 packets * 256 Bytes = 2,560,000 Bytes per flush
Total of extra 153MB per hour for the continuous flushing. or 1.2GB for 8 Hours is fairly reasonable.
Default system behavior is flushing when capture is idle for second (1e9 nanoseconds).
Support for FEC is built into all 100G capable Packet Capture systems. By default its turned off and requires manual setting to enable the port to link up.
Enabling FEC support requires editing the file
Add in cap0 and cap1 interface settings into the file if they do not exist, such as the following
Address and MAC settings are not required, only if you need the capture port to have an IP/MAC address
After updating the file check the syntax is correct by running the following command
The output should look like the following, without any errors or warnings about the syntax.
Once complete, please reboot the system and FEC should be enabled on boot.
FEC settings can be overridden and set manually per the following commands
Force FEC on all ports
Disable FEC on all ports
FMADIO devices run exclusively from pseudo-ROM where any changes on the file system between reboots is lost. This ROM approach provides consistency and system predictability making maintenance simpler.
One problem with this approach is shell customization becomes quite difficult. To allow small modifications in the shell environment when a user logs into the system it can run the shell script for each SSH session. Configuration file is:
Please do not use this excessively, typically its used for setting ENV variables.
Example:
This file is usually located in ~/.ssh/ directory. As that is part of the volatile file system, the persistent version of this is placed into
This allows SSH keys to be used in a persistent way across reboots and power cycles. Note the file in /opt/fmadio/etc/authorized_keys is only copied during bootup. Updates made after reboot are not copied to the user .ssh directory.
A customized sshd configuration file can be used by placing the customized configuration into
This is helpful for example to force exclusive RSA based login / disable password login. Which is a good practice if the device is on a public network.
In many cases using the default fmadio SSH RSA ID is not a good security practice. As such custom SSH RSA keys both public and private can be copied into
These will copied into ~/.ssh/idrsa and idrsa_pub on boot
FW: 7974+
The FMADIO Packet Capture device may not be conveniently accessible on a network. Ability to form persistent SSH tunnels both to and from the FMADIO Packet Capture Device is important.
We use autossh for this feature.
One typical example is pushing rsyslog traffic to a centralized location for ingest and processing. In this example we show how to configure such a tunnel that remains persistent across reboots.
Storing the key in /mnt/store0/etc/sshtunnel.id without any password
Add the sshtunnel.id.pub key to the remote servers authorized_keys. This allows password-less ssh access to the remote server.
To ensure it logs in correctly manually.
This gets run automatically on system boot. Because autossh handles reconnects, no cronjob or monitoring is required.
Create the file
Paste the contents as follows. This is a standard lua script, in this case its running autossh with some command line arguments. Anything can be run in the script for boot time setup
Can test the functionality of the boot script as follows. Correct output is similar to this
On reboot the boot.lua script is executed and a persistent ssh tunnel to the remote system is formed.
A customized persistant /etc/sudoers file can be created at the following location
On boot the system will override the default /etc/sudoers file with the file created per above
FMADIO devices timestamp all incoming packets on the first byte of payload data received at the PCS layer. The granularity of the timestamp depends on the port speed which selects the PCS clock frequency. The table below shows the granuality of the timestamp based on Port Speed
2x100G
3.103ns
2x40G
3.2ns
8x10G
6.4ns
Timestamps are always written in nanoseconds, which requires the system to constantly discipline the FPGA internal clock. This provides highly accurate timestamps against global world time down to the nanosecond level.
Below is a summary on the accuracy that can be achieved using FMADIO packet capture systems
NTP
100 milliseconds
PTPv2
100 nanoseconds
Pulse Per Second
10 nanoseconds
The following is FMADIO internal test setup on how to measure the accuracy of our hardware timestamps.
The picture above shows we are verifying the time accuracy of the system by comparing it against FMADIO Gen1 system that uses Solarflare NIC for its capture and timestamping. As the Solarflare also has a TXCO clock and running the full PTPv2 and PPS output master the Solarflare is timestamping packets directly with the TXCO master clock. This creates an excellent test.
PTPv2 time accuracy of PTPv2 only is is about +/- 100 nanoseconds. This is the measured results of the above testing setup, the resulting histogram below shows the time delta in nanoseconds between the FMADIO Gen1 (Solarflare master clock timestamp) and the FMADIO Gen3 (8x10G) timestamp as the slave clock.
Per the above time histogram, the support width is about +/- 100nsec this is the expectation of PTPv2 synchronized clocks. Its pretty good considering it requires no dedicated clock hardware, just time synchronization over 10G ethernet
For the ultimate global time accuracy, Pulse Per Second timing gives the most accurate clock synchronization results. This requires dedicated Clock hardware typically using Coax cable and SMA/BNC connectors. Typical setup is a PTPv2 Grandmaster synchronized using a GPS antenna, this Grandmaster has PTPv2 master to sync against but it also has PPS Coax output for slave clocks to achieve the absolute best global time accuracy.
Below is a time histogram of our FMADIO Gen1 (Solarflare NIC as the master clock) compared against FMADIO 40G Gen3 (8x10G) system as the slave clock. The clock are synchronized using PTPv2 and PPS coax cable. The result is impressive
In the above histogram the time bins are 1 nanosecond. As you can see the support width of the histogram is +/- 10ns. This means the global time accuracy is within 10 nanoseconds.
This the extreme level of clock synchronization as required by extreme customers. This is particle physics level accuracy as required by the ultimate most high end applications.
Mounting a remote NFS file system on the FMADIO capture device can be extremely useful in many cases.
It provides a simple way to process PCAPs on the Capture Device then writing the results out to a remove storage system.
Provides a simple and robust way to push PCAP data automatically from the capture device to an NFS share
Please edit the configuration file in
Create an entry named "NFSDisk" as shown below, be careful all punctuation is correct. The mount point should be "remote0" or "remote1" or "remote2" which maps to /mnt/remote0 /mnt/remote1 on the capture device.
Mount options can be specified after a double colon : e.g. to mount using NFSv3 or specify locking behavior
After configuration file has been updated, run the following command to confirm the config file has no syntax errors.
Above is the correct operation. Once complete please reboot the system to confirm the new mount points.
NOTE: if the NFS mount fails it may take 5 min to boot the system. This is the time it takes for the NFS client timeout and give up on mounting the remote partition. If long boot delays are seen please check the log messages in /mnt/store0/log/messages for reasons the NFS mount failed.
Below is a a full disk.lua example configuration file for reference
We use nginx as our WWW front end interface, enabling all the standard features and stability this provides.
Custom TLS/SSL WWW x509 certificates are supported. Steps as follows
STEP 1)
Copy <you cert>.pem to /opt/fmadio/etc/fmadio_cert.pem
Copy <you cert>.key to /opt/fmadio/etc/fmadio_cert.key
STEP 2)
Stop the current nginx process
STEP 3)
Wait for nginx to re-spawn (takes ~ 60 sec) and your certs should be used for all HTTPS/TLS communication.
Any problems check the log files in /mnt/store0/log/nginx_error.log
In many environments different Authentication is required. By default FMADIO capture systems using built in BASIC authentication over HTTP. As this makes configuration and setup simple but is very weak security setting.
BAISC (insecure)
HTTPS Only + BASIC
RADIUS
Active Directory (SSO via OAUTH 2.0)
Google Cloud (SSO via OAUTH 2.0)
Ping Identity Cloud (SSO via OAUTH 2.0)
By default HTTP and HTTPS are enabled on the GUI. In any security setting HTTP needs to be disabled, as its an unsecure protocol. To disable HTTP edit the config file
Find the "Security" section as follows
Change the "HTTPAccess" section from "enable" to false as follows
Save the file
Then restart nginx as follows
NGINX will restart automatically within 60 seconds with the updated configuration. Only HTTPS access is possible.
SSO configuration is more complicated, please contact support@fmad.io and we can walk you thru the setup personally
FW: 7563+
We support RADIUS authentication using the freeradius client. Configuration is as follow
Edit the configuration file
Find the "Security" section, example shown below
Change the following, this disabled the HTTP protocol
Changes the following, this enables RADIUS as the authentication method
Configure your RADIUS login information
Finally the Timeout, this is how long the system waits until it will automatically logout the user and requirement them to re-authenticate. Value is in nanoseconds, scientific notation and formula is no problem. Per below, 24 hours * 60 min * 60 sec * 1e9 (nanos)
Restart nginx as follows, it will re-spawn within 60sec automatically
You should see a login page when accessing FMADIO as follows
If there is some problems, please confirm on CLI using radclient, example as follows.
FW:7608+
FMADIO Capture devices can authenticate the users using Active Directory via the OAUTH 2.0 protocol. This enable Single Sign On with ADFS.
In the follow example we have used a reverse SSH tunnel to temporarily put FMADIO system on a public IP, as Azure Active Directory services require internet accessible devices for the redirect_uri.
For an On Premise Active Directory server this is not required.
Example Reverse SSH Tunnel
NOTE: SSH tunnel should not use localhost, as all localhost sourced requests bypass authentication. Instead use the IP address of the management interface
Start by editing the general FMADIO configuration file
Then setting HTTP (un-encrypted) access to "disable", and Auth method to "OAUTH", example shown below. The other security fields can be left as is.
Save the file and ensure there are no parse errors by running fmadiolua /opt/fmadio/etc/time.lua
Next create a file name
This file contains the ADFS OAUTH End points as follows
These fields are from the ADFS Endpoint URI information, for example as follows. We created a fmadio sign in entry, this has the following client_id entered above.
The "discovery" config in the above needs to be the OpenID Connect Metadata document, as seen below.
the "client_id" is the shown below
The "client_secret" in the above config needs to be the Value shown below, not the secretID
Finally the "redirect_uri" needs to be registered as follows.
Once config is complete, please confirm no syntax errors by running
Correct output is as follows, if there are any syntax errors please correct.
Restart nginx to load in the new configuration file, by killing the process as below. It will reswpan on a 1min cron job automatically
Next point a browser to the FMADIO device, it should redirect you to the Active Directory login page as follows.
Login to the system using your Azure / Microsoft credentials. Then the FMADIO device dashboard will be shown as below
Logout is the same, using the logout button shown below
Then choose an account to sign out of
FW:7608+
While less practical as its typically for publicly accessible sites, it can be used with a Google Cloud VPC to tunnel authentication requests from a private network to Google Cloud infrastructure.
In this example we just reverse ssh tunnel an FMADIO system onto the public internet (strongly discouraged) for demonstration purposes only.
Start by editing the general FMADIO configuration file
Then setting HTTP (un-encrypted) access to "disable", and Auth method to "OAUTH", example shown below. The other security fields can be left as is.
Save the file and ensure there are no parse errors by running fmadiolua /opt/fmadio/etc/time.lua
Next create a file name
This file contains the Google Cloud OAUTH End points as follows.
The "clientid" and "client_secret" need to be replaced with the generated authentication information from google per below. The above is a throw away example only
Next generate Google OAUTH credentials as follows.
Then fill in the information, as follows. Google is a bit more strict and requires TLD endpoints not raw IPs
Which results in the following secret information
Update the oauth_opts.lua file above with the information
Restart nginx to load in the new configuration file, by killing the process as below. It will reswpan on a 1min cron job automatically
Next point the browser to the FMADIO device and it will redirect to Google Sign in account
Login using your Google account information, and it will re-direct you to the FMADIO dashboard.
Any further questions please contact support@fmad.io for assistance.
FW:7608+
Ping Identity is a popular onprem authentication system, typically used in large organizations. We support Single Sign On with their product suite, below is an example configuration example setup using the Cloud Services. This example uses a reverse SSH tunnel to put the FMADIO device on a publicly accessible IP (we strongly discourage) for demonstration purposes only, to replicate setting up an On Premise install.
Start by editing the general FMADIO configuration file
Then setting HTTP (un-encrypted) access to "disable", and Auth method to "OAUTH", example shown below. The other security fields can be left as is.
Save the file and ensure there are no parse errors by running fmadiolua /opt/fmadio/etc/time.lua
Next create a file name
This file contains the Ping Identity OAUTH End points as follows.
The "clientid" and "client_secret" need to be replaced with the generated authentication information from Ping Identity interface per below. The above is a throw away example only
We setup a web application using Ping Identity interface as follows. The key fields are shown in red.
These fields are mapped directly into the oauth_opts.lua configuration file above.
Restart nginx to load in the new configuration file, by killing the process as below. It will reswpan on a 1min cron job automatically.
Next point the browser to the FMADIO device and it will redirect to Ping Idneitty SSO account as follows
After a successful authentication the FMADIO dashboard is seen
Any further questions or problems, please contact us support@fmad.io
FW: 8529+
FMADIO systems support Linux PAM ( https://github.com/linux-pam/linux-pam ) as an authetication method. One option for centralized authentication is to use LDAP via PAM.
1) First run fmadiocli settings to set the authentication method
https://docs.fmad.io/fmadio-documentation/cli-reference/fmadiocli#config-security-auth
2) We also strongly recommend to disable HTTP access as all username / passwords are sent over un-encrypted HTTP
https://docs.fmad.io/fmadio-documentation/cli-reference/fmadiocli#config-security-http
3) Configure LDAP client nslcd. Copy the default config file as follows
The default config looks like the following
NOTE: ensure the permissions of
Are set as root.root and user only read/write
Otherwise nslcd will fail to start due to in-secure permissions
Modify the uri, base and any other LDAP specific configs to the enviroment and save it
4) reboot system
5) check LDAP connectivity
Changing the username/domain/ip address etc to match your environment
Successful authentication looks like the following
Once this is working, both SSH, WWW-Admin and WWW-User LDAP posix group members can login to the system.
The LDAP posixGroups are
fmadio-ssh-admin - for SSH access
fmadio-www-admin - for WWW admin access (can change anything)
fmadio-www-user - for WWW user access (monitoring and pcap downloading)
6) Both SSH and WWW now fully configured using LDAP as centralized authentication
Some environments require a notice when logging in, such as the following
This can be customized as follows
1) copy the default template
2) Edit the content of
3) restart nginx
kill nginx and wait 60sec for it to restart
Configuration usually does not go as planned, as such heres some tips to try
1) run nslcd in the foreground
This will check the /etc/nslcd.conf configuration file is working correctly, either config typeo or LDAP server problems.
Once its running ensure local lookups work correctly as follows
2) check nginx config files
The nginx logfiles are located in
Any errors there might help understand the issues
3) check syslog file for PAM logs
This will print out logs of all PAM messages and may help debugging
FW: 8325+
FMADIO Packet Capture system have 16 to 96 CPUs, this can make configuration and allocation of all these CPU resources tedious and error prone.
To assist the configuration file located in
There is a section name CPUMap, an example is shown below
CPUS which are dedicated to FMADIO Packet Capture system can not modified, these are marked "System" in the comments section per below
Any change to system CPUs will be overwritten.
By default the system allocates all CPUs to the FMADIO Capture System. To enable a customer CPU Mapping change Enable = true per the below setting
After enabling a FW update is required. As the enable re-writes the isolcpu Linux kernel boot parameters.
NOTE: after Enable = true is set, all subsequent firmware updates will modify the isolcpu setting. To return to stock default setting. Set to false, and update the firmware again.
To assign a CPU to a specific LXC container, use the naming convention
For example Market Data Gap detector LXC (mdgap) for CME is allocated to CPU 84
This also requires re-running the LXC install script to correctly allocate the CPUs in the container.
After the configuration files have been updated, a firmware update and reboot cycle are required. Its required as the Linux kernel boot command parameters (isolcpus) gets modified requiring the reboot.
Start by finding the current Firmware binary, in almost all cases this is the last upload firmware
In this case the latest firmware is named
Next re-install the firmware as follows. An upload is not required as the firmware is already on the system
Example output as follows, note the "Update CPU Map" output
Then reboot the system, and the new CPU mapping is reflected.
Various settings related to downloading PCAP.
Depending on the packet capture system the Download interface may be in Legacy mode. This modes functionality is limited in the range of API calls, specifically /api/* calls can not be used
To enable API Version 1, edit the configuration file
Find the section ["PCAP"] example as below
Please change ["DownloadAPI"]= "legacy" to "v1" as below. If theres no entry, create the entry.
It requires a browser refresh for the setting to become active.
FMADIO Devices use our internal FPGA NIC card for hardware timestamping on all packets. This may not be sufficient for some customers as Tap/Aggregation layers add additional non-deterministic latency due to egress port buffering.
FMADIO Packet Capture has the option to use Packet Brokers and Switch's metadata either in headers or footers. This allows ingress timestamping at the Tap/Agg layer plus gives additional port visibility and filtering capabilities.
By default FMADIO uses our internal hardware timestamps when downloading PCAPs. We support the following formats
FMADIO Capture Card (Default)
Cisco ERSPAN v3
Arista 7130(Metamako)
Arista 7150 (Overwrite mode)
Arista 7150 (Insert Mode)
Use the Config page on the FMADIO Capture system to select as follows
NOTE: This can be done retro-actively. e.g. Downloaded PCAPs are generated based on the timestamp selection mode. This setting does not effect the raw captured data.
FMADIO System can de-encapsulate various formats both in PacketScope, Search and via the API. Please contact support if you require an additional format.
De-encapsulation means the system will strip away the encapsulation, and BPF filters are run on the inner most packet.
VLAN Single Tag
VLAN Double Tag Old style (Ether Proto 0x9100)
VLAN Double Tag Old style (Ether Proto 0x9200)
VLAN Double Tag QinQ (Ether Proto 0x88a8)
VNTag Cisco (Ether Proto 0x8926)
VXLAN (Tunnel over UDP)
MPLS Single
MPLS Double
MPLS Triple
Cisco ERSPAN v1
Cisco ERSPAN v2 (Over GRE tunnel)
Cisco ERSPAN v3 (Over GRE tunnel)
CAPWAP (Wireless Encapsulation over UDP)
Arista 7150 (Insert Mode)
Arista 7150 (Overwrite Mode)
Arista 7130 (Metamako)
Ixia xStream
Exablaze Footer
By default PacketScope enables the de-encapsulation shown below as the checkbox enabled. To disable de-encapsulation uncheck the checkbox and re-run the filters.
On select models of FMADIO capture systems full disk encryption is available. When available it uses the SSD drivers controller firmware to provide AES256 encryption with the OPAL interface standard. States of the system is as follows Power Off: All data is encrypted accessing requires a password First Power On: Drives are accessible but data remains encrypted First Power On Unlock: Each drive in the system is unlocked by a shared password. This allows the drives media to be written/read from Warm Reboot: After Unlock the drives remain unlocked Power Off: On power loss to the disks, all data becomes un-accessible and fully encrypted Data is encrypted using AESS256 and a random key generated by the SSD Controller. The Password specified encrypts/decrypts this AES256 key allowing the controller to read/write from the media. This encryption key is only kept in volatile RAM, thus when power to the drive is removed, the encryption key is lost. Once the encryption key is gone all data on the storage media can not be read. Drives can never be "bricked" as the drives can be reset by creating a new AES256 key. This reset however will remove all data previously written to the drive.
This Operation displays the drive encryption state, documentation found here
https://docs.fmad.io/fmadio-documentation/cli-reference/fmadiocli#show-disk-status
This sets a new encryption password on all the disks. It will also WIPE ALL DATA ON THE CURRENT DISK. Please ensure there is no critical data on the system before running this command. After the operation the disks are in an UNLOCKED state.
https://docs.fmad.io/fmadio-documentation/cli-reference/fmadiocli#config-disk-sanitize Setting Password
Drives always have a password set, either a default password or a custom password the following command sets the disk password used for unlocking
By default disks are not a locked state. e.g. they do not require a password to be usable. This setting puts the disks into a locked state which when ever it loses power requires a password to unlock and make usable
There are cases where disabling the Locking function without destoying the data is helpful, note this is different from unlocking the disks.
This function disables requring a password to access the disks persistently.
Unlockingly the disks allows disk access but is not persistant
The functions are similar but quite different, documentation on how to perform this is
https://docs.fmad.io/fmadio-documentation/cli-reference/fmadiocli#config-disk-no-lock
Command will unlock and verify correct functionality of the disks. Starts by unlocking each disk using the supplied password, then reloading all storage components of the FMADIO system which reference the disks.
Reference documentation is
Generally entering passwords manually is fairly cumbersome / time intensive. The system has the ability to unlock the disks on boot.
The general purpose boot setup file /opt/fmadio/etc/boot.lua can be used to fetch the password from a key server, e.g. via CURL or copy a password file from the local disk
Below is an example boot.lua file to automatically unlock the disks at boot time, using the saved password in /opt/fmadio/etc/disk-password
For debugging the logfile is located in
/mnt/store0/log/boot.log
example output is shown below
FMADIO100 Capture system supports multiple data "Disk Packs" per capture system.
The system at boot time detect which "Disk Pack" that is currently installed, and uses that as the currently active config file.
FMADIO100Gv2 capture system has 10 x NVMe U.2 drive slots, of these 9 of the disks are used for data, and one disk is used for the OS. Below is the location of the OS disk. The OS disk can not be removed or swapped, must always be inserted at the location below (top left hand corner).
The location of the remaining 9 data disks when installed can be in any order.
Power down the system
Remove all 9 data disks from the current chassis and place into the provided storage case.
Insert the new OS disk into the top left slot, then each of the new 9 data disks into any location. Be careful to not use excessive force as it may damage the backplane and device.
Power on the device, it will take 90-120sec to fully boot. Once fully booted check the disks have been recognized by running "ls" in the directory
Correct listing will look like below, ssd0 - ssd7 and par0 are mapped to a physical end point. If any disk is missing, please reseat the disk and reboot the system.
System is now ready for capture and operation
When a Disk Pack is first delivered, it requires configuration. This is a one time operation that can be done in a lab environment.
The steps are as follows
Per the above instructions, install the new disk pack hardware. Including OS and 9 data disks.
Power on the system, it will have no network connectivity, but serial port and VGA are operational.
The system will be behaving oddly, and is partially operational. Connect to the VGA or Serial port and login with the default passwords.
NOTE: SSH network connectivity will not be functional.
Manually configure the 1G management interface with ifconfig, example below
The interface may be eth0 or eth1 depending on the config
Once 1G connectivity is established ssh and scp work no problem. SSH into the system
Update the udev rules with the vendor provided file. Either scp or use a text editor. The file is located in
After copying the file reboot the system. The system should have normal operation.
IPTables running on FMADIO Capture systems
FW: 7650+
IPTables the linux statefull firewall software runs on the FMADIO devices. By default iptables is disabled / ACCEPT for everything. In some scenarios a tighter security policy may be needed
The iptables command works same on a standard Linux system. Please refer to the following link for documentation
FMADIO uses nginx and fcgi backens as well as proxy pass settings. As such IPTABLES requires an INPUIT localhost ACCEPT rule such as the following.
which can be added as follows
Without this INPUT ACCEPT rule the FMADIO GUI dashboards and status settings can not be retrieved.
After configuring the IPTABLES setup on the FMADIO Packet Capture device, the settings will be lost each time the system is rebooted.
We use iptables-save and iptables-restore command with the configuration file located
To save the current state run
This will generate a the looks like the following. This is a baseline recommended setting for SSH, HTTP and HTTPS access only.
To remove persistent IPTABLES setting, delete the /opt/fmadio/etc/iptables,conf file and reboot the system
The firewall on the IPMI/BMC is a bit trickier, as there's no direct access to iptables and manipulation needs to be done using a very unfriendly ipmitool raw access.
IPMI does have firewall manipulation GUI but its impossible to use due to how it works. Theres no way to set a Policy on INPUT rules, instead you need to drop everything and build up the chain. Below is the final iptables rules we want to create.
As you can see its a bit bastardized.. but theres no choice as each rule is always added to the top of chain.
Our goal is to DROP everything, except SSH, HTTPS and IPMITOOL traffic.
Start by resetting the BMC firewall state entirely. This effectively resets iptables to the default state
And then confirm this by listing the total number of Firewall rules as follows
The value returned should be 00 indicating there are NO custom firewall rules.
NOTE: This can be used to clear/reset firewall settings if a mistake is made
Next we need to drop everything, as we are building the rules backwards. This is also the reason we cant use the GUI. It wont let you set a network of 0.0.0.0/0 and once you set that the GUI is no longer accessible.
As such we need to use ipmitool on the FMADIO Packet Capture device directly as we build up the rules.
The command above adds the drop everything rule to the system, this equates to the following in iptables.
Can confirm its working correctly by checking the total number of firewall rules as follows. The returned value should be 1
Next we will add SSH access to the firewall rules. This allows SMASH or shell access to the BMC device itself.
The following command opens TCP Port 22 (0x16 0x00 == 22 in hex bigedian format)
This adds the following iptables rule
Then confirm there are 2 firewall rules enabled.
At this point you can SSH into the BMC to confirm access is working correctly
Next add HTTPS access enabling the IPMI BMC Web client to be accessed.
This equates to the following iptables rules
At this point the IPMI BMC Webpage can be used such as the following
Finally add ipmitool access which is on UDP port 623
And the related iptables rule
This enable ipmitool to work over the network, which can be extremely critical and helpful when troubleshooting problems. Such as the following
While its quite cumbersome to use ipmitool raw mode to add and remove all these filters, the net result is a fairly secure BMC locked down with standard linux iptables.
Any questions or trouble please contact support.
FMADIO 100G Gen2 System,
SKU Capture (64G RAM + 32 CPUs)
FMADIO100GV2-1U-16T
FMADIO100GV2-1U-38T
FMADIO100GV2-1U-76T
FMADIO100Gv2-1U-156T
SKU Analytics ( 576GB RAM + 96CPU)
FMADIO100GV2-1U-16T-ANALYTICS
FMADIO100GV2-1U-38-ANALYTICS
FMADIO100GV2-1U-76T-ANALYTICS
FMADIO100Gv2-1U-156T-ANALYTICS
High Level Architecture
FW: 7738+
FMADIO Packet capture systems have the ability to push specific PCAPs to a external 3rd Party application as follows
This workflow enables a simple approach, using a URI to push a PCAP over HTTP using a POST request to a remote end application, then following a JSON redirect.
In this example, we are using our internally developed FMADIO Packet Scope and FMADIO Shark (FShark) as a reference example. This is for demonstration purposes only, any 3rd party web application will work.
The workflow process is as follows,
Web Application A generates a en.loader.html URI. For example Epoch Start/End and an BPF Filter
Web Application A directs the Client to this URI
FMADIO Packet Capture System presents the Client with the FMADIO Loader web page. This shows the progress of filtering and upload to Application B and any potential errors.
Application B completes upload, and returns a redirect URI
FMADIO Loader web page follows the redirect, allowing the client to load seamlessly into Web Application B with the new PCAP data.
The example is using FMADIO developed software for this, however there is no limitation, any 3rd party application will work.
In the following example we are using FMADIO Packet Scope as as (Application A) and FMADIO Shark as (Application B).
In this example, Web Application A is "FMADIO Packet Scope" with Web Application B "FMADIO Shark". PacketScope generates the loader link request as follows
The above example has an Epoch Start time and Epoch End time as well as a BPF Filter applied, the end result is the following URI
This redirects to the FMADIO Loader page which processes and pushes the above Filter specification to the Target "fshark"
Expanding on the details
Path indicates how the PCAP should be generated, in this case its a specific capture name with filters.
Target, informs what the End Point target is. "fshark" is an internally defined EP. Due to security reasons End Point definitions can only be configured on the FMADIO System. Only the enumerated name of the End Point is used in the URI.
StreamName specifies the name of the Capture to process.
TSBegin is the Epoch in Nanoseconds for the start of the Filter
TSEnd is the Epoch in Nanoseconds for the end of the Filter
FilterBPF is the Escape Encoded BPF filter, in this case "udp and port 53" e.g. extract DNS traffic.
For full details please check the API v1 Documentation page
Once clicked the following Search page is visible.
In addition we added the following URI to prevent automatic reload, this can be helpful for debugging purposes.
Internally the FMADIO Device is issuing the following HTTP POST command thru CURL on the filtered PCAP. This URL generator is configuration on the FMADIO Device, almost anything is possible. This example the "fshark" Target is built into the system firmware.
Once completed the above HTTP POST request into Web Application B is completed, it returns a redirect as follows to FMADIO Packet Capture System
This redirect URL is forwarded to the Web Client is shown below
The 3rd Party Application returns the redirect as the response to the POST upload request. This is where the redirect URI is specific, or any error condition
After the HTTP Post has been completed the 3rd Party Application B returns a redirect page. The FMADIO Loader then redirects to this location.
In this example it loads FMADIO Shark with the URI
The following page is what the 3rd Party Application displays.
NOTE: Due to FMADIO Shark being internally developed the Web page does look like FMADIO products. This page could show anything, there's no iframes etc.
The recommended URI is to use the /api/v1/pcap/timerange URI endpoint. As this does not require any stream names, just Epoch start/end times and a BPF filter.
Example as follows, please remember to Escape Encode the FilterBPF string.
Requires FW:6979+
FMADIO Packet Capture systems provide a built in Push mode to transfer capture PCAP data on a regular schedule to a remote system. An example is pushing 1minute PCAPs to a remote NFS share or an S3 storage bucket.
Configuration is via configuration scripts located:
If there is no such file above, please copy the basic example from the following location
An example is shown as follows:
Multiple push targets can be specified, there is no real limit however throughput does get effected.
In the above example there are 2 push rules
This push target sends all PCAP data the remote NFS share mounted on
/mnt/remote0
See NFS mount configuration section for details on setting up /mnt/remote0 mounting points.
The sepcified is "FilterBPF=nil" meaning there is no filter, thus all traffic is pushed
The second example shows pushing all TCP data on the network 192.168.1.0/24 to the specified /mnt/remote0/push/ directory with a PCAP file prefix of "tcp_*"
and many more, see the rclone documentation for full list of endpoints supported
Following is a description of each option for per push target.
Required
Provides a text human readable description for each push target. It is also used for for log file description
For example the above push logfiles will go to /mnt/store0/log/push_pcap-all_* this can be helpful for troubleshooting any problems
Default (FILE)
Specifies how the output files are written. Currently there are 2 modes, standard linux file "File" and rclone which provides multiple end points such as FTP, S3, Google Drive, Azure Cloud and many more.
Required
Full remote path of the target PCAPs + the leading prefix of the remote output.
The above example uses the "FILE" mode, which specifies a full linux system file path.
Required
This specifies how to split the incoming PCAP data, either by Bytes or by Time. Following example is splitting by Time
Required
Specifies how to split filename is encoded. Different downstream applications require specific encodings. If your downstream applications need an encoding not listed, please contact us for support.
Default (nil)
Full libpcap BPF filter can be applied to reduce the total PCAP size or segment specific list of PCAPs . The system uses the native libpcap library, everything that tcpdump supports FilterBPF also supports.
The above is an example BPF filter "net 192.168.1.0/24 and tcp" its a slightly more complicated BPF and shows the flexibility and wide range of options available. Technically there is no limit on the complexity of the BPF filter, we recommend to keep it as simple as possible to reduce the CPU load
Default (true)
In addition to FilterBPF full packet de-encapsulation is performed by default before the BPF filter is applied. This for example can decode VLAN, ERSPAN, GRE tunnels and many more. It enables the BPF filter is applied on the inner payload instead of the encapsulated output,
Example to disable automatic De-encapsulation
Configuration is a simple boolean type only
Requires FW:7157+
Default (nil)
Pipe commands are processed on a per PCAP split basis before the end transport is applied. Examples to use this are to GZIP or compress files before sending to the endpoint.
This is a generic stdin/stdout linux application, gzip, lz4 are current examples, Other options are possible, please contact us for more details
The above runs gzip with compression level 1 on the split PCAP before sending to the output location. Some examples are shown below
Requires FW:7157+
Default (nil)
By default the split PCAP filename suff is .pcap For most operations that is sufficient, however for more complicated operations such as GZIP compressing with PipeCmd a .pcap.gz file suffix is more appropriate. The Following is an example config target that compresses and outputs splits in .pcap.gz file format
The above example pushes gzip 1minute PCAP splits to an S3 protocol storage device
Requires FW: 7355+
Default: (nil)
Chunked mode is a more optimized processing mode. It increases the aggregate thoughput of the Push operation specifically for network traffic profiles skewed towards small packets.
By default its disabled.
Example as follows
Requires FW: 7355+
FollowStart forces the push to start from the beginning of the capture. If its disabled it will push from the current capture position.
Default is "false" push from the current capture position
Example as follows
Requires FW: 7750+
Sets a specific CPU for stream_cat to run on by overriding the default CPU setting. This is helpful when multiple pushes are running in parallel.
Default: the system assigned CPU number for push (typically CPU 23)
Example as follows
The default behavior of the system is to constantly re-try sending data downstream without loss. In some cases its better to restart the push process and reset the start sending position, when the consumer application restarts.
An example is, if an application shuts down between 1AM and 6AM but the capture process runs 24/7. The application wants to only receive data starting at 6AM when the application starts up.
Another use case is, if the application has an error for an unspecified amount of time. The application requires real-time processing, and requires FMADIO Capture system to send data from the current time now without sending old historical data.
Adding the following configuration to the push_realtime.lua config will cause stream_cat to exit if the downstream consumer is unable to process data. FMADIO system scheduler will constantly restart the push process from the current time, until the consumer process starts processing data.
NOTE: Please keep the additional white space at the end of the command.
In addition to configuration of
/opt/fmadio/etc/push_pcap.lua
To specify when the Push operation occurs the Analytics scheduler must be configured. This is on the "CONFIG" tab of the FMADIO GUI. An Example configuration to push files 24/7,
The "Analytics Engine" field must be exactly the following text.
Screenshot of 24/7 schedule is shown below
Troubleshooting
Configuration problems often occour when setting up the system. The following log files can be used to debug
Monitoring the output can be as follows
In addition each Push entry has a log file with the following format. The Desc value is described
Example output of correct functionality is as follows
In addition to log files its sometimes easier to debug via the CLI interface, by manually starting the push on specific capture files. This can also be helpful to push historical PCAP files.
This is done as the following CLI command
Replace capture name with the complete name of the capture. Also ensure push scheduler has been disabled
Example output of successful offline mode run is shown below.
Note the following repeated status line indicates the push is operating successfully
For problems per push target, the logfile shown in the above command line here /mnt/store0/log/push_pcap-all.cur
A good way to debug that is running tail -F /mnt/store0/log/push_pcap-all.cur to monitor it such as the following
Multiple push_pcap schedules can be added to the system, for example
Push A) Realtime 1min push
used for realtime monitoring throughout the day
Push B) End of Day, recon push.
Used for End of Day Recon pushing data to back office systems
This can be achieved with the following steps
All files in this directory are sym linked to the /opt/fmadio/analytics directory used by the scheduler.
Create a new push_pcap_eod loader as follows
Then Edit the file to load a different configuration
In this case the config file is called push_pcap_eod.lua
Configure the new push_pcap_eod.lua file, it will require hand editing of the file, as fmadiocli only operates on the default configuration
Going to the GUI -> Config page add the new loader file into the schedule with the new loader file push_pcap_eod
Example is shown below
Log files for (push_pcap_eod) are named
Push performance is critical and subject to multiple factors. The following provides a baseline test of different variables.
A first initial step is to confirm the writing to the remote file system has sufficient bandwidth. This is simply achieved running the commands
Example run
The above command writes a 20GB file to the remote file location. In this case its writing @ 678MB/sec ( 5.4Gbps throughput)
To force the PCAP link layer type to Ethernet use the following CLI command on the PCAP
The symptoms of this is unusual TCPDUMP output such as the following
After setting the PCAP Link Layer setting using the above command the output is as follows
Which contains the correctly decoded packets.
Example Configuration files for refence
FW: 7936+
Example pushes a single UDP multicast group 1001 at 1 minute snapshots using an Hour Min Sec with Timezone filename.
Example pushes 1min PCAPs with a BPF filter (port 80) and applying LZ4 compression. LZ4 compression is fast and reasonably good compression rates.
Example pushes 1min PCAPs with a BPF filter (port 80) and applying ZSTD compression. ZSTD is a new compression format with performance close to LZ4 but compression rates close to GZIP.
FW: 7936+
Example pushes the raw data to a remote NFS/CIFS (Windows Share) splitting by 1GB file size writing a gzip compressed PCAP file to the remote location.
Example pushes different VLAN traffic to seperate lxc_rings
Pushing captured PCAP data from the local device to AWS S3 Bucket can be done using the RCLONE support.
Below is an example push_pcap.lua config file for that
This uses gzip to compress the data. Also note we added a PreCapture filter to 64B Slice all traffic to AWS S3 IP address. This prevents the capture size for a run-away explosion.
Below is the resulting output in AWS S3
This does require RCLONE S3 Config to be configured before using.
Performance of push varies by protocol filter and compression mode. Typically the remote push locations IO bandwidth is typically the bottleneck.
Here we are testing against a RAM disk on the local system. This removes the network and remote IO performance from the benchmark. Focusing entirely on the FMADIO fetch filter and compress performance.
Parallel GZIP (pigz) running on an FMADIO 100Gv2 Analytics System (96 CPUs total). Making good use of all the CPUs
XZ compression using 64 CPUs, more utilization than Parallel GZ
The dataset we are testing with is a WAN connection that has an ISP like L2-L7 packet distribution. As its ISP like traffic the data compression rate is not high.
The data set tested is a full days worth of OPRA A+B Feed dataset, raw uncompressed data size is just under 1TB. Financial data typically gets x2 to x3 compression ratio with xz maxing out at x5.
FMADIO System uses a default cronjob that is located as below
Direct edits to this file will be lost every reboot, as the directory is on the RAMDisk and not persistant.
This can be customized and made persistent across reboots, for anything such as scheduled weekly reboots. By creating the cronjob file as bellow.
This puts the crontab.root file into persistent storage at
Allowing custom cronjobs to run on the system.
One example of this is to schedule a system reboot every Sunday night. To do this please see the below example.
Adding in the last line in the cronjob will schedule the reboot at midnight every Sunday.
NOTE: Please reboot the system or manually reload cron for the new setting to take effect
FMADIO capture systems support PTPv2 using the linux PTP open source project. ( ) running only on the 10G management interface. 1G links are supported using 1G RJ45 SFP transceiver.
Enabling PTPv2 via GUI is as follows
In addition the 10G Bridged networking config must be disabled. As LinuxPTP must run directly on the bare metal NIC. Please edit the file
Change the "phy10" values as shown below to "man10"
Changed to "man10" as follows
System must be rebooted for the above to take effect.
In some environments PTPv2 is run on its own dedicated VLAN interface. To enable this update the configuration file
Specifically setting the VLANID as follows, in the below example its set to VLANID "123"
After editing save and confirm the file syntax is correct, ensure there are no syntax or parse errors. The correct output is shown below.
Either reboot the system, or kill all ptp processes (e.g. sudo killall ptp4l; sudo killall phc2sys)
On reboot a man10.123 VLAN interface that matches the set VLANID should be populated as follows
And the linux ptp daemo should have binded to the newly created instance as follows
And the PHC interface also
The above config assumes PTPv2 is running on the primary 10G management port. There is not requirement for using the primary port, instead the 2nd 10G management port can also be used. Below are the differences required
Use phy11 interface
Rename to man11
man11 needs to be configured with an IP address, either editing network config file directly
or using the fmadiocli command line utility
PTPv2 can be run on the 2nd 1G management interface, as it supports both a PHC clock and IEE1588 hardware timestamping.
NOTE: PTPv2 can not be run on the primary/first 1G interface as this is a software bridged interface and does not support hardware timestamping.
Set the phy1 interface renamed to man1 as shown below
man1 needs to be configured with an IP address that the PTP master can reach. Set the network information either editing network config file directly
or using the fmadiocli command line utility
FMADIO system use the standard LinuxPTP project for time synchronization
As such the PTP configuration is highly customizable. For example Broadcast environments use a non standard domain and update rate. FMADIO support this by using a custom LinuxPTP configuration file.
Custom config file can be located below
If this file is present the system will use it when starting the application ptp4l.
After creating or editing the file, need to stop ptp4l as follows
The system will then automatically start it on a 60sec timer.
Logfiles for debugging can be found in
FMADIO Devices have a PPS input connect that expect 1 1PPS 50ohm 5V signal. The start of second boundary occurs on the rising edge of the PPS clock. Required hold time is 100usec or more.
Enabling of PPS is automatic, when the system detects PPS input, it automatically selects PPS to discipline the timestamp clock.
Note FilterBPF=net 192.168.1.0/24 and tcp This applies a full BPF (Berkley Packet Filter ) with the filter "tcp" on the packets before writing it to the location. This results in only TCP data written to the /mnt/remote0/push/tcp_*.pcap output files
Mode
Description
linux file
linux file on FMADIO capture system
NFS
remote NFS mountpoint on FMADIO capture system
SFTP
remote SSH file system via rclone ( https://rclone.org/sftp/ )
FTP
FTP push via rclone ( https://rclone.org/ftp/ )
S3
S3 protocol via rclone ( https://rclone.org/s3/ )
Google Drive
Google drive via rclone ( https://rclone.org/drive/ )
Digital Ocean
Digital Ocan Spaces via rclone ( https://rclone.org/s3/#digitalocean-spaces )
Azure Blob
Microsoft Azure Blob via rclone ( https://rclone.org/azureblob/ )
Dropbox
Dropbox via rclone ( https://rclone.org/dropbox/ )
Hadoop HDFS
Hadoop file system via rclone ( https://rclone.org/hdfs/ )
Ceph
Ceph S3 interface via rclone ( https://rclone.org/s3/ )
Command
Description
FILE
output a regular linux file. this can be ln the local file system or over a remote NFS mount
RCLONE
use rclone as the end point file. Note rclone needs to be setup and configured before remote push is started
For RCLONE Config please see their documentation
https://rclone.org/commands/rclone_config/
FMADIO by default stores config file into
/opt/fmadio/etc/rclone.conf
Requires FW:7157+
LXC
Writes output to the LXC ring buffer. Location of the ring buffer is the Path variable e.g
/opt/fmadio/queue/lxc_ring0
Requires FW:7738+
CURL
Pipe down a CURL pipe, e.g. to FTP or HTTP POST
Command
Description
/mnt/remote0/push/all
FILE mode output PCAP files will be written for example as /mnt/remote0/push/all_20210101_010101.cap
gdrive://pcap/all
RCLONE mode output PCAP files written be written to the rclone configured google drive endpoint into the google drive directory /pcap/
Command
Description
--split-time <nano seconds>
Splits PCAP data by time, argument is in nanoseconds Scientific notation can be used
--split-byte <bytes>
Splits PCAP data by Size. argument is in bytes, Scientific notation can be used
Command
Description
--filename-epoch-sec-startend
writes the sec epoch start/end time as the file name
e.g March 21, 2021 1:50:55
1616334655-1616334755.pcap
--filename-epoch-sec
writes the sec epoch start time as the file name.
e.g March 21, 2021 1:50:55
1616334655.pcap
--filename-epoch-msec
Writes the epoch start time in milliseconds as the file name
e.g for April 22 02:48 GMT
fmadio_1650595712592.pcap
--filename-epoch-usec
Writes the epoch start time in microseconds
e.g For April 22 02:48 GMT
fmadio_1650585598007301.pcap
--filename-epoch-nsec
Writes the epoch start time in nanoseconds
e.g For April 22 02:48 GMT
fmadio_1650585598007301462.pcap
--filename-tstr-HHMM
writes the YYYYMMDD_HHMM style file name.
e.g. 2021 Dec 1st 23:50
20211201_2350.pcap
--filename-tstr-HHMMSS
writes the YYYYMMDD_HHMMSS style file name.
e.g. 2021 Dec 1st 23:50:59
20211201_235059.pcap
--filename-tstr-HHMMSS_NS
writes the YYYYMMDD_HHMMSS.MSEC.USEC.NSEC style file name.
e.g. 2021 Dec 1st 23:50:59 123456789nsec20211201_235059.123.456.789.pcap
--filename-tstr-HHMMSS_TZ
Wrties the filename in Hour Min Sec with a local timezone suffix
e.g 2022 April 22 19:59 CST
fmadio__2022-04-21_19:59:58-04:00.pcap
--filename-strftime <time string>
Generic strftime print
e.g command line
--filename-strftime "%Y%m%d%H%M%S"
Output is as follows
fmadio__20220421224832.pcap
Command
Description
FilterBPF
Enter a full tcpdump equivlent BPF filter expression
example host filter
FilterBPF="host 192.168.1.1"
Command
Description
Decap
boolean value of "true" enables Packet De-encapsulation (Default true)
Command
Description
gzip -c -1
Run GZIP on split PCAPs with fastest compression mode
gzip -c -9
Run GZIP on split PCAPs in maximum compression mode
lz4 -c
Run LZ4 compression on split PCAPs for fast compression
Command
Description
.pcap
Default suffix.
.pcap.gz
GZIP Compressed PCAP
.pcap.lz4
LZ4 compressed PCAP
true
Enables chunk mode
false
Push from the current capture position
true
Push from the start of the currently active capture
nil
Uses the default system CPU for push operations
<numeric value>
Literal Numeric value indicating which CPU to run stream_cat on
raw (not compressed)
6.9 Gbps
45.5GB
1.0
lz4
2.5Gbps
41.2GB
x 1.104
gzip -1 (fast)
0.16Gbps
40.5GB
x 1..123
gzip default
0.15Gbps
40.2GB
x 1.131
pigz 64 CPU (Parallel GZIP with 64 CPUs)
4.8Gbps
40.3GB
x 1.129
pigz 32 CPU (Parallel GZIP with 32 CPUs)
4.4Gbps
40.3GB
x 1.129
pigz 8 CPU (Parallel GZIP with 8 CPUs)
1.04Gbps
40.3GB
x 1.129
zstd default
0.77Gbps
39.5GB
x 1.15
zstd --fast
2.4Gbps
40.3GB
x 1.129
xz default
0.019Gbps
39.2GB
x 1.160
xz 64 CPU (-T 64)
0.909Gbps
38.9GB
x 1.17
xz 32 CPU (-T 32)
0.502Gbps
39.3GB
x 1.157
raw (not compressed)
4.29Gbps
0.6H
979GB
x 1.0
lz4
1.65Gbps
1.3H
372GB
x 2.629
gzip -1 (fast)
0.341Gbps
6.3H
311GB
x 3.14
gzip (default)
0.129Gbps
16.6H
266GB
x 3.67
pigz 64 CPU (Parallel GZIP)
3.92Gbps
0.55H
264GB
x 3.70
pigz 32 CPU (Parallel GZIP)
3.92Gbps
0.55H
264GB
x 3.70
pigz 16 CPU (Parallel GZIP)
1.98Gbps
1.09H
264GB
x 3.70
pigz 8 CPU (Parallel GZIP)
0.997
2.1 H
264GB
x 3.70
zstd default
1.118Gbps
1.9H
255GB
x 3.83
zstd --fast
1.90Gbps
1.13H
294GB
x 3.32
xz default
0.012Gbps
181H
184GB
x 5.32
xz 64 CPU
0.604Gbps
3.6H
184GB
x 5.32
xz 32 CPU
0.372Gbps
5.8H
184GB
x 5.32
xz 16 CPU
0.192Gbps
11.3H
184GB
x 5.32
xz 8 CPU
0.096Gbps
22.6H
184GB
x 5.32
Advanced capture settings require changing the configuration file directly as follows. As many of these options are customizations, please discuss with support@fmad.io any issues or side effects seen
After modifying time.lua configuration file, please confirm syntax is correct by running the following
The correct output looks like the following
By default the Web user account has full permissions. There may be cases where Web access is for download and analyzing PCAP data only. Where configuration can be not modified.
To disable Web Admin mode SSH into the system and change the setting in
In the ["Security"] section
Change ConfigAccess setting to be "user"
The config file should be checked for syntax errors. Then restart the system. Once restarted the Web GUI no longer has Admin privileges'.
One the massive benefits of a full line rate PCAP replay feature is, you can generate PCAPs at any speed, upload them to the FMAD Packet Replay Device and then replay them at any speed you required. There are many ways to generate PCAP files for replay, we will use our builtin utility pcap_genflow ( https://github.com/fmadio/pcap_genflow however many other tools such as tcpreplay, iperf3 and others which can output to a PCAP file for upload.
is to generate the PCAP using various toolchains
Upload the PCAP into the FMADIO Capture System
Replay the capture at any speed
Some example using pcap_genflow as follows
Generate 1 billion packets, with 1M unique TCP flows using 64B packets @ 100Gbps
Generate 100M packets, with 1M unique TCP flows using 1500B packets @ 100Gbp
Generate 100M packets, with 1M unique TCP flows using IMIX packet size distribution @ 100Gbps
Typically when generating PCAP files the output is written to a linux pipe, the FMAD PCAP File upload function always reads PCAPs from stdin. The tool to upload PCAP into the FMAD capture system is stream_upload. The syntax is
Example of uploading the previously generated file flow_1M_IMIX_100G.pcap. PCAP can be scp to the device, or streamed over an SSH connection
Or this can be generated via on the FMADIO device itself, by piping the output of pcap_genflow directly to stream_upload
Once the PCAP has been uploaded into the capture system, it can be replayed with various options as discussed here. It should be clear PCAP Generation + Upload + Replay is a powerful tool for any Network Engineer. If you have suggestions or questions feel free to contact us.
FMADIO Packet Capture Systems use the default login and password when the system is shipped. Additional WebGUI users can be added manually using the htpasswd utility. To set a new password "password" for the fmadio account use the following command line:
By default this utility overwrites the existing user account, so only 1 user account is possible.
Additional users are added by appending to the /opt/fmadio/etc/htpasswd file. The following shows creating a user account "test" with the password "newpassword".
Please be careful duplicate usernames are not in the /opt/fmadio/etc/htpasswd file. Use a text editor to adjust the file if needed.
The new users and passwords can now access the GUI. In addition for logging the nginx access logs will show the username for all URL requests.
Unfortunately adding additional SSH usernames is not possible, as the permissions may be incorrectly set causing undefined system behavior. However multiple people can login to the system using different SSH keys via the .authorized_keys config file.
The authorized ssh keys file is located in
Please note, the authorized_keys file in the users .ssh account directory does not persist across reboots. Keys must be added to the above location.
Changing the default SSH password uses the standard linux utilit "passwd".
Example below
After setting the new password, it needs to be copied to persistent storage
FMADIO can replay PCAP traffic at low speed while simultaneously capturing at full line rate. This can be very helpful for debugging or other troubleshooting and does not require changing the FPGA firmware from Capture to Replay mode. This Replay mode uses PIO (Programmed IO) who's bandwidth is quite low, and is not suitable for full high bandwidth packet relay. Usage is as follows
Example command on the 20G Packet Capture system
Example below pipes a PCAP previously captured on the device back down the the capture interfaces.
For SKUs FMADIO100v2 and FMADIO40v3
You can pipe a PCAP from a local file system or use stream_cat to pipe from the capture system. Example below pipes from a previous captures.
The following example pipes a PCAP on the localfile system (/mnt/store0/tmp/imix.pcap) to replay the traffic
Scratch Disks are helpful for keeping temporary or persistant data outside of the FMADIO host capture system. Many times the OS disks is unable to be used due to capacity or location.
Scratch Disks can be run as a RAID0 or RAIDx group of drives, allow you the system flexible allocation of the total storage capacity of the system between Capture vs Analytics.
Start by editing the disk configuration file
Then moving serial numbers from the CacheDisk section to the ScratchDisk section.
For example moving a single disk (serial number "22443E9DC54E") from ssd8 -> scr0
Before:
renameing ssd8 to scr0
After:
Save the file.
This can be repeated for as many disks as you require, please keep the numbering sequential e.g. scr0, scr1, scr2, scr3 etc so the system can map it correctly.
Rebooting is required as the system needs to rename the mount points in /opt/fmadio/disk
Create the RAID0 partition using mdmadm as follows. Change the appropriate fields to create a RAID0 2 disk partition or more.
Or for a 2 disk Scratch Partition
The output will look something like below
The partition now exists on /dev/md1 we need to create a filesystem next so it can be used in a general purpose way. Run the following command to create an EXT4 based Scratch Disk partition
Example output looks like below
Next confirm th file system mounts correctly, by creating a test file on the partition
Then creating a file and confirming, such as below
Step 5) Confirm automatic mounting
Reboot the system and confirm the ScratchDisk gets mounted automatically on boot to
The test file created above should exit, example below
In some cases its best to locate the Capture Filesystems Metadata onto the scratch disk. A few extra steps in addition to the above steps are required.
Create a stream directory on the ScratchDisk such as below
Example output looks like the following
Edit the file
Near the end of the file create a one line entry as follows
Example output looks like
Reboot the system
Run the command
Confirm the /opt/fmadio/stream symblic link is pointing to the correct location, example below.
Run a quick format via the GUI to reset the storage partition. This changes the number of drives used for the capture file system.
The above will take about 5 minutes and reboot 2 times by itself.
Check the directory listing
Example looks as follows
After this the file system can be moved along with the diskpacks.
In many cases disabling SSDP is required. This reduces the public visibility of the Packet Capture system.
To do this please use the Redfish API as follows
Get the current BMC Network profile as follows
The output will be similar to this
The import part is extracting the ekey required for the next step. In this case the value is "1655790673"
Next disable SSDP using the ekey above using the command
There is no return value from the call
Use the same command in Step 1 to check the status of SSDP service. It should now be "false"
A summary for developers to access a FMADIO device using the API.
The FMADIO API is simple and designed for easy scripting integration.
Note: Replace the IP 127.0.0.1 with the host IP of your FMADIO device.
GET /sysmaster/capture_start?StreamName=<capture name>
This Command starts a capture running on the device.
GET /sysmaster/capture_stop
Stops any currently capturing process. NOTE: this does NOT stop scheduled captures.
GET /sysmaster/capture_status
Returns status of the capture in JSON format
GET /sysmaster/status
Returns Capture status of currently active capture.
GET /sysmaster/stats_summary
Get system status information. Partial example output in parsed JSON format below
GET /pcap/del?StreamName=<full capture name>
Deletes capture off the system
The FMADIO V1 API uses endpoints with parameters. All V1 versions of the API endpoints shall have the format:
/api/v1/<operation>/<task>?<params=x>
The V1 API has advantages of the previous API. These are:
Multiple downloads can occur at the same time on the same device (up to 4 concurrently).
Improved download performance
Ability to use compressed and bpf filter parameters on most downloads.
Note: All original API url's shall be available as well as the new V1 endpoints.
GET /api/v1/pcap/single
Download entire capture as a single file. Piping to a file or any other analysis tools is possible.
GET /api/v1/pcap/splittime
Gets PCAP from the specified StreamName with Start/Stop EPOCH time with an optional BPF filter
GET /api/v1/pcap/timerange
Download a timerange of pcap data that can cross over a multiple pcap files. The timerange results may be a portion of a single pcap stream, or a portion of multiple streams that share a connected time series.
Examples
TSBegin and TSEnd
curl -u fmadio:xxx "http://127.0.0.1/api/v1/pcap/timerange?TSBegin=1621772572136996000&TSEnd=1621774913584264000"
TSBegin, TSEnd and FilterBPF
curl -u fmadio:xxx "http://127.0.0.1/api/v1/pcap/timerange?TSBegin=1621772572136996000&TSEnd=1621774913584264000" -G --data-urlencode "FilterBPF=tcp"
TSBegin, TSEnd, FilterBPF and Compression
curl -u fmadio:xxx "http://127.0.0.1/api/v1/pcap/timerange?TSBegin=1621772572136996000&TSEnd=1621774913584264000&Compression=fast" -G --data-urlencode "FilterBPF=tcp"
GET /api/v1/system/time_current
Returns the current time on the fpga in epoch nanoseconds. This time/clock is used directly to timestamp packets on the FPGA.
GET /api/v1/system/version
Returns the current system version.
Example output:
fmadio@fmadio100v2-228U:$ curl http://127.0.0.1/api/v1/system/version {"version":"9120","device":"fmadio100v2","build":"Wed Sep 27 03:08:27 2023"} fmadio@fmadio100v2-228U:$
GET /api/v1/system/port_stats
returns RMON1 capture port statistics
Example:
fmadio@fmadio100v2-228U:$ curl -s http://127.0.0.1/api/v1/system/port_stats | jq
{
"cap0":
{ "Pkt": 0, "Byte": 0, "Pkt_RUNT": 0, "Pkt_64": 0, "Pkt_65_127": 0, "Pkt_128_255": 0, "Pkt_256_511": 0, "Pkt_512_1023": 0, "Pkt_1024_1518": 0, "Pkt_1024_2047": 0, "Pkt_2048_4095": 0, "Pkt_4096_8191": 0, "Pkt_8192_9216": 0, "Pkt_OVER": 0 },
"cap1":
{ "Pkt": 0, "Byte": 0, "Pkt_RUNT": 0, "Pkt_64": 0, "Pkt_65_127": 0, "Pkt_128_255": 0, "Pkt_256_511": 0, "Pkt_512_1023": 0, "Pkt_1024_1518": 0, "Pkt_1024_2047": 0, "Pkt_2048_4095": 0, "Pkt_4096_8191": 0, "Pkt_8192_9216": 0, "Pkt_OVER": 0 },
"cap2":
{ "Pkt": 0, "Byte": 0, "Pkt_RUNT": 0, "Pkt_64": 0, "Pkt_65_127": 0, "Pkt_128_255": 0, "Pkt_256_511": 0, "Pkt_512_1023": 0, "Pkt_1024_1518": 0, "Pkt_1024_2047": 0, "Pkt_2048_4095": 0, "Pkt_4096_8191": 0, "Pkt_8192_9216": 0, "Pkt_OVER": 0 },
"cap3":
{ "Pkt": 0, "Byte": 0, "Pkt_RUNT": 0, "Pkt_64": 0, "Pkt_65_127": 0, "Pkt_128_255": 0, "Pkt_256_511": 0, "Pkt_512_1023": 0, "Pkt_1024_1518": 0, "Pkt_1024_2047": 0, "Pkt_2048_4095": 0, "Pkt_4096_8191": 0, "Pkt_8192_9216": 0, "Pkt_OVER": 0 },
"cap4":
{ "Pkt": 0, "Byte": 0, "Pkt_RUNT": 0, "Pkt_64": 0, "Pkt_65_127": 0, "Pkt_128_255": 0, "Pkt_256_511": 0, "Pkt_512_1023": 0, "Pkt_1024_1518": 0, "Pkt_1024_2047": 0, "Pkt_2048_4095": 0, "Pkt_4096_8191": 0, "Pkt_8192_9216": 0, "Pkt_OVER": 0 },
"cap5":
{ "Pkt": 0, "Byte": 0, "Pkt_RUNT": 0, "Pkt_64": 0, "Pkt_65_127": 0, "Pkt_128_255": 0, "Pkt_256_511": 0, "Pkt_512_1023": 0, "Pkt_1024_1518": 0, "Pkt_1024_2047": 0, "Pkt_2048_4095": 0, "Pkt_4096_8191": 0, "Pkt_8192_9216": 0, "Pkt_OVER": 0 },
"cap6":
{ "Pkt": 7237, "Byte": 1936022, "Pkt_RUNT": 0, "Pkt_64": 55, "Pkt_65_127": 559, "Pkt_128_255": 257, "Pkt_256_511": 6136, "Pkt_512_1023": 180, "Pkt_1024_1518": 50, "Pkt_1024_2047": 50, "Pkt_2048_4095": 0, "Pkt_4096_8191": 0, "Pkt_8192_9216": 0, "Pkt_OVER": 0 },
"cap7":
{ "Pkt": 0, "Byte": 0, "Pkt_RUNT": 0, "Pkt_64": 0, "Pkt_65_127": 0, "Pkt_128_255": 0, "Pkt_256_511": 0, "Pkt_512_1023": 0, "Pkt_1024_1518": 0, "Pkt_1024_2047": 0, "Pkt_2048_4095": 0, "Pkt_4096_8191": 0, "Pkt_8192_9216": 0, "Pkt_OVER": 0 },
"port_config": "8x10G"
}
fmadio@fmadio100v2-228U:$
GET /api/v1/system/io_stats
Example:
fmadio@fmadio100v2-228U:$ curl -s http://127.0.0.1/api/v1/system/io_stats | jq
{
"timestamp": "1696925340903886080",
"ioqueue_active": "1"
}
fmadio@fmadio100v2-228U:$
GET /api/v1/system/status
Returns the system status, this is identical to the Telemetry data
Example output:
fmadio@fmadio100v2-228U:~$ curl -s http://127.0.0.1/api/v1/system/status
Resulting JSON blob
{"timestamp":1695785446,"ver":"9120","temperature":{"module":"system","subsystem":"temperature","timestamp":1695785446,"ver":"9120","Temperature_CPU0":55.00,"Temperature_CPU1":70.00,"Temperature_PCH":46.00,"Temperature_SYS":42.00,"Temperature_PER":24.00,"Temperature_NIC":49.00,"Temperature_AirIn":24.00,"Temperature_AirOut":0.00,"Temperature_Transceiver0":42.00,"Temperature_Transceiver1":42.00},"fan":{"module":"system","subsystem":"fan","timestamp":1695785446,"ver":"9120","Fan_SYS0":21450,"Fan_SYS1":21450,"Fan_SYS2":21300,"Fan_SYS3":21450,"Fan_SYS4":21450,"Fan_SYS5":21450,"Fan_SYS6":21600,"Fan_SYS7":21450},"disk":{"module":"system","subsystem":"disk","timestamp":1695785446,"ver":"9120","FreeGB_System":8.977,"FreeGB_Store0":4720.779,"FreeGB_Store1":0.000,"FreeGB_Remote0":46349.530,"FreeGB_Remote1":46349.530,"DiskPresent_os0":true,"DiskTemperature_os0":40,"DiskSMART_os0":0,"DiskPresent_ssd0":true,"DiskTemperature_ssd0":36,"DiskSMART_ssd0":0,"DiskPresent_ssd1":true,"DiskTemperature_ssd1":34,"DiskSMART_ssd1":0,"DiskPresent_ssd2":true,"DiskTemperature_ssd2":35,"DiskSMART_ssd2":0,"DiskPresent_ssd3":true,"DiskTemperature_ssd3":34,"DiskSMART_ssd3":0,"DiskPresent_ssd4":true,"DiskTemperature_ssd4":35,"DiskSMART_ssd4":0,"DiskPresent_ssd5":true,"DiskTemperature_ssd5":40,"DiskSMART_ssd5":0,"DiskPresent_ssd6":true,"DiskTemperature_ssd6":36,"DiskSMART_ssd6":0,"DiskPresent_ssd7":true,"DiskTemperature_ssd7":37,"DiskSMART_ssd7":0,"DiskPresent_par0":true,"DiskTemperature_par0":34,"DiskSMART_par0":0},"link":{"module":"system","subsystem":"link","timestamp":1695785446,"ver":"9120","cap0_link":true,"cap1_link":true,"cap2_link":true,"cap3_link":true,"cap4_link":true,"cap5_link":true,"cap6_link":true,"cap7_link":true,"man0_link":true,"man10_link":true},"io":{"module":"system","subsystem":"io","timestamp":1695785446,"ver":"9120","DiskRdGbps":0.40,"DiskWrGbps":0.25},"capture":{"module":"system","subsystem":"capture","timestamp":1695785446,"ver":"9120","CaptureEnb":true,"CapturePkt":10497009,"CaptureByte":14691374307,"CaptureDrop":0,"CaptureFCSError":0,"CaptureRateGbps":0.237402,"CaptureRateMpps":0.020893,"CaptureName":"wan_colo0_20230927_0320","CapturePort0_Byte":0,"CapturePort0_Pkt":0,"CapturePort1_Byte":14734141814,"CapturePort1_Pkt":10527209,"CapturePort2_Byte":0,"CapturePort2_Pkt":0,"CapturePort3_Byte":0,"CapturePort3_Pkt":0,"CapturePort4_Byte":0,"CapturePort4_Pkt":0,"CapturePort5_Byte":0,"CapturePort5_Pkt":0,"CapturePort6_Byte":0,"CapturePort6_Pkt":0,"CapturePort7_Byte":0,"CapturePort7_Pkt":0},"power":{"module":"system","subsystem":"power","timestamp":1695785446,"ver":"9120","PSU0_Status":false,"PSU1_Status":true,"PSU_PowerWatt":370},"other":{"module":"system","subsystem":"other","timestamp":1695785446,"ver":"9120","UptimeHour":0.33,"MemFree":352248299520,"MemErrorECC":0,"MemCached":17432817664,"MemMapped":5480136704,"MemBuffer":57147392,"MemDirty":0,"PageInByte":0,"FDCnt":1198,"WritebackB":0,"WritebackPct":0.000000,"WritebackDropTotalG":0.00,"WritebackDropG":0.00,"CacheSize":30726047137792,"StoreSize":30725994708992,"CPULoad":5.10,"SerialNo":"undef-undef-e0d55e5d2150","PortConfig":"8x10G","Version":"fmadio100v2:9120pcap2json:715"},"cat":{"module":"system","subsystem":"cat","timestamp":1695785446,"ver":"9120","cat_0_Enable":true,"cat_0_Mode":"FMADRing","cat_0_CPUMain":0,"cat_0_TSPCAP":1695785445,"cat_0_ReadPkt":21098,"cat_0_ReadByte":30071136,"cat_0_ReadTotalPkt":10510527,"cat_0_ReadTotalByte":14779806736,"cat_0_ReadGbps":0.231257,"cat_0_ReadMpps":0.020281,"cat_0_WritePkt":42196,"cat_0_WriteByte":59997961,"cat_0_WriteTotalPkt":10510527,"cat_0_WriteTotalByte":14779806736,"cat_0_WriteGbps":0.461405,"cat_0_WriteMpps":0.040563,"cat_0_PendingByte":30670848,"cat_0_PktDiscard":0,"cat_0_PktDiscardTotal":0,"cat_0_PktSlice":0,"cat_0_IOPriority":20,"cat_0_ChunkID":10996718,"cat_0_CmdLine":"/opt/fmadio/bin/stream_cat--uidpush_pcap_1695784865262465024--follow-start--nop-truncate--ring-eof--ring/opt/fmadio/queue/pcap_ring_all1sec--ring-cpu/opt/fmadio/queue/pcap_ring_all1sec23--ring-filter-bpf/opt/fmadio/queue/pcap_ring_all1sec--ring-filter-frame/opt/fmadio/queue/pcap_ring_all1sec--ring/opt/fmadio/queue/pcap_ring_icmp--ring-cpu/opt/fmadio/queue/pcap_ring_icmp23--ring-filter-bpf/opt/fmadio/queue/pcap_ring_icmpicmp--ring-filter-frame/opt/fmadio/queue/pcap_ring_icmp","cat_0_StreamName":"wan_colo0_20230927_0320","cat_0_FilterBPF":"","cat_0_CPUIdle":0.7622,"cat_0_CPUFetch":0.0154,"cat_0_CPUProcess":0.0154,"cat_0_CPUSend":0.0000,"cat_1_Enable":true,"cat_1_Mode":"FMADRing","cat_1_CPUMain":0,"cat_1_TSPCAP":1695785443,"cat_1_ReadPkt":199179,"cat_1_ReadByte":283074384,"cat_1_ReadTotalPkt":9743856,"cat_1_ReadTotalByte":13763158272,"cat_1_ReadGbps":0.226272,"cat_1_ReadMpps":0.019901,"cat_1_WritePkt":199179,"cat_1_WriteByte":281761649,"cat_1_WriteTotalPkt":9743856,"cat_1_WriteTotalByte":13763158272,"cat_1_WriteGbps":0.225223,"cat_1_WriteMpps":0.019901,"cat_1_PendingByte":33030144,"cat_1_PktDiscard":0,"cat_1_PktDiscardTotal":0,"cat_1_PktSlice":0,"cat_1_IOPriority":20,"cat_1_ChunkID":10996554,"cat_1_CmdLine":"/opt/fmadio/bin/stream_cat--uidpush_lxc_1695784924583593984--follow--ring/opt/fmadio/queue/lxc_market2json_euronext_sbe--ring-filter-bpf/opt/fmadio/queue/lxc_market2json_euronext_sbevlan177andnet224.0.208.0/24--ring-filter-frame/opt/fmadio/queue/lxc_market2json_euronext_sbe--cpu23-v--print-period10e9","cat_1_StreamName":"wan_colo0_20230927_0320","cat_1_FilterBPF":"","cat_1_CPUIdle":0.9921,"cat_1_CPUFetch":0.0071,"cat_1_CPUProcess":0.0071,"cat_1_CPUSend":0.0000,"cat_2_Enable":false,"cat_2_Mode":"","cat_2_CPUMain":0,"cat_2_TSPCAP":0,"cat_2_ReadPkt":0,"cat_2_ReadByte":0,"cat_2_ReadTotalPkt":0,"cat_2_ReadTotalByte":0,"cat_2_ReadGbps":0.000000,"cat_2_ReadMpps":0.000000,"cat_2_WritePkt":0,"cat_2_WriteByte":0,"cat_2_WriteTotalPkt":0,"cat_2_WriteTotalByte":0,"cat_2_WriteGbps":0.000000,"cat_2_WriteMpps":0.000000,"cat_2_PendingByte":0,"cat_2_PktDiscard":0,"cat_2_PktDiscardTotal":0,"cat_2_PktSlice":0,"cat_2_IOPriority":0,"cat_2_ChunkID":0,"cat_2_CmdLine":"","cat_2_StreamName":"","cat_2_FilterBPF":"","cat_2_CPUIdle":0.0000,"cat_2_CPUFetch":0.0000,"cat_2_CPUProcess":0.0000,"cat_2_CPUSend":0.0000,"cat_3_Enable":false,"cat_3_Mode":"","cat_3_CPUMain":0,"cat_3_TSPCAP":0,"cat_3_ReadPkt":0,"cat_3_ReadByte":0,"cat_3_ReadTotalPkt":0,"cat_3_ReadTotalByte":0,"cat_3_ReadGbps":0.000000,"cat_3_ReadMpps":0.000000,"cat_3_WritePkt":0,"cat_3_WriteByte":0,"cat_3_WriteTotalPkt":0,"cat_3_WriteTotalByte":0,"cat_3_WriteGbps":0.000000,"cat_3_WriteMpps":0.000000,"cat_3_PendingByte":0,"cat_3_PktDiscard":0,"cat_3_PktDiscardTotal":0,"cat_3_PktSlice":0,"cat_3_IOPriority":0,"cat_3_ChunkID":0,"cat_3_CmdLine":"","cat_3_StreamName":"","cat_3_FilterBPF":"","cat_3_CPUIdle":0.0000,"cat_3_CPUFetch":0.0000,"cat_3_CPUProcess":0.0000,"cat_3_CPUSend":0.0000,"cat_4_Enable":false,"cat_4_Mode":"","cat_4_CPUMain":0,"cat_4_TSPCAP":0,"cat_4_ReadPkt":0,"cat_4_ReadByte":0,"cat_4_ReadTotalPkt":0,"cat_4_ReadTotalByte":0,"cat_4_ReadGbps":0.000000,"cat_4_ReadMpps":0.000000,"cat_4_WritePkt":0,"cat_4_WriteByte":0,"cat_4_WriteTotalPkt":0,"cat_4_WriteTotalByte":0,"cat_4_WriteGbps":0.000000,"cat_4_WriteMpps":0.000000,"cat_4_PendingByte":0,"cat_4_PktDiscard":0,"cat_4_PktDiscardTotal":0,"cat_4_PktSlice":0,"cat_4_IOPriority":0,"cat_4_ChunkID":0,"cat_4_CmdLine":"","cat_4_StreamName":"","cat_4_FilterBPF":"","cat_4_CPUIdle":0.0000,"cat_4_CPUFetch":0.0000,"cat_4_CPUProcess":0.0000,"cat_4_CPUSend":0.0000,"cat_5_Enable":false,"cat_5_Mode":"","cat_5_CPUMain":0,"cat_5_TSPCAP":0,"cat_5_ReadPkt":0,"cat_5_ReadByte":0,"cat_5_ReadTotalPkt":0,"cat_5_ReadTotalByte":0,"cat_5_ReadGbps":0.000000,"cat_5_ReadMpps":0.000000,"cat_5_WritePkt":0,"cat_5_WriteByte":0,"cat_5_WriteTotalPkt":0,"cat_5_WriteTotalByte":0,"cat_5_WriteGbps":0.000000,"cat_5_WriteMpps":0.000000,"cat_5_PendingByte":0,"cat_5_PktDiscard":0,"cat_5_PktDiscardTotal":0,"cat_5_PktSlice":0,"cat_5_IOPriority":0,"cat_5_ChunkID":0,"cat_5_CmdLine":"","cat_5_StreamName":"","cat_5_FilterBPF":"","cat_5_CPUIdle":0.0000,"cat_5_CPUFetch":0.0000,"cat_5_CPUProcess":0.0000,"cat_5_CPUSend":0.0000,"cat_6_Enable":false,"cat_6_Mode":"","cat_6_CPUMain":0,"cat_6_TSPCAP":0,"cat_6_ReadPkt":0,"cat_6_ReadByte":0,"cat_6_ReadTotalPkt":0,"cat_6_ReadTotalByte":0,"cat_6_ReadGbps":0.000000,"cat_6_ReadMpps":0.000000,"cat_6_WritePkt":0,"cat_6_WriteByte":0,"cat_6_WriteTotalPkt":0,"cat_6_WriteTotalByte":0,"cat_6_WriteGbps":0.000000,"cat_6_WriteMpps":0.000000,"cat_6_PendingByte":0,"cat_6_PktDiscard":0,"cat_6_PktDiscardTotal":0,"cat_6_PktSlice":0,"cat_6_IOPriority":0,"cat_6_ChunkID":0,"cat_6_CmdLine":"","cat_6_StreamName":"","cat_6_FilterBPF":"","cat_6_CPUIdle":0.0000,"cat_6_CPUFetch":0.0000,"cat_6_CPUProcess":0.0000,"cat_6_CPUSend":0.0000,"cat_7_Enable":false,"cat_7_Mode":"","cat_7_CPUMain":0,"cat_7_TSPCAP":0,"cat_7_ReadPkt":0,"cat_7_ReadByte":0,"cat_7_ReadTotalPkt":0,"cat_7_ReadTotalByte":0,"cat_7_ReadGbps":0.000000,"cat_7_ReadMpps":0.000000,"cat_7_WritePkt":0,"cat_7_WriteByte":0,"cat_7_WriteTotalPkt":0,"cat_7_WriteTotalByte":0,"cat_7_WriteGbps":0.000000,"cat_7_WriteMpps":0.000000,"cat_7_PendingByte":0,"cat_7_PktDiscard":0,"cat_7_PktDiscardTotal":0,"cat_7_PktSlice":0,"cat_7_IOPriority":0,"cat_7_ChunkID":0,"cat_7_CmdLine":"","cat_7_StreamName":"","cat_7_FilterBPF":"","cat_7_CPUIdle":0.0000,"cat_7_CPUFetch":0.0000,"cat_7_CPUProcess":0.0000,"cat_7_CPUSend":0.0000,"cat_EnableCnt":2,"cat_ReadPkt":220277,"cat_ReadByte":313145520,"cat_ReadTotalPkt":20254383,"cat_ReadTotalByte":28542965008,"cat_ReadGbps":0.457529,"cat_ReadMpps":0.457529,"cat_WritePkt":241375,"cat_WriteByte":341759610,"cat_WriteTotalPkt":20254383,"cat_WriteTotalByte":28542965008,"cat_WriteGbps":0.686627,"cat_WriteMpps":0.686627},"ptp":{"module":"system","subsystem":"ptp","timestamp":1695785446,"ver":"9120","TimeFPGA":1695785446482977592,"TimeSYS":1695785446286598912,"GMOffset":0.00,"GMSync":false,"GMMaster":"","SysOffset":0.00,"SysSync":false,"iSysOffset":0.00,"iSysSync":true,"NTPOffset":0.00,"NTPSync":false,"NTPMaster":"","GMUpTime":0,"SysUptime":0,"iSysUptime":1061,"PPSUptime":690,"NTPUptime":0,"PPSPeriod":6.399925850,"PPSOffset":-867,"PPSdPhase":2065,"PPSdZero":-867,"PPSCnt":1225,"Clk156Period":0.000000,"Clk156Offset":0,"Clk250Period":0.000000,"Clk250Offset":0,"Clk322Period":0.000000,"Clk322Offset":0}}
Pretty print output
NOTE: These interfaces are legacy, recommend using the V1 API interfaces
GET /stream/list
Lists all captures on the device.
GET /stream/ssize?StreamName=<capture sname>&StreamView=<split mode>
Lists splits for a specific capture based on file size. Usually this is a 2 step process of 1) get the split list 2) download a specific split.
GET /stream/stime?StreamName=<capture sname>&StreamView=<split mode>
Lists splits for a specific capture based on a time unit. Usually this is a 2 step process of 1) get the split list 2) download a specific split
GET /pcap/single?StreamName=<capture name>&FilterRE=<string>
Download entire capture as a single file. Piping to a file or any other analysis tools is possible.
Compression example:
curl -u fmadio:100g "http://192.168.2.75/pcap/single?StreamName=TestCapture_20180702_1127&Compression=fast"
FilterBPF example:
curl -u fmadio:100g "http://192.168.2.75/pcap/single?StreamName=hitcon_20180702_1503_58&" -G --data-urlencode "FilterBPF=tcp"
GET /pcap/splittime?StreamName=<string>&Start=<int>&Stop=<int>&FilterBPF=<string>&FilterPort=<int>
Download the capture with a time filter. Note: the nanosecond Epoch Start is 1530498788000000000. Removing the nanosecond part convert epoch to date/time.
GET /pcap/timerange?TSBegin=<epoch start>&TSEnd=<epoch stop>&TSMax=<size>&TSMode=<nanos or msecs>
Download a timerange of pcap data without any capture file referenced. The system will search all captures for the specified timerange. At most it can cross two pcap files
fmadiocli is a command line interface using a switch style interface to operate FMADIO Packet Capture devices.
Brief help description is provided using ? command
Verbose help description is provided using ??? command
Command history is stored in the file
It can be deleted to clear the history
FW: 7856+
Shows the current state of the interfaces
Example below shows port status on an FMADIO100Gv2 Analytics system
FW: 8336+
Shows the currently configured IP address information for the management and BMC/IPMI/Capture ports
Example below shows the status on an FMADIO20Gv3 system
Shows RMON1 counter information on each capture port.
Example below is FMADIO20Gv3 system output
Example below shows FMADIO100v2 in 8x10G port output
FW: 7856+ support for 100Gv2 2x100G 2x40G
This shuts down a specific capture interface as specified, usually this is cap0 or cap1 and depends on the SKU and Port configuration on which ports can be shutdown
FW: 7856+ support for 100Gv2 2x100G 2x40G
Re-enables the specified capture interface from shutdown status. Depending on the link peer, the link peer might need to be bounced as it may be in a shutdown error state.
FW: 8224+ ( 2x100G only)
This forces FEC on the specific capture port. FEC Autoneg is disabled. This setting is persistent across reboots.
FW: 8224+ ( 2x100G only)
This disables the forced FEC setting where the system will try autoneg if FEC is enabled or not. Setting is persistent across reboots.
Configures the IP address of the specified network port. Typically this is used for setting the management/BMC IP address of the system
Example below sets the man0 port to IP address 192.168.187.10. The exact output may vary between SKUs
Example below shows setting the IP address of the BMC/IPMI port
Sets the netmask of the specified interface
Sets the default gateway for the specified interface
Example below sets the man0 management interfaces default gateway address
Sets the DNS server for the specified interface
Example below sets the DNS server for man0 interface to be 1.1.1.1
Shows the current capture status
Shows the current capture schedule
Displays list of all captures on the system
FW: 8367+
Shows the current capture roll setting
FW: 8367+
Shows the current capture flushing behaviour
Starts a capture with the specified name
Use config capture status to verify the current state
Stops the currently active capture
NOTE: This will only stop captures manually started, for scheduled captures please disable the schedule entry to stop the capture
Use config capture status to verify the current state
FW: 8367+
Sets the capture flushing behavior.
Default setting is flush 1sec after capture is idle
Flush always every 1 second. NOTE: 1sec is very aggressive mode and will consume additional storage. However it does provide low latency when watching low bandwidth captures.
Flush when capture is idle for >= 1sec
FW: 8367+
Configures the capture rolling behavior. Default (0) is roll at midnight.
Example configures capture to roll every 1 hour.
Shows the current PCAP timestamp mode. e.g. from the FMADIO FPGA or extract timing information from a packet broker
Example below shows the PCAP timestamp uses the Arista 7130 (Metamako) footer timestamp
Configures the default PCAP timestamp mode when downloading PCAP data. this value can be overidden by URI option TSMode.
Supported Timestamp Modes
nic - FMADIO FPGA timestamp
arista7130 - Arista 7130 (Metamako) Footer
arista7150_overwrite - Arista 7150 Overwite FCS + Keyframes
arista7150_insert - Arista 7150 Insert 32bit + Keyframes
arista7280_mac48 - Arista 7280 Source MAC 48bit Overwrite
arista7280_eth64 - Arista 7280 Ethernet Insert 64bit
erspanv3 - Cisco ERSPANv3 Encapsulation
cisco3550 - Cisco 3550 (Exablaze) Footer
Help command
Example to set the default behavior to use Arista 7130 footer. It takes 60sec to restart the processes after the setting.
Following provides commands for configuration and managing LXC containers on the system
Shows the current LXC container status of the system
Example shows 2 containers "suricata" and "centos7"
Suricata is enabled to start at boot time.
Adds an already installed container to the configuration file
In the example below adding an already installed container named "ubuntu20" to the system
Removes the specified container from the configuration
NOTE it does not delete the container on disk. Only removes it from the configuration files
Example deletes the container "ubuntu20" from the configuration files
Adds a human description to the lxc to provide context
Example set a human readable description for the container "ubuntu20" this is visibile when using the show lxc command
Sets the specified container to boot on startup
Example set the "ubuntu20" container to boot on system startup
Sets the specified LXC container to not boot at startup
Example sets the LXC container "ubuntu20" to not boot on startup
Starts the specified container named <lxc name> if the container starts successfully system will return back to the prompt
Example starts the fshark2 container on the system
When a container fails to start the output will look similar to below.
Stops the specified container from running
Example stops the fshark2 container running
TBD lists all available containers in the public repo
contact support@fmad.io for more info
TBD installs the specified container from the public repo
contact support@fmad.io for more info
TBD removes the specified container from the configuration and disk
contact support@fmad.io for more info
Configure and monitor the automatic push generation of PCAPs to storage locations.
FW: 7963+
Shows the currently configured automatic push pcaps
FW: 7963+
Creates a new push pcap target called <push target>
NOTE: all target names should be unique
Deletes the current push pcap entry name <push target>
Renames the specified <push target> entry with an updated one <new name>
Updates the push write path to the specified <new write path>. Typically this is the NFS remote path or rclone write path.
Configure PCAPs to be split by the specified time value. By default <value> is scientific notation in nanoseconds. In addition s (seconds) m (minutes) h (hours) suffix can be used also
Example configure to split every 1 minute
Configure PCAPs to be split by total byte size <value>
Example below shows splitting on 1GB boundaries
Specifies the filename format for each individual split PCAP
Example uses a simple Hour Min Sec format
Sets the specified push with a BPF filter.
NOTE: the BPF filter must be enclosed in double quotes
Example sets for udp and port 1900
Shutsdown the current push procesess and restarts them
Example output
The system can push automatically into a lxc_ring enabling a container to consume the data. These functions are to add/delete/modify these push functions.
NOTE this requires the push_lxc analytics script to be running
Shows the current push lxc targets configured on the system
Example shown below, indicates a single suricata ring is enabled with a BPF filter to remove all traffic from subnet 192.168.100.0/24
This adds a new LXC push to the ring named <ring name>.
By default the push is disabled when created.
Example below shows adding a push to the ring named "general"
NOTE this does not create the ring, it only creates the push to the specified ring
Removes the specified LXC push target
Example removes the push lxc target named "general"
Enables the specified lxc ring push target. By default when adding a new target the state is disabled.
Example enables the push lxc ring target named "general"
Disables the specified lxc push <ring name>
Example disables the lxc ring named "general"
Adds the specified BPF filter to the LXC push to the ring.
NOTE The filter must be enclosed in double quotes ""
Example adds a subnet "192.168.0.0/24" filter to the ring named "general"
Adds the specified frame filter to the lxc push to the ring.
NOTE the filter must be enclosed in double quotes ""
Example add a Frame filter of capture port 0 only to the ring named "general"
Sets the push to start from the current capture position into the lxc ring.
This is the default behaviour
Example sets the ring "general" to push data into the ring from now.
Sets the push to start from the beginning of the capture.
Example sets the ring "general" to start from the begnining of the capture
This shutsdown and then restarts the push lxc processes.
Example output
Various functions for monitoring the ring status both push pcap and push lxc
Shows all rings status information
Example, this can be helpful for monitoring data is being produced and consumed correctly.
Various functions for configuration and monitoring time
Shows the current timezone the system is configured
Example showing the current timezone
Configures the timezone by searching the timezone list for the location named "<city>"
System uses the first found match
For cities with spaces in the name, ensure to use double quotes around the city name
Example set the timezone to New York
NOTE change only takes effect on next reboot
FW: 8336+
FMADIO Web GUI supports multiple users with 2 levels of access
Using fmadiocli to setup and configure is shown below
This shows the currently configured list of users on the system
Example output, it shows 2 users fmadio (full access), bob (user access)
Adds a new user with default permissions and no password
Example adds the username "bob" to the system
Deletes the specified username
Example below deletes the username "bob"
Sets the WEB user password. This has no effect on SSH access to the system
Example below sets the web password for user "bob"
Sets the userlevel permission for the specified username
Level types are 2
full - provides full unrestricted GUI access
user - provides download and analysis only access (no configuration or capture state change)
Example below shows setting the username "bob" to be a "user" level (e.g. can not change system configuration or capture states)
Example below shows setting username "bob" to be a full access user (e.g. can change any configuration using the GUI)
Various commands to set and modify the security settings of the system
Shows the current security settings
This sets the authentication method of the system. Number of options as follows
BASIC - this is basic authencation, low security level
OAUTH - OAUTH 2.0 includding Active Directory, Google, Ping Identity
RADIUS - Use Radius based authentication
PAM-LDAP - Use the linux PAM system with an LDAP authentication mode
Example to set PAM-LDAP as follows
Output as follows
For some authentication methods it requires a system reboot. In this case a reboot is required as the system needs to start LDAP client daemons.
This enables/disables HTTP as a mode of access to the device. HTTP is plain clear text transport protocol, meaning all private data such as username and password are sent in the clear.
For private and secure networks this is ok(ish) for most situations HTTP should be disabled, allowing only HTTPS as the mode of access.
To disable HTTP access (HTTPS only)
Example output
This sets the SSH idle timeout timeout value. Use "show security" to validate the value is correct.
Time units supported are
An example of setting a 30 sec idle timeout as follows
With the following output
NOTE: the system requires a reboot for the changes to take effect.
This sets the WWW session timeout value. This is the maximum session duration. Once the session duration is reached the web interface will require a re-login.
Time units supported are
An example setting a 1 hour maximum session timeout
Example output
NOTE: a the system requires a reboot for the changes to take effect.
Configuration and status information for disks and disk encryption
NOTE: the default password is stored in
This may or maynot include whitespace charaters such as 0xa. Which may cause confusion about the password entered vs the saves on disk password.
When editing this file in VIM we recommend setting, to avoid any additional whitespace charaters in the pawwrod
Shows the current disk status information
Example below shows a fully setup 100Gp3 system with PSID and Encryption enabled
Using TCG OPAL2 sedutils the system will factory reset the device using the PSID values, initialize the drives for encryption and set a default password.
When complete the drives data is encrypted with a default password to access a randomly generated AES256 encryption key.
When complete the drives are in the unlocked state. To enable locking use the config disk lock comand
Example shows a partial log of the 100G systems sanitize operation. Entire operation takes about 60 seconds
Changes the password used for all encryption related disk operations.
Enter Old Password may be ENTER/NULL in which case the default password will be used
Enter Old Password can be read without keyboard input from the file /tmp/disk-password-old if the file exists
Enter New Password can be read without keyboard input from the file /tmp/disk-password if the file exists
Example of setting a new password from the default password
This sets the disks into the locked state requiring a password.
On any power cycle command (disks loose power) the disks will need to be unlocked using the config disk unlock command and a password
The password can be read without keyboard input from the file/tmp/disk-password if the file exists
Example locks all data disks
This removes the disk locking function of the drives.
Example below shows a disk no-lock operation
Unlocks the drives using the specified password
The password can be read without keyboard input from the file /tmp/disk-password if the file exists
Example of unlocking
This configures the systems FPGA firmware, currently support modes
capture-2x100G (2 x100G packet capture mode)
capture-2x40G (2x40G packet capture mode)
capture-8x10G (8x10G packet capture mode)
NOTE: this only sets up the system. Reboot is required to start the configuration change
Example:
Output
stream_cat is the core utility to extract data off the system. By default it outputs a standard nanosecond PCAP to stdout. This can be piped in multiple ways per the unix philosophy
CLI argument reference
Pins stream_cat to a specific CPU number.
Writes PCAP to the specified LXC <ring path> when the <bpf filter> matches.
Multiple rings can be specified
NOTE: if no BPF is used <bpf filter> needs to be ""
NOTE: If no CPU is specified, use setting of "0"
All fields must be populated.
Example:
Reset the LXC Ring read/write pointers
Adjusted the size of the ring FIFO depth. This value must be a Power of 2. Maximum value is 1024
Default value 8
Default: 30e9 (30 seconds)
Sets the default LXC Ring timeout value in nanoseconds.
Used in combination with capinfos2 Generates a histogram of the time between packets displaying it in a vertical histogram form.
Default bin size is 1nsec
Default offset is 0nsec
Example output is shown below:
Above example uses stream_cat with an epoch and BPF filter to isolate the packet histogram deltas between packets. This is particularly useful for checking QoS SLAs
Used with capinfos2 it specifes the width of each timebin (e.g. the histogram resolution). By default it uses 1nsec. Example usage below, this uses a 1e6 (1 millisecond) time bin with a 10msec offset.
As the number of timebins is limited, it may be nessecarry to offset the histogram to where the data is. The example below offsets it by 10msec with a time bin of 1msec.
Filters the specified capture using start time specified argument epoch time value.
Value of 0 means filter is disabled
NOTE: typically --epoch-start and --epoch-stop are used together
Example: filter from epoch 1497015595000000000. This uses capinfos2 to verify the first packed (Time First) is as specified in the filter
Filters the specified capture using and end time specified argument epoch time value.
Value of 0 means filter is disabled
NOTE: typically --epoch-start and --epoch-stop are used together
Example: filter up to epoch time 1497015594000000000. This example uses capinfos2 to verify the last packet (Time Last) meets the specified filter value.
The following section shows how to use stream_cat on the command line in various different ways.
Where test_capture is used, replace with a stream capture name from your fmadio system.
Where sample_file.pcap is used, replace with your own pcap filename.
To create a whole pcap of an entire fmadio system capture use the following:
To choose a selection of time for a pcap on the fmadio system the following can be used. The following example selects a time period using epoch nano seconds. 1000 nanoseconds of capture time will be extracted - assuming the stream was captured during this epoch period.
The examples here show some simple filter examples.
Stream_cat with a IP and UDP filter:
Stream_cat with a UDP port 80 filter:
Stream_cat with a complex filter - select port 80 packets with tcp range selectors :
Stream_cat is very useful for piping output to other programs to process the data. Examples are shown in the stream_cat --help. The example here shows stream_cat used with gzip to compress the output pcap into a smaller sized file.
streamcat can write directly to an lxc ring buffer located in /opt/fmadio/queue/lxc_ring0
It can also write to multiple lxcring buffers from a single stream_cat instance by issuing the --ring command line multiple times.
This example writes all captured data to a single LXC Ring with no BPF filter applied
NOTE: the "" arguments are required. This indicates a NULL BPF filter
The following example writes to a single LXC Ring with a simple "tcp" BPF filter
Example below shows a single stream_cat instance writing to 4 separate LXC Ring rings each with a different BPF Filter
NOTE: above should be a single line
Advanced capture settings require changing the configuration file directly as follows. As many of these options are customizations, please discuss with support@fmad.io any issues or side effects seen
After modifying time.lua configuration file, please confirm syntax is correct by running the following
The correct output looks like the following
By default the capture system will automatically stop all captures at midnight per the local timezone set on the capture device. This can be disabled which is helpful for capture periods which are < 24H but cross the midnight boundary.
In the ["Scheduler"] section
Add the entry set to true, to disable the Midnight roll
The config requires capture to be stopped and started for the the new setting to load.
In some cases capturing across midnight and rolling at a different time has advantages, such as Futures/Options exchanges who typically run 23H a day restarting very early in the morning.
This is achieved using the ManualOffset to offset when the capture rolls. To roll the capture as 5AM every day, setting 5 H * 60min * 60sec * 1e9 nanoseconds
Then setting the scheduler to start/stop at 00:00:59 and 23:59:59 as follows
This will create captures from 05:00AM until 04:59:59 the next day
StartTime = ManualOffset (5H) + Scheduler Start Time 00:00 = 05:00 AM
StopTime = ManualOffset (5H) + Scheduler Stop Time 23:59:59 = 04:59AM + 1 Day
FMADIO100G Capture system is for 100% lossless half duplex 100G link.
The system can write 100Gbps+ to Disk and 148M Packets+ all without any loss. This assumes data is on a single 1x100G capture port
As the system has 2x100G capture ports there is a point where due to data rate or per packet rate, the system will drop packets, once it exceed 148M packets / sec or 100Gbps data rate.
The following graphs show the performance characteristics when exceeding 100Gbps line rate
FMADIO100G Packet Capture system is fully lossless on a single 1 x 100G capture port across all packet sizes. Including up to 9218 Jumbo frame sizes.
Microburst of 100 packets at 2 x 100G line rate from 64B to 1500B packets.
Microbursts @ 200Gbps up to 100packets in length is 100% full capture no loss.
Microbust of 200 packets per port 2 x 100G line rate from 64B to 1500B packets. Slight capture loss on the larger packet sizes.
Microburst of 500packets per port 2 x 100G line rate from 64B to 1500B packets. Start to see more significant packet loss.
Microburst of 1K packets per port 2 x 100G line rate from 64B to 1500B packets.
Microburst of 10K packets per port 2 x 100G line rate 64B to 1500B packets
Microburst of 100K packets per port 2 x 100G line rate 64B to 1500B packets
Microburst of 1M Packets per port 2 x 100G line rate 64B to 1500B packets
Stream_cat can be executed with packet filtering commands. These are similar to the filter methods used by . Example filters are also available in the .
StreamName
string
full name of the capture file to be deleted
FilterBPF
string
BPF Filter to be applied to the stream.
Compression
string
Compress the returned stream with gzip. 'fast' Fastest compression but not smallest. 'best' Slowest compression smallest size. 1-9 The range from 'fast' to 'best'
StreamName*
string
Stream capture name.
FilterFrame
String
Filter on the Packet Frame
a7130.device==<device id>
a7130.srcport==<port id>
c3550.srcport==<portid>
capture.port==<portid>
TSMode
String
Sets the Timestamp of the PCAP
nic - FMADIO FPGA timestamp
arista7130 - Arista 7130 (Metamako)
arista7150_overwrite - Arista 7150 FCS Overwrite
arista7150_insert - Arista 7150 Insert 32bit
arista7280_eth64 - Arista 7280 Ethernet 64bit header
arista7280_mac48 - Arista 7280 SrcMAC 48bit Overwrite
cisco_erspan3 - Cisco ERPSANv3
cisco3550 - Cisco 3550 (Exablaze)
FilterBPFDecap
string
true : Run BPF filter on a De-Encapsulated version of the packet
false : Run BPF on raw packet
Default: false
StreamName*
String
Capture name to fetch from
Start*
String
EPOCH Nanosecond start time
Stop*
String
EPOCH Nanosecond stop time
FilterBPF
String
Escape Encoded BPF filter
FilterFrame
String
Filter on the Packet Frame
a7130.device==<device id>
a7130.srcport==<port id>
c3550.srcport==<portid>
capture.port==<portid>
TSMode
String
Sets the Timestamp of the PCAP
nic - FMADIO FPGA timestamp
arista7130 - Arista 7130 (Metamako)
arista7150_overwrite - Arista 7150 FCS Overwrite
arista7150_insert - Arista 7150 Insert 32bit
arista7280_eth64 - Arista 7280 Ethernet 64bit header
arista7280_mac48 - Arista 7280 SrcMAC 48bit Overwrite
cisco_erspan3 - Cisco ERPSANv3
cisco3550 - Cisco 3550 (Exablaze)
FilterBPF
string
Compression
string
Compress the returned stream with gzip. 'fast' Fastest compression but not smallest. 'best' Slowest compression smallest size. 1-9 The range from 'fast' to 'best'
TSMax
integer
Maximum nanosecond of packets to download.
TSUnit
string
Time Range mode to use for time selection
nanos : Nanoseconds (default)
msecs : Milliseconds
sec : Seconds
YYYYMMDD_HHMMSS: this year year month day hour min second time format
TSBegin*
integer
Start time in nanoseconds epoch.
TSEnd*
integer
Stop time in nanoseconds epoch.
FilterFrame
String
Filtering based on frame parameters
capture.port==0,1
(fetch data for capture ports 0 and 1 only)
capture.port!=0
(fetch data for capture ports NOT equal to 0)
a7130.srcdevice!=0
(fetch data for anything that has a valid Arista 7130 device id)
a7130.srcdevice==0
(fetch anything that does NOT have a valid Arista 7130 device id)
a7130.srcdevice=54931 and a7130.srcport=1
(fetch data only for Arista 7130 device id 54931 and Arista 7130 port id 1)
a7130.srcdevice==54931 and a7130.srcport=1,2,3,4
(fetch data for Arista 7130 device id 54931 and Arista 7130 ports 1, 2, 3, 4)
a7130.srcdevice==54931 and a7130.srcport!=1
(fetch data for Arista 7130 device ie 54931 and all ports except port 1)
TSMode
String
Sets the Timestamp of the PCAP
nic - FMADIO FPGA timestamp
arista7130 - Arista 7130 (Metamako)
arista7150_overwrite - Arista 7150 FCS Overwrite
arista7150_insert - Arista 7150 Insert 32bit
arista7280_eth64 - Arista 7280 Ethernet 64bit header
arista7280_mac48 - Arista 7280 SrcMAC 48bit Overwrite
cisco_erspan3 - Cisco ERPSANv3
cisco3550 - Cisco 3550 (Exablaze)
FilterBPFDecap
string
true : Run BPF filter on a De-Encapsulated version of the packet
false : Run BPF on raw packet
Default: false
StreamView
string
Stream time slice name split_10MB split_100MB split_250MB split_1GB split_2GB split_5GB split_10GB split_100GB split_1TB
StreamName
string
Stream capture name
StreamView
string
Split options for the time split split_1sec split_10sec split_1min split_10min split_15min split_1hour split_2hour split_4hour split_6hour split_8hour split_12hour
StreamName
string
Stream capture name.
FilterRE
string
Download the capture with using a RegEx DPI filter.
FilterBPF
string
BPF Filter to be applied to the stream
Compression
string
Compress the returned stream with gzip. 'fast' Fastest compression but not smallest 'best' Slowest compression smallest size 1-9 The range from 'fast' to 'best'
StreamName
string
Stream capture name.
FilterPort
integer
Download the capture specifying the port capture number.
FilterBPF
string
BPF Filter to be applied to the stream.
StreamName*
string
Stream capture name.
Stop*
integer
Stop time in nanoseconds epoch.
Start*
integer
Start time in nanoseconds epoch.
TSMax
integer
Maximum nanosecond of packets to download.
TSMode
string
Time Range mode to use for time: nsec | epoch in nano seconds (default) usec | epoch in micro seconds msec | epoch in milli seconds sec: | epoch in seconds
TSBegin*
integer
Start time in epoch (default nano seconds)
TSEnd*
integer
Stop time in epoch (default nano seconds).
1e9
1 second in scientific notation
60s
60 seconds
1m
1 minute
1h
1 hour
1e9
1 Gigabyte specified in scientific notation
100M
100 Megbyte
5G
5 Gigabyte
epoch-sec
_1654610221.pcap
Second Epoch
epoch-sec-startend
_1654610221-1654620221.pcap
Epoch start and End
epoch-msec
_1654610221012.pcap
Epoch in msec
epoch-usec
_1654610221012345.pcap
Epoch is micro sec
epoch-nsec
_1654610221012345678.pcap
Epoch in Nano sec
HHMM
_20200101_1201.pcap
Hour Min
HHMMSS
_20200101_120159.pcap
Hour Min Sec
HHMMSS_TZ
2020-01-01_12:01:59+09:00.pcap
House Min Sec + Timezone
HHMMSS_NS
_20200101_120159.012345678.pcap
House Min Sec Nanos
full
Provides full admin level access to all functions
user
User level only, no ability to modify config and start/stop captures
StreamName
string
Stream capture name
Mounting the host systems file system into the LXC container can be very helpful in many situations.
Easiest way is via the lxc config file example shows mounting the host /mnt/store1 directory directly into the container. Please ensure rootfs/mnt/store1 directory has been created within the container.
NOTE that it is critical to have no leading "/" in the container mount point (making it a relative mount point)
https://wiki.debian.org/LXC#Bind_mounts_inside_the_container
Download performance examples using the CLI interface to extract PCAP data to downstream systems.
Please use "stream_dump" to list all captures which can be used with
the output of stream_cat is a standard PCAP via the stdout pipe
NOTE: throughput is heavily dependent on the packet size mix of the capture. Lager packet sizes get a better over all throughput
Transport
Physical
Interface
64B
Packet
9200B Jumbo
Packet
IMIX
Packet
Local File System
local
8.8Gbps
Local File System (rclone)
local
2.4Gbps
NFS Remote
10G
5.4Gbps
SSH Pipe
1G
0.48Gbps
SSH Pipe
10G
048Gbps
SSH Pipe (arcfour)
10G
1.2Gbps
NetCat TCP
10G
8.8Gbps
HTTP localhost
local
6.2Gbps
HTTPS localhost
local
3.3Gbps
HTTP remote
1G
0.9Gbps
HTTP remote
10G
4.0Gbps
HTTP remote (MTU 9182)
10G
4.4Gbps
HTTPS remote
10G
1.7Gbpbs
S3 (MINIO Server)
10G
2.4Gbps
Standard pipe to file = 8.8Gbps using linux pipe to file
local file system writes ~ 2.4Gbps
Commands to write data to an NFS share over a 10G management interface ~ 5Gbps throughput
Stock SSH over 1G management port = 0.48Gbps
Stock SSH over 10G management port also ~ 0.48 Gbps
Using arcfour cipher "-c arcfour" performance is about 1.2Gbps
using netcat on a 10G Management interface over TCP results in near line rate throughput
local host HTTP download ~ 6.2Gbps
local host HTTPS download = 3.3Gbps
1G link is the bottleneck ~ 0.9Gbps
10G link HTTP download ~ 4.0 Gbps
10G HTTP download ~ 4.4Gbps
10G link HTTPS download ~ 1.7Gbps
Remove NGINX web server from the setup, throughput only slightly faster 4.8Gbps
Push over HTTP (S3 EP is HTTP ) using a 10G interface
All FMADIO Packet capture systems can also uploaded raw 3rd party PCAP files into the system. This allows Packetscope, Tcpscope, Analysis and Replay plugins to work on external and archived historical data. The upload functionality is heavily used internally for our own testing and regression frameworks.
Please note:
Capturing must be stopped. Running Capture and Upload simultaneously results in undefined behavior
If the PCAP your uploading is small, you can
Step 1)
scp the PCAP onto the OS disk. e.g. /mnt/store0/tmp2/
Step 2)
Upload using the utility stream_upload. The upload fetchs data via stdin allowing a wide range of options from a local PCAP file, to remote PCAP, to a curl URL or PCAP generation utility running on the system. The following example is a simple upload a PCAP thats on the local filesystem.
Note: the timestamp resolution of the uploaded PCAP is automatically detected and converted to FMADIO native nanosecond format.
Step 3)
Confirm upload.
Sometimes you need to upload very large multi TB PCAP to the FMADIO Packet Capture System. In such cases there isn't enough local storage on the OS disk for the scp method to work. To upload a large PCAP use the streaming/pipe functionality of the stream_upload utility.
In this example we are uploading a raw PCAP over SSH into the FMADIO Packet Capture system. From an SSH shell on the capture system the command SSH`s into the remote system where the PCAP is stored and issues a "cat" command on the PCAP to be uploaded.
Effectively piping the remote PCAP down the ssh connection. This is then read by the stream_upload command in --stdin mode, instead of reading from the local file system. For maximum performance its best to use the 10G management port for the connection.
Using this approach the PCAP is streamed onto the system via SSH, with no temporarily files created. The maximum PCAP that can be uploaded is limited by the capture systems total storage capacity.
Alerts can be generated by system automatically either
EMail alerts
Syslog alerts
SNMP Traps
Alert configuration file is located in
By default all Alert triggers are disabled.
An example alert.lua file is shown below. If the file does not exist please create.
System has can trigger an a small but well defined list of critical Events. The following is a description and example for each item. Triggers are enabled or disabled in the following part of the configuration file. Each line enables/disabled or puts a threshold on the trigger
Each trigger is described below.
Monitoring the capture link status is critical to ensure no data is lost. Enabling this option will alert when a capture link goes up or down.
Capture State shows the capture is active or in-active. When using in alert mode it will trigger anytime the capture state changes
Bytes Cached indicates how much capture data has been written to SSD, but not written back into long term storage yet. e.g. Its the delta between the capture SSD rate, and the HDD magnetic storage writeback. Trigger on for example 3TB here provides a good indication the HDD writeback process is running too slow for the sustained incoming capture rate.
(example trigger once Cache goes overt 1TB)
Any time Bytes Over increases an alert is generated. This typically a symptom of capture rates being too high, or HDD writeback too slow (or failing)
Counts FCS errors received on the interface. Any time packet error counts changes an alert is generated. Typically occurs when there are Layer1 link stability issues
Alerts generated when packets are dropped on the capture device.
When space on /mnt/store0 partition is less than this amount (scientific notation) in bytes. Alerts are generated.
In the below example, an alert is generated when less than 4e9 (4GB) of space is free on /mnt/store0 partition
When space on /mnt/store1 (scratch analytics workspace) is less than this amount (scientific notation) in bytes an Alert is generated
When space on the /mnt/remote0 (typically NFS mount partition) is less than this threshold an Alert is generated
Alerts when there is a disk error or RAID error on the device. For example a disk has been lost or HDD RAID redundancy has been reduced.
Alerts on the total number of disk SMART errors. The value is aggregated across all disks, please check the system log files for more details about which specific disk is having an issue.
Minimum number of seconds between alert generation. This is to prevent spamming of alerts due to unexpected system conditions.
Alert events are always output to SYSLOG regardless of the other transport modes (email/snmp etc)
SYSLOG logfile is found in
An example syslog alerts as follows.
Email alerts can be setup as the following, please add the ["Email"] section in the alet configuration file
An example that sends alerts to the address "alerts@fmad.io" is shown below.
In addition fmadio packet capture system uses msrtp as the email client, it requires smtp configuration file
Example configuration as follows. Please edit to match the email smtp provider
FW: 7611+
FMADIO devices can operate in SNMP Broadcast mode. In this mode the system will periodically broadcast all SNMP counter values at a fixed time interval to an SNMP target.
Latest MIB file is found (last updated 2021/12/25)
The general configuration file is used for config
Please edit the section titles ["SNMP"] as follows
The above config enables SNMP Broadcast mode only, while SNMP Trap(Alert) mode is disabled. Broadcast frequency is 60e9 nanoseconds, e.g. every 1 minute.
Broadcast and Trap mode can be use simultaneously if required.
Please update ["Target"] = setting to the correct SNMP collector address. Multiple SNMP targets can be specified separated by spaces. For example
Example output in broadcast mode is as follows, from the /mnt/store0/log/monitor_alert.cur logfile
This translates to
Logfiles are found /mnt/store0/log/monitor_alert.cur
Verbose mode above can be set to "true" to allow additional logging.
FW: 7611+
FMADIO Devices can send SNMP Traps based on the alert triggers described above. This may be preferable to email alerts for infrastructure management.
Latest MIB file is found (last updated 2021/12/25)
The general configuration file is used for config
Please edit the section titles ["SNMP"] as follows
The above config enables SNMP TRAP mode only, SNMP Broadcast mode is disabled. This configuration will only send SNMP TRAP events when a Trigger is alerted.
Please update ["Target"] = setting to the correct SNMP collector address.
An easy way to trouble shoot traps is to se the DiskFreeStore0 threshold to a very large number. In this setup the SNMP TRAP event will be constantly generated (every 1 minute).
Logfiles are found in /mnt/store0/log/monitor_alert.cur
FMADIO Syslog provides rich telemetry stream for monitoring the health of the system.
Telemetry is provided in JSON format for easy ingestion in monitoring and observability infrastructure. Each syslog moniting log event is described below, these are categorized into subsystems as follows
Generic SYSLOG events
FMADIO Temperature (Physical Thermal monitoring)
FMADIO Time (NTP, PTPv2, PPS)
FMADIO Other (uncategorized sensors)
FMADIO Power (status of power supply and consumption)
FMADIO Capture (monitoring state of the Packet Capture )
FMADIO IO (metrics related to Disk IO)
FMADIO Link (capture and management link status)
FMADIO Fan (physical status of FAN and cooling)
FMADIO Disk (status of both capture and OS disks)
FMADIO Cat (status of the stream_cat process used for extracting data off the disks)
FMADIO Push PCAP (status of the PCAP Push process)
FMADIO Push PCAP Split (event indicating each PCAP split completion)
FMADIO Alert (Custom onbox system alerts)
FMADIO Telemetry service is included without charge on all support contracts. It provides a dashboard for each system, example screen shot shown below.
All FMADIO JSON messages have the following header fields
This provides a high level granularity on what the event/status is for, these map the sections below.
This provides more granular view on what kind of event/status this is for
This is the epoch time in seconds, where the FMADIO Monitoring system generated the event. Not the rsyslog time, although under normal situations they should be the same.
This is the Firmware Version, it allows easy backwards compatibility as the system updates and improves as data is ingested into the monitoring system.
This includes generic syslog messages in free form plain text events. Example shown below
This provides thermal monitoring of the system to ensure its running within operating ranges.
Example JSON Pretty
Provides monitoring of time synchronization
Pretty JSON
Miscellaneous other fields
Pretty JSON
Provides power status and utilization information
Pretty JSON
Provides status information around the capture process
Pretty JSON
Provides status and performance of IO disk/network related systems
Pretty JSON
Status information around port link status
Pretty JSON
Status information around fans and cooling
Pretty JSON
Status around the capture and os disk
Pretty JSON
stream_cat is the core FMADIO utility for extracting packets off the storage system. Its used heavily for download, realtime processing and troubleshooting. This provides statistics and state of the currently running stream_cat instances.
Up to 8 simultaniousle stream_cat instances can be run at the same time. cat_*_ is per instance.
Pretty JSON
Provides status information of the currently active Push PCAP proceses
Pretty JSON
Called after completing a PCAP file split event. Helpful to monitoring each and every PCAP split leaving the system
Pretty JSON
The system has capibility to generate system alerts based on configuration files. These alerts can be sent to SYSLOG for ingestion by a monitoring system
Pretty JSON
Setting up automatic telemetry as follows
FMADIO devices by default have a pre-installed ssh key. To correctly secure and uniquely identify the system generate your own SSH key as follows.
Using a password less key ensures the automatic setup requires no manual intervention.
Example output per below
Send the above public key (below) to support@fmad.io
The SSH public/private keys are on the volatile file system. Copy the keys to the persistent storage.
NOTE: the key is renamed with an fmadio_* prefix. The system copies the keys from this location and renames them in the .ssh/id_rsa .ssh/id_rsa.pub directory during the boot process.
There is a reference boot script located in
Copy this to to the /opt/fmadio/etc/boot.lua file to automatically establish ssh tunnel to the telemetry service.
After copying replace the "username" to the username provided by fmadio support and save the file.
In addition to ssh tunnel setup, rsyslog configuration to forward syslog messages to the SSH tunnel.
Copy the reference config to /opt/fmadio/etc/ directory as follows
No modifications are required.
Reboot the system to check all the above steps are executed correctly
After rebooting log into the Grafana monitoring site with the assigned username and confirm data is being recevied.
Any problems please contact support@fmad.io
In addition to syslog, the systems telemetry information snapshot can be fetched using the JSON API.
Example command:
With the pretty formatted output as follows. This output will always match the above syslog information.
As a raw JSON blob
The FMADIO 100G Packet capture dashboard provides a high level overview of the capture system. Example is shown below.
NOTE: Different hardware platforms 20G, 40G, 100G have slightly different dashboard settings.
Going over the above in more detail as follows
Total number of packets received and successfully stored on the capture device
Total number of bytes successfully captured and stored on the device.
Errors with packet data seen on the wire. This includes Frame Check Sequence (FCS) errors.
Total number of packets dropped / unable to make it to the capture storage system
Total number of days worth of capture on the system. Calculated as difference between the oldest packet on the system vs the newest packet on the system
Total number of packets generated / transmitted on the device. This includes packet blaster and packet replay counters
Total number of bytes generated / transmitted on the device.
Total number of errors occurred during packet generation
Total number of new SMART Disk errors
Current RAID status, GOOD or DEGRADED or FAILED
Counter of system related warning or errors. Currently this is unused
Total number of DDR4 System RAM ECC Errors
Total uptime of the system
Most of the counters can be reset by clicking on the small circle arrow highlighted in red below.
All Gen2+ FMADIO devices have a built in packet blaster / Layer 2 packet generator. This allows a single system to be entirely self contained for unit and system testing. In addition FMADIO devices can also load test network devices such as switching and firewalls, checking for physical layer links and measuring network path latency.
Packet blaster is a layer 2 (Ethernet level) packet generator that runs at full line rate @ 64B to 9218 Jumbo sized packets. Generation is performed entirely on the FPGA Capture card thus up to full 100Gbps @ 64B 148Mpps packets can be generated without generation variance. Packet generation and capture can run simultaneously, thus verification the capture device is operating correctly is achieved.
The payload of each packet is a per physical port MAC Address followed by a 32bit incrementally increasing sequence number. This sequence number is used later post capture to ensure data of all packets has been captured correctly without error. An example packet is shown in Wireshark below.
In the above wireshark picture, you can see 2 different MAC address 11:11:11:11:11:11 (Physical Port 1) and 22:22:22:22:22 (Physical Port 2). The payload is a 32bit little endian sequence numbers
This sequence number and MAC address allow the analysis software to not only check the total number of packets captured by the device, but also check every byte of the payload has been captured without error. As the analysis software knows exactly what the packet payload data should be via the sequence number.
Packet blaster is operated only by the CLI interface, each FMAD SKU has a slightly different operation
Example operation, generate 1 billion 64B packets on a single 100G port at full line rate
Payload data verification on the device is achieved by "linux-cat-ing" a capture down a linux pipe to the builtin utility capinfos2. The syntax looks as follows
In many scenarios there is a requirement for simultaneous capture and analysis of data in pseudo-realtime. The target performance is bursting (up to 156TB worth) of 100Gbps line rate traffic, with an average sustained data rate of ~ 50Gbps. We target 50Gbps as per the graphs below is the hardware limitation of simultaneous sustained 50Gbps write + sustained 50Gbps read.
In the below image shows Capture/Write only, or Analyze/Read only FMADIO 100G Gen2 system easily does 100Gbps worth of IO bandwidth
The ideal IO profile is an simultaneous sustained 50Gbps Capture/Write and 50Gbps Analyze/Read profile as shown below. This allows the capture to burst to 100Gbps without significantly effecting the Analyze/Read performance of the system.
Due to hardware IO limitations (~ 150Gbps max aggregate bandwidth) at a sustained 100Gbps Capture/Write only a 26Gbps sustained Analyze/Read performance is achievable.
Please note all the above numbers, are the maximum limits. Typically the Analyze software performance is impacted more by the packet rate than the data rate, As such the Analysis software performance is usually the bottleneck, not the raw IO hardware limits.
FMADIO FShark2 is a full Ubuntu desktop accessiable via RDP or HTTP client. This has the latest Wireshark binary plus additional utilis enabling full packet investigations on the system.
In many enviroments creating an additional IP for FShark2 is problematic. Instead port fowarding ports on the FMADIO Capture Appliance to the FShark2 device is a simpler apporach.
Download latest fshark2 release
Example
Extract to LXC directory
Example output:
Or download an extract at the same time
Change directory to the /opt/fmadio/lxc/fshark2-<insert version>/
Run the install script. If no IP address for the container is used (e.g. fully NATed / port forward) leave the IP info blank
Example output
Step 3) Configure for NAT / Port forwarding
Comment out the lxc.net.1 (bridged interface) in the Config and set the default gateway to 192.168.255.2 (hosts internal interface)
Example Config
To enable automatic starting of the FSHAK2 container on system boot
Example output:
To start the container
Example output
If it prints any messages it means there is a configuration error somewhere
Check the port 3000 (HTTP browser) and 3389 (RDP) are open
Example output, can see both ports are listed
Copy the following iptables to the configuration directory
Example output:
Manually load the iptables setting
Example output:
For reference the /opt/fmadio/etc_ro/iptables_fshark2_portfwd.conf file looks like the following
Output the iptables information
Example output:
Point the browser to port 7000 or RDP to port 7001 to confirm FSHARK2 is accessible
FMADIO Shark is a self contained LXC container that runs on the FMADIO Packet Capture systems. It provides a "Wireshark Lite" way to navigate thats fully compatible with Wireshark, as the backend is Wireshark.
To configure and run as follows
Copy the tarball to /tmp
Unpack the tarball into the /opt/fmadio/lxc directory
Symlink the latest release. Optionally removing the symlink of any older version
Edit the file
Setting the following ["FShark"] = true, if the field does not exist then create it
By default FShark does not start on boot, enabling this in the config uses the generic LXC container framework.
Edit the config file
Near the bottom section of the config there is a "Container" section. The example below shows a basic FShark only configuration, depending on your usage there may be additional containers configured to run.
After configuration update, reboot the system
Check FShark is running using the fmadiocli utility as follows
FShark should be installed and running as hilighted in red below.
PacketBrowser and PacketScope should be visible on the GUI as follows
FMADIO LXC system high level architecture is shown below.
This provides a full lossless filtered version of data directly into the LXC container for processing.
Backpressure is provided thru the LXC Ring structure, allowing the application to consume data as fast or slow as possible without dropping anything.
During development its typically easiest to send data into the LXC container in a one time replay operation. This allows sending the same data repeatedly to the application allow the engineers to debug and test the code.
In this example we will use the application "fmadio2pcap" which converts the LXC ring into a standard PCAP formatted stream of data.
Its recommended to start the data consumer aka LXC application first, then start the replay.
Run the following on the LXC container to generate TCPDUMP output thru the lxc ring named "general"
It will look similar to the following on startup, it stops because no packets have been sent to the ring.
On the FMADIO Host system find a capture you want to replay. In this example we are using capture named "inetsample_20230615_1513" filename can be found using "stream_dump"
The command is as follows
Example output of the command shown below
And on the LXC client side the output will show ICMP packets printed in tcpdump format as shown below
For a custom application to directly ingest data, the following reference code is provided
Primary header file with all functions and strcutures inline
Core example code to retreive packets
Core loop snippet, this converts from LXC Ring into standard PCAP Packet format
fnic_test is a general purpose utility to configure and run settings specific to FMADIO Packet Capture NIC card
Reference LX containers can be found here
See our public github repo for details
The container can be configured to have a independent public IP address (fully bridged), a private internal IP (internal NAT) or both. All options have Pros/Cons and need to be considered for final deployment.
A common setup is to have the container entirely private using nginx as a proxy or iptables to port forward to the container.
Block diagram below shows Private only Network Topology
With external access using NGINX proxy such as below.
Or use iptables to port forward packets thru the NAT, such as below
To enable the NATed bridge between man0 (public) network and the fmad0 (private)network the following IPTables config needs to be set
Alternatively the following /opt/fmadio/etc/iptables.conf file can be used (requires a system reboot or iptables-restore) to take effect
The above is general setup, to forward a specific port from the Host IP to the LXC container IP run as follows.
NOTE: if using 10G management interface replace man0 with man10
1) Forwarding port 9000 on the host to port 3000 on the LXC
(in this case LXC is configured as 192.168.255.191)
Container network settings need to be the following
List of private container addresses
The following are generalized steps to install setup and start the specified container. Please consult any container specific documentation in addition to this process.
Download the LXC into the directory below
Check the MD5 sum against the published value
Untar the tarball in that directory as root.
Example below shows the steps for the mdgap lxc container
Most LXCs have an install script in the root directory, it will configure the container requiring some information such as IP Netmask. Gateway etc.
Modify the containers configuration. such as adding rings or shared mount points
Example is enabling the Eurex lxc ring to the container by uncommenting in the configuration file, the line
In the configuration below
Ensure to save the changes
Generally containers require LXC Rings to feed data to. In this example we create an lxc ring named "mdgap_eurex" as the container references this ring in step 3) it must be created for the container to start correctly.
Example output as follows
Add the container to the system configuration file using fmadiocli utility
Add the container using the command
Example shown below
Start the container using fmadiocli utlity
Example starting the MDGap container
Optionally attach to the containers console using the command
Example of MDGap container
A guide for developers to access a FMADIO using the web based API. Examples are provided for all endpoints.
The examples show how to use the different parameters for the uri endpoint.
Note: Replace the IP 127.0.0.1 with the host IP of your FMADIO device.
StreamName only:
StreamName and FilterBPF
StreamName and Compression
StreamName and FilterRE
StreamName, Compression and FilterBPF
StreamName, Start and Stop
StreamName, Start, Stop and FilterBPF
StreamName, Start, Stop and FilterPort
TSBegin and TSEnd
TSBegin, TSEnd, TSMode and TSMax
The examples show how to use the different parameters for the uri endpoint.
Note: Replace the IP 127.0.0.1 with the host IP of your FMADIO device.
The Time Range function is very useful as the FMADIO system will work out which (or multiple) captures to check based on the Epoch Time stamp value.
Using the TSUnit option can use a more friendly time selection.
By default it uses the TimeZone configured on the system
Fetch PCAP from 3AM to 4AM on 2023 / 10 (October) / 1st
Same as above but specifying the timezone.
NOTE: if using the full TSZone = Asia/Singapore for example, CURL will append a ? to the URL. Its recommended to use the City name only to avoid confusing CURL.
Specifying all of the above with a BPF Filter, with BPF De-encapsulation enabled
Filter based on FMADIO Capture port number
Filter based on multiple FMADIO Capture port numbers
Filter based on exclude FMADIO Capture port numbers
Filter for a specific 7130 Device 54932 (any port)
Filter for everything except a specific 7130 Device (not device id 54932)
Filter for a specific 7130 Port number 1
Filter for multiple 7130 Port numbers 1, 2, 3, 5, 10
Filter for everything except 7130 Port number 10
Filter on a specific 7130 Port number and use the 7130 Footer Timestamp as the PCAP timestamp. Overriding the current TimeStamp setting
Filter on a specific ingress port of the Cisco 3550, and use the Footer timestamp as the PCAP timestamp.
Many times the exact packet encapsulation is unclear, the following uses a wireshark filter expression to extract and show the full encapsulation format of the packet. From this a high speed BPF filter can be used to process the data.
In the below example we are using the Wireshark filter "ip.addr == 192.168.1.1" on a historical capture.
Alternatively running on the currently running capture via SSH on the fmadio box looks like the following. This example filters on any UDP traffic.
The output looks like the following
The above output shows there is a single VLAN tag in the packet. Making the equivalent BPF filter
With the final BPF filter using a CURL request
Output per below
192.168.255.2
FMADIO Host
FMADIO Host IP Address
192.168.255.10
FShark
FMADIO Internal Wireshark Lite
192.168.255.100
Ubuntu Desktop
Ubuntu Desktop
192.168.255.110
Elastic Search 7.x
Elastic Search 7.x Container
192.168.255.111
Elastic Search 8.x
Elastic Search 8.x Container
192.168.255.120
Suricata 6.x
Suricata 6.x Container (CentOS)
192.168.255.130
Zeek
Zeek Container (CentOS)
FMADIO packet capture systems all have serial port either virtual or physical connectors. Typically serial port access over LAN is the preferred method as it provides an out of bands interface to the system that only requires a SSH terminal (No KVM or HTM or Java).
Serial port access is archived using the IPMITOOL located at the following link:
https://github.com/ipmitool/ipmitool
We are using IPMITOOL 1.8.16 in this example
Start by checking the system status via IPMITOOL with the sdr argument. Contact support for the default password and replace the IP address (192.168.1.123) with the IP address of the BMC device
In the above we see the sensor status of the FMADIO device, thus the tool is connecting to the device and BMC interface is working correctly.
To connect to the serial port use the following command
ipmitool -U admin -P <password> -H <host IP address> -I lanplus sol activate
Example as below
And you now have full serial port access to the system.
The serial port is mutually exclusive access, e.g.. only one user at a time can access it. If a previous session is still connected the following will be shown
In such cases, please disconnect the previous session as follows
Then reconnect
Sometimes GUI logfile generation is more difficult than a CLI version. As the GUI calls a script on the system to generate the logfile report, SSHing into the fmadio packet capture system and running the script directly is possible. The command to run in as follows
After the report has completed, the final log file is located at
Please scp off the device and send to support,
Upgrading the BMC software is simple process but requires physical access to the system. Physical access is required as the power cables need to be disconnected after BMC upgrade has been completed.
Retrieve the current BMC version as follows
Example output
The key value difference is shown below, the firmware gets upgraded to version 0xd shown below. If the system already shows version 0xd there is no need to upgrade the BMC software
Run the BMC update in the following directory
Then run the update program
Enter Y for preserve configuration settings.
The process will take several minutes to complete
Disconnect the AC power from the system. Wait for 1 minute
Reconnect AC power to the system
Wait 5minutes for the BMC to fully reboot and host system boot
Reboot the linux Host server
After host linux system has rebooted, Check BMC version is updated. It should show version 0xD per image below
Update is complete
BMC upgrade to version 12.60.39 resolves CVEs and makes possible boot without prompt when enabling full disk encryption.
From FW: 8940 the BMC firmware is located in
As the version jump from the factory installed to this version is very large, all BMC settings are lost durning the upgrade, and need to be set again.
This includes BMC Network information, meaning BMC network connectivity will be lost during this process
In addition BMC passwords and other items are also lost. The way to update this is via the FMADIO x86 Host system, where the host is always powered on enabling it to set the BMC Network settings and User passwords directly.
After updating the FMADIO Firmware the following files are located at
Copy the files as follows
NOTE: -L in the cp forces a literal copy (no sym links)
Start the BMC update process using the following commands on the FMADIO host system
Example output shown below, it will take about 5minutes to run.
Note the warning, Do you want preseve configuration, enter N
After the BMC is flashed and has rebooted confirm the new BMC version is 12.61.01 using the following command
The output should look like the following
Set the new network configuration information.
Set to use static ip per below
Set a new IP address, netmask and gateway, replace addresses with the assigned BMC network address
At this point the network should be reachable via ping, however the username password will be reverted to the default setting.
Confirm the network settings using the command
Example output is shown below
The BMC webpage should be accessible at this point.
The BMC update will delete all the settings, these need to be added back
First one is to set the admin password as follows, replacing "secret" with the password chosen
After setting this, logging into the BMC using the admin account and above password.
As all BMC settings are disabled, the first critical setting is a custom FAN profile. Usually this is set at the factory however it needs to be created again after the BMC update
Then copy the default fan profile as follows
The new fan profile is named "fmadio100v2" with the following settings. Ensure all CPU sensor and all FANs are selected
Click Save
Finally activate the profile
Disable all remote media settings as follows
BIOS update, bios update is located in
Files look like the following
To update the BIOS run the command
It will take a few minutes to update, the console output looks like the following
Then power off the system + power it on
A full power off is required to load the new BIOS
After BIOS update all settings are lost and need to be set-again, the system will fail to boot also as the BIOS settings have not been configured
Setting the Boot settings as follows
Advanced -> Trusted computing set the following
Advanced -> Serial Port Console
Advance -> PCIe Subsystem, configure as follows
Advanced -> Network Stack
Advance -> NVMe Configuration, configure as follows
Advanced -> Chipset Configuration
Boot configure as follows
Boot -> UEFI Application Boot Priorities
Save changes and exit
System boot will look different as it now boots via UEFI, similar to the following.
After boot completation the usual prompt will be shown on both the VGA and Serial ports
The system should boot normally now without any BIOS password prompt.
Finished... horary.
All FMADIO Packet Capture systems have an onboard BMC, this allows remote power on/off and HTML5 based KVM utility which is very helpful when troubleshooting issues. In addition the BMC provides temperature voltage, current sensors and many other functions.
The chipset is an ASPEED2500 ( http://www.aspeedtech.com/server_ast2500/ ) which runs its own linux kernel + management software. Logging into the BMC via web interface looks like the following
There are times where the BMC after long uptime or other issues becomes unstable and requires a reboot
The easiest way to reset the BMC is via IPMITOOL. Start by confirming the BMC is up and responding by issuing the command
Example output is shown below
This shows the BMC is responding correctly, Next issue the BMC cold reset function using the command
Example run as follows
It will take a few minutes for the BMC Web interface and IPMITOOL tools to become active again.
NOTE: Please power cycle the Host system after BMC reboot to fully clear the error condition
If the above fails, a secondary option is possible using SSH into the BMC directly and issuing a reboot. Please contact support@fmad.io for password
NOTE: Please power cycle the Host system after BMC reboot to fully clear the error condition. If a powercycle is not possible (due to BMC), a warm reboot can be run first.
In some cases ipmitool is not responsive, thus the only option is a full power cable removal. Using the "power cycle" option on the BMC only resets the x86 server, it does not reboot the BMC. This is a last resort as physical access to the machine can be difficult in some cases.
SSD Replacement is quite simple, the FMADIO device needs to be powered down, but does not need to be un-racked. Front access to the system is all thats required.
Below is a picture of the FMADIO100G 1U
Its preferred the system is powered down when replacing SSDs
Remove the SSD by pressing on the orange tab such as picture below
Remove the SSD + Caddie entirely from the chassis
Remove the SSD from the Caddie via 2 screws on each side of the drive
The first step for any problem resolution is generating detailed log files for analysis to understand the exact nature of the problem. Our system automatically generates logfile information using the following steps.
NOTE: Typically logfile sizes are 200-500MB in size. It contains detailed system logs and may contain snippets of captured, packet data.
Select the Tools menu highlighted in green below.
Start System Log generation, by clicking on the icon highlighted in green below.
Logfile generation starts with status information shown in area highlighted in green. Depending on the size of logfile this may take from 1 - 15 minutes to complete.
When completed the status will change as highlighted in green below.
You can now download the report via the icon highlighted in Green. An example downloaded log file is shown in blue below. After download, transfer to us for further analysis.
FW: 8224+
Getting 100G links link up either works no problem out of the box or be fickle to setup and get a link.
Usually the main source of problems is FEC vs non-FEC setting on both the switch and on the FMADIO system. FMADIO Packet Capture systems will try detect if FEC is on the link by alternating FEC enable / FEC disabled when trying to link up.
This can cause problems if the switch its connected to is also cycling thru the different combinations.
To check the current status the fmadiocli utility provides the easiest way to monitor the state. Use the command, below to get the current link state.
In the above we see capture port 0 (cap0) is link down and capture port 1(cap1) is link up
The link may be bounced by shutting down using the command as follows
It may take 1min for the link to go down, use the show interface status command to wait for link down to complete.
Once the link is down, disable the shutdown using the command
The default setting of FMADIO Packet Capture system is FEC running in "auto" mode. This means the system will try link up without FEC, if it fails FEC is enabled and link up is tried again. This process is repeated indefinitely.
For some switchs and NICs this works for others it has problems. e.g. if both end points are cycling like this its possible it will ever link up due to phase/offset of the cycles.
Its best to force the FEC setting if your expecting FEC to run on the link. This is done as follows
After forcing FEC on the links the interface status will show a "force-" option as follows
This indicates FEC has been forced on for the specific interfaces.
NOTE: bouncing the port on the switch may be required, it depends what state the switch port is. It may have entered an error mode, or backed off on the autoneg. Bouncing the port on the switch most of the time will resolve the issue.
If the link is still failing to link up, you can trace the state machine for link up procedure using the command
By default it runs on cap0 using the --port 1 can run the trace on the other port. This prints out the realtime event history of the link up process, below is an example of a link bounce.
In the above output we see the link state go from State 1 -> State 7. Where state 7 is a stable link up.
For additional assistance on link problems, please run the above and pipe to a logfile. Send the logfile to support @ fmad . io for further assistance.
FMADIO systems include a full full Keyboard Video Mouse (KVM) remote access using the onboard AST2500/AST2600 BMC chipset. This enables remote power cycles, reboots and others all using a simple HTTPS HTML5 web browser (No requirement for Java)
To log into the system via the KVM please run the following steps
STEP 1
Log into the IPMI/BMC system, please contact support@fmad.io for default IP and credentials.
It uses a self signed certificate unless a certified certificate has been uploaded. Please click thru to the login page
Login page looks like the following. Please contact support@fmad.io for default credentials
STEP 2
Select "Remote Control" on the left menu
STEP 3
Select "H5Viewer" to launch the web HTML KVM page.
NOTE: this uses normal HTTPS connection no additional firewall rules or ports need to be opened
STEP 4
HTML5 KVM viewer is launched. Looks like the following below. From here please login to the FMADIO device to setup and configuration.
Due to screen saver, it may appear blank. Please press any key to disable the screen saver.
NOTE: the currently configured IPs are shown at boot.
In the unlikely event of a complete boot failure, system can be recovered by booting via the Virtual CDROM interface over a HTML BMC connection
Start by going to the BMC interface (default IP is 192.168.0.93) contact us for default login/password
Start the Remote HTML KVM
Will look like this. Select Brose Files, selecting an ISO image + Start the Media
System will boot Ubuntu (for example), we are using ( systemrescue 8.01 amd64)
System will boot as follows, it may take several minutes depending on the speed of the HTML <-> FMADIO System connection. Recommend the closer the HTML instance is to the FMAD device the better.
If a particular boot stage is taking too long Ctrl-C can skip it
After SystemRescue CD has booted, the above is seen. Note the total number of bytes transfered over the Virtual ISO.
First step is to find the FMADIO OS and Persistant storage devices, Use the "lsblk" tool
Looking foor a small (15GB) partition as the OS boot disk. In this case its sda1 and a large (224GB or larger) partition for the Persistent storage
Sometimes its easier to work over SSH. To access the system find or assign an IP address to the a reachable interface
SystemRescuelCD by default has iptables setup. Disable all iptables as follows
Then setup a password for the root account
Then ssh access to the system is possible
Next mount the FMAD OS and Persistant storage disks. They may be sda* or nvme0n1p* in this example its mapped to sda
Next check the contents, it should look roughtly like this
Generating the network traffic profile can help debug many issues
FMADIO Capture System has built in utility "capinfos2" which can provide fast and compact summary information about a PCAP.,
For network profile information please run as follows
This will generate a packet size histogram such as the following
With a more compact histogram version shown below. Note the first graph is the packet size histogram of packets on the wire, the second is size based on the packets captured. In this case packets have been sliced @ 128B thus the histogram is quite short
Finally at the end there is a number list, which can be sent to FMADIO Support so we can replicate your network traffic profile locally with our synthetic packet generator.
To reset the SSH password system the system requires KVM either directly with a physical montior and keyboard of via the BMC HTML viewer. We recommend HTML viewer and will show the steps here. The process is the same for a physical monitor and keyboard
1) Connect to the BMC login and launch the HTML5 KVM
2) The KVM viewer looks similar to below. If the screen is blank press a few random keys to disable the screensaver
3) Reboot the system using the Power menu
4) Need to catch the system boot prompt here. One way is to tap space key or letter "r" every few secconds. This will prevent the bootloader from automatically starting.
5) Enter system "rescue" mode at the prompt and hit enter. This will boot the system into rescue mode.
6) Rescue mode drop to a default prompt as follows
7) Set a new password for fmadio user as follows
8) Mount the persistent storage volume as follows. The exact path depends on the system SKU
FMADIO100v2 or FMADIO100p3:
FMADIO20v3 or FMADIO40v3:
9) Copy the shadow password file to persistant storage
10) reboot the system
11) SSH login fmadio has the newly set password
In addition to the above web user password and access list can be configured using fmadiocli tool. Documentation is here
FMADIO Systems use a RAM based linux distribution, where all persistent files have to be explicitly specified. This approach allows easy forward and backward changing of the system firmware.
Forward updates to new FW is documented below
Below is how to revert to a previous image.
Check is the reverted firware is already uploaded on the system
Example output shown below
If the firmware file is shown with correct MD5 sum skip to Step 3
Upload the target firmware onto the system. There is no harm in overwriting an existing firmware file. In the below example using firmware FW:8475 fmadio100v2_20230307_1925.bin
Example output similar to this
Next install the firmware image as follows
Example output looks similar to this
Reboot the system, it make take up to 5 minutes. The system will reboot 2 times, once for fpga update, and again for the final pass.
Verify the firmware version is correct
Example output
Its possible when differrence in firmware version numbers is extremely large there may be additional steps. Please contact support@fmad.io for assistance in such cases.
System Firmware Update
Upload Firmware into the system is the following process
scp or cul the *.bin firmware file to the system
upload the firmware into the system
install the firmware on the system
reboot the system
Its fairly straight forward to do, in many instances CLI based update is easier than a GUI
This is either scp or curl -O the firmware to the home directory, example below uses curl directly from the webpage
While the .bin file may be on the system, It needs to be uploaded and processed by the system to make it accessible. Using the following command line on the FW file from 1)
NOTE: the firmware filename must be exactly as downloaded, e.g. no (1) or other suffix appended.
Below example filename is "fmadio20v3_20210831_1136.bin"
Example upload process
Next the firmware needs to be installed using the command
Example output is shown below. Note if the capture is running as shown, it will eventually timeout and complete the update.
System needs to be rebooted, It will power cycle also after the 1st reboot to complete the update process.
Example
