Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
PCAP2JSON can do basic packet de-encapsulation. Specifically ICMP and IPv4 GRE, these fields are populated in the decap* JSON fields. Please note the de-encapsulation is include in the unique half duplex flow calculation.
Example output
By default pcap2json uses its builtin JSON field definition. This can however be overridden with a custom JSON format and file using
option in /opt/fmadio/etc/pcap2json.lua Section "backend"
Default JSON format (2021/7/24)
The following is a reference for all the fields
Outputs timestamp in MSec epoch time
PCAP2JSON can calculate RTT and Byte accurate TCP Window information following is a description of each field. TCP Window sizes are real sizes in bytes that are scaled correctly using the SYN/SYNACK negotiated window scaling factor
This field indicates which side of the tcp connection the Half Duplex flow is on. Its particularly useful when used in combination with tcpRTTNetHalf to monitor the response time of the server or the WAN latency of a client.
Server highlighted in Aqua
Client highlighted in Aqua
Outputs the full TCP round trip time in milliseconds. . This is a full duplex value, both sides of the TCP connection have the same value. This value approximates an ICMP ping between the two hosts.
Calculation is Full Duplex
Unit is Milliseconds
Show below, the aqua colored paths are used in the calculation. Total caluclation is
tcpNetRTTFull = P2 - P0
Example JSON output, note field tcpRTTNetFull, in this example the Full RTT round trip is 856msec.
Both half duplex hash have the exact same tcpRTTNetFull value.
This calculates the partial RTT value. Note depending on where the capture device is located indicates which side of the RTT it shows. This value is used in conjunction with @TCP_MODE@ to monitor Server response time or Client speed of light latency.
In this mode tcpRTTNetPartial is the round trip between request to connect to the server, and the services accept() of the connection. Shown below in aqua
tcpRTTNetPartial = P1 - P0
In the client mode, tcpRTTNetPartial is (usually) the speed of light network latency between the Server (e.g. HTTP web server) and the Client a desktop / mobile end user. Its not strictly the case, this is the typical use case as FMADIO Capture device is usually co-located with the Web Server.
tcpRTTNetPartial = P2 - P1
Example JSON below. tcpRTTPartial is different for each half duplex flow.
Outputs the Application Request -> Response network round trip in milliseconds
Calculation is Half Duplex
Unit is Milliseconds
Time different between between application Push to Push. e.g. HTTP Get -> 200 OK Response
tcpRTTApp = TS1 - TS0
Outputs the minimum TCP window size in Bytes
Calculation is Half Duplex
NOTE: This does take into account full Window Scaling from the SYN/SYNACK connection. If no SYN/SYNCACK has been processed this filed outputs NULL
Outputs the maximum TCP window size in Bytes
Calculation is Half Duplex
NOTE: This does take into account full Window Scaling from the SYN/SYNACK connection. If no SYN/SYNCACK has been processed this filed outputs NULL
Outputs the arithmetic mean TCP window size in Bytes
Calculation is Half Duplex
NOTE: This does take into account full Window Scaling from the SYN/SYNACK connection. If no SYN/SYNCACK has been processed this filed outputs NULL
PCAP2JSON will de-encapsulation some packets and payloads. Following is a description of each field
If packet de-encapsulation was performance, indicates what kind of encapsulation
For de-encapsulated packets, the IPv4 Source address of the INNER packet
For de-encapsulated packets, the IPv4 Destination address of the INNER packer
For de-encapsulated packets the IPv4 Protocol of the INNER packet
Version:608+
Value (In Hex)
Example JSON
For de-encapsulated packets the Source Port of the INNER TCP or UDP packet
For de-encapsulated packets the Destination Port of the INNER TCP or UDP packet
Uses the GEO IP Mapping to search a a textual description of the De-Encapsulated source IPv4 address. If a mapping is not found, the standard IPV4 numeric address is used.
De-encapsulated Source IP GeoIP Location. This includes the Longitude / Latitude value
De-Encapsulated Source IP GeoIP Country
De-Encapsulated Source IP GeoIP 2 letter country code
De-Encapsulated Source IP GeoIP City
De-Encapsulated Source IP ASN
De-Encapsulated Source IP Organization
De-Encapsulated Source IP ISP Name
See @DECAP_SRCIP_HOSTNAME@ for description
See @DECAP_SRCIP_LOCATION@ for description
See @DECAP_SRCIP_COUNTRY@ for description
See @@DECAP_DSTIP_COUNTRY_CODE@
See @DECAP_SRCIP_CITY@ for description
See @DECAP_SRCIP_ASN@ for description
See @DECAP_SRCIP_ORG@ for description
See @DECAP_SRCIP_ISP@ for description
FW: 7297+
PCAP2JSON v2: Ver:606+
By default pcap2json does not support HTTPS / TLS connections as it severely reduces the ES push performance. As a work around you can use NGINX as a proxy to receive HTTP and send HTTPS to the destination end point, As shown below
In this example we are configuring FMADIO100G system running pcap2json v2 pushing over TLS to a cloud ElasticStack instance hosted by elastic.io
Configuring pcap2json for this is fairly simple start by configuring the nginx proxy as follows in the file
If the WWW section has not been created, please add as shown below. Replace the Endpoint + API Key with your own
Please refer to https://www.elastic.co/guide/en/kibana/master/api-keys.html for documentation on how to generate Elastic Search API Keys
For the above configuration to take effect, need to reload nginx by doing the following
After killing the process it may take up to 1 minute for nginx to respawn and be up and running
Next check the TLS proxy is working correctly by querying the ES Node for a valid response.
Note we are using http to get the cluster status, the URI prefix uses the nginx running on the FMADIO system to proxy and output HTTPS connection to our ES EndPoint usually on a WAN or internet facing IP.
A correct example output looks like the following
If this is not the correct result, check the nginx logfile per below for trouble shooting
Configuring pcap2json is quite simple, because HTTPS/TLS push is less performant, we need to restrict the total number of con-current ES push operations, as follows
Next change the --es-host target as follows
In addition we need to prefix all URI requests so FMADIO nginx will route it correctly
Save it and pcap2json will run as normal but push to your remote ES cluster using HTTPS/TLS.
Any issues check the logfile below for problems + the same nginx error logfile as above
Following is a full backend example configuration file for reference.
pcap2json Kafka Integration
pcap2json can push flow snapshot JSON information to a Kafka bus using the libkafka client library. This is an alternative flow to directly pushing to Elastic Search, there are Pros/Cons of doing this which depends on customers infrastructure.
Example configuration file located in /opt/fmadio/etc/pcap2json.lua
In the above example its pushing to Kafaka broker at "192.168.2.82:9092" per
Also pushing to Kafka topic "pcap2json" per
Please modify based on you infrastructure, the other items can remain the same.
Running pcap2json offline is as follows, in the example we are using the FMADIO Capture "interop17_20220927_0348"
Execute the command in the /opt/fmadio/analytics directory
Example output is as follows
Depending on the size of the capture, this may take 10sec, or 10 minutes.
Any errors with the configuration can be checked in the log file.
NOTE: this file is a symbolic link of the latest run of pcap2json.
After confirming operation in offline mode works correctly, enabling realtime via the GUI as follows
Below is the current default JSON Field mappings. Customization of this can is possible using a custom Template option.
v728+
pcap2json has a built in watchdog timer that automatically restarts the system if no Input PCAP data is received, or Output JSON data is sent.
By default the function is disabled
Configuration to enable is in
Example configuration shown below
Key configuration is "WatchdogTimeout" which is set to number of seconds. In the above example it waits for 30 seconds of null activity before restarting the pcap2json system.
This can be tuned based on the network profile of the captured data. Recommend 60sec to 5 minutes.
When a watchdog restart is generated it creates a sentinel file
When pcap2json restarts it checks if this file exists, if so disables "FollowStart" mode and instead always starts processing the currently capturing PCAP at the current time, not from the start of the capture.
While the data will have gaps, it does not generate duplicate data downstream that would result in incorrect network statistics.
Value
Description
server
The server side of a half duplex flow. Calculated by the half duplex flow that receives the TCP SYN
client
TheClient side of a half duplex flow. Calculated by the flow which receives the TCP SYNACK
Value
Description
ICMP Destination Unreachable
ICMP Code 3
ICMP Time Exceeded
ICMP Code 11
GRE
GRE Tunnel
This is the INNER IPv4 DSCP field following table shows the current enumeration. If an unknown value is found the text output is a hexadecimal string literal.
JSON String
0x2e
EF
0x0a
AF11
0x0c
AF12
0x0e
AF13
0x12
AF21
0x14
AF22
0x16
AF23
0x1a
AF31
0x1c
AF32
0x1e
AF33
0x22
AF41
0x24
AF42
0x26
AF43
0x08
CS1
0x10
CS2
0x18
CS3
0x20
CS4
0x28
CS5
0x30
CS6
0x38
CS7
default
0xXX (hexadecimal print)
From Elastic Search 8.0 usernames and passwords are required for all HTTP bulk uploads. pcap2json from Version 642 supports basic authentication using configured passwords
In the configuration file
In the "Backend" configuration section add the following configuration
"Basic" is the type of authentication used, this is configurable.
"dXNlcm5hbWU6cGFzc3dvcmQ=" is the BASE64 encoding of "username:password"
Example encoding
This translates to the following HTTP header
And provides a range of options for different authentication, e.g API keys can use "APIKey" instead of "Basic" for specific cloud services.
Debugging pcap2json can be tricky, logfiles are key to understanding what has happened.
The key files are as follows all in the /mnt/store0/log/ directory
Troubleshooting ES problems please check
NOTE: These files are symbolic links and change each time pcap2json is run.
Performance baselines
GEOIP Stats for all tests
GEOIP Enable
Output ESNULL
Result
53.941 Gbps
38.332 Mpps
pcap2json
GEOIP Enable
Output ESNULL
Result:
52.685 Gbps
37.440 Mpps
pcap2json
GEOIP Enable
Output ESNULL
Result:
55.737 Gbps
39.609 Mpps
pcap2json
GEOIP Enable
Output ES NULL
Result
51.910 Gbps
36.887 Mpps
pcap2json
GEOIP Enable
Output ESNULL
Result:
22.768 Gbps
16.180 Mpps
pcap2json
GEOIP Enable
Output ESNULL
Result
28.115 Gbps
19.98 Mpps
pcap2json
GEOIP Enable
Output ESNULL
Result
31.823Gbps
22.614Mpps
pcap2json
GEOIP Enable
Output ESNULL
Result:
42.069Gbps
29.895Mpps
pcap2json
GEOIP enable
Output: ESNULL
Result:
51.99Gbps
36.945Mpps
Flow Generation
pcap2json
GEOIP Enable
Output: ES NULL
Result
79.9 Gbps
28.21 Mpps
Flow generation
pcap2json
GEOIP Enable
Output: ES NULL
Result:
Data rate 69.98Gbps
Packet rate 28.18Mpps
Flow generation
Capture size
Backend performance
GEOIP Enable
Output: ESNULL
Results
39.99Gbps
39.0Mpps
Flow Generation
pcap2json
GEOIP Enable
Output: ESNULL
Results
19.99Gbps
39.05Mpps
Flow Generation
pcap2json
GEOIP Enable
Output ESNULL
Results
19.98Gbps
39.056Mpps
Flow Generation
pcap2json
GEOIP Enabled
Output ESNULL
Results
109.98Gbps
38.799Mpps
Flow Generation
pcap2json
pcap2json
Good visibility and understanding of how traffic moves though your infrastructure is extremely critical to any Network Operations Center (NOC) and Security Operations Center (SOC) yet there are many approaches. Some are easy, some are hard, some are expensive and some are free.
FMADIO 10G 40G 100G Packet Capture systems combined with Open Source software enables a new kind of visibility and monitoring capabilities using our full packet capture systems. Our packet capture systems are multipurpose, used for troubleshooting, deep dives with Wireshark but also excellent monitoring capabilities!
FMADIO developed pcap2json utility which converts a PCAP into ElasticSearch bulk upload JSON, it even uploads fully compressed JSON directly into ElasticSearch without LogStash. Its free and OpenSource on GitHub, and can use any PCAP file.
configuring pcap2json
pcap2json is a high speed utility that converts PCAP data into JSON flow snapshots. This allows highspeed 10Gbps-100Gbps of capture data size be reduced by x10 into JSON records which can be ingested by an Elastic Stack cluster.
pcap2json pushes data directly to the Elastics Stack cluster, typically over HTTP and port 9200. There is no load balancer, LogStash, Reddis or other system between FMADIO pcap2json and the ES Cluster. as shown below.
7.17.12
pcap2json is created using FMADIO Packet Capture plugin architecture. It requires the plugin file to load the plugin as follows using the command
Example load operation shown below
When loading plugin the first time on a new system, a reboot may be required to fully load
Verify the plugin is loaded by checking the version number as follows
Here we see pugin is at version 306
Next setup the configuration file located in
Example config file is shown below, if no pcap2json.lua file is in the directory please create one with vi or nano
The above config is a good start, we have left some of options in ["backend"] commend out to help with debugging per below..
The most import config setting here is
Please replace the IP and Port with the appropriate IP Port address of your ES Cluster. You can add multiple lines as above for each ES Node.
Once complete run the below command to confirm no syntax errors in the configuration file.
Instead of turning everything on at one time and trying to debug the system, a good way is to start with running pcap2json in "offline" mode using stdout with a small PCAP.
What this does is run pcap2json and instead of pushing to the ES Cluster, it writes the JSON to a local file. This is good as we can confirm pcap2json is functioning correctly first, then look at confirming pcap2json + ES is working correctly.
First ensure the option, per the pcap2json.lua config file above is set
Next find a small ish sized capture file, using the command
Output of this command will look similar to this
In the above example we will use the file named "interop17_20210716_0716"
Next move to a temporary directory
Then start pcap2json in offline mode with the above capture
Example output as follows.
After it has completed running there will be a file name "stdout" in the directory
This is the JSON records PCAP2JSON generated, but written to a file. Example shown below
If your stdout file looks similar, ready to connect the system to Elastic Stack. If not please check the above steps again until stdout file looks correct.
After the Test Run writing to file "stdout" is functioning correctly, next step is to connect and push that JSON data to Elastic Stack.
First is checking the ES version, via CLI as below. We are using Elastic Stack 7.10+ (7.10.1)
Next create a default ES mapping per the following command. Please replace 192.168.1.100/9200 with your ES node address
The default mapping file is located below
Example output below
Next push a small part of stdout file manually to ES as follows. This writes the first 10 lines in the JSON file to the ES cluster
The correct output showing indexing has been successful as below
If the above is functioning correctly, can next push the output to ES
Pushing captures to ES in offline mode is good for lab/debug/troubleshooting a system. As its easier to control than the 24/7 running mode
Start by updating the config file /opt/fmadio/etc/pcap2json.lua as follows. The only real difference is "--output-espush" is set instead of "--output-stdout".
Ensure the "--es-host <es node hostname>:<port>:1e6" is set correctly
Next run the same offline command as the STDOUT version above
The output will look the same, except it will push data to the ES Host now instead of writing to a local file.
Trouble shooting and debug please see the logfiles https://docs.fmad.io/fmadio-documentation/v/pcap2json/logfiles
Generating a MAP requires the following. e.g. from Maxmind GeoIP Lite database + custom hosts file
In this case we have the Maxmind City Geo database
Then the MaxMind ASN database
Then the MaxMind ISP database
Then a custom Hosts file
Then a scratch-file (ensure that path to file is a ) as the process will require 200-500GB of scratch space to complete processing.
All resulting in a final command line as follows
The output file is shown below
Typically this is 400MB-2GB in size depending on the input dataset. Enabling it in pcap2json.lua configuration file requires
