Provides general purpose scriptable way to download PCAP from the system
/api/v1/pcap/timerange
Downloads a BPF and Frame filtered PCAP using start / end time only. The system will merge multiple captures transparently across the Start/End time specified.
Address
/api/v1/pcap/timerange
Verb
GET
Notes
The system will merge multiple captures transparently across the Start/End time specified.
Arguments
Following is a list of arguments
Time Range
Below selects the time range for download. If no TSUnit is specified, the system will attempt to detect what the unit of measure is for TSBegin/TSEnd.
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
TSBegin | Integer | Yes | Start time in EPOCH units of TSUnit (default nanoseconds) | |
TSEnd | Integer | Yes | Stop time in EPOCH units of TSUnit (default nanoseconds) | |
TSUint | String | Optional | Time Unit used in TSBegin / TSEnd selection | |
nanos (default) | Epoch is in Nanoseconds (1e9) | |||
msecs | Epoch is in Milliseconds (1e6) | |||
sec | Epoch is in Seconds | |||
YYYYMMDD_HHMMSS | Year Month Day Hour Min Second time format |
Time Limit
In some use cases (e.g. API integration) it can be useful to set a maximum amount of PCAP time the API call will download.
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
TSMax | Integer | Optional | Maximum nanosecond of packets to download. (this sets an upper bound on the TSMax value) |
FilterBPF
Core filtering mechanism, this uses the Berkley Packet Filter syntax (e.g. tcpdump) to filter packets
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
FilterBPF | String | Optional | BPF Filter to be applied to the stream. |
Filter BPFDecap
Enables or Disabled the FilterBPF to run after de-encapsulation. This helps simplify the FilterBPF to the inner payload
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
FilterBPFDecap | String | Optional | Enables or Disables FilterBPF run after de-encapsulation | |
true | FilterBPF run after packet de-encapsluation (e.g. on inner payload) | |||
false (default) | FilterBPF run on the raw packet without de-encapsulation |
Filter Frame
Filter the frame, e.g. selection of physical capture port of Tap Aggregation port number
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
FilterFrame | String | Optional | Filter on the Packet Frame | |
capture.port==<portid> | Selects FMADIO Capture FPGA Capture Port ID | |||
a7130.device==<device id> | Selects a specific Arista 7130 Device ID | |||
a7130.srcport==<port id> | Selects a specific Arista 7130 Ingress Port ID | |||
c3550.srcport==<portid> | Selects a specific Cisco 3550 Ingress Port ID |
Timestamp Mode
Allows selecting what timestamp gets written to each packet in the PCAP. Supporting different header and footer formats.
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
TSMode | String | Optional | Sets the Timestamp of the PCAP | |
nic (default) | FMADIO Capture FPGA timestamp (Default) | |||
arista7130 | Arista 7130 (Metamako) | |||
arista7150_overwrite | Arista 7150 FCS Overwrite | |||
arista7150_insert | Arista 7150 Insert 32bit | |||
arista7280_eth64 | Arista 7280 Ethernet 64bit header | |||
arista7280_mac48 | Arista 7280 SrcMAC 48bit Overwrite | |||
cisco_erspan3 | Cisco ERPSANv3 | |||
cisco3550 | Cisco 3550 (Exablaze) |
Compression
Optionally compress the PCAP, recommendation is zstd due to speed and compression ratio.
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
Compression | String | Optional | Compress the returned stream with gzip. | |
none (default) | No compression | |||
fast | Fastest compression but not smallest | |||
best | Best compression but slow | |||
1,2,3,4,5,6,7,8,9 | argument to gzip -1 -2 … -9 9 = Best | |||
zstd | Use ZSTD compressor with default settings | |||
zstd_fast | Use ZSTD compression with fast setting | |||
zstd_ultra | Use ZSTD compression with ultra maximum setting
|
Slicing
Optionally, slices (sets the maximum packet size in the PCAP) for all packets download
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
Slice | Number | Optional | 1-12288 | Amount of Bytes to slice the packet by |
Example
Example shows extracting data from a specific start/end nanosecond epoch time.
curl -u fmadio:xxx "http://192.168.2.75/pcap/timerange?TSBegin=1530498788000000000&TSEnd=1530498789000000000&" | tcpdump -r - -nn | headExample output shown below
curl -u fmadio:xxx "http://192.168.2.75/pcap/timerange?TSBegin=1530498788000000000&TSEnd=1530498789000000000&" | tcpdump -r - -nn | head
11:33:08.000000 66:77:88:99:aa:bb > 00:44:44:44:44:44 Null Information, send seq 22, rcv seq 1, Flags [Poll], length 54
0x0000: 0000 2c03 153a 2d03 153a 2e03 153a 2f03 ..,..:-..:...:/.
0x0010: 153a 3003 153a 3103 153a 3203 153a 3303 .:0..:1..:2..:3.
0x0020: 153a 3403 153a 3503 153a 3603 153a 3703 .:4..:5..:6..:7.
0x0030: 153a a878 4e26 .:.xN&
11:33:08.000000 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 22, rcv seq 1, Flags [Poll], length 54
0x0000: 0000 2c03 152a 2d03 152a 2e03 152a 2f03 ..,..*-..*...*/.
0x0010: 152a 3003 152a 3103 152a 3203 152a 3303 .*0..*1..*2..*3.
0x0020: 152a 3403 152a 3503 152a 3603 152a 3703 .*4..*5..*6..*7.
0x0030: 152a 7b57 491d .*{WI.
.
.
.
Example - Nanosecond Epoch
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1621772572136996000&
TSEnd=1621774913584264000"Example - Second Epoch
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSUnit=sec&
TSBegin=1621772572&
TSEnd=1621774913"Example - Year Month Day Hour Min
Using the TSUnit option can use a more friendly time selection.
By default it uses the TimeZone configured on the system
Fetch PCAP from 3AM to 4AM on 2023 / 10 (October) / 1st
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSUnit=YYYYMMDD_HHMMSS&
TSBegin=20231001_030000&
TSEnd=20231001_040000"Example - Year Month Day Hour Min with Timezone
Same as above but specifying the timezone.
NOTE: if using the full TSZone = Asia/Singapore for example, CURL will append a ? to the URL. Its recommended to use the City name only to avoid confusing CURL.
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSUnit=YYYYMMDD_HHMMSS&
TSZone=Singapore&
TSBegin=20231001_030000&
TSEnd=20231001_040000"Example - Year Month Day Hour Min Timezone and BPF Filter with De-encapsulation
Specifying all of the above with a BPF Filter, with BPF De-encapsulation enabled
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSUnit=YYYYMMDD_HHMMSS&
TSZone=Singapore&
TSBegin=20231001_030000&
TSEnd=20231001_040000&
FilterBPFDecap=true&"
-G --data-urlencode "FilterBPF=icmp"Example - Year Month Day Hour Min Timezone with BPF bi-direction host filter
Extracts a bi-directional flow of traffic between 2 hosts 1.2.3.4 and 5.6.7.8
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSUnit=YYYYMMDD_HHMMSS&
TSZone=Singapore&
TSBegin=20231001_030000&
TSEnd=20231001_040000&
-G --data-urlencode "FilterBPF=host 1.2.3.4 and host 5.6.7.8"Example - TSBegin, TSEnd and TSMax
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1621772572136996000&
TSEnd=1621774913584264000&
TSMax=100000Example - TSBegin, TSEnd and FilterBPF
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1621772572136996000&
TSEnd=1621774913584264000"
-G --data-urlencode "FilterBPF=tcp" Example - TSBegin, TSEnd and Compression
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1621772572136996000&
TSEnd=1621774913584264000&
Compression=fast"Example - TSBegin, TSEnd, FilterBPF and Compression
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1621772572136996000&
TSEnd=1621774913584264000&
Compression=fast"
-G --data-urlencode "FilterBPF=tcp"Example - Frame Filters Capture Port
Filter based on FMADIO Capture port number
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&"
-G --data-urlencode "FilterFrame=capture.port==0"
| tcpdump -r - -nn
| headMultiple capture ports
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&"
-G --data-urlencode "FilterFrame=capture.port==0,1,2,3"
| tcpdump -r - -nn
| headExcluding ports
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&"
-G --data-urlencode "FilterFrame=capture.port!=0"
| tcpdump -r - -nn
| headExample - Frame Filter Arista 7130
Filter for a specific 7130 Device 54932 (any port)
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&"
-G --data-urlencode "FilterFrame=a7130.srcdevice==54932"
| tcpdump -r - -nn
| headFilter for everything except a specific 7130 Device (not device id 54932)
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&"
-G --data-urlencode "FilterFrame=a7130.srcdevice!=54932"
| tcpdump -r - -nn
| headFilter for a specific 7130 Port number 1
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&"
-G --data-urlencode "FilterFrame=a7130.srcport==1"
| tcpdump -r - -nn
| headFilter for multiple 7130 Port numbers 1, 2, 3, 5, 10
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&"
-G --data-urlencode "FilterFrame=a7130.srcport==1,2,3,5,10"
| tcpdump -r - -nn
| headFilter for everything except 7130 Port number 10
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&"
-G --data-urlencode "FilterFrame=a7130.srcport!=10"
| tcpdump -r - -nn
| headFilter on a specific 7130 Port number and use the 7130 Footer Timestamp as the PCAP timestamp. Overriding the current TimeStamp setting
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&
TSMode=arista7130"
-G --data-urlencode "FilterFrame=a7130.srcport!=10"
| tcpdump -r - -nn
| headExample - Frame Filter Cisco 3550 (Exablaze)
Filter on a specific ingress port of the Cisco 3550, and use the Footer timestamp as the PCAP timestamp.
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&TSMode=cisco3550"
-G --data-urlencode "FilterFrame=c3550.srcport==48"
| tcpdump -r - -nn
| headExample - Packet Slicing TCP data
Filter on all TCP data slicing everything to 64 Bytes
curl "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1658744408270221800&
TSEnd=1658744501189259300&Slice=64"
-G --data-urlencode "FilterBPF=tcp"
| tcpdump -r - -nn
| head/api/v1/pcap/single
Downloads a single capture as a PCAP with optional time, port and BPF filtering
Address
/api/v1/pcap/single
Verb
GET
Notes
Arguments
Parameter | Type | Required | Value | Description |
|---|---|---|---|---|
StreamName | String | Yes | Name of the capture | |
FilterBPF | String | BPF Filter to be applied to the stream. | ||
FilterBPFDecap | String | Enables or Disables FilterBPF run after de-encapsulation | ||
true | FilterBPF run after packet de-encapsluation (e.g. on inner payload) | |||
false (default) | FilterBPF run on the raw packet without de-encapsulation | |||
FilterFrame | String | Filter on the Packet Frame | ||
capture.port==<portid> | Selects FMADIO Capture FPGA Capture Port ID | |||
a7130.device==<device id> | Selects a specific Arista 7130 Device ID | |||
a7130.srcport==<port id> | Selects a specific Arista 7130 Ingress Port ID | |||
c3550.srcport==<portid> | Selects a specific Cisco 3550 Ingress Port ID | |||
TSMode | String | Sets the Timestamp of the PCAP | ||
nic (default) | FMADIO Capture FPGA timestamp (Default) | |||
arista7130 | Arista 7130 (Metamako) | |||
arista7150_overwrite | Arista 7150 FCS Overwrite | |||
arista7150_insert | Arista 7150 Insert 32bit | |||
arista7280_eth64 | Arista 7280 Ethernet 64bit header | |||
arista7280_mac48 | Arista 7280 SrcMAC 48bit Overwrite | |||
cisco_erspan3 | Cisco ERPSANv3 | |||
cisco3550 | Cisco 3550 (Exablaze) | |||
Compression | String | Compress the returned stream with gzip. | ||
none (default) | No compression | |||
fast | Fastest compression but not smallest | |||
best | Best compression but slowest | |||
1,2,3,4,5,6,7,8,9 | Compression levels sent to gzip. 9 = Best | |||
zstd | Use ZSTD compressor with default settings | |||
zstd_fast | Use ZSTD compression with fast setting | |||
zstd_ultra | Use ZSTD compression with ultra maximum setting
|
Example
Example shows extracting TCP data with FilterBPF and outputing via tcpdump. Command is below
curl -u fmadio:xxx "http://192.168.2.75/api/v1/pcap/single?StreamName=TestCapture_20180702_1127&&FilterBPF=tcp" | tcpdump -r - -nn | headExample output shown below
$ curl -u fmadio:xxx "http://192.168.2.75/api/v1/pcap/single?StreamName=TestCapture_20180702_1127&&FilterBPF=tcp" | tcpdump -r - -nn | head
11:33:08.000000 66:77:88:99:aa:bb > 00:44:44:44:44:44 Null Information, send seq 22, rcv seq 1, Flags [Poll], length 54
0x0000: 0000 2c03 153a 2d03 153a 2e03 153a 2f03 ..,..:-..:...:/.
0x0010: 153a 3003 153a 3103 153a 3203 153a 3303 .:0..:1..:2..:3.
0x0020: 153a 3403 153a 3503 153a 3603 153a 3703 .:4..:5..:6..:7.
0x0030: 153a a878 4e26 .:.xN&
11:33:08.000000 66:77:88:99:aa:bb > 00:33:33:33:33:33 Null Information, send seq 22, rcv seq 1, Flags [Poll], length 54
0x0000: 0000 2c03 152a 2d03 152a 2e03 152a 2f03 ..,..*-..*...*/.
0x0010: 152a 3003 152a 3103 152a 3203 152a 3303 .*0..*1..*2..*3.
0x0020: 152a 3403 152a 3503 152a 3603 152a 3703 .*4..*5..*6..*7.
0x0030: 152a 7b57 491d .*{WI.
.
.
.
.
. Example - StreamName only.
All arguments as defaults
curl -u fmadio:xxxxx "http://127.0.0.1/api/v1/pcap/single?
StreamName=stream_test"Example - StreamName and FilterBPF
With a FilterBPF (without de-encapsulation)
curl -u fmadio:xxxxx "http://127.0.0.1/api/v1/pcap/single?
StreamName=stream_test_001"
-G --data-urlencode "FilterBPF=tcp"Example - StreamName and Compression
With fast gzip compression
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/single?
StreamName=stream_test_001&
Compression=fast"Example - StreamName, Compression and FilterBPF
Compression and BPF Filter
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/single?
StreamName=stream_test_001&
Compression=fast"
-G --data-urlencode "FilterBPF=tcp"Encapsulation Debugging
Many times the exact packet encapsulation is unclear, the following uses a wireshark filter expression to extract and show the full encapsulation format of the packet. From this a high speed BPF filter can be used to process the data.
In the below example we are using the Wireshark filter "ip.addr == 192.168.1.1" on a historical capture.
curl -u fmadio:xxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1666706401000000000&
TSEnd=1666706401010000000"
| tshark -r - -T fields -e frame.protocols -e ip.src -e ip.dst
-Y "ip.addr == 192.168.1.1"Alternatively running on the currently running capture via SSH on the fmadio box looks like the following. This example filters on any UDP traffic.
sudo stream_cat
| tshark -r - -T fields -e frame.protocols -e ip.src -e ip.dst -Y "udp"
| headThe output looks like the following
eth:ethertype:vlan:ethertype:ip:udp:ntp 106.10.186.200 192.168.133.10
eth:ethertype:vlan:ethertype:ip:udp:ntp 106.10.186.201 192.168.133.10
eth:ethertype:vlan:ethertype:ip:udp:ntp 167.172.70.21 192.168.133.10
eth:ethertype:vlan:ethertype:ip:udp:ntp 106.10.186.200 192.168.133.10
eth:ethertype:vlan:ethertype:ip:udp:ntp 106.10.186.201 192.168.133.10
eth:ethertype:vlan:ethertype:ip:udp:ntp 167.172.70.21 192.168.133.10
.
.
The above output shows there is a single VLAN tag in the packet. Making the equivalent BPF filter
vlan and udpWith the final BPF filter using a CURL request
curl -u fmadio:xxxxx "http://127.0.0.1/api/v1/pcap/timerange?
TSBegin=1671407102&
TSEnd=1671407752&
TSMode=sec&"
-G --data-urlencode "FilterBPF=vlan and udp"
| tcpdump -r - -n
| headOutput per below
23:47:45.409489 IP 106.10.186.201.123 > 192.168.133.10.123: NTPv4, Server, length 48
23:52:14.407364 IP 167.172.70.21.123 > 192.168.133.10.123: NTPv4, Server, length 48
23:55:42.405072 IP 106.10.186.200.123 > 192.168.133.10.123: NTPv4, Server, length 48