API interface for Packetscope2
The following API end points use PCAP interface and a custom target.lua configuration file. This allows custom processing of the streamed PCAP files
Utilities file is located at
https://static.fmad.io/download/container/pscope2_utils_20250319.tar.gz
MD5 sum bd775af211d9bb113cb67d8a8e790e9e
1) Download the above
2) copy to /mnt/store0/tmp2
3) Extract the archive
cd /mnt/store0/tmp2/
tar xfzv pscope2_utils_20250319.tar.gz4) Overwrite the target.lua file
cp /mnt/store0/tmp2/pscope2/target.lua /opt/fmadio/etc/target.lua/api/v1/capture/pcap/single?Target=pscope2_process
This will process a full capture named StreamName which resides on the capture system into the Packetscope 2 container.
Address
/api/v1/capture/pcap/single?Target=pscope2_process
Verb
GET
Notes
Arguments
Parameter | Description |
|---|---|
StreamName | Name of the capture |
Example
Example show processing a specific capture file previously captured (in this case the capture is named ny2_election_20250204_2245
fmadio@fmadio100v2-228U:~$ curl -u fmadio:xxx -k "https://127.0.0.1/api/v1/pcap/single?Target=pscope2_process&StreamName=ny2_election_20250204_2245"
│{"tstr":"20250319_123841", "timestamp":1742359121,"PktCnt": 857960770, "PktByte": 119093146198, "ChunkID":60272653,"PCAPTS":"14:31:07.685.743.690","PendingB":745013248, "Read_bps":4827339222,"Read_pps":4359387, "Write_bps":9652734495,"Write_pps":8718775, "FwdPct":2.000 }It will likely takes several minutes to process the file.
To monitor its progress running the following
sudo lxc-attach -n <pscope2-yyyymmdd-hhmm> -- journalctl -f
Example output shown below
Mar 19 04:48:13 fmadio100v2-228U-pscope2 pscope2.lua[2810]: [14:30:31.876.139.708] 1730903431876139708 59435732702 63882 55.354/0.096 GB 3.22 Mpps 3.48 Gbps | cat 0 MB 0.00 0.00 0.00 | Flows/Snap: 0: 0 FlowCPU:0.00 0.00 | Output 0.000 K Lines/sec 0.000 Gbps
Mar 19 04:48:14 fmadio100v2-228U-pscope2 pscope2.lua[2810]: [14:30:32.167.900.052] 1730903432167900052 59946010772 64458 55.829/0.097 GB 3.63 Mpps 4.08 Gbps | cat 0 MB 0.00 0.00 0.00 | Flows/Snap: 0: 0 FlowCPU:0.00 0.00 | Output 0.000 K Lines/sec 0.000 Gbps/api/v1/capture/pcap/single?Target=pscope2_dump
This will dump the full processed JSON data from the specified StreamName to the standard out processing
Address
/api/v1/capture/pcap/single?Target=pscope2_dump
Verb
GET
Notes
Arguments
Parameter | Description |
|---|---|
StreamName | Name of the capture |
Example
Example shows the first few lines of an output. Please note the amount of output data can be an order of magnitude larger than the source PCAP file.
fmadio@fmadio100v2-228U:~$ curl -u fmadio:xxx -k "https://127.0.0.1/api/v1/pcap/single?Target=pscope2_dump&StreamName=ny2_election_20250204_2245" | head | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0{
"TS": "0",
"timestamp": "2024-11-06 14:31:07",
"system": "fmadio100v2-228U-pscope2",
"site": "",
"capture": "ny2_election_20250204_2245",
"hashHalf": "34a64b47bb0fbb4e7cc8722d70e175f7353e8e3d",
"frameProto": ":ether:ipv4:udp",
"ether_0_type": "0x0800",
"ether_0_src": "ec:38:73:5b:eb:40",
"ether_0_dst": "01:00:5e:00:cc:4a",
"ether_1_type": "",
"ether_1_src": "",
"ether_1_dst": "",
"ether_2_type": "",
"ether_2_src": "",
"ether_2_dst": "",
.
.
.
Alternatively writing this to a JSON file is another approach
fmadio@fmadio100v2-228U:~$ curl -u fmadio:xxx -k "https://127.0.0.1/api/v1/pcap/single?Target=pscope2_dump&StreamName=ny2_election_20250204_2245" > output.json
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 330M 0 330M 0 0 87.7M 0 --:--:-- 0:00:03 --:--:-- 87.7M
fmadio@fmadio100v2-228U:~$
/api/v1/capture/pcap/single?Target=push_offline
This runs an offline Push PCAP using the specific StreamName. This requires the Push PCAP functions to be pre-configured before running.
Address
/api/v1/capture/pcap/single?Target=push_offline
Verb
GET
Notes
Arguments
Parameter | Description |
|---|---|
StreamName | Name of the capture |
Example
Example of running
fmadio@fmadio100v2-228U:~$ curl -u fmadio:xxx -k "https://127.0.0.1/api/v1/pcap/single?Target=push_offline&StreamName=ny2_election_20250204_2245"
Monitoring of the process is in the logfile
$ tail -F /mnt/store0/log/stream_command.curOr in the specific push_pcap logfile, for example using a Push Target named “Test”
fmadio@fmadio100v2-228U:~$ tail -F /mnt/store0/log/push_pcap_Test.cur
[0.000 H][2024-11-06 22:30:11] /mnt/store0/pushpcap/Test20241106_223011.pcap : Total Bytes 28161431034 167681290 28.161 GB Speed: 1.730 Gbps 1.371 Mpps : TotalSplit 284 PCAPTS: 1730903411926565522
fmad fmadlua Sep 7 2024 (/opt/fmadio/bin/fmadiolua --nocal /opt/fmadio/analytics/push_pcap_close.lua /mnt/store0/pushpcap/Test20241106_223011.pcap 2507091460 15576073 12149790000 1000000024 1730903411000000000 1730903412000000000 1730903411000000023 1730903411999999833 1730903412000000047 )
Disable cycle calibration
done 0.001318Sec 0.000022Min
[0.038 H][2024-11-06 22:30:12] /mnt/store0/pushpcap/Test20241106_223011.pcap : Finished : Split Bytes 2507091460 (2.507 GB) Split Pkts: 15576073 WallTime: 12149790000 PCAPTime: 1000000024
Script [/opt/fmadio/analytics/push_pcap_close.lua /mnt/store0/pushpcap/Test20241106_223011.pcap 2507091460 15576073 12149790000 1000000024 1730903411000000000 1730903412000000000 1730903411000000023 1730903411999999833 1730903412000000047]
[cat > '/mnt/store0/pushpcap/Test20241106_223012.pcap.pending']
[0.038 H][2024-11-06 22:30:12] /mnt/store0/pushpcap/Test20241106_223012.pcap : Total Bytes 28.319 GB Speed: 1.642 Gbps : New Split
[0.000 H][2024-11-06 22:30:12] /mnt/store0/pushpcap/Test20241106_223012.pcap : Total Bytes 28411275524 169301099 28.411 GB Speed: 1.675 Gbps 1.357 Mpps : TotalSplit 285 PCAPTS: 1730903412036390276
[0.000 H][2024-11-06 22:30:12] /mnt/store0/pushpcap/Test20241106_223012.pcap : Total Bytes 28658312092 170883233 28.658 GB Speed: 1.656 Gbps 1.326 Mpps : TotalSplit 285 PCAPTS: 1730903412142087319