Batch Processing

Re-processing already captured data with a new version of Packetscope2 can be useful. The following steps demonstrate how to do this.

Step 1) Generate list of captures to process

Using the command

fmadiocli "show capture list"

This will list all captures on the system.

Example shown below

fmadio@fmadio100v2-228U:/mnt/store0/tmp2$ fmadiocli "show capture list"
fmad fmadlua Sep  7 2024 (/opt/fmadio/bin/fmadiolua --nocal /opt/fmadio/bin/fmadiocli show capture list )
Disable cycle calibration
[Thu Mar 27 17:43:50 2025] CmdLine [show capture list]
[Thu Mar 27 17:43:50 2025] Cmd [show capture list]
[Thu Mar 27 17:43:50 2025] Showing captures
[Thu Mar 27 17:43:50 2025] [l1_25G_20250207_2300                                                            ]            2,621,440 B (Fri . 23:59:58 . 07-02-2025)
[Thu Mar 27 17:43:50 2025] [l1_25G_20250207_2000                                                            ]      354,467,446,784 B (Fri . 20:15:16 . 07-02-2025)
[Thu Mar 27 17:43:50 2025] [l1_25G_20250207_1900                                                            ]    2,174,831,558,656 B (Fri . 19:25:15 . 07-02-2025)
[Thu Mar 27 17:43:50 2025] [ny2-election_20250125_1550                                                      ]    3,273,740,255,232 B (Wed . 23:00:01 . 06-11-2024)
[Thu Mar 27 17:43:50 2025] [teleco_20241218_1922                                                            ]      110,280,835,072 B (Fri . 21:08:03 . 06-10-2023)
[Thu Mar 27 17:43:50 2025] [euronext_20241219_1638                                                          ]       81,132,519,424 B (Thu . 20:00:01 . 03-08-2023)
[Thu Mar 27 17:43:50 2025] [interop_full_20250109_1538                                                      ]      100,260,642,816 B (Fri . 21:45:03 . 09-06-2017)
[Thu Mar 27 17:43:50 2025] [interop17_20241218_1917                                                         ]        5,014,290,432 B (Fri . 21:40:10 . 09-06-2017)
.
.

In this example we want to process 2 captures named

ny2-election_20250125_1550
interop_full_20250109_1538

Step 2) list of captures in a file

create a file, in this case file_list and place the names of captures to be processed one line at a time.

An example is shown below

fmadio@fmadio100v2-228U:/mnt/store0/tmp2$ cat file_list
ny2-election_20250125_1550
interop_full_20250109_1538

Step 3) Run batch processing script

Run the batch processing script with the format

pscope2_batch.lua  <file path of list>

Example is shown below

fmadio@fmadio100v2-228U:/mnt/store0/tmp2$ pscope2_batch.lua  ./file_list
fmad fmadlua Sep  7 2024 (/opt/fmadio/bin/fmadiolua /opt/fmadio/bin/pscope2_batch.lua ./file_list )
Batch Processing [./file_list]
StreamList 2
   1 : interop_full_20250109_1538
   2 : sky_red_short_20250322_1554
Processing    1 : interop_full_20250109_1538
reset [sudo /mnt/store0/tmp2/pscope2/pscope2_ingress.lua interop_full_20250109_1538]
fmad fmadlua Sep  7 2024 (/opt/fmadio/bin/fmadiolua /mnt/store0/tmp2/pscope2/pscope2_ingress.lua interop_full_20250109_1538 )
1       pscope2-20250323-0403
2       RUNNING
3       0
4       -
5       192.168.255.50
6       -
7
echo "interop_full_20250109_1538" > /opt/fmadio/lxc/pscope2-20250323-0403/rootfs/tmp/capture_name
systemctl restart fmadio-pscope2.service
done 0.104781Sec 0.001746Min
Process [sudo /opt/fmadio/bin/stream_cat --uid pscope2_batch -v --ring /opt/fmadio/queue/lxc_pscope2 interop_full_20250109_1538 2>&1 ]
[interop_full_20250109_1538]   0.04 Min Pending:   100.260 GB PCAPTS:00:00:00.000.000.000 Processing:   0.000 Gbps
[interop_full_20250109_1538]   0.05 Min Pending:    99.196 GB PCAPTS:13:39:56.966.909.082 Processing:  16.737 Gbps
[interop_full_20250109_1538]   0.07 Min Pending:    98.350 GB PCAPTS:13:39:59.482.726.685 Processing:  13.329 Gbps
[interop_full_20250109_1538]   0.09 Min Pending:    97.497 GB PCAPTS:13:40:02.095.980.759 Processing:  13.451 Gbps
[interop_full_20250109_1538]   0.10 Min Pending:    96.645 GB PCAPTS:13:40:04.732.637.386 Processing:  13.413 Gbps
[interop_full_20250109_1538]   0.12 Min Pending:    94.861 GB PCAPTS:13:40:10.255.663.677 Processing:  28.121 Gbps
.
.
.
.

This will continue to run until all files have been processed and updated. This may take some time depending on how much data has been processed.

Step 4) Confirm data

Confirm all data is present in Grafana after processing has completed.