Operation

  • Updated on Feb 21, 2025
  • Published on Feb 21, 2025
  • 5 minute(s) read
  • aA
 Prev Next

Resetting Database

Sometimes too much data ends up in the database, requiring a database reset to clear out all contents. This can be done at any time and helpful when investigating new problems.

The following steps will reset and delete all database content, please use with caution.

Step 1) Attach to the LXC

Attach to the LXC container from the host. Example as below

 sudo lxc-attach -n pscope2-20250221-1052

Step 2) Change directory

(run inside the LXC container)

Move to the /root/pscope2 directory per

cd /root/pscope2

Step 3) Run the reset script

(run inside the LXC container)

Running the reset script as follows

 ./pscope2-reset.lua

Example output as follows. It will take 30-60sec to run depending on how much data is in the database

root@fmadio100v2-228U-pscope2:~/pscope2# ./pscope2-reset.lua
clickhouse stop
/var/run/clickhouse-server/clickhouse-server.pid file exists and contains pid = 167.
The process with pid = 167 is running.
Sent terminate signal to process with pid 167.
Waiting for server to stop
/var/run/clickhouse-server/clickhouse-server.pid file exists and contains pid = 167.
The process with pid = 167 is running.
Waiting for server to stop
/var/run/clickhouse-server/clickhouse-server.pid file exists and contains pid = 167.
The process with pid = 167 is running.
.
.
clickhouse-client --password fmad-secret --query "ALTER TABLE system.query_log MODIFY SETTING ttl_only_drop_parts=1"
clickhouse-client --password fmad-secret --query "ALTER TABLE system.part_log MODIFY SETTING ttl_only_drop_parts=1"
Received exception from server (version 24.4.1):
Code: 60. DB::Exception: Received from localhost:9000. DB::Exception: Could not find table: part_log. (UNKNOWN_TABLE)
(query: ALTER TABLE system.part_log MODIFY SETTING ttl_only_drop_parts=1)
clickhouse-client --password fmad-secret --query "ALTER TABLE system.asynchronous_metric_log MODIFY SETTING ttl_only_drop_parts=1"
Received exception from server (version 24.4.1):
Code: 60. DB::Exception: Received from localhost:9000. DB::Exception: Could not find table: asynchronous_metric_log. (UNKNOWN_TABLE)
(query: ALTER TABLE system.asynchronous_metric_log MODIFY SETTING ttl_only_drop_parts=1)
clickhouse-client --password fmad-secret --query "ALTER TABLE system.metric_log MODIFY SETTING ttl_only_drop_parts=1"
clickhouse-client --password fmad-secret --query "set input_format_skip_unknown_fields=1"
Done.
root@fmadio100v2-228U-pscope2:~/pscope2#

Step 4) Done

Database reset has been completed

Restarting Daemon

Sometimes its best to restart the processing daemon. Such as after resetting the database and working on a completely difference capture file. This will clear out any state the decoder process currently.

Step 1) Attach to the LXC

Attach to the LXC container from the host. Example as below

 sudo lxc-attach -n pscope2-20250221-1052

Step 2) Restart the process

(run inside the LXC container)

Using the systemd utilities restart the process with

 systemctl restart fmadio-pscope2.service

Example output

root@fmadio100v2-228U-pscope2:~/pscope2# systemctl restart fmadio-pscope2.service
root@fmadio100v2-228U-pscope2:~/pscope2#

Step 3) Confirm Operation is good

(run inside the LXC container)

Confirm the operation is good using

systemctl status fmadio-pscope2.service

Example output, green status shows correct operation

root@fmadio100v2-228U-pscope2:~/pscope2# systemctl status fmadio-pscope2.service
* fmadio-pscope2.service - FMADIO Packetscope2
     Loaded: loaded (/etc/systemd/system/fmadio-pscope2.service; enabled; vendor preset: enabled)
    Drop-In: /run/systemd/system/service.d
             `-zzz-lxc-service.conf
     Active: active (running) since Fri 2025-02-21 11:11:51 UTC; 52s ago
   Main PID: 2103 (pscope2.lua)
     CGroup: /system.slice/fmadio-pscope2.service
             |-2103 /usr/bin/bash /opt/fmadio/app/pscope2/bin/pscope2.lua
             |-2104 /opt/fmadio/bin/fmadio2pcap -i /opt/fmadio/queue/lxc_pscope2
             |-2105 /opt/fmadio/app/pscope2/bin/pcap2json2 -v --json-flow --decode-gtpc --decode-gtpu
             `-2106 /usr/bin/luajit /opt/fmadio/app/pscope2/bin/ingest.lua

Feb 21 11:12:35 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:35 2025] flushing (         0) Total:0.000M
Feb 21 11:12:36 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:36 2025] flushing (         0) Total:0.000M
Feb 21 11:12:37 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:37 2025] flushing (         0) Total:0.000M
Feb 21 11:12:38 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:38 2025] flushing (         0) Total:0.000M
Feb 21 11:12:39 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:39 2025] flushing (         0) Total:0.000M
Feb 21 11:12:40 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:40 2025] flushing (         0) Total:0.000M
Feb 21 11:12:41 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:41 2025] flushing (         0) Total:0.000M
Feb 21 11:12:42 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:42 2025] flushing (         0) Total:0.000M
Feb 21 11:12:43 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:43 2025] flushing (         0) Total:0.000M
Feb 21 11:12:44 fmadio100v2-228U-pscope2 pscope2.lua[2106]: [Fri Feb 21 11:12:44 2025] flushing (         0) Total:0.000M
root@fmadio100v2-228U-pscope2:~/pscope2#

Debugging

(run inside the LXC container)

If any of the above steps are showing problems the journal logfile can be quite useful. Run the following to follow the logfile

 journalctl -n 100 -f

This will output the last 100 current log lines

Sending data for Analysis

After install sending data to Packetscope2 for analysis is the next step. This is achieved using the host stream_cat command to the FMADIO Ring buffers.

Step 1) Find the capture

Find a capture to send for analysis using 

sudo stream_dump

Locate the full capture name and save it

Example output, in this example we will use the interop_full_20250109_1538 capture file

Step 2) stream_cat the capture

Using stream_cat send the specified file to Packetscope2 for analysis example command

sudo stream_cat -v --ring /opt/fmadio/queue/lxc_pscope2 interop_full_20250109_1538

Example output

fmadio@fmadio100v2-228U:~$ sudo stream_cat -v --ring /opt/fmadio/queue/lxc_pscope2 interop_full_20250109_1538
Create FMAD Ring: 0 [/opt/fmadio/queue/lxc_pscope2]
stream_cat ioqueue: 4
TimeStamp[0] --pcap
TimeStamp[1] --pcap
TimeStamp[2] --pcap
TimeStamp[3] --pcap
TimeStamp[4] --pcap
TimeStamp[5] --pcap
TimeStamp[6] --pcap
TimeStamp[7] --pcap
RING[/opt/fmadio/queue/lxc_pscope2                     ] 00 : CPU:   0 FilterBPF:[(null)] FilterFrame:[(null)]
StartChunkID: 44584040
StartChunk: 44584040 Offset: 0 Stride: 1
StartChunk: 44584040
RING[/opt/fmadio/queue/lxc_pscope2                     ] Size   : 12595200 16777216
RING[/opt/fmadio/queue/lxc_pscope2                     ] Version:      100      100
RING[/opt/fmadio/queue/lxc_pscope2                     ] Put:a2c5a659 259 0x7f494a8c1000
RING[/opt/fmadio/queue/lxc_pscope2                     ] Get:a2c5a659 259 0x7f494a8c2000
RING[/opt/fmadio/queue/lxc_pscope2                     ] thread:0
{"tstr":"20250221_191954", "timestamp":1740136794,"PktCnt":         6948846, "PktByte":      8001462074, "ChunkID":44614994,"PCAPTS":"13:40:18.704.542.861","PendingB":92145975296,"Read_bps":16880460611,"Read_pps":1839701,"Write_bps":33716658114,"Write_pps":3679402,"FwdPct":2.000,""CPUIdle":0.000,"CPUFetch":0.528, "CPUSend":0.000}
{"tstr":"20250221_191955", "timestamp":1740136795,"PktCnt":         8786681, "PktByte":     10108237494, "ChunkID":44623143,"PCAPTS":"13:40:25.156.191.475","PendingB":90009763840,"Read_bps":16853552914,"Read_pps":1837764,"Write_bps":33663168356,"Write_pps":3675528,"FwdPct":2.000,""CPUIdle":0.000,"CPUFetch":0.526, "CPUSend":0.000}
{"tstr":"20250221_191956", "timestamp":1740136796,"PktCnt":        10614757, "PktByte":     12201881962, "ChunkID":44631241,"PCAPTS":"13:40:31.756.566.192","PendingB":87886921728,"Read_bps":16747492366,"Read_pps":1827894,"Write_bps":33450859907,"Write_pps":3655789,"FwdPct":2.000,""CPUIdle":0.000,"CPUFetch":0.523, "CPUSend":0.000}
{"tstr":"20250221_191957", "timestamp":1740136797,"PktCnt":        12466309, "PktByte":     14316887110, "ChunkID":44639422,"PCAPTS":"13:40:38.399.475.552","PendingB":85742321664,"Read_bps":16917371775,"Read_pps":1851260,"Write_bps":33790309777,"Write_pps":3702520,"FwdPct":2.000,""CPUIdle":0.000,"CPUFetch":0.528, "CPUSend":0.000}
{"tstr":"20250221_191958", "timestamp":1740136798,"PktCnt":        14311107, "PktByte":     16437974669, "ChunkID":44647626,"PCAPTS":"13:40:45.039.519.369","PendingB":83591692288,"Read_bps":16965932870,"Read_pps":1844497,"Write_bps":33887115696,"Write_pps":3688994,"FwdPct":2.000,""CPUIdle":0.000,"CPUFetch":0.531, "CPUSend":0.000}
{"tstr":"20250221_191959", "timestamp":1740136799,"PktCnt":        16126410, "PktByte":     18516540292, "ChunkID":44655666,"PCAPTS":"13:40:51.703.297.459","PendingB":81484054528,"Read_bps":16628098274,"Read_pps":1815256,"Write_bps":33212186589,"Write_pps":3630513,"FwdPct":2.000,""CPUIdle":0.000,"CPUFetch":0.522, "CPUSend":0.000}
{"tstr":"20250221_192000", "timestamp":1740136800,"PktCnt":        17936687, "PktByte":     20553336476, "ChunkID":44663546,"PCAPTS":"13:40:57.923.855.449","PendingB":79418359808,"Read_bps":16293456763,"Read_pps":1810177,"Write_bps":32543864490,"Write_pps":3620353,"FwdPct":2.000,""CPUIdle":0.000,"CPUFetch":0.514, "CPUSend":0.000}
.
.

Step 3) Confirm Grafana

After starting the processing, data will by visible on the Grafana dashboard.

NOTE: in this example the PCAP is quite old (2017) thus the timerange selection in Grafana must be wide. The Time range is based on the PCAP timestamps (not the current system time)

Step 4) Done

Completed, multiple ingests can be done this can be quite helpful when analyzing a problem over multiple days / time periods.