Remote Push

Pushing PARQ files to a remote location enables a central datalake of PARQ metadata. This packet datalake can then be queried across an orginisations infrastructure regardless of the physical capture location.

Configuration File

Assuming a S3 Mount point of

/mnt/fmadio-metadata-ny2

Create a configuration file in

/opt/fmadio/etc/index.lua

Use the following configuration file

local Config = {}

-- where to write the data
Config.PathIndex = "/mnt/store1/index/"

-- copy location
Config.Copy = "/mnt/fmadio-meta-ny2/"

return Config

This will create the metadata on the local disk at location

/mnt/store1/index

And systematically copy all PARQ files as they are generated to the remote location (S3FS mount point) on

/mnt/fmadio-meta-ny2