Network Topology

Introduction

FMADIO Devices have a unique network topology. While we use the Linux operating system the combination of FPGA devices, LXCs, Time Synchronization and custom interfaces make the topology less standard.

Below is a generic network topology diagram showing the standard configuration.

FMADIO Network Topology

Description

Linux Physical Network Interface

This indicates a regular linux network interface, typically using the in-kernel network driver.

FMADIO network refer to these as “phy” physical interfaces, for example phy0. Typically in linux systems the physical network interface is the logical network interface, for example eth0. FMADIO systems its slightly more abstracted.

Generally

phy0, phy1 are the low speed 1G RJ45 interfaces on the system
phy10, phy11 are the SFP/SFP28/QSFP/QSFP28 10G - 100G high speed management interfaces

Example:

phy0
phy1
phy10
phy11

Linux External Bridge Interface

Linux has the concept of a “bridge” network interface. It runs and operates exactly the same as a physical network interface but has not physical network drivers attached. FMADIO systems refer to these External Bridge interfaces as management interface, and named man0, man1, man10, man11.

Typically a single physical interface is attached to a logical management interface. For example below is how phy0 and man0 are mapped.

fmadio@fmadio100v2-228U:~$ sudo brctl show man0
bridge name     bridge id               STP enabled     interfaces
man0            8000.e0d55e5d2150       no              phy0
fmadio@fmadio100v2-228U:~$

In the above we see man0 as the logical bridge interface, which a single phy0 interface atached.

The reason for this structure is described later, primary it allows an LXC Container to use bridged networking with its own IP address on the same physical network interface.

Example:

man0
man1
man10
man11

Linux Internal Bridge Interface

A linux internal bridge is a standard linux bridge but has no physical network ports attached. Its primary purpose is a private interface from the FMADIO Host system to the FMADIO LXC container system, where traffic will never egress from the box.

This interfaces are named “fmad0” and have a static IP allowing FMADIO Host and LXC containers to have pre-defined network topology without worry of the external physical network topology.

Example:

fmad0

FPGA Capture Interface

Finally the FPGA Capture interfaces are also shown in the above diagram. These interfaces have no linux networking connectivity and reside outside the usual linux network configuration.

FPGA Capture ports have no regular linux network interfaces

These interfaces are controlled entirely by the FMADIO OS, the usual Linux networking tools do not operate with them.

Example:

cap0
cap1
cap2
cap3
cap4
cap5
cap6
cap7

Interface List

bmc

The bmc interface is not present on the FMADIO Host system, however all systems have a physical BMC network interface which requires IP port, subnet, gateway etc configuration.

phy0

This is the 1G RJ45 based copper physical network interface.

phy1

This is the second 1G RJ45 based copper physical network interface.

man0

This is the default 1G low speed management port. Typically its configured with a static IP address and the primary management interface. Its usually bridged with phy0 physical interface

man1

This can be a secondary interface, one typical use case is to run the PTPv2 network on the man1 interface so it does not interfere with the general management network

phy10

High speed physical interface, typically an Intel 710 or 810 based 10/25/40/100G physical interface.

phy11

High speed physical interface, typically an Intel 710 or 810 based 10/25/40/100G physical interface.

man10

Usually a 10G, 40G or 100G high speed management interface. This provides high speed data transfer and access to the system.

man11

Also a 10G, 40G, 100G high speed management interface. One typical example is man10 provides general core network access, while man11 provides dedicated cloud network access.

fmad0

This is the internal bridged network, it runs on the subnet 192.168.255.0/24

cap0

FPGA capture port either a physical SFP/SFP28/QSFP28 interface. In some scenarios this may be a lane within a QSFP28/MPO12 physical connection.

cap1

FPGA capture port either a physical SFP/SFP28/QSFP28 interface. In some scenarios this may be a lane within a QSFP28/MPO12 physical connection.