Occasionally a firmware upgrade or system transport has a problem resulting in the FPGA image getting corrupted in some form. When that occurs the FPGA defaults back to the factory Xilinx golden image. The steps here are how to recover the system from that state.
1) FPGA PCIe Address
The initial requirement is to confirm the FPGA is on the PCIe Bus
Model 1U Gen4
3214
3414
3514
5214
5214A
5414
5414A
5514
5514
Confirm what device is on bus
e1:00
e2:00
Depending on the firmware installed and port configuration one or both PCI endpoints will have a device
command is
sudo lspci -s e1:00 -v2) Check PCI device
Run the lspci command from step 1) above
FPGA Good
Example below shows a GOOD result
fmadio@fmadio200v4-636:~$ sudo lspci -s e1:00
e1:00.0 Ethernet controller: FMAD Engineering FNIC200 (rev c0)Seeing the above means no recovery process is required. A normal FMADIO Firmware upgrade only.
BIOS Problem
Example below shows a FAIL BIOS setting
fmadio@fmadio200v4-636:~$ sudo lspci -s e1:00
e1:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PROSeeing the above means there is a BIOS problem and the FPGA Image problem. It requires going into the BIOS and changing the default settings
BIOS → Advanced → PCI Subsystem Settings
Change U2_P1_P0 Lanes to Auto
.png?sv=2022-11-02&spr=https&st=2025-07-25T03%3A27%3A55Z&se=2025-07-25T03%3A42%3A55Z&sr=c&sp=r&sig=jrAPgKXk6E187PscjLQUoOVYiX2Sp9Yz9QsEHVFwJUQ%3D)
Then save and exit per below
Save & Exit → Save Changes and Exit
.png?sv=2022-11-02&spr=https&st=2025-07-25T03%3A27%3A55Z&se=2025-07-25T03%3A42%3A55Z&sr=c&sp=r&sig=jrAPgKXk6E187PscjLQUoOVYiX2Sp9Yz9QsEHVFwJUQ%3D)
( the problem is the Xilinx default factory image fails to link up in x8×8 PCIe mode)
Factory Default
Example output shows the system when the default FPGA Xilinx factory image is loaded.
fmadio@fmadio200v4-636:/mnt/store0/develop$ lspci -s e1:00 -v
e1:00.0 Processing accelerators: Xilinx Corporation Alveo U55C
Subsystem: Xilinx Corporation Alveo card
Flags: fast devsel, NUMA node 5
Memory at b00dc000000 (64-bit, prefetchable) [disabled] [size=32M]
Memory at b00de000000 (64-bit, prefetchable) [disabled] [size=128K]
Capabilities: <access denied>
fmadio@fmadio200v4-636:/mnt/store0/develop$This is the sate where the f200v4 factory recovery can be used. Please proceed to step 3)
Other
If none of these outputs match, please contact support @ fmad.io for further assitance.
Step 3) Flash recovery
After Confirming the Xilinx Processing Accelerator is shown on the PCI bus run the following to write the FMADIO FPGA Image onto the device
fmadio@fmadio200v4-636:/mnt/store0/develop$ lspci -s e1:00 -v
e1:00.0 Processing accelerators: Xilinx Corporation Alveo U55C
fmadio@fmadio200v4-636:/mnt/store0/develop$Change to directory
cd /opt/fmadio/firmware/Run the recovery process
sudo ./f200v4-recover.shA successful output is shown below. It takes about 5minutes to run
fmadio@fmadio200v4-636:/opt/fmadio/firmware$ sudo ./f200v4-recover.sh
About to flash below MCS bitstream onto card e1:00.0:
/opt/fmadio/firmware/fNICBoard_U55C_2x100G_fallback.bin
flashing via QSPI controller located at 0x1f06000 on BAR0
INFO: ***Found 794 ELA Records
Enabled bitstream guard. Bitstream will not be loaded until flashing is finished.
Preparing flash chip 0
Erasing flash.......................................
Programming flash.......................................
Cleared bitstream guard. Bitstream now active.
fmadio@fmadio200v4-636:/opt/fmadio/firmware$This programs a backup FMADIO Firmware onto the FPGA.
Step 4) Power cycle
Once the FPGA has been programmed, a full power cycle is required for the image to be loaded. This is achevied using the following command
sudo ipmitool power cycleExample output is shown below
fmadio@fmadio200v4-636:/opt/fmadio/firmware$ sudo ipmitool power cycle
IANA PEN registry open failed: No such file or directory
Chassis Power Control: Cycle
fmadio@fmadio200v4-636:/opt/fmadio/firmware$After a about 3 minutes log back into the system
Step 5) Confirm PCI device
Confirm the PCI device is correctly an FMADIO device. Run the following command
lspci -s e1:00The output looks like below.
fmadio@fmadio200v4-636:~$ lspci -s e1:00
e1:00.0 Ethernet controller: FMAD Engineering FNIC200 (rev c0)
fmadio@fmadio200v4-636:~$This shows the FMAIO fpga image is now loaded
Step 5) Flash the current firmware FPGA image
The above Step 3) writes a backup FMADIO image onto the FPGA which is always different from the FPGA image the firmware is expecting. To flash the correct FPGA image run the following command
sudo fnic_flash --write /mnt/system/boot/bitstream.romExample output shown below. The process will take about 3 minutes to complete
fmadio@fmadio200v4-636:~$ sudo fnic_flash --write /mnt/system/boot/bitstream.rom
fNIC Flash: Jul 20 2025 17:30:40
SysConfig: 00000081
U55C 2x100G
fNIC Flash SPI: Jul 20 2025 17:30:43
50434930 50434930
Version: 42303054
EchoData: 0000000012345678
FlashSize: 64MB
BaseAddr : 01002000
Loading [/mnt/system/boot/bitstream.rom]
Flash Write Length: 52873320
resetting....
JDECID [19] 20 bb 21 10 44 00 10 87 0f 00 12 09 00 08 00 c1 50 09 00
Nonvol Config Read : 3 ff7f
cycles : 15
xip : 7
output : 5
ddr_n : 1
hold : 1
quad_n : 1
dual_n : 1
seg_hi : 1
addr3 : 1
Vol Config Read : 3 00fb
cycles : 15
xip_n : 1
wrap : 3
Flag Status: 2 0081
prog_erase_busy_n: 1
erase_suspect : 0
erase_err : 0
program_err : 0
prog_suspend : 0
protection_err : 0
addr_4B : 1
issue erase
Erase 0 [01002000] 0.00 %
Erase 0 [01192000] 3.10 %
.
.
Erase 0 [04072000] 96.06 %
Erase 0 [04202000] 99.16 %
Program 0 [01040000] (1.00 sec) 0.48 % 2.022 Mbps
Program 0 [01080000] (1.06 sec) 0.98 % 2.002 Mbps
Program 0 [010c0000] (1.06 sec) 1.47 % 1.992 Mbps
Program 0 [01100000] (1.06 sec) 1.97 % 1.986 Mbps
Program 0 [01140000] (1.06 sec) 2.46 % 1.984 Mbps
Program 0 [01180000] (1.06 sec) 2.96 % 1.982 Mbps
Program 0 [011c0000] (1.06 sec) 3.46 % 1.982 Mbps
Program 0 [01200000] (1.06 sec) 3.95 % 1.981 Mbps
Program 0 [01240000] (1.06 sec) 4.45 % 1.981 Mbps
Program 0 [01280000] (1.06 sec) 4.94 % 1.980 Mbps
Program 0 [012c0000] (1.07 sec) 5.44 % 1.979 Mbps
.
.
Program 0 [04100000] (1.06 sec) 97.16 % 1.992 Mbps
Program 0 [04140000] (1.06 sec) 97.66 % 1.992 Mbps
Program 0 [04180000] (1.06 sec) 98.15 % 1.992 Mbps
Program 0 [041c0000] (1.05 sec) 98.65 % 1.992 Mbps
Program 0 [04200000] (1.05 sec) 99.14 % 1.992 Mbps
Program 0 [04240000] (1.05 sec) 99.64 % 1.992 Mbps
317.680 Sec
fmadio@fmadio200v4-636:~$
Step 6) Power cycle
After the FMADIO systems expected FPGA firmware has been flashed, power the cycle system one last time
sudo ipmitool power cycleExample output shown below
fmadio@fmadio200v4-636:~$ sudo ipmitool power cycle
IANA PEN registry open failed: No such file or directory
Chassis Power Control: Cycle
fmadio@fmadio200v4-636:~$Wait for the system to return
Step 7) Update BIOS Setting
At this point the update has been completed. Check the system is running correctly.
If the BIOS settings were changed in Step 2). Please update per below. Otherwise jump to Step 8)
Change U2_P1_P0 Lanes to x8×8
As shown below.
.png?sv=2022-11-02&spr=https&st=2025-07-25T03%3A27%3A55Z&se=2025-07-25T03%3A42%3A55Z&sr=c&sp=r&sig=jrAPgKXk6E187PscjLQUoOVYiX2Sp9Yz9QsEHVFwJUQ%3D)
Step 8) Finished
System is completed updating and read for use
Any additional questions contact support@fmad.io for assistance