Memory Test

For server hardware particularly as hardware ages is DDR memory failures over time. These can show up as un-explainable problems and unusual behaviour, sometimes its very predictiable other times it seldom occurs.

The procedure below describes how to run a memory test on the system to confirm no issues.

Step 1) Download Memtest86 Free

Memtest86 Free version is free to download, the current version is v11.7

Link to the download page is shown below

https://www.memtest86.com/downloads/memtest86-usb.zip

Copy this file onto the FMADIO system in the directory

/mnt/store0/tmp2/

Step 2) Extract zip

Unzip the tarball as follows

 unzip memtest86-usb.zip

Example output

fmadio@fmadio200v4-636:/mnt/store0/tmp2$ unzip memtest86-usb.zip
Archive:  memtest86-usb.zip
  inflating: memtest86-usb.img
  inflating: readme.txt
  inflating: imageUSB.exe
  inflating: ReadMe_imageUSB.txt
  inflating: MemTest86_User_Guide_UEFI.pdf
   creating: Help/
   creating: Help/HTML/
  inflating: Help/HTML/zoom_pageinfo.js
  inflating: Help/HTML/hmkwindex.htm
  inflating: Help/HTML/cicon_loadindex_ani.gif
  inflating: Help/HTML/jquery.js
  inflating: Help/HTML/usage.htm
  inflating: Help/HTML/imageusb-banner.jpg
  inflating: Help/HTML/search.php
  inflating: Help/HTML/zoom_search.js
  inflating: Help/HTML/hmcontent.htm
  inflating: Help/HTML/settings.js
  inflating: Help/HTML/introduction_and_overview.htm
  inflating: Help/HTML/hmcontextids.js
  inflating: Help/HTML/contacting_passmark_software.htm
  inflating: Help/HTML/purchasing_information.htm
  inflating: Help/HTML/helpman_navigation.js
  inflating: Help/HTML/search_template.html
  inflating: Help/HTML/system_requirements.htm
  inflating: Help/HTML/gui.jpg
  inflating: Help/HTML/helpman_settings.js
  inflating: Help/HTML/default.css
  inflating: Help/HTML/zoom_index.js
  inflating: Help/HTML/hmftsearch.htm
  inflating: Help/HTML/highlight.js
  inflating: Help/HTML/cicon9.png
  inflating: Help/HTML/helpman_topicinit.js
  inflating: Help/HTML/index.html
fmadio@fmadio200v4-636:/mnt/store0/tmp2$

Step 3) Mount the USB Image

Mount the memtes86-usb.img USB image using the following command

sudo losetup -fP memtest86-usb.img

Then find the loopback device.

NOTE: need to use the absolute path as FMADIO shell by default ignores all loopback block devices

/usr/local/bin/lsblk | grep loop  | tail

Example output

fmadio@fmadio200v4-636:/mnt/store0/tmp2$ /usr/local/bin/lsblk | grep loop  | tail
loop177       7:177  0   308K  1 loop  /tmp/tcloop/libevent
loop178       7:178  0   6.5M  1 loop  /tmp/tcloop/glibc_i18n_locale
loop179       7:179  0   248K  1 loop  /tmp/tcloop/tmux
loop180       7:180  0   2.1M  1 loop  /tmp/tcloop/net-snmp
loop181       7:181  0     8K  1 loop  /tmp/tcloop/libestr
loop182       7:182  0  69.1M  1 loop  /tmp/tcloop/fmadio200v4_current
loop183       7:183  0     1G  0 loop
|-loop183p1 259:16   0   256M  0 part
|-loop183p2 259:17   0   255M  0 part
`-loop183p3 259:18   0   512M  0 part
fmadio@fmadio200v4-636:/mnt/store0/tmp2$

The mount point is the first loopback partition mount, in this case its

NOTE: this will vary depending on the system and firmware version

/dev/loop183p1

Mount this loopback device

sudo mkdir memtest

Then

sudo mount /dev/loop183p1 /mnt/store0/tmp2/memtest

The expected output of the directory is shown below

fmadio@fmadio200v4-636:/mnt/store0/tmp2$ ls -al memtest
total 64
drwxr-xr-x    4 root     root         16384 Jan  1  1970 ./
drwxrwxrwx   13 root     root          4096 Jun 25 00:39 ../
drwxr-xr-x    3 root     root          4096 May  4 01:50 EFI/
drwxr-xr-x    2 root     root          4096 May  4 01:50 help/
-rwxr-xr-x    1 root     root         35815 May  4 01:50 license.rtf
fmadio@fmadio200v4-636:/mnt/store0/tmp2$

Step 4) Copy the EFI image to System

Ensure an EFI directory is present on the boot system partition. On older systems the directory is not there

sudo mkdir /mnt/system/EFI/

Copy the EFI image to the FMADIO system partition

sudo cp -R /mnt/store0/tmp2/memtest/EFI /mnt/system/EFI/memtest86

Ensure it copied correctly

ls -al /mnt/system/EFI/memtest86/BOOT

Expected output

fmadio@fmadio200v4-636:/mnt/store0/tmp2$ ls -al /mnt/system/EFI/memtest86/BOOT
total 6808
drwxr-xr-x    3 root     root          8192 Jun 25 00:42 ./
drwxr-xr-x    3 root     root          8192 Jun 25 00:42 ../
-rwxr-xr-x    1 root     root       1369640 Jun 25 00:42 BOOTAA64.efi
-rwxr-xr-x    1 root     root       1553960 Jun 25 00:42 BOOTIA32.efi
-rwxr-xr-x    1 root     root       1631784 Jun 25 00:42 BOOTX64.efi
drwxr-xr-x    2 root     root          8192 Jun 25 00:42 Benchmark/
-rwxr-xr-x    1 root     root          5949 Jun 25 00:42 blacklist.cfg
-rwxr-xr-x    1 root     root          6823 Jun 25 00:42 mt86.png
-rwxr-xr-x    1 root     root       2355408 Jun 25 00:42 unifont.bin
fmadio@fmadio200v4-636:/mnt/store0/tmp2$

Step 4a) Copy EFI Shell

For Gen2 and Gen3 FMADIO system by default EFI boot is not included.

Download the following file

https://firmware.fmad.io/download/tmp/shellx64.efi

Confirm MD5sum

md5sum shellx64.efi

Expected result

37586bf70c618caa7c30c66a73dcd840  shellx64.efi

Copy the file to

sudo cp shellx64.efi /mnt/system/

Step 5) Force BIOS boot

Force the system to boot into the system BIOS, using the below IPMITool command

sudo ipmitool chassis bootdev bios

Expected output

fmadio@fmadio200v4-636:/mnt/store0/tmp2$ sudo ipmitool chassis bootdev bios
Set Boot Device to bios
fmadio@fmadio200v4-636:/mnt/store0/tmp2$

Step 6) Reboot system

Start the BMC KVM to monitor the boot process

Reboot the system, ensure it loads the BIOS page when completed

sudo reboot

When it completes BIOS screen looks per below

Step 7) Boot into EFI Shell

Navigate to the Save & Exit page

FMADIO Gen4 Systems

boot UEFI: Built in EFI Shell

FMADIO Gen2 and Gen3 Systems

Use the Launch EFI Shell from file system device per below

This will drop a prompt similar to below.

NOTE: on Gen4 systems press ESC to cancel the usual boot process

Step 8) Navigate to memtest86 dir

Continue to the copied device

fs0:

cd EFI/memtest86/BOOT

Step 9) Launch memtest

Run the BOOTX86.EFI image per

BOOTX64.EFI

It will start similar to below

Then the full memtest will run as shown below

Progress of the test is shown in the top Green PASS %

Do note it will take 10H or longer for the full test to complete.

Step 10) Wait for Result

It will take ~10 hours for the complete test to run. It van be left to run overnight

The final output result is shown below.