TITLE: HPE Superdome Flex Server Firmware Bundle (for installation from RMC)

VERSION:
Bundle Version: 3.25.46

 

VERSION 3.25.46 contains:  

   COMPLEX_METADATA: 3.25.46
    FWU: 2.20.3-20190829-142308

    NPAR_METADATA: 3.25.46

    RMC: 2.57.105

    RMC_EMMC: 2.57.105

    UV400_BIOS: 7.2.132.20200423_212505

    UV400_BMC: 2.57.105                      

    UV400_BMC_BASEIO_P0003171_002_PLD: 12.2.32

    UV400_BMC_BASEIO_P0003171_003_PLD: 12.3.35

    UV400_BMC_BMC_P0004912_001_PLD: 10.1.8

    UV400_BMC_EMMC: 2.57.105

    UV400_BMC_FWU_TOOLS: 2.57.105

    UV400_BMC_HARP_P0003240_001_PLD: 11.1.19

    UV400_BMC_NODE_P0001924_003_PLD: 13.3.57                                

    UV400_BMC_PWR_BRD_1_ENV_PSOC: 36.1.1

    UV400_BMC_PWR_BRD_1_PS_PSOC: 35.1.16

    UV400_HARP_FPGA_1590_B100: 1590_b100-0a

    UV400_HARP_FPGA_1590_B101: 1590_b101-0a

    UV400_HARP_FPGA_1590_B102: 1590_b102-0a

 

DESCRIPTION:
This bundle contains the firmware file for updating the HPE Superdome Flex server firmware from the RMC. This file updates the server BIOS firmware as well as firmware on the RMC (Rack Management Controller) and on the BMCs (Board Management Controller). 

Note: To comply with Open Source requirements, the Open Source used in Superdome Flex RMC/BMC firmware is provided in the tar file foss_SDFlex_2.3.tar.gz (only included for Open Source documentation purpose).

 

UPDATE RECOMMENDATION: Critical

 [X ] Critical                                              

    [X ] Panic, [ ] Hang, [ ] Abort, [ ] Corruption, [ ] Memory Leak, [ ] Performance

    [X  ] Security  [ ] Other

    [ ] Hardware Enablement, [ ] Software Enablement

[ ] Required

[ ] Recommended

[ ] Optional

    [ ] Hardware Enablement, [ ] Software Enablement, [ ] non-critical

[ ] Initial Customer Release

 

SUPERSEDES:                                
Version: 3.20.206  

 

PRODUCT MODEL(S):     
HPE Superdome Flex Server

 

OPERATING SYSTEMS: 
Supported on Superdome Flex systems with Intel Xeon® Scalable processors 62XX/82XX (Cascade Lake):
      Supported with DDR4:

• SUSE Linux Enterprise Server 12 SP4, 12 SP5, 15, 15 SP1, 15 SP2 – up to 32 socket/48 TB
• Red Hat Enterprise Linux 7.6, 7.7, 7.8, 7.9, 8.0, 8.1, 8.2, 8.3 - up to 32 socket/48TB
• Oracle Linux 7, Oracle Linux 8
• Oracle Linux 7 with UEK Release 5 Update 2 – up to 32 socket/48TB
• Oracle Linux 7.8, 8.1 with UEK6 (*)
• Oracle VM 3.4.4,  3.4.5, 3.4.6  - up to 8 sockets
• VMware 6.7 U3
• Windows Server 2016 & 2019 –  up to 16 socket

Supported with PMM (HPE Persistent Memory): 

• SUSE Linux Enterprise Server 12 SP4, 12 SP5, 15, 15 SP1, 15 SP2
• Red Hat Enterprise Linux 7.6, 7.7, 7.8, 7.9, 8.0, 8.1, 8.2, 8.3
• Oracle Linux 7, Oracle Linux 8
• Oracle Linux 7.7, 7.8 with UEK Release 5 update 2 (up to 8 socket, storage mode only)
• Oracle Linux 7.8, 8.1 with UEK6 (storage mode only) (*)
• VMware 6.7 U3
• Windows Server 2019 - up to 16 socket/24TB HPE Persistent Memory

 

 (*) Note:

 - UEK6 will be included in the ISO only starting with RHEL version 8.2; for prior versions (e.g. 7.8, 8.1), you need to install the Red Hat Compatible Kernel for that version, then use YUM   to install UEK6, then select UEK6 to load at boot.

- You can directly update from OL 7.7 to OL 7.8 while remaining on UEK5 because UEK5 is included in RHEL 7.8 ISO, like on prior RHEL versions.

 

Note: For latest information on configurations supported with PMM (HPE Persistent Memory), see the HPE Persistent Memory Guide for HPE Superdome Flex as well as the white papers “Installing and Running Microsoft Windows Server 2019 on HPE Superdome Flex Server” and “Running Linux on HPE Superdome Flex Server”.

Supported on Superdome Flex systems with Intel Xeon® Scalable processors 61XX/81XX (Skylake):
      Supported with DDR4 DIMMs only (no support of PMM):

• SUSE Linux Enterprise Server 12 SP2, 12 SP3, 12 SP4, 12 SP5, 15, 15 SP1, 15 SP2 (up to 32 socket)
• Red Hat Enterprise Linux 7.3z, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 8.0, 8.1, 8.2 (up to 32 socket)
• Oracle Linux 7.5, 7.6, 7.7
• Oracle Linux 8.0,8.1
• Oracle Linux 7 with UEK Release 5  update 2
• Oracle Linux 7 with UEK 4 & UEK5 (Unbreakable Enterprise Kernel) (up to 32 socket)
• Oracle VM 3.4.4,  3.4.5, 3.4.6 on systems with up to 8 sockets
• VMWare 6.5 U1/U2, 6.7, 6.7 U1/U3
• Windows Server 2016 on systems up to 16 sockets

 

     Note: For more details on VMware support and certifications, check the “Running VMware vSphere on HPE Superdome Flex Server” white paper.

LANGUAGES:
International English

ENHANCEMENTS:    

·       Added support for RHEL 7.8

·       Added iSCSI software initiator based boot support with HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter (872726-B21) on SLES 12 SP3, SP4 , SLES 15 SP1 and supported RHEL releases

·       Support Openstack HTTP boot and Openstack T release

·       Support sector mode for HPE Persistent Memory

·       Supports Windows Server 2019 with 16 socket/24 TB of HPE Persistent Memory 

·       Added IPMI watchdog functionality; the CLI command 'set ipmi_watchdog os_managed' command may be used to enable it

·       Support in-band IPMI power operations such as power cycle, power off, power soft

·       Includes some uvdmp improvements

·       June 2020 update:

o   Added support for RHEL 8.2

o   Added support for OL 7.8 and OL 8.1 with UEK6

o   Removed Windows 2019 support on 61XX/81XX (unsupported)

·       September 2020 update:

o   Added support for SLES 15 SP2

·       October 2020 update:

o   Added support for RHEL 7.9

o   Added support for RHEL 8.2 and OL 8.1 on systems with Intel Xeon® Scalable processors 81XX/61XX

·       December 2020 update:

o   Added support for RHEL 8.3  on systems with Intel Xeon® Scalable processors 82XX/62XX

o   Added support for VMware 6.7 U3 on systems with Intel Xeon® Scalable processors 82XX/62XX and Persistent Memory

o   Added Intel microcode issue under "Known Issues"

 

FIXES:  

·       The BIOS in firmware version 3.25.46 includes updated Intel microcode that addresses some crashes caused by DDR4 memory errors. In particular, this updated microcode addresses a Machine Check Exception timeout failure when Fast Fault Tolerant Memory Mode (ADDDC) is enabled. This issue is not unique to HPE servers.

·       The updated microcode (0x2f00 on systems with 62xx/82xx processors and 0x6901 for systems with 61xx/81xx processors) also addresses security vulnerability CVE-2020_0549

·       System now reports event 1210 in case of MCA due to some uncorrected memory errors

·       The grub2 boot loader menu now successfully launches when booting grub2 boot loader via UEFI HTTP boot and when booted via either fully qualified domain name URL or via IP address URL

·       Addressed boot issues. Monarch chassis would fail to boot with erroneous "PLD or PSOC not detected" message. Other chassis would boot but nPar would be missing the chassis resources

·       Users can now successfully mount ISO images in Jviewer when the RMC/eRMC password contains one or more of %, #, or & characters

·       Dot ​(.) ​character is ​now ​accepted ​as ​part ​of ​LDAP ​User/Group ​names ​in ​Superdome ​Flex ​Server ​RMC/eRMC ​CLI

·       FRU data for HPE PCIe cards installed in slot 16 are now properly read and "SHOW CHASSIS INFO” no longer returns fields as “Unknown”

• Addressed additional issues where deconfigured PMMs in any chassis would prevent Windows Server 2019 from accessing any PMM devices (physical or logical), resulting in users being unable to see any physical or logical PMM devices in device manager, or in the output of Windows PowerShell cmdlets Get-PmemPhysicalDevice or Get-PmemDisk, and PMM disk data being inaccessible
• Some BMC logs are no longer corrupted or missing data right after AC loss
• Addressed an issue where show logs audit would fail if the file contained non ASCII characters

 

COMPATIBILITY:

·       Monitoring and management of HPE Superdome Flex servers with firmware version 3.20.186 or later is supported only starting from OneView version 5.00.

·       It is recommended to use this firmware along with HPE Superdome Flex I/O Service Pack version 2020.05 and HFS (HPE Foundation Software) version 2.3.2 (Linux only) as well as the latest DCD version:

o   See the Superdome Flex support matrix: HPE Superdome Flex Release Sets

·       For additional OS specific information, please see:

o   For VMware, the “Running VMware vSphere on HPE Superdome Flex Server” white paper.

o   For Windows, the “Running Microsoft Windows Server on HPE Superdome Flex Server” white paper (for Windows 2016) and “Installing and Running Microsoft Windows Server 2019 on HPE Superdome Flex Server” (for Windows 2019).

o   For Linux, the “Running Linux on HPE Superdome Flex Server” white paper as well as the “HPE Superdome Flex Server OS Installation Guide” at https://www.hpe.com/support/superdome-flex-os

PREREQUISITES:  

• HPE recommends using the kernel versions that prevent the Transactional Asynchronous Abort (TAA - CVE-2019-11135). Links to updated packages are available in the SUSE CVE-2019-11135 page.  Please see Intel Security advisory Intel-SA-00270  for additional information.
• Prior to updating the firmware, make sure that you know the RMC administrator password
• HPE requires the following security best practices for correct operation and to help reduce both known and future security vulnerability risks:

1.       Isolate the management network from the normal corporate LAN. This management network should limit and restrict access to your RMC management interfaces using firewall, Accesses control lists (ACLs), or VPN.  This will greatly reduce a large group of security risks, (for example Denial of Service attacks).

2.       Patch and maintain web servers.

3.       Run the up-to-date virus and malware scanners in your network environment

4.       Apply HPE firmware updates as recommended.

 

INSTALLATION INSTRUCTIONS:

Please review all instructions and the "Hewlett Packard Enterprise Support Tool License Terms" or your Hewlett Packard Enterprise support terms and conditions for precautions, scope of license, restrictions, and limitation of liability and warranties, before installing this package. It is important that you read and understand these instructions completely before you begin. This can determine your success in completing the firmware update.

Note: It is highly recommended that firmware updates be executed by Hewlett Packard Enterprise support personnel.

 

Note: HPE provides three methods for updating the server firmware, from the RMC CLI, using SUM or using Oneview. If you need help selecting a method, please see the “Firmware Update” section in the Superdome Flex Server Manageability white paper.

 

Online Firmware Update is available when updating from FW 3.0.512 or later:

·       Server management capabilities will become unavailable until the update completes.

·       The new Partition firmware only gets programmed during the update, and a partition reboot is required to activate it. Therefore it may get activated on one partition at a time.

·       Operation with mixed complex/partition firmware is supported, providing that the complex firmware version is always greater than or equal to the partition firmware.

·       Online firmware downgrade is supported when downgrading to firmware version 3.0.512 or later.

 

Important:

·       DO NOT abort the firmware update once started as this may cause the system to get in an un-usable state. In particular, DO NOT turn off system AC power during a Firmware update.

·       In case a firmware mismatch is displayed after the update, retry the update. If you continue to see failures, please contact HPE support

 

INSTALLATION

To update server firmware from the RMC CLI:
1. Copy the firmware file sd-flex-3.25.46-fw.tars to your local computer.
 

2. Follow the instructions below to update the firmware to version 3.25.46 on your system.

 

a.       Please verify that the system date is set. If not, set it and check if you have a NTP server up and running as it is used to set the date.

 

b.      For offline FW update only (Skip step b. for online firmware update):
Log into the HPE Superdome Flex Server operating system as the root user, and enter the following command to stop the operating system:
          # shutdown

c.       Login to the RMC as administrator user, provide the password when prompted.

d.      Use of DNS is recommended:
- If using DNS, verify that the RMC is configured to use DNS access by running:
          RMC cli> show dns
   If not, you may use the command “add dns” to configure DNS access (or you can’t use DNS).

- If not using DNS, you will need to specify IP address in the <path_to_firmware>

e.       For offline update only! (Skip step e. for online FW update):
Enter the following command to power off the system
        - If there is only 1 partition, partition 0 is the default:
                 RMC cli> power off npar pnum=0  
         - In case of multiple partitions, enter show npar to find the partition number, then enter:
                RMC cli> power off npar pnum=x, where x is the partition number

f.        Update the firmware by running the command: 
          RMC cli> update firmware url=<path_to_firmware> [exclude_npar_fw]

                   Where <path_to_firmware>  specifies the location to the firmware file that you previously
                   downloaded. You can use https, sftp or scp with an optional port. For instance:
                         RMC cli> update firmware url=scp://username@myhost.com/sd-flex-<version>-fw.tars
                         RMC cli> update firmware url=sftp://username@myhost.com/sd-flex-<version>-fw.tars
                         RMC cli> update firmware url=https://myhost.com/sd-flex-<version>-fw.tars
                         RMC cli> update firmware url=https://myhost.com:123/sd-flex-<version>-fw.tars

                    And where exclude_npar_fw is used to not update the BIOS firmware running on an nPar.

                    Note: The CLI does not accept clear text password, the password has to be manually typed in.
                    Note: To use a hostname like ‘myhost.com’, RMC must be configured for DNS for name
                             resolution, otherwise you need to specify the IP address of ‘myhost.com’ instead. See
                             the command ‘add dns’ for more information.

g.       Wait for RMC to reboot after a successful firmware update, then check the new firmware version installed by running:
         RMC cli> show firmware verbose
 Note: The nPar firmware version will not be updated until the next nPar reboot. See output under “DETERMINING CURRENT VERSION” below.

h.      For online FW update, reboot the Partition when convenient to activate the new nPar firmware:
- To reboot a partition or multiple partitions, enter:
                RMC cli>  reboot npar pnum=x, where x is the partition number or 0 for a single
                                 partition chassis numbered 0

For offline FW update, Power on the system or partition:
 - To power on a system configured with all chassis in one large nPartition numbered 0, enter:
                    RMC cli>  power on npar pnum=0.
 - If you have multiple npars, each npar can be powered on separately using:
                   RMC cli>  power on npar pnum=x, where x is the partition number.

 

DETERMINING CURRENT VERSION:
To check or verify the current firmware levels on the system, from the CLI, enter the RMC command:  
         RMC cli> show firmware
                Configured complex bundle version: 3.25.46

                Configured npar bundle version: 3.25.46

 

         Firmware on all devices matches the configured version.

 

         nPars running a different version until their next reboot:

                    nPar 0: 3.20.206        (Note: One line is displayed for any nPar not rebooted yet)

 

   Note: If online FW update was used and some nPars have not been rebooted yet, the command
               lists the prior FW version still active on these nPar’s.

   Note: If you want to see all the components’ versions, you may use “show firmware verbose”.

 

Downgrading firmware:

Note: Downgrading firmware is not recommended as it may cause a loss of functionality and expose the system to vulnerabilities fixed in later versions. If you need to downgrade, here are some guidelines:

- After online downgrade completes, you will need to reboot the nPAR using the “reboot npar pnum=x” command to activate the newly installed BIOS.

- After offline downgrade completes, power on the system or nPAR using the “power on” command.

 

KNOWN ISSUES & WORKAROUNDS:  

·       Due to faulty Intel microcode (02006a08 for Skylake, 04003003 or 05003003 for Cascade Lake), Superdome Flex servers running with these microcode versions may stop running, fail to boot or experience random de-configuration of processors and/or processor cores. The following OS versions are affected:

o   SLES 15 SP2: ucode-intel-20201110-2.10.1.x86_64,  ucode-intel-20201118-2.13.1.x86_64

o   SLES 12 SP5: ucode-intel-20201110-3.23.1.x86_64,  ucode-intel-20201118-2.13.1.x86_64

o   RHEL : microcode_ctl-2.1-73.2.el7_9.x86_64.rpm,  microcode_ctl-20200609-2.el8.x86_64.rpm

o   Windows Server 2016: KB4589210

o   Windows Server 2019: KB4589208 

For details and workaround, see Customer Advisory a00108899

·       On systems running with SLES 15 SP1 or 15 SP2, an IO card may rarely go offline with the console log showing a message “Device recovery failed”. An OS reboot is required to recover.

·       Installation of Windows Server 2019 on a 16 socket partition with Hyper-Threading enabled may take hours on servers with hundreds of logical processors. Workaround: Either disable Hyper-Threading prior to installing the OS (and re-enable it after OS is installed), or modify to an 8 (or less) socket partition, install the OS, and change back to a 16 socket partition after the OS is installed.

·       On systems (8 or 16 socket) configured with PMM (HPE Persistent Memory) and running with Windows 2019, a BSOD (Blue Screen of Death) may be intermittently encountered while creating or deleting namespaces using new-pmemdisk or remove-pmemdisk powershell cmdlets. When this occurs, the following bug check happens: STOP 0x00000101 CLOCK_WATCHDOG_TIMEOUT. Workaround: Retry the operation after the system comes back up.

·       From the Device Manager’s iSCSI configuration menu, “Add an attempt” is used to create iSCSI boot attempt entries. The firmware currently supports a maximum of 8 entries. Adding a 9^th entry results in a SFW assertion.

·       A BIOS ASSERT may be encountered when performing PXE boot from Microsoft Windows Deployment Services, if the user attempts to select ESC in the menu displayed by Windows Boot Manager. Workaround: Do not exit the menu using the ESC key.

·       After a RMC reboot, Insight Remote Support (IRS) may not collect all the inventory data. Workaround: Rediscover the missing device in IRS (in IRS console, under Devices, check all the devices from the Device Summary TAB and select Discover); for information on Insight Remote Support, see “Manage Devices” in the IRS Help.

·       A Superdome Flex server may rarely crash during boot due to a timeout occurring in the BIOS. When this happens, the system will automatically reboot and recover.

·       Firmware update with certificate checking enabled is not supported with OneView or SUM (SUM does not support providing a certificate for client verification). Workaround: disable certificate checking prior to updating firmware with SUM or OneView.

·       Superdome Flex server registration with IRS fails if the system is configured with certificate checking enabled. Workaround: Install IRS patch 7.10 to fix the issue.

·       Deconfigured PMMs in the expansion chassis may at times result in multiple persistent memory regions being inaccessible to the OS. User should call HPE Support to check on the reason for the deconfigured PMM and get it addressed.  

·       After a PMM becomes deconfigured, Windows Server 2019 may not be able to access PMM logical devices if the PMM region is setup in the default AppDirect (interleaved) mode, causing the PMM disk data to be inaccessible. Workaround: Do not use the interleaved mode. Also call HPE Support to check on the reason for the deconfigured PMM and get it addressed. 

·       Whenever an nPAR is created, removed, or modified that results in a change in monarch chassis, all existing Redfish sessions will immediately expire. Redfish client will need to re-authenticate with the Superdome Flex Redfish to establish a new Redfish session.

·       The UEFI shell accepts invalid directory names such as ‘..’ appended to a valid directory name. Workaround: Retry using a proper directory name.

·       The UEFI shell command ‘dh’ fails to display EFI_DEVICE_PATH_PROTOCOL information for some device handles. It also displays “IPv6 (Not Available)” for devices including IPv6 device path node information.

·       If the BMC stops responding, a watchdog timer resets it to recover automatically with an event BMC_RESET_BY_WATCHDOG. This event reflects that normal BMC operation has resumed and it may be ignored. 

·       The BMC may occasionally reboot to recover from a software error condition. If a BMC_KERNEL_PANIC is logged, you may ignore it as it does not affect system operation.

·       Virtual Media connection may cause slow-boot. Workaround: Disconnect Virtual Media to resolve the slow boot. In general, it is recommended to disable Virtual Media when not needed.

·       Virtual Media does not support UEFI reconnect -r command.
Workaround: After attaching Virtual Media to the partition, use POWER RESET to reset the partition and activate Virtual Media.

·       By default, the Virtual Media instance setting for CD/DVD and hard disk are set to 0. To use virtual media, CD/DVD must be set to at least 1. For more details, refer to the Superdome Flex OS installation guide (http://www.hpe.com/support/superdome-flex-os)

·       The JViewer application VMedia hard disk size must be between 4 MB and 512 MB. If any larger size is needed, create an ISO and mount it via the CD/DVD tab.

·       Mac OS browsers (Safari, etc) are not supported on KVM and JViewer. Use Windows Internet Explorer, Firefox and Chrome instead.

·       Right after nPar creation, the partition status may show as Unknown due to a delay in status update. Workaround: Run SHOW NPAR command to check the nPar status.

·       Failure due to timeout during directed pxe boot when directed PXE boot is requested and the PXE server is not enabled to respond to the DHCPINFORM message request. You must use a PXE server that supports DHCPINFORM message requests.

·       When upgrading firmware from version 2.4.98, a unique certificate per RMC/eRMC is re-generated. For systems using OneView, the RMC’s older certificate residing in OneView’s trust store will become stale and communication with the RMC will not succeed.
Workaround: To restore OneView to RMC communication after updating from version 2.4.98, follow the steps below (to address it before update, see pre-requisites):

In OneView instance,

i) go to Settings -> Security

ii) Click Manage certificates button.

iii) Delete the RMC certificate from the list.

iv) Initiate rack manager refresh

·       The CLI provides a convenient ‘ipmi’ wrapper script. However, serial over lan (SOL) is not supported by this convenient ‘ipmi’ command. Attempting to activate partition console via ‘ipmi command=”sol activate”’ will fail with the message: “Error: This command is only available over the lanplus interface”. User should use CLI ‘connect npar’ or ‘uvcon’ to connect to partition console.

 

For more details on accessing and managing the system, see the HPE Superdome Flex user documentation located at this link.

 

DISCLAIMER:
The information in this document is subject to change without notice.
Hewlett Packard Enterprise makes no warranty of any kind with regard to this material, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett Packard Enterprise shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this material.

This document contains proprietary information that is protected by copyright. All rights are reserved. No part of this document may be reproduced, photocopied, or translated to another language without the prior written consent of Hewlett Packard Enterprise.

(C) Copyright 2017-2020 Hewlett Packard Enterprise Development L.P.

 

SUPERSEDES HISTORY:
Version 3.20.186/3.20.206:
ENHANCEMENTS:  

February 2020 update:

·       Server firmware 3.20.186 and later supports also Oracle VM 3.4.6

December 17, 2019 update:

·       Firmware version 3.20.186 and later includes mitigation for CVE-2019-14607 in the Intel Microcode.

In firmware version 3.20.206:

·       POWER_SUPPLY_INPUT_LOST (event 1106) is now generated in case of loss of AC input to the chassis power supply.

·       When inserting a power supply, POWER_SUPPLY_TYPE_PROBE_FAILED (event 1111) is now generated in case the power supply is detected but FRU data cannot be read, likely because the power supply needs reseating.

 

In Firmware version 3.20.186 (not available as web release):

·       Supports 128/256/512 GB Intel PMM (HPE Persistent Memory Module):

o   For details on supported PMM configurations, see the HPE Persistent Memory Guide for HPE Superdome Flex.

§  Supported with Intel Xeon processors 82xx/62xx only

§  Support of App direct mode on 4/8/16 socket systems  

§  For OS support, see the Operating Systems section above

o   Supports management and configuration of PMM’s

o   Supports PMM related events and logging as well as health monitoring and reporting

o   Supports creation of XFS root file system for Linux on PMM

·       Supports additional CPUs: 6226, 8253

·       Supports additional OS’es: RHEL 8.1, Oracle UEK 7.7, SLES 12 SP5 

·       Supports new I/O cards:

o   RTx 8000 GPU

o   HPE Infiniband HDR/Ethernet 200GB 1-port  & HDR100/Ethernet 100GB 1-port/2-port 940QSFP56

·       Improved manageability using Redfish and Openstack Ironic release (see HPE Superdome Flex Manageability white paper)

·       Improved security: supports certificate checking on web console and Redfish connections

o   Note: Certificate checking is not supported with SUM or OneView; you need to disable certificate checking prior to updating firmware

·       Improved BMC stability by significantly reducing occurrences of random BMC reboots

·       Improved diagnosing of damaged cables while preventing mis-diagnosis of other parts

·       Improved chassis power supply AC loss handling and reporting

·       Improved power supply redundancy reporting

·       Supports RMC power supply fault as well as thermal fault reporting

 

FIXES:
Fixes in 3.20.206:  

·       Addresses an issue where an error occurring on PMMs (HPE Persistent Memory) in the expansion chassis would rarely result in unexpected behavior, including an OS crash

·       Fixed an issue where deconfigured PMMs in any chassis would prevent Windows Server 2019 from accessing any PMM devices (physical or logical), resulting in users being unable to see any physical or logical PMM devices in device manager, or in the output of Windows PowerShell cmdlets Get-PmemPhysicalDevice or Get-PmemDisk, and PMM disk data being inaccessible.

Fixes in 3.20.186 (not available as web release):

·       Critical - Addressed an issue seen with 64GB DDR4 DIMMs in socket 0 and using 2 DIMMs per channel, where ADDDC (Adaptive Double Device Data Correction) bank sparing would result in a crash due to an uncorrected memory error.

·       The System ROM in firmware version 3.20.186 or later includes the latest revision of the Intel Reference Code that provides mitigations for security vulnerabilities. The following vulnerability has been addressed in this release: CVE-2019-0152. This issue is not unique to HPE servers.

·       The system ROM in firmware version 3.20.186 or later includes the latest revision of the Intel microcode which provides mitigation for CVE-2017-5715, CVE-2019-11135, CVE-2019-11139 and CVE-2019-14607, as well as mitigation for an Intel sighting where under complex micro-architectural conditions, executing X87 or AVX or integer divide instructions may result in unpredictable system behavior. These issues are not unique to HPE servers.

·       Addressed an issue where, after power cycling all the chassis in the complex, the system boot would fail with a BIOS ASSERT DETECTED due to an ASSERT EFI ERROR

·       Indicted or deconfigured FRUs are now acquitted upon AC power-cycling of the entire complex (all BMCs and RMC)

·       Firmware now sends service events or SNMP traps when a Rack Management Controller (RMC) hardware error occurs

·       Superdome Flex now supports IPMI over LAN via any IPv6 IP address listed within the subnet, not just the first one configured

·       Fixed an issue where uncorrected memory errors would rarely occur without deconfiguration or indictment

 

Version 3.10.164/3.10.174:

ENHANCEMENTS:

Note: System firmware 3.10.164 includes all of the enhancements defined for 3.10.174.

Enhancements in firmware 3.10.164 / 3.10.174:

• Changed the directory format default setting to “Extra-compressed” mode (1/64^th size of the main memory) for optimal performance; this value is configurable, see advisory a00082677 for details
• On Superdome Flex servers, OneView Managed Mode is supported starting with OneView version 5.0 and firmware version 3.0.512 (see OneView related fix in the Fixes section).
• Online firmware update is supported when updating from firmware version 3.0.512 (or later) to a later version
• Added support for Intel Xeon® Scalable processors 6240 and 6246 with firmware 3.10.164 and later
• Added support of IPv6 addresses in UEFI, BMC and RMC

FIXES:

Note: System firmware 3.10.164 includes all of the fixes defined for 3.10.174.

Fixes in version 3.10.164 / 3.10.174:

• Latest revision of the Intel microcode which, in combination with operating system and/or hypervisor updates, provides mitigation for a new group of side-channel vulnerabilities known as Micro architectural Data Sampling (MDS). This includes support for mitigating the following vulnerabilities:
• CVE-2018-12126 - Micro architectural Store Buffer Data Sampling
• CVE-2018-12130 - Micro architectural Fill Buffer Data Sampling
• CVE-2018-12127 - Micro architectural Load Port Data Sampling
• CVE-2019-11091 - Micro architectural Data Sampling Uncacheable Memory.
• Firmware manages setvar settings to further reduce the likelihood of PCIe Completion Timeout (CTO) issues that could lead to MCA’s
• Addressed an issue where high CPU utilization on the BMC would rarely cause the NL (Numa Link) initialization sequence to run slower than required and result in NL_PORT_LINK_FAIL and “grid initialization failure” events during the reboot of an nPars.
• Addresses a limitation (with firmware 3.0.512 or 3.0.542), when using BIOS settings in OneView 5.00 Server Profile where, if the “DHCPv4” property was enabled, either the system would boot from “IPv4 PXE Server Address” (if configured to a non-default value), or the system would erroneously ignore “Boot from URL 1” and “Boot from URL 2” and proceed to fall back to existing UEFI PXE boot options.
• Addressed error messages such as “Entry attributes invalid: RO and XP bits both cleared” that were observed at boot time due to some OS versions using tighter access controls
• IPMI serial over LAN (SOL) now works with IPv6 on the RMC
• The VGA preview pane on the Dashboard page is no longer blank when a FireFox or Chrome browser is used
• LDAP users can now login successfully into the RMC after a LDAP field or group is added using SET LDAP or ADD LDAP_GROUP command respectively
• SHOW NETWORK and BASEIOLIST commands now display the IPv6 addresses for the base IO management port
• The RMC and eRMC will now use the proper clock time even if they are configured to use DHCP and NTP is not explicitly configured
• SHOW NETWORK and BASEIOLIST commands now display both the IPv4 and IPv6 addresses for the base IO management port
• The UEFI shell command “mv” no longer fails to move a file to a destination that already exists
• When making changes using the UEFI menus, you no longer need to save the change before moving to another menu page in order for the change to be saved

·       Fixed an issue where web console login would fail if the RMC network was not properly configured or not connected to the site network prior to RMC boot

Version 3.0.542:

ENHANCEMENTS:  

June 2019: Release notes were updated to add some fixes and known issues and add a link to the Superdome Flex Support Matrix (Release Sets).

 

FIXES:

Version 3.0.542:

• Addressed an issue, on OS’es running with secure boot, where an OS update would fail when trying to update the secure boot keys.

 

Version 3.0.512:

ENHANCEMENTS:

·       Added support for Intel Xeon® Scalable processors 8280, 8276, 8270, 8268, 8260, 8256, 6254, 6252, 6248, 6244, 6242, 6240, 6230; requires firmware version 3.0.512 or later

·       Added support for Windows 2019 on systems with Intel Xeon® Scalable processors 62XX/82XX

·       Added support for IPv6 USGv6

·       On 4 socket systems, TEST FABRIC now displays a warning instead of an error message when the 3 Numa Link loop back cables are not installed.

·       Superdome Flex internal management network uses by default 172.16.0.0/16, 172.30.50.0/24 and 172.30.60.0/24 subnets. However, the RMC “set network internal” command now allows to change these subnets to any legal subnet.

·       APPWT limit has been increased to the 29 WT (Weighted Teraflops) threshold, effective since October 2018

 

Note:

·       Default NIC naming differs on systems with Intel Xeon® Scalable processors 62XX/82XX versus 61XX/81XX. On servers with 62XX/82XX processors, NIC naming is based on udev property “ID_NET_NAME_SLOT”. This was introduced with firmware 3.0.542. On systems with 61XX/81XX processors, NIC naming continues to be based on “ID_NET_NAME_PATH” and there is no impact after updating to 3.0.542. However, if customers wish to use the consistent device naming standard on systems with  61XX/81XX processors, then they can follow the steps in the DETAILS section of the customer notice https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00075568en_us

·       The RMC factory default password has been changed from a common password to a random password, unique on each system, provided on a label on the rear of the RMC or eRMC base chassis. For details, see advisory a00073630.

·       The web console, virtual media, and JViewer default login now use the RMC/eRMC administrator account and password. The user 'admin' can no longer login to the web console.

FIXES:

• Implemented CPU Hardware P-States (HWP) to improve performance and also to address errors such as,
• “Performance power management features on processor “xxz” in group “g” are disabled due to a firmware problem.”
• “Collaborative processor power controls on processor “xyz” in group “g” are disabled due to a firmware problem.”

·       Provided enhancements and multiple fixes in memory and IO error handling to help prevent MCAs

• Significantly improves BMC stability by reducing 'linked but not configured' occurrences
• Fixed an occurrence of MCA (Machine Check Abort) observed during the early boot phase on systems running with RHEL 7.5, with logs displaying an error such as "Halt on fatal HW error" or "BMC r001i01b: HWERR: BIOS HALT detected!."

·       Addressed occurrences of MCAs, BIOS HALTs due to HWERR, and ASSERTs at boot time

·       Addressed a number of false critical alerts that were seen at boot or after a power cycle

·       Addressed an eRMC hang after a MC warm reset is triggered from ipmitool

• USB error messages that used to slow down the shutdown of Linux partitions are no longer displayed and the shutdown time is back to normal
• Fixed an issue where “acquit all” did not clear all the de-configured IO cards
• Fixed a shadow group issue causing IRS credentials to be rejected on the eRMC
• CAE and IEL now report the correct location of a faulty physical drive connected to a MegaRAID 9361-4i Controller (requires DCD version 2.0)
• System no longer erroneously reports a FAN_FATAL event for both PSU fans if an AC power loss occurs while the system power is off, and the PSU containing the fans is no longer indicted
• Fixed an issue where a VOLTAGE_FATAL event was generated during boot to EFI
• Fixed an issue where incorrect PSU fan speed would be reported after a PSU removal/insertion
• Fixed an issue where the power on/off command would rarely fail to change the system power state and terminate without any indication that the state was not changed
• UPI_PORT_LINK_FAIL is no longer reported as a Critical event upon successful automatic UPI link recovery
• After changing the BMC admin password, if the chassis is AC cycled or the BMC is power reset/cycled using CLI, the BMC admin password no longer reverts to the previously set password
• Pasting more than 8 characters at a time into an Extensible Firmware Interface (EFI) partition console no longer causes one or more characters to be dropped
• You can now delete the final remaining Key Exchange Key (KEK) certificate via the "Delete KEK" boot manager menu (reached via the Secure Boot Configuration menu and selecting "Customize Secure Boot" as "Customized")
• RHEL installation using virtual media is now supported. It no longer fails with a message such as “Media check failure, SQUASHFS error on the RHEL install console, Input/Output error”.
• Virtual Media: OS installation could silently fail part way through the installation.  Failure messages are now provided and connection timeout issues have been addressed.
• KVM console may not accept keystrokes, until toggling the KVM’s “Mouse Mode” to a different mode.
• Web GUI: Some settings and buttons were not effective and have been removed.

 

SUM 8.4.0 fixes:

·       SUM version 8.4.0 fixed an issue where SUM would falsely report that update had completed less than 2 minutes after deployment had started.

 

Version 2.5.314:

ENHANCEMENTS:

·       None

FIXES:

·       CRITICAL: Fixed an issue where a system under heavy workload would sometimes MCA with error messages such as "Fatal Link Timeout to PCIe Device" and "LER_ENTERED".

·       CRITICAL: Optimized memory controller for RAS features to prevent MCA’s when a system is under a heavy workload and a DIMM sparing operation is required.

·       Addressed the privilege escalation vulnerability CVE-2018-12204

 

Version 2.5.300:

ENHANCEMENTS:

• Added support for Red Hat Enterprise Linux 7.6
• Added support for SUSE Linux Enterprise Server 12 SP4
• Added support for SUSE Linux Enterprise Server 15

FIXES:

·       Updated to latest Intel microcode

·       Firmware update no longer reports an error when updating chassis with rack number greater than 9

·       Extended range of rack numbers supported from 0 to 254

 

Version 2.5.90:

FIXES:

·       Firmware version 2.5.290 resolves certain potential unexpected system behavior when operating the Superdome Flex system or nPar(s) in HPC mode. For systems or partitions running in HPC mode, the frequency of system memory errors may increase after the update. These memory errors will be fully visible in memlog.  HPE strongly recommends that customers run in RAS mode, but if they elect to continue using HPC mode, they should update to this firmware version to eliminate the risk of unintended side effects from memory correction including possible system crashes. Refer to this advisory for more details.

 

Version 2.5.80:

ENHANCEMENTS:

• Added support for Nvidia P40 GPU
• System now recognizes the Nvidia V100-32GB GPU card so that it can detect its presence and adjust its GPU cooling calculation accordingly

FIXES:

The following issue has been addressed in firmware version 2.5.280:

·       Fixed an issue where a bugcheck or unexpected process termination would be seen after an uncorrectable memory error on systems running Microsoft Windows Server 2016.

 

The following issues have been addressed in firmware version 2.5.270 and later:

• Fixed an issue where RHEL 7.5 kernel would panic in presence of saved error records
• Fixed an issue where a system under significant load would crash when running certain commands
• Fixed an issue where, after detecting an uncorrected memory error, the faulty DIMM would get indicted but not de-configured, potentially causing the OS/partition to stop responding or the system to crash after reboot if the faulty DIMM had not been replaced.
• The CLI SHOW SENSOR TARGET=<chassis GEOID> command no longer erroneously displays the DIMM temperature reading as "N/A" when the chassis is powered on.
• Fixed an issue where the TEST FABRIC command would incorrectly display that all chassis have a Base IO Board
• Fixed an issue where an incorrect FAN_FATAL Integrated and Service Events would be sent upon removing the AC power cord
• When the Superdome Flex Rack Management Controller (RMC) reboots, the RMC command CLI SHOW CHASSIS INFO now correctly reports the number of installed power supplies in each chassis
• Fixed a stack trace issue observed when using the IDK Error Injection tool
• Fixed an error reporting problem where the register would report the Requester ID incorrectly
• PCIe Error injection using EINJ (Error Injection) now works for all PCIe segments
• System will no longer ASSERT when injecting PCIE error using EINJ (Error Injection) with invalid PCIE address

Version 2.5.256:

FIXES:

• Fixed an issue where the Superdome Flex BMC inadvertently ignored GPU thermal data for its cooling algorithm, which might cause the GPU to run at an elevated temperature and result in the GPU running much slower to avoid overheating. HPE strongly recommends that systems with GPU do not run with firmware version 2.5.246 but update to version 2.5.256 or later where this issue is fixed.
• Version 2.5.246 and later fixed a DIMM_LOAD_ORDER_VIOLATION issue caused by checking the DIMM loading order for disabled sockets.
• Version 2.5.246 and later addressed the following L1 Terminal Faults:

-       L1 Terminal Fault - OS, SMM (CVE-2018-3620). Please note this mitigation also requires operating system software updates.

-       L1 Terminal Fault - OS, VMM (CVE-2018-3646). Please note this mitigation also requires operating system software updates, and VMM software updates.

-       For more information, see the bulletin a00055017en  

Version 2.5.246:

ENHANCEMENTS:  

·       Added support for CPU models 8170M, 8170, 8168, 6140, 6140M, 6150, 6142M, 6142, 6138

·       Added support for 24 and 28 socket configurations (in addition to 4, 8, 12, 16, 20 and 32)

·       Added partitioning (nPAR) support with ability to convert non partitioned systems to partitionable

·       Added support for HPE Ethernet 10Gb 2-port 562T adapter, 32Gb Fibre Channel SN1600Q, SN1600E HBAs

·       Added support for Windows Server 2016 with up to 16 sockets

·       Added support for RHEL 7.5

·       Added support for VMware 6.5 U2

·       Added support for Oracle VM 3.4.4

·       Added new security and management features (secure boot, SSH upgrade, reduced port usage)

·       Added support for offline firmware update via SUM (Smart Update Manager)

·       Added support for OneView monitoring (requires OneView version 4.1 or later)

·       Added support for provisioning OS with redfish using the OpenStack Ironic  (Requires Openstack Ironic version ‘Pike’  or later)

·       Enhances security with SSH

FIXES:

·       New BIOS addresses the following known vulnerabilities, CVE-2018-3639 and CVE-2018-3640.

·       Fixed an issue where the firmware update would fail on rare occasions to update the BIOS image.

·       Some IO errors no longer cause an incorrect decoding to be logged in the Integrated Event Log with the string “[physloc_err=5]”.

·       Fixed an issue where rebooting the Board Management Controller (BMC) when the Operating System was Running would cause the BMC to stop responding.

·       Fixed an issue on 2-socket clump systems (i.e. chassis with Intel Intel Xeon® Scalable 61xx series processors installed) where the fans would jump to maximum speed and remain there if the BMC was rebooted with the system power on.

·       The ‘SHOW UVDMP’ command always displayed one screen at a time and require user to interact with the keyboard to move to the next page, even with the CLI in script mode. This is now fixed.

·       IPMI watchdog is unsupported and can no longer be enabled. This prevents an issue seen in prior versions where a multi-chassis reboot from OS would fail when IPMI watchdog was enabled.

·       Fixed a syntax issue allowing to use the CLI ADD LOCATION command with “module=rmc” on eRMC.

Version 2.4.98:

FIXES:
BIOS:

·       Updated Intel microcode to address CVE-2017-5715

·       Some I/O Fatal errors (e.g. Malformed TLP, RxOverflow, FlowCntl, DLLP, etc) detected at the End Point device no longer cause an MCA and the system now allows OS recovery instead of rebooting.

 

Version 2.3.132:

FIXES:

·       Removed the Intel microcode that was issued to address the Spectre/Meltdown security vulnerability, which Intel then asked vendors not to use (see Intel guidance here).

 

Version 2.3.122:   REMOVED due to Intel microcode issue.

FIXES:

·       Fixed an issue where the eRMC SET FACTORY command could cause the eRMC to become unusable while trying to initialize the configuration flash partition. The SET FACTORY command is now supported on eRMC.

·       Fixed an issue where CAE service event id #306 (uncorrectable memory data read error) incorrectly encoded DIMM group number, causing the wrong DIMM to be indicted.

Version 2.3.110: REMOVED due to Intel microcode issue.
FIXES:

The following issues were fixed:

·       Addresses security vulnerability CVE-2017-5715; see updates in this advisory.

·       DCD was not supported with firmware version 2.3.94.

·       The eRMC uses NTP daemon internally to keep the management times synchronized. The internal NTP daemon usage is very limited in scope, but is of older ntpd version (4.2.6p5). As a result of older ntpd version, security scanner may falsely flag vulnerabilities that are not applicable to Superdome Flex eRMC system. To mitigate security impact, follow HPE required security best practices.

·        When BIOS de-configures a DIMM, the eRMC will correctly record the data, but will incorrectly return no de-configuration the next time BIOS boots. This incorrect information causes BIOS to retrain the DIMM and attempt to use it. Marginal DIMM may sometime pass the retrain and be included in the system for OS use. Because the DIMM is marginal, it may fail at a later time and cause the OS to crash. To minimize the chance of marginal DIMM being used at next boot, run SHOW DECONFIG and SHOW INDICT after the system is booted and replace any DIMM that has been indicted and de-configured.

·       Memory on some sockets may be in SDDC mode instead of the intended ADDDC mode

 

Version: 2.3.94: Initial version.

FEEDBACK
As we are continuing to improve the firmware management process we welcome your feedback on this document and on the firmware update process: TEAM-FWupdateFeedback@groups.ext.hpe.com