TITLE: HPE Superdome Flex Server Firmware Bundle (for installation from RMC)

VERSION:
Bundle Version: 2.5.290

 

VERSION 2.5.290 contains:  
rmc.2.20.227 
bmc.2.20.227 
bios.6.2.74.20181023_161036
fwu.1.20.8-20180516-163547
rmc-emmc.2.20.227
  

DESCRIPTION:
This bundle contains the firmware file for updating the HPE Superdome Flex server firmware from the RMC. This file updates the server BIOS firmware as well as firmware on the RMC (Rack Management Controller) and on the BMCs (Board Management Controller). 

Note: To comply with Open Source requirements, the Open Source used in Superdome Flex RMC/BMC firmware is provided in the tar file foss_SDFlex_1.2.tar.gz (only included for Open Source documentation purpose).

 

UPDATE RECOMMENDATION: Critical

 [X ] Critical                                                       

    [ ] Panic, [ ] Hang, [ ] Abort, [ ] Corruption, [ ] Memory Leak, [ ] Performance

    [  ] Security  [X ] Other: Unexpected system behavior

    [ ] Hardware Enablement, [ ] Software Enablement

[ ] Required

[ ] Recommended

[ ] Optional

    [ ] Hardware Enablement, [ ] Software Enablement, [ ] non-critical

[ ] Initial Customer Release

 

SUPERSEDES:                                
Version: 2.5.280
 

PRODUCT MODEL(S):     
HPE Superdome Flex Server

 

OPERATING SYSTEMS: 

• Windows Server 2016 on systems with up to 16 sockets 
• Red Hat Enterprise Linux 7.3, 7.4, 7.5
• SUSE Linux Enterprise Server 12 SP2, 12 SP3
• Oracle Linux 7 UEK (Unbreakable Enterprise Kernel) 4 update 5
• Oracle VM 3.4.4,  3.4.5 on systems with up to 8 sockets
• VMWare 6.5 U1, U2 and 6.7 on systems with up to 8 sockets

LANGUAGES:
International English

ENHANCEMENTS:  

·         None

 

FIXES:

o   Firmware version 2.5.290 resolves certain potential unexpected system behavior when operating the Superdome Flex system or nPar(s) in HPC mode. For systems or partitions running in HPC mode, the frequency of system memory errors may increase after the update. These memory errors will be fully visible in memlog.  HPE strongly recommends that customers run in RAS mode, but if they elect to continue using HPC mode, they should update to this firmware version to eliminate the risk of unintended side effects from memory correction including possible system crashes. Refer to this advisory for more details.

COMPATIBILITY:

·         To enhance security on the RMC/eRMC, ssh has been upgraded starting with firmware version 2.5.246, OpenSSH version 6.5 or later and Putty version 0.68 or later are required. Older OpenSSH or Putty will fail to connect to the RMC/eRMC.

·         It is recommended to use HPE Superdome Flex I/O Service Pack version 2018.09 (or later) with this server firmware version as well as HPE Foundation Software version 1.2.2 or later (for Linux only) and DCD version 1.2 or later (for Linux).

o   DCD for Linux is part of HPE Foundation Software (supported on Linux only)

o   DCD for Oracle VM is posted under the Superdome Flex server download page

o   DCD for VMware is posted on Vibsdepot

·         For OS specific information, please see:

o   For VMware, the “Running VMware vSphere on HPE Superdome Flex Server” white paper.

o   For Windows, the “Running Microsoft Windows Server on HPE Superdome Flex Server” white paper.

o   For Linux, the “HPE Superdome Flex Server Software Installation and Configuration Guide” available under https://support.hpe.com/hpesc/public/home/documentHome?sp4ts.oid=1010323142.

PREREQUISITES:  

• The following applies if you are upgrading firmware from version 2.4.98 and are using OneView to monitor the Superdome Flex server:
• A unique certificate per RMC/eRMC is re-generated. This disrupts communication from OneView to RMC. To avoid this, follow the steps below prior to upgrading the firmware:

    In OneView instance,

    i) Go to Settings -> Security

    ii) Click Manage certificates button.

    iii) Delete the RMC certificate from the list.

After upgrading the firmware, a new re-generated certificate would be added to OneView’s trust store after a refresh of that RMC.

• Prior to updating the firmware, make sure that you know the RMC administrator password
• HPE requires the following security best practices for correct operation and to help reduce both known and future security vulnerability risks:

1.       Isolate the management network from the normal corporate LAN. This management network should limit and restrict access to your RMC management interfaces using firewall, Accesses control lists (ACLs), or VPN.  This will greatly reduce a large group of security risks, (for example Denial of Service attacks).

2.       Patch and maintain web servers.

3.       Run the up-to-date virus and malware scanners in your network environment

4.       Apply HPE firmware updates as recommended.

 

INSTALLATION INSTRUCTIONS:

Please review all instructions and the "Hewlett Packard Enterprise Support Tool License Terms" or your Hewlett Packard Enterprise support terms and conditions for precautions, scope of license, restrictions, and limitation of liability and warranties, before installing this package. It is important that you read and understand these instructions completely before you begin. This can determine your success in completing the firmware update.

Note: It is highly recommended that firmware updates be executed by Hewlett Packard Enterprise support personnel.

 

INSTALLATION
1. Copy the firmware file sd-flex-2.5.290-fw.tars to your local computer.
 

2. Follow the instructions below to update the firmware version 2.5.290 on your system.

 

a.       Log into the HPE Superdome Flex Server operating system as the root user, and enter the following command to stop the operating system:
          # shutdown

b.      Login to the RMC as administrator user, provide the password when prompted.

c.       Verify that the RMC is configured to use DNS access by running:
          RMC cli> show dns
If not, you may use the command “add dns” to configure DNS access (or you can’t use DNS).

d.      Enter the following command to power off the system
        - If there is only 1 partition, partition 0 is the default:
                 RMC cli> power off npar pnum=0  
         - In case of multiple partitions, enter show npar to find the partition number, then enter:
                RMC cli> power off npar pnum=x, where x is the partition number

e.      Update the firmware by running the command: 
          RMC cli> update firmware url=<path_to_firmware>

                   Where <path_to_firmware>  specifies the location to the firmware file that you previously
                   downloaded. You can use https, sftp or scp with an optional port. For instance:
                         RMC cli> update firmware url=scp://username@myhost.com/sd-flex-<version>-fw.tars
                         RMC cli> update firmware url=sftp://username@myhost.com/sd-flex-<version>-fw.tars
                         RMC cli> update firmware url=https://myhost.com/sd-flex-<version>-fw.tars
                         RMC cli> update firmware url=https://myhost.com:123/sd-flex-<version>-fw.tars

                    Note: The CLI does not accept clear text password, the password has to be manually typed in.
                    Note: To use a hostname like ‘myhost.com’, RMC must be configured for DNS for name
                             resolution, otherwise you need to specify the IP address of ‘myhost.com’ instead. See
                             the command ‘add dns’ for more information.

f.        Wait for RMC to reboot after a successful FW update, then check the new firmware version installed by running:
         RMC cli> show firmware verbose

g.       Power on the system or partition:
         - To power up a system configured with all chassis in one large nPartition numbered 0, enter:
                    RMC cli>  power on pnum=0.
          - If you have multiple npars, each npar can be powered on separately using:
                   RMC cli>  power on npar pnum=x, where x is the partition number.

 

DETERMINING CURRENT VERSION:
To check or verify the current firmware levels on the system, from the CLI, enter the RMC command:  
         RMC cli>  show firmware
                Configured version: 2.5.290
                Firmware on all devices matches the configured version.

 

    Note: If you want to see all the components’ versions, you may use “show firmware verbose”.

 

KNOWN ISSUES & WORKAROUNDS:  

 

·         Firmware update of the eRMC chassis may hang indefinitely. See advisory a00056833 for recovery steps.

·         When upgrading firmware from version 2.4.98, a unique certificate per RMC/eRMC is re-generated. For systems using OneView, the RMC’s older certificate residing in OneView’s trust store will become stale and communication with the RMC will not succeed.
Workaround: To restore OneView to RMC communication after updating from version 2.4.98, follow the steps below (to address it before update, see pre-requisites):

In OneView instance,

i) go to Settings -> Security

ii) Click Manage certificates button.

iii) Delete the RMC certificate from the list.

iv) Initiate rack manager refresh

·         The CLI provides a convenient ‘ipmi’ wrapper script. However, serial over lan (SOL) is not supported by this convenient ‘ipmi’ command. Attempting to activate partition console via ‘ipmi command=”sol activate”’ will fail with the message: “Error: This command is only available over the lanplus interface”. User should use CLI ‘connect npar’ or ‘uvcon’ to connect to partition console.

·         The Superdome Flex BMC (Board Management Controller) GUI page will always first display the AMI logo, then automatically replaces the AMI logo with HPE logo. This behavior is seen at login and each time the GUI page is refreshed. The GUI functionality is unaffected by dual logo display.

·         VMedia does not support UEFI reconnect -r command.
Workaround: After attaching VMedia to the partition, use POWER RESET to reset the partition and activate VMedia.

·         Superdome Flex internal management network uses 172.16.0.0/16, 172.30.50.0/24, 172.30.60.0/24, and 10.1.0.0/16 subnets. As a result, IP addresses inside of these 4 subnets cannot be assigned to the RMC, eRMC, or BMC management port.

 

For more details on accessing and managing the system, see the HPE Superdome Flex user documentation located at this link.

 

DISCLAIMER:
The information in this document is subject to change without notice.
Hewlett Packard Enterprise makes no warranty of any kind with regard to this material, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett Packard Enterprise shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this material.

This document contains proprietary information that is protected by copyright. All rights are reserved. No part of this document may be reproduced, photocopied, or translated to another language without the prior written consent of Hewlett Packard Enterprise.

(C) Copyright 2017-2018 Hewlett Packard Enterprise Development L.P.

 

SUPERSEDES HISTORY:
Version 2.5.80:

ENHANCEMENTS:

• Added support for Nvidia P40 GPU
• System now recognizes the Nvidia V100-32GB GPU card so that it can detect its presence and adjust its GPU cooling calculation accordingly

FIXES:

The following issue has been addressed in firmware version 2.5.280:

·         Fixed an issue where a bugcheck or unexpected process termination would be seen after an uncorrectable memory error on systems running Microsoft Windows Server 2016.

 

The following issues have been addressed in firmware version 2.5.270 and later:

• Fixed an issue where RHEL 7.5 kernel would panic in presence of saved error records
• Fixed an issue where a system under significant load would crash when running certain commands
• Fixed an issue where, after detecting an uncorrected memory error, the faulty DIMM would get indicted but not de-configured, potentially causing the OS/partition to stop responding or the system to crash after reboot if the faulty DIMM had not been replaced.
• The CLI SHOW SENSOR TARGET=<chassis GEOID> command no longer erroneously displays the DIMM temperature reading as "N/A" when the chassis is powered on.
• Fixed an issue where the TEST FABRIC command would incorrectly display that all chassis have a Base IO Board
• Fixed an issue where an incorrect FAN_FATAL Integrated and Service Events would be sent upon removing the AC power cord
• When the Superdome Flex Rack Management Controller (RMC) reboots, the RMC command CLI SHOW CHASSIS INFO now correctly reports the number of installed power supplies in each chassis
• Fixed a stack trace issue observed when using the IDK Error Injection tool
• Fixed an error reporting problem where the register would report the Requester ID incorrectly
• PCIe Error injection using EINJ (Error Injection) now works for all PCIe segments
• System will no longer ASSERT when injecting PCIE error using EINJ (Error Injection) with invalid PCIE address

Version 2.5.256:

FIXES:

• Fixed an issue where the Superdome Flex BMC inadvertently ignored GPU thermal data for its cooling algorithm, which might cause the GPU to run at an elevated temperature and result in the GPU running much slower to avoid overheating. HPE strongly recommends that systems with GPU do not run with firmware version 2.5.246 but update to version 2.5.256 or later where this issue is fixed.
• Version 2.5.246 and later fixed a DIMM_LOAD_ORDER_VIOLATION issue caused by checking the DIMM loading order for disabled sockets.
• Version 2.5.246 and later addressed the following L1 Terminal Faults:

-         L1 Terminal Fault - OS, SMM (CVE-2018-3620). Please note this mitigation also requires operating system software updates.

-         L1 Terminal Fault - OS, VMM (CVE-2018-3646). Please note this mitigation also requires operating system software updates, and VMM software updates.

-         For more information, see the bulletin a00055017en  

Version 2.5.246:

ENHANCEMENTS:  

·         Added support for CPU models 8170M, 8170, 8168, 6140, 6140M, 6150, 6142M, 6142, 6138

·         Added support for 24 and 28 socket configurations (in addition to 4, 8, 12, 16, 20 and 32)

·         Added partitioning (nPAR) support with ability to convert non partitioned systems to partitionable

·         Added support for HPE Ethernet 10Gb 2-port 562T adapter, 32Gb Fibre Channel SN1600Q, SN1600E HBAs

·         Added support for Windows Server 2016 with up to 16 sockets

·         Added support for RHEL 7.5

·         Added support for VMware 6.5 U2

·         Added support for Oracle VM 3.4.4

·         Added new security and management features (secure boot, SSH upgrade, reduced port usage)

·         Added support for offline firmware update via SUM (Smart Update Manager)

·         Added support for OneView monitoring (requires OneView version 4.1 or later)

·         Added support for provisioning OS with redfish using the OpenStack Ironic  (Requires Openstack Ironic version ‘Pike’  or later)

·         Enhances security with SSH

FIXES:

·         New BIOS addresses the following known vulnerabilities, CVE-2018-3639 and CVE-2018-3640.

·         Fixed an issue where the firmware update would fail on rare occasions to update the BIOS image.

·         Some IO errors no longer cause an incorrect decoding to be logged in the Integrated Event Log with the string “[physloc_err=5]”.

·         Fixed an issue where rebooting the Board Management Controller (BMC) when the Operating System was Running would cause the BMC to stop responding.

·         Fixed an issue on 2-socket clump systems (i.e. chassis with Intel SkyLake 61xx series processors installed) where the fans would jump to maximum speed and remain there if the BMC was rebooted with the system power on.

·         The ‘SHOW UVDMP’ command always displayed one screen at a time and require user to interact with the keyboard to move to the next page, even with the CLI in script mode. This is now fixed.

·         IPMI watchdog is unsupported and can no longer be enabled. This prevents an issue seen in prior versions where a multi-chassis reboot from OS would fail when IPMI watchdog was enabled.

·         Fixed a syntax issue allowing to use the CLI ADD LOCATION command with “module=rmc” on eRMC.

Version 2.4.98:

FIXES:
BIOS:

·         Updated Intel microcode to address CVE-2017-5715

·         Some I/O Fatal errors (e.g. Malformed TLP, RxOverflow, FlowCntl, DLLP, etc) detected at the End Point device no longer cause an MCA and the system now allows OS recovery instead of rebooting.

 

Version 2.3.132:

FIXES:

·         Removed the Intel microcode that was issued to address the Spectre/Meltdown security vulnerability, which Intel then asked vendors not to use (see Intel guidance here).

 

Version 2.3.122:   REMOVED due to Intel microcode issue.

FIXES:

·         Fixed an issue where the eRMC SET FACTORY command could cause the eRMC to become unusable while trying to initialize the configuration flash partition. The SET FACTORY command is now supported on eRMC.

·         Fixed an issue where CAE service event id #306 (uncorrectable memory data read error) incorrectly encoded DIMM group number, causing the wrong DIMM to be indicted.

Version 2.3.110: REMOVED due to Intel microcode issue.
FIXES:

The following issues were fixed:

·         Addresses security vulnerability CVE-2017-5715; see updates in this advisory.

·         DCD was not supported with firmware version 2.3.94.

·         The eRMC uses NTP daemon internally to keep the management times synchronized. The internal NTP daemon usage is very limited in scope, but is of older ntpd version (4.2.6p5). As a result of older ntpd version, security scanner may falsely flag vulnerabilities that are not applicable to Superdome Flex eRMC system. To mitigate security impact, follow HPE required security best practices.

·          When BIOS de-configures a DIMM, the eRMC will correctly record the data, but will incorrectly return no de-configuration the next time BIOS boots. This incorrect information causes BIOS to retrain the DIMM and attempt to use it. Marginal DIMM may sometime pass the retrain and be included in the system for OS use. Because the DIMM is marginal, it may fail at a later time and cause the OS to crash. To minimize the chance of marginal DIMM being used at next boot, run SHOW DECONFIG and SHOW INDICT after the system is booted and replace any DIMM that has been indicted and de-configured.

·         Memory on some sockets may be in SDDC mode instead of the intended ADDDC mode

 

Version: 2.3.94: Initial version.

FEEDBACK
As we are continuing to improve the firmware management process we welcome your feedback on this document and on the firmware update process: TEAM-FWupdateFeedback@groups.ext.hpe.com