TITLE: HPE Integrity MC990 X Server Firmware Bundle

VERSION:
Bundle Version: 2020.06
RMC version: 1.2.6
BMC Version: 1.2.4
BIOS Version: 5.2.13
PSOC Version: 20.1.9
MB PLD1 Version:  4.1.18 or 10.1.17 or 10.1.18 (*)
MB PLD2 Version:  5.1.12 or 7.1.12 (*)
HARP PLD3 Version: 3.1.19

 

*: Versions 10.1.17 (or 10.1.18) and 7.1.12 are returned on systems with boards that support Intel EX v4 family processors

 

DESCRIPTION:
This bundle contains the RMC, BMC and BIOS firmware files for HPE Integrity MC990 X Servers. These files allow updating the MC990 X server firmware (RMC, Motherboard BMC, Motherboard BIOS, Motherboard and HARP board PLD, Power board PSOC). The PLD and power board PSOC firmware are updated using the BMC file.

 

Note: Firmware bundle version 2016.06 or later is required to support servers running with Intel processor EX v4 family.

 

Important: The default admin login password for the BMC has been changed starting with firmware version 2018.03. If the user had changed the password before updating the firmware, it will remain to what the user had set it to. Otherwise, it is now set to HPE1nnov by default instead of admin. The same applies to the IPMI admin password (now HPE1nnov instead of admin). Note: HPE recommends changing the default password after initial setup of the system.

 

UPDATE RECOMMENDATION: Recommended

[ ] Critical                                                       

    [ ] Panic, [ ] Hang, [ ] Abort, [ ] Corruption, [ ] Memory Leak, [ ] Performance

    [ ] Security

    [ ] Hardware Enablement, [ ] Software Enablement

[ ] Required

[X ] Recommended

[ ] Optional

    [ ] Hardware Enablement, [X] Software Enablement, [ ] non-critical

[ ] Initial Customer Release

 

SUPERSEDES:                                
Bundle 2019.08

 

PRODUCT MODEL(S):     
HPE Integrity MC990 X Server

 

OPERATING SYSTEMS: 


LANGUAGES:

International English

ENHANCEMENTS:

·       Added support for RHEL 8.1, 8.2 and RHEL 7.8

·       Added support for SLES 12 SP5

·       Added support for OL 7.8, 8.1, 8.2

·       November 2020 update:

o   Added support for SLES 15 SP2

o   Added support for RHEL 7.9, RHEL 8.3 and OL 7.9

·       March 2021 update:

o   Fixed a directory name in the installation instructions

FIXES:
RMC 1.2.6:

·       Uv3dmp improvements:

·       Now generates primary and secondary FRU events for LH_SA AOERRs due to NDC messages from UPI

·       Improved handling of Hub UCE indications when multiple DIMMS on a socket have failed with correctable mem errors

·       Corrected problem with LH_DA_DBC_DATA_UCE_TSTAMP and LH_SC_DBC_DATA_UCE_TSTAMP naming

·       Other miscellaneous fixes

·       Added full support for HPE system serial number format

 

COMPATIBILITY:

The table below lists all the IO cards and firmware versions supported with the HPE Integrity MC990 X server firmware bundle versions 2016.05 or later.

Note: For IO Firmware updates, please contact Hewlett Packard Enterprise support.
Note: With VMware, the maximum number of adapters supported is 8. 

 

Product

Description

Minimum Recommended IO FW versions

H7B97A

HPE SN1100E 16Gb 2p Fibre Channel Adapter

Universal boot 10.7.110.31

H7B68A

HPE MC990 16Gb 2p Fibre Channel Adapter

10.6.144.21

H7B69A

HPE MC990 6Gb 16p Ext SAS Controller

20.00.04.00-IT

H7B70A

HPE MC990 12Gb 8p Ext SAS Controller

10.00.00.00-IT

H7B71A

HPE MC990 6Gb 4p Internal SAS Controller (9260-4i)

12.14.0-0167 / 2.130.393-2551 (Note 4)

Q2U17A

HPE MC990 6Gb 4p Internal SAS Controller (9361-4i)

4.670.00-6500

H7B72A

HPE MC990 10GbE Fiber 2p Adapter

n/a (Note 1)

H7B74A

HPE MC990 10GBASE-T 2p Adapter

n/a (Note 1)

H7B75A

HPE MC990 1000BASE-T 4p 5719 Adapter

n/a (Note 1)

H7B76A

HPE MC990 1000BASE-T 4p Adapter

n/a (Note 1)

H7B78A

HPE MC990 10GbE 2p SFP+ RDMA Adapter

n/a (Note 1) (Note 5)

H7B79A

HPE MC990 IB FDR/EN 40Gb 2p QSFP Adptr

PSID (Note 2): MCX354A-FCBT MT_1090120019
Firmware: fw-ConnectX3-rel-2_36_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.718.bin

H7B80A

HPE MC990 800GB P3700 SSD Accelerator

n/a (Note 3)

H7B81A

HPE MC990 2TB P3700 SSD Accelerator

n/a (Note 3)

 

Note 1: No minimum firmware requirement established. Note that PXE boot is only supported on the BASE I/O NIC (Intel i210)

Note 2: PSID (Parameter-Set Identification) is a 16-ascii character string embedded in the firmware image which provides a unique identification for the configuration of the firmware.
Note 3: No minimum firmware requirement established for Intel SSD S3700 products.  
Note 4: 12.14.0-0167 is a package version, firmware version is 2.130.393-2551
Note 5: H7B78A is not supported with VMware

 

PREREQUISITES:  

·       HPE recommended using BIOS version 5.2.13 with BMC version 1.2.4 and RMC version 1.2.6.

·       It is recommended to use this firmware along with SFS (System Foundation Software) version 2.23 (or later) and the following OS patches that address the MDS (Microarchitectural Data Sampling) CVE-2018-12126, CVE-2018-12127,CVE-2018-12130, CVE-2019-11091 vulnerabilities:

o   SLES 15,12: https://www.suse.com/support/kb/doc/?id=7023736

o   RHEL 8, 7.X: https://access.redhat.com/security/vulnerabilities/mds

o   Oracle Linux / UEK: https://linux.oracle.com/cve/CVE-2018-12130.html

o   VMware: https://www.vmware.com/security/advisories/VMSA-2019-0008.html

·       Prior to updating the firmware, make sure that you know the RMC root password. It will be requested several times during the RMC and BMC firmware update.

 

INSTALLATION INSTRUCTIONS:

Please review all instructions and the "Hewlett Packard Enterprise Support Tool License Terms" or your Hewlett Packard Enterprise support terms and conditions for precautions, scope of license, restrictions, and limitation of liability and warranties, before installing this package. It is important that you read and understand these instructions completely before you begin. This can determine your success in completing the firmware update.

Note: It is highly recommended that firmware updates be executed by Hewlett Packard Enterprise support personnel.

 

INSTALLATION
1. Copy the rpm files that need to be installed to your system. 

2. Follow the instructions below to install the firmware version 2020.06 on your system. Here is a summary of the steps to install the firmware:

Note: Do not leave the system unattended during the update as the RMC root password will be requested multiple times during the update process.

 

Note: Only the RMC has changed from last version. You may skip the installation steps if any of the firmware versions are already up to date.

A.       Flashing the RMC firmware (file: hpe-mc990x-rmc-1.2.6_3-20200601.noarch.rpm)

B.       Flashing the BMC firmware (file: hpe-mc990x-bmc-1.2.4_3-20190523.noarch.rpm)

C.    Updating the BIOS firmware (file: hpe-mc990-bios-REL_5.2.13-2019-07-30_17.26.40-5.2.13-1.noarch.rpm)

D.       Flashing the PLD firmware (uses the BMC file above)

E.     Flashing the power board PSOC firmware (uses the BMC file above)

 

A. Flashing the RMC firmware

To flash the RMC firmware:

1. Install the hpe-mc990x-rmc-1.2.6_3-20200601.noarch.rpm RPM on a system running Linux that has network access to the RMC.

         linux# rpm -ivh hpe-mc990x-rmc-1.2.6_3-20200601.noarch.rpm

2. Change to the directory that contains the flashrmc file:

         linux# cd /usr/local/sysco/firmware/uv300/rmc/1.2.6

 

3. Run the flashrmc flash utility:

        linux# ./flashrmc -r mc990x_rmc.bin <RMC hostname/IP>

 

4. Confirm that the RMC firmware was updated:

      rmc> version

B.       Flashing the BMC firmware

To flash the BMC firmware:
1. Install the hpe-mc990x-bmc-1.2.4_3-20190523.noarch.rpm RPM on a system running Linux that has network access to the RMC.
         linux# rpm -ivh hpe-mc990x-bmc-1.2.4_3-20190523.noarch.rpm

 

2. Change to the directory that contains the flashbmc file:
         linux# cd /usr/local/sysco/firmware/mc990x/bmc/1.2.4

 

3. Cycle the system power manually:

a. Log into the Integrity MC990 X system as the root user, and enter the following command to stop the operating system:
      # halt

b. On the RMC, enter the power off command to power down the Integrity MC990 X system.
For example:
            RMC:r1i1c> power off
            ==== r001i01c (PRI) ====

 

4. Run the flashbmc flash utility:
        linux# ./flashbmc –r mc990xbmc.ima <CMC hostname/IP>

    

  Warning: Flashing the BMC takes approximately 10 minutes. Do not interrupt the process or power cycle the system while flashing the BMCs.

 

5. After the flashbmc utility has completed, on the RMC, enter the power on command to power up the Integrity MC990 X system. The power up takes only a few minutes.
For example:
         RMC:r001i01c> power on

 

5. Confirm that the BMC firmware was updated:
         rmc> bmc version

 

C.     Updating the BIOS firmware

The BIOS is updated by copying a new image to the RMC, creating an entry point in the HWCFG file, and rebooting the system.

To update the motherboard BIOS firmware:
1. Install the hpe-mc990-bios-REL_5.2.13-2019-07-30_17.26.40-5.2.13-1.noarch.rpm RPM on a system running Linux that has network access to the RMC.
           linux# rpm -ivh hpe-mc990-bios-REL_5.2.13-2019-07-30_17.26.40-5.2.13-1.noarch.rpm

 

2. Change to the directory that contains the BIOS image file:
          linux# cd /usr/local/sysco/firmware/uv3/bios/rel-5.2.13

 

3. Copy the BIOS image file to the RMC:
         linux# scp hpe-mc990-bios-REL_5.2.13-2019-07-30_17.26.40.fd root@<RMC hostname/IP>:/work/bmc/common

 

4. Log in to the RMC and link the image file to bios.fd:
         rmc> cd /work/bmc/common
         rmc> ln –s hpe-mc990-bios-REL_5.2.13-2019-07-30_17.26.40.fd   bios.fd

 

5. Create an entry point to the image in the HWCFG file:
        rmc> hwcfg BIOS_FILE=/common/bios.fd

 

6. Reboot the system:
         rmc> power —c reset

 

7. Confirm that the BIOS firmware was updated:
        rmc> bios —v

D.       Flashing the PLD firmware

The PLD firmware is included in the BMC RPM: hpe-mc990x-bmc-1.2.4_3-20190523.noarch.rpm

To flash the motherboard and HARP board programmable logic device (PLD) firmware:

1. Power down the MC990 X server:

            rmc> power off

2. Flash the PLD:

             rmc> bmc flash_pld

 

E.     Flashing the power board PSOC firmware

The PSOC firmware is included in the BMC RPM: hpe-mc990x-bmc-1.2.4_3-20190523.noarch.rpm
To flash the power board PSOC firmware:
      1. Power down the MC990 X server:
               rmc> power off
      2. Flash the PSOC:
               rmc> bmc flash_psoc

 

DETERMINING CURRENT VERSION:
To check or verify the current firmware levels on the system, enter these RMC commands:

RMC> version 
                           HPE Integrity MC990 X RMC, Rev. 1.2.6 [Bootloader 1.1.4]

RMC> bmc version
                          HPE Integrity MC990 X BMC, Firmware Rev. 1.2.4

RMC> bios or RMC> bios —v 
                           Integrity MC990 X BIOS 5.2.13 built in rel-5.2.13

RMC> bmc version —v
           
HPE Integrity MC990 X BMC, Firmware Rev. 1.2.4   [Bootloader 0.6.0]
            Power Board PSOC:   20.1.9
           
MB PLD1:                    4.1.18 or 10.1.17 or 10.1.18 (*)

            MB PLD2:                    5.1.12 or 7.1.12 (*)

            HARP PLD3:                3.1.19

 

 *: Versions 10.1.17 (or 10.1.18) and 7.1.12 are returned on systems with boards that support Intel EX v4 family processors

 

KNOWN ISSUES & WORKAROUNDS:  

·       In rare cases, after updating from a BIOS version prior to version 5.2.10, the system may not boot due to APPWT key issues. If that happens, please contact HPE support.

 

DISCLAIMER:
The information in this document is subject to change without notice.
Hewlett Packard Enterprise makes no warranty of any kind with regard to this material, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this material.

This document contains proprietary information that is protected by copyright. All rights are reserved. No part of this document may be reproduced, photocopied, or translated to another language without the prior written consent of Hewlett Packard Enterprise.

(C) Copyright 2016-2021 Hewlett Packard Enterprise Development L.P.

SUPERSEDES HISTORY:
Firmware bundle 2019.08:
ENHANCEMENTS:

·       Added support for RHEL 8 and 7.7

·       Added support for SLES 15 SP1

·       Added support for VMware 6.7 U3


FIXES:

BIOS 5.2.13: 

§ Latest revision of the Intel microcode which, in combination with operating system and/or hypervisor updates, provides mitigation for a new group of side-channel vulnerabilities known as Micro architectural Data Sampling (MDS). This includes support for mitigating the following vulnerabilities:
CVE-2018-12126 - Micro architectural Store Buffer Data Sampling,
CVE-2018-12130 - Micro architectural Fill Buffer Data Sampling,
CVE-2018-12127 - Micro architectural Load Port Data Sampling, and
CVE-2019-11091 - Micro architectural Data Sampling Uncacheable Memory.

§ Dmidecode no longer reports the same UUID in DMI/SMBIOS for all systems
 

§ Added a new BIOS setup  screen option "Serial number/UUID compatibility" that supports two values, "Unique serial numbers/UUIDs" and "Common serial numbers/UUIDs":

·       When the BIOS setup  screen option is set to the value "Unique serial numbers/UUIDs", the system will report globally unique values in the DMI (a.k.a. SMBIOS) Type 1 records:
       Serial Number:  << system serial number >> 
       UUID: An ID corresponding to a combination of the system serial number and  partition number
      Note: This behavior may be enabled to improve compatibility with third party software which depends on the uniqueness of these values.

·       When BIOS setup  screen option is set to the value "Common serial numbers/UUIDs", the system will report fixed, non-unique values in the DMI (a.k.a. SMBIOS) Type 1 records:
       Serial Number:  MC990X------ 
       UUID: 8B2C9555-5C8E-4221-BF48-DFF302443042                  
       Note:

o   This behavior is the default so as to retain compatibility with third party software which depend on the previously implemented behavior.

·       Note: The setting selected via the "Serial number/UUID compatibility" option in the BIOS setup screens will be cleared and revert to the default value and behavior of "Common serial numbers/UUIDs" anytime the RMC "power -c" command is used.

          RMC 1.2.5 / BMC 1.2.4:

§ Updated to OpenSSH 7.6p1 and OpenSSL 1.0.2n which removed several weak ciphers/MACs

§ Added ability to disable the IPMI (RMCP) interface. Full instructions are provided in this notice: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00081767en_us

Firmware bundle 2019.03:
ENHANCEMENTS:

·       None

FIXES:
BIOS 5.2.12:

·       CRITICAL: Fixed an issue where a system under heavy workload would sometimes MCA with error messages such as "Fatal Link Timeout to PCIe Device" and "LER_ENTERED"

·       Addressed the privilege escalation vulnerability CVE-2018-12204


Firmware bundle 2018.12:
ENHANCEMENTS:

·       Added support for RHEL 7.6, SLES 12 SP4, SLES 15

FIXES:
BIOS 5.2.11:

·       Resolved intermittent BIOS boot TRB_TIMEOUT errors seen after an OS reboot

BMC 1.2.3 (rpm dated 20181031):

Firmware bundle 2018.06:
ENHANCEMENTS:

·       N/A

FIXES:
BIOS 5.2.10:

·       Addressed Speculative Store Bypass or Variant 4 vulnerabilities (CVE-2018-3639 and CVE-2018-3640).

·       Fixed a defect on systems with Broadwell-EX processors where BIOS incorrectly calculated the APPWT value, allowing the systems to boot even though the APPWT value exceeded the RMC default limit of 1.5 WT. Note: That limit was increased to 16 WT (effective as of September 2017) starting with RMC version 1.2.1

RMC 1.2.4:

Firmware bundle 2018.03:

FIXES:

BIOS 5.2.9:

·       Updated Intel microcode to address CVE-2017-5715

·       Changed threshold VMSE read ECC errors on the memory risers

BMC 1.2.3:

·       Fixed an issue where the BMC would hang with a message "BMC linked but not configured"

·       Fixed a Problem where web GUI would incorrectly flag discrete sensors as failed (RED) even though the sensors are nominal and the system fully functional

·       Fixed an issue where bios –s output would show “Hub not present” with 48xx processors

·       Allow customer configuration of the RMC root account SSH authorized_keys file (/root/.ssh/authorized_keys) to control RMC access

·       Addressed issue were Webgo died during a port scan

·       Addressed several bash shell vulnerabilities CVE-2014-7169, CVE-2014-6271,  CVE-2014-6277, CVE-2014-6278, CVE-2014-6277, and CVE-2014-6278. Changes default root password 

RMC 1.2.1:

·       Block external access to unneeded RMC ports that were reported open by security scans

 

Firmware bundle 2017.11:

FIXES:

BIOS 5.2.7:

·       Upstream Linux has made changes to the TSC synchronization code. As those changes flow to distro kernels, installing kernel updates may begin to hit TSC failures. This bios change improves TSC synchronization by making use of the TSC_ADJUST MSR when available. It also averages the adjustment across all threads of a socket, and installs the same TSC adjustment for all threads of a socket, which makes the difference between threads on a socket be zero (when TSC_ADJUST is used).

 

Firmware bundle 2017.06:

FIXES:
BMC:

-      Intel: Xeon Phi 7120p Coprocessor

-      Nvidia: Tesla K40/K80/M40/M60/P100 GPU

-      NOTE: Quadro M6000 is actively cooled and does not need the fan speed adjustment

RMC:

·       Fix invalid node temp sensor reading -126 reported intermittently

 

ENHANCEMENTS:
October 2017:

·       Added support for VMware 6.5 and 6.5 U1

 

Firmware bundle 2017.02:
BIOS 5.2.6:

·       Handle and report PCU SVID errors and memory buffer chip errors to aid in diagnosing VMSE channel versus DDR ECC errors.
For a full list of changes, see the README* file (after installing the rpm file)


Firmware bundle 2016.11:

FIXES:

RMC 1.1.99:

·       Fixed uv3dmp issues related to invalid data in HAx_iMCx_ADDR 

 

ENHANCEMENTS:

Firmware bundle 2016.09:

FIXES:
BIOS 5.2.5:

·       Addresses an issue where correctable memory errors could cause process termination

ENHANCEMENTS:

Firmware bundle 2016.06:

FIXES:

·       None

Enhancements:

 

Firmware bundle 2016.05.25:

FIXES:

BIOS 5.2.03:

FEEDBACK
As we are continuing to improve the firmware management process we welcome your feedback on this document and on the firmware update process: TEAM-FWupdateFeedback@groups.ext.hpe.com