HP iLO에서 오류 메시지 발생시 Embedded media manager failed initialization

 

 

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c04996097

Version: 10

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors

NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2016-11-22

Last Updated: 2018-09-18

---

DESCRIPTION

Document Version	Release Date	Details
10	09/18/2018	Updated Resolution section with information on iLO 4 Firmware 2.61.
9	06/28/2018	Updated link in Resolution.
8	06/08/2018	Updated Resolution, including updated version of the iLO 4 firmware and Intelligent Provisioning considerations.
7	01/23/2018	Added online Intelligent Provisioning update to Resolution.
6	08/22/2017	Updated the Resolution with specific steps to be completed.
5	05/24/2017	Updated the Resolution with additional details on running the script.
4	12/05/2016	Updated the Resolution with additional details on running the script.
3	08/08/2016y	Updated the Resolution with details on running the script.
2	07/21/2016	Updated the Description with additional errors that are observed and updated Resolution to include Step 2 through Step 4.
1	03/31/2016	Original Document Release.

On an HPE Gen8-series or HPE ProLiant Gen9-series servers with HPE Integrated Lights-Out 4 (iLO 4), the NAND flash device may not initialize or mount properly, which may cause a variety of symptoms, which are listed below:

AHS Errors

• AHS Logs display a blank date when performing the following:
  "Select a range of the Active Health log in days From: _______ To:_________"
  AHS file system mount may fail with (I/O Error) or (No Such Device).
• The HP Active Health System (AHS) Logs are unable to be downloaded. AHS Data is not available due to a filesystem error.
• The iLO Diagnostic tab will display the following error message: "Embedded media manager failed initialization" or "The AHS file system mount failed with (No such device) or "The AHS file system mount failed with (I/O error)" or "Controller firmware revision 2.09.00 Could not partition embedded media device."
• Unable to download AHS log and "bb_dl_disabled" is displayed.

Embedded Media Errors

• Controller Firmware Version 2.09.00 may fail to restart.
• Unable to partition Embedded media device.
• Embedded media manager may fail initialization.

Intelligent Provisioning Errors

• Intelligent Provisioning will not execute when selecting F10.

OneView Errors

• Remote Insight/Integrated Lights-Out Self-Test Error 8192.
• Unable to register this HP OneView instance with iLO: There was a problem with posting a command to the iLO.
• Unable to register this HP OneView instance with iLO: The iLO initialization was unable to complete.
• Unable to determine if this server hardware is being managed by another management system. Received an error from iLO <ip address of iLO> with Error: Blob Store is not yet initialized. and Status: 126

SCOPE

Any HPE ProLiant Gen8-series or HPE ProLiant Gen9-series server running iLO 4.

RESOLUTION

The resolution for this issue may take several steps that need to be completed in the order specified below.

OVERVIEW

Step 1) Upgrade the iLO 4 firmware to version 2.61
Step 2) Perform a NAND format
Step 3) Check the iLO status If the iLO status is normal, then skip to Step 6 If the iLO status is still degraded, continue to Step 4
Step 4) Schedule downtime; AC power-cycle and repeat the NAND format
Step 5) Check the iLO status If the iLO status is normal, continue to
Step 6 If the iLO status is still degraded, then skip to Step 7
Step 6) Perform these final steps if the system board does not need to be replaced: Reboot the server; reinstall IP; and refresh the server in OneView (if server is managed by OneView)
Step 7) If steps 1-4 did not resolve the degraded iLO, replace the system board.

Note: The 2.61 iLO 4 firmware is a critical update. As such, HPE requires users to update to this version immediately. Install this update to take advantage of significant improvements to the write algorithm for the embedded 4 GB non-volatile flash memory (also known as the NAND). These improvements increase the NAND lifespan.

Additional considerations before performing a NAND format

A NAND format can be performed while the server is online in most cases.

Exception: For ESXi hosts booting from the Embedded SD Card - it isstrongly recommended to perform the NAND format with the ESXi OSshutdown.This recommendation also applies when updating the iLO 4firmware or resetting the iLO for ESXi hosts booting from the Embedded SDCard

A server AC power removal may be required (prior to the NAND format) in order for the NAND format to be successful. This can be accomplished for ML or DL servers by shutting down the server and disconnecting the power cables for a few seconds. For blade servers, an E-fuse can be accomplished by logging into the OA CLI and typing "reset server #" where "#" is the bay number of the blade.

An AuxPwrCycle feature was added in iLO 4 firmware version 2.55 so that the equivalent of an AC power removal can be performed remotely on a server.
Refer to Customer Notice "HPE Integrated Lights Out (iLO) 4 - RESTful Command to Allow an Auxiliary Power-Cycle Is Available in Firmware Version 2.55 (and Later)" located at the following URL:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00047494en_us

Required steps to perform after a successful NAND format

• If the server is powered on when performing the NAND format, a server reboot is required after a successful NAND format. This reboot repopulates the RIS and RESTful data on the server during the next server POST with iLO 4 firmware version 2.53 or newer installed. If iLO 4 firmware version 2.50 or older is installed, connect to the iLO CLI (using putty) and use the following command "oemHP_clearRESTapistate" before rebooting the server to repopulate the RIS and RESTful data on the server.
• Reinstall Intelligent Provisioning (IP) to ensure that it is working properly and reported as installed in the iLO 4 GUI "System Information - Firmware Information" page.
• If using OneView: After the server is rebooted (to repopulate the RIS and RESTful data on the server), perform a server refresh in OneView; any existing errors in OneView will need to be marked as cleared after the refresh.

Detailed steps

Step 1. Upgrade the iLO 4 firmware to version 2.61. To download the firmware. The latest version of the iLO 4 firmware is available as follows:

Note: For ESXi hosts booting from the Embedded SD Card (Gen8/Gen9)- it is strongly recommended to perform this step with the ESXi OS shutdown.

1. Click the following link:
   https://support.hpe.com/hpesc/public/home
2. Enter a product name (e.g., "DL380 Gen9") in the text search field and wait for a list of products to populate. From the products displayed, identify the desired product and click on the Drivers & software icon to the right of the product.
3. From the Drivers & software dropdown menus on the left side of the page:
   • Select the Software Type - (e.g. Firmware)
   • Select the Software Sub Type - (e.g. Network)
   • For further filtering if needed - Select the specific Operating System from the Operating Environment.
4. Select the latest release of Firmware - Lights-Out Management iLO 4 firmware version 2.61 (or later). Note: To ensure the latest version will be downloaded, click on the Revision History tab to check if a new version of the firmware/driver is available.
5. Click Download.

Step 2. Perform NAND format using one of the methods detailed in the customer advisory: using one of the methods detailed in the customer advisory: HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning
Find the document at:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00048622en_us

Note: iLO 4 firmware version 2.44 (or later) is required to format the NAND using the following steps.

Step 3. Check the iLO status using the options available in the customer advisory: HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning (find the document at the link in Step 2).

If the iLO status is normal based on the above criteria, then skip to Step 6 If the iLO status is still degraded, continue to Step 4

Step 4. If the iLO status is still degraded, perform the following steps:

a) Schedule a maintenance window
b) Shut down the server
c) Perform an E-fuse (server blade) or AC Power Pull (DL / ML series servers) d) Perform the NAND format again (refer to the instructions in Step 2 above)

Step 5. Check the iLO status (refer to the instructions in Step 3 above)
If the iLO status is normal, continue to Step 6
If the iLO status is still degraded, then skip to Step 7

Step 6. Perform these final steps after the NAND format is successful:
a) Reboot the server
b) Reinstall Intelligent Provisioning (see additional details below)
c) If the server is managed by HPE OneView, perform a server refresh to bring the server back under management.

Note: Any existing errors in OneView will need to be marked as cleared after the refresh.

Step 7. If steps 1-4 did not resolve the degraded iLO, contact HPE support to arrange a system board replacement. Follow these steps to complete the remediation:

a) Open an HPE support case to arrange for a replacement system board / arrange a maintenance window
b) During the maintenance window, shutdown the server
c) Unassign the OneView profile (if the server is under OneView management)
d) Replace the system board
e) Enter Server Model and Serial Number via RBSU
f) Update to iLO 4 firmware 2.61 (or later)
g) Check iLO status (refer to Step 3 above)
h) Reassign the OneView profile (if the server is under OneView management)

Note: Any existing errors in OneView will need to be marked as cleared after the OneView profile is applied.

If additional assistance is needed, contact HPE support and reference Advisory a00019495en_us as follows:

Click on the following URL to locate the HPE Customer Support phone number in your country:

https://h20195.www2.hpe.com/v2/Getdocument.aspx?docname=A00039121ENW .

OneView Considerations

OneView relies on the NAND to be accessible to perform many operations such as adding a new server or applying a profile. Because of this there are several things that need to be understood when dealing with this issue in an OneView environment.

1. OneView interaction with the NAND - OneView uses a portion of the NAND called the iLO blob store. If the blob store is not accessible, adding a server to OneView or assigning a profile to the server cannot be completed.
2. Impact of an inaccessible NAND when managed by OneView - If the NAND becomes inaccessible after the server was added in OneView, there are several things that can cause an outage:
   1. E-fuse - If an E-fuse reset is performed, the server will not be able to be brought back under management until the NAND issue is remediated and this will cause an unexpected outage.
   2. Server is removed and reinserted - This would essentially be the same as performing an E-fuse, so the same information in the E-fuse section applies here.
   3. Un-assign a server profile - If a server profile is unassigned, the server profile will not be able to be reassigned until the NAND issue is remediated.
3. Migration from Virtual Connect to OneView - If an enclosure is migrated from Virtual Connect and a server has an inaccessible NAND, that server will not be able to be added properly until the NAND issue is remediated. If an in-service migration is attempted on a server with an inaccessible NAND, an unexpected outage will occur. Reference advisoryhttps://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c05384185for more detail.
4. Formatting of the NAND - The NAND format wipes the iLO blob store that is used by OneView. It is important to issue a server refresh after a successful NAND format to recover the blob that OneView uses.

Reinstalling Intelligent Provisioning Considerations

There are online and offline options for reinstalling Intelligent Provisioning

• Intelligent Provisioning can be reinstalled online if the server is running Windows (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers only) or Linux (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers). The available online Windows packages are older versions of Intelligent Provisioning.
• Intelligent Provisioning must be reinstalled offline if the server is running VMware (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers).
• The Intelligent Provisioning software download links are provided below. Intelligent Provisioning versions
• HPE ProLiant Gen8-series servers are only supported with Intelligent Provisioning 1.x
• " HPE ProLiant Gen9-series servers are only supported with Intelligent Provisioning 2.x

Note: Intelligent Provisioning version 3.x is for HPE ProLiant Gen10-series servers only.

Offline method considerations:

• Run the Intelligent Provisioning Restore Media to restore the Intelligent Provisioning data. Instructions to restore Intelligent Provisioning are as follows:
• Instructions to create a bootable DVD with the IP image are provided on the HPE Intelligent Provisioning recovery media download site under the Installation Instructions tab.

Reference:
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_170e0ac3cb6e4439bf313f137a#tab3

Note: The DVD can be used multiple times.

Please note the Intelligent Provisioning Recovery Media DVD may be remotely mounted using HPE Integrated Lights-Out 4 (iLO 4) Virtual Media functionality, in order to reinstall Intelligent Provisioning. Additional information is available in the HPE iLO 4 User Guide at the following URL regarding how to mount an ISO image (federated or un-federated) and perform basic virtual media operations. Reference Pages 189 and 223-237:

HPE iLO 4 User Guide:
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c03334051-16

In addition, it is possible to write a script that utilizes HPE Integrated Lights-Out 4 (iLO 4) to reinstall Intelligent Provisioning on multiple servers. HPE Lights-Out management processors support an advanced scripting interface for group configuration and server actions. Scripts would need to be customized for the specific environment and task. Sample scripts are available for customers to reference at the following URL: HPE Lights-Out XML Scripting Sample for Windows: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_459b8adc29c04317ad1d6a6752

Intelligent Provisioning software download links

HPE ProLiant Gen8-series servers are only supported with Intelligent Provisioning 1.x
HPE ProLiant Gen9-series servers are only supported with Intelligent Provisioning 2.x

a.) Linux RPM located at the following URLs: For HPE Gen8-series servers: http://downloads.linux.hpe.com/SDR/repo/ip/rhel/current/x86_64/gen8/

For HPE Gen9-series servers: http://downloads.linux.hpe.com/SDR/repo/ip/rhel/current/x86_64/gen9/firmware-intelligentprovisioning-ip-2.71-1.1.x86_64.rpm

b.) The Windows files are located at the following URLs:

For HPE ProLiant Gen8-series servers: Intelligent Provisioning for Microsoft Windows 64-bit Operating Systems 1.64.0.0 (2 Mar 2017) cp031302.exe https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_01b900175afc4ae68be6c87b39
For HPE ProLiant Gen9-series servers: Intelligent Provisioning for Windows x64 2.50.0.0(6 Jan 2017) cp031091.exe https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_b031497eafd94b9ea2156cdef6

c) ISO images (e.g. for VMware) on Hewlett Packard Enterprise Support Center located at the following: https://support.hpe.com/hpesc/public/home/driverHome?sp4ts.oid=1008862660

For HPE ProLiant Gen8-series servers: Intelligent Provisioning Recovery Media, version 1.70(9 Oct 2017)
For HPE ProLiant Gen9-series servers: Intelligent Provisioning for Gen9 Servers, version 2.70(b)(28 Feb 2018)