X3850 X5 / X3950 X5 PCI Reboot 발생시 해결법

PCIE adapters may be loose - IBM System x3850 X5, x3950 X5
PCI슬롯이 루즈해져서 그런현상이 생깁니다

아래 리테인 확인하시면 됩니다

Source

RETAIN tip: H197448

Symptom

PCIe adapters may appear loose and may wobble at the top if moved side-to-side. With sufficient strain placed on the card from external cables, the card may be lifted from its socket sufficiently to disconnect.

Affected configurations

The system may be any of the following IBM servers:

• System x3850 X5, type 7145, any model
• System x3850 X5, type 7146, any model
• System x3950 X5, type 7145, any model

This tip is not software specific.
This tip is not option specific.

Solution

PCIe adapters may appear loose and may wobble at the top if moved from side-to-side. With sufficient strain placed on the card from external cables, the card may be lifted from its socket sufficiently to disconnect. One or more of the following symptoms may be observed for unseated adapters:

• During system start up, Power On Self Test (POST) intermittently does not recognize an installed PCIe adapter
• During runtime, a PCI error occurs.
• "All PCI Err" event in Integrated Management Module (IMM) log
• The system resets because of a PCIe adapter driver software Non-Maskable Interrupt (NMI) as diagnosed by the logged events.
• Intel Network Interface Controller (NIC) Light Emitting Diode (LED) indicates it linked at 100 Mb instead of 1 Gb as expected, i.e. the "1Gb Link" LED is out.
• When not connected to a switch, the three LEDs on a QLogic FC Host Bus Adapter (HBA) remain solidly lit ("Power on before firmware initializes") instead of blinking as expected ("Power on after firmware initialized / Beacon")
• An installed PCIe adapter may be seen to be partially lifted of the rear of the PCI slot by 1 to 2 MM, or about 1/10 inch, or more. When the adapter is seated properly, the top of the adapter bracket should sit flush down against the I/O Shuttle chassis.
• The bottom edge, or components along the bottom edge, of an installed PCIe adapter may be making contact with one of the two (2) large heat sinks on the input/output hub (IOH) chips that are adjacent to the PCIe slots in the I/O board.
• An adapter may unexpectedly pop up when the PCIe Adapter retention latch is opened.
• A PCIe adapter bracket tail stock may be stubbed on its slot on the I/O shuttle as shown in the picture below. This can cause the adapter to be not fully seated by about 1 MM which is enough to cause one or more of the symptoms above.

The I/O Shuttle slots and PCIe slot label are covered by chassis shielding metal, so it's not readily apparent whether the adapter bracket tailstock is stubbed when the I/O shuttle is fully engaged in the system chassis. After powering off the server, the tab slots can be made visible by camming the entire I/O Shuttle up off its connectors.

Affected Configuration

The system may be any of the following IBM servers:

• System x3850 X5, type 7145, any model
  
• System x3850 X5, type 7146, any model
  
• System x3950 X5, type 7145, any model
  

This tip is not software specific.
This tip is not option specific.
The system has the symptom described above.

Workaround

To ensure the symptoms were not caused by stubbed adapters, the following steps should be taken to reseat adapters.

1. Re-seat the adapter so that the tailstock does not catch on the I/O shuttle slot. This can be done by applying pressure the top part of the bracket against the I/O Shuttle bulkhead as the adapter is being inserted into its PCIe slot.

2. Follow the procedure below to check for stubbed adapters.

3. It may be more convenient to remove the power supplies and install adapters with the I/O Shuttle raised because the adapter bracket tailstock is then visible as it slides into I/O Shuttle slot and it is easier to determine the minor adjustments needed to position the adapter correctly to avoid stubbing. If this method is chosen, care should be taken to insure the I/O Shuttle is fully raised with the handle in the full upright position before installing adapters. Care should also be taken not to exert too much downward pressure when installing an adapter to cause the I/O Shuttle to lower unexpectedly. A second person may be enlisted to hold the I/O Shuttle handle in the upright position to further insure the I/O Shuttle does not lower inadvertently.

Note: Complete removal of the I/O Shuttle from the server chassis is not required and should not be done for the purposes of this tip.

4. The top of the adapter bracket should sit flush down on the I/O Shuttle chassis. It should not be raised by 1 MM or more. If the adapter is stubbed, it may be raised by 2 or 3 MM, and the symptoms described above may occur when the server initializes, during operating system load, or during runtime.

5. After the adapter is installed, insure that the adapter printed circuit board or its components do not contact or slides between the fins of either of the two large IOH chip heat sinks that are adjacent to the PCIe slots on the I/O board. If contact exists between the adapter and the heat sinks, this indicates that the adapter may be stubbed, and there may be unexpected results. Reseat the adapter to avoid contact between the adapter and the heat sinks.

Additional information

This tip was significantly updated in November, 2010, because new information became available regarding unseated PCIe adapters in the field. Adapter stubbing was identified and confirmed to be the root cause for reported unseated adapters.

If the adapters are not stubbed, then the original adapter retention latches prevent any vertical movement, i.e. unseating, even though the adapters may wobble from side-to-side. Additional, more strenuous vibration and drop tests have been performed with the original adapter retention latches to confirm their designed functionality that prevents adapters from becoming unseated.

Ensure that all unpopulated slots have PCI blanks installed. PCIe adapters must be secured by neighboring brackets, else they will experience lateral movement. If the neighbor slots are unpopulated, then the unpopulated slot must have a blank bracket to provide lateral support to neighboring adapters or blanks.

Proper cable management must always be followed for secure adapter installation. Follow the recommended procedures in the Installation and User's Guide and the Rack Installation Instructions. Ensure that the Cable Management Arm is properly installed, and that there is sufficient cable slack when the server is fully extended from the rack.

Prior to installing cables in a new installation, all PCIe adapters should be checked to insure proper seating in their connector slots as described in the "Installing a PCI Express Adapter" section of the System x3850 X5 Installation and User's Guide.

First, open the Adapter Retention Latch, and make note of the "pin," as shown in the picture below:

Ensure that the adapter is properly seated. If in doubt, then re-seat the adapter, then close the adapter retention latch. The latch pin should fit securely within the adapter so that the latch is flush with the adapter as shown below:

If the latch pin is not properly inserted in the adapter, then the latch will be skewed as shown in the picture below, and the adapter will be loose.



Field Replaceable Unit (FRU) kit part number 59Y4983 contains a set of improved adapter retention latches, which reduce the possibility of lateral adapter movement, but do not prevent or automatically fix stubbed adapters. This FRU kit should be used only to repair adapter retention latches that are visibly broken. Proactive replacement of adapter retention latches is not supported.