Technote (troubleshooting)
Problem(Abstract)
This document introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when configuring and using the IBM System networking switch. This guide provides basic troubleshooting and log collecting guidelines in a switch crash scenario
Diagnosing the problem
Introduction
This document introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when configuring and using the IBM System networking switch. This guide provides basic troubleshooting and log collecting guidelines in a switch crash scenario
1. Troubleshooting Basics Overview
2. Troubleshooting and log collecting on a Switch Crash
A. ISCLI Mode
· Check the reason for Crash
· Check the memory statistics
· Check the CPU utilization
· Report the crash to IBM support
B. IBMNOS Mode
· Check the reason for Crash
· Check the memory statistics
· Check the CPU utilization
· Report the crash to IBM support
C. BBI(Browser based Interface) Mode
3. Best Practices
4. Generic Log collecting steps using below Terminal Emulators
· Putty
· Tera Term
· Secure CRT
1. Troubleshooting Basics Overview
The following are the basic steps for troubleshooting:
Step 1 Gather information that defines the specific symptoms.
Step 2 Identify all potential problems that could be causing the symptoms.
Step 3 Systematically eliminate each potential problem (from most likely to least likely)
To identify the possible problems, understand the overall configuration and have a good understanding of your end to end setup.
2. Troubleshooting and log collecting on a Switch Crash
A. ISCLI Mode
When a switch crashes, the cause might be from the failure of a process, and results in a reload of the switch. A crash is usually recorded on the switch.
· When you are logging into the switch you can see a banner message, which would indicate a system dump exists
Select Command Line Interface mode (ibmnos-cli/iscli): iscli
------------------------------------------------------------------
NOTE: System dump exists in FLASH.
The dump was saved at 9:55:15 Sun Jan 6, 2013.
Use 'show flash-dump-uuencode' or 'copy flash-dump' to
extract the dumps for analysis and 'clear flash-dump' to
clear the FLASH region.
------------------------------------------------------------------
switch>en
switch#
· Check the reason for Crash
· Use the sh sys-info command to display the reason of the crash
Note: The output for this command would be slightly different for an IBM embedded switch
switch#sh sys-info
System Information at 4:03:15 Mon Jan 7, 2013
Time zone: No timezone configured
Daylight Savings Time Status: Disabled
IBM Networking Operating System RackSwitch G8000
Switch has been up for 0 days, 18 hours, 8 minutes and 1 second.
Last boot: 9:55:15 Sun Jan 6, 2013 (software PANIC)
MAC address: 08:17:f4:c2:48:00 IP (If 1) address: 192.168.1.211
Hardware Revision: *
Board Revision:
Switch Serial No: CH4131004F
Spare Part No: BAC-00017-00
Manufacturing date: 11/22
Software Version 6.8.6.0 (FLASH image1), active configuration.
Fans are in Forward AirFlow, Warning at 55 C and Recover at 80 C
Temperature Sensor 1: 27.0 C
Temperature Sensor 2: 30.0 C
Temperature Sensor 3: 34.0 C
Temperature Sensor 4: 36.0 C
Speed of Fan 1: 16265 RPM (255 PWM)
Speed of Fan 2: 16023 RPM (255 PWM)
Speed of Fan 3: 16265 RPM (255 PWM)
Speed of Fan 4: 16216 RPM (255 PWM)
Speed of Fan 5: 15882 RPM (255 PWM)
State of Power Supply 1: Off
State of Power Supply 2: On
· Use the copy flash-dump tftp to export the flash dump to a tftp server
#copy flash-dump tftp
Address or name of remote host: a.b.c.d
Destination file name: crashfile
FLASH dump successfully tftp'd to a.b.c.d:crashfile
· Use the sh flash-dump-uuencode and log the session in absence of tftp server
#sh flash-dump-uuencode
Confirm proceeding with large dump (execution
time can exceed 20 minutes!) [y/n]:
· Obtain the timestamp of the crash with the show logging command, review the events that happened just before the crash.
switch# show logging
Jan 6 9:54:19 hostname ERROR mgmt: SIGSEGV(11) in Non-STEM pthread(tid=245)
Jan 6 9:54:25 hostname NOTICE system: System Reset from software panic
Jan 6 9:55:26 hostname NOTICE system: rebooted (software PANIC)
Jan 6 9:55:26 hostname INFO system: booted version 6.8.6 from FLASH image2, active configuration
· Check the memory statistics
· Use sh mp memory or sh processes memory to capture the memory consumption, when you see the MemFree decreasing consistently it indicates a memory leak, you can look allocs and frees memory for respective processes
#sh mp memory
------------------------------------------------------------------
Memory utilization:
MemTotal: 1033692 kB
MemFree: 785504 kB
Buffers: 172 kB
Cached: 57608 kB
SwapCached: 0 kB
Active: 145164 kB
Inactive: 13680 kB
Active(anon): 140300 kB
Inactive(anon): 0 kB
Active(file): 4864 kB
Inactive(file): 13680 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 262144 kB
HighFree: 86160 kB
LowTotal: 771548 kB
LowFree: 699344 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 24 kB
Writeback: 0 kB
AnonPages: 101064 kB
Mapped: 13480 kB
Shmem: 39236 kB
Slab: 3696 kB
SReclaimable: 292 kB
SUnreclaim: 3404 kB
KernelStack: 344 kB
PageTables: 488 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 516844 kB
Committed_AS: 203380 kB
VmallocTotal: 241584 kB
VmallocUsed: 34316 kB
VmallocChunk: 206084 kB
Percentage used 24
Memory tracing: enabled
Extended Memory tracing: disabled
High-water monitoring: enabled
Memory high-water: 28 percent (at 157 seconds from boot)
Memory stats:
allocs: 44800317
frees: 44794094
current: 6223
alloc fails: 0
STEM thread memory stats:
thid name allocs frees current * largest
0 INIT 5102 486 33636138 7680000
1 STEM 0 0 0 0
2 STP 7 0 18551856 7511616
3 MFDB 2 0 12 6
4 TND 3 0 175 140
5 CONS 106668 106661 774708419 2097152
6 TNET 3427 3430 18256384 4194304
7 TNET 1 0 6119 6119
8 TNET 1 0 6119 6119
9 TNET 1 0 6119 6119
10 LOG 57 56 12848003 262233
11 TRAP 952170 952170 82730714 544
13 NTP 0 0 0 0
14 RMON 0 0 0 0
17 IP 7177 7142 623676 5164
18 RIP 0 0 0 0
19 AGR 19639 18587 4692842 6119
20 EPI 0 0 0 0
22 PORT 46 0 37168 808
24 BGP 0 2 0 0
31 MGMT 24 20 3024 340
34 OSPF 0 4 0 0
36 VRRP 1 0 16 16
37 SNMP 245406 245400 18956447 6119
38 SNMP 116 115 54743894 4194304
40 SSHD 94 0 3834 340
43 DT1X 0 0 0 0
57 TEAM 0 0 0 0
58 I2C 27 6 1090 140
59 LACP 0 0 0 0
60 SFP 0 0 0 0
63 L3HS 0 0 0 0
71 SWKY 0 0 0 0
77 HLNK 0 0 0 0
79 LLDP 43405814 43352730 295080050 3072
80 IPV6 12 0 432 36
81 RTM6 0 0 0 0
82 PNG6 0 0 0 0
84 OSP3 0 0 0 0
87 VMAC 0 0 0 0
88 MEMM 0 0 0 0
90 UDLD 0 0 0 0
91 FCOE 0 0 0 0
93 SFLO 0 0 0 0
94 PROX 0 0 0 0
95 OAM 0 0 0 0
96 PIM 0 0 0 0
99 DHV6 2 0 38 24
100 DCBX 53324 106152 6718824 126
102 NBOO 1 0 6119 6119
105 VLAG 0 0 0 0
108 MLD6 0 0 0 0
110 ETMR 0 0 0 0
111 IKE2 0 12 0 0
116 UPTM 0 0 0 0
118 ECPT 3 0 168532 168000
119 ECPR 0 0 0 0
121 VFDB 0 3 0 0
124 HIST 0 0 0 0
125 SLP 19 4 266729 262140
128 IGMP 0 0 0 0
129 NORM 0 0 0 0
130 DONE 1273 1214 5364963 2097152
Total 44800417 44794194 1327417736
Non-STEM allocs 0
Non-STEM frees 0
Overhead 2132
· Check the CPU utilization
· Use sh processes cpu to review the cpu being utilized for respective process
#sh processes cpu
---------------------------------------------------------------------
CPU Utilization at 2:27:20 Mon Feb 4, 2013
Total CPU Utilization: For 1 second: 1.20%
For 5 second: 1.53%
For 1 minute: 1.21%
For 5 minute: 1.79%
Highest CPU Utilization: thread 11 (TRAP) at 23:59:03 Fri Feb 1, 2013
---------------------------------------------------------------------
Thread Thread Utilization Status
ID Name 1sec 5sec 1Min 5Min
---------------------------------------------------------------------
1 STEM 0.00% 0.00% 0.00% 0.00% idle
2 STP 0.98% 0.80% 0.83% 0.93% idle
3 MFDB 0.00% 0.00% 0.00% 0.00% idle
4 TND 0.00% 0.00% 0.00% 0.00% idle
5 CONS 0.03% 0.02% 0.00% 0.00% running
6 TNET 0.00% 0.00% 0.00% 0.00% idle
7 TNET 0.00% 0.00% 0.00% 0.00% idle
8 TNET 0.00% 0.00% 0.00% 0.00% idle
9 TNET 0.00% 0.00% 0.00% 0.00% idle
10 LOG 0.00% 0.00% 0.00% 0.00% idle
11 TRAP 0.00% 0.00% 0.00% 0.02% idle
13 NTP 0.00% 0.00% 0.00% 0.00% idle
14 RMON 0.02% 0.02% 0.02% 0.02% idle
17 IP 0.00% 0.01% 0.01% 0.01% idle
18 RIP 0.00% 0.00% 0.00% 0.00% idle
19 AGR 0.00% 0.00% 0.00% 0.00% idle
20 EPI 0.01% 0.01% 0.02% 0.03% idle
22 PORT 0.00% 0.00% 0.00% 0.00% idle
24 BGP 0.00% 0.00% 0.00% 0.00% idle
31 MGMT 0.01% 0.01% 0.02% 0.02% idle
34 OSPF 0.00% 0.00% 0.00% 0.00% idle
36 VRRP 0.00% 0.00% 0.00% 0.00% idle
37 SNMP 0.00% 0.00% 0.00% 0.00% idle
38 SNMP 0.00% 0.00% 0.00% 0.00% idle
40 SSHD 0.00% 0.00% 0.00% 0.00% idle
43 DT1X 0.00% 0.00% 0.01% 0.01% idle
57 TEAM 0.00% 0.00% 0.00% 0.00% idle
58 I2C 0.00% 0.36% 0.04% 0.02% idle
59 LACP 0.02% 0.02% 0.07% 0.07% idle
60 SFP 0.00% 0.00% 0.00% 0.00% idle
63 L3HS 0.00% 0.00% 0.00% 0.00% idle
71 SWKY 0.00% 0.00% 0.00% 0.00% idle
77 HLNK 0.00% 0.00% 0.00% 0.00% idle
79 LLDP 0.00% 0.12% 0.03% 0.05% idle
80 IPV6 0.00% 0.00% 0.00% 0.02% idle
81 RTM6 0.00% 0.00% 0.00% 0.00% idle
82 PNG6 0.00% 0.00% 0.00% 0.00% idle
84 OSP3 0.00% 0.00% 0.00% 0.00% idle
87 VMAC 0.00% 0.00% 0.00% 0.00% idle
88 MEMM 0.00% 0.00% 0.00% 0.00% idle
90 UDLD 0.00% 0.00% 0.00% 0.00% idle
91 FCOE 0.01% 0.00% 0.00% 0.00% idle
93 SFLO 0.00% 0.00% 0.00% 0.00% idle
94 PROX 0.00% 0.00% 0.00% 0.44% idle
95 OAM 0.00% 0.00% 0.00% 0.00% idle
96 PIM 0.00% 0.00% 0.00% 0.00% idle
99 DHV6 0.00% 0.00% 0.00% 0.00% idle
100 DCBX 0.00% 0.00% 0.00% 0.00% idle
102 NBOO 0.00% 0.00% 0.00% 0.00% idle
105 VLAG 0.00% 0.00% 0.00% 0.00% idle
108 MLD6 0.00% 0.00% 0.00% 0.00% idle
110 ETMR 0.11% 0.14% 0.15% 0.15% idle
111 IKE2 0.00% 0.00% 0.00% 0.00% idle
116 UPTM 0.00% 0.00% 0.00% 0.00% idle
118 ECPT 0.00% 0.00% 0.00% 0.00% idle
119 ECPR 0.00% 0.00% 0.00% 0.00% idle
121 VFDB 0.00% 0.00% 0.00% 0.00% idle
124 HIST 0.00% 0.00% 0.00% 0.00% idle
125 SLP 0.00% 0.00% 0.00% 0.00% idle
128 IGMP 0.00% 0.00% 0.00% 0.00% idle
129 NORM 0.00% 0.00% 0.00% 0.00% idle
130 DONE 0.00% 0.00% 0.00% 0.00% idle
---------------------------------------------------------------------
· Report the crash to IBM support
· If you would like to report the crash event to IBM support attach the following to the PMR
1. The Flash dump file captured
2. show tech-support
a. #sh tech-support
b. dumping all information, statistics, and configuration (y/n) ?
3. A port level topology Diagram
4. The configuration of the directly connected devices to the switch which witnessed the crash
B. IBMNOS Mode
When a switch crashes, the cause might be from the failure of a process, and results in a reload of the switch. A crash is usually recorded on the switch.
· When you are logging into the switch you can see a banner message, which would indicate a system dump exists
Select Command Line Interface mode (ibmnos-cli/iscli): ibmnos
------------------------------------------------------------------
NOTE: System dump exists in FLASH.
The dump was saved at 9:55:15 Sun Jan 6, 2013.
Use /maint/uudmp or /maint/ptdmp to extract the dumps for
analysis and /maint/cldmp to clear the FLASH region.
------------------------------------------------------------------
switch - Main#
· Check the reason for Crash
· Use the /info/sys/general command to display the reason of the crash
Note: The output for this command would be slightly different for an IBM embedded switch
switch - System# /info/sys/general
System Information at 4:03:15 Mon Jan 7, 2013
Time zone: No timezone configured
Daylight Savings Time Status: Disabled
IBM Networking Operating System RackSwitch G8000
Switch has been up for 0 days, 18 hours, 8 minutes and 1 second.
Last boot: 9:55:15 Sun Jan 6, 2013 (software PANIC)
MAC address: 08:17:f4:c2:48:00 IP (If 1) address: 192.168.1.211
Hardware Revision: *
Board Revision:
Switch Serial No: CH4131004F
Spare Part No: BAC-00017-00
Manufacturing date: 11/22
Software Version 6.8.6.0 (FLASH image1), active configuration.
Fans are in Forward AirFlow, Warning at 55 C and Recover at 80 C
Temperature Sensor 1: 27.0 C
Temperature Sensor 2: 30.0 C
Temperature Sensor 3: 34.0 C
Temperature Sensor 4: 36.0 C
Speed of Fan 1: 16265 RPM (255 PWM)
Speed of Fan 2: 16023 RPM (255 PWM)
Speed of Fan 3: 16265 RPM (255 PWM)
Speed of Fan 4: 16216 RPM (255 PWM)
Speed of Fan 5: 15882 RPM (255 PWM)
State of Power Supply 1: Off
State of Power Supply 2: On
· Use the /maint/ptdmp to export the flash dump to a tftp server
switch- Main # /maint/ptdmp
Address or name of remote host: a.b.c.d
Destination file name: crashfile
FLASH dump successfully tftp'd to a.b.c.d:crashfile
· Use the /maint/uudmp and log the session in absence of tftp server
switch- Main # /maint/uudmp
Confirm proceeding with large dump (execution
time can exceed 20 minutes!) [y/n]:
· Obtain the timestamp of the crash with the /info/sys/log command, review the events that happened just before the crash.
switch- Main# /info/sys/log
Jan 6 9:54:19 hostname ERROR mgmt: SIGSEGV(11) in Non-STEM pthread(tid=245)
Jan 6 9:54:25 hostname NOTICE system: System Reset from software panic
Jan 6 9:55:26 hostname NOTICE system: rebooted (software PANIC)
Jan 6 9:55:26 hostname INFO system: booted version 6.8.6 from FLASH image2, active configuration
· Check the memory statistics
· Use /stats/mp/mem to capture the memory consumption, when you see the MemFree decreasing consistently it indicates a memory leak, you can look allocs and frees memory for respective processes
switch- Main# /stats/mp/mem
------------------------------------------------------------------
Memory utilization:
MemTotal: 1033692 kB
MemFree: 785504 kB
Buffers: 172 kB
Cached: 57608 kB
SwapCached: 0 kB
Active: 145164 kB
Inactive: 13680 kB
Active(anon): 140300 kB
Inactive(anon): 0 kB
Active(file): 4864 kB
Inactive(file): 13680 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 262144 kB
HighFree: 86160 kB
LowTotal: 771548 kB
LowFree: 699344 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 24 kB
Writeback: 0 kB
AnonPages: 101064 kB
Mapped: 13480 kB
Shmem: 39236 kB
Slab: 3696 kB
SReclaimable: 292 kB
SUnreclaim: 3404 kB
KernelStack: 344 kB
PageTables: 488 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 516844 kB
Committed_AS: 203380 kB
VmallocTotal: 241584 kB
VmallocUsed: 34316 kB
VmallocChunk: 206084 kB
Percentage used 24
Memory tracing: enabled
Extended Memory tracing: disabled
High-water monitoring: enabled
Memory high-water: 28 percent (at 157 seconds from boot)
Memory stats:
allocs: 44800317
frees: 44794094
current: 6223
alloc fails: 0
STEM thread memory stats:
thid name allocs frees current * largest
0 INIT 5102 486 33636138 7680000
1 STEM 0 0 0 0
2 STP 7 0 18551856 7511616
3 MFDB 2 0 12 6
4 TND 3 0 175 140
5 CONS 106668 106661 774708419 2097152
6 TNET 3427 3430 18256384 4194304
7 TNET 1 0 6119 6119
8 TNET 1 0 6119 6119
9 TNET 1 0 6119 6119
10 LOG 57 56 12848003 262233
11 TRAP 952170 952170 82730714 544
13 NTP 0 0 0 0
14 RMON 0 0 0 0
17 IP 7177 7142 623676 5164
18 RIP 0 0 0 0
19 AGR 19639 18587 4692842 6119
20 EPI 0 0 0 0
22 PORT 46 0 37168 808
24 BGP 0 2 0 0
31 MGMT 24 20 3024 340
34 OSPF 0 4 0 0
36 VRRP 1 0 16 16
37 SNMP 245406 245400 18956447 6119
38 SNMP 116 115 54743894 4194304
40 SSHD 94 0 3834 340
43 DT1X 0 0 0 0
57 TEAM 0 0 0 0
58 I2C 27 6 1090 140
59 LACP 0 0 0 0
60 SFP 0 0 0 0
63 L3HS 0 0 0 0
71 SWKY 0 0 0 0
77 HLNK 0 0 0 0
79 LLDP 43405814 43352730 295080050 3072
80 IPV6 12 0 432 36
81 RTM6 0 0 0 0
82 PNG6 0 0 0 0
84 OSP3 0 0 0 0
87 VMAC 0 0 0 0
88 MEMM 0 0 0 0
90 UDLD 0 0 0 0
91 FCOE 0 0 0 0
93 SFLO 0 0 0 0
94 PROX 0 0 0 0
95 OAM 0 0 0 0
96 PIM 0 0 0 0
99 DHV6 2 0 38 24
100 DCBX 53324 106152 6718824 126
102 NBOO 1 0 6119 6119
105 VLAG 0 0 0 0
108 MLD6 0 0 0 0
110 ETMR 0 0 0 0
111 IKE2 0 12 0 0
116 UPTM 0 0 0 0
118 ECPT 3 0 168532 168000
119 ECPR 0 0 0 0
121 VFDB 0 3 0 0
124 HIST 0 0 0 0
125 SLP 19 4 266729 262140
128 IGMP 0 0 0 0
129 NORM 0 0 0 0
130 DONE 1273 1214 5364963 2097152
Total 44800417 44794194 1327417736
Non-STEM allocs 0
Non-STEM frees 0
Overhead 2132
· Check the CPU utilization
· Use /stats/mp/cpu to review the cpu being utilized for respective process
switch - Main# /stats/mp/cpu
---------------------------------------------------------------------
CPU Utilization at 2:27:20 Mon Feb 4, 2013
Total CPU Utilization: For 1 second: 1.20%
For 5 second: 1.53%
For 1 minute: 1.21%
For 5 minute: 1.79%
Highest CPU Utilization: thread 11 (TRAP) at 23:59:03 Fri Feb 1, 2013
---------------------------------------------------------------------
Thread Thread Utilization Status
ID Name 1sec 5sec 1Min 5Min
---------------------------------------------------------------------
1 STEM 0.00% 0.00% 0.00% 0.00% idle
2 STP 0.98% 0.80% 0.83% 0.93% idle
3 MFDB 0.00% 0.00% 0.00% 0.00% idle
4 TND 0.00% 0.00% 0.00% 0.00% idle
5 CONS 0.03% 0.02% 0.00% 0.00% running
6 TNET 0.00% 0.00% 0.00% 0.00% idle
7 TNET 0.00% 0.00% 0.00% 0.00% idle
8 TNET 0.00% 0.00% 0.00% 0.00% idle
9 TNET 0.00% 0.00% 0.00% 0.00% idle
10 LOG 0.00% 0.00% 0.00% 0.00% idle
11 TRAP 0.00% 0.00% 0.00% 0.02% idle
13 NTP 0.00% 0.00% 0.00% 0.00% idle
14 RMON 0.02% 0.02% 0.02% 0.02% idle
17 IP 0.00% 0.01% 0.01% 0.01% idle
18 RIP 0.00% 0.00% 0.00% 0.00% idle
19 AGR 0.00% 0.00% 0.00% 0.00% idle
20 EPI 0.01% 0.01% 0.02% 0.03% idle
22 PORT 0.00% 0.00% 0.00% 0.00% idle
24 BGP 0.00% 0.00% 0.00% 0.00% idle
31 MGMT 0.01% 0.01% 0.02% 0.02% idle
34 OSPF 0.00% 0.00% 0.00% 0.00% idle
36 VRRP 0.00% 0.00% 0.00% 0.00% idle
37 SNMP 0.00% 0.00% 0.00% 0.00% idle
38 SNMP 0.00% 0.00% 0.00% 0.00% idle
40 SSHD 0.00% 0.00% 0.00% 0.00% idle
43 DT1X 0.00% 0.00% 0.01% 0.01% idle
57 TEAM 0.00% 0.00% 0.00% 0.00% idle
58 I2C 0.00% 0.36% 0.04% 0.02% idle
59 LACP 0.02% 0.02% 0.07% 0.07% idle
60 SFP 0.00% 0.00% 0.00% 0.00% idle
63 L3HS 0.00% 0.00% 0.00% 0.00% idle
71 SWKY 0.00% 0.00% 0.00% 0.00% idle
77 HLNK 0.00% 0.00% 0.00% 0.00% idle
79 LLDP 0.00% 0.12% 0.03% 0.05% idle
80 IPV6 0.00% 0.00% 0.00% 0.02% idle
81 RTM6 0.00% 0.00% 0.00% 0.00% idle
82 PNG6 0.00% 0.00% 0.00% 0.00% idle
84 OSP3 0.00% 0.00% 0.00% 0.00% idle
87 VMAC 0.00% 0.00% 0.00% 0.00% idle
88 MEMM 0.00% 0.00% 0.00% 0.00% idle
90 UDLD 0.00% 0.00% 0.00% 0.00% idle
91 FCOE 0.01% 0.00% 0.00% 0.00% idle
93 SFLO 0.00% 0.00% 0.00% 0.00% idle
94 PROX 0.00% 0.00% 0.00% 0.44% idle
95 OAM 0.00% 0.00% 0.00% 0.00% idle
96 PIM 0.00% 0.00% 0.00% 0.00% idle
99 DHV6 0.00% 0.00% 0.00% 0.00% idle
100 DCBX 0.00% 0.00% 0.00% 0.00% idle
102 NBOO 0.00% 0.00% 0.00% 0.00% idle
105 VLAG 0.00% 0.00% 0.00% 0.00% idle
108 MLD6 0.00% 0.00% 0.00% 0.00% idle
110 ETMR 0.11% 0.14% 0.15% 0.15% idle
111 IKE2 0.00% 0.00% 0.00% 0.00% idle
116 UPTM 0.00% 0.00% 0.00% 0.00% idle
118 ECPT 0.00% 0.00% 0.00% 0.00% idle
119 ECPR 0.00% 0.00% 0.00% 0.00% idle
121 VFDB 0.00% 0.00% 0.00% 0.00% idle
124 HIST 0.00% 0.00% 0.00% 0.00% idle
125 SLP 0.00% 0.00% 0.00% 0.00% idle
128 IGMP 0.00% 0.00% 0.00% 0.00% idle
129 NORM 0.00% 0.00% 0.00% 0.00% idle
130 DONE 0.00% 0.00% 0.00% 0.00% idle
---------------------------------------------------------------------
· Report the crash to IBM support
· if you would like to report the crash event to IBM support attach the following to the PMR
1. The Flash dump file captured
2. Tech-support
a. # /maint/tsdmp
b. Confirm dumping all information, statistics, and configuration [y/n]: y
3. The configuration of the directly connected devices to the switch which witnessed the crash
4. A port level topology Diagram
C. BBI(Browser Based Interface)
When a switch crashes, the cause might be from the failure of a process, and results in a reload of the switch. A crash is usually recorded on the switch.
· When you are logging into the switch you can see a banner message, which would indicate a system dump exists
· Navigate to IBM Switch>>Sytem>>Config/Image Control on the left hand side. To export the flash dump to a tftp server give a name to file and hit “Put FLASH Dump” button, make sure the transfer was successful
Status of Previous Transfer 02/13/13 01:05:38 Put main FLASH dump a.b.c.d:IBM Switch Flash dump – Successful
· Navigate to “Show Log” tab to capture the timestamp, review the events that happened just before the crash.
Jan 6 9:54:19 hostname ERROR mgmt: SIGSEGV(11) in Non-STEM pthread(tid=245)
Jan 6 9:54:25 hostname NOTICE system: System Reset from software panic
Jan 6 9:55:26 hostname NOTICE system: rebooted (software PANIC)
Jan 6 9:55:26 hostname INFO system: booted version 6.8.6 from FLASH image2, active configuration
· if you would like to report the crash event to IBM support attach the following to the PMR
1. The Flash dump file captured
2. Navigate to IBM Switch>>Sytem>>Config/Image Control on the left hand side. To export the tech support dump to a tftp server give a name to file and hit “Put TS Dump” button, make sure the transfer was successful.
Status of Previous Transfer 02/13/13 00:54:03 Put TS dump a.b.c.d:IBMSwitch tech Suppot - Successful
3. A port level topology diagram
4. The configuration of the directly connected devices to the switch which witnessed the crash
3. Best Practices
Best practices are the recommended steps you should take to ensure the proper operation of your switch.
• Maintain a latest IBMNOS release across all your IBM switches.
• Refer to the release notes for your IBM switches for the latest features, limitations, and caveats.
• Enable system message logging.
• Troubleshoot any new configuration changes after implementing the change.
4. Generic Log collecting steps using below Terminal Emulators
· Putty
To collect a log using PuTTY:
1. Start the PuTTY program (downloaded from http://www.putty.org)
2. Connect to the switch using Telnet or SSH.
3. Once connected, enable the session logging:
· Right click on the title bar
· Select the “Change Settings…” command
· Select “Logging” from the “Session” Category
· Select “Printable output”
· Under “Log file name” enter the name of the output file (i.e. Switch.log)
· Select the “Browse” button and select the location for the log file to be written to
· Hit the “Apply” button
4. Issue the appropriate CLI command to dump the switch technical information
· ISCLI: #show tech ibmnosCLI: # /maint/tsdmp
· Tera Term
To collect a log using Tera Term:
1. Start the Tera Term program (download for free from: http://ttssh2.sourceforge.jp)
2. Connect to the switch using Telnet or SSH.
3. Once connected, enable the session logging:
· Click on the “File” item in the menu bar
· Select the “Log…” item
· In the “Tera Term: Log” pop-up, select the file name to log to
· Hit the “Save” button
4. Issue the appropriate CLI command to dump the switch technical information:
· ISCLI: show tech ibmnosCLI: /maint/tsdmp
· SecureCRT
To collect a log using SecureCRT
1. Start the Tera Term program (download from http://www.vandyke.com/download/securecrt/download.html)
2. Connect to the switch using Telnet or SSH.
3. Once connected, enable the session logging:
· Click on the “File” item in the menu bar
· Select the “Log Session” item
· In select Log file pop up select the file name to log to
· Hit the “Save” button
4. Issue the appropriate CLI command to dump the switch technical information:
· ISCLI: show tech ibmnosCLI: /maint/tsdmp
| Segment | Product | Component | Platform | Version | Edition |
|---|---|---|---|---|---|
| Computer Networking Hardware | IBM RackSwitch G8052 | Platform Independent | Version Independent | ||
| Computer Networking Hardware | IBM RackSwitch G8124 | Platform Independent | Version Independent | ||
| Computer Networking Hardware | IBM RackSwitch G8264 | Platform Independent | Version Independent | ||
| Computer Networking Hardware | IBM RackSwitch G8264CS | Platform Independent | Version Independent | ||
| Computer Networking Hardware | IBM RackSwitch G8264T | Platform Independent | Version Independent | ||
| Computer Networking Hardware | IBM RackSwitch G8316 | Platform Independent | Version Independent | ||
| Computer Networking Hardware | IBM RackSwitch G7028 | Platform Independent | Version Independent |