Troubleshooting Router Crashes - Cisco
When we refer to a "system crash", we mean a situation where the system
has detected an unrecoverable error, and has restarted itself.
The errors that cause crashes are typically detected by processor
hardware, which automatically branches to special error handling code in the
ROM monitor. The ROM monitor identifies the error, prints a message, saves
information about the failure, and restarts the system.
There are no specific requirements for this document.
This document is not restricted to specific software and hardware
The information in this document was created from the devices in a
specific lab environment. All of the devices used in this document started with
a cleared (default) configuration. If your network is live, make sure that you
understand the potential impact of any command.
For more information on document conventions, see the
When the router crashes, it is extremely important to gather as much
information as possible about the crash before you manually reload or
power-cycle the router. All information about the crash, except that which has
been successfully stored in the crashinfo file, is lost after a manual reload
or power-cycle. The following outputs give some indication and information on
the crash.
If you have the output of a show version,
show stacks, show
context, or show tech support command
from your Cisco device, you can use
to display potential issues and fixes. To use , you must be a
customer, be logged in, and have JavaScript enabled.
Description
show version
This command first appeared in Cisco IOS& Software Release10.0.
The show version EXEC command displays the
configuration of the system hardware, the software version, the names and
sources of configuration files and software images, the router uptime, and
information on how the system has been restarted.
IMPORTANT: If the router is reloaded after the crash (for
example, if it has been power-cycled or the reload
command has been issued), this information will be lost, so try to collect it
before reloading!
show stacks
This command first appeared in Cisco IOS Software Release 10.0.
The show stacks EXEC command is used to monitor the
stack usage of processes and interrupt routines. The show
stacks output is one of the most indispensable sources of
information to collect when the router crashes.
IMPORTANT: If the router is reloaded after the crash (for
example, through power-cycle or the reload command),
this information will be lost so try to collect it before reloading!
show context
This command first appeared in Cisco IOS Software Release 10.3.
The show context EXEC command is used to display
information stored in nonvolatile RAM (NVRAM) when an exception occurs. Context
information is specific to processors and architectures, whereas software
version and uptime information are not. Context information for different
router types could therefore differ. The output displayed from the
show context command includes:
the reason for the system reboot.
stack trace.
software version.
signal number, code, and router uptime
information.
all the register contents at the time of the
show tech-support
This command first appeared in Cisco IOS Software Release 11.2.
This command is useful in collecting general information about the router when
you report a problem. It includes:
show version
show running-config
show stacks
show interface
show controller
show process cpu
show process memory
show buffers
console log
If you are connected to the console of the router at the time
of the crash, you will see something like this during the crash:
*** System received a Software forced crash ***
signal= 0x17, code= 0x24, context= 0x
PC = 0x602e59dc, Cause = 0x4020, Status Reg = 0x
DCL Masked Interrupt Register = 0x
DCL Interrupt Value Register = 0x
MEMD Int 6 Status Register = 0x
Keep this information and the logs before it. Once the router
comes up again, do not forget to get the show stacks
If the router is set up to send logs to a syslog server, you
will see some information on what happened before the crash on the syslog
server. However, when the router is crashing, it may not be able to send the
most useful information to this syslog server. So most of the time,
syslog output is not very useful for troubleshooting
The crashinfo file is a collection of useful information
related to the current crash, stored in bootflash or flash memory. When a
router crashes due to data or stack corruption, more reload information is
needed to debug this type of crash than just the output from the normal
show stacks command.
The crashinfo is written by default to
bootflash:crashinfo on the Cisco 12000 Gigabit
Router Processor (GRP), the Cisco 7000 and 7500 Route Switch Processors (RSPs),
and the Cisco 7200 series routers. For the Cisco 7500 Versatile Interface
Processor 2 (VIP2), this file is stored by default to
bootflash:vip2_slot_no_crashinfo
is the VIP2
slot number. For the Cisco 7000 Route Processor (RP), the file is stored by
default to flash:crashinfo.
For more details, see.
A core dump is a full copy of the router's memory image. This
information is not necessary for troubleshooting most types of crashes, but it
is highly recommended when filing a new bug. You may need to enable some debugs
to add more information into the core dump such as debug sanity, scheduler
heapcheck process, and memory check-interval 1.
For more details, see
rom monitor
The router might end up in ROM monitor after a crash when its
config-register setting ends with 0. If the processor is a 68k, the prompt will
be "&". You can get the stack trace with the k
command. If the processor is a reduced instruction set computing (RISC), the
prompt will be "rommon 1&". Get the output of stack
50 or show context.
The show version and show
stacks commands provide you with output that gives you an
indication of the type of the crash that occurred, such as bus error, or
software forced crash. You can also get crash type information from the
crashinfo and show context
commands. For some later Cisco IOS Software versions, the crash
reasons are not clearly indicated (for example, you see "Signal = x" where x is
a number). Refer to
to translate this number into
something meaningful. For example, "Signal = 23" translates to a software
forced crash. Follow these links to troubleshoot the specific type of crash
your router is experiencing:
Sometimes, only a specific router module crashes, and not the router
itself. Here are some documents that describe how to troubleshoot crashes on
some router modules:
Router#show version
Cisco Internetwork Operating System Software
IOS (tm) RSP Software (RSP-PV-M), Version 12.0(10.6)ST, EARLY DEPLOYMENT
MAINTENANCE INTERIM SOFTWARE
Copyright (c)
by cisco Systems, Inc.
Compiled Fri 23-Jun-00 16:02 by richv
Image text-base: 0x, data-base: 0x60D96000
ROM: System Bootstrap, Version 12.0(725), DEVELOPMENT SOFTWARE
BOOTFLASH: RSP Software (RSP-BOOT-M), Version 12.0(9)S, EARLY DEPLOYMENT
RELEASE SOFTWARE (fc1)
Router uptime is 20 hours, 56 minutes
System returned to ROM by error - a Software forced crash, PC 0x60287EE8
System image file is "slot0:rsp-pv-mz.120-10.6.ST"
cisco RSP8 (R7000) processor with 16K bytes of memory.
R7000 CPU at 250Mhz, Implementation 39, Rev 1.0, 256KB L2, 2048KB L3 Cache
Last reset from power-on
G.703/E1 software, Version 1.0.
G.703/JT2 software, Version 1.0.
X.25 software, Version 3.0.0.
Chassis Interface.
1 EIP controller (6 Ethernet).
1 VIP2 R5K controller (1 FastEthernet)(2 HSSI).
6 Ethernet/IEEE 802.3 interface(s)
1 FastEthernet/IEEE 802.3 interface(s)
2 HSSI network interface(s)
2043K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
16384K bytes of Flash internal SIMM (Sector size 256K).
No slave installed in slot 7.
Configuration register is 0x2102
Router#show stacks
Minimum process stacks:
CEF Reloader
RADIUS INITCONFIG
MDFS Reload
RSP memory size check
DHCP Client
Interrupt level stacks:
Called Unused/Size
Network Interrupt
Network Status Interrupt
OIR interrupt
PCMCIA Interrupt
Console Uart
Error Interrupt
NMI Interrupt Handler
System was restarted by error - a Software forced crash, PC 0x602DE884 at 05:07:31
UTC Thu Sep 16 1999
RSP Software (RSP-JSV-M), Version 12.0(7)T,
RELEASE SOFTWARE (fc2)
Compiled Mon 06-Dec-99 19:40 by phanguye
Image text-base: 0x, database: 0x
Stack trace from system failure:
FP: 0x61F73C30, RA: 0x602DE884
FP: 0x61F73C30, RA: 0x6030D29C
FP: 0x61F73D88, RA: 0x6025E96C
FP: 0x61F73DD0, RA: 0x
FP: 0x61F73E30, RA: 0x602B94BC
FP: 0x61F73E48, RA: 0x602B94A8
When a crashinfo is available in bootflash, the following is displayed
at the end of the show stacks command:
***************************************************
******* Information of Last System Crash **********
***************************************************
Using bootflash:crashinfo_850. 2000
CMD: 'sh int fas' 03:23:41 UTC Thu Mar 2 2000
CMD: 'sh int fastEthernet 6/0/0' 03:23:44 UTC Thu Mar 2 2000
CMD: 'conf t' 03:23:56 UTC Thu Mar 2 2000
CMD: 'no ip cef di' 03:23:58 UTC Thu Mar 2 2000
CMD: 'no ip cef distributed ' 03:23:58 UTC Thu Mar 2 2000
Router#show context
System was restarted by error - a Software forced crash, PC 0x602DE884 at
05:07:31 UTC Thu Sep 16 1999
RSP Software (RSP-JSV-M), Version 12.0(7)T,
RELEASE SOFTWARE (fc2)
Compiled Mon 06-DEC-99 19:40 by phanguye
Image text-base: 0x, database: 0x
Stack trace from system failure:
FP: 0x61F73C30, RA: 0x602DE884
FP: 0x61F73C30, RA: 0x6030D29C
FP: 0x61F73D88, RA: 0x6025E96C
FP: 0x61F73DD0, RA: 0x
FP: 0x61F73E30, RA: 0x602B94BC
FP: 0x61F73E48, RA: 0x602B94A8
Fault History Buffer:
RSP Software (RSP-JSV-M), Version 12.0(7)T,
RELEASE SOFTWARE (fc2)
Compiled Mon 06-DEC-99 19:40 by phanguye
Signal = 23, Code = 0x24, Uptime 3w0d
$0 : , AT : 619A0000, v0 : , v1 :
a0 : , a1 : 61A309A4, a2 : , a3 :
t0 : 61F6CD80, t1 : 8000FD88, t2 : , t3 : FFFF00FF
t4 : , t5 : 3E840024, t6 : , t7 :
s0 : 0000003C, s1 : , s2 : , s3 : 61F73C48
s4 : , s5 : 61993A10, s6 : 61982D00, s7 :
t8 : 0000327A, t9 : , k0 : 61E48C4C, k1 : 602E7748
gp : , sp : 61F73C30, s8 : , ra : 6030D29C
EPC : 602DE884, SREG : , Cause :
Error EPC : BFC00000, BadVaddr : 40231FFE
If you still need assistance after following the
troubleshooting steps above, and want to open a service request with the Cisco
TAC, be sure to include the following information for troubleshooting a router
Troubleshooting performed before opening the service
show technical-support output (if
possible, in enable mode).
show log output or console
captures, if available.
(if present, and not already included in the show
technical-support output).
show region output (if not already
included in the show technical-support
Please attach the collected data to your service request in
non-zipped, plain text format (.txt). You can attach information to your
service request by uploading it using the
( customers only)
If you cannot access the Service Request tool, you can attach the relevant
information to your service request by sending it to
case number in the subject line of your message.
Note:&Please do not manually reload or power-cycle the router
before collecting the above information unless required to troubleshoot a
router crash, as this can cause important information to be lost that is needed
for determining the root cause of the problem.
Was this document helpful?
Thank you for your feedback.
(Requires a .)
Related Cisco Support Community Discussions
is a forum for you to ask and answer questions, share suggestions, and
collaborate with your peers.
for information on conventions used in this document.
Document ID: 7900