Oracle Clusterware Diagnosis - Expert Oracle RAC Performance Diagnostics and Tuning - page 567

■ Occasionally, once the restore or import is completed, clusterware could go into a panic mode and reboot the
server. this is normal behavior, and it happens when Crs has not completely shut down and has a hanging daemon on
one of the servers.

Note

This should restart the clusterware and all of the applications managed by the CRS. Once the clusterware
components have been verified and the database has been checked, it would be a good practice to export or manually
back up the OCR and archive it as a good copy.

EVMD Verification
EVMD plays a very important function in the RAC architecture; it sends and receives actions regarding resource state
changes to and from all other nodes in the cluster. To determine whether the EVMD for a node can send and receive
messages from other nodes, the set of tests outlined here should help.
Activities of the EVMD could be verified using the evmwatch monitor utility. The evmwatch monitor is a
background process that constantly watches for actions, which are then parsed to the evmshow utility for formatting
and display.
For example, evmwatch -A -t "@timestamp @@" will monitor for actions sent and received, and such
information will be displayed on standard output. The display in this example is from evmshow; however, it is
automatically started when the -A switch is specified. @timestamp will list the date and time at which actions are sent
and received by the node.
Additional details regarding the actions received or sent could also be obtained using additional switches.
For example, evmwatch -A -t "@timestamp @priority @" will return the designated priority of the event received,
and the third “ @name ” will display the name (shown in the following output) of the service, resource, or application.
See the following:

[root@PRDDB1 home]# $GRID_HOME/bin/evmwatch -A -t "@timestamp @priority @name"
"05-Apr-2011 22:00:51 200 sys.ora.clu.cluster_evt"
"05-Apr-2011 22:01:22 200 sys.ora.clu.cluster_evt"
"05-Apr-2011 22:01:52 200 sys.ora.clu.cluster_evt"
"05-Apr-2011 22:02:22 200 sys.ora.clu.cluster_evt"
"05-Apr-2011 22:02:53 200 sys.ora.clu.cluster_evt"

This output illustrates two types of actions sent and received. An imcheck action is sent to determine the state of
the resources defined in the OCR, and a subsequent response is received that provides the current state ( imup ) of the
resource (similar to a reply message for the initial verification request). All action/responses are user-defined (user is
identified by ora ) HA services (identified by ha ), and all communications are performed at priority 200 .
Other types of actions sent and received by the EVMD are listed in Table 16-5 .