Oracle Clusterware Diagnosis - Expert Oracle RAC Performance Diagnostics and Tuning - page 548

Resources that are managed by the CRS include global service daemon (GSD), Oracle notification services (ONS)
daemon, virtual Internet Protocol (VIP), listeners, databases, instances, and services. Resources are grouped based
on the level at which they apply to the environment. For example, some of these resources are referred to as node
applications ( nodeapps ) and pertain to individual nodes in the cluster. Nodeapps are needed on a per-node basis
independent of the number of databases on the node. GSD, ONS, VIPs, and listeners are nodeapps . Nodeapps are
created and registered with the OCR during installation of the Oracle Clusterware. Listener, database, and service
resources are formed during the database creation process.
Cluster/Node Failures/Evictions
Like any hardware and software component, clusters, nodes in the cluster, or clusterware components can fail due
to various reasons. For example, when a specific node does not have sufficient resources (CPU starvation) for the
clusterware components to complete its activity, it could be evicted from the cluster. There are several reasons for
evictions and reboots to occur in an Oracle clusterware.
Node eviction due to missing NHB
When the CSS on one of the nodes sends a heartbeat message to another node indicating its health, a similar
heartbeat is received back from the other node. If this does not happen in the set misscount interval, a potential node
eviction is possible. The following output (formatted to fit the page) is extracted from the cssd.log file. The misscount
entries in the log file are visible when the CSSD logging is set to level 3 and is searching for “ PolllingThread ” string.
cat ocssd.log | grep PollingThread.