Oracle Clusterware Diagnosis - Expert Oracle RAC Performance Diagnostics and Tuning - page 539

Step 3
Since the olr.loc file still exists, the next step is to look for any entries in the client directory. Before the actual start
of the OHASD process, the init.d tries to read the OLR file as a client process. During this step, it opens a log file in
the $GRID_HOME/log/ <node name>/ client / folder. Verifying the client log files indicates that the client process was
unable to locate the OLR file.

[root@ssky1l4p2 client]# more crsctl_29422.log
Oracle Database 12c Clusterware Release 12.1.0.1.0 - Production Copyright 1996, 2013 Oracle.
All rights reserved.
2014-09-06 16:33:32.103: [ GIPC][2565203488] gipcCheckInitialization: possible incompatible
non-threaded init from [prom.c : 730], original from [clsCrsctlUtil.cpp : 2934]
2014-09-06 16:33:32.105: [ OCRMSG][2565203488]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2014-09-06 16:33:32.105: [ OCRMSG][2565203488]GIPC error [29] msg [gipcretConnectionRefused]
2014-09-06 16:33:32.105: [ OCRMSG][2565203488]prom_connect: error while waiting for connection
complete [24]
2014-09-06 16:33:32.105: [ OCROSD][2565203488]utopen:6m': failed in stat OCR file/disk
/u01/app/12.1.0.1/grid/cdata/ssky1l4p2.olr, errno=2, os err string=No such file or directory
2014-09-06 16:33:32.105: [ OCROSD][2565203488]utopen:7: failed to open any OCR file/disk,
errno=2, os err string=No such file or directory
2014-09-06 16:33:32.105: [ OCRRAW][2565203488]proprinit: Could not open raw device
2014-09-06 16:33:32.105: [ default][2565203488]a_init:7!: Backend init unsuccessful : [26]
[root@ssky1l4p2 client]#

Step 4
As seen in Step 3, it's clear that the clusterware stack on the ssky1l4p2 node would not start because of an issue with
the OLR file.
Based on the information obtained from Step 2, verify whether the OLR file exists in the specified location. From
the following command, it's determined that the OLR file is missing from the location.

[root@ssky1l4p2 client]# ls -ltr /u01/app/12.1.0.1/grid/cdata/ssky1l4p2.olr

ls: /u01/app/12.1.0.1/grid/cdata/ssky1l4p2.olr: No such file or directory

Step 5
Based on the information collected thus far, it's understood that the clusterware would not start because of the
missing OLR file. At this stage, ideally, it would be good if a backup of the OLR file were available. The backups are
verified using the following command:

ocrconfig -local -showbackup

No backups of the OLR file are found. Unlike the OCR file, the clusterware does not back up the OLR file on a
regular basis. In this situation, the only backup of the OLR file available was from the time of the installation; which is
outdated and does not contain the updates to the clusterware performed on the node.

■ Note it's critical to include the backup of the Olr file from each node in the cluster in the backup strategy.