Network Administration - Oracle Exadata Recipes: A Problem-Solution Approach - page 297

Solution
In this recipe, you will learn how to list your InfiniBand network details on the compute nodes, understand how the
InfiniBand network is built, and validate that your Oracle RAC interconnect is using the InfiniBand network.
On Exadata, the Oracle RAC private interconnect runs on the InfiniBand network, which is the same network
that Exadata's storage network runs on. This InfiniBand private interconnect runs on a bonded InfiniBand interface,
bondib0 . To validate this, log in to your compute server as root and run ifconfig on the bonded InfiniBand interface:

[root@cm01dbm01 ~]# ifconfig bondib0
bondib0 Link encap:InfiniBand HWaddr
80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.10.1 Bcast:192.168.11.255 Mask:255.255.252.0
... Additional interface details omitted
[root@cm01dbm01 ~]#

The Oracle RAC interconnect is deployed using Oracle's Cluster High Availability IP framework, or HAIP. We can
find the details of the cluster HAIP by querying the relevant cluster resource:

[root@cm01dbm01 ~]# /u01/app/11.2.0.3/grid/bin/crsctl stat res -init -w "TYPE = ora.haip.type"
NAME=ora.cluster_interconnect.haip
TYPE=ora.haip.type
TARGET=ONLINE
STATE=ONLINE on cm01dbm01
[root@cm01dbm01 ~]#

■ in this crsctl query, the -init argument lists the core oracle Crs processes, such as ora.crsd , ora.cssd ,
ora.cluster_interconnect.haip , and so forth. the -w argument provides a means to display only a specific section
of the resource, by name. in the previous example, we're displaying the oracle Crs processes where
TYPE = ora.haip.typ e.

Note

Using oifcfg , you can find where Oracle places the InfiniBand network. Following, we see that the cluster
interconnect is on network 192.168.8.0/22 , the bondib0 interface. This is essentially a non-routable network
dedicated for cluster interconnect traffic.

[root@cm01dbm01 ~]# /u01/app/11.2.0.3/grid/bin/oifcfg getif
bondib0 192.168.8.0 global cluster_interconnect
bondeth0 172.16.10.0 global public
[root@cm01dbm01 ~]#

You can also log in to either a database instance or ASM instance to validate the cluster_interconnects
initialization parameter. Following, we see the 192.168.10.1 and 192.168.10.2 IP addresses, which in our case
correspond to both of the compute nodes in our Exadata Quarter Rack.

[grid@cm01dbm01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Fri Jul 27 17:54:42 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.