Network Administration - Oracle Exadata Recipes: A Problem-Solution Approach - page 298

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> select * from gv$cluster_interconnects;

INST_ID NAME IP_ADDRESS IS_ SOURCE
---------- --------------- ---------------- --- -------------------------
1 bondib0 192.168.10.1 NO cluster_interconnects parameter
2 bondib0 192.168.10.2 NO cluster_interconnects parameter
SQL>

As stated, the Oracle interconnect traffic is over the InfiniBand network. Confirm this by running skgxpinfo from
the 11gR2 Grid Infrastructure software directory:

[grid@cm01dbm01 ~]$ which skgxpinfo
/u01/app/11.2.0.3/grid/bin/skgxpinfo
[grid@cm01dbm01 ~]$ /u01/app/11.2.0.3/grid/bin/skgxpinfo -v
Oracle RDS/IP (generic)
[grid@cm01dbm01 ~]$

This shows that the communication protocol is using Oracle RDS/IP—RDS is the InfiniBand protocol.

■ if you have a need to change your infiniBand network configuration on exadata, it is best to engage oracle
support and/or follow my oracle support note 283684.1 to ensure that the cluster interconnect information is modified
appropriately. Changing your interconnect information carries implications with your storage network as well.

Note

How It Works
As documented in the Oracle Exadata Owner's Manual and other sources, the InfiniBand network serves as both
the storage network and Oracle RAC private interconnect on Exadata. Oracle binaries in both the RDBMS and Grid
Infrastructure homes are linked with the ipc_rds flags on Exadata, which you can see by examining the contents of
$ORACLE_HOME/rdbms/lib/ins_rdbms.mk :

ipc_rds:
-$(RMF) $(LIBSKGXP)
$(CP) $(LIBHOME)/libskgxpr.$(SKGXP_EXT) $(LIBSKGXP)

ipc_relink:
(if $(ORACLE_HOME)/bin/skgxpinfo | grep rds;\
then \
$(MAKE) -f $(MAKEFILE) ipc_rds; \
else \
$(MAKE) -f $(MAKEFILE) ipc_g; \
fi)

Note than an Exadata DMA can mistakenly choose to relink binaries without RDS on Exadata; there is nothing on
the compute nodes that prevents this from happening and, in fact, Oracle only decides to link with RDS if the output of