How Oracle Works on Exadata - Oracle Exadata Recipes: A Problem-Solution Approach - page 68

First, log in to a compute node as the Grid Infrastructure software owner and validate your Oracle RAC
interconnect details using oifcfg :

[grid@cm01dbm01 ~]$ oifcfg getif
bondib0 192.168.8.0 global cluster_interconnect
bondeth0 172.16.1.0 global public
[grid@cm01dbm01 ~]$

Next, check your routing tables to determine which interface this network resides on:

[root@cm01dbm01 ~]# route -v | grep 192.168
192.168.8.0 * 255.255.252.0 U 0 0 0 bondib0
[root@cm01dbm01 ~]#

We can see from this output that we are using the bondib0 interface. Use ifconfig to display details of
this interface:

[root@cm01dbm01 ~]# ifconfig bondib0
bondib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.10.1 Bcast:192.168.11.255 Mask:255.255.252.0
inet6 addr: fe80::221:2800:1a1:25c5/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1
RX packets:1970084 errors:0 dropped:0 overruns:0 frame:0
TX packets:1923066 errors:0 dropped:69 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:998645331 (952.3 MiB) TX bytes:1168770033 (1.0 GiB)

[root@cm01dbm01 ~]#

The Link encap:InfiniBand text confirms we have an InfiniBand link on our private interconnect, but it
doesn't show us whether we're running RDS over InfiniBand or possibly UDP. To determine this, use the
skgxpinfo command:

[grid@cm01dbm01 ~]$ $ORACLE_HOME/bin/skgxpinfo -v
Oracle RDS/IP (generic)
[grid@cm01dbm01 ~]

For reference, on a traditional non-Exadata Oracle 11gR2 RAC Cluster, it would be common to see this command
display udp :

[oracle@rac1 ~]$ $ORACLE_HOME/bin/skgxpinfo
udp
[oracle@rac1 ~]

On Exadata, you can also use ibdump to examine packets from your bonded InfiniBand interface to confirm the
RAC interconnect traffic is using this network.

■ Note the traditional tcpdump command is not InfiniBand-aware and would tag its output as Udp packets.