How Oracle Works on Exadata - Oracle Exadata Recipes: A Problem-Solution Approach - page 71

To identify whether you have bottlenecks or performance issues, concentrate on overall time spent and time-
per-call for the I/O-related system calls, io_getevents and io_submit . You would ideally strive for sub-5 millisecond
service times or lower. Divide the usecs/call value for io_getevents and io_submit by 1,000 to determine if this is
the case. In the previous example, a 135 usecs/call for io_getevents represents 0.135 ms.
Here is some of the key information about the results: The •

recvmsg lines indicate
iDB messages received over InfiniBand from the compute servers; these would have
corresponding sendmsg calls from the database grid.
The

•

sendmsg lines represent iDB messages returned to the compute grid from the storage
server, typically indicating the completion of an I/O.

•

io_getevents and io_submit work in tandem to collect asynchronous I/O requests and
perform the actual I/Os being requested.

In short, cellsrv on the storage servers are handling database I/O and communicating via messaging over the
InfiniBand interconnect. In Recipe 3-5, we discussed the I/O path from the compute node servers; this recipe
is a complement to Recipe 3-5.
If you look more closely into sendmsg and recvmsg system calls, you will be able to see the InfiniBand IP
addresses of the two compute nodes. If you took the time to weed through the output, you would learn that
the IP addresses for the recvmsg and sendmsg calls only come from and to the InfiniBand IP addresses of the
compute servers:

[root@cm01cel01 ~]# grep recvmsg /tmp/cellsrv.trc |more
[pid 1185] recvmsg(127, <unfinished ...>
[pid 1185] <... recvmsg resumed> {msg_name(16)={sa_family=AF_INET, sin_port=htons(20938),
sin_addr=inet_addr("192.168.10.2")}, msg_iov(3)=
[{"\4\3\2\1\240\226\26\27\0\0\0\0MRON\4\3\0\0\0\0\0\0\237>\343W\0\0\0\0"..., 76},
{"\1\0\0\0\215\330\201\242(\0\0\0\0\0\0\0\0\0\0\0\2\2\0\0\0\0\0\0", 28},
{"\4\3\2\1\31\0\0\0004\214\231\35\0\0\0\0\373\177\0\0\233\323\252\2\0\10\250\300\2\2\2\2"..., 576}],
msg_controllen=0, msg_
flags=0}, 0) = 144
[pid 1185] recvmsg(127, <unfinished ...>
[pid 1185] <... recvmsg resumed> {msg_name(16)={sa_family=AF_INET, sin_port=htons(28810),
sin_addr=inet_addr("192.168.10.1")}, msg_iov(3)=
[{"\4\3\2\1\234\226\26\27\0\0\0\0MRON\4\3\0\0\0\0\0\0\350\217\225\30\0\0\0\0"..., 76},
{"\1\0\0\0\215x\240\376(\0\0\0\0\0\0\0\0\0\0\0\1\2\0\0\0\0\0\0", 28},
{"\4\3\2\1\31\0\0\0004\214\231\35\0\0\0\0\374\177\0\0tP8b\0\10\250\300\1\2\1\2"..., 576}],
msg_controllen=0, msg_flags=0}

root@cm01cel01 ~]# grep sendmsg /tmp/cellsrv.trc |more
[pid 1185] sendmsg(127, {msg_name(16)={sa_family=AF_INET, sin_port=htons(20938),
sin_addr=inet_addr("192.168.10.2")}, msg_iov(3)=[{"\4\3\2\1\2
5\360C\5\0\0\0\0MRON\4\3\0\0\0\0\0\0004\214\231\35\0\0\0\0"..., 76},
{"\1\0\0\0\0\0\0\0,\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 28}, {"\4\3\2\
1\24\0\0\0\237>\343W\0\0\0\0004\214\231\35\373\177\0\0\233\323\252\2\0\10\250\300"..., 44}],
msg_controllen=0, msg_flags=0}, 0 <unfinished ...>
[pid 1185] <... sendmsg resumed> ) = 148
[pid 1185] sendmsg(127, {msg_name(16)={sa_family=AF_INET, sin_port=htons(28810),
sin_addr=inet_addr("192.168.10.1")}, msg_iov(3)=[{"\4\3\2\1\2