How Oracle Works on Exadata - Oracle Exadata Recipes: A Problem-Solution Approach - page 64

3-5. Tracing Oracle I/O Requests on Exadata Compute Nodes
Problem
You wish to confirm where physical database I/O is performed on Oracle Exadata. You've learned that the Exadata
Storage Servers are responsible for handling physical I/O, and you wish to confirm via operating system tracing that
this indeed is the case.
Solution
In this recipe, you will learn how to use strace to trace system calls from an Oracle instance process. Start by logging
in to an Oracle Exadata compute server and trace system calls issued by the DBWR process for an instance. Find
the process ID of a DBWR process and then use strace to examine the system calls it makes during a period of
DBWR activity:
[root@cm01dbm01 ~]# ps -ef|grep visx|grep dbw
oracle 22251 1 0 Jul27 ? 00:00:03 ora_dbw0_visx1
oracle 22259 1 0 Jul27 ? 00:00:03 ora_dbw1_visx1
oracle 22261 1 0 Jul27 ? 00:00:02 ora_dbw2_visx1
[root@cm01dbm01 ~]# strace -c -p 22251
Process 22251 attached - interrupt to quit
Process 22251 detached
% time seconds usecs/call calls errors syscall
------ ---------- ----------- --------- --------- --------------
nan 0.000000 0 1 read
nan 0.000000 0 1 open
nan 0.000000 0 1 close
nan 0.000000 0 1 poll
nan 0.000000 0 14 sendmsg
nan 0.000000 0 57 1 recvmsg
nan 0.000000 0 28 14 setsockopt
nan 0.000000 0 6 getrusage
nan 0.000000 0 13 times
nan 0.000000 0 4 3 semtimedop
------ ---------- ----------- --------- --------- ---------------
100.00 0.000000 126 18 total
[root@cm01dbm01 ~]#
This shows us that the DBWR process used the sendmsg system call to package an I/O request, which ships
the iDB message over the InfiniBand network in the case of Exadata, and a recvmsg call to receive the data from the
storage servers.
How It Works
With Oracle Exadata, database I/O is handled exclusively by the cellsrv process/threads on the storage servers. The
database tier nodes, operating with asynchronous I/O, bundle I/O requests into iDB messages and transmit these over
the InfiniBand network grid to the Exadata Storage Servers. The storage servers perform the actual I/O and send data
back to the compute nodes over InfiniBand. Compute servers ingest this data and process it according to the nature
of the I/O request in an identical fashion to how physical I/O is consumed by the server process or database in
non-Exadata systems. Figure 3-3 depicts this flow.