Hardware Reference
In-Depth Information
25.2.1 MPI-IO Instrumentation
The MPI message-passing libraries provide a name-shifted interface that
permits tools, including TAU, to intercept calls using the profiling message-
passing interface (PMPI) name-shifted interface. TAU additionally uses this
support to create a wrapper library for MPI-IO calls (e.g.,
MPIFileread
)
that internally calls the name-shifted interface (e.g.,
PMPIFileread
). Like
before, the wrappers can examine the arguments that flow through the I/O
calls to compute volume and bandwidth of individual I/O operations. In addi-
tion to TAU, this instrumentation technique is used in a wide variety of HPC
tools including Scalasca [9], VampirTrace [6], Score-P [8], mpiP [1], and In-
tegrated Performance Monitoring (IPM) (see Chapter 26). However, library
interposition through name-shifted interfaces is only available as a technique
if such interfaces are implemented in the library. This is not the case with
POSIX I/O.
25.2.2 Runtime Preloading of Instrumented Library
Many HPC operating systems such as Linux, Cray Compute Node Linux
(CNL), IBM Blue Gene Compute Node Kernel (CNK), and Solaris permit pre-
loading of a library in the address space of an executing application specifying
a shared object (DSO) in an environment variable (
LDPRELOAD
). It is possible
to create a tool based on this technique that can intercept all I/O operations
by means of a wrapper-library where the POSIX I/O calls are redefined to call
the global routine (identified using the
dlsym
system call) internally. Preload-
ing instrumented libraries is a powerful technique implemented by the runtime
linker and is used in TAU [4] and VampirTrace [6]. While it can resolve all
POSIX-I/O calls and operates on un-instrumented executables, it only sup-
ports dynamic executables. Static executables are used by default on IBM
Blue Gene and Cray XE6 and XK6 systems, although dynamic executables
may be created using the
dynamic
command line flag. A different technique
will be necessary to support static binaries.
25.2.3 Linker-Based Instrumentation
A linker can redirect references to a wrapped routine when it is invoked
with a special flag on the command line (
-Wl,-wrap,functionname
). In this
case, the application does need to be re-linked to use the wrapped library, but
this instrumentation technique overcomes the limitation of the previous ap-
proach provided by the runtime linker and may be used with both static and
dynamic executables. TAU has applied this approach to instrument POSIX
I/O calls by creating a wrapper library. Since the number of wrapped rou-
tines that may be present in a library might be potentially large, listing each
routine on the linker's command line can interfere with predened system
limits for command line length. Instead, a linker may read a file that contains
