Introduction to High-Performance Computing Using MPI - High-Throughput Image Reconstruction and Analysis - page 117

Line 12 of the program contains a call to the MPI Comm rank function. Finally,
lines 14 and 15 print the ranks and sizes along with the message.
Figure 5.8 also shows a session of running the program mpi-hello.c, on a 4,096-
node Blue Gene/L system system and excerpts of its output. The first line shows the
mpirun command, used to launch the executable binary file mpi-hello.rts on
the Blue Gene/L system. The first argument wR0 represents a 4,096-node partition
(i.e., it specifies the specific set of processes on the Blue Gene/L System where this
job is to be run). The argument -np 4000 specifies the number of MPI processes
that need to be created. The argument -exe mpi-hello.rts specifies the com-
piled executable mpi-hello.rts of the mpi-hello.c program to be run. The job
launch command and parameters are not standardized and vary from system to sys-
tem. Even for a single system, there may be many ways to launch a job. One must
consult the system administrator to get details of how MPI jobs may be created.
Each MPI process of the program prints its message simultaneously to the stan-
dard output. MPI does not define any ordering in which the outputs of individual
MPI processes are displayed. These outputs are generally printed in a first come
first serve (FCFS) manner. As a result of this, the output of the MPI processes is
garbled as shown in Figure 5.8.
How can this problem be fixed? Since the output is printed in FCFS manner,
imposing an explicit ordering by appropriately timing the output of the MPI pro-
cesses may solve this problem. One naive (though not advisable) way to impose
order is by introducing different delays at different MPI processes before printing.
The amount of delay at each MPI process could be made proportional to its rank
(as in the program mpi-hello2.c listed in Figure 5.9), thereby ensuring that the MPI
processes print their message in a staggered way. In mpi-hello2.c, lines 2 and 15
were added to stagger the printf by introducing delays proportional to the rank
of the MPI processes. In the output of this program, messages from different MPI
processes are printed in separate lines, as desired.
Instead of adding delays proportional to the ranks, one could add random
delays as in mpi-hello3.c listed in Figure 5.10. Quite surprisingly, the output of
this program was garbled and looked like that of mpi-hello.c. This happens due
to the fact that when an MPI job is started, identical copies of the same program
are created. The calls to the rand function return a sequence of pseudo random
numbers determined by a seed . When the MPI job starts, all its processes are
identical and therefore have the same seed. Thus, the pseudo random sequence
generated at each process is the same. So each of the MPI processes, waits for
exactly the same time, and then prints out the message at exactly the same time,
resulting in the garbled output.
This problem can be fixed by initializing the seeds of the random number
generators at each process with different numbers. One can use the rank of the
process to carry out this initialization. Adding the call srand(rank+1) before line
16 solves this problem. With this change, output of mpi-hello3.c program is nicely
staggered with no mixing of messages printed by different MPI processes.
Any solution that requires processes to wait, without synchronizing with each
other, is not guaranteed to work. The delays required to stagger the output are
highly dependent on the system and its state at the time the processes print their
output. The MPI subsystem guarantees that identical copies of the program will