Database Reference
In-Depth Information
for today's I/O systems for supporting ecient and scalable data movement
between disks and memories. With the increasing number of cores in future
computer systems, the task for designing scalable I/O systems will be even
tougher. Hence, it is important to understand the applications, I/O require-
ments and reexamine the I/O strategies used by the current file systems.
A recent survey conducted by Shan and Shalf 37 studied the current practice
and future requirements of I/O systems from the user community of scien-
tific parallel applications. The results revealed several interesting I/O char-
acteristics commonly appearing in parallel applications. First, the majority
of I/O patterns are append-only writes. Parallel applications often have very
long execution time and perform data checkpointing at a regular interval for
restart purposes. Many applications also keep the checkpoint files for postrun
data analysis, such as visualization and feature extraction. However, the data
analysis tasks are usually carried out on different machines, because their
computation power requirement is not as critical as the parallel applications
themselves. The read operations occur mostly at the beginning of the run for
data initialization. Compared to the write, the read amount is much smaller
and the cost is negligible.
The survey also indicated that the most popular I/O method uses one-file-
per-process strategy. This method is easy to program and often gives satis-
factory performance when applications run on a small number of processes.
However, the immediate drawback is that a restart must use the same num-
ber of processes as the run that produced the checkpoint files. Note that if
a different number of processes were used in a restart, the I/O is no longer
one-file-per-process. A more serious problem is that this method can create a
management nightmare for file systems when an application runs on a large
number of processes. Thousands of processes can produce hundreds of thou-
sands or millions of files. Because modern parallel file systems employ one of
only a small number of metadata servers, serious network tra c congestion
can form at the metadata servers when thousands of files are being created
simultaneously. Furthermore, after a parallel job exits, millions of newly cre-
ated files immediately become a challenge for file systems to manage. An
alternative solution is the parallel shared-file I/O.
In a parallel application, the problem domain is often represented by a set
of global data structures, for example, multidimensional arrays. These global
data structures are partitioned among multiple processes so that each process
operates on the data in its subdomain. During a data checkpoint, it makes
more sense to store the data in global canonical order. Such file layouts often
make postrun data analysis and visualization easier. However, shared-file I/O
may result in poor performance if the requests are not handled carefully. In or-
der to address this concern, the MPI-IO standard defines a set of programming
interfaces for concurrent shared-file access. MPI-IO consists of collective and
independent function calls. The purpose of collectives is to enable the MPI-
IO implementation to collect together processes to generate the I/O requests
that perform faster. An example is the two-phase I/O strategy. 21 , 23
Another
Search WWH ::




Custom Search