Hardware Reference
In-Depth Information
19.7 Conclusion
This study successfully undertook an unprecedented trillion-particle sim-
ulation on 120,000 cores on Hopper. The fully functional simulation produced
30 TB to 42 TB of data on a per-timestep basis resulting in major I/O chal-
lenges. While the study utilized a well-tuned production I/O stack consisting
of HDF5, Cray MPI-IO, and Lustre, the study faced and addressed a number
of issues at scale. Terabytes of data were successfully written to a single file on
the Lustre file system using the available peak bandwidth. This performance
was comparable with the fpp model, where each MPI process writes data into
one file. The fpp model is often considered the gold standard for writing large
data to parallel file systems. The study also analyzed the particle datasets,
which led to various scientific discoveries answering the science objectives men-
tioned in Section 19.2. More details of analysis and visualization of the data
are available in works by Byna et al. [2, 3, 12].
19.8 Acknowledgments
Contributions of this study and chapter also come from V. Roytershteyn,
K. Wu, O. Rubel, J. Chou, K-W. Lin, A. Uselton, Y. He, and D. Knaak. This
work is supported by the Director, Oce of Science, Oce of Advanced Scien-
tific Computing Research, of the U.S. Department of Energy under Contract
No. DE-AC02-05CH11231. This research used resources of the National En-
ergy Research Scientific Computing Center (NERSC). The authors would like
to thank NERSC and Cray staff for troubleshooting I/O issues on Hopper.
We would also like to thank members of the HDF Group for their advice on
HDF5 I/O optimizations, and Burlen Loring for his advice and support.
Bibliography
[1] K. J. Bowers, B. J. Albright, L. Yin, B. Bergen, and T. J. T. Kwan.
Ultrahigh Performance Three-Dimensional Electromagnetic Relativistic
Kinetic Plasma Simulation. Physics of Plasmas, 15(5):7, 2008.
[2] Surendra Byna, Jerry Chou, Oliver Rubel, Prabhat, Homa Karimabadi,
William S. Daughton, Vadim Roytershteyn, E. Wes Bethel, Mark How-
ison, Ke-Jou Hsu, Kuan-Wu Lin, Arie Shoshani, Andrew Uselton, and
