Database Reference
In-Depth Information
Chapter 2
Parallel Data Storage and Access
Robert Ross, 1 Alok Choudhary, 2 Garth Gibson, 3 and Wei-keng Liao 2
1 Argonne National Laboratory
2 Northwestern University
3 Carnegie Mellon University
Contents
2.1
From Disk Arrays to Parallel Data Storage
..........................
36
2.1.1
Data Consistency and Coherence
..............................
37
2.1.2
Fault Tolerance
................................................
39
2.2
Parallel File Systems
..................................................
40
2.2.1
Block-Based Storage Systems
..................................
40
2.2.1.1
General Parallel File System
.........................
41
2.2.2
Object-Based Storage Systems
................................
43
2.2.2.1
PanFS
.................................................
43
2.2.2.2
PVFS
..................................................
47
2.2.2.3
Lustre
.................................................
49
2.3
Example Parallel File System Deployments
..........................
50
2.3.1
PVFS on the IBM Blue Gene/P
..............................
50
2.3.1.1
Argonne Blue Gene/P System
........................
51
2.3.1.2
Blue Gene/P I/O Infrastructure
.....................
51
2.3.1.3
Tolerating Failures
....................................
52
2.3.2
PanFS on Roadrunner
.........................................
53
2.4
Interfacing with Applications
.........................................
55
2.4.1
Access Patterns
................................................
55
.....................................................
2.4.2
POSIX I/O
57
.........................................................
2.4.3
MPI-IO
58
2.4.4
High-Level I/O Libraries
......................................
60
2.4.4.1
Parallel netCDF
......................................
60
2.4.4.2
HDF5
.................................................
61
2.5
Future Trends and Challenges
........................................
63
2.5.1
Disk Failures
...................................................
63
2.5.2
Solid State and Hybrid Devices
...............................
66
2.5.3
Extreme-Scale Devices
.........................................
66
2.6
Summary
..............................................................
68
Acknowledgments
...........................................................
69
References
..................................................................
69
35
Search WWH ::




Custom Search