Database Reference
In-Depth Information
1.6.4 Data Integrity
Media integrity is often confused as being equivalent to data integrity, but the
important and consequential differences are often underappreciated. Compos-
ite devices only protect against data-integrity problems with the disk media,
but that does not ensure the integrity of data produced by a scientific ap-
plication. From the application standpoint, any intermediate step along the
complex pathway to storage can compromise data integrity before it arrives
at the storage device, which will dutifully record the incorrect values to per-
manent storage. Unfortunately, commonly deployed storage hierarchies (for
example, memory to disk to tape) provide no such end-to-end data integrity
checking, so it is left to the application developer to employ various forms of
checksums and other tests to verify the integrity of their data.
CERN (the European Organization for Nuclear Research) recently per-
formed a study of data integrity issues on their own systems
10
by writing
3,000 special 2 GB files of a predefined pattern every two hours, and then
reading them back to check for errors every five weeks. What they found was
that even with RAID to protect against media errors, they observed 300 un-
corrected and unreported errors in the 2.4 petabytes of data that were stored
on the data volumes. In all, after examining 8.7 TB of user data for corrup-
tion (33,700 files), they found 22 corrupted files that had not been detected by
any part of the storage infrastructure. In some cases, the corruption remained
undetected because it was caused by errors in various tiers of the software in-
frastructure rather than random bit-flips. Ultimately, CERN found an overall
byte error rate of one in 3*10
7
bytes, which is considerably higher than the
media error rate of one byte in 10
14
. Without some form of end-to-end mon-
itoring of data integrity it is clear that the larger disk-based storage systems
will observe data inconsistencies. Focusing on media alone for data integrity
protection is insucient.
There are emerging storage technologies such as zettabyte file system
(ZFS),
11
that are capable of ensuring end-to-end data integrity. However, until
such systems are widely deployed, it is important for scientists to incorporate
their own checksum checks (such as cyclic redundancy check [CRC] or mes-
sage digest algorithm #5 [MD5] and error detection mechanisms into their
data storage practices to protect against silent data corruption.
1.7 Emerging Technologies
This section describes technology that is poised to compete with mainstream
tape and disk technology, but for various reasons has not yet overtaken these
technologies. The reasons include manufacturing economics in a volume mar-
ket and technologies that are still in development. Many of these technologies
