Database Reference
In-Depth Information
These are blocks that do not meet their target replication for the file they belong to.
HDFS will automatically create new replicas of under-replicated blocks until they meet
the target replication. You can get information about the blocks being replicated (or
waiting to be replicated) using
hdfs dfsadmin -metasave
.
Misreplicated blocks
These are blocks that do not satisfy the block replica placement policy (see
Replica
three replicas of a block are on the same rack, then the block is misreplicated because
the replicas should be spread across at least two racks for resilience. HDFS will auto-
matically re-replicate misreplicated blocks so that they satisfy the rack placement
policy.
Corrupt blocks
These are blocks whose replicas are all corrupt. Blocks with at least one noncorrupt
replica are not reported as corrupt; the namenode will replicate the noncorrupt replica
until the target replication is met.
Missing replicas
These are blocks with no replicas anywhere in the cluster.
Corrupt or missing blocks are the biggest cause for concern, as they mean data has been
lost. By default,
fsck
leaves files with corrupt or missing blocks, but you can tell it to per-
form one of the following actions on them:
▪ Move the affected files to the
/lost+found
directory in HDFS, using the
-move
option. Files are broken into chains of contiguous blocks to aid any salvaging ef-
forts you may attempt.
▪ Delete the affected files, using the
-delete
option. Files cannot be recovered
after being deleted.
Finding the blocks for a file
The
fsck
tool provides an easy way to find out which blocks are in any particular file. For
example:
%
hdfs fsck /user/tom/part-00007 -files -blocks -racks
/user/tom/part-00007 25582428 bytes, 1 block(s): OK
0. blk_-3724870485760122836_1035 len=25582428 repl=3 [/default-rack/
10.251.43.2:
50010,/default-rack/10.251.27.178:50010, /default-rack/
10.251.123.163:50010]
