Database Reference
In-Depth Information
hist
.
write
(
To
.
textFile
(
outputPath
),
Target
.
WriteMode
.
OVERWRITE
);
pipeline
.
done
();
Crunch compares the timestamps of the input files with those of the checkpoint files; if
any inputs have later timestamps than the checkpoints, then it will recompute the depend-
ent checkpoints automatically, so there is no risk of using out-of-date data in the pipeline.
Since they are persistent between pipeline runs, checkpoints are not cleaned up by
Crunch, so you will need to delete them once you are happy that the code is producing the
expected results.
