Motivating example and problem analysis - Reliability Assurance of Big Data in the Cloud - page 22

Figure 3.2 Data flow graph of a pulsar searching instance for 8 minutes of observation
step generates 5 to 10 accelerated de-dispersion files (with the same total size of the original
de-dispersion file) for each de-dispersion file. Hence the total size of the accelerated di-
dispersion files is also at least 135.72 GB ( ∼ 78,000-156,000 files).
• Third, in the pulsar seeking step, based on the (accelerated) de-dispersion files, three dif-
ferent seeking algorithms: FFT, FFA, and single pulse are applied to search for pulsar
candidates. Each algorithm generates one seek result file (16 KB) for each de-dispersion/
accelerated de-dispersion file. Therefore in total, approximately 0.7488 to 7.488 GB pulsar
candidates' lists ( ∼ 46,800-468,000 files) are generated. Next, by comparing the candidates
generated from 13 beam files (13 “txt” files with a total size of 13 KB), the final 100 pulsar
candidates are selected. These candidates are then folded in XML files with their feature
signals, so 100 XML files are generated with a total size of 2.5 MB.
In summary, in addition to the raw observation data stored on tapes, for the pul-
sar searching application instance without the accelerated step, this 8-minute pulsar
searching application instance generates a total of 31,326 files with a size of about
163 GB. If the accelerated step is included, this 8-minute pulsar searching application