Motivating example and problem analysis - Reliability Assurance of Big Data in the Cloud - page 21

conduct the pulsar searching survey, one observation duration is currently from 4 minutes to
1 hour [11] . The raw signal data are preprocessed by a local cluster at Parkes in real time and
archived in tapes for future analysis. These data are then delivered by post to the Swinburne
Centre for Astrophysics and Supercomputing located in Melbourne.
2. Data preparation:
At the beginning of the pulsar searching application work flow, different beam files are
extracted from the raw data files and compressed. The outcomes are 13 extracted and com-
pressed beam files. Each of the files is normally approximately 1 to 20 GB in size depending
on the observation duration. The beam files contain the pulsar signals, which are dispersed
by the interstellar medium. Therefore, a de-disperse step needs to be conducted to counteract
the effects. In the de-dispersion process, a large number of de-dispersion files are generated
with different dispersion trials. For one dispersion trial of one beam file, the size of the de-
dispersion file is approximately 4.6 to 80 MB, depending on the size of the input beam file
( ∼ 1-20 GB). A minimum of 1200 dispersion trials are conducted in parallel for each beam
file, which in total take 1 to 13 hours to finish and generate approximately 5 to 90 GB of de-
dispersion files. Next, for binary pulsar searching specifically, every de-dispersion file needs
a further accelerated step for processing. This step generates the accelerated de-dispersion
files of the similar size compared with the de-disperse files.
3. Pulsar seeking:
Based on the de-dispersion files, different seeking algorithms can be applied to search pulsar
candidates, such as fast Fourier transform (FFT), fast folding algorithm (FFA), and single
pulse. Take the FFT seeking algorithm as an example, it takes approximately 7 to 80 min-
utes to process the approximately 5 to 90 GB of de-dispersion files, and the outcome of
each pulsar seeking algorithm is a seek result file, which is normally 16 KB in size. After
that, the candidate lists of pulsars in “txt” format with the size of 1 KB each generated for
each beam file in the same time session are compared so that interferences can be detected
and eliminated. For the final pulsar candidates, their feature signals are obtained from the
corresponding de-dispersion files, which are then folded into XML files (25 KB for each
pulsar candidate). This step takes up to 1 hour according to the number of candidates found.
Finally, the XML files are visually displayed for making final decisions on whether a pulsar
has been found or not.
3.1.2 The pulsar searching application data flow
In order to facilitate analyzing the data storage of the pulsar searching application, the
data flow of the pulsar searching application also needs to be described. Figure 3.2
shows the data flow graph of a typical pulsar searching application instance for pro-
cessing 8 minutes of raw telescope data [83] .
In Figure 3.2 , the amount of data involved in each step of the process is clearly
indicated:
• First, in the raw data recording step, the telescope real-time raw data stream is downloaded
at the speed of 1 GB per second for 8 minutes, and hence 480 GB of raw data are recorded.
• Second, in the data preparation step, 13 extracted and compressed beam files are generated.
For the 8-minute observation, the size of each beam file is 2.1 GB. Hence the total size of the
beam files is 27.3 GB. Next, to counteract the dispersion effect by the interstellar medium,
the de-dispersion is conducted at a minimum of 1200 different dispersion trials. Each disper-
sion trial generates a de-dispersion file of 8.7 MB. Hence the total size of dispersion files
is at least 135.72 GB (15,600 files). Optionally, for binary pulsar searching, the accelerate