Database Reference
In-Depth Information
their computing infrastructures almost exclusively toward a few flavors
of Linux distributions. Many other disciplines, however, find themselves
with applications that will only run on platforms of other types. NFSv4
would help in this regard as well, but SRM clients and other data man-
agement utilities would still have to be ported to other popular plat-
forms.
3.5 Examples of Using File Streaming
in Real Applications
In the previous example, WLCG has chosen to have all files have a “per-
manent” type, and therefore file placement and removal are managed by ad-
ministrators of VOs. In this section, we describe two example applications
where file streaming is quite useful. In such applications, the streaming files
are
volatile
; that is, they have a lifetime, and each file can be released by the
client when it is done accessing it.
3.5.1 Robust File Replication in the STAR Experiment
Robust file replication of thousands of files is an extremely important task
in data-intensive scientific applications. Moving the files by writing scripts is
too tedious since the scientist needs to monitor for failures over long periods
of time (hours, even days), and recover from such failures. Failures can oc-
cur in staging files from a MSS, or because of transient transfer failures over
networks. This mundane, seemingly simple task is extremely time-consuming
and prone to mistakes. We realized that SRMs are perfectly suited to per-
form such tasks automatically. The SRMs monitor the staging, transfer, and
archiving of files, and recover from transient failures.
Figure 3.6 shows the sequence of actions for each file as a request of mul-
tiple files between two MSSs, such as NCAR-MSS and HPSS (a real use case
between NCAR and LBNL). The request is made to the target SRM, and
it issues a request for files to the source SRM. The source SRM stages files
from its MSS into its disk cache and notifies the target SRM. The target SRM
allocates space, notifies the source, and the transfer takes place. File stream-
ing works as follows: After each file gets transferred, the space it occupies at
the source cache is released, making space for additional files. Likewise, after
every file gets archived, the space it occupies in the target cache is released,
making space for additional files to be transferred. File streaming is essential
for this operation, where files are pinned and released, since the number of
files to be transferred can often be much larger than the space available in
the disk cache, especially when the disk cache is shared by multiple users.
