Database Reference
In-Depth Information
or delete it. That is, the SRM can make the decision about which files
to delete when space is needed based on the history of requests to each
file or based on the requests currently queued. For example, it is advan-
tageous to keep files that are accessed often (“hot” files) longer in the
disk cache. There are many such caching algorithms, the simplest but
effective one being to remove the least recently used (LRU) files.
Pinning and releasing files is an essential function for supporting file
streaming, described next.
7. File streaming.
Suppose that a client running an analysis program re-
quests 500 files, each 1 GB in size. Also, suppose that the analysis can
start as soon as a subset of the files is available. Some or all of the files
could be already in the SRM disk cache, or they have to be brought
in from an MSS or remote sites. When a request is made to the SRM,
it may allocate a quota that is smaller than the total size needed. For
example, assume that the SRM allocated 100 GBs to this client. Obvi-
ously, this job cannot be accomplished if the client wishes to have all 500
files in the SRM cache before proceeding. For such tasks, the concept of
file streaming is useful.
Basically, once the request is made, the SRM brings as many files
as will fit into the user's quota space. If some files are already in the
SRM's disk cache (for instance, they were brought in for another user
previously), they are made available immediately. The client can check
which files are there by issuing a status call, which returns TURLs for
all the files currently available. The client can then get each file and re-
lease them when finished. For every file released, the SRM can bring in
another file not yet accessed, thus streaming the files to the client. This
is shown schematically in Figure 3.5, where requested files are staged
from two near-line storage systems (could be local or remote) into the
client's quota space.
Some projects choose not to support file streaming since they require
all the files before analysis can start. However, even for such applica-
tions, file streaming can be used effectively by using the SRM disk cache
as a temporary staging space for files on the way to the client's local disk.
All files requested
from near-line storage
Some files cached into client's
quota space on on-line storage
by SRM
Figure 3.5
File streaming: stage, use, release, stage next.
