Database Reference
In-Depth Information
We discuss such examples in some detail in Section 3.5.2 for systems in
use by the ESG project and by the STAR HEP experiment.
8. File sharing.
The term
file sharing
is used to convey the concept that
the same physical copy of a file is shared by multiple users. Of course,
this can be done only because these files cannot be modified, which is
the typical use case that SRMs deal with. The value of file sharing is
naturally saving storage resources, by avoiding the replication of the
same file in the SRM cache, one for each user requesting it. However,
supporting this feature is not straightforward because of the need to
manage the lifetime of such a file.
The lifetime of a file is associated with a particular request by a par-
ticular client. Suppose that a file was already brought to cache and was
assigned a lifetime of an hour. Now, suppose that another client makes
a request for the same file half an hour later, requesting a lifetime of
another hour. This can be simply supported by extending the lifetime
of the file. However, how would the SRM know when all lifetimes ex-
pired or the files' pins released so that the file can be disposed of? This
requires that each lifetime is kept track of and marked as either
active,
expired,
or
released
. A file can only be removed if all of its lifetimes either
expired or were released.
Another aspect is how to count a shared file against the quota of
clients. Let us suppose that a file is already used by a client and a sec-
ond client requests it. Should the space required for the file be counted
against the second client's quota? In general, the answer is yes. However,
such a consideration is a choice of the VO policies.
Since it is complicated to manage file sharing, some implementations
choose to avoid that altogether and replicate each file in the client's
quota space. While this is wasteful of space, it only has a serious effect
if files are shared heavily by clients. Since many applications focus on a
subset of the files for their analysis (e.g., a group of scientists investigat-
ing jointly a certain phenomena), it is advisable to support file sharing.
9. File lifetime expiration mode.
So far we have described files that are
brought into a disk space on a temporary basis, thus having a lifetime
associated with them. We refer to these files as
volatile
because they can
be removed when the lifetime expires. If space is not extremely tight,
volatile files are usually not removed immediately, thus having a high
probability of being on the SRM cache if requested again often. Another
use for volatile files is a case when an administrator for a group of re-
searchers wants to make a set of files available to all for their analysis
over a relatively long period. This is common in applications where data
from experiments or simulations is shared by many researchers. In such a
case, one can setup an SRM per region (e.g., southern Europe) and repli-
cate files to that SRM for an extended period of time. A special function
referred to as
bring online
permits long lifetimes (e.g., many months),
