Game Development Reference
In-Depth Information
standard compilation that includes uploading the results to the cache server. It is
up to the designer to allow the method of populating the server from users' work-
stations and dedicated machines. In general, any result is acceptable, independent
of its origin.
Establishing how the set of input dependencies are determined, stored, and
queried is a fundamental decision. Simple methods such as filenames and times-
tamps may be inaccurate as they do not fully represent the contents of the files.
Hashes, signatures, and file digests are popular alternatives. They can be calculated
using different variations of the CRC, MD5, or SHA algorithms. In general, any
method capable of converting a variable-length stream into a fixed-length key is
valid. For our practical needs, the methods need to generate keys with low collision
probability; that is, for two different files, the probability of generating the same
key is really low.
Hashes can also be combined together, so several dependencies can be repre-
sented by a single key. For a combined hash, a successful search in the database
will retrieve a file that has been compiled from all the inputs whose hashes were
merged.
Cached results can be stored in many ways. Two of the most common methods
are versioning systems and dedicated servers:
Versioning systems.
Compiled results are stored in a versioning system (Perforce,
Alienbrain, CVS, SVN, Git). Cache population and cache hits are, in fact, reposi-
tory check-ins and retrievals.
This approach works well in those cases where each file is compiled with a
single set of inputs and where the most recent version is generally usable by any
build. Every source and its compiled resultscanbecheckedinatthesametime.
Syncing to the latest version of the repository will mostly retrieve files that do not
need further recompilation. This approach is not so flexible when files are processed
with variable compiler flags and dependencies. In these cases, the versioning system
needs to be complemented with an external database capable of linking each hash
with a given version.
Dedicated servers.
The use of dedicated servers represents the most flexible ap-
proach. A cache server can integrate sophisticated algorithms, but in its most basic
configuration it can be built from a database and a file server. In some circum-
stances the hash management system is accessed through an ad hoc interface that
implements advanced querying features. They will be described in the following
sections.
Not to be overlooked: These machines must be able to support high disk/net-
work loads as well as to eciently handle concurrent uploads and downloads. In
general, file accesses will range from a few bytes to several gigabytes. However,
the distribution is usually biased to files containing metadata and art assets due
