Database Reference
In-Depth Information
SPARQL. The MonetDB team started work in 2008 to offer scalable RDF
storage and support for the W3C query language SPARQL to the
system.
7.5 Experience with SkyServer Data Warehouse
Using MonetDB
7.5.1 Application Description and Planned Experiments
To illustrate the advantages of vertical databases for scientific data manage-
ment we summarize the experiences from porting the SkyServer application
55
onto MonetDB. The SkyServer application is a good example of a read-
optimized database system with long periods of
ad hoc
querying of large data
volumes, and periodic bulk-loading of new data. In these settings a column-
store architecture offers more ecient data access patterns for disk-bound
queries, flexibility in the presence of changing workloads, and reduced storage
needs. The MonetDB/SkyServer project
56
started with the purpose of pro-
viding an experimentation platform to develop new techniques addressing the
challenges posed by scientific data management. Our intent was to examine
and demonstrate the maturity of column-store technology by providing the
functionality required by this real-world astronomy application. The project
shows the advantages of vertical storage architectures for scientific applica-
tions in a broader perspective. It goes way beyond micro benchmarks and sim-
ulations typically used to examine individual algorithms and techniques. Mon-
etDB/SkyServer allows testing the performance of the entire software stack.
The SkyServer application gives public access to data from the Sloan Digi-
tal Sky Survey
57
, an astronomy survey with the ambition to map one-quarter
of the entire sky in detail. The survey has already collected several terabytes
of data. The sky object catalog stored in a relational database reached the
volume of 4 TB for data release 6 in 2007. The database schema is orga-
nized in several sections among which
Photo
and
Spectro
contain the most
important photometric and spectroscopic factual data from the survey. The
Photo
section has a structure centered in the
PhotoObjAll
table. The table
contains more than 440 columns and more than 270 million rows, which al-
ready stresses the capabilities of most DBMSs. A single record in a row-store
representation occupies almost 2 KB, and the majority of the fields are real
numbers representing CCD measurements.
Porting of the SkyServer application to MonetDB was organized in three
phases. The goal of the first phase was to develop and enhance MonetDB's
features to handle the functionality requirements of the SkyServer application.
The target dataset during this phase was the so-called Personal SkyServer,
a 1% subset of the archive with a size of approximately 1.5 GB. Since this
