Database Reference
In-Depth Information
The mechanics from the BigQuery side were pretty much the same as
loading CSV or JSON data from Google Cloud Storage. In fact, in some
ways it was simpler because the table schema was not required because
it is derived from the contents of the backup. However, this final step of
loading a backup into BigQuery is a relatively small part of the overall
Datastore-BigQuery integration. This section covers the details on the
AppEngine side and covers important features and caveats in the
integration. Developers who have used Datastore will find that the material
is familiar, but it is still useful to see how the features of the service relate
to BigQuery. If you are not familiar with Datastore, the section can serve as
a quick introduction to its capabilities and help you evaluate if it could be
used as a component in your analytics solution.
Briefly, Datastore is a NoSQL data storage service that is well suited to
transactional workloads. It supports:
• Efficient writes, reads, and lookups of individual records
• A well-defined consistency model and transactions
• Secondary indexes
• Structured records
• A whole lot more
You
can
learn
all
about
the
service
at
The
two
most
important characteristics with respect to BigQuery are:
• Relatively slow and expensive full scan queries
• NoSQL, records are not required to adhere to a schema.
The cost of full scan queries is what makes BigQuery integration interesting.
In general, Datastore has a high cost for low selectivity queries where a
large fraction of rows have to be inspected. If a primary or secondary index
cannot be used to limit the amount of data that needs to be read from a large
table, the query is generally prohibitively slow and expensive on Datastore.
Further, the query language supported by BigQuery is much richer and
permits much more complex queries than the query language supported
by Datastore, which is intended mainly for record lookup operations. The
second point, the NoSQL nature of Datastore, is probably the main source
of complexity in the integration of the two services.
