Database Reference
In-Depth Information
By now you have seen that schemas are strictly enforced on BigQuery tables.
In Datastore terminology tables are
kinds
. A set of
entities
in Datastore
have the same
kind
if certain parts of their Datastore
key
are the same.
Although entities are structured, namely they have fields with values rather
than being a binary or text blob, they are not required to all have a uniform
structure. In fact, even entities of the same kind are not required to have
the same structure. However, it is common for entities of the same kind to
have a uniform structure. This is generally enforced at the application layer,
and the commonly used Datastore client libraries are geared to this kind of
usage. The service does not enforce this behavior, and there are examples
of applications that take advantage of this flexibility. When entities of a
given kind have a uniform structure, the integration with BigQuery is quite
transparent, and the kind maps neatly to a BigQuery table. Now start with
an example that fits this simple usage model and then explore what happens
when a non-uniform structure is introduced.
To fully work through the examples in this section, you need an AppEngine
project and a Google Cloud Storage project. Sample data is available so that
some of the examples can be tried with just a BigQuery project.
Simple Kind
As part of the sensor application developed for this topic, we stored
information about the devices registered with the application to stream logs.
We chose to store this data in Datastore as the
Device
kind. Each entity of
this kind represented a single device registered with the service and stored
information that was pretty much constant for a given device so that each
log record did not have to duplicate this information. Here is the Python
code used to describe the information stored in these entities:
class Device(ndb.Model):
"""Registration record for a device logging to the
service."""
owner = ndb.UserProperty()
added = ndb.DateTimeProperty(indexed=False,
auto_now_add=True)
type = ndb.StringProperty(indexed=False,
choices=('phone', 'tablet'))
make = ndb.StringProperty(indexed=False,
validator=_validate_str)
