Database Reference
In-Depth Information
Data Model for End-to-End Application
The BigQuery end-to-end sensor app uses the objects described in this
chapter to organize its data for querying and display. This section describes
the project setup, the datasets, the tables, and the schemas that are used for
the sensor application.
Project
The project used by the sensor data app has the project ID bigquery-e2e .
The project number and project name are visible in Figure 4.2 . The
BigQuery and Cloud Storage APIs have both been enabled via the services
tab (hence you can see them in the left panel), and billing has been enabled
via the billing tab (which is necessary to load data).
The project permissions configuration (refer to Figure 4.3 ) is relatively
straightforward: The two authors of this topic are both owners; the service
account from the BigQuery Sensors AppEngine app is a reader. The service
account is a reader instead of an editor as a security precaution that prevents
it from updating data unintentionally or maliciously. Having reader-level
access does allow a user to run queries and other jobs that may cost the
project owner money—so be careful when adding people to your team!
Datasets
There are four datasets used in the sensor app: logs , reference ,
dashboard , and backup . The logs dataset contains the streamed data
from mobile sensors, with one table per day. The dashboard contains a
cache of values used for rendering charts in the app. The reference dataset
contains reference tables that have been downloaded from external sources.
Finally, the backup dataset contains backups from the app's AppEngine
datastore. All these datasets have been locked down via custom ACLs.
The logs dataset contains sensitive data, so the default project reader and
project writer ACL entries have been removed. Project owners are still on
the ACL so that we (the owners) can query the data to diagnose problems
with the app. The logs are read by the sensor data AppEngine app, so
the app's service account ( bigquery-sensors@appspot.gserviceaccount.com )
has been added as a reader on the dataset ACL.
Search WWH ::




Custom Search