Database Reference
In-Depth Information
performing the actual work. Besides ensuring that the scheduled task
completes in a bounded amount of time, the use of queues also allows
the application to manage the concurrency of operations to ensure you
stay under BigQuery rate limits. Here we simply use the default queue
configuration, but App Engine allows custom queue configurations that rate
limit task execution.
Data Transformation
You now have data cached in the dashboard dataset, ready for delivery to
the user for rendering in the UI. In the sample application you have the
JavaScript client fetch the data from the web application rather than directly
from BigQuery so that you can control the access to the data. Also, you
want to make it simple for the JavaScript application to render the data,
so you want to supply it in a format that is convenient for your rendering
framework. To support this you can implement a handler in the web
application that enables you to transform the data returned by BigQuery
into different formats. This code is a little complex because it needs to be
flexible in input and output. Chapter 5 describes two different methods
for reading data. The data from a table can be read via the
bigquery.tabledata().list() operation, and data that is the result
of a query job can be read via the bigquery.jobs().query() and
bigquery.jobs().getQueryResults() methods. The former can read
data from any table. The latter which is limited to tables generated by query
jobs has the advantage that you can issue a query and read its result in a
single call for queries that run quickly and return a modest amount of data.
We have implemented the data handler in the application so that it supports
both methods as shown in Listing 8.7 . If an existing table is requested, the
bigquery.tabledata() API is used, and if the request requires running
a new query, the bigquery.jobs() API is invoked.
Listing 8.7 : Data formatting handler (dashboard.py)
class _Formatter(object):
'''Base class for formatting rows.'''
def mime_type(self):
'''Returns the mime type that the format conforms
to.'''
 
 
Search WWH ::




Custom Search