Database Reference
In-Depth Information
Programming Language Note
AppEngine is a general-purpose computing environment. In our
examples we have used Python, but all these features are also available
in the AppEngine Java SDK, so you could use Java if it is more
convenient for your application. Translating the Python APIs to Java is
mostly straightforward. You should consult the AppEngine
documentation for more information about the differences between the
Java and Python APIs.
The AppEngine SDK contains other reader and writer modules that let
you specify Datastore or Cloud SQL as MapReduce sources or destinations.
Using these, you can modify the sample presented in this chapter to move
data between BigQuery and AppEngine. This approach is more flexible than
the Datastore backup import method presented in Chapter 11 because it
allows you to transform the data before insertion into BigQuery. This can
be useful when your Datastore entities have a complicated or nonuniform
structure.
In addition, AppEngine also supports transferring data in the reverse
direction, from BigQuery to Datastore. This can be useful to mirror data
in both locations. If your application requires efficient individual object
lookups, Datastore is a perfect solution; if you also require more complex
analytics, having the same data also in BigQuery allows fast queries over the
entire dataset. Be aware, however, that you will incur the costs of Datastore
writes when performing this operation.
The AppEngine MapReduce SDK does not contain direct BigQuery input
readers and writers as of the time of publication. It is quite likely, however,
that they will become available in future releases. When they do the process
will become even simpler—instead of supplying BigQuery job configurations
in code, you will be able to specify them in the config file, like the GCS
example.
Using BigQuery with Hadoop
Hadoop is more or less synonymous with Big Data. It started out as a
clone of Google MapReduce and Google File System (GFS) but has since
