Database Reference
In-Depth Information
•
Source
: Location from which bytes will be read
•
Destination
: Table that will be modified on success
•
Format
: Instructions on how to turn the bytes into valid records
The reason it is useful to consider these separately is that they can, for
the most part, be varied independently. In addition, the settings you select
for each of these is driven by different considerations.
Source
is dictated
by where your data currently exists,
destination
by how you would like
to structure your data in the service, and
format
by what you can most
conveniently generate or the schema of your table. Now dive into each of
these sections to understand what BigQuery supports.
Listing 6.1a
:
(run_load.py)
'''Common functions used to execute load jobs.'''
import json
import time
def start_and_wait(jobs, project_id, load,
media_body=None):
'''Run a load job with the given specification.
jobs: client for the jobs collection in the
service.
project_id: project ID under which the job will
run.
load: the load job configuration.
media_body: optional media object, ie file, to
upload.
'''
start = time.time()
job_id = 'ch06_%d' % start
# Create the job.
result = jobs.insert(
projectId=project_id,
body={
'jobReference': {
'jobId': job_id
},
'configuration': {
