Database Reference
In-Depth Information
The Case for Specifying Your Own Job ID
After you submit a job insert request, BigQuery starts processing the
job on your behalf. A number of things can go wrong that would
prevent BigQuery from communicating back to you the status of the job
unless you have a pre-arranged name to refer to. For instance, your
wireless network could disconnect. Your wired network could have a
hiccup. A BigQuery server could crash. Your machine could crash. A
traffic management algorithm could decide to drop packets that are
crossing intercontinental fiber-optic cables in order to make way for
something more important.
All these things may look, to you, like errors starting the job. However,
because BigQuery runs the job asynchronously, when you get an error,
it might actually still complete successfully. If the job is a query, maybe
this isn't important; you can just run the job again. But if the job were
appending data to a table, you want to know whether you need to run it
again or whether the import actually succeeded.
The way to avoid these types of problems is to create your own job ID
for each task. When you run the
Jobs.insert()
API, you can specify
a unique ID for the job that will be created. This can give you a way to
refer to the job if a connection error occurs. Usually, coming up with a
unique name is easier than it sounds—just use the current time or the
pathname of the file you're importing.
If you have a known job ID, you have two ways of figuring out whether
the job ran successfully. The first is the simplest: Just try running it
again with the same ID. Because job IDs are enforced to be unique, you
cannot run a job with the same name twice. If the job actually started,
the second call to
Jobs.insert()
will fail with an Already Exists
error. The following transcript shows an example of what happens
when you try to create the same job twice. (We first generate a job ID
based on the date so that it will start out as unique.)
$
JOB_ID=job_$(date +"%s")
$
bq --job_id=${JOB_ID} query --max_rows=0 "SELECT
