Database Reference
In-Depth Information
examples of how table decorators can be used for data management; here
you see how they're useful in reading your data out of BigQuery.
Snapshot Decorators
Table snapshot decorators were discussed in Chapter 11. They can be used
anywhere a table is read (in the API, in a query, and so on) to refer to a
historical snapshot of a table at a particular time. You can use a snapshot
decorator by adding
@timestamp
to a table name, where
timestamp
is
the number of milliseconds since the POSIX epoch (that is, January 1st,
1970 GMT). For example,
publicdata:samples.wikipedia@1386465812000
is the BigQuery
public sample table as of 1:23 AM GMT on December 8, 2013. The snapshot
time must be within the last 7 days (this snapshot only worked until
December 15, 2013, for example), but 7 days should be plenty of time to read
a table. Note that if you're reading via page tokens, no snapshot decorator is
needed because the page token implicitly specifies a snapshot time.
If you're going to read a table in parallel via row indexes, and that table
might be changing, you should use a snapshot decorator. This way all the
reads will be based on the same version of the table. To find a starting
snapshot time, first read as of the current time. The Python
time.time()
method returns the number of seconds since the POSIX epoch as a
floating-point value. To get the current snapshot time, just multiply the
current time by 1000 and cast to an integer, as in:
timestamp =
int(1000 * time.time())
. Then, each parallel worker should use the
same snapshot time so that row 1,000,000 means the same thing for all
readers.
Listing 12.5
shows how to get a good snapshot time and use it in a parallel
TableData.list()
operation. The net effect of this listing is the same as
in 12.3; however, in this case you read the table directly instead of extracting
it first to GCS.
Listing
12.5
:
Reading
from
a
table
by
index
in
parallel
(tabledata_index.py)
import os
import sys
import time
