Database Reference
In-Depth Information
mentioned because you don't have to worry about creating subdirectories
before exporting a table.
There are a couple of other special options that you can use when specifying
a destination URI or URIs, which are described next.
Pattern Export Paths
BigQuery won't export files larger than 1 GB to a single filename. Because a
single file can't be written in parallel, a large destination file is inefficient to
emit. If you export large tables, you can use a pattern path, which allows as
many files to be written as necessary.
You shouldn't make any assumptions about the size or number of these files,
other than that they will all be smaller than 1 GB in size. Small tables will
likely just be exported as a single file, and larger tables will be broken into a
larger number of pieces. BigQuery often moves data around to make it easier
to query, so if you export the same table more than once, it might be split a
different way each time.
To create a pattern export path, specify the glob character
*
somewhere
in your
destinationUris
field. Glob patterns operate like command-line
glob matching (for example,
foo*.txt
), rather than regular expressions
(where you'd specify
foo.*\.txt
). The glob patterns will be replaced by a
shard number padded to twelve digits, which starts at
000000000000
and
increments by one for each file that is written out. For instance, if you pass
the filename
gs://bigquery-e2e/data/extract/zipcodes*.json
,
BigQuery may write out these objects:
gs://bigquery-e2e/data/extract/zipcodes0000000000.json
gs://bigquery-e2e/data/extract/zipcodes0000000001.json
gs://bigquery-e2e/data/extract/zipcodes0000000002.json
These files generally are written out in parallel (up to a limit, of course), so
using a glob pattern can drastically reduce the time it takes to export a table.
It is a best practice to always use a glob when exporting, since a glob pattern
allows you to export data without worrying about the size of the source table.
Partitioned Export
Nobody likes waiting around for data. Often the reason you want to export
your data is because you want to use it right away somewhere else. For
