Database Reference
In-Depth Information
a URI with a wildcard is called a
glob
. Note that wildcard characters are only
valid in the object name portion of the GCS URI. The glob features shown in
Table 6.1
are supported:
Table 6.1
Glob Patterns
Pattern
Matches
Any single character except for /
?
*
Any sequence of characters except for /
[
characters
]
Set of characters, which can include ranges, for example
[a-z]
[!
characters
]
Characters
not
in the list
Even though GCS does not support explicit directories, the wildcards still do
not match the “/” character so globs match files like they would in a regular
file system. For example,
gs://bucket/f*
will match
gs://bucket/
file
and exclude
gs://bucket/f/other
. There is an upper limit of
1000 files per load job,
after
glob expansion. Additional limits apply to the
number of bytes in a single file and the total number of bytes, which are
covered when discussing the limits and quotas that apply to load jobs.
Access control works as you would expect; the creator of the job must have
reader access for all the files you enumerate in the
sourceUris
list. If you
include a glob in the list, you must also have reader access on the bucket,
which grants permission to list the contents of a bucket.
Because GCS and BigQuery are both a part of the Google Cloud Platform,
it is easy to forget that loading data from GCS into BigQuery creates an
additional copy of your data. The data stored in GCS is unchanged by the
BigQuery load job, and after the job has completed, deleting the GCS files
will have no effect on the data in BigQuery. Again it is useful to consider the
analogy to your local file system. If you restore the contents of a database
from a backup file, the database ends up storing a copy of the data in a
manner suitable for its operation. The backup remains unchanged. GCS and
BigQuery operate in a similar manner.
Resumable Uploads
When using GCS as the data source, you need to have transferred the bytes
before issuing the load job creation request. BigQuery also supports issuing
