Database Reference
In-Depth Information
Limits and Quotas
Load jobs are subject to two types of limits by the service:
• Limits on the total amount of data processed by a single job
• The rate at which import jobs can be submitted to the service
The limits on a single job are rarely encountered in practice because they are
large enough that network transfer is usually the bottleneck. The second set
of limits requires more attention because a large number of tables requiring
regular updates or a single table being updated concurrently can run into
these limits. The values specified in this section are subject to change. You
should refer to the service documentation for up-to-date information
Start by looking at the limits on a single job because they are simple to
describe.
An individual load job is restricted to a maximum of:
• 10,000 input files, only relevant when loading multiple files from GCS
• 1 TB total input bytes
In addition, there are limits on each file to ensure that processing can
be sufficiently parallelized. As discussed in the section on formats, some
formats can be split up for parallel processing and others cannot, so these
limits vary by format. GZIP compressed input is limited to a maximum of 1
GB per file. For uncompressed data the following limits apply:
• 4 GB for CSV data that may contain quoted newlines
• 1 TB for CSV data that does not contain quoted newlines
• 1 TB for newline delimited JSON
Although each compressed file is limited to 1 GB, the total job can still
reference up to 1 TB of input data. It is common to achieve compression
ratios greater than 10x on CSV, so the limit of 1 GB of input data per file
can translate to more than 10 GB of uncompressed CSV per file or 10 TB
for the entire job, which is a substantial volume of data. So if your primary
requirement is loading a large volume of data, the most effective choice is
generating compressed files between 100 MB and 1 GB in GCS.
