Database Reference
In-Depth Information
is most useful when you run queries that are actually large. These limits do
mean, however, that you can never run more than one large query at once.
If your query has already been cached, it won't need to actually be executed
by the compute cluster, so it won't count against any of the rate limits.
It
will
however, count toward your daily limit. This is to prevent people
from abusing the cache. If you need to access the same data over and over
again, you should use
TableData.list()
to read the data directly without
running the query or caching the data locally.
Batch Query Limits
When you hit a rate limit in a query run at normal (
INTERACTIVE
) priority,
your query will fail, and you will need to try again later. The reasoning is that
INTERACTIVE
queries are intended to be, well, interactive. The assumption
is that if the queries were queued, people would experience degraded query
performance and think the queries were just running slowly rather than
realize that their queries are stuck in the queue.
Sometimes, however, you aren't actively waiting for the query to return;
maybe you're updating a dashboard that runs a large number of queries
once an hour. In these cases, it would be nice to just “fire and forget” your
queries; that is, start all your queries at once and not have to worry about
query pipes and large query sizes. Batch queries can help with this issue.
When you run a query at
BATCH
priority, it is not subject to any of the
rate limits discussed in the previous section. The only limitation is the total
number of allowed batch queries per day, which is set at 10,000.
That said, when you run
BATCH
queries, BigQuery runs them at a lower
priority. They are queued and execute when the system has spare capacity.
To make sure that your queries make forward progress, if a batch query
hasn't executed within 3 hours, it gets promoted to
INTERACTIVE
priority
and will run as soon as it can be scheduled.
Other Query Limitations
BigQuery works on Big Data, so there should be no limitations, right? Well,
not exactly. BigQuery can perform many operations faster than a relational
database because of its architecture (which is described in more detail in
Chapter 9). That said, there are some trade-offs—things that insert it may
seem like it should do, but it doesn't do well.
