Database Reference
In-Depth Information
Resources Exceeded
Considering that every computational system has its limitations, it should
not come as a surprise that there are some queries that simply do not fit
within the resource constraints of BigQuery. However, the mode of failure
may be somewhat surprising. Many database systems will not by default
fail a query because it is too expensive. Instead they will attempt to make
progress given the existing resources even if it means consuming all the
resources at the expense of other queries in the system, and they may take
an unreasonably long time. BigQuery aims to execute every query within
a time frame that is proportional to the data being scanned, so it fails the
query relatively quickly rather than processing it indefinitely. This approach
has merits, but you also have to cope with its drawbacks.
It is useful to understand at a high level why some queries fail to fit within
the constraints of the system. Chapter 10 covers the execution model in
detail, and it is useful to keep it in mind when considering resource errors.
To achieve good performance, the system avoids using a disk once the data
has been read from a disk. The intermediate results in each node are kept
in volatile memory, and every query is subject to a limit on the memory it
uses in a single node. When a query exceeds this limit on any single node,
the query fails and reports a resources exceeded exception.
errorResult": {
"reason": "resourcesExceeded",
"message": "Resources exceeded."
}
Retrying the unit of work on a different node is not going to help matters
because the limit is a static value, so any other node will encounter the same
limit. For the same reason, retrying the query is not going to help matters.
There are a few different query features that have high memory
requirements within a single execution node:
GROUP BY clauses that generate a large number of distinct groups
• Aggregation functions that require memory proportional to the number
of input values
• Join operations that generate a greater number of outputs than inputs
Search WWH ::




Custom Search