Database Reference
In-Depth Information
Table 10-4. MapReduce job memory properties (set by the client)
Property name
Type
Default
value
Description
1024
The amount of memory for map containers.
mapreduce.map.memory.mb
int
1024
The amount of memory for reduce containers.
mapreduce.reduce.memory.mb int
The JVM options used to launch the container
process that runs map and reduce tasks. In addi-
tion to memory settings, this property can in-
clude JVM properties for debugging, for ex-
ample.
mapred.child.java.opts
String -
Xmx200m
The JVM options used for the child process that
runs map tasks.
mapreduce.map.java.opts
String -
Xmx200m
The JVM options used for the child process that
runs reduce tasks.
mapreduce.reduce.java.opts String -
Xmx200m
For example, suppose
mapred.child.java.opts
is set to
-Xmx800m
and
mapre-
duce.map.memory.mb
is left at its default value of 1,024 MB. When a map task is
run, the node manager will allocate a 1,024 MB container (decreasing the size of its pool
by that amount for the duration of the task) and will launch the task JVM configured with
an 800 MB maximum heap size. Note that the JVM process will have a larger memory
footprint than the heap size, and the overhead will depend on such things as the native lib-
raries that are in use, the size of the permanent generation space, and so on. The important
thing is that the physical memory used by the JVM process, including any processes that
it spawns, such as Streaming processes, does not exceed its allocation (1,024 MB). If a
container uses more memory than it has been allocated, then it may be terminated by the
node manager and marked as failed.
YARN schedulers impose a minimum or maximum on memory allocations. The default
minimum is 1,024 MB (set by
yarn.scheduler.minimum-allocation-mb
), and
the default maximum is 8,192 MB (set by
yarn.scheduler.maximum-
allocation-mb
).
There are also virtual memory constraints that a container must meet. If a container's vir-
tual memory usage exceeds a given multiple of the allocated physical memory, the node
manager may terminate the process. The multiple is expressed by the
yarn.nodemanager.vmem-pmem-ratio
property, which defaults to 2.1. In the ex-
ample used earlier, the virtual memory threshold above which the task may be terminated
is 2,150 MB, which is 2.1 × 1,024 MB.
