Database Reference
In-Depth Information
Listing 13-2.
Storage account key
<property>
<name>fs.azure.account.key.democluster.blob.core.windows.net</name>
<value>YourStorageAccountKey</value>
</property>
There are also several Azure throttling factors and blob IO buffer parameters that can be set through the
core-site.xml
file. They are outlined in Listing 13-3.
Listing 13-3.
Azure throttling factors
<property>
<name>fs.azure.selfthrottling.write.factor</name>
<value>1.000000</value>
</property>
<property>
<name>fs.azure.selfthrottling.read.factor</name>
<value>1.000000</value>
</property>
<property>
<name>fs.azure.buffer.dir</name>
<value>/tmp</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
■
Note
azure throttling is discussed in the section “Windows azure storage” later in this chapter.
mapred-site.xml
The
mapred-site.xml
file has the configuration settings for MapReduce services. It contains parameters for the
JobTracker and TaskTracker processes. These parameters determine where the MapReduce jobs place their
intermediate files and control files, the virtual memory usage by the Map and Reduce jobs, the maximum numbers of
mappers and reducers, and many such settings.
In the case of a poorly performing job, optimizations such as moving the intermediate files to a fast Redundant
Array of Inexpensive Disks (RAID) can be really helpful. Also, in certain scenarios when you know your job well, you
may want to control the number of mappers or reducers being spawned for your job or increase the default timeout
that is set for Map jobs. Listing 13-4 shows a few of the important attributes in
mapred-site.xml
.
