Database Reference
In-Depth Information
Hive
SELECT
commands with aggregate functions (
count
,
sum
, and so on), or having conditions with column
filters, invoke MapReduce jobs to get the command output. For example, if you execute the query
SELECT count(*)
from hivesampletable
, you would see output with MapReduce job details as shown in Listing 13-11.
Listing 13-11.
MapReduce Operation Log
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201311120315_0003,
Tracking URL =
http://jobtrackerhost:50030
/jobdetails.jsp?jobid=job_201311120315_0003
Kill Command = c:\apps\dist\hadoop-1.2.0.1.3.0.1-0302\bin\
hadoop.cmd job -kill job_201311120315_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-11-16 17:28:38,336 Stage-1 map = 0%, reduce = 0%
2013-11-16 17:28:42,354 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.093 sec
2013-11-16 17:28:43,363 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.093 sec
2013-11-16 17:28:44,376 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.093 sec
2013-11-16 17:28:45,388 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.093 sec
2013-11-16 17:28:46,395 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.093 sec
2013-11-16 17:28:47,401 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.093 sec
2013-11-16 17:28:48,409 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.093 sec
2013-11-16 17:28:49,416 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.093 sec
2013-11-16 17:28:50,423 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 3.093sec
2013-11-16 17:28:51,429 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 3.093sec
2013-11-16 17:28:52,445 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 5.514 sec
2013-11-16 17:28:53,453 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 5.514 sec
2013-11-16 17:28:54,462 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 5.514 sec
MapReduce Total cumulative CPU time: 5 seconds 514 msec
Ended Job = job_201311120315_0003
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 Cumulative CPU: 5.514 sec
HDFS Read: 245 HDFS Write: 6 SUCCESS
Total MapReduce CPU Time Spent: 5 seconds 514 msec
OK
59793
Time taken: 48.899 seconds, Fetched: 1 row(s)
As we see from the preceding output, the job that is created is
job_201311120315_0003
. Now, take a look
at the folder
C:\apps\dist\hadoop-1.2.0.1.3.0.1-0302\logs\
. In that folder, you should have a file named
job_201311120315_0003_conf.xml.
The content of that file gives information about all the environment variables
and configuration details for that MapReduce job.
