Logging in HDInsight - Pro Microsoft HDInsight: Hadoop on Windows - page 185

I am running all the services for my demo cluster in the name node itself. My set of Hadoop service log files for
cluster version 2.1 looks like those shown in Figure 11-2 .
Figure 11-2. Hadoop service log files
The service log files are common for all the services listed in Table 11-1 . That means that each of the
service-based projects, like Hive and so on, have these sets of service log files in their respective bin folders.
Hadoop log4j Log Files
When you consider that HDInsight is essentially a wrapper on top of core Hadoop; it is no surprise that it continues to
embrace and support the traditional logging mechanism by Apache. You should continue to investigate these log files
for most of your job failures, authentication issues, and service communication issues.
In the HDInsight distribution on Azure, these logs are available in the C:\apps\dist\hadoop-1.2.0.1.3.1.0-06\
logs directory of the respective nodes for Hadoop. By default, the log files are recycled daily at midnight; however,
historical versions are preserved. The old file names are appended with a _<timestamp> value each time they are
purged and rolled over. The most current log files are in the format hadoop-namenode-<Hostname>.log ,
hadoop-datanode-<Hostname>.log , hadoop-secondarynamenode-<Hostname>.log , and so on. The Hostname is the
host where the service is running on. These are pretty similar to the service error-log files discussed in the previous
section and record the stack traces of the service failures. A typical name node log looks similar to the following
snippet after a successful startup.
2013-08-16 21:32:39,324 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = <HostName>/<IP Address>
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.2.0
STARTUP_MSG: build = [e-mail]:hortonworks/hadoop-monarch.git on branch (no branch)
-r 99a88d4851ce171cf57fa621910bb293950e6358; compiled by 'jenkins' on Fri Jul 19 22:07:17
Coordinated Universal Time 2013
************************************************************/