Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can't find the MapReduce Log files

avatar
Explorer

I can't find the log files from my MapReduce jobs. I'm using MR2 in HortonWorks 2.4.3 sandbox I got from here.

In an effort to try to create the logs in one directory, I have set the following environment variables

export HADOOP_MAPRED_HOME=/home/hadoop
export HADOOP_YARN_HOME=/hadoop/yarn
export YARN_LOG_DIR=/hadoop/yarn/log
export HADOOP_LOG_DIR=/hadoop/yarn/log
export HADOOP_MAPRED_LOG_DIR=/hadoop/yarn/log

I'm not sure if setting these in my environment session has any effect when running the job. Presumably I have to set this in Ambari?

I do see the job history logs in /var/log/hadoop-mapreduce/mapred/. But I don't see the logs from my map reduce program itself. Following the link in Ambari for MapReduce JobHistory UI takes me to http://<>:19888/jobhistory which shows no jobs.

I have tried starting my mapreduce job using

yarn jar ./lib/<my>.jar <mapreduce driver class name> <input file name> <output hdfs dir name> <properties file>

and

hadoop jar ./lib/<my>.jar <mapreduce driver class name> <input file name> <output hdfs dir name> <properties file>

Same result with both. Except with the second I get a warning message

WARNING: Use "yarn jar" to launch YARN applications.

I see nothing in the resource manager (http://<>:8088) or node manager (http://<>:8042) UI

According to Simplifying user-logs management and access in YARN, I should be using Application Ids with the yarn commands. But where are these Application Ids set when I invoke the Map Reduce program as I do above?

Secondly how am I setting properties like yarn.nodemanager.log-dirs By default it's ${yarn.log.dir}/userlogs Where is ${yarn.log.dir} set?

I think this is what I'm missing (along with how to set/get an application Id from a MapReduce program).

I think I'm missing something obvious so a nudge in the right direction would be appreciated.

1 ACCEPTED SOLUTION

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Super Guru

can you check with yarn logs -applicationId <application ID>

avatar
Explorer

Rajkumar, that's the problem I have. Where do I get the Application Id from? Is there some different command that I need to use to start the MapReduce job or some flags I need to use? The same job used to work in MapR hadoop 1 with the logs going to the syslog directory. In Ambari, I'm seeing no applications at all.

avatar
Expert Contributor

Hi Milind Rao. You can use Ambari -> YARN -> Quick Links -> ResourcesManager UI for ApplicationId

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login