where are hadoop log files ?

[#1]

the hadoop-env.sh file is showing HADOOP_HOME to be "/usr/hdp/current/hadoop-client" but on hadoop forum I am seeing notes that hdp home is /usr/hdp/<current-release> in my case "/usr/hdp/2.5.0.0-124" .

so two questions :

1- where should HADOOP_HOME point to ?

2- where should HADOOP_CON_DIR point to ?

3- where are the hadoop log files ? as I am seeing to debug YARN Isues I need to look into hadoop log files

thanks

[#2]

@Sami Ahmad

1- where should HADOOP_HOME point to ?

HADOOP_HOME usually points to hadoop installation dir. For hortonworks it is /usr/hdp/current/hadoop/ or /usr/hdp/current/hadoop-client/

2- where should HADOOP_CON_DIR point to ?

/etc/hadoop/conf or some sym linc in /etc/hadoop/

3- where are the hadoop log files ? as I am seeing to debug YARN Isues I need to look into hadoop log files

By default all logs are under /var/log (Ex /var/log/hadoop)

[#3]

thanks

where would be yarn log files ? as the yarn directory is empty

[root@hadoop1 ~]# ls -ltr /var/log/hadoop/yarn total 0 [root@hadoop1 ~]#

[#4]

You have to do this on the node where resource manager runs. on other nodes, this directory would be empty only. so this should be good.

[#5 · Accepted solution]

Hi @Sami Ahmad

For Question 3, The log files location can be found out by checking hadoop-env.sh or yarn-env.sh file which are present in HADOOP_CONF_DIR which is usually /etc/hadoop/conf/ .

Sample yarn-env.sh

export HADOOP_YARN_HOME=/usr/hdp/current/hadoop-yarn-nodemanager   
export YARN_LOG_DIR=/grid/0/log/yarn/$USER       
export YARN_PID_DIR=/var/run/hadoop-yarn/$USER 

yarn.nodemanager.log-dirs present in yarn-site.xml(Inside HADOOP_CONF_DIR): Determines where the container-logs are stored on the node when the containers are running

[#6]

@Sami Ahmad

I believe that you are trying to debug yarn service components such as node manager, resource manager, if so you could find the logs inside /var/log/hadoop-yarn/$USER.

For instance if your yarn process user account is yarn then the logs would be in /var/log/hadoop-yarn/yarn