question where are hadoop log files ? in Support Questions

where are hadoop log files ?

aliyesami — Fri, 06 Jan 2017 04:39:14 GMT

the hadoop-env.sh file is showing HADOOP_HOME to be "/usr/hdp/current/hadoop-client" but on hadoop forum I am seeing notes that hdp home is /usr/hdp/<current-release> in my case "/usr/hdp/2.5.0.0-124" .

so two questions :

1- where should HADOOP_HOME point to ?

2- where should HADOOP_CON_DIR point to ?

3- where are the hadoop log files ? as I am seeing to debug YARN Isues I need to look into hadoop log files

thanks

Re: where are hadoop log files ?

ajay_kumar — Fri, 06 Jan 2017 07:21:17 GMT

@Sami Ahmad

1- where should HADOOP_HOME point to ?

HADOOP_HOME usually points to hadoop installation dir. For hortonworks it is /usr/hdp/current/hadoop/ or /usr/hdp/current/hadoop-client/

2- where should HADOOP_CON_DIR point to ?

/etc/hadoop/conf or some sym linc in /etc/hadoop/

3- where are the hadoop log files ? as I am seeing to debug YARN Isues I need to look into hadoop log files

By default all logs are under /var/log (Ex /var/log/hadoop)

Re: where are hadoop log files ?

aliyesami — Sat, 07 Jan 2017 00:41:59 GMT

thanks

where would be yarn log files ? as the yarn directory is empty

[root@hadoop1 ~]# ls -ltr /var/log/hadoop/yarn total 0 [root@hadoop1 ~]#

Re: where are hadoop log files ?

ssathish — Sat, 07 Jan 2017 03:58:57 GMT

Hi @Sami Ahmad

For Question 3, The log files location can be found out by checking hadoop-env.sh or yarn-env.sh file which are present in HADOOP_CONF_DIR which is usually /etc/hadoop/conf/ .

Sample yarn-env.sh

export HADOOP_YARN_HOME=/usr/hdp/current/hadoop-yarn-nodemanager   
export YARN_LOG_DIR=/grid/0/log/yarn/$USER       
export YARN_PID_DIR=/var/run/hadoop-yarn/$USER

yarn.nodemanager.log-dirs present in yarn-site.xml(Inside HADOOP_CONF_DIR): Determines where the container-logs are stored on the node when the containers are running

Re: where are hadoop log files ?

ssivachandran — Mon, 09 Jan 2017 14:02:59 GMT

@Sami Ahmad

I believe that you are trying to debug yarn service components such as node manager, resource manager, if so you could find the logs inside /var/log/hadoop-yarn/$USER.

For instance if your yarn process user account is yarn then the logs would be in /var/log/hadoop-yarn/yarn

Re: where are hadoop log files ?

Throw — Wed, 08 Jan 2020 09:46:52 GMT

You have to do this on the node where resource manager runs. on other nodes, this directory would be empty only. so this should be good.