Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

where are hadoop log files ?

Solved Go to solution

where are hadoop log files ?

Master Collaborator

the hadoop-env.sh file is showing HADOOP_HOME to be "/usr/hdp/current/hadoop-client" but on hadoop forum I am seeing notes that hdp home is /usr/hdp/<current-release> in my case "/usr/hdp/2.5.0.0-124" .

so two questions :

1- where should HADOOP_HOME point to ?

2- where should HADOOP_CON_DIR point to ?

3- where are the hadoop log files ? as I am seeing to debug YARN Isues I need to look into hadoop log files

thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: where are hadoop log files ?

Hi @Sami Ahmad

For Question 3, The log files location can be found out by checking hadoop-env.sh or yarn-env.sh file which are present in HADOOP_CONF_DIR which is usually /etc/hadoop/conf/ .

Sample yarn-env.sh

export HADOOP_YARN_HOME=/usr/hdp/current/hadoop-yarn-nodemanager   
export YARN_LOG_DIR=/grid/0/log/yarn/$USER       
export YARN_PID_DIR=/var/run/hadoop-yarn/$USER 

yarn.nodemanager.log-dirs present in yarn-site.xml(Inside HADOOP_CONF_DIR): Determines where the container-logs are stored on the node when the containers are running

4 REPLIES 4
Highlighted

Re: where are hadoop log files ?

Rising Star
@Sami Ahmad

1- where should HADOOP_HOME point to ?

HADOOP_HOME usually points to hadoop installation dir. For hortonworks it is /usr/hdp/current/hadoop/ or /usr/hdp/current/hadoop-client/

2- where should HADOOP_CON_DIR point to ?

/etc/hadoop/conf or some sym linc in /etc/hadoop/

3- where are the hadoop log files ? as I am seeing to debug YARN Isues I need to look into hadoop log files

By default all logs are under /var/log (Ex /var/log/hadoop)

Re: where are hadoop log files ?

Master Collaborator

thanks

where would be yarn log files ? as the yarn directory is empty

[root@hadoop1 ~]# ls -ltr /var/log/hadoop/yarn total 0 [root@hadoop1 ~]#

Re: where are hadoop log files ?

Hi @Sami Ahmad

For Question 3, The log files location can be found out by checking hadoop-env.sh or yarn-env.sh file which are present in HADOOP_CONF_DIR which is usually /etc/hadoop/conf/ .

Sample yarn-env.sh

export HADOOP_YARN_HOME=/usr/hdp/current/hadoop-yarn-nodemanager   
export YARN_LOG_DIR=/grid/0/log/yarn/$USER       
export YARN_PID_DIR=/var/run/hadoop-yarn/$USER 

yarn.nodemanager.log-dirs present in yarn-site.xml(Inside HADOOP_CONF_DIR): Determines where the container-logs are stored on the node when the containers are running

Re: where are hadoop log files ?

Cloudera Employee

@Sami Ahmad

I believe that you are trying to debug yarn service components such as node manager, resource manager, if so you could find the logs inside /var/log/hadoop-yarn/$USER.

For instance if your yarn process user account is yarn then the logs would be in /var/log/hadoop-yarn/yarn

Don't have an account?
Coming from Hortonworks? Activate your account here