Reply
New Contributor
Posts: 4
Registered: ‎02-23-2017

How log files are organized and managed

[ Edited ]

Hi,

 

I have below basic questions on log file location in a ideal HW cluster setup,

 

1. Where exactly the hadoop system log of hadop master(s) and data nodes are stored?
for example,

a) where can i see consolidated system logs of Name node & individual Data nodes? what is the path?
can i see consolidated log on name node or should i refer both name node and data nodes?

 

b) where can i see consolidated system logs of RM (resource mgr) & NM (Node Mgr)? what is the path?
can i see consolidated log on RM & NM or should i refer both RM and NM?

 

2. Where exactly the hadoop application log of MR, Spark, YARN are stored?

 

a) where can i see consolidated YARN application logs?
assume that there are 100 YARN applications (Hive queries, Pig scripts etc) running in a day.
Can i see all the logs on one centralized location?

 

b) where can i see consolidated SPARK application logs?
assume that there are 100 Spark applications running in a day.
Can i see all the logs on one centralized location?

 

I hear, system & application logs can be stored on HDFS? is it a default HW behavior?

 

Apologies for this dump question. Appreciate your response.

 

Champion
Posts: 595
Registered: ‎05-16-2016

Re: How log files are organized and managed

I am going to give you some best place to start looking for hadoop dameon logs 

 

Log rotation and Logger Level information. 

/etc/hadoop/conf/log4j.properties. 

in Yarn-site.xml for Resource manager and Nodemanager Logs in the below tag

yarn.nodemanager.log-dirs
/var/log/hadoop-mapreduce/mapred/ (for History logs)

or

/var/log / - do a ls
you should see folders.

hadoop-0.20-mapreduce
hadoop-hdfs
hadoop-mapreduce
hadoop-yarn


Pig 

 

 By default, your failed job log is saved in the same directory you're in. 

 

Meantime  - You can always go on to Resourcemanger , Namenode web  UI to check all the logs. 

 

 

Posts: 394
Topics: 11
Kudos: 60
Solutions: 35
Registered: ‎09-02-2016

Re: How log files are organized and managed

@geramkumar1

 

The default location for log is /var/log. But it is customizable, so if you are using Cloudera Manager, I would recommend you to go to each service and check configuration path to double confirm

 

Ex: Cloudera Manager -> HDFS -> Configuration -> Category (left menu) -> Logs

 

Repeat this for all the required services

Announcements