Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Edit Logs piling up under nn current directory and using lot of space on filesystem!!!1

Edit Logs piling up under nn current directory and using lot of space on filesystem!!!1

Contributor

Hi,

 

 

The edit logs are growing under the nn current directory, which is causing lot of space to be occupied on file system on which name node resides, I am worried this may lead to break down of cluster? It is been growing from the day we installed cdh 5.3.2.......Please help!!!!!

 

 

What can I do with the edit logs, shall I move them to some other place ? Please suggest, or there is a way to reduce the increasing edit logs day by day........Any suggestions!!!!!!!

1 REPLY 1

Re: Edit Logs piling up under nn current directory and using lot of space on filesystem!!!1

Cloudera Employee

Hello,
You may want to look into your secondary/standby NameNode (NN). There are 2 files to keep in mind: fsimage and edits. Fsimage contains the snapshot of the filesystem namespace and edits contains all the changes of the current fsimage file.

 

The secondary NN talks to active NN if either of the following conditions are met: pre-configured time has elapsed since last checkpoint or number of edits. When this event triggers, the active NN rolls a new edit file and secondary NN combines (in its memory) fsimage and edits, and send updated fsimage to active NN. If for any reason secondary NN is not able to peform checkpoints (or talk to the primary NN), active NN won't be able to roll the edits file and it would keep on increasing.


You can read more about this process in this blog: http://blog.cloudera.com/blog/2014/03/a-guide-to-checkpointing-in-hadoop/

 

If that is not the cause, then there is a tunable to hown many edits files are retained
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml.