Skip to main content
sen_ke
sen_ke
Gold ⭐️
January 21, 2019
Solved

May i delete /var/log/hadoop/hdfs files

Hi,

there are many log files in /var/log/hadoop/hdfs, and file name like hdfs-audit.log.2018-06-01,

and some file name like hadoop-hdfs-namenode-hostname are very large.

may i delete it all?

(this host running namenode, ranger, datanode flume and node manager.)

and if any configure can setting those log as auto cycling or auto delete?

Thanks

jsensharma
jsensharmaAnswer
Platinum ⭐️⭐️
January 21, 2019

@Sen Ke

Logs for certain period is always useful to retain so that in case of any analysis we have the logs to verify if anything went unexpected.

Some of the logs like "hdfs-audit.log" are important as they contains all the auditing data of HDFS access.

However if you want to delete old data Logs then you can delete them as it wont cause any service interruption.

But the best approach will be to implement the Log4j Extra functionality for your logging so that the old logs will be automatically compressed and saved + rolled. As those logs are basically Text files so the compression happens greatly and compressed log size is 10-15 times lower than the original logs.

Please refer to the following article to know more about it:

https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th.html

View in thread

2 replies

jsensharma
Platinum ⭐️⭐️
jsensharmaAnswer
Platinum ⭐️⭐️
January 21, 2019

@Sen Ke

Logs for certain period is always useful to retain so that in case of any analysis we have the logs to verify if anything went unexpected.

Some of the logs like "hdfs-audit.log" are important as they contains all the auditing data of HDFS access.

However if you want to delete old data Logs then you can delete them as it wont cause any service interruption.

But the best approach will be to implement the Log4j Extra functionality for your logging so that the old logs will be automatically compressed and saved + rolled. As those logs are basically Text files so the compression happens greatly and compressed log size is 10-15 times lower than the original logs.

Please refer to the following article to know more about it:

https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th.html

Translate
sen_ke
sen_keAuthor
Gold ⭐️
January 22, 2019

@Jay Kumar SenSharma

Thanks for your reply, i deleted some logs to recover Namenode service status to running (service can't start when log space full),

and my logs over 50GB per year, so i will study log4j extra to save space usage.

Translate
Terms & ConditionsAccessibility statement