Controlling the hadoop logs using log 4j

[#1]

Do any have the log 4j properties for controlling the hadoop logs

[#2]

@khadeer mhmd

References:

https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.h...

https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th....

You can get the logging config from ambari UI.

1. Login to Ambari
2. Goto the config tab of HDFS component.
3. Filter for Advanced hdfs-log4j

In case of ambari managed cluster it should be managed via ambari only else it will override out manual changes made to the log4j.propeties file on component restart.

/etc/hadoop/conf/log4j.properties 

.

[#3]

@khadeer mhmd

Additional references: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0-Win/bk_HDP_Install_Win/content/ref-8896fe3b...

• log4j.properties:

  Use the log4j.properties file to modify the log purging intervals of the HDFS logs. This file defines logging for all the Hadoop services. It includes, information related to appenders used for logging and layout. For more details, see the log4j documentation.

.

[#4 · Accepted solution]

@khadeer mhmd - You should use the following properties of the RollingFileAppender to efficiently control the Size and the number of backup index of old log files:

maxFileSize:This is the critical size of the file above which the file will be rolled. Default value is 10 MB.

maxBackupIndex: This property denotes the number of backup files to be created. Default value is 1.

More details in :

https://community.hortonworks.com/questions/89171/hdfs-audit-log-file-size-issues.html#comment-89639