question Re: Controlling the hadoop logs using log 4j in Archives of Support Questions (Read Only)

Controlling the hadoop logs using log 4j

khadeermhmd — Fri, 31 Mar 2017 02:36:46 GMT

Do any have the log 4j properties for controlling the hadoop logs

Re: Controlling the hadoop logs using log 4j

jsensharma — Fri, 31 Mar 2017 02:39:58 GMT

@khadeer mhmd

References:

https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.html

https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th.html

You can get the logging config from ambari UI.

Login to Ambari
Goto the config tab of HDFS component.
Filter for Advanced hdfs-log4j

In case of ambari managed cluster it should be managed via ambari only else it will override out manual changes made to the log4j.propeties file on component restart.

/etc/hadoop/conf/log4j.properties

Re: Controlling the hadoop logs using log 4j

jsensharma — Fri, 31 Mar 2017 02:43:33 GMT

@khadeer mhmd

Additional references: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0-Win/bk_HDP_Install_Win/content/ref-8896fe3b-8788-4a88-b7cc-d77d6f2481fc.1.html

log4j.properties:
Use the log4j.properties file to modify the log purging intervals of the HDFS logs. This file defines logging for all the Hadoop services. It includes, information related to appenders used for logging and layout. For more details, see the log4j documentation.

Re: Controlling the hadoop logs using log 4j

namaheshwari — Fri, 31 Mar 2017 03:12:48 GMT

@khadeer mhmd - You should use the following properties of the RollingFileAppender to efficiently control the Size and the number of backup index of old log files:

maxFileSize:This is the critical size of the file above which the file will be rolled. Default value is 10 MB.

maxBackupIndex: This property denotes the number of backup files to be created. Default value is 1.

More details in :

https://community.hortonworks.com/questions/89171/hdfs-audit-log-file-size-issues.html#comment-89639