Log4j Settings for HDP services

[#1]

Log4j settings

Ambari uses sets of properties called Log4j properties to control logging activities for each service running in your Hadoop cluster. Initial, default values for each property reside in a <service_name>-log4j template file.

To access the default Log4j settings for a service; in Ambari Web, you can browse to <Service_name> > Configs > Advanced <service_name>-log4j.

Often, user might want to configure the number of backup log files or limit the maximum size of a log file. Prior to Ambari-2.5, this had to be done by manually searching for the property in the log4j template Text Area and modifying it. This is quite inconvenient.

Ambari-2.4 and before

Ambari-2.5

With Ambari-2.5, a simplified log rotation configuration is provided i.e Log4j properties and values that limit the size and number of backup log files for each service now appear above the log4j template file as separate fields.

Steps

1.Edit the values for the <service_name> backup file size and <service_name> # of backup files properties.

2.Click Save.

Typically, once you modify these properties a restart indicator shows up next to the relevant components. Restarting components in the service pushes the configuration properties to each host running components for that service.

Fresh Install Ambari-2.5

These settings are readily available for any fresh install of Ambari-2.5

Ambari 2.x (x<5)

Stack Upgrade

These settings will be added during any stack upgrade.

Ambari-2.x to Ambari-2.5

After upgrading Ambari from version 2.x to 2.5.0, if you want to utilize the simplified log rotation configuration, you must update configurations for all services in your cluster, using the following steps

HDFS

1. In Ambari Web, browse to HDFS > Configs > Advanced.
2. Scroll down to Custom HDFS-log4j.
3. In Custom HDFS-log4j, click Add Property.
4. In Add Property, type the following properties and values: hadoop_security_log_max_backup_size = 256 hadoop_security_log_number_of_backup_files = 20 hadoop_log_max_backup_size = 256 hadoop_log_number_of_backup_files = 10
5. Click Add.
6. Browse to Advanced hdfs-log4j .
7. In Advanced hdfs-log4j content section, find and replace the following properties and values: Find: hadoop.security.log.maxfilesize= <value>MB Replace: hadoop.security.log.maxfilesize={{hadoop_security_log_max_backup_size}}MB Find: hadoop.security.log.maxbackupindex=20 Replace: hadoop.security.log.maxbackupindex={{hadoop_security_log_number_of_backup_files}} Find: log4j.appender.RFA.MaxFileSize=<value>MB Replace: log4j.appender.RFA.MaxFileSize={{hadoop_log_max_backup_size}}MB Find: log4j.appender.RFA.MaxBackupIndex=<value> Replace: log4j.appender.RFA.MaxBackupIndex={{hadoop_log_number_of_backup_files}}
8. In Configs, click Save.Restart HDFS, as prompted.

For more information refer :

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-upgrade/content/upgrading_log_rot...