@Michael Bronson
MapReduce uses RFA appender. Which is defined inside the "Advanced hdfs-log4j"
Ambari UI --> HDFS --> Configs --> Advanced --> "Advanced hdfs-log4j" --> hdfs-log4j template (Text area)
You will find something like this. By default you can see it like following:
#
# Rolling File Appender
#
log4j.appender.RFA=org.apache.log4j.RollingFileAppender
log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
# Logfile size and and 30-day backups
log4j.appender.RFA.MaxFileSize={{hadoop_log_max_backup_size}}MB
log4j.appender.RFA.MaxBackupIndex={{hadoop_log_number_of_backup_files}}
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
NOTE:
However making changes to "RFA" appender will not be good because this
appender is also being used by many other components.
So if you want just want to limit the number of logfiles based on size for MapReduce hostory server only then you can try this :
1. Create a new Appender like "RFA_MAPRED" inside the somewhere just below the RFA appender which we posted above.
Ambari UI --> HDFS --> Configs --> Advanced --> "Advanced hdfs-log4j" --> hdfs-log4j template (Text area)
Add "RFA_MAPRED" as following:
#
# MapReduce Rolling File Appender (ADDED MANUALLY)
#
log4j.appender.RFA_MAPRED=org.apache.log4j.RollingFileAppender
log4j.appender.RFA_MAPRED.File=${hadoop.log.dir}/${hadoop.log.file}
log4j.appender.RFA_MAPRED.MaxFileSize=100MB
log4j.appender.RFA_MAPRED.MaxBackupIndex=3
log4j.appender.RFA_MAPRED.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA_MAPRED.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n
log4j.appender.RFA_MAPRED.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
2. Now Save the config changes. Click Save Button.
3. Now we need to edit the "Advanced mapred-env"
Ambari UI --> MapReduce --> Configs --> Advanced --> "Advanced mapred-env" --> mapred-env template
Change the following line to use the "RFA_MAPRED" appender instead of default "RFA" (Single line change)
Old value:
export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA
New value:
export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA_MAPRED
4. Save and restart all required services to get this change reflected.
5. You can also verify that MapReduce is now using the correct Appender by running the following Command on Job History Server host.
# ps -ef | grep JobHistoryServer
. You should see -Dhadoop.root.logger=DEBUG,RFA_MAPRED in the output of the above command.
