log4g + logs not rotated and zipped

[#1]

hi all

we configured the following properties in Advanced ams-hbase-log4j

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}.log
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}.%i.log.gz
log4j.appender.RFA.triggeringPolicy.MaxFileSize=1000000
log4j.appender.RFA.rollingPolicy.maxIndex=10

but after metrics collector restart

we not see that hbase-ams rotated and sipped to .gz

please advice what could be wrong with the log4j properties ?

example of the logs that we get ,

-rw-r--r--  1 ams hadoop     45434111 Aug 16 17:07 hbase-ams-master-master.isys.com.log
-rw-r--r--  1 ams hadoop     2802 Aug 16 17:07 hbase-ams-master-master.isys.com.out

[#2]

OK no problem , we wait for you

[#3]

do you have some direction ? , if this issue cant be solved then it is ok , I will not do the log4j with zip option

[#4 · Accepted solution]

@Michael Bronson

Please try the below steps and check if it works or not? (I tested this on Ambari 2.6.2.2 and it worked fine without any issue)

Step1). Download and Setup the log4j extras Jar:

# mkdir /tmp/log4j_extras
# curl http://apache.mirrors.tds.net/logging/log4j/extras/1.2.17/apache-log4j-extras-1.2.17-bin.zip -o /tmp/log4j_extras/apache-log4j-extras-1.2.17-bin.zip
# cd /tmp/log4j_extras
# unzip apache-log4j-extras-1.2.17-bin.zip
# cp -f /tmp/log4j_extras/apache-log4j-extras-1.2.17/apache-log4j-extras-1.2.17.jar  /usr/lib/ams-hbase/lib

(OR) in ambari UI you will find the property "hbase_classpath_additional" inside the AMS configs to point to the log4j extras jar.

.

Step2). Edit the "ams-hbase-log4j template"

From ambari side navigate to amb ari UI

Ambari UI --> Ambari Metrics --> Configs --> Advanced --> "Advanced ams-hbase-log4j" --> ams-hbase-log4j template (text area)

And then "Comment out the following part:"

# Rolling File Appender
#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
#log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}
#log4j.appender.RFA.MaxFileSize=${hbase.log.maxfilesize}
#log4j.appender.RFA.MaxBackupIndex=${hbase.log.maxbackupindex}
#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n

.

Now add the following part to define your own RFA appender as following:

log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}-.%d{yyyyMMdd}.log.gz

.

Then restart the AMS service and then wait for some time to see the rolling.

I can see the following kind of Zip "hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz" creation at my end next day:

#  ls -lart /var/log/ambari-metrics-collector/
-rw-r--r--.  1 ams  hadoop  9177961 Aug 18 00:00 hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz
-rw-r--r--.  1 ams  hadoop  7520915 Aug 18 19:43 gc.log-201808170101
-rw-r--r--.  1 ams  hadoop  2506556 Aug 18 19:47 collector-gc.log-201808170220
-rw-r--r--.  1 ams  hadoop  1869725 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.log
drwxr-xr-x.  2 ams  hadoop     8192 Aug 19 01:14 .
-rw-r--r--.  1 ams  hadoop      234 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.out
-rw-r--r--.  1 ams  hadoop      678 Aug 19 01:16 ambari-metrics-collector-startup.out
-rw-r--r--.  1 ams  hadoop  6485518 Aug 19 01:16 SecurityAuth.audit
-rw-r--r--.  1 ams  hadoop     2008 Aug 19 01:17 ambari-metrics-collector.out
-rw-r--r--.  1 ams  hadoop 40038004 Aug 19 01:18 ambari-metrics-collector.log
-rw-r--r--.  1 ams  hadoop     6077 Aug 19 01:18 collector-gc.log-201808190114
-rw-r--r--.  1 ams  hadoop    89210 Aug 19 01:18 gc.log-201808190114

.

[#5]

@Jay - this already worked according to pappu ( see his answer about  cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ams-hbase/lib/  ) (  , my problem is about - ambari-metrics-collector.log that rotated but not zipped ) , see also my current conf ams-log4j.txt

[#6]

@Michael Bronson

We see that https://bz.apache.org/bugzilla/show_bug.cgi?id=36384. which says that "Configuring triggering/rolling policies should be supported through properties" hence you will need to make sure that your are using the log4j JAR of version "log4j-1.2.17.jar" (instead of using the "log4j-1.2.15.jar")

Hence make sure that your AMS collector is not using old version of log4j

# mv /usr/lib/ambari-metrics-collector/log4j-1.2.15.jar /tmp/
# cp -f /usr/lib/ams-hbase/lib/log4j-1.2.17.jar /usr/lib/ambari-metrics-collector/

.

Now also make sure to copy the "log4j-extras-1.2.17.jar"

# cp -f /tmp/log4j_extras/apache-log4j-extras-1.2.17/apache-log4j-extras-1.2.17.jar  /usr/lib/ambari-metrics-collector/

.

Now edit the"ams-log4j" via ambari as following;

Ambari UI --> Ambari Metrics --> Configs --> Advanced --> "Advanced ams-log4j" --> ams-log4j template (text area)

.

OLD Value

# Direct log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=${ams.log.dir}/${ams.log.file}
log4j.appender.file.MaxFileSize={{ams_log_max_backup_size}}MB
log4j.appender.file.MaxBackupIndex={{ams_log_number_of_backup_files}}
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

.

CHANGED VALUE

log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.file.rollingPolicy.maxIndex={{ams_log_number_of_backup_files}}
log4j.appender.file.rollingPolicy.ActiveFileName=${ams.log.dir}/${ams.log.file}
log4j.appender.file.rollingPolicy.FileNamePattern=${ams.log.dir}/${ams.log.file}-%i.gz
log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.file.triggeringPolicy.MaxFileSize=1048576
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

Notice: Here i am hard coding the value for property "log4j.appender.file.triggeringPolicy.MaxFileSize"for testing to something like "1048576" (around 1MB) because triggering policy does not accept values in KB/MB format hence i am putting the values in Bytes. You can have your own value defined there.

.