Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

log4g + logs not rotated and zipped

avatar

hi all

we configured the following properties in Advanced ams-hbase-log4j

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}.log
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}.%i.log.gz
log4j.appender.RFA.triggeringPolicy.MaxFileSize=1000000
log4j.appender.RFA.rollingPolicy.maxIndex=10

but after metrics collector restart

we not see that hbase-ams rotated and sipped to .gz

please advice what could be wrong with the log4j properties ?

example of the logs that we get ,

-rw-r--r--  1 ams hadoop     45434111 Aug 16 17:07 hbase-ams-master-master.isys.com.log
-rw-r--r--  1 ams hadoop     2802 Aug 16 17:07 hbase-ams-master-master.isys.com.out
Michael-Bronson
1 ACCEPTED SOLUTION

avatar
Master Mentor

@Michael Bronson

Please try the below steps and check if it works or not? (I tested this on Ambari 2.6.2.2 and it worked fine without any issue)

Step1). Download and Setup the log4j extras Jar:

# mkdir /tmp/log4j_extras
# curl http://apache.mirrors.tds.net/logging/log4j/extras/1.2.17/apache-log4j-extras-1.2.17-bin.zip -o /tmp/log4j_extras/apache-log4j-extras-1.2.17-bin.zip
# cd /tmp/log4j_extras
# unzip apache-log4j-extras-1.2.17-bin.zip
# cp -f /tmp/log4j_extras/apache-log4j-extras-1.2.17/apache-log4j-extras-1.2.17.jar  /usr/lib/ams-hbase/lib

(OR) in ambari UI you will find the property "hbase_classpath_additional" inside the AMS configs to point to the log4j extras jar.

.

Step2). Edit the "ams-hbase-log4j template"

From ambari side navigate to amb ari UI

Ambari UI --> Ambari Metrics --> Configs --> Advanced --> "Advanced ams-hbase-log4j" --> ams-hbase-log4j template (text area)

And then "Comment out the following part:"

# Rolling File Appender
#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
#log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}
#log4j.appender.RFA.MaxFileSize=${hbase.log.maxfilesize}
#log4j.appender.RFA.MaxBackupIndex=${hbase.log.maxbackupindex}
#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n

.

Now add the following part to define your own RFA appender as following:

log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}-.%d{yyyyMMdd}.log.gz

.

Then restart the AMS service and then wait for some time to see the rolling.

I can see the following kind of Zip "hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz" creation at my end next day:

#  ls -lart /var/log/ambari-metrics-collector/
-rw-r--r--.  1 ams  hadoop  9177961 Aug 18 00:00 hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz
-rw-r--r--.  1 ams  hadoop  7520915 Aug 18 19:43 gc.log-201808170101
-rw-r--r--.  1 ams  hadoop  2506556 Aug 18 19:47 collector-gc.log-201808170220
-rw-r--r--.  1 ams  hadoop  1869725 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.log
drwxr-xr-x.  2 ams  hadoop     8192 Aug 19 01:14 .
-rw-r--r--.  1 ams  hadoop      234 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.out
-rw-r--r--.  1 ams  hadoop      678 Aug 19 01:16 ambari-metrics-collector-startup.out
-rw-r--r--.  1 ams  hadoop  6485518 Aug 19 01:16 SecurityAuth.audit
-rw-r--r--.  1 ams  hadoop     2008 Aug 19 01:17 ambari-metrics-collector.out
-rw-r--r--.  1 ams  hadoop 40038004 Aug 19 01:18 ambari-metrics-collector.log
-rw-r--r--.  1 ams  hadoop     6077 Aug 19 01:18 collector-gc.log-201808190114
-rw-r--r--.  1 ams  hadoop    89210 Aug 19 01:18 gc.log-201808190114

.

View solution in original post

24 REPLIES 24

avatar

ams-log4j.txtafter I set the following and copy the files , and restart the service , still logs not rotated and ziped

can you please check if I wrong here ?

log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.file.rollingPolicy.ActiveFileName=${ams.log.dir}/${ams.log.file}.log
log4j.appender.file.rollingPolicy.FileNamePattern=${ams.log.dir}/${ams.log.file}.%i.log.gz
log4j.appender.file.triggeringPolicy.MaxFileSize=1000
log4j.appender.file.rollingPolicy.maxIndex=10

<br>
Michael-Bronson

avatar

@Michael Bronson any WARNINGS in ambari-metrics-collector.out ?

avatar
more ambari-metrics-collector.out
log4j:WARN Failed to set property [triggeringPolicy] to value "org.apache.log4j.rolling.SizeBasedTriggeringPolicy".
log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.FixedWindowRollingPolicy".
log4j:WARN Please set a rolling policy for the RollingFileAppender named 'file'
log4j:ERROR No output stream or file set for the appender named [file].
Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider as a provider class
Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices as a root resource class
Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices as a root resource class
Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
Aug 16, 2018 7:04:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.11 12/09/2011 10:27 AM'
Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices to GuiceManagedComponentProvider with the scope "Singleton"
Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices to GuiceManagedComponentProvider with the scope "Singleton"
Michael-Bronson

avatar

seems like last 3 lines are messed up. can you try like below?

log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.file.rollingPolicy.ActiveFileName=${ams.log.dir}/${ams.log.file}.log
log4j.appender.file.rollingPolicy.FileNamePattern=${ams.log.dir}/${ams.log.file}.%i.log.gz
log4j.appender.file.triggeringPolicy.MaxFileSize=1000
log4j.appender.file.rollingPolicy.maxIndex=10

avatar

yes I already set this lines see the previos attcment file

Michael-Bronson

avatar

what is mean - log4j:ERROR No output stream or file set for the appender named [file]. ?

Michael-Bronson

avatar

also


log4j:WARN Failed to set property [triggeringPolicy] to value "org.apache.log4j.rolling.SizeBasedTriggeringPolicy". log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.FixedWindowRollingPolicy". log4j:WARN Please set a rolling policy for the RollingFileAppender named 'file'
Michael-Bronson

avatar

Can you make sure this file is present?

/usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar

avatar
ls -ltr /usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar
-rw-r--r-- 1 root root 448794 Aug 16 18:51 /usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar
Michael-Bronson

avatar

@Michael Bronson am also seeing same issue in my setup- will try and let you know.