Created 08-16-2018 05:35 PM
hi all
we configured the following properties in Advanced ams-hbase-log4j
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}.log log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}.%i.log.gz log4j.appender.RFA.triggeringPolicy.MaxFileSize=1000000 log4j.appender.RFA.rollingPolicy.maxIndex=10
but after metrics collector restart
we not see that hbase-ams rotated and sipped to .gz
please advice what could be wrong with the log4j properties ?
example of the logs that we get ,
-rw-r--r-- 1 ams hadoop 45434111 Aug 16 17:07 hbase-ams-master-master.isys.com.log -rw-r--r-- 1 ams hadoop 2802 Aug 16 17:07 hbase-ams-master-master.isys.com.out
Created 08-19-2018 01:20 AM
Please try the below steps and check if it works or not? (I tested this on Ambari 2.6.2.2 and it worked fine without any issue)
Step1). Download and Setup the log4j extras Jar:
# mkdir /tmp/log4j_extras # curl http://apache.mirrors.tds.net/logging/log4j/extras/1.2.17/apache-log4j-extras-1.2.17-bin.zip -o /tmp/log4j_extras/apache-log4j-extras-1.2.17-bin.zip # cd /tmp/log4j_extras # unzip apache-log4j-extras-1.2.17-bin.zip # cp -f /tmp/log4j_extras/apache-log4j-extras-1.2.17/apache-log4j-extras-1.2.17.jar /usr/lib/ams-hbase/lib
(OR) in ambari UI you will find the property "hbase_classpath_additional" inside the AMS configs to point to the log4j extras jar.
.
Step2). Edit the "ams-hbase-log4j template"
From ambari side navigate to amb ari UI
Ambari UI --> Ambari Metrics --> Configs --> Advanced --> "Advanced ams-hbase-log4j" --> ams-hbase-log4j template (text area)
And then "Comment out the following part:"
# Rolling File Appender #log4j.appender.RFA=org.apache.log4j.RollingFileAppender #log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file} #log4j.appender.RFA.MaxFileSize=${hbase.log.maxfilesize} #log4j.appender.RFA.MaxBackupIndex=${hbase.log.maxbackupindex} #log4j.appender.RFA.layout=org.apache.log4j.PatternLayout #log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n
.
Now add the following part to define your own RFA appender as following:
log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file} log4j.appender.RFA.layout=org.apache.log4j.PatternLayout log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file} log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}-.%d{yyyyMMdd}.log.gz
.
Then restart the AMS service and then wait for some time to see the rolling.
I can see the following kind of Zip "hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz" creation at my end next day:
# ls -lart /var/log/ambari-metrics-collector/ -rw-r--r--. 1 ams hadoop 9177961 Aug 18 00:00 hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz -rw-r--r--. 1 ams hadoop 7520915 Aug 18 19:43 gc.log-201808170101 -rw-r--r--. 1 ams hadoop 2506556 Aug 18 19:47 collector-gc.log-201808170220 -rw-r--r--. 1 ams hadoop 1869725 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.log drwxr-xr-x. 2 ams hadoop 8192 Aug 19 01:14 . -rw-r--r--. 1 ams hadoop 234 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.out -rw-r--r--. 1 ams hadoop 678 Aug 19 01:16 ambari-metrics-collector-startup.out -rw-r--r--. 1 ams hadoop 6485518 Aug 19 01:16 SecurityAuth.audit -rw-r--r--. 1 ams hadoop 2008 Aug 19 01:17 ambari-metrics-collector.out -rw-r--r--. 1 ams hadoop 40038004 Aug 19 01:18 ambari-metrics-collector.log -rw-r--r--. 1 ams hadoop 6077 Aug 19 01:18 collector-gc.log-201808190114 -rw-r--r--. 1 ams hadoop 89210 Aug 19 01:18 gc.log-201808190114
.
Created 08-16-2018 06:52 PM
ams-log4j.txtafter I set the following and copy the files , and restart the service , still logs not rotated and ziped
can you please check if I wrong here ?
log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.file.rollingPolicy.ActiveFileName=${ams.log.dir}/${ams.log.file}.log log4j.appender.file.rollingPolicy.FileNamePattern=${ams.log.dir}/${ams.log.file}.%i.log.gz log4j.appender.file.triggeringPolicy.MaxFileSize=1000 log4j.appender.file.rollingPolicy.maxIndex=10 <br>
Created 08-16-2018 07:06 PM
@Michael Bronson any WARNINGS in ambari-metrics-collector.out ?
Created 08-16-2018 07:11 PM
more ambari-metrics-collector.out log4j:WARN Failed to set property [triggeringPolicy] to value "org.apache.log4j.rolling.SizeBasedTriggeringPolicy". log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.FixedWindowRollingPolicy". log4j:WARN Please set a rolling policy for the RollingFileAppender named 'file' log4j:ERROR No output stream or file set for the appender named [file]. Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider as a provider class Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices as a root resource class Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices as a root resource class Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Aug 16, 2018 7:04:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.11 12/09/2011 10:27 AM' Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider to GuiceManagedComponentProvider with the scope "Singleton" Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices to GuiceManagedComponentProvider with the scope "Singleton" Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices to GuiceManagedComponentProvider with the scope "Singleton"
Created 08-16-2018 07:56 PM
seems like last 3 lines are messed up. can you try like below?
log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.file.rollingPolicy.ActiveFileName=${ams.log.dir}/${ams.log.file}.log log4j.appender.file.rollingPolicy.FileNamePattern=${ams.log.dir}/${ams.log.file}.%i.log.gz log4j.appender.file.triggeringPolicy.MaxFileSize=1000 log4j.appender.file.rollingPolicy.maxIndex=10
Created 08-16-2018 08:09 PM
yes I already set this lines see the previos attcment file
Created 08-16-2018 08:21 PM
what is mean - log4j:ERROR No output stream or file set for the appender named [file]. ?
Created 08-16-2018 08:25 PM
also
log4j:WARN Failed to set property [triggeringPolicy] to value "org.apache.log4j.rolling.SizeBasedTriggeringPolicy". log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.FixedWindowRollingPolicy". log4j:WARN Please set a rolling policy for the RollingFileAppender named 'file'
Created 08-16-2018 08:51 PM
Can you make sure this file is present?
/usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar
Created 08-16-2018 09:05 PM
ls -ltr /usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar -rw-r--r-- 1 root root 448794 Aug 16 18:51 /usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar
Created 08-17-2018 12:31 AM
@Michael Bronson am also seeing same issue in my setup- will try and let you know.