Support Questions

Find answers, ask questions, and share your expertise

log4g + logs not rotated and zipped

hi all

we configured the following properties in Advanced ams-hbase-log4j

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}.log
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}.%i.log.gz
log4j.appender.RFA.triggeringPolicy.MaxFileSize=1000000
log4j.appender.RFA.rollingPolicy.maxIndex=10

but after metrics collector restart

we not see that hbase-ams rotated and sipped to .gz

please advice what could be wrong with the log4j properties ?

example of the logs that we get ,

-rw-r--r--  1 ams hadoop     45434111 Aug 16 17:07 hbase-ams-master-master.isys.com.log
-rw-r--r--  1 ams hadoop     2802 Aug 16 17:07 hbase-ams-master-master.isys.com.out
Michael-Bronson
1 ACCEPTED SOLUTION

Super Mentor

@Michael Bronson

Please try the below steps and check if it works or not? (I tested this on Ambari 2.6.2.2 and it worked fine without any issue)

Step1). Download and Setup the log4j extras Jar:

# mkdir /tmp/log4j_extras
# curl http://apache.mirrors.tds.net/logging/log4j/extras/1.2.17/apache-log4j-extras-1.2.17-bin.zip -o /tmp/log4j_extras/apache-log4j-extras-1.2.17-bin.zip
# cd /tmp/log4j_extras
# unzip apache-log4j-extras-1.2.17-bin.zip
# cp -f /tmp/log4j_extras/apache-log4j-extras-1.2.17/apache-log4j-extras-1.2.17.jar  /usr/lib/ams-hbase/lib

(OR) in ambari UI you will find the property "hbase_classpath_additional" inside the AMS configs to point to the log4j extras jar.

.

Step2). Edit the "ams-hbase-log4j template"

From ambari side navigate to amb ari UI

Ambari UI --> Ambari Metrics --> Configs --> Advanced --> "Advanced ams-hbase-log4j" --> ams-hbase-log4j template (text area)

And then "Comment out the following part:"

# Rolling File Appender
#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
#log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}
#log4j.appender.RFA.MaxFileSize=${hbase.log.maxfilesize}
#log4j.appender.RFA.MaxBackupIndex=${hbase.log.maxbackupindex}
#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n

.

Now add the following part to define your own RFA appender as following:

log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}-.%d{yyyyMMdd}.log.gz

.

Then restart the AMS service and then wait for some time to see the rolling.

I can see the following kind of Zip "hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz" creation at my end next day:

#  ls -lart /var/log/ambari-metrics-collector/
-rw-r--r--.  1 ams  hadoop  9177961 Aug 18 00:00 hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz
-rw-r--r--.  1 ams  hadoop  7520915 Aug 18 19:43 gc.log-201808170101
-rw-r--r--.  1 ams  hadoop  2506556 Aug 18 19:47 collector-gc.log-201808170220
-rw-r--r--.  1 ams  hadoop  1869725 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.log
drwxr-xr-x.  2 ams  hadoop     8192 Aug 19 01:14 .
-rw-r--r--.  1 ams  hadoop      234 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.out
-rw-r--r--.  1 ams  hadoop      678 Aug 19 01:16 ambari-metrics-collector-startup.out
-rw-r--r--.  1 ams  hadoop  6485518 Aug 19 01:16 SecurityAuth.audit
-rw-r--r--.  1 ams  hadoop     2008 Aug 19 01:17 ambari-metrics-collector.out
-rw-r--r--.  1 ams  hadoop 40038004 Aug 19 01:18 ambari-metrics-collector.log
-rw-r--r--.  1 ams  hadoop     6077 Aug 19 01:18 collector-gc.log-201808190114
-rw-r--r--.  1 ams  hadoop    89210 Aug 19 01:18 gc.log-201808190114

.

View solution in original post

24 REPLIES 24

@Michael Bronson

Hbase master does not come with apache-log4j-extras-1.2.17.jar jar file - hence it fails with error as "RollingFileAppender classnotfound error"

As a workaround I copied the JAR file hiveserver2 to hbase master and restart the service then it starts working.

cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/hdp/current/hbase-master/lib/ 

Please note that this is temporary workaround - If you will have to do this again in case of HDP upgrade.

@Michael Bronson

I am sorry - that workaround is for HBase-master - let me share the step for AMS Hbase master

@Michael Bronson

You can try with below step and restart the service and check

cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ambari-metrics-collector/ 

under /usr/hdp/current , we have only the follwing links ( not hbase-master folder )

hadoop-client            hadoop-hdfs-portmap             hadoop-yarn-client           hive-server2        livy-client           spark2-thriftserver  zookeeper-server
hadoop-hdfs-client       hadoop-hdfs-secondarynamenode   hadoop-yarn-nodemanager      hive-server2-hive2  pig-client            spark-client
hadoop-hdfs-datanode     hadoop-hdfs-zkfc                hadoop-yarn-resourcemanager  hive-webhcat        shc                   spark_llap
hadoop-hdfs-journalnode  hadoop-httpfs                   hadoop-yarn-timelineserver   kafka-broker        slider-client         storm-slider-client
hadoop-hdfs-namenode     hadoop-mapreduce-client         hive-client                  livy2-client        spark2-client         tez-client
hadoop-hdfs-nfs3         hadoop-mapreduce-historyserver  hive-metastore               livy2-server        spark2-historyserver  zookeeper-client



Michael-Bronson

@Michael Bronson yes, you have hive-server2 - can copy from that to ams folder

cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ams-hbase/lib/

I copy the file and restart the service but without good results

Michael-Bronson

its works -:)

Michael-Bronson

another one question please , I see that ambari-metrics-collector.log created with huge size , can we rote it also and zipped it ?

-rw-r--r--  1 ams hadoop     2008 Aug 16 18:32 ambari-metrics-collector.out
-rw-r--r--. 1 ams hadoop 40776116 Aug 16 18:33 ambari-metrics-collector.log
Michael-Bronson

@Michael Bronson

Thanks for confirming that it worked - you can update the log4j configurations and copy the jar file and then check it?

cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ambari-metrics-collector/

ams-log4j.txtafter I set the following and copy the files , and restart the service , still logs not rotated and ziped

can you please check if I wrong here ?

log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.file.rollingPolicy.ActiveFileName=${ams.log.dir}/${ams.log.file}.log
log4j.appender.file.rollingPolicy.FileNamePattern=${ams.log.dir}/${ams.log.file}.%i.log.gz
log4j.appender.file.triggeringPolicy.MaxFileSize=1000
log4j.appender.file.rollingPolicy.maxIndex=10

<br>
Michael-Bronson

@Michael Bronson any WARNINGS in ambari-metrics-collector.out ?

more ambari-metrics-collector.out
log4j:WARN Failed to set property [triggeringPolicy] to value "org.apache.log4j.rolling.SizeBasedTriggeringPolicy".
log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.FixedWindowRollingPolicy".
log4j:WARN Please set a rolling policy for the RollingFileAppender named 'file'
log4j:ERROR No output stream or file set for the appender named [file].
Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider as a provider class
Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices as a root resource class
Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices as a root resource class
Aug 16, 2018 7:04:28 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
Aug 16, 2018 7:04:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.11 12/09/2011 10:27 AM'
Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices to GuiceManagedComponentProvider with the scope "Singleton"
Aug 16, 2018 7:04:29 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices to GuiceManagedComponentProvider with the scope "Singleton"
Michael-Bronson

seems like last 3 lines are messed up. can you try like below?

log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.file.rollingPolicy.ActiveFileName=${ams.log.dir}/${ams.log.file}.log
log4j.appender.file.rollingPolicy.FileNamePattern=${ams.log.dir}/${ams.log.file}.%i.log.gz
log4j.appender.file.triggeringPolicy.MaxFileSize=1000
log4j.appender.file.rollingPolicy.maxIndex=10

yes I already set this lines see the previos attcment file

Michael-Bronson

what is mean - log4j:ERROR No output stream or file set for the appender named [file]. ?

Michael-Bronson

also


log4j:WARN Failed to set property [triggeringPolicy] to value "org.apache.log4j.rolling.SizeBasedTriggeringPolicy". log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.FixedWindowRollingPolicy". log4j:WARN Please set a rolling policy for the RollingFileAppender named 'file'
Michael-Bronson

Can you make sure this file is present?

/usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar

ls -ltr /usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar
-rw-r--r-- 1 root root 448794 Aug 16 18:51 /usr/lib/ambari-metrics-collector/apache-log4j-extras-1.2.17.jar
Michael-Bronson

@Michael Bronson am also seeing same issue in my setup- will try and let you know.

OK no problem , we wait for you

Michael-Bronson
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.