Support Questions

mike_bronson7 · ‎08-16-2018

hi all

we configured the following properties in Advanced ams-hbase-log4j

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}.log
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}.%i.log.gz
log4j.appender.RFA.triggeringPolicy.MaxFileSize=1000000
log4j.appender.RFA.rollingPolicy.maxIndex=10

but after metrics collector restart

we not see that hbase-ams rotated and sipped to .gz

please advice what could be wrong with the log4j properties ?

example of the logs that we get ,

-rw-r--r--  1 ams hadoop     45434111 Aug 16 17:07 hbase-ams-master-master.isys.com.log
-rw-r--r--  1 ams hadoop     2802 Aug 16 17:07 hbase-ams-master-master.isys.com.out

Michael-Bronson

jsensharma · ‎08-19-2018

@Michael Bronson

Please try the below steps and check if it works or not? (I tested this on Ambari 2.6.2.2 and it worked fine without any issue)

Step1). Download and Setup the log4j extras Jar:

# mkdir /tmp/log4j_extras
# curl http://apache.mirrors.tds.net/logging/log4j/extras/1.2.17/apache-log4j-extras-1.2.17-bin.zip -o /tmp/log4j_extras/apache-log4j-extras-1.2.17-bin.zip
# cd /tmp/log4j_extras
# unzip apache-log4j-extras-1.2.17-bin.zip
# cp -f /tmp/log4j_extras/apache-log4j-extras-1.2.17/apache-log4j-extras-1.2.17.jar  /usr/lib/ams-hbase/lib

(OR) in ambari UI you will find the property "hbase_classpath_additional" inside the AMS configs to point to the log4j extras jar.

.

Step2). Edit the "ams-hbase-log4j template"

From ambari side navigate to amb ari UI

Ambari UI --> Ambari Metrics --> Configs --> Advanced --> "Advanced ams-hbase-log4j" --> ams-hbase-log4j template (text area)

And then "Comment out the following part:"

# Rolling File Appender
#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
#log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}
#log4j.appender.RFA.MaxFileSize=${hbase.log.maxfilesize}
#log4j.appender.RFA.MaxBackupIndex=${hbase.log.maxbackupindex}
#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n

.

Now add the following part to define your own RFA appender as following:

log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}-.%d{yyyyMMdd}.log.gz

.

Then restart the AMS service and then wait for some time to see the rolling.

I can see the following kind of Zip "hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz" creation at my end next day:

#  ls -lart /var/log/ambari-metrics-collector/
-rw-r--r--.  1 ams  hadoop  9177961 Aug 18 00:00 hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz
-rw-r--r--.  1 ams  hadoop  7520915 Aug 18 19:43 gc.log-201808170101
-rw-r--r--.  1 ams  hadoop  2506556 Aug 18 19:47 collector-gc.log-201808170220
-rw-r--r--.  1 ams  hadoop  1869725 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.log
drwxr-xr-x.  2 ams  hadoop     8192 Aug 19 01:14 .
-rw-r--r--.  1 ams  hadoop      234 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.out
-rw-r--r--.  1 ams  hadoop      678 Aug 19 01:16 ambari-metrics-collector-startup.out
-rw-r--r--.  1 ams  hadoop  6485518 Aug 19 01:16 SecurityAuth.audit
-rw-r--r--.  1 ams  hadoop     2008 Aug 19 01:17 ambari-metrics-collector.out
-rw-r--r--.  1 ams  hadoop 40038004 Aug 19 01:18 ambari-metrics-collector.log
-rw-r--r--.  1 ams  hadoop     6077 Aug 19 01:18 collector-gc.log-201808190114
-rw-r--r--.  1 ams  hadoop    89210 Aug 19 01:18 gc.log-201808190114

.

View solution in original post

apappu · ‎08-16-2018

@Michael Bronson

Hbase master does not come with apache-log4j-extras-1.2.17.jar jar file - hence it fails with error as "RollingFileAppender classnotfound error"

As a workaround I copied the JAR file hiveserver2 to hbase master and restart the service then it starts working.

cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/hdp/current/hbase-master/lib/

Please note that this is temporary workaround - If you will have to do this again in case of HDP upgrade.

apappu · ‎08-16-2018

@Michael Bronson

I am sorry - that workaround is for HBase-master - let me share the step for AMS Hbase master

apappu · ‎08-16-2018

@Michael Bronson

You can try with below step and restart the service and check

cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ambari-metrics-collector/

mike_bronson7 · ‎08-16-2018

under /usr/hdp/current , we have only the follwing links ( not hbase-master folder )

hadoop-client            hadoop-hdfs-portmap             hadoop-yarn-client           hive-server2        livy-client           spark2-thriftserver  zookeeper-server
hadoop-hdfs-client       hadoop-hdfs-secondarynamenode   hadoop-yarn-nodemanager      hive-server2-hive2  pig-client            spark-client
hadoop-hdfs-datanode     hadoop-hdfs-zkfc                hadoop-yarn-resourcemanager  hive-webhcat        shc                   spark_llap
hadoop-hdfs-journalnode  hadoop-httpfs                   hadoop-yarn-timelineserver   kafka-broker        slider-client         storm-slider-client
hadoop-hdfs-namenode     hadoop-mapreduce-client         hive-client                  livy2-client        spark2-client         tez-client
hadoop-hdfs-nfs3         hadoop-mapreduce-historyserver  hive-metastore               livy2-server        spark2-historyserver  zookeeper-client

Michael-Bronson

apappu · ‎08-16-2018

@Michael Bronson yes, you have hive-server2 - can copy from that to ams folder

cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ams-hbase/lib/

mike_bronson7 · ‎08-16-2018

I copy the file and restart the service but without good results

Michael-Bronson

mike_bronson7 · ‎08-16-2018

its works -:)

Michael-Bronson

mike_bronson7 · ‎08-16-2018

another one question please , I see that ambari-metrics-collector.log created with huge size , can we rote it also and zipped it ?

-rw-r--r--  1 ams hadoop     2008 Aug 16 18:32 ambari-metrics-collector.out
-rw-r--r--. 1 ams hadoop 40776116 Aug 16 18:33 ambari-metrics-collector.log

Michael-Bronson

apappu · ‎08-16-2018

@Michael Bronson

Thanks for confirming that it worked - you can update the log4j configurations and copy the jar file and then check it?

cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ambari-metrics-collector/

Cloudera Community

Support Questions

log4g + logs not rotated and zipped