- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
log4g + logs not rotated and zipped
Created 08-16-2018 05:35 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
hi all
we configured the following properties in Advanced ams-hbase-log4j
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file}.log log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}.%i.log.gz log4j.appender.RFA.triggeringPolicy.MaxFileSize=1000000 log4j.appender.RFA.rollingPolicy.maxIndex=10
but after metrics collector restart
we not see that hbase-ams rotated and sipped to .gz
please advice what could be wrong with the log4j properties ?
example of the logs that we get ,
-rw-r--r-- 1 ams hadoop 45434111 Aug 16 17:07 hbase-ams-master-master.isys.com.log -rw-r--r-- 1 ams hadoop 2802 Aug 16 17:07 hbase-ams-master-master.isys.com.out
Created 08-19-2018 01:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please try the below steps and check if it works or not? (I tested this on Ambari 2.6.2.2 and it worked fine without any issue)
Step1). Download and Setup the log4j extras Jar:
# mkdir /tmp/log4j_extras # curl http://apache.mirrors.tds.net/logging/log4j/extras/1.2.17/apache-log4j-extras-1.2.17-bin.zip -o /tmp/log4j_extras/apache-log4j-extras-1.2.17-bin.zip # cd /tmp/log4j_extras # unzip apache-log4j-extras-1.2.17-bin.zip # cp -f /tmp/log4j_extras/apache-log4j-extras-1.2.17/apache-log4j-extras-1.2.17.jar /usr/lib/ams-hbase/lib
(OR) in ambari UI you will find the property "hbase_classpath_additional" inside the AMS configs to point to the log4j extras jar.
.
Step2). Edit the "ams-hbase-log4j template"
From ambari side navigate to amb ari UI
Ambari UI --> Ambari Metrics --> Configs --> Advanced --> "Advanced ams-hbase-log4j" --> ams-hbase-log4j template (text area)
And then "Comment out the following part:"
# Rolling File Appender #log4j.appender.RFA=org.apache.log4j.RollingFileAppender #log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file} #log4j.appender.RFA.MaxFileSize=${hbase.log.maxfilesize} #log4j.appender.RFA.MaxBackupIndex=${hbase.log.maxbackupindex} #log4j.appender.RFA.layout=org.apache.log4j.PatternLayout #log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n
.
Now add the following part to define your own RFA appender as following:
log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender log4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file} log4j.appender.RFA.layout=org.apache.log4j.PatternLayout log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy log4j.appender.RFA.rollingPolicy.ActiveFileName=${hbase.log.dir}/${hbase.log.file} log4j.appender.RFA.rollingPolicy.FileNamePattern=${hbase.log.dir}/${hbase.log.file}-.%d{yyyyMMdd}.log.gz
.
Then restart the AMS service and then wait for some time to see the rolling.
I can see the following kind of Zip "hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz" creation at my end next day:
# ls -lart /var/log/ambari-metrics-collector/ -rw-r--r--. 1 ams hadoop 9177961 Aug 18 00:00 hbase-ams-master-newhwx2.example.com.log-.20180817.log.gz -rw-r--r--. 1 ams hadoop 7520915 Aug 18 19:43 gc.log-201808170101 -rw-r--r--. 1 ams hadoop 2506556 Aug 18 19:47 collector-gc.log-201808170220 -rw-r--r--. 1 ams hadoop 1869725 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.log drwxr-xr-x. 2 ams hadoop 8192 Aug 19 01:14 . -rw-r--r--. 1 ams hadoop 234 Aug 19 01:14 hbase-ams-master-newhwx2.example.com.out -rw-r--r--. 1 ams hadoop 678 Aug 19 01:16 ambari-metrics-collector-startup.out -rw-r--r--. 1 ams hadoop 6485518 Aug 19 01:16 SecurityAuth.audit -rw-r--r--. 1 ams hadoop 2008 Aug 19 01:17 ambari-metrics-collector.out -rw-r--r--. 1 ams hadoop 40038004 Aug 19 01:18 ambari-metrics-collector.log -rw-r--r--. 1 ams hadoop 6077 Aug 19 01:18 collector-gc.log-201808190114 -rw-r--r--. 1 ams hadoop 89210 Aug 19 01:18 gc.log-201808190114
.
Created 08-16-2018 05:56 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hbase master does not come with apache-log4j-extras-1.2.17.jar jar file - hence it fails with error as "RollingFileAppender classnotfound error"
As a workaround I copied the JAR file hiveserver2 to hbase master and restart the service then it starts working.
cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/hdp/current/hbase-master/lib/
Please note that this is temporary workaround - If you will have to do this again in case of HDP upgrade.
Created 08-16-2018 06:02 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am sorry - that workaround is for HBase-master - let me share the step for AMS Hbase master
Created 08-16-2018 06:05 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can try with below step and restart the service and check
cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ambari-metrics-collector/
Created 08-16-2018 06:07 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
under /usr/hdp/current , we have only the follwing links ( not hbase-master folder )
hadoop-client hadoop-hdfs-portmap hadoop-yarn-client hive-server2 livy-client spark2-thriftserver zookeeper-server hadoop-hdfs-client hadoop-hdfs-secondarynamenode hadoop-yarn-nodemanager hive-server2-hive2 pig-client spark-client hadoop-hdfs-datanode hadoop-hdfs-zkfc hadoop-yarn-resourcemanager hive-webhcat shc spark_llap hadoop-hdfs-journalnode hadoop-httpfs hadoop-yarn-timelineserver kafka-broker slider-client storm-slider-client hadoop-hdfs-namenode hadoop-mapreduce-client hive-client livy2-client spark2-client tez-client hadoop-hdfs-nfs3 hadoop-mapreduce-historyserver hive-metastore livy2-server spark2-historyserver zookeeper-client
Created 08-16-2018 06:12 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Michael Bronson yes, you have hive-server2 - can copy from that to ams folder
cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ams-hbase/lib/
Created 08-16-2018 06:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I copy the file and restart the service but without good results
Created 08-16-2018 06:32 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
its works -:)
Created 08-16-2018 06:33 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
another one question please , I see that ambari-metrics-collector.log created with huge size , can we rote it also and zipped it ?
-rw-r--r-- 1 ams hadoop 2008 Aug 16 18:32 ambari-metrics-collector.out -rw-r--r--. 1 ams hadoop 40776116 Aug 16 18:33 ambari-metrics-collector.log
Created 08-16-2018 06:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for confirming that it worked - you can update the log4j configurations and copy the jar file and then check it?
cp /usr/hdp/current/hive-server2/lib/apache-log4j-extras-1.2.17.jar /usr/lib/ambari-metrics-collector/