https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221301#M183175 <P>@Jay can yoiu help me regarding my last notes ?</P> Thu, 16 Aug 2018 01:34:57 GMT mike_bronson7 2018-08-16T01:34:57Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221297#M183171 <P>hi all</P><P>we configure the HIVE , and log4j with RollingFileAppender </P><PRE>log4j.appender.DRFA=org.apache.log4j.RollingFileAppender log4j.appender.DRFA.MaxBackupIndex=10 log4j.appender.DRFA.MaxFileSize=1KB</PRE><P>full details: </P><PRE># Define some default values that can be overridden by system properties hive.log.threshold=ALL hive.root.logger=INFO,DRFA hive.log.dir=${java.io.tmpdir}/${user.name} hive.log.file=hive.log # Define the root logger to the system property "hadoop.root.logger". log4j.rootLogger=${hive.root.logger}, EventCounter # Logging Threshold log4j.threshold=${hive.log.threshold} # # Daily Rolling File Appender # # Use the PidDailyerRollingFileAppend class instead if you want to use separate log files # for different CLI session. # # log4j.appender.DRFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender #log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA.File=${hive.log.dir}/${hive.log.file} # Rollver at midnight #log4j.appender.DRFA.DatePattern=.yyyy-MM-dd log4j.appender.DRFA=org.apache.log4j.RollingFileAppender log4j.appender.DRFA.MaxBackupIndex=10 log4j.appender.DRFA.MaxFileSize=1KB # 30-day backup log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout # Pattern format: Date LogLevel LoggerName LogMessage #log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n # Debugging Pattern format log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n</PRE><P>the logs from the machine are:</P><PRE>-rw-r--r-- 1 hive hadoop 1113 Aug 15 13:10 hivemetastore.log.10 -rw-r--r-- 1 hive hadoop 1028 Aug 15 13:10 hivemetastore.log.9 -rw-r--r-- 1 hive hadoop 1070 Aug 15 13:11 hivemetastore.log.8 -rw-r--r-- 1 hive hadoop 1239 Aug 15 13:12 hiveserver2.log.10 -rw-r--r-- 1 hive hadoop 1154 Aug 15 13:13 hivemetastore.log.7 -rw-r--r-- 1 hive hadoop 1133 Aug 15 13:13 hivemetastore.log.6 -rw-r--r-- 1 hive hadoop 1055 Aug 15 13:15 hiveserver2.log.9 -rw-r--r-- 1 hive hadoop 1203 Aug 15 13:15 hiveserver2.log.8 -rw-r--r-- 1 hive hadoop 1098 Aug 15 13:15 hiveserver2.log.7 -rw-r--r-- 1 hive hadoop 1028 Aug 15 13:15 hiveserver2.log.6 -rw-r--r-- 1 hive hadoop 1239 Aug 15 13:15 hiveserver2.log.5 -rw-r--r-- 1 hive hadoop 1113 Aug 15 13:16 hivemetastore.log.5 -rw-r--r-- 1 hive hadoop 1028 Aug 15 13:16 hivemetastore.log.4 -rw-r--r-- 1 hive hadoop 1070 Aug 15 13:16 hivemetastore.log.3 -rw-r--r-- 1 hive hadoop 1048 Aug 15 13:18 hiveserver2.log.4 -rw-r--r-- 1 hive hadoop 1173 Aug 15 13:18 hiveserver2.log.3 -rw-r--r-- 1 hive hadoop 1157 Aug 15 13:18 hiveserver2.log.2 -rw-r--r-- 1 hive hadoop 1239 Aug 15 13:18 hiveserver2.log.1 -rw-r--r-- 1 hive hadoop 503 Aug 15 13:18 hiveserver2.log -rw-r--r-- 1 hive hadoop 1154 Aug 15 13:19 hivemetastore.log.2 -rw-r--r-- 1 hive hadoop 1133 Aug 15 13:19 hivemetastore.log.1 -rw-r--r-- 1 hive hadoop 292 Aug 15 13:19 hivemetastore.log -rw-r--r-- 1 hive hadoop 4904 Aug 15 13:20 hivemetastore-report.json.tmp -rw-r--r-- 1 hive hadoop 4273 Aug 15 13:20 hiveserver2-report.json.tmp</PRE><P>my question is - all roteated logs as hiveserver2.log.1 , hiveserver2.log.2 , etc</P><P>are not zipped log</P><P>what is the change in log4j that I need to do in order to ziped the roteated fuiles?</P> Wed, 15 Aug 2018 20:22:43 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221297#M183171 mike_bronson7 2018-08-15T20:22:43Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221298#M183172 <P><A rel="user" href="https://community.cloudera.com/users/26229/uribarih.html" nodeid="26229">@Michael Bronson</A></P><P>Please define a "rollingPolicy" inside the RollingFileAppender something as described in the following article:</P><P><A href="https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th.html" target="_blank">https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th.html</A></P><P>Create a new appender like ZIPRFA like the mentioned article</P><PRE>log4j.appender.XXXXXX.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}-.%d{yyyyMMdd}.log.gz</PRE> <P>.</P> Wed, 15 Aug 2018 20:31:11 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221298#M183172 jsensharma 2018-08-15T20:31:11Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221299#M183173 <P>hi Jay</P><P>we configured the follwing , but still rotated files are not ziped </P><P>whart is wrong in my log4j? </P><PRE>hive.log.threshold=ALL hive.root.logger=INFO,DRFA hive.log.dir=${java.io.tmpdir}/${user.name} hive.log.file=hive.log # Define the root logger to the system property "hadoop.root.logger". log4j.rootLogger=${hive.root.logger}, EventCounter # Logging Threshold log4j.threshold=${hive.log.threshold} # # Daily Rolling File Appender # # Use the PidDailyerRollingFileAppend class instead if you want to use separate log files # for different CLI session. # # log4j.appender.DRFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender #log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA.File=${hive.log.dir}/${hive.log.file} # Rollver at midnight #log4j.appender.DRFA.DatePattern=.yyyy-MM-dd log4j.appender.DRFA=org.apache.log4j.RollingFileAppender log4j.appender.DRFA.MaxBackupIndex=10 log4j.appender.DRFA.MaxFileSize=1KB log4j.appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}-.%d{yyyyMMdd}.log.gz</PRE> Wed, 15 Aug 2018 21:08:03 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221299#M183173 mike_bronson7 2018-08-15T21:08:03Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221300#M183174 <P>I also configured the follwing ( without date ) but without zip alos</P><PRE>log4j.appender.DRFA=org.apache.log4j.RollingFileAppender log4j.appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}-.%i.log.zip log4j.appender.DRFA.MaxBackupIndex=10 log4j.appender.DRFA.MaxFileSize=1KB</PRE> Wed, 15 Aug 2018 21:31:13 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221300#M183174 mike_bronson7 2018-08-15T21:31:13Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221301#M183175 <P>@Jay can yoiu help me regarding my last notes ?</P> Thu, 16 Aug 2018 01:34:57 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221301#M183175 mike_bronson7 2018-08-16T01:34:57Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221302#M183176 <P>Please use below config for size based rotation.</P><PRE>log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout log4j.appender.DRFA=org.apache.log4j.rolling.RollingFileAppender log4j.appender.DRFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.DRFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.DRFA.rollingPolicy.ActiveFileName =${hive.log.dir}/${hive.log.file}.log log4j.appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}-.%i.log.gz log4j.appender.DRFA.triggeringPolicy.MaxFileSize=10000 log4j.appender.DRFA.rollingPolicy.maxIndex=10 </PRE><P>I have tried this and it works perfectly fine.</P><P>For some reason maxfilesize is not working if used like 1MB like that - so use live above.</P><P>Note: please check hive-server2.err for any WARNINGS if does not work</P> Thu, 16 Aug 2018 01:58:27 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221302#M183176 apappu 2018-08-16T01:58:27Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221303#M183177 <P>do you mean that I need to replace my lines as</P><PRE>log4j.appender.DRFA=org.apache.log4j.RollingFileAppender log4j.appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}-.%i.log.zip log4j.appender.DRFA.MaxBackupIndex=10 log4j.appender.DRFA.MaxFileSize=1KB</PRE><P>to</P><OL> <LI>log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout</LI><LI>log4j.appender.DRFA=org.apache.log4j.rolling.RollingFileAppender</LI><LI>log4j.appender.DRFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy</LI><LI>log4j.appender.DRFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy</LI><LI>log4j.appender.DRFA.rollingPolicy.ActiveFileName=${hive.log.dir}/${hive.log.file}.log</LI><LI>log4j.appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}-.%i.log.gz</LI><LI>log4j.appender.DRFA.triggeringPolicy.MaxFileSize=10000</LI><LI>log4j.appender.DRFA.rollingPolicy.maxIndex=10</LI></OL><P>?</P><PRE></PRE><OL></OL> Thu, 16 Aug 2018 02:05:28 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221303#M183177 mike_bronson7 2018-08-16T02:05:28Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221304#M183178 <P>second</P><P>my ambari is based on linux machines cluster </P><P>so</P><P>what is the line:</P><OL> <LI>log4j.appender.DRFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy</LI></OL> Thu, 16 Aug 2018 02:08:01 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221304#M183178 mike_bronson7 2018-08-16T02:08:01Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221305#M183179 <P><A rel="user" href="https://community.cloudera.com/users/26229/uribarih.html" nodeid="26229">@Michael Bronson</A> yes that is correct - please see attached my log4j configuration in ambari.<A href="https://community.cloudera.com/legacyfs/online/attachments/85648-log4j.txt">log4j.txt</A></P> Thu, 16 Aug 2018 02:11:15 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221305#M183179 apappu 2018-08-16T02:11:15Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221306#M183180 <P><A rel="user" href="https://community.cloudera.com/users/26229/uribarih.html" nodeid="26229">@Michael Bronson</A> see my above comment - "FixedWindowRollingPolicy" (its not windows its window)is policy name - its nothing to do with OS</P> Thu, 16 Aug 2018 02:12:38 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221306#M183180 apappu 2018-08-16T02:12:38Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221307#M183181 <P>ok , let me try to set it on my ambari , I will update ....</P> Thu, 16 Aug 2018 02:15:43 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221307#M183181 mike_bronson7 2018-08-16T02:15:43Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221308#M183182 <P>this are the new logs that I get now</P><PRE><BR />-rw-r--r-- 1 hive hadoop 389 Aug 15 19:28 hivemetastore.log-.10.log.gz -rw-r--r--. 1 hive hadoop 0 Aug 15 19:29 hive.err -rw-r--r--. 1 hive hadoop 31 Aug 15 19:29 hive.out -rw-r--r--. 1 hive hadoop 0 Aug 15 19:29 hive-server2.out -rw-r--r--. 1 hive hadoop 0 Aug 15 19:29 hive-server2.err -rw-r--r-- 1 hive hadoop 1719 Aug 15 19:34 hivemetastore.log-.9.log.gz -rw-r--r-- 1 hive hadoop 4357 Aug 15 19:34 hiveserver2-report.json -rw-r--r-- 1 hive hadoop 4315 Aug 15 19:34 hivemetastore-report.json -rw-r--r-- 1 hive hadoop 1240 Aug 15 19:42 hiveserver2.log-.10.log.gz -rw-r--r-- 1 hive hadoop 1038 Aug 15 19:43 hivemetastore.log-.8.log.gz -rw-r--r-- 1 hive hadoop 1159 Aug 15 19:48 hiveserver2.log-.9.log.gz -rw-r--r-- 1 hive hadoop 1038 Aug 15 19:49 hivemetastore.log-.7.log.gz -rw-r--r-- 1 hive hadoop 1150 Aug 15 19:54 hiveserver2.log-.8.log.gz -rw-r--r-- 1 hive hadoop 1019 Aug 15 19:58 hivemetastore.log-.6.log.gz -rw-r--r-- 1 hive hadoop 1168 Aug 15 20:00 hiveserver2.log-.7.log.gz -rw-r--r-- 1 hive hadoop 1174 Aug 15 20:06 hiveserver2.log-.6.log.gz -rw-r--r-- 1 hive hadoop 1016 Aug 15 20:07 hivemetastore.log-.5.log.gz -rw-r--r-- 1 hive hadoop 1156 Aug 15 20:12 hiveserver2.log-.5.log.gz -rw-r--r-- 1 hive hadoop 1019 Aug 15 20:13 hivemetastore.log-.4.log.gz -rw-r--r-- 1 hive hadoop 1159 Aug 15 20:18 hiveserver2.log-.4.log.gz -rw-r--r-- 1 hive hadoop 1028 Aug 15 20:22 hivemetastore.log-.3.log.gz -rw-r--r-- 1 hive hadoop 1171 Aug 15 20:24 hiveserver2.log-.3.log.gz -rw-r--r-- 1 hive hadoop 1167 Aug 15 20:30 hiveserver2.log-.2.log.gz -rw-r--r-- 1 hive hadoop 1017 Aug 15 20:31 hivemetastore.log-.2.log.gz -rw-r--r-- 1 hive hadoop 233 Aug 15 20:36 hiveserver2.log -rw-r--r-- 1 hive hadoop 1166 Aug 15 20:36 hiveserver2.log-.1.log.gz -rw-r--r-- 1 hive hadoop 1003 Aug 15 20:37 hivemetastore.log-.1.log.gz -rw-r--r-- 1 hive hadoop 1218 Aug 15 20:37 hivemetastore.log -rw-r--r-- 1 hive hadoop 4248 Aug 15 20:39 hivemetastore-report.json.tmp -rw-r--r-- 1 hive hadoop 4363 Aug 15 20:39 hiveserver2-report.json.tmp [root@master01 hive]# [root@master01 hive]# [root@master01 hive]# [root@master01 hive]# du -sh * 0 hive.err 4.0K hivemetastore.log 4.0K hivemetastore.log-.10.log.gz 4.0K hivemetastore.log-.1.log.gz 4.0K hivemetastore.log-.2.log.gz 4.0K hivemetastore.log-.3.log.gz 4.0K hivemetastore.log-.4.log.gz 4.0K hivemetastore.log-.5.log.gz 4.0K hivemetastore.log-.6.log.gz 4.0K hivemetastore.log-.7.log.gz 4.0K hivemetastore.log-.8.log.gz 4.0K hivemetastore.log-.9.log.gz 8.0K hivemetastore-report.json 8.0K hivemetastore-report.json.tmp 4.0K hive.out 0 hive-server2.err 4.0K hiveserver2.log 4.0K hiveserver2.log-.10.log.gz 4.0K hiveserver2.log-.1.log.gz 4.0K hiveserver2.log-.2.log.gz 4.0K hiveserver2.log-.3.log.gz 4.0K hiveserver2.log-.4.log.gz 4.0K hiveserver2.log-.5.log.gz 4.0K hiveserver2.log-.6.log.gz 4.0K hiveserver2.log-.7.log.gz 4.0K hiveserver2.log-.8.log.gz 4.0K hiveserver2.log-.9.log.gz 0 hive-server2.out 8.0K hiveserver2-report.json 8.0K hiveserver2-report.json.tmp</PRE> Thu, 16 Aug 2018 03:40:40 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221308#M183182 mike_bronson7 2018-08-16T03:40:40Z https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221309#M183183 <A rel="user" href="https://community.cloudera.com/users/26229/uribarih.html" nodeid="26229">@Michael Bronson</A><P>thats good - please mark the correct answer.</P> Thu, 16 Aug 2018 03:53:46 GMT https://community.cloudera.com/t5/Support-Questions/How-to-make-log-files-rotate-based-on-size-using-log4j/m-p/221309#M183183 apappu 2018-08-16T03:53:46Z