Created 08-10-2017 01:27 PM
In HDP 2.6.0.3 ( ATLAs version 0.8.0 ), cluster is generating log files of approx 1 GB/day. ATLAS log settings are set to INFO level. Is there any way to reduce volume of entries in log files ?
Created 08-10-2017 01:46 PM
You can try using the standard "log4j" Filtering mechanism to tell the log4j to not log the lines containing few Strings "blah blah"
<filter class="org.apache.log4j.varia.StringMatchFilter"> <param name="StringToMatch" value="blah blah" /> <param name="AcceptOnMatch" value="false" /> </filter>
- Here you can write many strings (part of a line) that you are seeing repeatedly in your log file and are useless.
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/varia/StringMatchFilter.html
- The above filter you can add inside the "Atlas"
Ambari UI --> Atlas --> Configs --> Advanced --> Advanced atlas-log4j
Your desired Appender Tag something like following:
<appender .....> . . <filter class="org.apache.log4j.varia.StringMatchFilter"> <param name="StringToMatch" value="blah blah" /> <param name="AcceptOnMatch" value="false" /> </filter> </appender>
.
Reference: http://middlewaremagic.com/jboss/?p=296
.
Other option will be to change to Logging level from INFO to WARN ( but that will then show only WARNING, ERROS, CRITICAL,FATAL) messages, No INFO messages.
Created 08-10-2017 01:46 PM
You can try using the standard "log4j" Filtering mechanism to tell the log4j to not log the lines containing few Strings "blah blah"
<filter class="org.apache.log4j.varia.StringMatchFilter"> <param name="StringToMatch" value="blah blah" /> <param name="AcceptOnMatch" value="false" /> </filter>
- Here you can write many strings (part of a line) that you are seeing repeatedly in your log file and are useless.
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/varia/StringMatchFilter.html
- The above filter you can add inside the "Atlas"
Ambari UI --> Atlas --> Configs --> Advanced --> Advanced atlas-log4j
Your desired Appender Tag something like following:
<appender .....> . . <filter class="org.apache.log4j.varia.StringMatchFilter"> <param name="StringToMatch" value="blah blah" /> <param name="AcceptOnMatch" value="false" /> </filter> </appender>
.
Reference: http://middlewaremagic.com/jboss/?p=296
.
Other option will be to change to Logging level from INFO to WARN ( but that will then show only WARNING, ERROS, CRITICAL,FATAL) messages, No INFO messages.
Created 08-10-2017 01:57 PM
I think this link will show you how to change INFO to WARN or ERROR, as well as adjust rotating log sizes
Created 08-15-2017 05:40 PM
Is there a specific exception that is logged over and over again? If yes, can you please let me know what the exception is?