Is there any way to reduce log file sizes in ATLAS version 0.8.0 ?

[#1]

In HDP 2.6.0.3 ( ATLAs version 0.8.0 ), cluster is generating log files of approx 1 GB/day. ATLAS log settings are set to INFO level. Is there any way to reduce volume of entries in log files ?

[#2 · Accepted solution]

@Shishir Saxena

You can try using the standard "log4j" Filtering mechanism to tell the log4j to not log the lines containing few Strings "blah blah"

    <filter class="org.apache.log4j.varia.StringMatchFilter">
      <param name="StringToMatch" value="blah blah" />
      <param name="AcceptOnMatch" value="false" />
    </filter>

- Here you can write many strings (part of a line) that you are seeing repeatedly in your log file and are useless.
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/varia/StringMatchFilter.html

- The above filter you can add inside the "Atlas"

Ambari UI --> Atlas --> Configs --> Advanced --> Advanced atlas-log4j

Your desired Appender Tag something like following:

<appender  .....>
   .
   .
    <filter class="org.apache.log4j.varia.StringMatchFilter">
      <param name="StringToMatch" value="blah blah" />
      <param name="AcceptOnMatch" value="false" />
    </filter>   
</appender>

.

Reference: http://middlewaremagic.com/jboss/?p=296

.

Other option will be to change to Logging level from INFO to WARN ( but that will then show only WARNING, ERROS, CRITICAL,FATAL) messages, No INFO messages.

[#3]

I think this link will show you how to change INFO to WARN or ERROR, as well as adjust rotating log sizes

https://community.hortonworks.com/content/supportkb/49455/atlas-default-logging-is-filling-up-file-s...

[#4]

Is there a specific exception that is logged over and over again? If yes, can you please let me know what the exception is?