question Is there any way to reduce log file sizes in ATLAS version 0.8.0 ? in Archives of Support Questions (Read Only)

Is there any way to reduce log file sizes in ATLAS version 0.8.0 ?

shishir_saxena4 — Thu, 10 Aug 2017 20:27:57 GMT

In HDP 2.6.0.3 ( ATLAs version 0.8.0 ), cluster is generating log files of approx 1 GB/day. ATLAS log settings are set to INFO level. Is there any way to reduce volume of entries in log files ?

Re: Is there any way to reduce log file sizes in ATLAS version 0.8.0 ?

jsensharma — Thu, 10 Aug 2017 20:46:35 GMT

@Shishir Saxena

You can try using the standard "log4j" Filtering mechanism to tell the log4j to not log the lines containing few Strings "blah blah"

    <filter class="org.apache.log4j.varia.StringMatchFilter">
      <param name="StringToMatch" value="blah blah" />
      <param name="AcceptOnMatch" value="false" />
    </filter>

- Here you can write many strings (part of a line) that you are seeing repeatedly in your log file and are useless.
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/varia/StringMatchFilter.html

- The above filter you can add inside the "Atlas"

Ambari UI --> Atlas --> Configs --> Advanced --> Advanced atlas-log4j

Your desired Appender Tag something like following:

<appender  .....>
   .
   .
    <filter class="org.apache.log4j.varia.StringMatchFilter">
      <param name="StringToMatch" value="blah blah" />
      <param name="AcceptOnMatch" value="false" />
    </filter>   
</appender>

Reference: http://middlewaremagic.com/jboss/?p=296

Other option will be to change to Logging level from INFO to WARN ( but that will then show only WARNING, ERROS, CRITICAL,FATAL) messages, No INFO messages.

Re: Is there any way to reduce log file sizes in ATLAS version 0.8.0 ?

gkeys — Thu, 10 Aug 2017 20:57:36 GMT

I think this link will show you how to change INFO to WARN or ERROR, as well as adjust rotating log sizes

https://community.hortonworks.com/content/supportkb/49455/atlas-default-logging-is-filling-up-file-system-ho.html

Re: Is there any way to reduce log file sizes in ATLAS version 0.8.0 ?

amestry — Wed, 16 Aug 2017 00:40:09 GMT

Is there a specific exception that is logged over and over again? If yes, can you please let me know what the exception is?