Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cleanup NiFi logs which are older than x days?

avatar

Hi,

My NiFi is hosted on 3 node clustered mode, NiFi is maintaining logs and might be the chances of disk overflow. I wanted to cleanup NiFi logs which are older than some day.

What is the best approach to do this?

Thanks,

1 ACCEPTED SOLUTION

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar

@Matt Clarke, can you please post me the changes which i need to do in logback.xml in order to delete logs which are older than 10 days?

avatar
Super Mentor

@Rahoul A

Here is a typical logback appender entry for the nifi-app.log:

      <appender name="APP_FILE">
            <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
            <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
            For daily rollover, use 'app_%d.log'.
            For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
            To GZIP rolled files, replace '.log' with '.log.gz'.
            To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
            <!-- optional setting for keeping 10GB total of log files
            <totalSizeCap>10GB</totalSizeCap>
            -->
            </rollingPolicy>
            <immediateFlush>true</immediateFlush>
            <encoder>
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
      </appender>

-

Their are couple of lines you want to pay attention to here:

<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

The above line defines what type of policy is being used by this appender. Here is will roll based in size and time.

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

In the above line you can see log is designed to roll on the hour based on yyyy-MM-dd_HH.

<maxFileSize>100MB</maxFileSize>

The above lines states no single log file should be larger then 100 MB, so you must account for scenarios where logs for a single hour may exceed 100 MB. That is what the "%i" is for in the fileNamePattern line above.

<maxHistory>30</maxHistory>

The above lines dictates how many "hours" of logs to retain. Each hour may contain 1 to many log files because of the max size constraint.

- - -

So in your case, using above as an example you would set maxHistory to "240".

- - -

You will also see an additional option property (commented out in above example):

            <!-- optional setting for keeping 10GB total of log files
            <totalSizeCap>10GB</totalSizeCap>
            -->

This property is important if you have limited logging disk space, It will start rolling off the oldest logs if you exceed this configured max threshold.

-

Thank you,

Matt

avatar

@Matt Clarke, Really appreciate your detailed answers as always.