Cleanup NiFi logs which are older than x days?

[#1]

Hi,

My NiFi is hosted on 3 node clustered mode, NiFi is maintaining logs and might be the chances of disk overflow. I wanted to cleanup NiFi logs which are older than some day.

What is the best approach to do this?

Thanks,

[#2 · Accepted solution]

@Rahoul A

The rules that govern log rotation and retention are all configured in the NiFi logback.xml file.
-
If you find that log files are not being deleted, verify that the user running the NiFi service has sufficient open files handles allocated it to it. Not having enough file handles can result in failed log rotations and clean-up.
-

While logged in as the user who owns the NiFi process, execute the following command:

ulimit -a

-

Recommended starting value is 50000. Depending on NiFi dataflow and flow volumes, this value may need to be even larger like 999999.

-

Thanks,

Matt

[#3]

@Matt Clarke, can you please post me the changes which i need to do in logback.xml in order to delete logs which are older than 10 days?

[#4]

@Rahoul A

Here is a typical logback appender entry for the nifi-app.log:

      <appender name="APP_FILE">
            <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
            <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
            For daily rollover, use 'app_%d.log'.
            For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
            To GZIP rolled files, replace '.log' with '.log.gz'.
            To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
            <!-- optional setting for keeping 10GB total of log files
            <totalSizeCap>10GB</totalSizeCap>
            -->
            </rollingPolicy>
            <immediateFlush>true</immediateFlush>
            <encoder>
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
      </appender>

-

Their are couple of lines you want to pay attention to here:

<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

The above line defines what type of policy is being used by this appender. Here is will roll based in size and time.

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

In the above line you can see log is designed to roll on the hour based on yyyy-MM-dd_HH.

<maxFileSize>100MB</maxFileSize>

The above lines states no single log file should be larger then 100 MB, so you must account for scenarios where logs for a single hour may exceed 100 MB. That is what the "%i" is for in the fileNamePattern line above.

<maxHistory>30</maxHistory>

The above lines dictates how many "hours" of logs to retain. Each hour may contain 1 to many log files because of the max size constraint.

- - -

So in your case, using above as an example you would set maxHistory to "240".

- - -

You will also see an additional option property (commented out in above example):

            <!-- optional setting for keeping 10GB total of log files
            <totalSizeCap>10GB</totalSizeCap>
            -->

This property is important if you have limited logging disk space, It will start rolling off the oldest logs if you exceed this configured max threshold.

-

Thank you,

Matt

[#5]

@Matt Clarke, Really appreciate your detailed answers as always.