Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi Log Rotation

avatar
Contributor

I need to configure nifi for log rotation. I have seen this article: Understanding how the logback.xml configuration in... - Cloudera Community - 248662

Here I have few doubts regarding the property.

<maxHistory>30</maxHistory>

 It's mentioned in comments as

<!-- keep 30 log files worth of history -->

But on checking I have come across a document logback.xml Example - Mkyong.com where its mentioned as 

 

 

 

            <!-- each archived file, size max 10MB -->
            <maxFileSize>10MB</maxFileSize>
            <!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
            <totalSizeCap>20GB</totalSizeCap>
            <!-- 60 days to keep -->
            <maxHistory>60</maxHistory>

 

 

 

So, my doubt is does maxhistory refers to number of file or number of days. Also do we really need to add totalSizeCap because anyway we are adding maxhistory and maxfilesize

11 REPLIES 11

avatar
Contributor

@MattWho 
I also noticed these comments and raised a concern in Apache Nifi Slack Channel. Let see whether they will have a fix on it. 

Thanks again for your inputs😊.

avatar
Master Mentor

@Alexy 

MaxHistory is relative to timedBased Policy date pattern:
in the example shown in this thread you mentioned:
https://mkyong.com/logging/logback-xml-example/

The full rollingPolicy example is:

        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <!-- each archived file, size max 10MB -->
            <maxFileSize>10MB</maxFileSize>
            <!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
            <totalSizeCap>20GB</totalSizeCap>
            <!-- 60 days to keep -->
            <maxHistory>60</maxHistory>
        </rollingPolicy>

Within the fileNamePattern we see the configured date pattern "%d{yyyy-MM-dd}" which is based on days.  So, maxHistory of 60 means in this specific example to retain "60" days.

Now if that date pattern instead was "%d{yyyy-MM-dd_HH}" which is based on hours instead, the same maxHistory=60 would now mean 60 hours of logs.  But since this policy also archives on size, the fileNamePattern also includes "%i" which roles the log incrementally each time it reaches the configured maxFileSize= 10MB.  So it will keep 60 days of logs; however, each of those 60 days could have 1 too many rolled logs depending on how much logging is output. 
app.2024-01-18.1.log.gz
app.2024-01-18.2.log.gz
app.2024-01-18.3.log.gz
app.2024-01-18.4.log.gz
app.2024-01-18.5.log.gz
...

So if your application decides to unexpectedly start producing large amounts of log output, you could quickly consume all your disk space with incremental logs.  This is where the TotalSizeCap setting comes into the picture. It's job is to say regardless of number of logs you want to keep (60 days in this specific configuration example), start deleting old archives when total log archives exceeds this configured size cap.

maxHistory is relative to the configured date pattern and is always days.

So comment in above exact example is correct.  it is not about number of log files and since pattern is based in days, it is correct.

A better generic comment is:

<!-- Control the maximum number of log archive files kept and asynchronously delete older files -->

since it makes no specific relative to the dat pattern being used. It simply states that the maxHistory value correlates to number of log archives that are retained.  What that equates (minutes, hours, days, etc) to depends on the user defined data pattern.

I modified the example in the article you liked that I created with above comment instead of days even though I called it out later in my article for clarity.

In my opinion, TotalSizeCap is something you should use anytime you use the SizeAndTimeBasedRollingPolicy.  That is because maxHistory setting does not take into account the incrementally (%i) created log files.  So using above example where it is creating daily logs (%d{yyyy-MM-dd}) the maxHistory= 60 says to keep 60 days of logs.  

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt