Gold ⭐️

Question

Nifi Log Rotation

13 replies
13557 views
Translate All

I need to configure nifi for log rotation. I have seen this article: Understanding how the logback.xml configuration in... - Cloudera Community - 248662

Here I have few doubts regarding the property.

<maxHistory>30</maxHistory>

It's mentioned in comments as

But on checking I have come across a document logback.xml Example - Mkyong.com where its mentioned as

            <!-- each archived file, size max 10MB -->
            <maxFileSize>10MB</maxFileSize>
            <!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
            <totalSizeCap>20GB</totalSizeCap>
            <!-- 60 days to keep -->
            <maxHistory>60</maxHistory>

So, my doubt is does maxhistory refers to number of file or number of days. Also do we really need to add totalSizeCap because anyway we are adding maxhistory and maxfilesize.

Apache NiFi

MattWho

Titanium ⭐️⭐️⭐️

@Alexy

The answer to your question is neither...
The following property is relative to the configuration of the "fileNamingPattern" property.

<maxHistory>60</maxHistory>

Let's say we have the following fileNamingPattern property:

<fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.log.gz</fileNamePattern>

This pattern would create one log file per day and then maxHistory would would dictate number of days to retain as 60. In this specific example the number of days retained and number of files and number of days both equal 60.

and another example:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log</fileNamePattern>

Here you will notice the date pattern is based on hourly logs. So the "maxHistory" property in this example would dictate the number of hours worth of logs to retain (not days).

Now lets look at a slightly different example:

<fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<maxFileSize>10MB</maxFileSize>

In this example you'll notice "%i" has been added to pattern. "%i" works in conjunction with the "maxFileSize" property. throughout a single day any time the log file sizes reaches 10MB, an incremental number log for the same day is created.
app.2024-01-02.1.log.gz
app.2024-01-02.2.log.gz
app.2024-01-02.3.log.gz
...
The MaxHistory does not take into consideration the "%i" incremental logs. It is still based on the %d pattern to retain 60 days of logs. So in this case you would have any number of files potentially created across those 60 days.

The same applies to the hours based example if "%i" is added to the fileNamingPattern.

in order to protect disk usage you will often see the following used:

<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<maxFileSize>100MB</maxFileSize>
<maxHistory>30</maxHistory>
<totalSizeCap>10GB</totalSizeCap>
</rollingPolicy>

What this does is create hourly logs that will incrementally role during each hour if the log file reaches 100MB in size.
Then to prevent excessive disk usage should logging unexpectedly increase in log generation, the "TotalSizeCap" will start deleting the oldest rolled logs until max usage is below 10GB. This may mean that 30 hours of logs are no longer retained, but you avoid disk usage issues when the unexpected happens.

Hopefully this clears this up for you.

NOTE: Above is based on the "SizeAndTimeBasedRollingPolicy". LogBack as other rolling Policy providers.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

AlexyAuthor

Gold ⭐️

@MattWho Really appreciate your detailed answer

I need few clarifications.
You mentioned "TotalSizeCap" will start deleting the oldest rolled logs until max usage is below 10GB" is it calculated per day?
We are haven't used the property "TotalSizeCap" property. This is What we were using:

                <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
                <maxFileSize>50MB</maxFileSize>
                <!-- keep 30 log files worth of history -->
                <maxHistory>10</maxHistory>

So, another doubt is regarding "maxHistory" property. This means latest 10hrs of log is kept, is it? if so, will the older 12th & 11th hour log be deleted? and is it calculated per day?

Currently I'm observing the log rotation after editing few values. Added "totalsizecap" too. Your inputs and suggestions will be valuable.

Translate

MattWho

Titanium ⭐️⭐️⭐️

@Alexy

1. TotalSizeCap is across all log files, it is not based on anything else. So with this set depending on number of incremental (%i) 50 MB logs are created each hour, you may ned up retaining less then 10 hours of logs.

2. Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created.

Logback is not developed by NiFi and has some oddities to it. It works well when using both maxHistory and TotalSizeCap, but you may see odd behavior when only using MaxHistory. When only MaxHistory is used, Logback does not look back further than configured maxHistory. So while a continually running service works great, things get weird when the service is down for a period of time the extends beyond the maxHistory.

Lets say you have logs for last 10 hours (hours 1 -10) already and you bring down NiFi for a couple hours. So now you have logs that go back 12 hours (hours 3-12)when you restart the service. Logback will ignore the hour 11 and hour 12 logs since it is only looking to purge the hour 10 logs when the hour rolls over. So a few hours later you would still see the logs for hour 11 and 12 lingering around with a continuous 10 hour block of logs logback is still tracking on. The TotalSizeCap property however would make sure those old logs get deleted. This odd behavior to me was also brought up with logback community, but ultimately community decided to keep this behavior.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

AlexyAuthor

Gold ⭐️

@mat I'm good the functionality of "TotalSizeCap", but still in a confusion with " Max History".

In the first reply you mentioned like this:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log</fileNamePattern>

"Here you will notice the date pattern is based on hourly logs. So the "maxHistory" property in this example would dictate the number of hours worth of logs to retain (not days)."

Last reply you have mentioned like this: Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created."

Please have a look at the above bolded characters.

Nifi version - 1.17.0
This is a sample of Nifi app log file which is being generated now.
nifi-app_2024-01-09_11.0.log
nifi-app_2024-01-09_11.1.log
nifi-app_2024-01-09_11.2.log
nifi-app_2024-01-09_11.3.log

I understood that its writted in an hourly basis. And this is the 11th hour log file.
Suppose I give "Max History" as 2 and there is no " TotalSizeCap" What should ideally happen? (I'm asking this because I need to make sure we are on the same page. According to me it will keep only latest 2hr of log only🙄)

Translate

MattWho

Titanium ⭐️⭐️⭐️

@Alexy

Sorry about the confusion, it shoudl have said hours instead of days in my response:

2. Max History does not count incremental (%i) logs generated.  It is based on the date pattern used %d{yyyy-MM-dd_HH}.  So you role logs every hour, so MaxHistory would retain 10 hours unless TotalSizeCap is reached before 10 hours of logs are created.

So you example here:

nifi-app_2024-01-09_11.0.log
 nifi-app_2024-01-09_11.1.log
 nifi-app_2024-01-09_11.2.log
nifi-app_2024-01-09_11.3.log

Implies you are using the pattern:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

This also means you are using "maxFileSize" property as well that controls when each incremental hour log is rolled.

Without a "TotalSizeCap" property set, logs will be retained for 2 hours or until you run out of disk space since there is no boundary on the number of incremental logs that may roll within each of those 2 hours.

nifi-app_2024-01-09_11.0.log
nifi-app_2024-01-09_11.1.log
nifi-app_2024-01-09_11.2.log
nifi-app_2024-01-09_11.3.log
nifi-app_2024-01-09_11.4.log
nifi-app_2024-01-09_11.5.log
...
nifi-app_2024-01-09_12.0.log
nifi-app_2024-01-01_12.1.log
nifi-app_2024-01-09_12.2.log
nifi-app_2024-01-09_12.3.log
nifi-app_2024-01-09_12.4.log
...

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

AlexyAuthor

Gold ⭐️

@MattWho Appreciate your detailed information in clearing the doubts.

Log rotation is working fine. But problem happens when we are switching off&on the NIFI. I good with Nifi being ignoring the logs when it's switched off. So I have used a new property called:

<cleanHistoryOnStart>true</cleanHistoryOnStart>

I have taken this from the document: Chapter 4: Appenders (qos.ch)

But the problem is all the old logs are still present. I believe some additional configurations are to be added in order to remove previous days logs(last 2 hours logs only). Correct me if I'm wrong. This is my current configuration.

        <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
            <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
            <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                <!--
                For daily rollover, use 'app_%d.log'.
                For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
                To GZIP rolled files, replace '.log' with '.log.gz'.
                To ZIP rolled files, replace '.log' with '.log.zip'.
                -->
                <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log.gz</fileNamePattern>
                <!-- keep 30 log files worth of history -->
                <maxHistory>2</maxHistory>
                <totalSizeCap>1GB</totalSizeCap>
                <cleanHistoryOnStart>true</cleanHistoryOnStart>
            </rollingPolicy>
            <immediateFlush>true</immediateFlush>
            <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
                <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
        </appender>

I switched off the EKS cluster and on next day when I switched on, the log of last two hours of previous day is still present.

These are the jars available:

log4j-over-slf4j-1.7.36.jar
logback-classic-1.2.11.jar
logback-core-1.2.11.jar

Translate

MattWho

Titanium ⭐️⭐️⭐️

@Alexy

I have never used cleanHistoryOnStart and the docs are not super clear.
Archive removal normally happens at time of log rotation. This is why older logs beyond window in which they should be retained linger around when application is down and then started up.

I cam across this post that has some interesting responses:
https://stackoverflow.com/questions/54680434/the-old-log-files-didnt-get-removed-when-using-logback-rollingfileappender

Implied from that that post it is not removing the files because of your rolling policy

%d{yyyy-MM-dd_HH}

If what they are saying is correct, it will not remove your old logs because they are form a different day and your log rotation is based on hour within current day.

I more commonly uses "totalSizeCap" setting a value here larger than what i expect to see retained in my desired history time window or even larger based on available disk space. This has worked for me to clean old stuff up so it is not around forever.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

AlexyAuthor

Gold ⭐️

@MattWho In my case "totalSizeCap" is also not working as expected. On checking further I understood that it's fixed in 1.3.x version of logback. This is a similar issue: logging - Logback: SizeAndTimeBasedRollingPolicy applies totalSizeCap to each day in maxHistory - Stack Overflow

Since I can't change Nifi version I'm moving forward with initcontaier functionality which will delete logs older than specified time when cluster starts up. Also need to change the comment related to maxhistory. Still its given as log file rather than hour or day. Are you aware of any of the Nifi maintaniers ?

Translate

MattWho

Titanium ⭐️⭐️⭐️

@Alexy

I looked at the default logback.xml bundled with the latest NiFi 2.0 releases and see which is much better description of what it does:

            <!-- Control the maximum number of log archive files kept and asynchronously delete older files -->
            <maxHistory>30</maxHistory>

Control maximum number does not mean it is exactly the configured number being retained since maxHistory applies the TimeBasedRolling policy.

older versions of NiFi have following which is misleading:

            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>

I know a lot of committers and they are not going to go back to older releases and update it there.

NiFi went to Logback 1.3 as of Apache NiFi release 1.20 as part of:
https://issues.apache.org/jira/browse/NIFI-10580
later upgraded to:
1.3.8 ---> Apache NiFi 1.23.1 and 1.24.0 NIFI-11881
1.3.14 --> Apache NiFI 1.25 and 2.0.0-M2 NIFI-12462

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

ayu_dev

Silver ⭐️

</contextListener>

<fileNamePattern>${logDir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

<fileNamePattern>${logDir}/nifi-user_%d.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

<file>C:/nifi-2.0.0/logs/nifi-request.log</file>

<fileNamePattern>${logDir}/nifi-request_%d.log</fileNamePattern>

</rollingPolicy>

</encoder>

</appender>

<file>C:/nifi-2.0.0/logs/nifi-bootstrap.log</file>

<fileNamePattern>${logDir}/nifi-bootstrap_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

<file>C:/nifi-2.0.0/logs/nifi-deprecation.log</file>

<fileNamePattern>${logDir}/nifi-deprecation_%d.%i.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger %msg%n</pattern>

</encoder>

</appender>

<sift>

<file>C:/nifi-2.0.0/logs/nifi-app-${logFileSuffix}.log</file>

<fileNamePattern>${logDir}/nifi-app-${logFileSuffix}_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

</sift>

</appender>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

<appender-ref ref="DEPRECATION_FILE"/>

</logger>

<appender-ref ref="APP_FILE"/>

</logger>

<!--

Logger for capturing user events. We do not want to propagate these

log events to the root logger. These messages are only sent to the

user-log appender.

-->