question Re: Nifi Log Rotation in Support Questions

Nifi Log Rotation

Alexy — Tue, 02 Jan 2024 11:26:42 GMT

I need to configure nifi for log rotation. I have seen this article: Understanding how the logback.xml configuration in... - Cloudera Community - 248662

Here I have few doubts regarding the property.

<maxHistory>30</maxHistory>

It's mentioned in comments as

<!-- keep 30 log files worth of history -->

But on checking I have come across a document logback.xml Example - Mkyong.com where its mentioned as

<maxFileSize>10MB</maxFileSize>  <totalSizeCap>20GB</totalSizeCap>  <maxHistory>60</maxHistory>

So, my doubt is does maxhistory refers to number of file or number of days. Also do we really need to add totalSizeCap because anyway we are adding maxhistory and maxfilesize.

Re: Nifi Log Rotation

MattWho — Tue, 02 Jan 2024 14:16:01 GMT

@Alexy

The answer to your question is neither...
The following property is relative to the configuration of the "fileNamingPattern" property.

Let's say we have the following fileNamingPattern property:

<fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.log.gz</fileNamePattern>

This pattern would create one log file per day and then maxHistory would would dictate number of days to retain as 60. In this specific example the number of days retained and number of files and number of days both equal 60.

and another example:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log</fileNamePattern>

Here you will notice the date pattern is based on hourly logs. So the "maxHistory" property in this example would dictate the number of hours worth of logs to retain (not days).

Now lets look at a slightly different example:

<fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern> <maxFileSize>10MB</maxFileSize>

In this example you'll notice "%i" has been added to pattern. "%i" works in conjunction with the "maxFileSize" property. throughout a single day any time the log file sizes reaches 10MB, an incremental number log for the same day is created.
app.2024-01-02.1.log.gz
app.2024-01-02.2.log.gz
app.2024-01-02.3.log.gz
...
The MaxHistory does not take into consideration the "%i" incremental logs. It is still based on the %d pattern to retain 60 days of logs. So in this case you would have any number of files potentially created across those 60 days.

The same applies to the hours based example if "%i" is added to the fileNamingPattern.

in order to protect disk usage you will often see the following used:

<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <maxHistory>30</maxHistory> <totalSizeCap>10GB</totalSizeCap> </rollingPolicy>

What this does is create hourly logs that will incrementally role during each hour if the log file reaches 100MB in size.
Then to prevent excessive disk usage should logging unexpectedly increase in log generation, the "TotalSizeCap" will start deleting the oldest rolled logs until max usage is below 10GB. This may mean that 30 hours of logs are no longer retained, but you avoid disk usage issues when the unexpected happens.

Hopefully this clears this up for you.

NOTE: Above is based on the "SizeAndTimeBasedRollingPolicy". LogBack as other rolling Policy providers.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi Log Rotation

Alexy — Tue, 09 Jan 2024 11:29:46 GMT

@MattWho Really appreciate your detailed answer

I need few clarifications.
You mentioned "TotalSizeCap" will start deleting the oldest rolled logs until max usage is below 10GB" is it calculated per day?
We are haven't used the property "TotalSizeCap" property. This is What we were using:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>50MB</maxFileSize>  <maxHistory>10</maxHistory>

So, another doubt is regarding "maxHistory" property. This means latest 10hrs of log is kept, is it? if so, will the older 12th & 11th hour log be deleted? and is it calculated per day?

Currently I'm observing the log rotation after editing few values. Added "totalsizecap" too. Your inputs and suggestions will be valuable.

Re: Nifi Log Rotation

MattWho — Tue, 09 Jan 2024 14:06:39 GMT

@Alexy

1. TotalSizeCap is across all log files, it is not based on anything else. So with this set depending on number of incremental (%i) 50 MB logs are created each hour, you may ned up retaining less then 10 hours of logs.

2. Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created.

Logback is not developed by NiFi and has some oddities to it. It works well when using both maxHistory and TotalSizeCap, but you may see odd behavior when only using MaxHistory. When only MaxHistory is used, Logback does not look back further than configured maxHistory. So while a continually running service works great, things get weird when the service is down for a period of time the extends beyond the maxHistory.

Lets say you have logs for last 10 hours (hours 1 -10) already and you bring down NiFi for a couple hours. So now you have logs that go back 12 hours (hours 3-12)when you restart the service. Logback will ignore the hour 11 and hour 12 logs since it is only looking to purge the hour 10 logs when the hour rolls over. So a few hours later you would still see the logs for hour 11 and 12 lingering around with a continuous 10 hour block of logs logback is still tracking on. The TotalSizeCap property however would make sure those old logs get deleted. This odd behavior to me was also brought up with logback community, but ultimately community decided to keep this behavior.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi Log Rotation

Alexy — Wed, 10 Jan 2024 03:23:18 GMT

@mat I'm good the functionality of "TotalSizeCap", but still in a confusion with " Max History".

In the first reply you mentioned like this:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log</fileNamePattern>

"Here you will notice the date pattern is based on hourly logs. So the "maxHistory" property in this example would dictate the number of hours worth of logs to retain (not days)."

Last reply you have mentioned like this: Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created."

Please have a look at the above bolded characters.

Nifi version - 1.17.0
This is a sample of Nifi app log file which is being generated now.
nifi-app_2024-01-09_11.0.log
nifi-app_2024-01-09_11.1.log
nifi-app_2024-01-09_11.2.log
nifi-app_2024-01-09_11.3.log

I understood that its writted in an hourly basis. And this is the 11th hour log file.
Suppose I give "Max History" as 2 and there is no " TotalSizeCap" What should ideally happen? (I'm asking this because I need to make sure we are on the same page. According to me it will keep only latest 2hr of log only🙄)

Re: Nifi Log Rotation

MattWho — Wed, 10 Jan 2024 21:11:37 GMT

@Alexy

Sorry about the confusion, it shoudl have said hours instead of days in my response:

2. Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 hours unless TotalSizeCap is reached before 10 hours of logs are created.

So you example here:

nifi-app_2024-01-09_11.0.log nifi-app_2024-01-09_11.1.log nifi-app_2024-01-09_11.2.log nifi-app_2024-01-09_11.3.log

Implies you are using the pattern:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

This also means you are using "maxFileSize" property as well that controls when each incremental hour log is rolled.

Without a "TotalSizeCap" property set, logs will be retained for 2 hours or until you run out of disk space since there is no boundary on the number of incremental logs that may roll within each of those 2 hours.

nifi-app_2024-01-09_11.0.log nifi-app_2024-01-09_11.1.log nifi-app_2024-01-09_11.2.log nifi-app_2024-01-09_11.3.log nifi-app_2024-01-09_11.4.log nifi-app_2024-01-09_11.5.log ... nifi-app_2024-01-09_12.0.log nifi-app_2024-01-01_12.1.log nifi-app_2024-01-09_12.2.log nifi-app_2024-01-09_12.3.log nifi-app_2024-01-09_12.4.log ...

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi Log Rotation

Alexy — Mon, 15 Jan 2024 13:35:56 GMT

@MattWho Appreciate your detailed information in clearing the doubts.

Log rotation is working fine. But problem happens when we are switching off&on the NIFI. I good with Nifi being ignoring the logs when it's switched off. So I have used a new property called:

I have taken this from the document: Chapter 4: Appenders (qos.ch)

But the problem is all the old logs are still present. I believe some additional configurations are to be added in order to remove previous days logs(last 2 hours logs only). Correct me if I'm wrong. This is my current configuration.

<appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">  <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log.gz</fileNamePattern>  <maxHistory>2</maxHistory> <totalSizeCap>1GB</totalSizeCap> <cleanHistoryOnStart>true</cleanHistoryOnStart> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender>

I switched off the EKS cluster and on next day when I switched on, the log of last two hours of previous day is still present.

These are the jars available:

log4j-over-slf4j-1.7.36.jar
logback-classic-1.2.11.jar
logback-core-1.2.11.jar

Re: Nifi Log Rotation

MattWho — Wed, 17 Jan 2024 15:28:08 GMT

@Alexy

I have never used cleanHistoryOnStart and the docs are not super clear.
Archive removal normally happens at time of log rotation. This is why older logs beyond window in which they should be retained linger around when application is down and then started up.

I cam across this post that has some interesting responses:
https://stackoverflow.com/questions/54680434/the-old-log-files-didnt-get-removed-when-using-logback-rollingfileappender

Implied from that that post it is not removing the files because of your rolling policy

%d{yyyy-MM-dd_HH}

If what they are saying is correct, it will not remove your old logs because they are form a different day and your log rotation is based on hour within current day.

I more commonly uses "totalSizeCap" setting a value here larger than what i expect to see retained in my desired history time window or even larger based on available disk space. This has worked for me to clean old stuff up so it is not around forever.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi Log Rotation

Alexy — Thu, 18 Jan 2024 08:52:01 GMT

@MattWho In my case "totalSizeCap" is also not working as expected. On checking further I understood that it's fixed in 1.3.x version of logback. This is a similar issue: logging - Logback: SizeAndTimeBasedRollingPolicy applies totalSizeCap to each day in maxHistory - Stack Overflow

Since I can't change Nifi version I'm moving forward with initcontaier functionality which will delete logs older than specified time when cluster starts up. Also need to change the comment related to maxhistory. Still its given as log file rather than hour or day. Are you aware of any of the Nifi maintaniers ?

Re: Nifi Log Rotation

MattWho — Thu, 18 Jan 2024 14:14:00 GMT

@Alexy

I looked at the default logback.xml bundled with the latest NiFi 2.0 releases and see which is much better description of what it does:

<maxHistory>30</maxHistory>

Control maximum number does not mean it is exactly the configured number being retained since maxHistory applies the TimeBasedRolling policy.

older versions of NiFi have following which is misleading:

<maxHistory>30</maxHistory>

I know a lot of committers and they are not going to go back to older releases and update it there.

NiFi went to Logback 1.3 as of Apache NiFi release 1.20 as part of:
https://issues.apache.org/jira/browse/NIFI-10580
later upgraded to:
1.3.8 ---> Apache NiFi 1.23.1 and 1.24.0 NIFI-11881
1.3.14 --> Apache NiFI 1.25 and 2.0.0-M2 NIFI-12462

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi Log Rotation

Alexy — Thu, 18 Jan 2024 16:26:54 GMT

@MattWho
I also noticed these comments and raised a concern in Apache Nifi Slack Channel. Let see whether they will have a fix on it.

Thanks again for your inputs😊.

Re: Nifi Log Rotation

MattWho — Thu, 18 Jan 2024 17:55:12 GMT

@Alexy

MaxHistory is relative to timedBased Policy date pattern:
in the example shown in this thread you mentioned:
https://mkyong.com/logging/logback-xml-example/

The full rollingPolicy example is:

<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>  <maxFileSize>10MB</maxFileSize>  <totalSizeCap>20GB</totalSizeCap>  <maxHistory>60</maxHistory> </rollingPolicy>

Within the fileNamePattern we see the configured date pattern "%d{yyyy-MM-dd}" which is based on days. So, maxHistory of 60 means in this specific example to retain "60" days.

Now if that date pattern instead was "%d{yyyy-MM-dd_HH}" which is based on hours instead, the same maxHistory=60 would now mean 60 hours of logs. But since this policy also archives on size, the fileNamePattern also includes "%i" which roles the log incrementally each time it reaches the configured maxFileSize= 10MB. So it will keep 60 days of logs; however, each of those 60 days could have 1 too many rolled logs depending on how much logging is output.
app.2024-01-18.1.log.gz
app.2024-01-18.2.log.gz
app.2024-01-18.3.log.gz
app.2024-01-18.4.log.gz
app.2024-01-18.5.log.gz
...

So if your application decides to unexpectedly start producing large amounts of log output, you could quickly consume all your disk space with incremental logs. This is where the TotalSizeCap setting comes into the picture. It's job is to say regardless of number of logs you want to keep (60 days in this specific configuration example), start deleting old archives when total log archives exceeds this configured size cap.

maxHistory is relative to the configured date pattern and is always days.

So comment in above exact example is correct. it is not about number of log files and since pattern is based in days, it is correct.

A better generic comment is:

since it makes no specific relative to the dat pattern being used. It simply states that the maxHistory value correlates to number of log archives that are retained. What that equates (minutes, hours, days, etc) to depends on the user defined data pattern.

I modified the example in the article you liked that I created with above comment instead of days even though I called it out later in my article for clarity.

In my opinion, TotalSizeCap is something you should use anytime you use the SizeAndTimeBasedRollingPolicy. That is because maxHistory setting does not take into account the incrementally (%i) created log files. So using above example where it is creating daily logs (%d{yyyy-MM-dd}) the maxHistory= 60 says to keep 60 days of logs.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi Log Rotation

ayu_dev — Mon, 03 Feb 2025 04:31:33 GMT

</contextListener>

<fileNamePattern>${logDir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

<fileNamePattern>${logDir}/nifi-user_%d.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

<file>C:/nifi-2.0.0/logs/nifi-request.log</file>

<fileNamePattern>${logDir}/nifi-request_%d.log</fileNamePattern>

</rollingPolicy>

</encoder>

</appender>

<file>C:/nifi-2.0.0/logs/nifi-bootstrap.log</file>

<fileNamePattern>${logDir}/nifi-bootstrap_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

<file>C:/nifi-2.0.0/logs/nifi-deprecation.log</file>

<fileNamePattern>${logDir}/nifi-deprecation_%d.%i.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger %msg%n</pattern>

</encoder>

</appender>

<sift>

<file>C:/nifi-2.0.0/logs/nifi-app-${logFileSuffix}.log</file>

<fileNamePattern>${logDir}/nifi-app-${logFileSuffix}_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

</rollingPolicy>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

</sift>

</appender>

<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

</encoder>

</appender>

<appender-ref ref="DEPRECATION_FILE"/>

</logger>

<appender-ref ref="APP_FILE"/>

</logger>

<!--

Logger for capturing user events. We do not want to propagate these

log events to the root logger. These messages are only sent to the

user-log appender.

-->