Nifi logs rolling policy check....Please help

[#1]

I am new to nifi and I want nifi logs to roll daily with no retention policy

Is this configuration correct?

Another question was do we need to restart nifi after updating logback.xml file?

<appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!---<file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>-->
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log</fileNamePattern>
<maxFileSize>1GB</maxFileSize>
<!-- keep 30 log files worth of history -->
<maxHistory>1</maxHistory>
</rollingPolicy>
<immediateFlush>true</immediateFlush>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>

Version - 1.11.4

Thank you

[#2 · Accepted solution]

@hkh 

The appender you shared is not valid.
You have configured your appender rolling policy to use:

SizeAndTimeBasedRollingPolicy

However, your file naming patten only supports time based pattern

${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log

This leaves you will two options:
Option1:
- Keep using the "SizeAndTimeBasedRollingPolicy", but change your file naming pattern
- A pattern like "${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd}.%i.log" The "{yyyy-MM-dd}" is option but allows you to specify the date format.
- With above pattern logback will retain 1 day of log history, but may or may not have more then 1 log in a given day depending on volume of logging that is occurring.  If a daily log reaches the configured "maxFileSize" the log with roll.  This allows you to keep you logs at manageable sizes.  When the log rolls it will get a one up number applied per this new file naming pattern.
For example:
nifi.app_2021-04-28.1.log
nifi.app_2021-04-28.2.log
- While this can still result in an unbounded number fo incremental logs files created in a single day, you can control overall disk usage by adding another property within the "rollingPolicy" section that will start purging incremental rolled logs if max amount of space consumed by these rolled logs exceeds this set value.  Add this line below your "<maxHistory>1</maxHistory>" line:

<totalSizeCap>3GB</totalSizeCap>

Note: Will only remove rolled/archive logs and will not remove active log.

Option 2:
- Change the rolling policy you are using to "ch.qos.logback.core.rolling.TimeBasedRollingPolicy"
- With this policy you can keep the file name pattern you already have "${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log"
- You will need to comment out remove the following line:

<maxFileSize>1GB</maxFileSize>

it is only valid for size based rolling policies.

- The downside of this setup is that a single daily log file can grow unbounded.

As far as your other question goes....
The logback.xml is the only configuration file in NiFi that you can edit and its changes will take affect without needing a NiFi restart.  Some caveats... NiFi did not write logback and it certainly has its quirks.  For example... If you enter a bad configuration, it may simply just stop logging.   The way maxHistory works in logback prevents the cleanup from looking at older rolled logs outside the maxHistory window, so you will need to cleanup the older rolled logs manually initially and not expect logback to do that clean-up for you after editing the logback.xml.  Since you are making big changes to file naming pattern and potentially the rolling policy, i'd encourage you to restart NiFi anyway so it cleanly starts writing to the new log format on startup.

Hope you found this helpful. If so take a moment to login and click accept on this solution,

Matt

[#3]

Thanks a lot for detailed solution. 

[#4]

Now my files are getting rolled on daily basis 

But they are not getting deleted.

<appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!---<file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>-->
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'app_%d.log'.
For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log</fileNamePattern>
<!-- keep 30 log files worth of history -->
<maxHistory>1</maxHistory>
</rollingPolicy>
<immediateFlush>true</immediateFlush>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>

[#5]

@hkh 

Did you happen to restart your NiFi instance at any time?
Can you share a versbose (ls -latrh) listing of your log directory?

I do recommend you compress on rollover. nifi-app.logs can get very large especially if you are only creating one log file per day.  To compress on rollover, you would add a ".gz" to the end of your 

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.gz</fileNamePattern>

Thanks,

Matt