Understanding how the logback.xml configuration in NiFi removes rolled logs

[#1]

Have you ever noticed some lingering old rolled log files in your nifi logs directory that never seem to get deleted?

This is a by-product of how logback works depending on how you have it configured.

-

Lets take a look at a default logback.xml configuration from NiFi:

            <appender name="APP_FILE">
            <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
            <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
            For daily rollover, use 'app_%d.log'.
            For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
            To GZIP rolled files, replace '.log' with '.log.gz'.
            To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- Control the maximum number of log archive files kept and asynchronously delete older files -->
            <maxHistory>30</maxHistory>
            <!-- optional setting for keeping 10GB total of log files
            <totalSizeCap>10GB</totalSizeCap>
            -->
            </rollingPolicy>
            <immediateFlush>true</immediateFlush>
            <encoder>
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
            </appender>

The above app log configuration will log to a file named nifi-app.log. Once that file reaches either 100 MB in size or crest the top of the hour, it will be rolled. You may end up with numerous log files within a single hour if there is an excessive amount of logging occurring in your NiFi.

-

A "maxHistory" of 30 means that the logger will only keep 30 hours (HH) of rolled logs. But that is not the full story here with how logback works. Not only does it control the number of hours to keep but also controls the max age of logs to evaluate for deletion. So the log files being left around that are more then 30 hours in age would be ignored when deletion thread ran.

-

So this naturally raises the question of how did these files get left behind in the first place? Typically this occurs if the file crest say 30 hours old while the application is stopped. When the application is restarted those older files end up getting ignored.

-

While the application is continuously running this works as one would normally expect. To simply clean-up these older rolled log files, you could run a touch command on them so their system file timestamp updates so they are no longer more then 30 hours old. They will then be considered within the 30 hour window and be deleted once the "maxHistory" count reaches 30.

-

However, above is not a permanent solution. I recommend instead to control file deletion by "totalSizeCap" setting (commented out by default in the NiFi logback.xml) It offers a couple of advantages:

1. The "%i" option in the fileNamePattern says to create sequential numbered log files every "maxFileSize" (100MB) within each hour. This help prevent any one log from getting to large, but has the downside of not being considered by "maxHistory" as individually counted files. So "maxHistory" set to 15 is 15 hours of logs even if each hour contains 2000 100MB log files. So you can see under heavy logging you can end up using a lot of logs space.

2. "TotalSizeCap" will start deleting old rolled log files as long as the log file date is less then "maxHistory" age. So lets say we want to retain up to 100GB of log history. We would set "maxHistory" to some very large value like 8760 (~1 year of hours) and set "totalSizeCap" to 100GB. Provided you hot 100GB before your hit 8760 hours.

-

Here is an example configuration:

            <appender name="APP_FILE">
            <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
            <rollingPolicy>
            <!--
            For daily rollover, use 'app_%d.log'.
            For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
            To GZIP rolled files, replace '.log' with '.log.gz'.
            To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>10MB</maxFileSize>
            <!-- keepup to 8,760 hours worth of log files -->
            <maxHistory>8760</maxHistory>
            <!-- optional setting for keeping 100 GB total of log files -->
            <totalSizeCap>100GB</totalSizeCap>
           <!-- archive removal will be executed on appender start up -->
            <cleanHistoryOnStart>true</cleanHistoryOnStart>
            </rollingPolicy>
            <immediateFlush>true</immediateFlush>
            <encoder>
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
            </appender>

-

Of course there is always a chance you could hit 8,760 hours worth of logs before reaching 100 GB of generated app logs, so you may need to tailor these setting based on app log sizes being generated by your particular running NiFi.

[#2]

Hi! Folks,

For some reason, it seems that the nifi service does not take the logging configurations of logback.xml and the service still generates the logs whose size exceed 100MB.

We simply use the default logback.xml settings such as follow:

<configuration scan="true" scanPeriod="30 seconds">
<contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
<resetJUL>true</resetJUL>
</contextListener>

<appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!--
For daily rollover, use 'app_%d.log'.
For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<maxFileSize>100MB</maxFileSize>
<!-- keep 30 log files worth of history -->
<maxHistory>10</maxHistory>
</rollingPolicy>
<immediateFlush>true</immediateFlush>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>

...

The file resides in the nifi service's conf folder "C:\nifi-1.15.3\conf\logback.xml".

According to the config, log "c:\nifi-1.15.3\logs\nifi-app.log" should rotate when its size reaches 100MB.

However, for some reason, the log rotation never happens.

I did some some research and it doesn't seem config file "logback.xml" needs to be referenced or included in other nifi property file (e.g., c:\nifi-1.15.3\nifi.properties) to be effective.

Please help.

[#3]

@myuintelli2021 

I just installed Apache NiFi 1.19.1 on my MAC and only modified the <maxFileSize> from default 100MB to 2 MB.  Log rotation is working as expected.
My current logback configuration for nifi-app.log:

<configuration scan="true" scanPeriod="30 seconds">
    <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" />

    <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
        <resetJUL>true</resetJUL>
    </contextListener>

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'app_%d.log'.
              For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>2MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
    </appender>

Log directory showing logs rotated based on size (adds .<num> before .log suffix) reaching max:

-rw-r--r--   1 nifiadmin  nifi   452B Jan 17 16:39 nifi-user.log
-rw-r--r--   1 nifiadmin  nifi   2.0M Jan 17 16:53 nifi-app_2023-01-17_16.0.log
-rw-r--r--   1 nifiadmin  nifi   2.0M Jan 17 16:58 nifi-app_2023-01-17_16.1.log
drwxr-xr-x  13 nifiadmin  nifi   416B Jan 17 16:58 .
-rw-r--r--   1 nifiadmin  nifi    96K Jan 17 16:58 nifi-request.log
-rw-r--r--   1 nifiadmin  nifi   178K Jan 17 16:58 nifi-app.log

I don't have access to a Windows environment to test there as it appears that is where you are testing, but it also works as expected in a linux environment I also have.

I recommend opening a community question if you are looking for assistance with your setup. A community article comments is not he correct place to troubleshoot configurations or environmental issues.  I encourage you to include details when you raise that question to include at least your complete OS version, complete java (JDK) version being used, and complete NiFi version.

Thank you,

Matt

[#4]

Hi! Matt,

Thanks for the help!

I tried your logback.xml configuration in our env and it did not make difference.

I noticed that the nifi version you used for testing is 1.19.1 while we use older version 1.15.3.
Do you know if the log rotation feature was implemented in v1.15.3 or only available in a later version?

[#5]

@myuintelli2021 
Logback is not something written by or for Apache NiFi.  
NiFi's default logback has changed very little for nifi-app.log, nifi-user.log, and nifi-bootstrap.log.  
All of which by default use the "RollingFileAppender". In the latest releases you may see that some additional appenders are now present; however, it is not uncommon for users to create additional appenders themselves in the logback.xml so that specific loggers can output to user defined log files rather than the defaults.  Logback supports numerous rolling policies:
https://logback.qos.ch/manual/appenders.html

These have existed along as NiFi has been using Logback.
Not sure if maybe the issue is specific to the version of Windows or JDK you are using?

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#6]

Hi! Matt,

Thanks for your follow up.

Our nifi 1.15.3 runs on windows 2019 datacenter edition and JDK version is 1.8.

We're aware that JDK version might be old. Do you think we should upgrade?

Thanks,
Ming

[#7]

@myuintelli2021 

Hello Ming,
NiFi 1.15.3 will support JDK 1.8 or 1.11.  We do strongly encourage users to be on the latest update version of either of those with NiFI.  So not sure what update release of JDK your are on and which JDK provider (Oracle, OpenJDK, etc) you are using.

I unfortunately do not have access ti a Windows 2019 datacenter edition to  see if I can reproduce myself to evaluate further.

I strongly encourage you to raise a community question to take your query further.  A new question will gather more attention rather than trying to diagnosis and resolve your specific issue with the comments of a community article.

Thank you,

Matt