Nifi Log Rotation

[#1]

I need to configure nifi for log rotation. I have seen this article: Understanding how the logback.xml configuration in... - Cloudera Community - 248662

Here I have few doubts regarding the property.

<maxHistory>30</maxHistory>

 It's mentioned in comments as

<!-- keep 30 log files worth of history -->

But on checking I have come across a document logback.xml Example - Mkyong.com where its mentioned as 

            <!-- each archived file, size max 10MB -->
            <maxFileSize>10MB</maxFileSize>
            <!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
            <totalSizeCap>20GB</totalSizeCap>
            <!-- 60 days to keep -->
            <maxHistory>60</maxHistory>

So, my doubt is does maxhistory refers to number of file or number of days. Also do we really need to add totalSizeCap because anyway we are adding maxhistory and maxfilesize. 

[#2]

<configuration scan="true" scanPeriod="10 seconds">

    <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">

        <resetJUL>true</resetJUL>

    </contextListener>

    <!-- Make sure the log directory exists, if not create it -->

    <property name="logDir" value="C:/nifi-2.0.0/logs" />

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>C:/nifi-2.0.0/logs/nifi-app.log</file>

        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

            <fileNamePattern>${logDir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

            <maxFileSize>1MB</maxFileSize>

            <maxHistory>1</maxHistory>

            <totalSizeCap>1MB</totalSizeCap> <!-- Increased total size cap -->

            <cleanHistoryOnStart>true</cleanHistoryOnStart>

        </rollingPolicy>

        <immediateFlush>true</immediateFlush>

        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">

            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

        </encoder>

    </appender>

    <appender name="USER_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>C:/nifi-2.0.0/logs/nifi-user.log</file>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

            <fileNamePattern>${logDir}/nifi-user_%d.log</fileNamePattern>

            <maxHistory>1</maxHistory> <!-- Increased maxHistory -->

            <totalSizeCap>1MB</totalSizeCap>

            <cleanHistoryOnStart>true</cleanHistoryOnStart>

        </rollingPolicy>

        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">

            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

        </encoder>

    </appender>

    <appender name="REQUEST_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>C:/nifi-2.0.0/logs/nifi-request.log</file>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

            <fileNamePattern>${logDir}/nifi-request_%d.log</fileNamePattern>

            <maxHistory>1</maxHistory>

            <totalSizeCap>1MB</totalSizeCap> <!-- Increased size cap for request logs -->

            <cleanHistoryOnStart>true</cleanHistoryOnStart>

        </rollingPolicy>

        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">

            <pattern>%msg%n</pattern>

        </encoder>

    </appender>

    <appender name="BOOTSTRAP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>C:/nifi-2.0.0/logs/nifi-bootstrap.log</file>

        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

            <fileNamePattern>${logDir}/nifi-bootstrap_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

            <maxFileSize>1MB</maxFileSize>

            <maxHistory>1</maxHistory>

            <totalSizeCap>1MB</totalSizeCap> <!-- Increased total size cap -->

            <cleanHistoryOnStart>true</cleanHistoryOnStart>

        </rollingPolicy>

        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">

            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

        </encoder>

    </appender>

    <appender name="DEPRECATION_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>C:/nifi-2.0.0/logs/nifi-deprecation.log</file>

        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

            <fileNamePattern>${logDir}/nifi-deprecation_%d.%i.log</fileNamePattern>

            <maxFileSize>1MB</maxFileSize>

            <maxHistory>1</maxHistory>

            <totalSizeCap>1MB</totalSizeCap> <!-- Increased total size cap -->

            <cleanHistoryOnStart>true</cleanHistoryOnStart>

        </rollingPolicy>

        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">

            <pattern>%date %level [%thread] %logger %msg%n</pattern>

        </encoder>

    </appender>

    <appender name="DEDICATED_LOGGING" class="ch.qos.logback.classic.sift.SiftingAppender">

        <discriminator class="org.apache.nifi.logging.NifiDiscriminator"/>

        <sift>

            <appender name="APP-${logFileSuffix}_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

                <file>C:/nifi-2.0.0/logs/nifi-app-${logFileSuffix}.log</file>

                <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

                    <fileNamePattern>${logDir}/nifi-app-${logFileSuffix}_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>

                    <maxFileSize>1MB</maxFileSize>

                    <maxHistory>1</maxHistory>

                    <totalSizeCap>1MB</totalSizeCap> <!-- Increased total size cap -->

                    <cleanHistoryOnStart>true</cleanHistoryOnStart>

                </rollingPolicy>

                <immediateFlush>true</immediateFlush>

                <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">

                    <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

                </encoder>

            </appender>

        </sift>

    </appender>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">

        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">

            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>

        </encoder>

    </appender>

       <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->

    <!-- Deprecation Log -->

    <logger name="deprecation" level="WARN" additivity="false">

        <appender-ref ref="DEPRECATION_FILE"/>

    </logger>

    <logger name="org.apache.nifi" level="INFO"/>

    <logger name="org.apache.nifi.processors" level="WARN"/>

    <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>

    <logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>

    <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" />

    <logger name="org.apache.parquet.hadoop.InternalParquetRecordReader" level="WARN" />

    <!-- Py4J set to WARN to avoid verbose socket communication messages -->

    <logger name="py4j" level="WARN" />

    <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />

    <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />

    <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" />

    <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" level="ERROR" />

    <logger name="org.apache.zookeeper.server.quorum" level="ERROR" />

    <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />

    <logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" />

    <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" level="ERROR" />

    <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />

    <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" />

    <logger name="org.apache.curator.ConnectionState" level="OFF" />

    <!-- Logger for managing logging statements for nifi clusters. -->

    <logger name="org.apache.nifi.cluster" level="INFO"/>

    <!-- Logger for logging HTTP requests received by the web server. -->

    <logger name="org.apache.nifi.server.JettyServer" level="INFO"/>

    <!-- Logger for managing logging statements for jetty -->

    <logger name="org.eclipse.jetty" level="INFO"/>

    <!-- Suppress non-error messages related to Logback ServletContainerInitializer -->

    <logger name="org.eclipse.jetty.ee10.annotations.AnnotationConfiguration" level="WARN"/>

    <!-- Suppress non-error messages due to excessive logging by class or library -->

    <logger name="org.springframework" level="ERROR"/>

    <logger name="org.springframework.security" level="INFO"/>

    <!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) -->

    <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>

    <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a large amount of WARNS. Issue described in NIFI-5479. -->

    <logger name="org.eclipse.jetty.annotations.AnnotationParser" level="ERROR"/>

    <!-- Suppress non-error messages from SSHJ which was emitting large amounts of INFO logs by default -->

    <logger name="net.schmizz.sshj" level="WARN" />

    <logger name="com.hierynomus.sshj" level="WARN" />

    <!-- Suppress non-error messages from SMBJ which was emitting large amounts of INFO logs by default -->

    <logger name="com.hierynomus.smbj" level="WARN" />

    <!-- Suppress non-error messages from AWS KCL which was emitting large amounts of INFO logs by default -->

    <logger name="software.amazon.awssdk.kinesis" level="WARN" />

    <!-- Suppress non-error messages from Apache Atlas which was emitting large amounts of INFO logs by default -->

    <logger name="org.apache.atlas" level="WARN"/>

    <!-- Suppress non-error messages from JetBrains Xodus FileDataWriter related to FileChannel -->

    <logger name="jetbrains.exodus.io.FileDataWriter" level="WARN" />

    <!-- Suppress warnings from Lucene Vectorization Provider related to JDK incubator features -->

    <logger name="org.apache.lucene.internal.vectorization.VectorizationProvider" level="ERROR" />

    <!-- Suppress information from Lucene Memory Segment Provider related to native access configuration -->

    <logger name="org.apache.lucene.store.MemorySegmentIndexInputProvider" level="WARN" />

    <!-- These log messages would normally go to the USER_FILE log, but they belong in the APP_FILE -->

    <logger name="org.apache.nifi.web.security.requests" level="INFO" additivity="false">

        <appender-ref ref="APP_FILE"/>

    </logger>

    <!--

        Logger for capturing user events. We do not want to propagate these

        log events to the root logger. These messages are only sent to the

        user-log appender.

    -->

    <logger name="org.apache.nifi.web.security" level="INFO" additivity="false">

        <appender-ref ref="USER_FILE"/>

    </logger>

    <logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false">

        <appender-ref ref="USER_FILE"/>

    </logger>

    <logger name="org.apache.nifi.authorization" level="INFO" additivity="false">

        <appender-ref ref="USER_FILE"/>

    </logger>

    <logger name="org.apache.nifi.cluster.authorization" level="INFO" additivity="false">

        <appender-ref ref="USER_FILE"/>

    </logger>

    <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" additivity="false">

        <appender-ref ref="USER_FILE"/>

    </logger>

    <logger name="org.opensaml" level="WARN" additivity="false">

        <appender-ref ref="USER_FILE"/>

    </logger>

    <!-- Web Server Request Log -->

    <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" additivity="false">

        <appender-ref ref="REQUEST_FILE"/>

    </logger>

    <!--

        Logger for capturing Bootstrap logs and NiFi's standard error and standard out.

    -->

    <logger name="org.apache.nifi.bootstrap" level="INFO" additivity="false">

        <appender-ref ref="BOOTSTRAP_FILE" />

    </logger>

    <logger name="org.apache.nifi.bootstrap.Command" level="INFO" additivity="false">

        <appender-ref ref="CONSOLE" />

        <appender-ref ref="BOOTSTRAP_FILE" />

    </logger>

    <!-- Everything written to NiFi's Standard Out will be logged with the logger org.apache.nifi.StdOut at INFO level -->

    <logger name="org.apache.nifi.StdOut" level="INFO" additivity="false">

        <appender-ref ref="BOOTSTRAP_FILE" />

    </logger>

    <!-- Everything written to NiFi's Standard Error will be logged with the logger org.apache.nifi.StdErr at ERROR level -->

    <logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false">

        <appender-ref ref="BOOTSTRAP_FILE" />

    </logger>

    <root level="INFO">

        <appender-ref ref="APP_FILE" />

        <appender-ref ref="DEDICATED_LOGGING" />

    </root>

</configuration>

Help me the configuration as log rotation is only working for bootstrap and user.log files.Above is my configuration of logback.xml. why its not working, what changes should I do to make it work?

[#3]

@ayu_dev 

Welcome to the Cloudera Community.

Please start a new Cloudera Community question rather then adding a new thread on an existing question from over a year ago.  You'll get much better traction there.  Your query is unrelated to the original question in this thread around log rotation and retention policies.

Fell free to ping me in your new community question and I'll be happy to take a look.

Thank you,
Matt

[#4]

@MattWho 
I also noticed these comments and raised a concern in Apache Nifi Slack Channel. Let see whether they will have a fix on it. 

Thanks again for your inputs😊.

[#5]

@Alexy 

MaxHistory is relative to timedBased Policy date pattern:
in the example shown in this thread you mentioned:
https://mkyong.com/logging/logback-xml-example/

The full rollingPolicy example is:

        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <!-- each archived file, size max 10MB -->
            <maxFileSize>10MB</maxFileSize>
            <!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
            <totalSizeCap>20GB</totalSizeCap>
            <!-- 60 days to keep -->
            <maxHistory>60</maxHistory>
        </rollingPolicy>

Within the fileNamePattern we see the configured date pattern "%d{yyyy-MM-dd}" which is based on days.  So, maxHistory of 60 means in this specific example to retain "60" days.

Now if that date pattern instead was "%d{yyyy-MM-dd_HH}" which is based on hours instead, the same maxHistory=60 would now mean 60 hours of logs.  But since this policy also archives on size, the fileNamePattern also includes "%i" which roles the log incrementally each time it reaches the configured maxFileSize= 10MB.  So it will keep 60 days of logs; however, each of those 60 days could have 1 too many rolled logs depending on how much logging is output. 
app.2024-01-18.1.log.gz
app.2024-01-18.2.log.gz
app.2024-01-18.3.log.gz
app.2024-01-18.4.log.gz
app.2024-01-18.5.log.gz
...

So if your application decides to unexpectedly start producing large amounts of log output, you could quickly consume all your disk space with incremental logs.  This is where the TotalSizeCap setting comes into the picture. It's job is to say regardless of number of logs you want to keep (60 days in this specific configuration example), start deleting old archives when total log archives exceeds this configured size cap.

maxHistory is relative to the configured date pattern and is always days.

So comment in above exact example is correct.  it is not about number of log files and since pattern is based in days, it is correct.

A better generic comment is:

<!-- Control the maximum number of log archive files kept and asynchronously delete older files -->

since it makes no specific relative to the dat pattern being used. It simply states that the maxHistory value correlates to number of log archives that are retained.  What that equates (minutes, hours, days, etc) to depends on the user defined data pattern.

I modified the example in the article you liked that I created with above comment instead of days even though I called it out later in my article for clarity.

In my opinion, TotalSizeCap is something you should use anytime you use the SizeAndTimeBasedRollingPolicy.  That is because maxHistory setting does not take into account the incrementally (%i) created log files.  So using above example where it is creating daily logs (%d{yyyy-MM-dd}) the maxHistory= 60 says to keep 60 days of logs.  

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt