NiFi logs not rolling over on Windows

[#1]

I manage NiFi installations for various clients within our company. On any servers or client systems running a Windows-based operating system, and on any version of NiFi I currently maintain (1.11.4, 1.16.3, 1.19.1), NiFi does not roll over logs using the default logback.xml.

Example:

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'app_%d.log'.
              For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
    </appender>

This works as expected on Linux-based operating systems, where the log is rolled over after an hour or when it exceeds the max file size.

On Windows systems, the only file that is created is nifi-app.log. It is never rolled over and keeps growing in size infinitely. This leads to performance issues and requires manual intervention and interrupting uptime once in a while, as we otherwise risk running out of disk space sooner or later.

I found this topic: https://community.cloudera.com/t5/Support-Questions/Nifi-1-2-0-on-Windows-2012-Server-is-not-rolling...

However, the solution provided within disables creation of e.g. nifi-app.log (without any date/time suffixes). This file being created and rolled over is a requirement by some of my clients as they have data flows that depend on it already running in place. The expectation is that NiFi works the same across Linux and Windows installations, so adapting these data flows is not an option.

Is there a way to have log rollover work on Windows the way it does on Linux by default? Ideally, nifi-app.log exists and is written to by default, and after a period of time or if the log grows bigger than a defined file size, its contents are rolled over.

[#2]

Any update on this? I can't imagine that this is the expected and accepted default behavior. There surely has to be a solution for something as fundamental as this.

[#3]

Hi @edaley,

I have a couple instances runnning on Windows and all of them rotates uses the TimeBasedRollingPolicy with the following configuration

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!-- file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.zip</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>

This configuration works over the nifi-app.log file and move it into a new file nifi-app_date.log.zip at midnight.

Hope this helps.

[#4]

Thank you very much for the response and suggestion. I tried to implement this in our test environment, which resulted in a nifi-app_2023-01-11.log file being created, but nifi-app.log is never created.

I tried changing the timestamp format of fileNamePattern to %d{yyyy-MM-dd_HH_mm} to trigger rollovers every minute. Again, log files including the timestamps are created, but nifi-app.log is not.

I tried uncommenting the <file> line. Now, nifi-app.log is created, but rollover is no longer working - nifi-app.log is the only file that is created.

It feels like I can either get nifi-app.log or rollover working at any one time, but never both at the same time. Any way to get both working at the same time?