Support Questions

edaley · ‎12-15-2022

I manage NiFi installations for various clients within our company. On any servers or client systems running a Windows-based operating system, and on any version of NiFi I currently maintain (1.11.4, 1.16.3, 1.19.1), NiFi does not roll over logs using the default logback.xml.

Example:

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'app_%d.log'.
              For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
    </appender>

This works as expected on Linux-based operating systems, where the log is rolled over after an hour or when it exceeds the max file size.

On Windows systems, the only file that is created is nifi-app.log. It is never rolled over and keeps growing in size infinitely. This leads to performance issues and requires manual intervention and interrupting uptime once in a while, as we otherwise risk running out of disk space sooner or later.

I found this topic: https://community.cloudera.com/t5/Support-Questions/Nifi-1-2-0-on-Windows-2012-Server-is-not-rolling...

However, the solution provided within disables creation of e.g. nifi-app.log (without any date/time suffixes). This file being created and rolled over is a requirement by some of my clients as they have data flows that depend on it already running in place. The expectation is that NiFi works the same across Linux and Windows installations, so adapting these data flows is not an option.

Is there a way to have log rollover work on Windows the way it does on Linux by default? Ideally, nifi-app.log exists and is written to by default, and after a period of time or if the log grows bigger than a defined file size, its contents are rolled over.

garb · ‎10-07-2025

Wanted to reply because I was also having this issue on Windows Server.

@templarian has the working solution.
Whenever you first start NiFi leave the logback configurations unchanged, however we set ours to debug to have more insight into the issue.

<configuration scan="true" scanPeriod="30 seconds" debug="true">
...
</configuration>

NiFi's PID is blocking the file constantly, never releasing it, this causes the RollingFileAppender to never be able to change the filename:

16:32:07,611 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - No removal attempts were made on account of totalSizeCap=3221225472
16:32:07,613 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - No removal attempts were made on account of totalSizeCap=3221225472
16:32:22,499 |-INFO in c.q.l.co.rolling.helper.RenameUtil - Renaming file [C:\nifi-2.5.0\logs\nifi-app.log] to [C:\nifi-2.5.0\logs\nifi-app_2025-10-07_16.0.log5423244956573500.tmp]
16:32:22,499 |-WARN in c.q.l.co.rolling.helper.RenameUtil - Failed to rename file [C:\nifi-2.5.0\logs\nifi-app.log] as [C:\nifi-2.5.0\logs\nifi-app_2025-10-07_16.0.log5423244956573500.tmp].
16:32:22,501 |-WARN in c.q.l.co.rolling.helper.RenameUtil - Please consider leaving the [file] option of RollingFileAppender empty.
16:32:22,502 |-WARN in c.q.l.co.rolling.helper.RenameUtil - See also https://logback.qos.ch/codes.html#renamingError

Our workaround was to leave it unchanged on the first launch of NiFi and then remove all the <file/> properties for each appender once each file was initially created. Logback automatically applies these settings so there should be no disruption to NiFi.

The appender configuration is identical to @templarian's suggestion. This got rotation and compression working for each of our appenders.

View solution in original post

MattWho · ‎10-08-2025

@garb

There is an existing Apache NiFi jira reporting this issue here:
https://issues.apache.org/jira/browse/NIFI-14729

It aligns with your observations above.

Thanks,
Matt

View solution in original post

edaley · ‎01-10-2023

Any update on this? I can't imagine that this is the expected and accepted default behavior. There surely has to be a solution for something as fundamental as this.

templarian · ‎01-10-2023

Hi @edaley,

I have a couple instances runnning on Windows and all of them rotates uses the TimeBasedRollingPolicy with the following configuration

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!-- file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d.log.zip</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>

This configuration works over the nifi-app.log file and move it into a new file nifi-app_date.log.zip at midnight.

Hope this helps.

edaley · ‎01-11-2023

Thank you very much for the response and suggestion. I tried to implement this in our test environment, which resulted in a nifi-app_2023-01-11.log file being created, but nifi-app.log is never created.

I tried changing the timestamp format of fileNamePattern to %d{yyyy-MM-dd_HH_mm} to trigger rollovers every minute. Again, log files including the timestamps are created, but nifi-app.log is not.

I tried uncommenting the <file> line. Now, nifi-app.log is created, but rollover is no longer working - nifi-app.log is the only file that is created.

It feels like I can either get nifi-app.log or rollover working at any one time, but never both at the same time. Any way to get both working at the same time?

garb · ‎10-07-2025