This is an issue I've been observing for quite a while now. I unfortunately don't have any concrete reproduction steps, but from what I can tell, this only seems to happen after rolling restarts of the cluster.   Just a few days ago, I performed a rolling restart of one of our NiFi clusters (running NiFi 1.16.3 on Ubuntu 18.04 LTS). Afterwards, a couple of our flows stopped processing data, despite all involved components appearing as started in the UI. The effects of this seem to be under the scope of process groups, as in, if one component within a process group is affected, all components within that group and subgroups are affected.   Stopping and then starting the affected components restores them to a functional state. This is of course not a long-term solution, so I'm curious to know if anyone else has encountered this problem before or could share any suggestions regarding troubleshooting or steps towards a solution. ... View more

I manage NiFi installations for various clients within our company. On any servers or client systems running a Windows-based operating system, and on any version of NiFi I currently maintain (1.11.4, 1.16.3, 1.19.1), NiFi does not roll over logs using the default logback.xml.   Example:   <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender>     This works as expected on Linux-based operating systems, where the log is rolled over after an hour or when it exceeds the max file size. On Windows systems, the only file that is created is nifi-app.log. It is never rolled over and keeps growing in size infinitely. This leads to performance issues and requires manual intervention and interrupting uptime once in a while, as we otherwise risk running out of disk space sooner or later.   I found this topic: https://community.cloudera.com/t5/Support-Questions/Nifi-1-2-0-on-Windows-2012-Server-is-not-rolling-over-its/td-p/224826 However, the solution provided within disables creation of e.g. nifi-app.log (without any date/time suffixes). This file being created and rolled over is a requirement by some of my clients as they have data flows that depend on it already running in place. The expectation is that NiFi works the same across Linux and Windows installations, so adapting these data flows is not an option.   Is there a way to have log rollover work on Windows the way it does on Linux by default? Ideally, nifi-app.log exists and is written to by default, and after a period of time or if the log grows bigger than a defined file size, its contents are rolled over. ... View more