Have you ever noticed some lingering old rolled log files in your nifi logs directory that never seem to get deleted? This is a by-product of how logback works depending on how you have it configured. - Lets take a look at a default logback.xml configuration from NiFi: <appender name="APP_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- Control the maximum number of log archive files kept and asynchronously delete older files --> <maxHistory>30</maxHistory> <!-- optional setting for keeping 10GB total of log files <totalSizeCap>10GB</totalSizeCap> --> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> The above app log configuration will log to a file named nifi-app.log. Once that file reaches either 100 MB in size or crest the top of the hour, it will be rolled. You may end up with numerous log files within a single hour if there is an excessive amount of logging occurring in your NiFi. - A "maxHistory" of 30 means that the logger will only keep 30 hours (HH) of rolled logs. But that is not the full story here with how logback works. Not only does it control the number of hours to keep but also controls the max age of logs to evaluate for deletion. So the log files being left around that are more then 30 hours in age would be ignored when deletion thread ran. - So this naturally raises the question of how did these files get left behind in the first place? Typically this occurs if the file crest say 30 hours old while the application is stopped. When the application is restarted those older files end up getting ignored. - While the application is continuously running this works as one would normally expect. To simply clean-up these older rolled log files, you could run a touch command on them so their system file timestamp updates so they are no longer more then 30 hours old. They will then be considered within the 30 hour window and be deleted once the "maxHistory" count reaches 30. - However, above is not a permanent solution. I recommend instead to control file deletion by "totalSizeCap" setting (commented out by default in the NiFi logback.xml) It offers a couple of advantages: 1. The "%i" option in the fileNamePattern says to create sequential numbered log files every "maxFileSize" (100MB) within each hour. This help prevent any one log from getting to large, but has the downside of not being considered by "maxHistory" as individually counted files. So "maxHistory" set to 15 is 15 hours of logs even if each hour contains 2000 100MB log files. So you can see under heavy logging you can end up using a lot of logs space. 2. "TotalSizeCap" will start deleting old rolled log files as long as the log file date is less then "maxHistory" age. So lets say we want to retain up to 100GB of log history. We would set "maxHistory" to some very large value like 8760 (~1 year of hours) and set "totalSizeCap" to 100GB. Provided you hot 100GB before your hit 8760 hours. - Here is an example configuration: <appender name="APP_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>10MB</maxFileSize> <!-- keepup to 8,760 hours worth of log files --> <maxHistory>8760</maxHistory> <!-- optional setting for keeping 100 GB total of log files --> <totalSizeCap>100GB</totalSizeCap> <!-- archive removal will be executed on appender start up --> <cleanHistoryOnStart>true</cleanHistoryOnStart> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> - Of course there is always a chance you could hit 8,760 hours worth of logs before reaching 100 GB of generated app logs, so you may need to tailor these setting based on app log sizes being generated by your particular running NiFi. ... View more