We have lots of loggers. Below is a few part from out logback.xml <logger name="org.apache.nifi" level="INFO"/> <logger name="org.apache.nifi.processors" level="WARN"/> <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/> <logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/> <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" /> We already have separated pvc mounted for storing logs for our Nifi pods. Right now I can see total size of 8GB of logs generated a day. I'm compressing it so having a control on it.   ... View more

So to answer your question, After, compressing we are having 2GB of logs files generated per day. Most part is by Nifi_app log files. Below is the configuration used as of now.  <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern> <maxFileSize>500MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> </rollingPolicy> And we are getting 80-100 files per day. ... View more

@MattWho  I also noticed these comments and raised a concern in Apache Nifi Slack Channel. Let see whether they will have a fix on it.  Thanks again for your inputs😊. ... View more

@MattWho In my case "totalSizeCap" is also not working as expected. On checking further I understood that it's fixed in 1.3.x version of logback. This is a similar issue: logging - Logback: SizeAndTimeBasedRollingPolicy applies totalSizeCap to each day in maxHistory - Stack Overflow Since I can't change Nifi version I'm moving forward with initcontaier functionality which will delete logs older than specified time when cluster starts up. Also need to change the comment related to maxhistory. Still its given as log file rather than hour or day. Are you aware of any of the Nifi maintaniers ? ... View more

@MattWho Appreciate your detailed information in clearing the doubts.  Log rotation is working fine. But problem happens when we are switching off&on the NIFI.  I good with Nifi being ignoring the logs when it's switched off. So I have used a new property called: <cleanHistoryOnStart>true</cleanHistoryOnStart> I have taken this from the document: Chapter 4: Appenders (qos.ch) But the problem is all the old logs are still present. I believe some additional configurations are to be added in order to remove previous days logs(last 2 hours logs only). Correct me if I'm wrong. This is my current configuration. <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log.gz</fileNamePattern> <!-- keep 30 log files worth of history --> <maxHistory>2</maxHistory> <totalSizeCap>1GB</totalSizeCap> <cleanHistoryOnStart>true</cleanHistoryOnStart> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> I switched off the EKS cluster and on next day when I switched on, the log of last two hours of previous day is still present.  These are the jars available: log4j-over-slf4j-1.7.36.jar logback-classic-1.2.11.jar logback-core-1.2.11.jar ... View more

@mat I'm good the functionality of "TotalSizeCap", but still in a confusion with " Max History".  In the first reply you mentioned like this:   <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log</fileNamePattern>   "Here you will notice the date pattern is based on hourly logs. So the "maxHistory" property in this example would dictate the number of hours worth of logs to retain (not days)." Last reply you have mentioned like this: Max History does not count incremental (%i) logs generated.  It is based on the date pattern used %d{yyyy-MM-dd_HH}.  So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created." Please have a look at the above bolded characters.  Nifi version - 1.17.0 This is a sample of Nifi app log file which is being generated now. nifi-app_2024-01-09_11.0.log  nifi-app_2024-01-09_11.1.log  nifi-app_2024-01-09_11.2.log nifi-app_2024-01-09_11.3.log I understood that its writted in an hourly basis. And this is the 11th hour log file.  Suppose I give "Max History" as 2 and there is no " TotalSizeCap" What should ideally happen? (I'm asking this because I need to make sure we are on the same page. According to me it will keep only latest 2hr of log only🙄) ... View more

@MattWho Really appreciate your detailed answer I need few clarifications.  You mentioned "TotalSizeCap" will start deleting the oldest rolled logs until max usage is below 10GB" is it calculated per day? We are haven't used the property "TotalSizeCap" property. This is What we were using: <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>50MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>10</maxHistory> So, another doubt is regarding "maxHistory" property. This means latest 10hrs of log is kept, is it? if so, will the older 12th & 11th hour log be deleted? and is it calculated per day? Currently I'm observing the log rotation after editing few values. Added "totalsizecap" too. Your inputs and suggestions will be valuable.  ... View more