Member since
03-01-2023
13
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1090 | 03-06-2023 05:19 AM |
06-12-2024
05:43 AM
We have lots of loggers. Below is a few part from out logback.xml <logger name="org.apache.nifi" level="INFO"/>
<logger name="org.apache.nifi.processors" level="WARN"/>
<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
<logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
<logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" /> We already have separated pvc mounted for storing logs for our Nifi pods. Right now I can see total size of 8GB of logs generated a day. I'm compressing it so having a control on it.
... View more
06-04-2024
05:24 AM
So to answer your question, After, compressing we are having 2GB of logs files generated per day. Most part is by Nifi_app log files. Below is the configuration used as of now. <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<maxFileSize>500MB</maxFileSize>
<!-- keep 30 log files worth of history -->
<maxHistory>30</maxHistory>
</rollingPolicy> And we are getting 80-100 files per day.
... View more
05-27-2024
03:23 AM
1 Kudo
Hi, Currently we have nifi logrotation in such a way that rotation is happening per day for 100mb of size. So, we have approximately 700 to 800 log files per day. So, my question is if I change the size from 100mb to 500mb /1GB will it affect the performance of Nifi. Or should I rotate logs per hour with 50mb of files size for better performance. Current log rotation is as follows: <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern>
<maxFileSize>100MB</maxFileSize>
<!-- keep 30 log files worth of history -->
<maxHistory>30</maxHistory>
... View more
Labels:
- Labels:
-
Apache NiFi
01-18-2024
08:26 AM
@MattWho I also noticed these comments and raised a concern in Apache Nifi Slack Channel. Let see whether they will have a fix on it. Thanks again for your inputs😊.
... View more
01-18-2024
12:51 AM
@MattWho In my case "totalSizeCap" is also not working as expected. On checking further I understood that it's fixed in 1.3.x version of logback. This is a similar issue: logging - Logback: SizeAndTimeBasedRollingPolicy applies totalSizeCap to each day in maxHistory - Stack Overflow Since I can't change Nifi version I'm moving forward with initcontaier functionality which will delete logs older than specified time when cluster starts up. Also need to change the comment related to maxhistory. Still its given as log file rather than hour or day. Are you aware of any of the Nifi maintaniers ?
... View more
01-15-2024
05:35 AM
@MattWho Appreciate your detailed information in clearing the doubts. Log rotation is working fine. But problem happens when we are switching off&on the NIFI. I good with Nifi being ignoring the logs when it's switched off. So I have used a new property called: <cleanHistoryOnStart>true</cleanHistoryOnStart> I have taken this from the document: Chapter 4: Appenders (qos.ch) But the problem is all the old logs are still present. I believe some additional configurations are to be added in order to remove previous days logs(last 2 hours logs only). Correct me if I'm wrong. This is my current configuration. <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'app_%d.log'.
For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log.gz</fileNamePattern>
<!-- keep 30 log files worth of history -->
<maxHistory>2</maxHistory>
<totalSizeCap>1GB</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<immediateFlush>true</immediateFlush>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender> I switched off the EKS cluster and on next day when I switched on, the log of last two hours of previous day is still present. These are the jars available: log4j-over-slf4j-1.7.36.jar logback-classic-1.2.11.jar logback-core-1.2.11.jar
... View more
01-09-2024
07:22 PM
@mat I'm good the functionality of "TotalSizeCap", but still in a confusion with " Max History". In the first reply you mentioned like this: <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.log</fileNamePattern> "Here you will notice the date pattern is based on hourly logs. So the "maxHistory" property in this example would dictate the number of hours worth of logs to retain (not days)." Last reply you have mentioned like this: Max History does not count incremental (%i) logs generated. It is based on the date pattern used %d{yyyy-MM-dd_HH}. So you role logs every hour, so MaxHistory would retain 10 days unless TotalSizeCap is reached before 10 days of logs are created." Please have a look at the above bolded characters. Nifi version - 1.17.0 This is a sample of Nifi app log file which is being generated now. nifi-app_2024-01-09_11.0.log nifi-app_2024-01-09_11.1.log nifi-app_2024-01-09_11.2.log nifi-app_2024-01-09_11.3.log I understood that its writted in an hourly basis. And this is the 11th hour log file. Suppose I give "Max History" as 2 and there is no " TotalSizeCap" What should ideally happen? (I'm asking this because I need to make sure we are on the same page. According to me it will keep only latest 2hr of log only🙄)
... View more
01-09-2024
03:29 AM
@MattWho Really appreciate your detailed answer I need few clarifications. You mentioned "TotalSizeCap" will start deleting the oldest rolled logs until max usage is below 10GB" is it calculated per day? We are haven't used the property "TotalSizeCap" property. This is What we were using: <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<maxFileSize>50MB</maxFileSize>
<!-- keep 30 log files worth of history -->
<maxHistory>10</maxHistory> So, another doubt is regarding "maxHistory" property. This means latest 10hrs of log is kept, is it? if so, will the older 12th & 11th hour log be deleted? and is it calculated per day? Currently I'm observing the log rotation after editing few values. Added "totalsizecap" too. Your inputs and suggestions will be valuable.
... View more
01-02-2024
03:26 AM
I need to configure nifi for log rotation. I have seen this article: Understanding how the logback.xml configuration in... - Cloudera Community - 248662 Here I have few doubts regarding the property. <maxHistory>30</maxHistory> It's mentioned in comments as <!-- keep 30 log files worth of history --> But on checking I have come across a document logback.xml Example - Mkyong.com where its mentioned as <!-- each archived file, size max 10MB -->
<maxFileSize>10MB</maxFileSize>
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
<totalSizeCap>20GB</totalSizeCap>
<!-- 60 days to keep -->
<maxHistory>60</maxHistory> So, my doubt is does maxhistory refers to number of file or number of days. Also do we really need to add totalSizeCap because anyway we are adding maxhistory and maxfilesize.
... View more
Labels:
- Labels:
-
Apache NiFi
09-13-2023
03:00 AM
I'm trying to set deploy Nifi in Kubernetes cluster. After deployment the status are shown as Init:0/2 and Init:1/2 and Init:Error. This all takes just 12s. Does anyone tried nifi in Kuberenetes version 1.27, is there any version compactability issue ? Nifi - 1.17.0 Nifi-Operator - 1.0.0 Cert-Manager - 1.11.0 Zookeeper - 3.8.0 This configuration works in older versions of Kubernetes Cluster. Hoping to get some help on this. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Zookeeper