Member since
03-01-2023
13
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1170 | 03-06-2023 05:19 AM |
06-14-2024
07:46 AM
1 Kudo
@Alexy Without seeing your logs, I have no idea which NiFi classes are producing the majority of your logging. But logback is functioning exactly as you have it configured. Each time the nifi-app.log reaches 500 MB within a single day it is compressed and rolled using an incrementing number. I would suggest changing the log level for the base class "org.apache.nifi" from INFO to WARN. The bulk of all NiFi classes begin with org.apache.nifi and by changing this to WARN to you will only see ERROR and WARN level log output from the bulk of the ora.apache.nifi.<XYZ...> classes. <logger name="org.apache.nifi" level="WARN"/> Unless you have a lot of exception happening within your NiFi processor components used in your dataflow(s), this should have significant impact on the amount of nifi-app.log logging being produced. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-18-2024
09:55 AM
@Alexy MaxHistory is relative to timedBased Policy date pattern: in the example shown in this thread you mentioned: https://mkyong.com/logging/logback-xml-example/ The full rollingPolicy example is: <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>logs/archived/app.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<!-- each archived file, size max 10MB -->
<maxFileSize>10MB</maxFileSize>
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
<totalSizeCap>20GB</totalSizeCap>
<!-- 60 days to keep -->
<maxHistory>60</maxHistory>
</rollingPolicy> Within the fileNamePattern we see the configured date pattern "%d{yyyy-MM-dd}" which is based on days. So, maxHistory of 60 means in this specific example to retain "60" days. Now if that date pattern instead was "%d{yyyy-MM-dd_HH}" which is based on hours instead, the same maxHistory=60 would now mean 60 hours of logs. But since this policy also archives on size, the fileNamePattern also includes "%i" which roles the log incrementally each time it reaches the configured maxFileSize= 10MB. So it will keep 60 days of logs; however, each of those 60 days could have 1 too many rolled logs depending on how much logging is output. app.2024-01-18.1.log.gz app.2024-01-18.2.log.gz app.2024-01-18.3.log.gz app.2024-01-18.4.log.gz app.2024-01-18.5.log.gz ... So if your application decides to unexpectedly start producing large amounts of log output, you could quickly consume all your disk space with incremental logs. This is where the TotalSizeCap setting comes into the picture. It's job is to say regardless of number of logs you want to keep (60 days in this specific configuration example), start deleting old archives when total log archives exceeds this configured size cap. maxHistory is relative to the configured date pattern and is always days. So comment in above exact example is correct. it is not about number of log files and since pattern is based in days, it is correct. A better generic comment is: <!-- Control the maximum number of log archive files kept and asynchronously delete older files --> since it makes no specific relative to the dat pattern being used. It simply states that the maxHistory value correlates to number of log archives that are retained. What that equates (minutes, hours, days, etc) to depends on the user defined data pattern. I modified the example in the article you liked that I created with above comment instead of days even though I called it out later in my article for clarity. In my opinion, TotalSizeCap is something you should use anytime you use the SizeAndTimeBasedRollingPolicy. That is because maxHistory setting does not take into account the incrementally (%i) created log files. So using above example where it is creating daily logs (%d{yyyy-MM-dd}) the maxHistory= 60 says to keep 60 days of logs. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-13-2023
03:00 AM
I'm trying to set deploy Nifi in Kubernetes cluster. After deployment the status are shown as Init:0/2 and Init:1/2 and Init:Error. This all takes just 12s. Does anyone tried nifi in Kuberenetes version 1.27, is there any version compactability issue ? Nifi - 1.17.0 Nifi-Operator - 1.0.0 Cert-Manager - 1.11.0 Zookeeper - 3.8.0 This configuration works in older versions of Kubernetes Cluster. Hoping to get some help on this. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Zookeeper
03-06-2023
10:16 PM
@Alexy Thank you for sharing the solution to your issue. Your contribution to our community is greatly appreciated, and we believe it will help other members who may face similar issues in the future.
We are glad to have you as a part of our community, and we hope that you will continue to participate and share your knowledge with us.
... View more
03-01-2023
02:00 AM
Initially I was using official Nifi-registry image of Apache. I used the official docker file and created new nifi-registry image. On deploying it in the cluster I'm getting the following error and the pod doesnot spin up. 2023-02-22 14:32:54,459 INFO [main] o.a.n.r.provider.StandardProviderFactory Instantiated FlowPersistenceProvider with class name org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider 2023-02-22 14:32:54,464 INFO [main] o.a.n.r.p.flow.git.GitFlowMetaData ./flow_git is not a directory or does not exist. git configaration has been given in the values file of nifi registry. I tried building the docker file whithout any change provided in the official git repo and I was not getting any error. I use another java image as base image and I'm getting the error Nifi-registry previous: 1.17.0 Nifi-registry present: 1.19.0 No DB I will be forever grateful if anyone can help me
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Zookeeper
-
NiFi Registry