question Re: Nifi Logrotation Policy in Support Questions

Nifi Logrotation Policy

Alexy — Mon, 27 May 2024 10:23:56 GMT

Hi,
Currently we have nifi logrotation in such a way that rotation is happening per day for 100mb of size. So, we have approximately 700 to 800 log files per day.

So, my question is if I change the size from 100mb to 500mb /1GB will it affect the performance of Nifi. Or should I rotate logs per hour with 50mb of files size for better performance.

Current log rotation is as follows:

<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern> <maxFileSize>100MB</maxFileSize>  <maxHistory>30</maxHistory>

Re: Nifi Logrotation Policy

ckumar — Tue, 28 May 2024 09:02:44 GMT

Log rotation settings have No/Driect impact on NiFi data flow processing performance.

It is just that there should be enough space available on the file system to store the log files.

Thank you

Re: Nifi Logrotation Policy

MattWho — Tue, 28 May 2024 13:49:45 GMT

@Alexy

100% agree with @ckumar

Why is your NiFi producing som much logging? Additional loggers? Increased log levels? Huge FlowFile volume?
Why are you not compressing (gz)on rollover to save disk space? Keep in mind that compression will take longer the larger the log file.
The performance is not going to change whether you are writing/appending to 100 MB or larger log files. But you do have disk I/O related to amount of logging you are producing.

Matt

Re: Nifi Logrotation Policy

Alexy — Tue, 04 Jun 2024 12:24:44 GMT

So to answer your question, After, compressing we are having 2GB of logs files generated per day. Most part is by Nifi_app log files.
Below is the configuration used as of now.

<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern> <maxFileSize>500MB</maxFileSize>  <maxHistory>30</maxHistory> </rollingPolicy>

And we are getting 80-100 files per day.

Re: Nifi Logrotation Policy

MattWho — Tue, 04 Jun 2024 16:03:00 GMT

@Alexy
Are you specifically needing to produce so much logging?
What loggers do you have added to your logback.xml?
How many are set to "INFO" level logging?
If you only want to log exceptions, you could change the "INFO" to "WARN" or "ERROR" to greatly reduce mount of INOF logging being produced.

As far as NiFi performance goes, it is all about managing CPU Load average and Disk I/O (Specifically disk I/O of the disks where NiFi's content, flowfile, and provenance repositories are located). You could make sure your logs are being written a separate disk to elevate that Disk I/) form impacting NiFi's repos disks.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi Logrotation Policy

Alexy — Wed, 12 Jun 2024 12:43:50 GMT

We have lots of loggers. Below is a few part from out logback.xml

We already have separated pvc mounted for storing logs for our Nifi pods. Right now I can see total size of 8GB of logs generated a day. I'm compressing it so having a control on it.

Re: Nifi Logrotation Policy

MattWho — Fri, 14 Jun 2024 14:46:10 GMT

@Alexy

Without seeing your logs, I have no idea which NiFi classes are producing the majority of your logging. But logback is functioning exactly as you have it configured. Each time the nifi-app.log reaches 500 MB within a single day it is compressed and rolled using an incrementing number. I would suggest changing the log level for the base class "org.apache.nifi" from INFO to WARN. The bulk of all NiFi classes begin with org.apache.nifi and by changing this to WARN to you will only see ERROR and WARN level log output from the bulk of the ora.apache.nifi.<XYZ...> classes.

Unless you have a lot of exception happening within your NiFi processor components used in your dataflow(s), this should have significant impact on the amount of nifi-app.log logging being produced.

Thank you,
Matt