Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi - Log Size Alert

NiFi - Log Size Alert

New Contributor

alert message :

NiFi Node(s) is/are having the log files size in GBs. Please delete the older log files to free some space.


Please help the resolution

1 REPLY 1
Highlighted

Re: NiFi - Log Size Alert

Master Guru

@BVS REDDY

NiFi does not produce the alert message you are showing above. Where did you see this alert displayed?

All NiFi's logging is controlled by the logback.xml file found in NiFi's conf directory.

There are three default logs that NiFi creates and writes log entries.

nifi-bootstrap.log <-- Should be relatively small by default and log info related to NiFi bootstrap process.

nifi-user.log <-- Only used when NiFi has been secured. This log contains logging about user/client authentication and authorizations requests.

nifi-app.log . <-- These logs can become very large depending on a couple factors. The number of components (processors, controller services, reporting tasks, etc.) added to NiFi canvas. The volume of FlowFiles passing through components. By default the log level is set to "INFO" in older NiFi releases which results in above mention potentially large output. The newest version of added a new logger to the logback to set processor specific logging to "WARN" log level. NiFi I would suggest editing your default NiFi logback.xml as follows:

  • Within the "appender" for the nifi-app.log add a ".gz" to the ned of the "fileNamePattern" line so that rolled logs are compressed to reduce disk usage.
  • Check for existence of this logger "<logger name="org.apache.nifi.processors" level="WARN"/>"
    • If it exists, make sure it is still set to "WARN"
    • If it does not exist, add it.

Tail your nifi-app.log file (tail -F <path to NiFi logs>/logs/nifi-app.log) and see if your NiFi flows are producing a lot of WARN or ERROR level log outputs. If so, address those ERRORs and WARNs to reduce amount of logging that is occurring.


Thank you,
Matt


If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.