How to log message at the Processor Group Level

[#1]

Dear Experts,

I would like to implement logging at the Processor Group level. Suppose we have two processor groups, A and B, and I want to maintain separate log files for each. When Processor Group A is started, its activities should be logged in a dedicated log file for Group A. Similarly, Processor Group B should have its own separate log file. How can I achieve this?

Thank you in advance for your assistance.

[#2 · Accepted solution]

@HoangNguyen 

As long as you are running a new enough version of Apache NiFi, you'll have an option with the process group configuration to set a logging suffix.

For each process group you want a separate log file, create a unique suffix.

In above example I used the suffix "extracted".
In my NiFi "logs" directory, I now have a new "nifi-app-extracted.log" file that contains the logging output of every component contained within that process group.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3 · Accepted solution]

@HoangNguyen 

As long as you are running a new enough version of Apache NiFi, you'll have an option with the process group configuration to set a logging suffix.

For each process group you want a separate log file, create a unique suffix.

In above example I used the suffix "extracted".
In my NiFi "logs" directory, I now have a new "nifi-app-extracted.log" file that contains the logging output of every component contained within that process group.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Hello @MattWho 
Thank you very much for your help.

I have a follow-up question. I would the log file to capture more than just errors. After researching, I found that we can configure the log level (e.g., TRACE, DEBUG, INFO) in the conf/logback.xml file. Is this correct? However, I'm unsure about how to configure it properly.

Here’s the relevant section from my logback.xml:

xml

<!-- Deprecation Log -->
<logger name="deprecation" level="WARN" additivity="false">
    <appender-ref ref="DEPRECATION_FILE"/>
</logger>

<logger name="org.apache.nifi" level="INFO"/>
<logger name="org.apache.nifi.processors" level="WARN"/>
<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
<logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
<logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN"/>
<logger name="org.apache.parquet.hadoop.InternalParquetRecordReader" level="WARN"/>

Would changing <logger name="deprecation" level="WARN" additivity="false"> to <logger name="deprecation" level="TRACE" additivity="false"> or <logger name="deprecation" level="DEBUG" additivity="false"> be correct?

Thank you for your guidance.

[#5 · Accepted solution]

@HoangNguyen 

The "Deprecation log" has nothing to do with your running dataflows on your NiFi canvas.  The deprecation log contains notifications about NiFi components you may be using that are deprecated.  Deprecated components get removed in future Apache NiFi versions.  This log is to make you aware of this so you can make dataflow design changes to stop using them before you migrate to newer Apache NiFi release. 

The NiFi Standard Log Files  include the "bootstrap log, app log, and user log". 
The app log is where you will find alll your dataflow component based logging.  In the logback.xml, "logger" will write to the nifi-app.log by default unless a specific "appender-ref is declared for the logger.  

NiFi app.log can produce a lot of logging, but to get it all you can adjust:

<logger name="org.apache.nifi" level="INFO"/>

to "DEBUG" instead of INFO.  It will be very noisy.

Logback standard log levels:

• OFF: This level turns off all logging. No messages will be outputted.
• ERROR: Indicates a serious error that might still allow the application to continue running, but requires attention.
• WARN: Indicates a potentially harmful situation that should be investigated, but does not necessarily prevent the application from continuing.
• INFO: Provides general information about the application's progress and significant events.
• DEBUG: Offers detailed information useful for debugging purposes, often including variable values and execution flow.

• TRACE: Provides even finer-grained information than DEBUG, typically used for extremely detailed tracing of execution paths.

• ALL: This level enables all logging, including messages at all other levels.

Keep in mind that just because you set DEBUG log level, does not mean every component will produce DEBUG level log messages. It all depends on what logging exists within the component class and dependent libraries.  When set to DEBUG, it will log DEBUG and all level below it (INFO, WARN, ERROR).  If you set "INFO", you also get WARN and ERROR logging.

NiFi user authorization logging will go to the nifi-user.log.  This is logging related to access to NiFi.  Nifi-bootstrap.log has logging for you rNiFi bootstrap process.  The bootstrap is what is lauched when you execute the nifi.sh start command.  The bootstrap then starts the nifi main child process whcih loads your NiFi and dataflows.  The bootstrap then monitors that child process to make sure it is still live (restarts it automatically if it dies).

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#6]

Hello @MattWho ,

Thank you so much. 

Very clear details and usefuls