question Re: How to log message at the Processor Group Level in Support Questions

How to log message at the Processor Group Level

HoangNguyen — Mon, 22 Sep 2025 02:42:51 GMT

Dear Experts,

I would like to implement logging at the Processor Group level. Suppose we have two processor groups, A and B, and I want to maintain separate log files for each. When Processor Group A is started, its activities should be logged in a dedicated log file for Group A. Similarly, Processor Group B should have its own separate log file. How can I achieve this?

Thank you in advance for your assistance.

Re: How to log message at the Processor Group Level

MattWho — Mon, 22 Sep 2025 12:32:07 GMT

@HoangNguyen

As long as you are running a new enough version of Apache NiFi, you'll have an option with the process group configuration to set a logging suffix.

For each process group you want a separate log file, create a unique suffix.

In above example I used the suffix "extracted".
In my NiFi "logs" directory, I now have a new "nifi-app-extracted.log" file that contains the logging output of every component contained within that process group.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: How to log message at the Processor Group Level

HoangNguyen — Tue, 23 Sep 2025 03:12:15 GMT

Hello @MattWho
Thank you very much for your help.

I have a follow-up question. I would the log file to capture more than just errors. After researching, I found that we can configure the log level (e.g., TRACE, DEBUG, INFO) in the conf/logback.xml file. Is this correct? However, I'm unsure about how to configure it properly.

Here’s the relevant section from my logback.xml:

xml

<!-- Deprecation Log -->
<logger name="deprecation" level="WARN" additivity="false">
    <appender-ref ref="DEPRECATION_FILE"/>
</logger>

<logger name="org.apache.nifi" level="INFO"/>
<logger name="org.apache.nifi.processors" level="WARN"/>
<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
<logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
<logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN"/>
<logger name="org.apache.parquet.hadoop.InternalParquetRecordReader" level="WARN"/>

Would changing <logger name="deprecation" level="WARN" additivity="false"> to <logger name="deprecation" level="TRACE" additivity="false"> or <logger name="deprecation" level="DEBUG" additivity="false"> be correct?

Thank you for your guidance.

Re: How to log message at the Processor Group Level

MattWho — Tue, 23 Sep 2025 12:26:32 GMT

@HoangNguyen

The "Deprecation log" has nothing to do with your running dataflows on your NiFi canvas. The deprecation log contains notifications about NiFi components you may be using that are deprecated. Deprecated components get removed in future Apache NiFi versions. This log is to make you aware of this so you can make dataflow design changes to stop using them before you migrate to newer Apache NiFi release.

The NiFi Standard Log Files include the "bootstrap log, app log, and user log".
The app log is where you will find alll your dataflow component based logging. In the logback.xml, "logger" will write to the nifi-app.log by default unless a specific "appender-ref is declared for the logger.

NiFi app.log can produce a lot of logging, but to get it all you can adjust:

<logger name="org.apache.nifi" level="INFO"/>

to "DEBUG" instead of INFO. It will be very noisy.

Logback standard log levels:

OFF: This level turns off all logging. No messages will be outputted.
ERROR: Indicates a serious error that might still allow the application to continue running, but requires attention.
WARN: Indicates a potentially harmful situation that should be investigated, but does not necessarily prevent the application from continuing.
INFO: Provides general information about the application's progress and significant events.
DEBUG: Offers detailed information useful for debugging purposes, often including variable values and execution flow.

TRACE: Provides even finer-grained information than DEBUG, typically used for extremely detailed tracing of execution paths.

ALL: This level enables all logging, including messages at all other levels.

Keep in mind that just because you set DEBUG log level, does not mean every component will produce DEBUG level log messages. It all depends on what logging exists within the component class and dependent libraries. When set to DEBUG, it will log DEBUG and all level below it (INFO, WARN, ERROR). If you set "INFO", you also get WARN and ERROR logging.

NiFi user authorization logging will go to the nifi-user.log. This is logging related to access to NiFi. Nifi-bootstrap.log has logging for you rNiFi bootstrap process. The bootstrap is what is lauched when you execute the nifi.sh start command. The bootstrap then starts the nifi main child process whcih loads your NiFi and dataflows. The bootstrap then monitors that child process to make sure it is still live (restarts it automatically if it dies).

Thank you,
Matt

Re: How to log message at the Processor Group Level

HoangNguyen — Wed, 24 Sep 2025 01:43:46 GMT

Hello @MattWho ,

Thank you so much.

Very clear details and usefuls