Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Apache Nifi PutHiveQL Extract error message

avatar
New Contributor

Apache Nifi PutHiveQL How do I extract error message into flowfile attribute when failure occurs?

1 ACCEPTED SOLUTION

avatar
Super Mentor

@kam1 

Processors are not coded to expose the exceptions to the FlowFile.

Typically how this handled is by Using the SiteToSIteBulletinReportingTask.   Anytime a component throws an exception the processor will display a bulletin in the upper right corner and that same bulletin will be sent to the bulletin board.  The same exception along with stack trace (if there is one) is also logged to the nifi-app.log.

The S2SBulletinReportingTasks allows you to capture all produced bulletins and send them via S2S to another NiFi or even back to same NiFi to be processed as FlowFiles for notification and alerting reasons through a dataflow.

Writing exceptions out to FlowFile attributes would be problematic for following reasons:
1. FlowFile attributes exist in NiFi heap memory and exceptions can be large.
2. A FlowFile may pass multiple processor which produce bulletin/exception. Is expectation we overwrite previous exception with new exception?  Or create a new FlowFile attribute based on processor uuid for each exception encounter during life of FlowFile? This leads back to point 1 issue.

It is better to handle exceptions as content within their own FlowFiles via the reporting task mentioned above.

 

Hope this helps,

Matt

View solution in original post

1 REPLY 1

avatar
Super Mentor

@kam1 

Processors are not coded to expose the exceptions to the FlowFile.

Typically how this handled is by Using the SiteToSIteBulletinReportingTask.   Anytime a component throws an exception the processor will display a bulletin in the upper right corner and that same bulletin will be sent to the bulletin board.  The same exception along with stack trace (if there is one) is also logged to the nifi-app.log.

The S2SBulletinReportingTasks allows you to capture all produced bulletins and send them via S2S to another NiFi or even back to same NiFi to be processed as FlowFiles for notification and alerting reasons through a dataflow.

Writing exceptions out to FlowFile attributes would be problematic for following reasons:
1. FlowFile attributes exist in NiFi heap memory and exceptions can be large.
2. A FlowFile may pass multiple processor which produce bulletin/exception. Is expectation we overwrite previous exception with new exception?  Or create a new FlowFile attribute based on processor uuid for each exception encounter during life of FlowFile? This leads back to point 1 issue.

It is better to handle exceptions as content within their own FlowFiles via the reporting task mentioned above.

 

Hope this helps,

Matt