Support Questions
Find answers, ask questions, and share your expertise

Apache Nifi PutHiveQL Extract error message

Solved Go to solution

Apache Nifi PutHiveQL Extract error message

New Contributor

Apache Nifi PutHiveQL How do I extract error message into flowfile attribute when failure occurs?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Apache Nifi PutHiveQL Extract error message

Master Guru

@kam1 

Processors are not coded to expose the exceptions to the FlowFile.

Typically how this handled is by Using the SiteToSIteBulletinReportingTask.   Anytime a component throws an exception the processor will display a bulletin in the upper right corner and that same bulletin will be sent to the bulletin board.  The same exception along with stack trace (if there is one) is also logged to the nifi-app.log.

The S2SBulletinReportingTasks allows you to capture all produced bulletins and send them via S2S to another NiFi or even back to same NiFi to be processed as FlowFiles for notification and alerting reasons through a dataflow.

Writing exceptions out to FlowFile attributes would be problematic for following reasons:
1. FlowFile attributes exist in NiFi heap memory and exceptions can be large.
2. A FlowFile may pass multiple processor which produce bulletin/exception. Is expectation we overwrite previous exception with new exception?  Or create a new FlowFile attribute based on processor uuid for each exception encounter during life of FlowFile? This leads back to point 1 issue.

It is better to handle exceptions as content within their own FlowFiles via the reporting task mentioned above.

 

Hope this helps,

Matt

View solution in original post

1 REPLY 1

Re: Apache Nifi PutHiveQL Extract error message

Master Guru

@kam1 

Processors are not coded to expose the exceptions to the FlowFile.

Typically how this handled is by Using the SiteToSIteBulletinReportingTask.   Anytime a component throws an exception the processor will display a bulletin in the upper right corner and that same bulletin will be sent to the bulletin board.  The same exception along with stack trace (if there is one) is also logged to the nifi-app.log.

The S2SBulletinReportingTasks allows you to capture all produced bulletins and send them via S2S to another NiFi or even back to same NiFi to be processed as FlowFiles for notification and alerting reasons through a dataflow.

Writing exceptions out to FlowFile attributes would be problematic for following reasons:
1. FlowFile attributes exist in NiFi heap memory and exceptions can be large.
2. A FlowFile may pass multiple processor which produce bulletin/exception. Is expectation we overwrite previous exception with new exception?  Or create a new FlowFile attribute based on processor uuid for each exception encounter during life of FlowFile? This leads back to point 1 issue.

It is better to handle exceptions as content within their own FlowFiles via the reporting task mentioned above.

 

Hope this helps,

Matt

View solution in original post