Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to do error handling in PutHiveQL Processor in Apache NIFI

Highlighted

How to do error handling in PutHiveQL Processor in Apache NIFI

Explorer

Hello,

I have design the flow in that I executed the query like CTAS,Insert into, truncate table in PutHiveQL Processor.

Depending on query failed or success I have to send email but PutHiveQL not generated any metadata.

I have set rollback property true but that data goes on queue.

How to send email depending upon PutHiveQL Processor success or failure.

4 REPLIES 4
Highlighted

Re: How to do error handling in PutHiveQL Processor in Apache NIFI

Super Guru

@Mitthu Wagh

You can add attributes after PutHiveQL processor by using Update Attribute processor before sending Email.

Flow:-

62456-flow.png

Update Attribute configs:-

Adding Flow_status=Success:-

Add new property

Flow_Status

Success

62457-flow-status-success.png

Adding Flow_status=Failure:-

62458-flow-status-failure.png

PutEmail processor Configs:-

62459-email.png

We can use our Flow_Status attribute value as subject so that we can easily detect is the process got completed successfully or failed.

Subject

${Flow_Status}

Then change property

Include All Attributes In Message

true

So that we can get all the attributes that are associated with the flowfile will be included in the body of message.

As you can add as many attributes as you need in update attribute processor.

Rollback property:-

1.If you rollback property is set to true, then failed FlowFiles will stay in the input relationship without penalizing it and being processed repeatedly until it gets processed successfully or removed by other means. It is important to set adequate 'Yield Duration' to avoid retrying too frequently.

2.By default rollback property is set to false, if an error occurs while processing a FlowFile, the FlowFile will be routed to 'failure' or 'retry' relationship based on error type, and processor can continue with next FlowFile.

Re: How to do error handling in PutHiveQL Processor in Apache NIFI

Super Guru
@Mitthu Wagh

From PutHiveQL processor transfers flowfile to Success relationship only when it is able to execute query without any errors.

For relationships Retry and Failure,

if you set Rollback On Failure

true

Then failed FlowFiles will stay in the input relationship without penalizing it and being processed repeatedly until it gets processed successfully or removed by other means. so if you get flowfiles into 'failure' or 'retry' then those flowfiles will stay in the input relationship.

Instead of keeping above property value as true keep it as false and try a retry loop to execute the failed (or) retry relation flowfiles like 3 times (or) some other number then transfer them to Failure queue to send mail.

Reference to retry loop:-

https://community.hortonworks.com/questions/65755/nifi-flowfile-retries.html

(or)

you can add the time attribute(${now:tonumber()}) to the failed(or) retry relationship flowfiles and do retry, keep routeonattribute to check the attribute value and add your delay time like(5mins..)for another retry of failed(or)retry relation flowfiles.

Then once the time attribute value reaches some time extent(like 15 mins) then transfer those failed(or)retry relation flowfiles to PutEmail processor

Reference to delay processing:-

https://community.hortonworks.com/questions/56167/is-there-wait-processor-in-nifi.html

Highlighted

Re: How to do error handling in PutHiveQL Processor in Apache NIFI

Explorer

thanks shu, it's working and thats helpfull for me.

Highlighted

Re: How to do error handling in PutHiveQL Processor in Apache NIFI

Explorer

thanks shu, I tried your approach

it is working fine with success relationship that time I received the mail

but if any query failed that time not receive any mail.

Don't have an account?
Coming from Hortonworks? Activate your account here