Silver ⭐️⭐️⭐️

Solved

The InvokeHttp Processor's 'Retry' option is causing the flow to queue up after the retry attempts.

7 replies
5885 views
Translate All

I have configured an InvokeHttp Processor with the 'Retry' relationship enabled to capture HTTP status codes between 500 and 599 and initiate retries. I connected the 'Retry' relationship back to the InvokeHTTP processor to enable automatic reattempts. The number of retry attempts is set to 3. However, after all retry attempts are exhausted, the process does not complete and ends up queuing in the Retry relationship, as shown below:

Could you please help how to fix it?

Apache NiFi

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@s198

If I am following your description correctly, It appears to be working as designed.

 The number of retry attempts is set to 3. However, after all retry attempts are exhausted, the process does not complete and ends up queuing in the Retry relationship.

When you set "Retry" on a relationship, it tell NiFi to attempt that many retries before on the component BEFORE routing the FlowFile to the relationship (So FlowFile stays on current inbound connection and you see the hour glass indicating a FlowFile is penalized on that connection). See in your dataflow InvokeHTTP will retry up to 3 times to process the inbound FlowFile. If processing of the FlowFile results in a "retry" relationship destination, it will get penalized again until configured number of retries completes and the route to "retry" relationship connection. And that is waht we see happening in your dataflow. After those three attempts the FlowFile get transferred to "retry" relationship which you have in a connection looping back on the processor. We then see the hour glass there meaning the 3 reties is happening again. This will continue forever unless that FlowFile is eventually successful.

I think what you really wanted to do is retry 3 times and if all three times result in routing to retry relationship, terminate the FlowFile (delete it). So in this case you would check both "terminate" and "retry" boxed on the retry relationship. After 3 unsuccess FlowFile get terminated.

If you want to retain the FlowFile after 3 failed retry attempts, don't loop the "retry" relationship back on the processor, but instead route to some other processor for your error handling/notification/etc...

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

DianaTorres

Community Manager

@s198 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @SAMSAL @MattWho who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate

Regards, Diana Torres, Senior Community Moderator

mburgess

Gold ⭐️⭐️⭐️

This may be related to https://issues.apache.org/jira/browse/NIFI-11783 which was fixed in CFM 2.1.6. Is upgrading your CFM an option?

Translate

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@s198

If I am following your description correctly, It appears to be working as designed.

 The number of retry attempts is set to 3. However, after all retry attempts are exhausted, the process does not complete and ends up queuing in the Retry relationship.

When you set "Retry" on a relationship, it tell NiFi to attempt that many retries before on the component BEFORE routing the FlowFile to the relationship (So FlowFile stays on current inbound connection and you see the hour glass indicating a FlowFile is penalized on that connection). See in your dataflow InvokeHTTP will retry up to 3 times to process the inbound FlowFile. If processing of the FlowFile results in a "retry" relationship destination, it will get penalized again until configured number of retries completes and the route to "retry" relationship connection. And that is waht we see happening in your dataflow. After those three attempts the FlowFile get transferred to "retry" relationship which you have in a connection looping back on the processor. We then see the hour glass there meaning the 3 reties is happening again. This will continue forever unless that FlowFile is eventually successful.

I think what you really wanted to do is retry 3 times and if all three times result in routing to retry relationship, terminate the FlowFile (delete it). So in this case you would check both "terminate" and "retry" boxed on the retry relationship. After 3 unsuccess FlowFile get terminated.

If you want to retain the FlowFile after 3 failed retry attempts, don't loop the "retry" relationship back on the processor, but instead route to some other processor for your error handling/notification/etc...

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

s198Author

Silver ⭐️⭐️⭐️

Thank you @MattWho for your valuable input.

Yes, as you mentioned, my requirement is to retry 3 times, and if all three attempts result in routing to the retry relationship, terminate the FlowFile.

I tried both options you mentioned.

The first option doesn’t seem to be possible. I tried checking both the "terminate" and "retry" boxes on the retry relationship, but an error popped up: "Cannot automatically terminate 'Retry' relationship because a connection already exists with this relationship." Since I’ve already connected the retry relationship back to the InvokeHttp processor, it’s not allowing me to enable "terminate."

Could you please provide more input on this option?

The second option is working fine. Instead of looping the "retry" relationship back to the processor, I routed it to another processor. After the 3 retry attempts, it routes to the next processor and terminates the FlowFile.

Translate

s198Author

Silver ⭐️⭐️⭐️

Hi @MattWho

Please disregard my previous response.

I removed the retry relationship which is connecting back to the InvokeHttp Processor and I am able to check both the "terminate" and "retry" boxes. The retry option is working as expected.

Translate

MattWho

Titanium ⭐️⭐️⭐️

@s198

It can get confusing. Relationships and Connections are two different things.

You can think of "relationships" as the component processor's internal code routing and "Connections" are the external NiFi framework routing of the FlowFile output by that processor.

A processor unique code says when X happens route FlowFile to relationship Y.
The framework says allows you to create a connection from one components to another and assign one or more of a processor's relationship to it. Or if the the relationship is allocated to "terminate", the framework can properly handle the removal/clean-up of that FlowFile. So the frameworks says: "Hey, you can't terminate a FlowFile I am suppose to route via an existing connection."

Matt

Translate

s198Author

Silver ⭐️⭐️⭐️

Thank you, @MattWho , for the additional information. I really appreciate you providing timely and clear solutions. You’re awesome!

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.