UpdateAttribute Warning.

[#1]

I am seeing below messages and it is messing up the SRC_OFFSET in this case, i have configured UpdateAttribute with 2 concurrent Tasks. mine is a 3 node cluster, Nifi version 1.9.0

[#2 · Accepted solution]

@Samar Aarkotti

The exception you are seeing can be expected because of the concurrent execution you have going on per node. With 2 concurrent tasks, you have the processor potentially executing its code in twice in parallel resulting in on thread updating what is in state before the other thread does.

-

In the event that the UpdateAttribute processor is unable to get the state at the beginning of the onTrigger, the FlowFile will be pushed back to the originating relationship and the processor will yield. If the processor is able to get the state at the beginning of the onTrigger but unable to set the state after adding attributes to the FlowFile, the FlowFile will be transferred to "set state fail". This is normally due to the state not being the most up to date version (another thread has replaced the state with another version). In most use-cases this relationship should loop back to the processor since the only affected attributes will be overwritten.

-

I would suggest when using state in the updateAttribute processor that you configure the processor to with only 1 concurrent task. Keep in mind that the processor settings are per node so each node in you cluster will still be executing this processor.

-

If throughput is not meeting your needs, make sure you have properly load-balanced the source FlowFile across all nodes in your cluster. If you are and throughput is still an issue, try adjusting the "Run Duration" in very small increments and still leave concurrent tasks to 1.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

[#3]

@Samar Aarkotti

*** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion.

It always best to leave your processor at default value for concurrent task unless there is a specific need to increment. Here is an article on this topic:

https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor....
and another on "Run Duration":
https://community.hortonworks.com/articles/221807/understanding-nifi-processors-run-duration-functio...

[#4]

Thanks @Matt Clarke i will set the configure concurrent task for this processor to be 1 , my rest of the processor's in the flow are set to 2 concurrent task's . which should be fine i guess.