Nifi - Dynamic attribute in the flow

[#1]

I have a dynamic attribute requirement.

Current Flow

1. GenerateFlow --> invokeHttp ( GenerateFlow is scheduled to run every 60 mins )
2. invokeHttp --> updateAttribute

I have stTime & endTime to invokeHttp which needs to be changed based on last invoke time.

Lets say, invoking http very first time, stTime = 04-01-2023 00:00 endTime = 04-01-2023 01:00

In the next schedule, it should be,

stTime = 04-01-2023 01:00 endTime = 04-01-2023 02:00

I tried to set attribute in the generate flow with initial value and incrementing using updateAttribute after the invokehttp is done. This value gets overwritten when the next schedule is starting by GenerateFlow.

[#2]

@udayabaski, Welcome to our community! To help you get the best possible answer, I have tagged in our Nifi experts @cotopaul @ckumar @steven-matison @MattWho  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@udayabaski  This sounds like a job for DistributedMapCache.  Using this, you can set key/value pairs for start and end time at one point in the flow, and access them in other parts of the flow.

Here are some important docs to get you started :

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-distributed-cache-services-na...

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apach...

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apach...

[#4]

Thanks for the links. Where to initialize the values in the flow? My requirement is to initialize somewhere and then update the value after invokeHttp.

1. GenerateFlow --> invokeHttp ( GenerateFlow is scheduled to run every 60 mins )
2. invokeHttp --> updateAttribute

As per the flow above, it gets initialized again before the invokeHttp and gets updated after the invokeHttp. But, should be intialized only once and keep updating after the invokeHttp.

[#5]

@udayabaski, as @steven-matison mentioned, the best solution would be to use the Distributed Map Cache. In order to implement it, you can follow these steps to initialize your Server: https://stackoverflow.com/questions/44590296/how-does-one-setup-a-distributed-map-cache-for-nifi/445...

Now, regarding the way you want to incorporate it in your flow, I would suggest the following:
- right after UpdateAttribute, you activate the PutDistributedMapCache. Within the processor, you will set the desired attribute at the property Cache Entry Identified.
- Before InvokeHTTP, you add a FetchDistributedMapCache with which you extract the value for your key:value pair. All you have to do next is to extract your attribute to further use in your invokeHTTP.

It is as simple as that and you do not need any fancy configurations 🙂