How to Add Dynamic Tags in NiFi

[#1]

Is it possible to use Expression language in UpdateAttribute processor?

Not for the Property Value but for the Property Name?

My mission is to get Dynamic Tag names.. If my JSON has 2 tags.. say Tag_1 and Tag_2.. I want to change them (tag_name not tag_value) to some other name, can that be done ?

OR

if existing tag name cannot be renamed, can I add a new tag? but again, the name of the Tag cannot be hardcoded in NiFi..

[#2]

Can you provide a little more detail on your use case? Are you trying to dynamically create new FlowFile Attributes or are you trying to update JSON tag names in the FlowFile content's JSON itself? The updateAttribute processor works with FlowFile Attributes using the expression language. The FlowFile's content is not read by this processor.

[#3]

UpdateAttribute will change the value of the JSON tag.. but it wont change the name of the Tag.. I am looking to change the name of the tag dynamically..

Like suppose in my JSON, i have a tag named CREDIT_FLAG with value 'C'.. I want the name of the tag to change from CREDIT_FLAG to say CR_FL.. assuming the data "CR_FL" is also present in our flow..

-----

and if the above cannot be done, can we create new FlowFile Attributes dynamically? or any other way

-----

Summary:

Change tag name "CREDIT_FLAG" to "CR_FL" in any possible way using NiFi built in processors

-----

Let me know if more clarification is required..

[#4 · Accepted solution]

@Anubhav Raikar

I think we have a terminology issue here. :) A NiFi FlowFile consists of two parts, the FlowFile attributes (Metadata about the FlowFile) and the FlowFile Content (In you case this would be the actual JSON).

NiFi FlowFile attributes consist of key value pairs (A property name and an associated value).

So if i am understanding you correctly, you have a FlowFile which has a FlowFile Attribute of property name of "CREDIT_FLAG" and a value of "C". You are looking for a way to change the FlowFile attribute Property name from "CREDIT_FLAG" to "CR_FL". You mention a criteria the "CR_FL" also exists in your flow? Are you saying you expect "CR_FL" to also exist in this same FlowFile or some other FlowFile elsewhere in your NiFi dataflow? NiFI processors interact with the FlowFile attributes and contents of a single FlowFile and do not cross reference with other processors or FlowFiles.

I cannot think of any processors that would allow you to dynamically create FlowFile attribute property names. Property names do accept Expression language. I am trying to understand the value of a dynamically created FlowFile Attribute property name. What use would that attribute have later in your dataflow. since it is dynamically created, how would you reference later in some other Expression language statement? FlowFile Attributes are not part of the FlowFile content that is written out when the FlowFile is processed by a putFile or putSFTP. They can be referenced as headers and such by processors like putSQL or invokeHTTP, but how would you do that if you don't know what the FlowFile attribute property name is?

Matt

[#5]

If there is a fixed mapping of names, you can use JoltTransformJSON to rename the fields. For more complex things (dynamic attribute creation), you can use ExecuteScript.

[#6]

there is no fixed mapping.. might go with Custom Processor to do all the tasks

[#7]

yes, you are right.. even if the attributes get created.. I have no way to further reference them.. I was looking to create statements in PutSQL but yes I wont be able to do it..