Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Dynamic property naming in NiFi

Solved Go to solution

Dynamic property naming in NiFi

Explorer

Hi there,

I'm in a situation where I need to be able to set the name of a property using Expression Language. Is there any way to do that?

I try to create an attribute using an UpdateAttribute processor and put in ${property1} as the property name. Now, property1 is in reference to an existing attribute containing the value "property2". What I'd hope is that the flowfile has a new property added named "property2" but what I end up with is a new property literally named "${property1}".

Is this even possible?

Thanks,

JP

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Dynamic property naming in NiFi

New Contributor

Hi,

Take a look at the JoltTransformJSON processor, it is capable to do the exact transformation you need.

More info about Jolt:

https://github.com/bazaarvoice/jolt

http://jolt-demo.appspot.com/

Cheers.

View solution in original post

5 REPLIES 5
Highlighted

Re: Dynamic property naming in NiFi

Explorer

Maybe there's another way to ask this; I shouldn't pre-suppose dynamic naming is necessary.

Say I have this JSON document:

{ "boxes": [{ "box": { "oranges": 2, "apples": 5 } }, { "box": { "oranges": 2, "bananas": 7 } } ] }

I need to then do some work to create something like this:

{ "boxes": [{ "box": 1, "fruits": 2, "count": 7 }, { "box": 2, "fruits": 2, "count": 9 } ] }

My thinking was first split by box to get a "box" attribute and 2 flowfiles. Then split by contents of each box to create 4 flowfiles each with a "fruit" attribute. But how do I merge back to construct pieces of the final output? The attribute names would be the same with different values. I need for the attribute names to be different (like box.1, box.2, fruit.1, fruit.2) which would be easy using fragment indexes, but I can't seem to name properties dynamically like that. Is there an alternative?

Does that make any sense?

Thanks,

JP

Highlighted

Re: Dynamic property naming in NiFi

Explorer

Alternately, is there a way to get the size of a JSON array using EvaluateJSONPath?

Highlighted

Re: Dynamic property naming in NiFi

New Contributor

Hi,

Take a look at the JoltTransformJSON processor, it is capable to do the exact transformation you need.

More info about Jolt:

https://github.com/bazaarvoice/jolt

http://jolt-demo.appspot.com/

Cheers.

View solution in original post

Highlighted

Re: Dynamic property naming in NiFi

@JP

One way of doing it is using ExecuteScript Processor. Follows a sample solution.

I generated a flow file using GenerateFlowFile Processor which has an attribute named "Property1" with the value set to "Property2".

64755-screen-shot-2018-03-22-at-121100-am.png

Now, this goes to an ExecuteScript Processor where I am using a Groovy script to create a new property "on the fly"!

64756-screen-shot-2018-03-22-at-121342-am.png

Follows the groovy script which is creating the property on the fly.

flowFile = session.get()
if(!flowFile) return
myAttr = flowFile.getAttribute('property1')
flowFile = session.putAttribute(flowFile, myAttr, 'myValue')
session.transfer(flowFile, REL_SUCCESS)

The flow file coming out of this processor looks something like this.

64757-screen-shot-2018-03-22-at-121458-am.png

Hope that helps!

Highlighted

Re: Dynamic property naming in NiFi

Explorer

As per Taras' answer; a JOLT transform ended up being exactly what we needed.

Don't have an account?
Coming from Hortonworks? Activate your account here