Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Multiple properties in EvaluateJsonPath processor in Nifi

Solved Go to solution

Multiple properties in EvaluateJsonPath processor in Nifi

Hi All,

I have a big payload coming in from Kafka topic to EvaluateJsonPath Processor in Nifi. I need to extract about 100 fields from the JSON payload that's coming in. I was wondering, if there was a way to specify these fields as property in EvaluateJsonProperty at once instead of specifying them one by one, which is time consuming.

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Multiple properties in EvaluateJsonPath processor in Nifi

Super Guru

@Krishna Sreenivas

As of now there is no built in processor in NiFi that could extract Json attribute dynamically because if we add significant number of attributes dynamically to the flow file, As attributes are hold in memory this will cause performance issues.

We need to add all the json keys/attributes and matching json path as value in EvaluateJsonPath processor and change the destination property to flowfile-attribute to keep them as a flow file attributes.

Please refer to This link for more details regarding dynamic attribute extract processor.

Refer to this link regards to similar kind of usecase and end of this thread there is script which you can leverage to extract key/values.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

View solution in original post

5 REPLIES 5
Highlighted

Re: Multiple properties in EvaluateJsonPath processor in Nifi

Super Guru

@Krishna Sreenivas

As of now there is no built in processor in NiFi that could extract Json attribute dynamically because if we add significant number of attributes dynamically to the flow file, As attributes are hold in memory this will cause performance issues.

We need to add all the json keys/attributes and matching json path as value in EvaluateJsonPath processor and change the destination property to flowfile-attribute to keep them as a flow file attributes.

Please refer to This link for more details regarding dynamic attribute extract processor.

Refer to this link regards to similar kind of usecase and end of this thread there is script which you can leverage to extract key/values.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

View solution in original post

Highlighted

Re: Multiple properties in EvaluateJsonPath processor in Nifi

Super Guru

Can you explain more about your use case? Why do you need to extract ~100 fields, how are they being used downstream? I wonder if a record-based processor or something would be more appropriate...

Highlighted

Re: Multiple properties in EvaluateJsonPath processor in Nifi

The 100 fields are to be used by another down stream application. I actually referred to your blog on writing groovy script to extract the fields. But facing issues with that as well.

Highlighted

Re: Multiple properties in EvaluateJsonPath processor in Nifi

Super Guru
@Krishna Sreenivas

Could you add more details regards to your use case for extracting 100 filed values?

If you are extracting the values and preparing csv file then sending the csv file into Downstream applications?
if this is the case then

-> if you are using NiFi-1.2+ use ConvertRecord processor to convert the Json Format data into CSV format then send to your down stream systems.

Highlighted

Re: Multiple properties in EvaluateJsonPath processor in Nifi

@ShuThank you very much for pointing me in the right direction. I wrote a script for it.

Don't have an account?
Coming from Hortonworks? Activate your account here