Multiple properties in EvaluateJsonPath processor in Nifi

[#1]

Hi All,

I have a big payload coming in from Kafka topic to EvaluateJsonPath Processor in Nifi. I need to extract about 100 fields from the JSON payload that's coming in. I was wondering, if there was a way to specify these fields as property in EvaluateJsonProperty at once instead of specifying them one by one, which is time consuming.

Thanks

[#2 · Accepted solution]

@Krishna Sreenivas

As of now there is no built in processor in NiFi that could extract Json attribute dynamically because if we add significant number of attributes dynamically to the flow file, As attributes are hold in memory this will cause performance issues.

We need to add all the json keys/attributes and matching json path as value in EvaluateJsonPath processor and change the destination property to flowfile-attribute to keep them as a flow file attributes.

Please refer to This link for more details regarding dynamic attribute extract processor.

Refer to this link regards to similar kind of usecase and end of this thread there is script which you can leverage to extract key/values.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

[#3]

Can you explain more about your use case? Why do you need to extract ~100 fields, how are they being used downstream? I wonder if a record-based processor or something would be more appropriate...

[#4]

The 100 fields are to be used by another down stream application. I actually referred to your blog on writing groovy script to extract the fields. But facing issues with that as well.

[#5]

@Krishna Sreenivas

Could you add more details regards to your use case for extracting 100 filed values?

If you are extracting the values and preparing csv file then sending the csv file into Downstream applications?
if this is the case then

-> if you are using NiFi-1.2+ use ConvertRecord processor to convert the Json Format data into CSV format then send to your down stream systems.

[#6]

@ShuThank you very much for pointing me in the right direction. I wrote a script for it.