question Re: Multiple properties in EvaluateJsonPath processor in Nifi in Archives of Support Questions (Read Only)

Multiple properties in EvaluateJsonPath processor in Nifi

krishna_sreeniv — Fri, 21 Sep 2018 05:46:32 GMT

Hi All,

I have a big payload coming in from Kafka topic to EvaluateJsonPath Processor in Nifi. I need to extract about 100 fields from the JSON payload that's coming in. I was wondering, if there was a way to specify these fields as property in EvaluateJsonProperty at once instead of specifying them one by one, which is time consuming.

Thanks

Re: Multiple properties in EvaluateJsonPath processor in Nifi

Shu_ashu — Fri, 21 Sep 2018 06:53:42 GMT

@Krishna Sreenivas

As of now there is no built in processor in NiFi that could extract Json attribute dynamically because if we add significant number of attributes dynamically to the flow file, As attributes are hold in memory this will cause performance issues.

We need to add all the json keys/attributes and matching json path as value in EvaluateJsonPath processor and change the destination property to flowfile-attribute to keep them as a flow file attributes.

Please refer to This link for more details regarding dynamic attribute extract processor.

Refer to this link regards to similar kind of usecase and end of this thread there is script which you can leverage to extract key/values.

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

Re: Multiple properties in EvaluateJsonPath processor in Nifi

mburgess — Fri, 21 Sep 2018 10:02:55 GMT

Can you explain more about your use case? Why do you need to extract ~100 fields, how are they being used downstream? I wonder if a record-based processor or something would be more appropriate...

Re: Multiple properties in EvaluateJsonPath processor in Nifi

krishna_sreeniv — Fri, 21 Sep 2018 22:54:10 GMT

@ShuThank you very much for pointing me in the right direction. I wrote a script for it.

Re: Multiple properties in EvaluateJsonPath processor in Nifi

krishna_sreeniv — Sat, 22 Sep 2018 02:07:54 GMT

The 100 fields are to be used by another down stream application. I actually referred to your blog on writing groovy script to extract the fields. But facing issues with that as well.

Re: Multiple properties in EvaluateJsonPath processor in Nifi

Shu_ashu — Sat, 22 Sep 2018 09:22:49 GMT

@Krishna Sreenivas

Could you add more details regards to your use case for extracting 100 filed values?

If you are extracting the values and preparing csv file then sending the csv file into Downstream applications?
if this is the case then

-> if you are using NiFi-1.2+ use ConvertRecord processor to convert the Json Format data into CSV format then send to your down stream systems.