how i extract attribute from json file using nifi

[#1]

I use gethttp processor to stream data using http request, so I need to get attribute from file json that I get from the first request and use it for another request, how I can do this with the existing nife processor.any help is appreciated

[#2 · Accepted solution]

@nejm hadj You can parse a JSON with the EvaluateJsonPath processor. You will need to make sure that the information that your getHTTP processor outputs is strictly JSON, then feed it into EvaluateJsonPath. From there, you can reference the fields in the JSON and forward them on to other processors.

Twitter_Dashboard.xml is a Twitter processing template that has a good example of evaluating JSON files in it. Import the template into your NiFi dashboard then instantiate it to check out the EvaluateJsonPath processor to see how it's done:

[#3]

@emaxwell

Hi emaxwell,

I have a same task to extract attributes from json file via gethttp with EvaluateJsonPath processor. However, the resulting data from the processor do not limit to the attributes, it pulled all attributes the original json data has. Is this meant to be so?

Thank you!

Shannon

[#4]

@shannon luo

You can use an EvaluateJSONPath processor to pull out the fields that you want in the flow. You will create a parameter for each field in the JSON you wish to put on the output flow file. The name will be what you want the field to be called on the output, and the value will be an expression equating to the field in the input JSON (e.g. Name = twitter.name, Value = $.user.screen_name takes the input user:screen_name value from the JSON and creates a variable called twitter.name on the output flow file).

Thanks

Erik

[#5]

@emaxwell

That was what I did. but still end up pulling all attribute. Going to Provenance, it showed EvaluateJsonPath pulled empty strings on all specified attribute. Would you be so kind an tell me where I got it wrong? -- I tried to pull "subject" property using value $.response.Matches.M

atch.Subject, $.Matches.Match.Subject, $.Match.Subject, and $.Subject. None of those worked. The source data looks like (see below):

Thanks !

Shannon

[#6]

@shannon luo

What are the configuration attributes of your EvaluateJSONPath processor? Is you destination set to "flowfile-content" or "flowfile-attribute"? I have a processor set up to evaluate Twitter JSON, and the destination is set to "flowfile-attribute" with a number of attributes identified. Can you take a look at the attached image and see if your attributes are configured similarly?

[#7]

@emaxwell

Yes, that was the template I used as the base for development. I used flowfile-attibute. Here is the screenshot for the properties. None of those properties added works.

Edit: May be this is what caused problem: Match has an array as its value. I tried $.response.Request.Actual.query, it worked fine. Do you know what to do to parse value from "Match"? (without going to a ExecuteScript).

Thanks so much for helping!

Shannon

[#8]

It would be nice to have a processor that specifically creates a list of attributes coming out of a JSON message.

[#9]

Hi, I am converting the output of ExecuteSQL processor in JSON format and trying the above steps to extract some fetched column values from it. For that I have follow the same steps mentioned here to make it work but its not working. Plz suggest. My main idea to implement is to extract some column from executeSQL and based on that value I need to do some file ingestion. Help will be appreciated

[#10]

Try this:

ExecuteSQL > SplitAvro > ConvertAvrotoJSON > EvaluateJsonPath

SplitAvro creates individual Avro records

ConvertAvrotoJSON creates JSON from Avro

EvaluateJsonPath allows you to create new FlowFile attributes from JSON path.

[#11]

hi is there any way to extract the attributes without nifi ! i mean , i need to extract an attribute from a collection of tweet i already have !

[#12]

Hi, We have extracted the required fields from Json file as below.

It is not with only one processor. It is with two processors.

1. EvaluateJsonPath : What ever the configurations above mentioned are correct. This processor will validate whether this field exists or not. If not, then it will show that filed value as null.

2. AttributesToJSON : In the Attributes list property, give what ever the fields you want to pull (with comma separated) from the fields added in the EvaluateJsonPath processor.

[say you have added field1 to 6 in EvaluateJsonPath and then you want only field 1,3,4. Then give field1,field3,field4 in the Attributes list in the AttributesToJSON processor.]

Hope this helps you.

[#13]

Hello, I have a json flow file as follows:

I used: EvaluateJsonPath:

But the result is: < Empty string set>

Thanks for your help and guide.

[#14]

You are using an array, this is why you got null values. You have 2 options:

1. Change the expressions to $[0].ARRIVAL_DATE and $[0].BOOK_ID, and it will work
2. Read the file and split the array using the SplitJson processor - set JsonPath expression to: $, and connect the SplitJson processor to your EvaluateJsonPath processor and it should work.