how i extract attribute from json file using nifi

[#1]

I use gethttp processor to stream data using http request, so I need to get attribute from file json that I get from the first request and use it for another request, how I can do this with the existing nife processor.any help is appreciated

[#2 · Accepted solution]

@nejm hadj You can parse a JSON with the EvaluateJsonPath processor. You will need to make sure that the information that your getHTTP processor outputs is strictly JSON, then feed it into EvaluateJsonPath. From there, you can reference the fields in the JSON and forward them on to other processors.

Twitter_Dashboard.xml is a Twitter processing template that has a good example of evaluating JSON files in it. Import the template into your NiFi dashboard then instantiate it to check out the EvaluateJsonPath processor to see how it's done:

[#3]

@emaxwell

Hi emaxwell,

I have a same task to extract attributes from json file via gethttp with EvaluateJsonPath processor. However, the resulting data from the processor do not limit to the attributes, it pulled all attributes the original json data has. Is this meant to be so?

Thank you!

Shannon

[#4]

@shannon luo

You can use an EvaluateJSONPath processor to pull out the fields that you want in the flow. You will create a parameter for each field in the JSON you wish to put on the output flow file. The name will be what you want the field to be called on the output, and the value will be an expression equating to the field in the input JSON (e.g. Name = twitter.name, Value = $.user.screen_name takes the input user:screen_name value from the JSON and creates a variable called twitter.name on the output flow file).

Thanks

Erik

[#5]

@emaxwell

That was what I did. but still end up pulling all attribute. Going to Provenance, it showed EvaluateJsonPath pulled empty strings on all specified attribute. Would you be so kind an tell me where I got it wrong? -- I tried to pull "subject" property using value $.response.Matches.M

atch.Subject, $.Matches.Match.Subject, $.Match.Subject, and $.Subject. None of those worked. The source data looks like (see below):

Thanks !

Shannon

[#6]

@shannon luo

What are the configuration attributes of your EvaluateJSONPath processor? Is you destination set to "flowfile-content" or "flowfile-attribute"? I have a processor set up to evaluate Twitter JSON, and the destination is set to "flowfile-attribute" with a number of attributes identified. Can you take a look at the attached image and see if your attributes are configured similarly?

[#7]

@emaxwell

Yes, that was the template I used as the base for development. I used flowfile-attibute. Here is the screenshot for the properties. None of those properties added works.

Edit: May be this is what caused problem: Match has an array as its value. I tried $.response.Request.Actual.query, it worked fine. Do you know what to do to parse value from "Match"? (without going to a ExecuteScript).

Thanks so much for helping!

Shannon

[#8]

It would be nice to have a processor that specifically creates a list of attributes coming out of a JSON message.

[#9]

Hi, I am converting the output of ExecuteSQL processor in JSON format and trying the above steps to extract some fetched column values from it. For that I have follow the same steps mentioned here to make it work but its not working. Plz suggest. My main idea to implement is to extract some column from executeSQL and based on that value I need to do some file ingestion. Help will be appreciated

[#10]

Try this:

ExecuteSQL > SplitAvro > ConvertAvrotoJSON > EvaluateJsonPath

SplitAvro creates individual Avro records

ConvertAvrotoJSON creates JSON from Avro

EvaluateJsonPath allows you to create new FlowFile attributes from JSON path.