Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Composing Json in Nifi

Solved Go to solution

Composing Json in Nifi

Expert Contributor

Hi,

I have this scenario where after reading json files I'm doing InvokeHttp against a url attribute in each json file. This returns a further list of json objects with url attribute which I later split and do InvokeHttp individually against each url for result.

Now the problem is at the end I need to have a composed json against each flow resulting from the inital json that I read from file along with the later json objects and the final result received after hitting individual url. I need to save this composed json as record in mongodb.

I 'm having troubling making this json, so need help regarding the flow and processors.

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Composing Json in Nifi

Super Guru

@Arsalan Siddiqi

One way to do this is extract all the content of the json file as attributes associated with the flowfile.

Then once you make invoke http extract the results of invokehttp to attributes again.

By following this process all the contents will be associated as attributes of the flowfile i.e the attributes associated with the flowfile includes the content of your json file, results of your invoke http processors because we are extracting every content and keeping them as attributes to the flowfile.

In final use AttributesToJson processor and keep all the attributes you need in that, this processor will result a json message and store that to mongodb.

Flow:-

1.EvaluateJsonProcessor //extract the content to flowfile attribute
2.invokehttp 
3.EvaluateJsonProcessor//extract the result of invoke http to flowfile attribute
4.AttributesToJSON //give all the list of attributes you need. so that this processor prepares new json message.

Example:-

Input Json file:-

{
"id": 1,
"name": "HCC",
"age": 20
}

extract all the content i.e id,name,age using Evaluatejsonpath change Destination property to flowfile-attribute processor, now all the content will be associated as attributes now.

Evaljsonpath configs:-

41501-evaluatejsonpath.png

Then do invokehttp now the result flowfile from invokehttp also will have id,name,age attributes associated with it.

use another evaluatejsonpath(if response from invokehttp is json) and extract the invokehttp results as attributes again.

lets consder invokehttpresult will have

{ "dept": "community"} 

extract dept as attribute.

Right now your flowfile will have id,name,age,dept as attributes to the flowfile.

you can do as many invokehttps as you want after each invoke http extract the content as attribute.

Now you need to prepare a json message use AttribiutestoJSON processor

Change the property

attributes list as

id,name,age,dept

This processor now prepares new json message with all the attributes you listed in the processor.

{ "id": 1, "name": "HCC", "age": 20,"dept": "community"}

Use the json result from Attributestojson processor to store into mongoDB.

AttributesToJSON Configs:-

41502-attributestojson.png

3 REPLIES 3

Re: Composing Json in Nifi

Expert Contributor

Re: Composing Json in Nifi

Super Guru

@Arsalan Siddiqi

One way to do this is extract all the content of the json file as attributes associated with the flowfile.

Then once you make invoke http extract the results of invokehttp to attributes again.

By following this process all the contents will be associated as attributes of the flowfile i.e the attributes associated with the flowfile includes the content of your json file, results of your invoke http processors because we are extracting every content and keeping them as attributes to the flowfile.

In final use AttributesToJson processor and keep all the attributes you need in that, this processor will result a json message and store that to mongodb.

Flow:-

1.EvaluateJsonProcessor //extract the content to flowfile attribute
2.invokehttp 
3.EvaluateJsonProcessor//extract the result of invoke http to flowfile attribute
4.AttributesToJSON //give all the list of attributes you need. so that this processor prepares new json message.

Example:-

Input Json file:-

{
"id": 1,
"name": "HCC",
"age": 20
}

extract all the content i.e id,name,age using Evaluatejsonpath change Destination property to flowfile-attribute processor, now all the content will be associated as attributes now.

Evaljsonpath configs:-

41501-evaluatejsonpath.png

Then do invokehttp now the result flowfile from invokehttp also will have id,name,age attributes associated with it.

use another evaluatejsonpath(if response from invokehttp is json) and extract the invokehttp results as attributes again.

lets consder invokehttpresult will have

{ "dept": "community"} 

extract dept as attribute.

Right now your flowfile will have id,name,age,dept as attributes to the flowfile.

you can do as many invokehttps as you want after each invoke http extract the content as attribute.

Now you need to prepare a json message use AttribiutestoJSON processor

Change the property

attributes list as

id,name,age,dept

This processor now prepares new json message with all the attributes you listed in the processor.

{ "id": 1, "name": "HCC", "age": 20,"dept": "community"}

Use the json result from Attributestojson processor to store into mongoDB.

AttributesToJSON Configs:-

41502-attributestojson.png

Highlighted

Re: Composing Json in Nifi

New Contributor

@rsalan Siddiqi how did you solve ur prblm ? I have same use case

,

@rsalan Siddiqi how did you solve this prblm ? I have same use case

Don't have an account?
Coming from Hortonworks? Activate your account here