Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Composing Json in Nifi

avatar
Expert Contributor

Hi,

I have this scenario where after reading json files I'm doing InvokeHttp against a url attribute in each json file. This returns a further list of json objects with url attribute which I later split and do InvokeHttp individually against each url for result.

Now the problem is at the end I need to have a composed json against each flow resulting from the inital json that I read from file along with the later json objects and the final result received after hitting individual url. I need to save this composed json as record in mongodb.

I 'm having troubling making this json, so need help regarding the flow and processors.

Thanks

1 ACCEPTED SOLUTION

avatar
Master Guru

@Arsalan Siddiqi

One way to do this is extract all the content of the json file as attributes associated with the flowfile.

Then once you make invoke http extract the results of invokehttp to attributes again.

By following this process all the contents will be associated as attributes of the flowfile i.e the attributes associated with the flowfile includes the content of your json file, results of your invoke http processors because we are extracting every content and keeping them as attributes to the flowfile.

In final use AttributesToJson processor and keep all the attributes you need in that, this processor will result a json message and store that to mongodb.

Flow:-

1.EvaluateJsonProcessor //extract the content to flowfile attribute
2.invokehttp 
3.EvaluateJsonProcessor//extract the result of invoke http to flowfile attribute
4.AttributesToJSON //give all the list of attributes you need. so that this processor prepares new json message.

Example:-

Input Json file:-

{
"id": 1,
"name": "HCC",
"age": 20
}

extract all the content i.e id,name,age using Evaluatejsonpath change Destination property to flowfile-attribute processor, now all the content will be associated as attributes now.

Evaljsonpath configs:-

41501-evaluatejsonpath.png

Then do invokehttp now the result flowfile from invokehttp also will have id,name,age attributes associated with it.

use another evaluatejsonpath(if response from invokehttp is json) and extract the invokehttp results as attributes again.

lets consder invokehttpresult will have

{ "dept": "community"} 

extract dept as attribute.

Right now your flowfile will have id,name,age,dept as attributes to the flowfile.

you can do as many invokehttps as you want after each invoke http extract the content as attribute.

Now you need to prepare a json message use AttribiutestoJSON processor

Change the property

attributes list as

id,name,age,dept

This processor now prepares new json message with all the attributes you listed in the processor.

{ "id": 1, "name": "HCC", "age": 20,"dept": "community"}

Use the json result from Attributestojson processor to store into mongoDB.

AttributesToJSON Configs:-

41502-attributestojson.png

View solution in original post

7 REPLIES 7

avatar
Expert Contributor

avatar
Master Guru

@Arsalan Siddiqi

One way to do this is extract all the content of the json file as attributes associated with the flowfile.

Then once you make invoke http extract the results of invokehttp to attributes again.

By following this process all the contents will be associated as attributes of the flowfile i.e the attributes associated with the flowfile includes the content of your json file, results of your invoke http processors because we are extracting every content and keeping them as attributes to the flowfile.

In final use AttributesToJson processor and keep all the attributes you need in that, this processor will result a json message and store that to mongodb.

Flow:-

1.EvaluateJsonProcessor //extract the content to flowfile attribute
2.invokehttp 
3.EvaluateJsonProcessor//extract the result of invoke http to flowfile attribute
4.AttributesToJSON //give all the list of attributes you need. so that this processor prepares new json message.

Example:-

Input Json file:-

{
"id": 1,
"name": "HCC",
"age": 20
}

extract all the content i.e id,name,age using Evaluatejsonpath change Destination property to flowfile-attribute processor, now all the content will be associated as attributes now.

Evaljsonpath configs:-

41501-evaluatejsonpath.png

Then do invokehttp now the result flowfile from invokehttp also will have id,name,age attributes associated with it.

use another evaluatejsonpath(if response from invokehttp is json) and extract the invokehttp results as attributes again.

lets consder invokehttpresult will have

{ "dept": "community"} 

extract dept as attribute.

Right now your flowfile will have id,name,age,dept as attributes to the flowfile.

you can do as many invokehttps as you want after each invoke http extract the content as attribute.

Now you need to prepare a json message use AttribiutestoJSON processor

Change the property

attributes list as

id,name,age,dept

This processor now prepares new json message with all the attributes you listed in the processor.

{ "id": 1, "name": "HCC", "age": 20,"dept": "community"}

Use the json result from Attributestojson processor to store into mongoDB.

AttributesToJSON Configs:-

41502-attributestojson.png

avatar
Explorer

@Shu_ashu / All,

I have a nested JSON (not an array).

I am able to read it using "EvaluateJsonPath" for all the direct attributes (IMEI or Counter in my case). The nested attributes($.Events.EvenType.Name or $.Events.Properties.Latitude) is not recognized and they are getting populated as NULL in my destination table. Can you please advise? 

 

Data Pipeline is as:

GetFile --EvaluateJSONPath -- ConvertJSONtoSQL -- PutSQL

 

EvaluateJSONPath Property (as attached):

EvaluateJSON_Property.PNG

Sample JSON Data:

{

"IMEI":"77777777777777",

"Events":

{

"EventType":{"Name":"EventName","Time":1577184434,"ReceievedTime":1577184434,"MsgId":1493134167},

"Properties":{"ThingUnitId":"779000000000062","Latitude":"24","Longitude":"46"

},"Counter":0

}

 

Thanks

avatar
Explorer

Hello All,

The below pipeline solved the issue.

The naming convention between the table and the Property/value in Nifi has to be set same.

If not, it cannot recognize and either it will fail or NULL gets loaded. It is not as simple as expanding the the JSON using SQLServer query!!!

 

Note: Value data in the EvoluteJSONPath is setup based on the FlattenJSON output. I had the "Seperator" as '_' and "Flattenmode" as 'dot notation' in the FlattenJSON setup.

 

JSON_NestedData_PipeLine.png

avatar

Hi @Siraj

 

Looking over your most recent post it appears that you solved your issue. Can you confirm by using the Accept as Solution button here so it can be of assistance to others?

 

cloudera community accept solution button created 2019-12-14_19-46-50.jpg

Bill Brooks, Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Explorer

@ask_bill_brooks ,

 

Sorry, i am not seeing the accept as solution option in my screen. 

 

 

Thanks. 

avatar
New Contributor

@rsalan Siddiqi how did you solve ur prblm ? I have same use case

,

@rsalan Siddiqi how did you solve this prblm ? I have same use case