One way to do this is extract all the content of the json file as attributes associated with the flowfile.
Then once you make invoke http extract the results of invokehttp to attributes again.
By following this process all the contents will be associated as attributes of the flowfile i.e the attributes associated with the flowfile includes the content of your json file, results of your invoke http processors because we are extracting every content and keeping them as attributes to the flowfile.
In final use AttributesToJson processor and keep all the attributes you need in that, this processor will result a json message and store that to mongodb.
Flow:-
1.EvaluateJsonProcessor //extract the content to flowfile attribute 2.invokehttp 3.EvaluateJsonProcessor//extract the result of invoke http to flowfile attribute 4.AttributesToJSON //give all the list of attributes you need. so that this processor prepares new json message.
Example:-
Input Json file:-
{
"id": 1,
"name": "HCC",
"age": 20
}extract all the content i.e id,name,age using Evaluatejsonpath change Destination property to flowfile-attribute processor, now all the content will be associated as attributes now.
Evaljsonpath configs:-
Then do invokehttp now the result flowfile from invokehttp also will have id,name,age attributes associated with it.
use another evaluatejsonpath(if response from invokehttp is json) and extract the invokehttp results as attributes again.
lets consder invokehttpresult will have
{ "dept": "community"} extract dept as attribute.
Right now your flowfile will have id,name,age,dept as attributes to the flowfile.
you can do as many invokehttps as you want after each invoke http extract the content as attribute.
Now you need to prepare a json message use AttribiutestoJSON processor
Change the property
attributes list as
id,name,age,dept
This processor now prepares new json message with all the attributes you listed in the processor.
{ "id": 1, "name": "HCC", "age": 20,"dept": "community"}Use the json result from Attributestojson processor to store into mongoDB.
AttributesToJSON Configs:-
One way to do this is extract all the content of the json file as attributes associated with the flowfile.
Then once you make invoke http extract the results of invokehttp to attributes again.
By following this process all the contents will be associated as attributes of the flowfile i.e the attributes associated with the flowfile includes the content of your json file, results of your invoke http processors because we are extracting every content and keeping them as attributes to the flowfile.
In final use AttributesToJson processor and keep all the attributes you need in that, this processor will result a json message and store that to mongodb.
Flow:-
1.EvaluateJsonProcessor //extract the content to flowfile attribute 2.invokehttp 3.EvaluateJsonProcessor//extract the result of invoke http to flowfile attribute 4.AttributesToJSON //give all the list of attributes you need. so that this processor prepares new json message.
Example:-
Input Json file:-
{
"id": 1,
"name": "HCC",
"age": 20
}extract all the content i.e id,name,age using Evaluatejsonpath change Destination property to flowfile-attribute processor, now all the content will be associated as attributes now.
Evaljsonpath configs:-
Then do invokehttp now the result flowfile from invokehttp also will have id,name,age attributes associated with it.
use another evaluatejsonpath(if response from invokehttp is json) and extract the invokehttp results as attributes again.
lets consder invokehttpresult will have
{ "dept": "community"} extract dept as attribute.
Right now your flowfile will have id,name,age,dept as attributes to the flowfile.
you can do as many invokehttps as you want after each invoke http extract the content as attribute.
Now you need to prepare a json message use AttribiutestoJSON processor
Change the property
attributes list as
id,name,age,dept
This processor now prepares new json message with all the attributes you listed in the processor.
{ "id": 1, "name": "HCC", "age": 20,"dept": "community"}Use the json result from Attributestojson processor to store into mongoDB.
AttributesToJSON Configs:-
@Shu_ashu / All,
I have a nested JSON (not an array).
I am able to read it using "EvaluateJsonPath" for all the direct attributes (IMEI or Counter in my case). The nested attributes($.Events.EvenType.Name or $.Events.Properties.Latitude) is not recognized and they are getting populated as NULL in my destination table. Can you please advise?
Data Pipeline is as:
GetFile --EvaluateJSONPath -- ConvertJSONtoSQL -- PutSQL
EvaluateJSONPath Property (as attached):
Sample JSON Data:
{
"IMEI":"77777777777777",
"Events":
{
"EventType":{"Name":"EventName","Time":1577184434,"ReceievedTime":1577184434,"MsgId":1493134167},
"Properties":{"ThingUnitId":"779000000000062","Latitude":"24","Longitude":"46"
},"Counter":0
}
Thanks
Hello All,
The below pipeline solved the issue.
The naming convention between the table and the Property/value in Nifi has to be set same.
If not, it cannot recognize and either it will fail or NULL gets loaded. It is not as simple as expanding the the JSON using SQLServer query!!!
Note: Value data in the EvoluteJSONPath is setup based on the FlattenJSON output. I had the "Seperator" as '_' and "Flattenmode" as 'dot notation' in the FlattenJSON setup.
Hi @Siraj
Looking over your most recent post it appears that you solved your issue. Can you confirm by using the Accept as Solution button here so it can be of assistance to others?
