question Re: Composing Json in Nifi in Archives of Support Questions (Read Only)

Composing Json in Nifi

arsalan_siddiqi — Tue, 24 Oct 2017 02:13:50 GMT

Hi,

I have this scenario where after reading json files I'm doing InvokeHttp against a url attribute in each json file. This returns a further list of json objects with url attribute which I later split and do InvokeHttp individually against each url for result.

Now the problem is at the end I need to have a composed json against each flow resulting from the inital json that I read from file along with the later json objects and the final result received after hitting individual url. I need to save this composed json as record in mongodb.

I 'm having troubling making this json, so need help regarding the flow and processors.

Thanks

Re: Composing Json in Nifi

arsalan_siddiqi — Tue, 24 Oct 2017 02:17:23 GMT

@Timothy Spann @Matt Burgess

Re: Composing Json in Nifi

Shu_ashu — Sun, 18 Aug 2019 00:28:24 GMT

@Arsalan Siddiqi

One way to do this is extract all the content of the json file as attributes associated with the flowfile.

Then once you make invoke http extract the results of invokehttp to attributes again.

By following this process all the contents will be associated as attributes of the flowfile i.e the attributes associated with the flowfile includes the content of your json file, results of your invoke http processors because we are extracting every content and keeping them as attributes to the flowfile.

In final use AttributesToJson processor and keep all the attributes you need in that, this processor will result a json message and store that to mongodb.

Flow:-

1.EvaluateJsonProcessor //extract the content to flowfile attribute
2.invokehttp 
3.EvaluateJsonProcessor//extract the result of invoke http to flowfile attribute
4.AttributesToJSON //give all the list of attributes you need. so that this processor prepares new json message.

Example:-

Input Json file:-

{
"id": 1,
"name": "HCC",
"age": 20
}

extract all the content i.e id,name,age using Evaluatejsonpath change Destination property to flowfile-attribute processor, now all the content will be associated as attributes now.

Evaljsonpath configs:-

Then do invokehttp now the result flowfile from invokehttp also will have id,name,age attributes associated with it.

use another evaluatejsonpath(if response from invokehttp is json) and extract the invokehttp results as attributes again.

lets consder invokehttpresult will have

{ "dept": "community"}

extract dept as attribute.

Right now your flowfile will have id,name,age,dept as attributes to the flowfile.

you can do as many invokehttps as you want after each invoke http extract the content as attribute.

Now you need to prepare a json message use AttribiutestoJSON processor

Change the property

attributes list as

id,name,age,dept

This processor now prepares new json message with all the attributes you listed in the processor.

{ "id": 1, "name": "HCC", "age": 20,"dept": "community"}

Use the json result from Attributestojson processor to store into mongoDB.

AttributesToJSON Configs:-

Re: Composing Json in Nifi

monikanaagardec — Mon, 11 Feb 2019 22:31:42 GMT

@rsalan Siddiqi how did you solve ur prblm ? I have same use case

@rsalan Siddiqi how did you solve this prblm ? I have same use case

Re: Composing Json in Nifi

Siraj — Tue, 24 Dec 2019 12:37:08 GMT

@Shu_ashu / All,

I have a nested JSON (not an array).

I am able to read it using "EvaluateJsonPath" for all the direct attributes (IMEI or Counter in my case). The nested attributes($.Events.EvenType.Name or $.Events.Properties.Latitude) is not recognized and they are getting populated as NULL in my destination table. Can you please advise?

Data Pipeline is as:

GetFile --EvaluateJSONPath -- ConvertJSONtoSQL -- PutSQL

EvaluateJSONPath Property (as attached):

Sample JSON Data:

{

"IMEI":"77777777777777",

"Events":

{

"EventType":{"Name":"EventName","Time":1577184434,"ReceievedTime":1577184434,"MsgId":1493134167},

"Properties":{"ThingUnitId":"779000000000062","Latitude":"24","Longitude":"46"

},"Counter":0

}

Thanks

Re: Composing Json in Nifi

Siraj — Tue, 24 Dec 2019 15:10:06 GMT

Hello All,

The below pipeline solved the issue.

The naming convention between the table and the Property/value in Nifi has to be set same.

If not, it cannot recognize and either it will fail or NULL gets loaded. It is not as simple as expanding the the JSON using SQLServer query!!!

Note: Value data in the EvoluteJSONPath is setup based on the FlattenJSON output. I had the "Seperator" as '_' and "Flattenmode" as 'dot notation' in the FlattenJSON setup.

Re: Composing Json in Nifi

ask_bill_brooks — Mon, 30 Dec 2019 04:37:20 GMT

Hi @Siraj

Looking over your most recent post it appears that you solved your issue. Can you confirm by using the Accept as Solution button here so it can be of assistance to others?

Re: Composing Json in Nifi

Siraj — Mon, 30 Dec 2019 04:42:00 GMT

@ask_bill_brooks ,

Sorry, i am not seeing the accept as solution option in my screen.

Thanks.