Skip to main content
totokogure
Silver ⭐️⭐️
August 11, 2022
Solved

SplitJSON doesnt split anything

Hello,

I'm new to nifi so my problem may be dumb. Sorry if it seems really basic.

I'm trying to use nifi to :

1/ Read a JSON response from a REST GET endpoint.
2/ Parse the JSON and split some JSON array into JSON objects

3/ insert objects in SQL.

I'm stuck at Step 2 because my SplitJSON doesnt return anything. No error as well.

Here is a sample of my JSON :

 

{
"context":"q%3D%2523all%26q.restriction%3Dclass%253Acr_hashtag%26of%3Djson",
"nhits":1000000,
"nmatches":1000000,
"start":0,
"ellql":"#query{nbdocs=26966678, text_relevance.expr=\"@term.score*@proximity+@b\", noproximity, term.score=NO_RANKING}(#and(#true() #category(categories,\"Top/datamodelclass/cr_hashtag\") ))",
"executor":null,
"estimated":false,
"autocorrected":false,
"spellCheckSuggestions":{
"origStr":null,
"suggestions":[

]
},
"hits":[
{
"did":1,
"url":"ID=10055&",
"buildGroup":"bg0",
"source":null,
"slice":3,
"score":0,
"sort":0,
"groups":[

],
"metas":[
{
"name":"created_by",
"type":2,
"value":"L37"
},
{
"name":"created_dttm",
"type":2,
"value":"2022/07/13 11:29:30"
},
{
"name":"last_modified_dttm",
"type":2,
"value":"2022/07/13 11:29:30"
},
{
"name":"legal_entity_id",
"type":2,
"value":"100000000000313"
},
{
"name":"source",
"type":2,
"value":"consolidation-cbx_bg0"
},
{
"name":"tags",
"type":2,
"value":"test10055"
},
{
"name":"id",
"type":2,
"value":"10055"
},
{
"name":"url",
"type":2,
"value":"ID=10055&"
}
]
},
{
"did":2,
"url":"ID=10119&",
"buildGroup":"bg0",
"source":null,
"slice":4,
"score":0,
"sort":0,
"groups":[

],
"metas":[
{
"name":"created_by",
"type":2,
"value":"L37"
},
{
"name":"created_dttm",
"type":2,
"value":"2022/07/13 11:29:30"
},
{
"name":"last_modified_dttm",
"type":2,
"value":"2022/07/13 11:29:30"
},
{
"name":"legal_entity_id",
"type":2,
"value":"100000000000313"
},
{
"name":"source",
"type":2,
"value":"consolidation-cbx_bg0"
},
{
"name":"tags",
"type":2,
"value":"test10119"
},
{
"name":"id",
"type":2,
"value":"10119"
},
{
"name":"url",
"type":2,
"value":"ID=10119&"
}
]
}
]
}

 


I want to split the "hits" array.
This is my nifi config so far. I'm able to get the JSON from the invokeHTTP

totokogure_0-1660233515123.png

totokogure_2-1660233609266.png

totokogure_1-1660233542248.png

Still, the output of the relation "split" is empty... The one of "failure" as well. I have no idea what's going on.

What am i doing wrong?

Thanks for your help

araujo
araujoAnswer
Platinum ⭐️⭐️⭐️
August 14, 2022

@totokogure ,

 

The EvaluateJsonPath that you added is extracting the $.hits array and storing it as an attribute. The next processor (SplitJson), does not even use that attribute, though. It extracts $.hits again from the flowfile content. The EvaluateJsonPath in this flow should be unnecessary. If you connected InvokeHTTP to SplitJson, things should work correctly.

 

Cheers,

André

-- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.

4 replies

araujo
Platinum ⭐️⭐️⭐️
Platinum ⭐️⭐️⭐️
August 12, 2022

@totokogure ,

 

I tested this on the same version of NiFi that you're using (1.17.0) and it worked fine for me.

The sample content that you provided was split in 2 flowfiles.

 

Cheers,

André

 

Translate
-- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.
totokogure
Silver ⭐️⭐️
August 12, 2022

@araujo 

Thanks for testing it. I also tried it with a different flow from raw JSON file read through GetFile Processor and managed to split the  JSON.

So as we can say it's not because of the SplitJSON processor, doest it mean i have some issue with the input format received from the SplitJSON aka the output of the InvokedHTTP?

Is there any chance that SplitJSON doesnt recognise the http response as a JSON and skips it?

totokogure_0-1660306472351.png

 

 




Translate
totokogure
Silver ⭐️⭐️
August 12, 2022

I managed to make it work by introducing a evaluateJsonPath between invokehttp and splitjson : 

 

totokogure_0-1660309523708.png

 

Config of evaluateJSONPath:

totokogure_1-1660309539268.png

Config of SplitJSON:

totokogure_2-1660309585805.png

 

Is it supposed to work like this?

 

Translate
araujo
Platinum ⭐️⭐️⭐️
araujoAnswer
Platinum ⭐️⭐️⭐️
August 14, 2022

@totokogure ,

 

The EvaluateJsonPath that you added is extracting the $.hits array and storing it as an attribute. The next processor (SplitJson), does not even use that attribute, though. It extracts $.hits again from the flowfile content. The EvaluateJsonPath in this flow should be unnecessary. If you connected InvokeHTTP to SplitJson, things should work correctly.

 

Cheers,

André

Translate
-- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.