question SplitJSON doesnt split anything in Support Questions

SplitJSON doesnt split anything

totokogure — Thu, 11 Aug 2022 16:02:44 GMT

Hello,

I'm new to nifi so my problem may be dumb. Sorry if it seems really basic.

I'm trying to use nifi to :

1/ Read a JSON response from a REST GET endpoint.
2/ Parse the JSON and split some JSON array into JSON objects

3/ insert objects in SQL.

I'm stuck at Step 2 because my SplitJSON doesnt return anything. No error as well.

Here is a sample of my JSON :

{ "context":"q%3D%2523all%26q.restriction%3Dclass%253Acr_hashtag%26of%3Djson", "nhits":1000000, "nmatches":1000000, "start":0, "ellql":"#query{nbdocs=26966678, text_relevance.expr=\"@term.score*@proximity+@b\", noproximity, term.score=NO_RANKING}(#and(#true() #category(categories,\"Top/datamodelclass/cr_hashtag\") ))", "executor":null, "estimated":false, "autocorrected":false, "spellCheckSuggestions":{ "origStr":null, "suggestions":[ ] }, "hits":[ { "did":1, "url":"ID=10055&", "buildGroup":"bg0", "source":null, "slice":3, "score":0, "sort":0, "groups":[ ], "metas":[ { "name":"created_by", "type":2, "value":"L37" }, { "name":"created_dttm", "type":2, "value":"2022/07/13 11:29:30" }, { "name":"last_modified_dttm", "type":2, "value":"2022/07/13 11:29:30" }, { "name":"legal_entity_id", "type":2, "value":"100000000000313" }, { "name":"source", "type":2, "value":"consolidation-cbx_bg0" }, { "name":"tags", "type":2, "value":"test10055" }, { "name":"id", "type":2, "value":"10055" }, { "name":"url", "type":2, "value":"ID=10055&" } ] }, { "did":2, "url":"ID=10119&", "buildGroup":"bg0", "source":null, "slice":4, "score":0, "sort":0, "groups":[ ], "metas":[ { "name":"created_by", "type":2, "value":"L37" }, { "name":"created_dttm", "type":2, "value":"2022/07/13 11:29:30" }, { "name":"last_modified_dttm", "type":2, "value":"2022/07/13 11:29:30" }, { "name":"legal_entity_id", "type":2, "value":"100000000000313" }, { "name":"source", "type":2, "value":"consolidation-cbx_bg0" }, { "name":"tags", "type":2, "value":"test10119" }, { "name":"id", "type":2, "value":"10119" }, { "name":"url", "type":2, "value":"ID=10119&" } ] } ] }

I want to split the "hits" array.
This is my nifi config so far. I'm able to get the JSON from the invokeHTTP

Still, the output of the relation "split" is empty... The one of "failure" as well. I have no idea what's going on.

What am i doing wrong?

Thanks for your help

Re: SplitJSON doesnt split anything

araujo — Fri, 12 Aug 2022 05:40:48 GMT

@totokogure ,

I tested this on the same version of NiFi that you're using (1.17.0) and it worked fine for me.

The sample content that you provided was split in 2 flowfiles.

Cheers,

André

Re: SplitJSON doesnt split anything

totokogure — Fri, 12 Aug 2022 12:17:42 GMT

@araujo

Thanks for testing it. I also tried it with a different flow from raw JSON file read through GetFile Processor and managed to split the JSON.

So as we can say it's not because of the SplitJSON processor, doest it mean i have some issue with the input format received from the SplitJSON aka the output of the InvokedHTTP?

Is there any chance that SplitJSON doesnt recognise the http response as a JSON and skips it?

Re: SplitJSON doesnt split anything

totokogure — Fri, 12 Aug 2022 13:06:44 GMT

I managed to make it work by introducing a evaluateJsonPath between invokehttp and splitjson :

Config of evaluateJSONPath:

Config of SplitJSON:

Is it supposed to work like this?

Re: SplitJSON doesnt split anything

araujo — Sun, 14 Aug 2022 22:58:01 GMT

@totokogure ,

The EvaluateJsonPath that you added is extracting the $.hits array and storing it as an attribute. The next processor (SplitJson), does not even use that attribute, though. It extracts $.hits again from the flowfile content. The EvaluateJsonPath in this flow should be unnecessary. If you connected InvokeHTTP to SplitJson, things should work correctly.

Cheers,

André