question Re: QueryElasticSearchHTTP does not return anything when a record is not found! in Support Questions

QueryElasticSearchHTTP does not return anything when a record is not found!

RB764 — Wed, 18 May 2022 07:06:38 GMT

Hi all,

I am hoping I can get some help with this. I have a process that's sending a set of JSON records to a NiFi flow. Each JSON record in the array is a set of criteria/conditions for which I run QueryElasticSearchHTTP to get matching records from ElasticSearch. I am using a Merge Content processor at the end to merge all of this data and send back the JSON array as response.

It works fine when for example I send 10 JSON criteria records in the array and QueryElasticSearchHTTP is able to find a document in the index for each criteria it receives. It bundles up the 10 resultant records from Elastic and sends the response through. However, if QueryElasticSearchHTTP finds only 8 records matching the criteria, it then waits forever at the Merge Content processor. I realise this is because it has received 10 records initially and split them into 10 flow files. When it looks to merge, it finds only 8 flow files and is waiting for the remaining 2 which never come through. This blocks up the entire flow and nothing works from here on.

Please see picture above. Can anyone help me with a solution to this, please? Thanks in advance.

Re: QueryElasticSearchHTTP does not return anything when a record is not found!

ckumar — Thu, 19 May 2022 12:18:09 GMT

I believe MergeContent Merging Strategy is set to Defragment ? If it is set to Defragment then MergeContent will wait to receive all flow file part of fragment.count. so what happened to the remaining 2 flow files? which relationships they move? failure? and terminated on Processor itself assuming as per screenshot . If you wish to merge with reamaning 8 then Merging Strategy needs to be changed but that could be lead to case where all 8 records can not be merged together all the time since there is no correlation, or if you just want to merge all records irrespective of failure or success , then you need to have failure or success both relationships configured for MergeContent .

Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: QueryElasticSearchHTTP does not return anything when a record is not found!

RB764 — Fri, 20 May 2022 20:20:55 GMT

Thanks for taking the time to respond @ckumar

Merge strategy is set to Defragment and yes it seems to be waiting for the remaining 2 files which never arrive. I have noticed that if a record isn't found in the Elastic index, QueryElasticSearchHTTP processor does not return a failure and so there is nothing returned through any relationship for the missing 2 records.