Consume Amazon ElasticSearch 6.0 documents using NiFi

[#1]

Hi,

I have a simple use case wherein I have to periodically fetch all records for a particular index from an Elasticsearch cluster into NiFi, while at the same time removing those documents from the ES cluster. But I cannot find the suitable processor(s) for the same.

Amazon Elasticsearch version 6.0 | Apache NiFi version 1.5.0

This might be a silly question but I am just not able to find the correct combination of processors to do this. Thanks!

[#2]

To fetch all records from a particular index, you should be able to use QueryElasticsearchHttp and ScrollElasticsearchHttp (see the documentation for usage and configuration information). Once they are fetched, you can use EvaluateJsonPath to get the ID into a flow file attribute, then specify that attribute in a PutElasticsearchHttp processor as "Identifier Attribute". Also you'd set "Index Operation" to "delete".