Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Consume Amazon ElasticSearch 6.0 documents using NiFi

Consume Amazon ElasticSearch 6.0 documents using NiFi


I have a simple use case wherein I have to periodically fetch all records for a particular index from an Elasticsearch cluster into NiFi, while at the same time removing those documents from the ES cluster. But I cannot find the suitable processor(s) for the same.

Amazon Elasticsearch version 6.0 | Apache NiFi version 1.5.0

This might be a silly question but I am just not able to find the correct combination of processors to do this. Thanks!


Re: Consume Amazon ElasticSearch 6.0 documents using NiFi

Super Guru

To fetch all records from a particular index, you should be able to use QueryElasticsearchHttp and ScrollElasticsearchHttp (see the documentation for usage and configuration information). Once they are fetched, you can use EvaluateJsonPath to get the ID into a flow file attribute, then specify that attribute in a PutElasticsearchHttp processor as "Identifier Attribute". Also you'd set "Index Operation" to "delete".

Don't have an account?
Coming from Hortonworks? Activate your account here