Support Questions

Find answers, ask questions, and share your expertise

Consume Amazon ElasticSearch 6.0 documents using NiFi

Hi,

I have a simple use case wherein I have to periodically fetch all records for a particular index from an Elasticsearch cluster into NiFi, while at the same time removing those documents from the ES cluster. But I cannot find the suitable processor(s) for the same.

Amazon Elasticsearch version 6.0 | Apache NiFi version 1.5.0

This might be a silly question but I am just not able to find the correct combination of processors to do this. Thanks!

1 REPLY 1

Super Guru

To fetch all records from a particular index, you should be able to use QueryElasticsearchHttp and ScrollElasticsearchHttp (see the documentation for usage and configuration information). Once they are fetched, you can use EvaluateJsonPath to get the ID into a flow file attribute, then specify that attribute in a PutElasticsearchHttp processor as "Identifier Attribute". Also you'd set "Index Operation" to "delete".

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.