Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NIFI PuElasticsearchHttp Update - how does it work

Highlighted

NIFI PuElasticsearchHttp Update - how does it work

New Contributor

I need to push json data to Elasticsearch. I am using PuElasticsearchHttp processor. I have a scenario where the json data can hold insert, update or upsert. I tried to use IndexOperation = index for all these operations, then the ES index is getting updated with new value. But then I use IndexOperation = update it fails with "update requires valid identifier value with rom a flow file attribute.

3 REPLIES 3
Highlighted

Re: NIFI PuElasticsearchHttp Update - how does it work

Super Guru

From the PutElasticsearchHttp documentation: "The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is "index", this property may be left empty or evaluate to an empty value, in which case the document's identifier will be auto-generated by Elasticsearch. For all other Index Operations, the attribute must evaluate to a non-empty value."

So for "update", Elasticsearch needs to know which document to update. As a general rule, if the document has an identifier, you should use EvaluateJsonPath or something to extract the identifier into an attribute (let's call it "es.id" for this example), then in PutElasticsearchHttp you can set the Identifier Attribute property to "es.id".

Highlighted

Re: NIFI PuElasticsearchHttp Update - how does it work

New Contributor

Thanks a lot Matt. I did figure it out finally. I introduced EvalJsonPath to extract the attribute and assign it to identifier. In case of single attribute, it worked. In my scenario i have multiple fields in my json - field1 and field2. I need to construct the _id field and send it to ES. I have used EvalJsonPath --> UpdateAttribute --> PutElastiSearchHttp. UpdateAttribute key is configured as _id and value is ${field1}_${field2}. During runtime I can see the attribute value getting constructed for example 111111_SmartAccount_700001, but I do not see the ES index getting populated. PutElastiSearchHttp processor is dropping this value. Any pointers?

Highlighted

Re: NIFI PuElasticsearchHttp Update - how does it work

New Contributor

@mburgess @ljonnavi  Thank you!

Don't have an account?
Coming from Hortonworks? Activate your account here