question [Nifi] Unify ReplaceText output FlowFile into one only record in Archives of Support Questions (Read Only)

[Nifi] Unify ReplaceText output FlowFile into one only record

jlmoreno — Thu, 13 Dec 2018 19:14:06 GMT

I have a ReplaceText that generates a Hive query. As an output flowfile, I have several records with the same Hive query, that I use to run a PutHiveQl processor. So that, the PutHiveQL processor is executed several times, but I only need it to run once.

How could I do that? Is there a way to remove all ReplaceText output records but one?

Any help would be really appreciated.

Re: [Nifi] Unify ReplaceText output FlowFile into one only record

Shu_ashu — Thu, 13 Dec 2018 21:33:30 GMT

@Jose Luis Moreno

Use DetectDuplicate processor, so that if you are having same hive query as this processor will detect that query transfers those flowfiles into duplicate connection and also you can use Age Off Duration property to age off cached flowfiles.

Refer to this link for more details regards to DetectDuplicate processor usage.

Another ways would be using ControlRate processor but this processor will release the flowfile that has been waiting in the queue after TimeDuration has been finished.

Refer this link for more details regards to ControlRate processor.

Re: [Nifi] Unify ReplaceText output FlowFile into one only record

jlmoreno — Fri, 14 Dec 2018 00:00:23 GMT

Age of duration seems to be a straight forward solution.

I thought in using DetectDuplicate, but we need here an external db to achieve that.

Thank you very much for the help, Shu.