Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

[Nifi] Unify ReplaceText output FlowFile into one only record

avatar
Explorer

I have a ReplaceText that generates a Hive query. As an output flowfile, I have several records with the same Hive query, that I use to run a PutHiveQl processor. So that, the PutHiveQL processor is executed several times, but I only need it to run once.

How could I do that? Is there a way to remove all ReplaceText output records but one?

Any help would be really appreciated.

1 ACCEPTED SOLUTION

avatar
Master Guru
@Jose Luis Moreno

Use DetectDuplicate processor, so that if you are having same hive query as this processor will detect that query transfers those flowfiles into duplicate connection and also you can use Age Off Duration property to age off cached flowfiles.

Refer to this link for more details regards to DetectDuplicate processor usage.

-

Another ways would be using ControlRate processor but this processor will release the flowfile that has been waiting in the queue after TimeDuration has been finished.

Refer this link for more details regards to ControlRate processor.

View solution in original post

2 REPLIES 2

avatar
Master Guru
@Jose Luis Moreno

Use DetectDuplicate processor, so that if you are having same hive query as this processor will detect that query transfers those flowfiles into duplicate connection and also you can use Age Off Duration property to age off cached flowfiles.

Refer to this link for more details regards to DetectDuplicate processor usage.

-

Another ways would be using ControlRate processor but this processor will release the flowfile that has been waiting in the queue after TimeDuration has been finished.

Refer this link for more details regards to ControlRate processor.

avatar
Explorer

Age of duration seems to be a straight forward solution.

I thought in using DetectDuplicate, but we need here an external db to achieve that.

Thank you very much for the help, Shu.