Support Questions
Find answers, ask questions, and share your expertise

Count flowfiles from Kafka Input and route them as per count

Highlighted

Count flowfiles from Kafka Input and route them as per count

New Contributor

Well I am new to Apache nifi, and was struggling with below scenario.

  1. Take input from Kafka topic using `ConsumeKafka_1_0` 
  2. Somehow add counter based on attribute which was extracted using `EvaluateJsonPathFields`
    1. Example  - serviceName = $.fields.service_name 
  3. Using `UpdateAttribute` I am again setting serviceName = ${serviceName:replaceNull("NoServiceName")}
  4. And then finally using `RouteOnAttribute` I am sending data as per properties to ControlRate and then finally to PublishKafka.

Not sure, how I can do something similar to - https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Processors/Ag... and aggregate the flowfiles using attribute `serviceName` ..it seems `MergeContent` processor can do same but not on window.

Also, once I am able to get count for all the serviceName = example 1 as 2000 ... I want to check this number based on some predefined values set in limits database or file.

Thus serviceName.Count > 2000  --> send to s3 else you can Publish to Kafka 

Any help is really appreciated.