Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi-Kafka: Message retrieval and commit process

NiFi-Kafka: Message retrieval and commit process

Rising Star

Let's say we put 5000 message files (each having 1000 records) into a partition

1) In ConsumeKafka processor, does "Max Poll Records" corresponds to number of records OR the number message files, consumed in a single batch? [Can you please confirm if Max Poll Records is valid property for ConsumeKafka which supports 0.9 version, or it is valid only with ConsumeKafka_0_10 processor, per below?]

http://mirror.fibergrid.in/apache/kafka/0.10.0.0/RELEASE_NOTES.html

https://issues.apache.org/jira/browse/KAFKA-3007

2) How and when does the ConsumeKafka processor initiates the commit? Does it commit as soon the Flowfile is generated and moved to the queue OR if immediate next is a PutHDFS processor, will it commit after that?