Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - slow reading from kafka topic only 100 messages

ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - slow reading from kafka topic only 100 messages

Contributor

NIFI 1.9

HDF 3.4.1

 

nifi ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - slow consuming from kafka topic which got 8 million messages

 

ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - reading only 100 messages at a time 

nifi singlenode 32core * 64 GB memory 

 

 

@MattWho, please advise

 

venkii_0-1600105545409.png

 

 

4 REPLIES 4
Highlighted

Re: ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - slow reading from kafka topic only 100 messages

Contributor

venkii_0-1600108109032.png

 

Highlighted

Re: ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - slow reading from kafka topic only 100 messages

Contributor

@Wynner @pvillard @bbende @MattWho @TimothySpann 

I have single node nifi, where the Kafka topic has 3 partitions but ConsumeKafkaRecord_2_0 though configured with 3 threads on single node nifi, it's pulling only 50 messages or even less.

 

i am assuming though nifi is a single node, as I have configured to use 3 threads based on the partitions on topic, it should consume data. in my case, it's pulling only 30-50 messages per second

 

any pointers would be greatly helpful. thanks!

 

HDF 3.4
nifi 1.9

Highlighted

Re: ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - slow reading from kafka topic only 100 messages

Super Guru

Upgrade to newer, make sure NiFi has 2GB+ of RAM.

 

On my laptop NiFi 1.11.4 is pulling in 1,500 records a second.

 

How large are those Kafka messages?   What format?  example?  Don't use threads for that.

Highlighted

Re: ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - slow reading from kafka topic only 100 messages

Contributor

@TimothySpann thanks for your response.

 

- i assume ConsumeKafkaRecord_2_0 1.9 will consume messages from the Kafka partitions based on "max poll records".in my case I have set it to 100000.  

please let me know if my assumption is wrong.

 

could you also confirm if there are any limitations to pull bulk messages like at least 10K messages in a single poll? with 1.9 nifi 

 

message size is 2 MB,

Avro format

and 3 topics have 2 million messages each 

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here