Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Consume Kafka processor not working

Solved Go to solution

Consume Kafka processor not working

New Contributor

consumer-processors.png

Hi,
I have HDF installed on my server. I want consume data from a kafka topic and write it to a file which can then be loaded into HDFS.

The problem is that the ConsumeKafka processor doesnt consume an messages from the topic, I tried different processors and also verified the hostname/ports. I added the group id by listing it from the zookeeper shell i.e. ls /consumers in the zookeeper shell and added shown group id in the processors group id.

THings I have done:

Verified the hostname and ports

Verified that data is consumed using the kafka-console-consumer.sh

Googled a lot!

Any help will be greatly appreciated.

I have attached the pics for reference

Thanks

kafka-consuemr-properties.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Consume Kafka processor not working

hy @Hamza,
if you are using no security protocol like kerberos/tsl then do use kafka processor version less than 0.9 which is getkafka and for those having kerberos security installed must have to use version above than 0.9 like consumekafka 0.10 and above.Its mandatroy.

9 REPLIES 9

Re: Consume Kafka processor not working

Super Guru
@Hamza Rashid

Change the below property value to

Offset Reset

earliest //automatically reset the offset to earliest

And if you are already consumed from the Group then change the

Group ID

nifi-cosumers-1

Run the processor again.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

Re: Consume Kafka processor not working

New Contributor

No effect whatsoever.

Re: Consume Kafka processor not working

New Contributor

Nope, it still doesn't work.

Re: Consume Kafka processor not working

@Hamza Rashid This should be broker hostname and port(default 6667, "listeners" property will have the configured port in your environment) instead of zookeeper port.

92485-screen-shot-2018-09-19-at-125845-pm.png

Re: Consume Kafka processor not working

@Hamza Rashid Did the above mentioned changes helped in addressing your issue?

Re: Consume Kafka processor not working

New Contributor

No sir, no luck yet.

Re: Consume Kafka processor not working

New Contributor

Can you please recommend some resources that might help me in this?

Highlighted

Re: Consume Kafka processor not working

hy @Hamza,
if you are using no security protocol like kerberos/tsl then do use kafka processor version less than 0.9 which is getkafka and for those having kerberos security installed must have to use version above than 0.9 like consumekafka 0.10 and above.Its mandatroy.

Re: Consume Kafka processor not working

New Contributor

Thank you.