question Re: Consume Kafka processor not working in Archives of Support Questions (Read Only)

Consume Kafka processor not working

hamza_rashid — Tue, 18 Sep 2018 19:49:04 GMT

Hi,
I have HDF installed on my server. I want consume data from a kafka topic and write it to a file which can then be loaded into HDFS.

The problem is that the ConsumeKafka processor doesnt consume an messages from the topic, I tried different processors and also verified the hostname/ports. I added the group id by listing it from the zookeeper shell i.e. ls /consumers in the zookeeper shell and added shown group id in the processors group id.

THings I have done:

Verified the hostname and ports

Verified that data is consumed using the kafka-console-consumer.sh

Googled a lot!

Any help will be greatly appreciated.

I have attached the pics for reference

Thanks

kafka-consuemr-properties.png

Re: Consume Kafka processor not working

Shu_ashu — Tue, 18 Sep 2018 20:29:31 GMT

@Hamza Rashid

Change the below property value to

Offset Reset

earliest //automatically reset the offset to earliest

And if you are already consumed from the Group then change the

Group ID

nifi-cosumers-1

Run the processor again.

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

Re: Consume Kafka processor not working

hamza_rashid — Wed, 19 Sep 2018 14:11:25 GMT

Nope, it still doesn't work.

Re: Consume Kafka processor not working

sandyy006 — Sun, 18 Aug 2019 05:43:17 GMT

@Hamza Rashid This should be broker hostname and port(default 6667, "listeners" property will have the configured port in your environment) instead of zookeeper port.

Re: Consume Kafka processor not working

sandyy006 — Mon, 24 Sep 2018 23:37:11 GMT

@Hamza Rashid Did the above mentioned changes helped in addressing your issue?

Re: Consume Kafka processor not working

hamza_rashid — Tue, 25 Sep 2018 01:41:46 GMT

No sir, no luck yet.

Re: Consume Kafka processor not working

hamza_rashid — Tue, 25 Sep 2018 16:15:44 GMT

Can you please recommend some resources that might help me in this?

Re: Consume Kafka processor not working

hamza_rashid — Tue, 25 Sep 2018 16:16:23 GMT

No effect whatsoever.

Re: Consume Kafka processor not working

Muhammad_Waqas — Mon, 15 Oct 2018 13:26:17 GMT

hy @Hamza,
if you are using no security protocol like kerberos/tsl then do use kafka processor version less than 0.9 which is getkafka and for those having kerberos security installed must have to use version above than 0.9 like consumekafka 0.10 and above.Its mandatroy.

Re: Consume Kafka processor not working

hamza_rashid — Mon, 15 Oct 2018 13:27:12 GMT

Thank you.