Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

SOLVED Go to solution
Highlighted

ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Expert Contributor

Hello,

I have enable kerberos on cluster. I can successfully connect to consumer through kafka-konsole-consumer.

But when i try to connect to topic through NiFi ConsumeKafka processor i have error:

WARN [Timer-Driven Process Thread-8] o.a.n.p.kafka.pubsub.ConsumeKafka_1_0 ConsumeKafka_1_0[id=504e5811-0168-1000-0000-000024c83cc5] Was interrupted while trying to communicate with Kafka with lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@51911c24. Will roll back session and discard any partially received data.   WARN [kafka-kerberos-refresh-thread-nifi/*@*] o.a.k.c.security.kerberos.KerberosLogin [Principal=nifi/*@*]: TGT renewal thread has been interrupted and will exit.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Mentor

@ Ruslan Fialkovsky

Can you log on both cluster as zookeeper run kinit then

$ /usr/hdp/current/zookeeper-server/bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] getAcl /config/topics

Compare the outputs there should be a difference between the 2 cluster

sample output

[zk: localhost:2181(CONNECTED) 1] getAcl /config/topics
'world,'anyone 
: r 
'sasl,'kafka 
: cdrwa

And on the other

[zk: localhost:2181(CONNECTED) 1] getAcl /config/topics 
'world,'anyone 
: cdrwa

If that's the case have a look at how to set the correct ACL for the /config/topics in this HCC document

Please let me know

16 REPLIES 16

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Expert Contributor

I can't connect with --bootstrap-server key, only with --zookeeper works. Obviously "ConsumeKafka processor" use "bootstrap" mod.

Can i use zookeeper with "ConsumeKafka processor" or how can i debug why i can't to connect direct to broker?

kafka-console-consumer.sh --bootstrap-server server:6667 --topic test5 --from-beginning --security-protocol SASL_PLAINTEXT

don't work
kafka-console-consumer.sh --zookeeper server:2181--topic test5 --from-beginning --security-protocol SASL_PLAINTEXT

work fine

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Mentor

@ Ruslan Fialkovsky

Is your nifi part of the kerberized cluster or is a separate HDF cluster? What are versions of the components NIF,HDP etc ?

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Expert Contributor

@Geoffrey Shelton Okot

I use HDF on HDP as one cluster, yes is kerberized. Ambari-2.6.2.2, HDP-2.6.5, HDF-3.1.2.

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Mentor

@ Ruslan Fialkovsky

Are you doing some lab, this looks familiar can you share the link, I want to see if I can reproduce it on my cluster?

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Expert Contributor

@Geoffrey Shelton Okot

It's real case at my work. I configured Test cluster from Hortonworks documentation and everything works good but at the same time Development cluster with same configuration doesn't work and i don't understand why it is. I don't have lab or some special instruction but i can to show you anything my config files or screenshots. Is it maybe some problem with zookeeper?

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Mentor

@ Ruslan Fialkovsky

Can you log on both cluster as zookeeper run kinit then

$ /usr/hdp/current/zookeeper-server/bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] getAcl /config/topics

Compare the outputs there should be a difference between the 2 cluster

sample output

[zk: localhost:2181(CONNECTED) 1] getAcl /config/topics
'world,'anyone 
: r 
'sasl,'kafka 
: cdrwa

And on the other

[zk: localhost:2181(CONNECTED) 1] getAcl /config/topics 
'world,'anyone 
: cdrwa

If that's the case have a look at how to set the correct ACL for the /config/topics in this HCC document

Please let me know

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Expert Contributor

@Geoffrey Shelton Okot

Working Test:

[zk: localhost:2181(CONNECTED) 0] getAcl /config/topics

'world,'anyone

: r

'sasl,'kafka

: cdrwa

[zk: localhost:2181(CONNECTED) 1]

Not working Dev:

[zk: localhost:2181(CONNECTED) 0] getAcl /config/topics

'world,'anyone

: cdrwa

[zk: localhost:2181(CONNECTED) 1]

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Mentor

@ Ruslan Fialkovsky

Now you see that on DEV world: everyone has cdrwa while for kafka there is no entry so you will need to setAcl for kafka

 setAcl /config/topics world:anyone:cdrwa,sasl:kafka:cdrwa

Did you grasp how to use the using superDigest to become a Zookeeper superuser, that's like the zookeeper root user who can add/change the ACL's ,let me know if you need help on that.

Re: ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Expert Contributor

@Geoffrey Shelton Okot

I set next acl on Dev cluster:

[zk: localhost:2181(CONNECTED) 1] getAcl /brokers
'world,'anyone
: cdrwa
'sasl,'kafka
: cdrwa
[zk: localhost:2181(CONNECTED) 2] getAcl /controller
'world,'anyone
: r
'sasl,'kafka
: cdrwa
[zk: localhost:2181(CONNECTED) 3] getAcl /config
'world,'anyone
: cdrwa
'sasl,'kafka
: cdrwa
[zk: localhost:2181(CONNECTED) 4] getAcl /config/topics
'world,'anyone
: cdrwa
'sasl,'kafka
: cdrwa

but kafka-console-consumer.sh --bootstrap-server still not work.